Xanathorn 2007-07-09 18:12:33 | I have been watching the behaviour of the latest client for a while now and every time there's a new best which was computed with client version 4.43c (like Herb's for example) my client just stalls sometimes for weeks on end until a 'lucky' optimization gives a breakthrough which my client can produce better results on. Anyone else noticed this behaviour? Sometimes I have the feeling this hinders progress and is the reason why on some occasions a lattice is just 'stuck' for a long while like last time for a few months, and in my eyes computing power is just wasted. I might have to go back to v4.43c to just check how that client handles the best muon percentages optimizations.. (or anyone can give a reason why this behaviour is, I'm not so into math . |
RGtx 2007-07-10 00:00:21 | Your observation has been noted before. On my single machine, I manually append the best v4.43d sample results from stephens update to my DecayRotB.txt sample file (Auto updating turned off). Probably not a lot of use to those with many machines, but it works for me. |
waffleironhead 2007-07-10 00:02:30 | I personally would go in and prune out any results from 4.43c and then turn off the sample files.(takes own advice) |
Herb[Romulus2] 2007-07-10 09:44:45 | Reason 1: the c version computes better! Reproducable, you can check it yourself do you own test queue and let it run repitevely with both versions, no matter what lattice, c is better. Reason 2: the strategy distribution which keeps me alsways high, TrialType ratios: Mutate=9;Crossover=3;Interpolate=3;Extrapolate=3;MuSpherical=3;MuOne=3;Extreme=1;LocalGrad=3;TopoSmooth=2; Reason 3: I'm cheating, from time to time I tweak a little bit the queue |
Stephen Brooks 2007-07-10 12:19:35 | I just had a look at (most of) the changes in the code between those two versions - there weren't many that could cause a score change. There was one where it's possible an extra particle was being read in from the input in c whereas it wasn't in d, but even that doesn't seem so likely. I think I'll put in a filter to only put results from v4.43d in the sample files, see if that improves things. |
RGtx 2007-07-10 12:54:02 | I have just run, Herb[Romulus2}'s best v4.43c result of 0.998112 (average of 5 runs) on the v4.43d client, getting: (0.990956 + 0.953544 + 0.987077 + 0.983856 + 1.021047)/5 = 0.987296 (Muon%). |
Stephen Brooks 2007-07-10 12:57:06 | Doesn't look a lon way off. What happens if you do exactly the same again? Does the result stay the same? |
RGtx 2007-07-10 13:01:10 | With the fixed seed values, the results will be identical. |
RGtx 2007-07-10 13:04:52 | If you could email me an archive copy of the v4.43c client, I could check whether the distribution of results over five or more runs are alike. |
Stephen Brooks 2007-07-10 13:48:56 | Well, having the fixed seed should mean they are identical, I just wanted to double-check. |
RGtx 2007-07-10 13:56:01 | I have performed this test in the past, and each trial does produce the same results, but I shall give it one more go. |
Stephen Brooks 2007-07-10 14:05:08 | Well, it's probably OK then. I've e-mailed you v4.43c so you can try and compare that. I'd be kind of interested in seeing the results too (whether it's like a different random seed, or there's a systematic difference). Especially as I can't see _that_ much that could have caused the change. I made the collision detection stricter at one point but that doesn't seem to have happened between those two versions. |
RGtx 2007-07-10 15:43:18 | Here are the two queue.txt files for v.4.43c and v4.43d respectively, for the identical design: d1l=068;d2l=533;d3l=000;d4l=546;d5l=116;pd1=015;pd10=149;pd11=005;pd12=000;pd13=443;pd14=972;pd15=000;pd16=330;pd17=665;pd18=006;pd19=000;pd2=001; pd20=007;pd21=000;pd22=995;pd23=380;pd24=958;pd25=389;pd26=082;pd27=363;pd28=594;pd29=021;pd3=864;pd30=586;pd31=093;pd32=267;pd33=781;pd34=012;pd35=618; pd36=683;pd37=002;pd38=317;pd39=000;pd4=000;pd40=827;pd5=001;pd6=292;pd7=036;pd8=005;pd9=004;prf10p=038;prf10v=164;prf11p=998;prf11v=321;prf12p=277; prf12v=869;prf13p=993;prf13v=361;prf14p=212;prf14v=997;prf15p=415;prf15v=998;prf16p=413;prf16v=793;prf17p=011;prf17v=411;prf18p=023;prf18v=463; prf19p=192;prf19v=780;prf1p=711;prf1v=514;prf20p=026;prf20v=213;prf21p=289;prf21v=868;prf22p=214;prf22v=999;prf23p=105;prf23v=991;prf24p=031;prf24v=120; prf25p=956;prf25v=036;prf26p=196;prf26v=956;prf27p=143;prf27v=861;prf28p=287;prf28v=792;prf29p=224;prf29v=993;prf2p=457;prf2v=729;prf30p=169;prf30v=992; prf31p=225;prf31v=995;prf32p=164;prf32v=997;prf33p=253;prf33v=889;prf34p=194;prf34v=996;prf35p=121;prf35v=987;prf36p=249;prf36v=995;prf37p=222; prf37v=999;prf38p=996;prf38v=000;prf39p=998;prf39v=114;prf3p=043;prf3v=025;prf40p=262;prf40v=993;prf4p=973;prf4v=024;prf5p=112;prf5v=616;prf6p=301; prf6v=424;prf7p=613;prf7v=927;prf8p=918;prf8v=129;prf9p=412;prf9v=983;ps10f=784;ps10l=943;ps11f=000;ps11l=000;ps12f=822;ps12l=547;ps13f=119;ps13l=370; ps14f=217;ps14l=803;ps15f=226;ps15l=532;ps16f=104;ps16l=250;ps17f=108;ps17l=319;ps18f=678;ps18l=539;ps19f=993;ps19l=172;ps1f=807;ps1l=526;ps20f=698; ps20l=373;ps21f=206;ps21l=887;ps22f=000;ps22l=095;ps23f=994;ps23l=236;ps24f=004;ps24l=446;ps25f=048;ps25l=367;ps26f=989;ps26l=144;ps27f=749;ps27l=958; ps28f=996;ps28l=060;ps29f=000;ps29l=025;ps2f=100;ps2l=617;ps30f=204;ps30l=997;ps31f=098;ps31l=736;ps32f=000;ps32l=321;ps33f=004;ps33l=129;ps34f=798; ps34l=589;ps35f=830;ps35l=673;ps36f=038;ps36l=153;ps37f=200;ps37l=663;ps38f=079;ps38l=196;ps39f=943;ps39l=398;ps3f=014;ps3l=134;ps40f=995;ps40l=994; ps4f=000;ps4l=041;ps5f=164;ps5l=244;ps6f=129;ps6l=266;ps7f=096;ps7l=102;ps8f=089;ps8l=169;ps9f=976;ps9l=019;s1f=998;s1l=709;s2f=996;s2l=450;s2r=574; s3f=999;s3l=013;s3r=868;s4f=184;s4l=626;s4r=653;s5f=104;s5l=667;s5r=709;s6f=980;s6l=582;s6r=906;tantalumrodr=000;tantalumrodz=455;#queued=1;#runs=5; 1.000595,0.976746,1.008544,0.994988,1.000125 (579.4 Mpts) [v4.43c] <DecayRotB> {E266F6A9312B25BE523D366D} d1l=068;d2l=533;d3l=000;d4l=546;d5l=116;pd1=015;pd10=149;pd11=005;pd12=000;pd13=443;pd14=972;pd15=000;pd16=330;pd17=665;pd18=006;pd19=000;pd2=001; pd20=007;pd21=000;pd22=995;pd23=380;pd24=958;pd25=389;pd26=082;pd27=363;pd28=594;pd29=021;pd3=864;pd30=586;pd31=093;pd32=267;pd33=781;pd34=012;pd35=618; pd36=683;pd37=002;pd38=317;pd39=000;pd4=000;pd40=827;pd5=001;pd6=292;pd7=036;pd8=005;pd9=004;prf10p=038;prf10v=164;prf11p=998;prf11v=321;prf12p=277; prf12v=869;prf13p=993;prf13v=361;prf14p=212;prf14v=997;prf15p=415;prf15v=998;prf16p=413;prf16v=793;prf17p=011;prf17v=411;prf18p=023;prf18v=463; prf19p=192;prf19v=780;prf1p=711;prf1v=514;prf20p=026;prf20v=213;prf21p=289;prf21v=868;prf22p=214;prf22v=999;prf23p=105;prf23v=991;prf24p=031;prf24v=120; prf25p=956;prf25v=036;prf26p=196;prf26v=956;prf27p=143;prf27v=861;prf28p=287;prf28v=792;prf29p=224;prf29v=993;prf2p=457;prf2v=729;prf30p=169;prf30v=992; prf31p=225;prf31v=995;prf32p=164;prf32v=997;prf33p=253;prf33v=889;prf34p=194;prf34v=996;prf35p=121;prf35v=987;prf36p=249;prf36v=995;prf37p=222; prf37v=999;prf38p=996;prf38v=000;prf39p=998;prf39v=114;prf3p=043;prf3v=025;prf40p=262;prf40v=993;prf4p=973;prf4v=024;prf5p=112;prf5v=616;prf6p=301; prf6v=424;prf7p=613;prf7v=927;prf8p=918;prf8v=129;prf9p=412;prf9v=983;ps10f=784;ps10l=943;ps11f=000;ps11l=000;ps12f=822;ps12l=547;ps13f=119;ps13l=370; ps14f=217;ps14l=803;ps15f=226;ps15l=532;ps16f=104;ps16l=250;ps17f=108;ps17l=319;ps18f=678;ps18l=539;ps19f=993;ps19l=172;ps1f=807;ps1l=526;ps20f=698; ps20l=373;ps21f=206;ps21l=887;ps22f=000;ps22l=095;ps23f=994;ps23l=236;ps24f=004;ps24l=446;ps25f=048;ps25l=367;ps26f=989;ps26l=144;ps27f=749;ps27l=958; ps28f=996;ps28l=060;ps29f=000;ps29l=025;ps2f=100;ps2l=617;ps30f=204;ps30l=997;ps31f=098;ps31l=736;ps32f=000;ps32l=321;ps33f=004;ps33l=129;ps34f=798; ps34l=589;ps35f=830;ps35l=673;ps36f=038;ps36l=153;ps37f=200;ps37l=663;ps38f=079;ps38l=196;ps39f=943;ps39l=398;ps3f=014;ps3l=134;ps40f=995;ps40l=994; ps4f=000;ps4l=041;ps5f=164;ps5l=244;ps6f=129;ps6l=266;ps7f=096;ps7l=102;ps8f=089;ps8l=169;ps9f=976;ps9l=019;s1f=998;s1l=709;s2f=996;s2l=450;s2r=574; s3f=999;s3l=013;s3r=868;s4f=184;s4l=626;s4r=653;s5f=104;s5l=667;s5r=709;s6f=980;s6l=582;s6r=906;tantalumrodr=000;tantalumrodz=455;#queued=1;#runs=5; 0.990956,0.953544,0.987077,0.983856,0.991417 (579.5 Mpts) [v4.43d] <DecayRotB> {B96D3A0C7410C96B78BF4230} ...1.000595,0.976746,1.008544,0.994988,1.000125 (579.4 Mpts) [v4.43c] <DecayRotB> {E266F6A9312B25BE523D366D} ...0.990956,0.953544,0.987077,0.983856,0.991417 (579.5 Mpts) [v4.43d] <DecayRotB> {B96D3A0C7410C96B78BF4230} [My assumption that the final result is the simple average of five runs is obviously wrong]. |
Stephen Brooks 2007-07-10 16:14:30 | It's actually the average of the central three of the five (i.e. excluding the highest and lowest). That result is pretty interesting because the "low" runs are in the same places in both sequences, so it looks like systematic difference... I wonder what it could be? |
RGtx 2007-07-10 16:18:43 | For a number of months the v4,43d client dominated the highest results, I shall repeat this test on an earlier result. |
Stephen Brooks 2007-07-10 16:19:55 | I just found the samplefiles generator was uploading blanks instead of all-v4.43d files. I think it's fixed now. |
RGtx 2007-07-10 20:05:03 | Here is a counterexample, demolishing Herb[Romulus]'s assertion that the V4.43c client gives higher valued results than those from the v4.43d client: d1l=361;d2l=427;d3l=000;d4l=563;pd1=000;pd10=156;pd11=000;pd12=000;pd13=425;pd14=958;pd15=000;pd16=322;pd17=765;pd18=315;pd19=000;pd2=002;pd20=000;pd21=000;pd22=949;pd23=413;pd24=739;pd25=391;pd26=127;pd27=573;pd28=588;pd29=000;pd3=856;pd30=755;pd31=029;pd32=288;pd33=729;pd34=002;pd35=586;pd36=817;pd37=051;pd38=669;pd39=000;pd4=424;pd40=786;pd5=000;pd6=000;pd7=055;pd8=022;pd9=000;prf10p=000;prf10v=198;prf11p=805;prf11v=586;prf12p=320;prf12v=873;prf13p=982;prf13v=921;prf14p=232;prf14v=000;prf15p=240;prf15v=637;prf16p=000;prf16v=766;prf17p=000;prf17v=324;prf18p=000;prf18v=438;prf19p=461;prf19v=833;prf1p=646;prf1v=457;prf20p=008;prf20v=201;prf21p=257;prf21v=713;prf22p=106;prf22v=000;prf23p=142;prf23v=990;prf24p=012;prf24v=084;prf25p=998;prf25v=301;prf26p=009;prf26v=000;prf27p=137;prf27v=839;prf28p=336;prf28v=744;prf29p=287;prf29v=609;prf2p=489;prf2v=591;prf30p=286;prf30v=345;prf31p=387;prf31v=811;prf32p=303;prf32v=704;prf33p=244;prf33v=867;prf34p=077;prf34v=341;prf35p=115;prf35v=999;prf36p=258;prf36v=999;prf37p=000;prf37v=554;prf38p=970;prf38v=054;prf39p=984;prf39v=127;prf3p=366;prf3v=012;prf40p=075;prf40v=644;prf4p=985;prf4v=000;prf5p=013;prf5v=613;prf6p=312;prf6v=280;prf7p=034;prf7v=944;prf8p=886;prf8v=923;prf9p=393;prf9v=849;ps10f=746;ps10l=922;ps11f=167;ps11l=000;ps12f=799;ps12l=597;ps13f=096;ps13l=382;ps14f=181;ps14l=836;ps15f=256;ps15l=443;ps16f=131;ps16l=274;ps17f=126;ps17l=282;ps18f=652;ps18l=464;ps19f=984;ps19l=202;ps1f=574;ps1l=499;ps20f=580;ps20l=419;ps21f=203;ps21l=836;ps22f=000;ps22l=020;ps23f=981;ps23l=270;ps24f=000;ps24l=478;ps25f=017;ps25l=350;ps26f=933;ps26l=083;ps27f=738;ps27l=991;ps28f=876;ps28l=000;ps29f=004;ps29l=020;ps2f=086;ps2l=609;ps30f=188;ps30l=865;ps31f=075;ps31l=812;ps32f=000;ps32l=307;ps33f=145;ps33l=109;ps34f=781;ps34l=591;ps35f=758;ps35l=631;ps36f=066;ps36l=127;ps37f=222;ps37l=699;ps38f=022;ps38l=149;ps39f=901;ps39l=425;ps3f=007;ps3l=112;ps40f=987;ps40l=999;ps4f=013;ps4l=008;ps5f=142;ps5l=357;ps6f=213;ps6l=187;ps7f=136;ps7l=072;ps8f=076;ps8l=157;ps9f=822;ps9l=000;s1f=954;s1l=581;s2f=978;s2l=498;s2r=654;s3f=999;s3l=000;s3r=753;s4f=241;s4l=601;s4r=698;s5f=129;s5l=888;s5r=602;tantalumrodr=000;tantalumrodz=342;#queued=1;#runs=5; 0.524439,0.522827,0.517182,0.517974,0.510776 (424.9 Mpts) [v4.43c] <DecayRotB> {5A07C87C54754F23ED466994} d1l=361;d2l=427;d3l=000;d4l=563;pd1=000;pd10=156;pd11=000;pd12=000;pd13=425;pd14=958;pd15=000;pd16=322;pd17=765;pd18=315;pd19=000;pd2=002;pd20=000;pd21=000;pd22=949;pd23=413;pd24=739;pd25=391;pd26=127;pd27=573;pd28=588;pd29=000;pd3=856;pd30=755;pd31=029;pd32=288;pd33=729;pd34=002;pd35=586;pd36=817;pd37=051;pd38=669;pd39=000;pd4=424;pd40=786;pd5=000;pd6=000;pd7=055;pd8=022;pd9=000;prf10p=000;prf10v=198;prf11p=805;prf11v=586;prf12p=320;prf12v=873;prf13p=982;prf13v=921;prf14p=232;prf14v=000;prf15p=240;prf15v=637;prf16p=000;prf16v=766;prf17p=000;prf17v=324;prf18p=000;prf18v=438;prf19p=461;prf19v=833;prf1p=646;prf1v=457;prf20p=008;prf20v=201;prf21p=257;prf21v=713;prf22p=106;prf22v=000;prf23p=142;prf23v=990;prf24p=012;prf24v=084;prf25p=998;prf25v=301;prf26p=009;prf26v=000;prf27p=137;prf27v=839;prf28p=336;prf28v=744;prf29p=287;prf29v=609;prf2p=489;prf2v=591;prf30p=286;prf30v=345;prf31p=387;prf31v=811;prf32p=303;prf32v=704;prf33p=244;prf33v=867;prf34p=077;prf34v=341;prf35p=115;prf35v=999;prf36p=258;prf36v=999;prf37p=000;prf37v=554;prf38p=970;prf38v=054;prf39p=984;prf39v=127;prf3p=366;prf3v=012;prf40p=075;prf40v=644;prf4p=985;prf4v=000;prf5p=013;prf5v=613;prf6p=312;prf6v=280;prf7p=034;prf7v=944;prf8p=886;prf8v=923;prf9p=393;prf9v=849;ps10f=746;ps10l=922;ps11f=167;ps11l=000;ps12f=799;ps12l=597;ps13f=096;ps13l=382;ps14f=181;ps14l=836;ps15f=256;ps15l=443;ps16f=131;ps16l=274;ps17f=126;ps17l=282;ps18f=652;ps18l=464;ps19f=984;ps19l=202;ps1f=574;ps1l=499;ps20f=580;ps20l=419;ps21f=203;ps21l=836;ps22f=000;ps22l=020;ps23f=981;ps23l=270;ps24f=000;ps24l=478;ps25f=017;ps25l=350;ps26f=933;ps26l=083;ps27f=738;ps27l=991;ps28f=876;ps28l=000;ps29f=004;ps29l=020;ps2f=086;ps2l=609;ps30f=188;ps30l=865;ps31f=075;ps31l=812;ps32f=000;ps32l=307;ps33f=145;ps33l=109;ps34f=781;ps34l=591;ps35f=758;ps35l=631;ps36f=066;ps36l=127;ps37f=222;ps37l=699;ps38f=022;ps38l=149;ps39f=901;ps39l=425;ps3f=007;ps3l=112;ps40f=987;ps40l=999;ps4f=013;ps4l=008;ps5f=142;ps5l=357;ps6f=213;ps6l=187;ps7f=136;ps7l=072;ps8f=076;ps8l=157;ps9f=822;ps9l=000;s1f=954;s1l=581;s2f=978;s2l=498;s2r=654;s3f=999;s3l=000;s3r=753;s4f=241;s4l=601;s4r=698;s5f=129;s5l=888;s5r=602;tantalumrodr=000;tantalumrodz=342;#queued=1;#runs=5; 0.543470,0.523723,0.521832,0.535450,0.525793 (424.9 Mpts) [v4.43d] <DecayRotB> {7EB9F16521D1F557D72D1174} The relative distribution of the results should also give Stephen something to think about. |
Stephen Brooks 2007-07-19 12:22:26 | Well now that the samplefiles only contain 4.43d results, progress seems pretty smooth on these optimisations, though it was also quite regular before. What confuses me a little about these Phase-Rotation-containing optimisations is how they evolve in a series of very small steps, unlike the ones that just contained solenoids, which tended to evolve very quickly. I think it's because there are a lot of parameters in the RF system each of which has a small effect on the overall result. I took apart the latest run on DecayRotB and found it actually consisted on four different "directions", every 0.15% or so in yield, it started heading off a different way. I'm still developing an analysis tool (and eventually will upgrade ViewResults) to figure out quite how different these directions really are, amongst other things. A few times the results distribution and evolution of the successive best ones has almost looked fractal(!), which is totally unexpected. |
Stephen Brooks 2007-07-19 18:11:01 | While looking at this I noticed that the samplefiles generator was regularly generating samples for about 12 different lattices (i.e. all of them), even the ones that are no longer active. So I've changed that because it was taking hours. Just keep an eye on the sample files in case I've made some more obscure error and they end up filled with garbage somehow. Should be fine... |
Zerberus 2007-07-19 23:59:48 | It seems that there are still people generating results for SolenoidsTo15ccm with 4.43d! Has this any scientifical value or are these results for inactive lattices simply ignored? |
Stephen Brooks 2007-07-20 14:46:36 | Sounds like they've kept upgrading their client but somehow turned "lattice updates" off and forgot to turn it back on! Some of the old lattices probably still had further to go, I just stopped them because it appeared it was best to return to them if I manage to make the optimiser work more efficiently (i.e. they were taking too long to get anywhere, or any higher). |
Stephen Brooks 2007-08-07 13:56:29 | Hmm now, PhaseRotDD has a strange problem. The v4.43d generated results in the samplefile right now only go up to 0.363 yield, whereas the "best" shown on the plots is 0.40-something. Does that mean only v4.43c clients are generating these higher scores? |
Xanathorn 2007-08-07 14:13:56 | I think the problem is somewhere with your best-of parser, because my banner says my second pc did 0.389 as best yield (and I use only 4.43d). |
Xanathorn 2007-08-07 14:16:11 |
Stephen Brooks 2007-08-07 16:11:26 | Yes, that was what I was worried about - I'll go and look in the code... |
Stephen Brooks 2007-08-07 16:13:25 | Er, LOL, appears I'd got an extra "!" character meaning the samples consisted of all results that *weren't* rechecked, rather than all results that were. |
Stephen Brooks 2007-08-07 16:14:53 | I've done an extra samplefile run and hopefully that shall improve things. It's interesting that the progress has been continuing at a fair rate even with this mistake (which I suspect I put into the code when I removed the v4.43c results). Should go even faster now |
RGtx 2007-08-07 17:11:26 | Rerunning that result: 0.363215 (994.0 Mpts) [v4.43d] <PhaseRotDD> #time=75906; by room101[Romulus2] with the v4.43c client gives: 0.406470,0.413708,0.351005,0.397824,0.389418 (993.9 Mpts) [v4.43c] <PhaseRotDD> {1C00FD3A441A779966E302D8} // or 0.397904%. So some of v4.43c's lead is due to this disparity between clients. |
Stephen Brooks 2007-08-07 17:53:02 | Nope I was wrong on my first assessment - the script was filtering against results containing "#queued=" in their genome - that's being part of an unfinished queue, not having been queued (which is #runs=). So I'm still confused. Looking at Xanathorn's results file reveals: 0.389750 (994.7 Mpts) [v4.43c] ...so it was a v4.43c result with the high score. |
Xanathorn 2007-08-07 18:43:55 | Weird, I was sure I installed 4.43d on that, maybe it's a rogue box somewhere so I'll look into that later. Well then it's odd, but Rgtx's finding is worrying aswell since there shoudln't be that much difference in results between those clients. |
Stephen Brooks 2007-08-08 11:04:58 | The effect is most severe with these fairly small yields, as changes to the statistical method between the clients can produce (relatively) large differences. Should I publish a utility to remove v4.43c and earlier results from your results.txt and .dat files? That could act as a stop-gap before I release another version (which could be told to ignore these earlier results if possible). I tried finding the difference between c and d in the muon code before; didn't find anything, but maybe I changed an underlying (library) routine or something. |