Stephen Brooks 2010-09-25 20:47:19 | Although it will be a while before any client updates (see my post in another thread on that topic), I've realised there's something else I want to do first, which can be done with the current client. I've never really taken a high-resolution scan of varying just one parameter over all its possible values - mainly because it requires 1000 simulations in a queue and that would take a long time on my workstation. This means I don't really have a clear idea of what the optimisation landscape looks like on a detailed level, particularly in the newer lattices. This sort of insight might let me get past some of the intermittent stumbling blocks that harder optimisations have. Anyway, I thought we could tackle this with some old-style workunit allocation distributed computing. I've made some queue.txt files which you can see here: http://stephenbrooks.org/muon1/hidden/scan/1/ What I hopefully want some volunteers to do is:
Hopefully as the blocks come back in, I should be able to build up a detailed graph - a cross-section across the optimisation of Linac900Ext10d2, centred around one of the recent best yield results, in the direction of the ls5l parameter (a solenoid length for one of the middle blocks of the linac). The choice of ls5l was somewhat arbitrary, mainly I chose it because the current value was 490 in the middle of the range and I expect changes in it to have a visible effect on beam transmission. |
tasburrfoot 2010-09-25 21:23:27 | I have queue975-999.txt Chocobo Navy of TeAm AnandTech Running using "muon1_background -q". Shouldn't affect the results, correct? [Edited by tasburrfoot at 2010-09-25 21:27:36] |
Stephen Brooks 2010-09-25 21:30:14 | I'm pretty sure muon1_background -q will just run Muon1 as usual. It'll probably do the queue but then it'll carry on making up its own results. Running muon1 -b -q ought to do the trick. |
tasburrfoot 2010-09-25 21:30:45 | How do I flush the results if there are less than 10? Just delete results.dat? (Sorry for causing problems) I use background, because I'm ususally using my machine while I have it running... [Edited by tasburrfoot at 2010-09-25 21:31:24] |
tasburrfoot 2010-09-25 21:43:04 | What isn't working? I flushed the initial cache I had (uploaded 16 results), then started on the first download. Well, I guess there is no results.txt generated, so nothing had really finished. Is it safe to let it run and see what happens? Or should I kill it completely and start over? I normally just run muon1_background and let it go... |
Stephen Brooks 2010-09-25 21:44:28 | OK, panic over - I'd stupidly forgot that the queue file for commandline operation was called "queuecli.txt". If you're running background, then "queue.txt" should be fine. I've modified the instructions accordingly... Your setup with queue.txt and background running should be fine, though I'd suggest running "muon -b -q" rather than muon1_background if you want it to stop after it's finished the queue (muon1_background is a stub for "muon1 -b"; it doesn't accept switches). |
tasburrfoot 2010-09-25 21:46:41 | EDIT: Nevermind. Re-read the instructions at the top. Will adjust accordingly... [Edited by tasburrfoot at 2010-09-25 21:59:32] |
Stephen Brooks 2010-09-25 21:49:33 | Ah yes, flushing less than 10 isn't possible - but don't delete results.txt, just store it somewhere and put it back after your test run. That way you can still get credit for it. Thanks for your help! I've got queue950-974.txt running on an old laptop I use as a server. It'll probably take a week to finish, but that's the only machine I have at home that runs 24/7! I'm off at a conference the next week so I won't have the desktop on. |
[TN]opyrt 2010-09-25 23:01:43 | Shouldn't you also move your autocli.sav (or similar for gfx, autogfx.sav, is it?) out of the way? |
Stephen Brooks 2010-09-25 23:15:51 | Um, yeah, that's also a good idea (maybe I should have recommended a clean install?!) Anyway I'll edit the instructions at the top. |
RGtx 2010-09-26 02:55:15 | I'll take queue0-99.txt. |
eveiga 2010-09-26 03:25:56 | I'll take queue100-199.txt. [Edited by eveiga at 2010-09-26 03:28:41] |
KWSN Grizzly 2010-09-26 08:47:40 | Taken 200-299.txt Regds Grizzly |
[DPC]white_panther 2010-09-26 12:19:48 | i'll take queue300-399.txt 0-99 RGtx 100-199 eveiga 200-299 KWSN Grizzly 300-399 [dpc]white_panther 400-499 500-549 550-599 600-649 650-699 700-749 750-799 800-824 825-849 850-874 875-899 900-924 925-949 950-974 Stephen Brooks 975-999 tasburrfoot |
tasburrfoot 2010-09-26 18:33:15 | Finished with 975-999. Downloaded 925-949 and running it now. |
tomaz 2010-09-26 18:44:26 | 0-99 RGtx 100-199 eveiga 200-299 KWSN Grizzly 300-399 [dpc]white_panther 400-499 500-549 TOMAZ 550-599 600-649 650-699 700-749 750-799 800-824 825-849 850-874 875-899 900-924 925-949 tasburrfoot 950-974 Stephen Brooks 975-999 tasburrfoot [Edited by tomaz at 2010-09-26 18:53:17] |
tasburrfoot 2010-09-26 19:12:07 | Finished with 975-999. Downloaded 925-949 and running it now. |
K`Tetch 2010-09-26 19:26:18 | I'll grab 600-649 and run it overnight. What sort of mpts are typical for these results? (I'm doing video editing/rendering on my main box, so I can only run overnight, don't want to leave it out of stuff) |
waffleironhead 2010-09-26 21:21:28 | 0-99 RGtx 100-199 eveiga 200-299 KWSN Grizzly 300-399 [dpc]white_panther 400-499 500-549 TOMAZ 550-599 600-649 K'Tetch 650-699 700-749 750-799 800-824 825-849 850-874 875-899 900-924 waffleironhead 925-949 tasburrfoot 950-974 Stephen Brooks 975-999 tasburrfoot [Edited by tomaz at 2010-09-26 18:53:17] |
waffleironhead 2010-09-26 22:18:28 | I just started up with a clean install. Some things to note if anyone else does this. 1) make sure you run the client once so it updates the lattice files before you set it to :nonet, or else muon1 will not know what to do with the queuecli.txt file you have. Linac900Ext10d2 lattice does not come packaged with the v444d file. 2) If you dont want to be running each of these tests five times I suggest you download the 100 result sample file and change the file to results.dat - that way you already have cached the current best optimizations and your client wont be running rechecks on -1.777777 results and wasting a bit of time. |
[TN]opyrt 2010-09-26 23:29:47 | A bit off-topic, but I'm wondering if number of threads each optimization is ran on has any relevance, as there will be particles in the optimization? |
tomaz 2010-09-27 05:20:48 | waffle, you can also download lattice file manualy and put it in your lattice directory. Stephen, wouldn't be a better comparasion if all results are run 5 times ? I thought -q would recheck every result regardles if it is the best or not. But I see now that -q is just for qiut after queue file is done. |
tasburrfoot 2010-09-27 05:45:20 | 925-924 is finished and uploaded. Starting 875-899 now. 0-99 RGtx 100-199 eveiga 200-299 KWSN Grizzly 300-399 [dpc]white_panther 400-499 500-549 TOMAZ 550-599 600-649 K'Tetch 650-699 700-749 750-799 800-824 825-849 850-874 875-899 tasburrfoot 900-924 waffleironhead 925-949 tasburrfoot 950-974 Stephen Brooks 975-999 tasburrfoot |
Stephen Brooks 2010-09-27 08:29:17 | OK, just found the wi-fi in the meeting room. Looks like these units are going well (making the table was a good idea!) K`Tetch - you might have been better with a 25 result file, these are probably quite long units (2000Mpts per simulation) but see how it goes on your computer. Tas - I've received your 975-999 via e-mail but not 925-949? I'm not keen on people just uploading these because I'll have to go and fish them out of the rest of your results in the database. It's possible, just a slight nuisance. waffleironhead - Yes, that's right. So adapting an existing install might not be so bad after all. I'd had in mind that most of these results are done as single runs, though the 5-run averages could be interesting for comparison, so it doesn't matter too much if you've done rechecked ones (I'll know which they are). [TN]opyrt - Number of threads shouldn't affect the simulation results, so I'd suggest using "auto" (the maximum). Tomaz - I might do a scan with #runs=5 later. This one I want mostly #runs=1 (and the occasional #runs=5 to see how far away it is). Most results the optmisation sees are single-run. |
RGtx 2010-09-27 10:48:00 | I'm using a small app. called "TheFolderSpy" to detect changes to queue.txt, then run a bat file to save and rename (appending time) a copy, permitting me to recover single run results for those run 5 times. |
tomaz 2010-09-27 11:34:50 | I grabbed 550-599 for my office computer I am running it with best so far result in result.dat, so there will be no 5 runners I think. 0-99 RGtx 100-199 eveiga 200-299 KWSN Grizzly 300-399 [dpc]white_panther 400-499 500-549 TOMAZ 550-599 TOMAZ 600-649 K'Tetch 650-699 700-749 750-799 800-824 825-849 850-874 875-899 tasburrfoot 900-924 waffleironhead 925-949 tasburrfoot 950-974 Stephen Brooks 975-999 tasburrfoot |
tasburrfoot 2010-09-27 12:45:07 | @Brooks: I sent them last night via email. When I get home tonight I will check and make sure it sent out correctly, and resend. |
Hydrox 2010-09-27 12:59:47 | I'll take 700-749 |
tomaz 2010-09-27 17:01:35 | 2.401777 (12782.0 Mpts) [v4.44d] <Linac900Ext10d2> {7D783F7DABB9D80E20BDC149} 2.333601 (2566.7 Mpts) [v4.44d] <Linac900Ext10d2> {E4A9204304076E42BB109D3D} 2.421967 (12782.0 Mpts) [v4.44d] <Linac900Ext10d2> {54D5950ED26250733969503C} 2.482482 (12781.0 Mpts) [v4.44d] <Linac900Ext10d2> {553C3CD7E899C310EEE93449} 2.381809 (2568.5 Mpts) [v4.44d] <Linac900Ext10d2> {7D9F54FEC0B19A96767D5433} Stephen, above are neighbouring results (500-504). It seems like a very tricky thing. Very small change of only one parameter results in rather large change in whole machine yield. It means problems not only in modelig but also when you want to built a real machine. What means ls5l=500 or 501 in real world ? Is it 100 m of some wired magnetic stuff or is it a 1 cm of something ? Off topic tought: Maybe parameters which are optimized to extreme values (000, 999) has much less influence. Can one asume that those with 000 might be unnecessary and those with 999 need to be expanded ? Can you try kind of selfbuilding lattice which would expand parameters acording to that logic (say to 9999 instead of 999) ? [Edited by tomaz at 2010-09-27 17:05:14] |
tasburrfoot 2010-09-28 03:49:08 | 0-99 RGtx 100-199 eveiga 200-299 KWSN Grizzly 300-399 [dpc]white_panther 400-499 500-549 TOMAZ 550-599 TOMAZ 600-649 K'Tetch 650-699 700-749 Hydrox 750-799 800-824 825-849 850-874 tasburrfoot 875-899 tasburrfoot 900-924 waffleironhead 925-949 tasburrfoot 950-974 Stephen Brooks 975-999 tasburrfoot I sent the two emails containing results. Outlook apparently didn't send the results before I closed it, so that email was actually sent this evening. I'm starting on 850-874 now. |
K`Tetch 2010-09-28 05:02:59 | Stephen, at 2000 per, thats 100,000 mpts, or about 66 hours work. not a big problem, but thanks for caring |
Stephen Brooks 2010-09-28 08:57:57 | TOMAZ - yes, this is the sort of thing I was looking for. It's most likely a features of the finite number of particles being simulated in Muon1, so you get statistical random effects when you change their paths. The 001 difference is perhaps 1mm in the length of a solenoid, but in this lattice it changes all solenoids in a block, maybe 10 of them, meaning differences in the trajectories of the particles will build up. For a very small difference, the real difference in yield might be small, but our simulations could get noise from some particles leaving and others entering the transmitted region. I expect the real machine to be quite sensitive to solenoid length but not THAT sensitive, and not in a "random" looking way. The parameters that have become 000 or 999 are running up against a physical or technological limitation (e.g. maximum feasible size to make solenoids for a reasonable cost). I'll also get someone to redo the same range twice to make sure Muon1 always produces the same output on deterministic lattices (that your client hasn't reverted to random somehow). Tas - I've got your results OK now! (about 8 hours ago) |
tasburrfoot 2010-09-29 04:52:34 | last one uploaded. I will take 400-499 now. 0-99 RGtx 100-199 eveiga 200-299 KWSN Grizzly 300-399 [dpc]white_panther 400-499 tasburrfoot 500-549 TOMAZ 550-599 TOMAZ 600-649 K'Tetch 650-699 700-749 Hydrox 750-799 800-824 825-849 850-874 tasburrfoot 875-899 tasburrfoot 900-924 waffleironhead 925-949 tasburrfoot 950-974 Stephen Brooks 975-999 tasburrfoot |
Cameron 2010-09-29 06:06:02 | Taking 750-799 |
Stephen Brooks 2010-09-29 09:21:43 | Some of the results have been sent back already: 0-99 RGtx 100-199 eveiga 200-299 KWSN Grizzly 300-399 [dpc]white_panther 400-499 tasburrfoot 500-549 TOMAZ 550-599 TOMAZ 600-649 K'Tetch 650-699 700-749 Hydrox 750-799 Cameron 800-824 825-849 850-874 tasburrfoot ---Received 875-899 tasburrfoot ---Received 900-924 waffleironhead ---Received 925-949 tasburrfoot ---Received 950-974 Stephen Brooks 975-999 tasburrfoot ---Received |
eveiga 2010-09-30 05:58:43 | Perhaps a dumb question but does the client give some indication when it is finished? I ran it as muon1 -c -q Will it exit after running the queue or will it just continue until I interrupt it? |
Stephen Brooks 2010-09-30 10:11:40 | If you've used the -q switch, it should exit when the queue is used up. |
tomaz 2010-09-30 18:31:24 | I took 825-849. |
[DPC]white_panther 2010-10-01 00:56:12 | i took 800-824 and have send the 300-399 results back 0-99 RGtx 100-199 eveiga 200-299 KWSN Grizzly 300-399 [dpc]white_panther 400-499 tasburrfoot 500-549 TOMAZ 550-599 TOMAZ 600-649 K'Tetch 650-699 700-749 Hydrox 750-799 Cameron 800-824 [DPC]white_panther 825-849 tomaz 850-874 tasburrfoot ---Received 875-899 tasburrfoot ---Received 900-924 waffleironhead ---Received 925-949 tasburrfoot ---Received 950-974 Stephen Brooks 975-999 tasburrfoot ---Received |
tasburrfoot 2010-10-01 02:11:34 | 400-499 uploaded, and I have 650-699... 0-99 RGtx 100-199 eveiga 200-299 KWSN Grizzly 300-399 [dpc]white_panther 400-499 tasburrfoot 500-549 TOMAZ 550-599 TOMAZ 600-649 K'Tetch 650-699 tasburrfoot 700-749 Hydrox 750-799 Cameron 800-824 [DPC]white_panther 825-849 tomaz 850-874 tasburrfoot ---Received 875-899 tasburrfoot ---Received 900-924 waffleironhead ---Received 925-949 tasburrfoot ---Received 950-974 Stephen Brooks 975-999 tasburrfoot ---Received |
Stephen Brooks 2010-10-01 09:08:54 | Some more have come in: 0-99 RGtx 100-199 eveiga 200-299 KWSN Grizzly 300-399 [dpc]white_panther ---Received 400-499 tasburrfoot ---Received 500-549 TOMAZ ---Received 550-599 TOMAZ ---Received 600-649 K'Tetch 650-699 tasburrfoot 700-749 Hydrox 750-799 Cameron 800-824 [DPC]white_panther 825-849 tomaz 850-874 tasburrfoot ---Received 875-899 tasburrfoot ---Received 900-924 waffleironhead ---Received 925-949 tasburrfoot ---Received 950-974 Stephen Brooks 975-999 tasburrfoot ---Received I'm in the final summary session of the conference at the Paul-Scherrer Institut in Switzerland right now. Might make a graph of these results on Saturday or Monday. I'd like one volunteer (other than TOMAZ) to do the 500-549 results a second time, just to ensure the client is giving consistent results on different machines. It should do, but it doesn't hurt to check! |
K`Tetch 2010-10-01 18:51:40 | I still have 12 of my 50 to go. do you want all of the blocks redone, or just 500-549? |
[DPC]white_panther 2010-10-01 19:20:08 | is it possible that i only get negative results on mine 800-824 block? |
K`Tetch 2010-10-01 19:33:14 | sure, if the yield is either too low, or not of the right energy (iirc -1.x is not enough, and -0.x is not right energy) |
Stephen Brooks 2010-10-02 00:59:46 | It's only really worth redoing one of the blocks. It's quite expected that some values will produce only negative results if they are too far from the optimum to work at all. My laptop is about 3/4 of the way through my small block of 25! |
[DPC]white_panther 2010-10-02 01:39:47 | i have send mine 800-824 block and start now on the recheck of block 500-549. |
tasburrfoot 2010-10-02 02:53:15 | 650-699 finished and uploaded. switched back into auto and running in the background. [Edited by tasburrfoot at 2010-10-02 02:54:52] |
[TN]steinrar 2010-10-02 19:41:50 | I'm also rechecking 500-549, hope it's not a waste of time Started yesterday, but forgot to post about it here. Not a big deal if i don't get credit for it. |
Stephen Brooks 2010-10-02 20:06:30 | You'll get Mpts credit for all of these so long as you remember to upload them. A 3-way check isn't a bad thing. 0-99 RGtx 100-199 eveiga 200-299 KWSN Grizzly 300-399 [dpc]white_panther ---Received 400-499 tasburrfoot ---Received 500-549 TOMAZ ---Received (Being rechecked by [DPC]white_panther and [TN]steinrar) 550-599 TOMAZ ---Received 600-649 K'Tetch 650-699 tasburrfoot ---Received 700-749 Hydrox 750-799 Cameron 800-824 [DPC]white_panther ---Received 825-849 tomaz ---Received 850-874 tasburrfoot ---Received 875-899 tasburrfoot ---Received 900-924 waffleironhead ---Received 925-949 tasburrfoot ---Received 950-974 Stephen Brooks 975-999 tasburrfoot ---Received |
Stephen Brooks 2010-10-03 01:28:07 | I've now plotted the graph including the results received so far: (red points are #runs=5 results, the optimum Muon1 had found when I started the scan is in green) |