stephenbrooks.orgForumMuon1Generalsimilar designs
Username: Password:
Search site:
Subscribe to thread via RSS
Joey2034
2003-05-28 12:17:54
Is there a system in place to measure how different the top simulations are from one another?  It might be helpful; at some point we are probably going to reach the maximum yield, because there are only so many combinations of numbers.  But due to the randomization of the pion decay, similar designs will give different yields, following some probability distribution.  So we may unwittingly end up testing the same designs over and over again in the hopes of getting a 'lucky run.' The top results might only slightly differ in a couple of the variables. 

I suggest taking a cue from statistics.  To measure the difference between two designs with variables V1i and V2i, take

sum( ( V1i - V2i ) ^ 2 / V1i )

Or maybe this would be better:

sum( ( V1i - V2i ) ^ 2 )

Then calculate this value using the #1 yield with each of the top couple hundred or so, just to see how unique the top yield really is. 
Although it might get a little tricky if the two designs have a different number of solenoids in the channel.

Any feedback?  Any statisticians out there?

[This message was edited by Joey2034 on 2003-May-28 at 20:27.]
[DPC]Stephan202
2003-05-29 02:00:03
I have to admit, I'm clueless about what you mean, or what results the calculations will be.  Statistics is not something I'm really good at (understatement Razz).

I may however be able to help you.  I think most of the top results are gathered at http://stephan202.qik.nl/. If you'd tell me exactly what I should do with these results (which variabeles to compare, what graph to draw), then I may try to create a PHP script that does these things.

---
Dutch Power Cow.
MOOH!
MaFi
2003-05-29 03:45:22
quote:

Or maybe this would be better:

sum( ( V1i - V2i ) ^ 2 )


sounds reasonable.  if you assume the results to be a vector in an n dimensional vector space (with an orthogonal radix), your sum is the square of the distance between those vectors... Wink maybe some weighting is nessesary if it turns out, that some components have a higher impact on the results than other.
but i like your idea.

markus
Joey2034
2003-05-29 07:33:20
quote:
Originally posted by [DPC]Stephan202:

If you'd tell me exactly what I should do with these results (which variabeles to compare, what graph to draw), then I may try to create a PHP script that does these things.





The calculation would go something like this:
For the two beamlines,

TANTALUMRODZ;TANTALUMRODR;S1L;S1F;D1L;S2L;S2R;S2F...S27L;S27R;S27F [...]
and
tantalumrodz;tantalumrodr;s1l;s1f;d1l;s2l;s2r;s2f...s27l;s27r;s27f [...]

Calculate:
(TANTALUMRODZ-tantalumrodz)^2 + (TANTALUMRODR-tantalumrodr)^2 + (S1L-s1l)^2 + ... + (S27F-s27f)^2 [+...]

The two beamlines must consist of the same number of units; otherwise they are incompatible for this calculation.  So you'll have to check that before calculating. 

Let's call the result of this calculation the D-value (D for difference).  I suggest taking the top design as one of the beamlines.  Then for every other design in the bestNNN file (or whatever), calculate the D-value with respect to it.  If you want, you can keep track of each design's D-value, or you can just take them as a whole and make a pretty graph of D-value vs frequency.

What the individual D-values will tell you is how different that design is from the top design.  In other words, how close together they are on the family tree.  A low D-value indicates similar designs (closely related), and a high D-value indicates fairly different designs (perhaps different families, coming from different evolutionary paths).

I really don't know what to expect.  I would say that D-values less than about 100 indicate very closely related designs.  To get 100, that would mean that, on average, each value of the beamline differs from the top beamline by just 1 or 2.

A D-value of 0 indicates identical designs.  Maybe you already have something in place to filter these out, I don't know.  If you get any zeros, though, their yields could conceivably be averaged together to get a more accurate figure. 

The highest possible D-value would be somewhere around 130.000.000, coming from two beamlines with 59 units, where one has all 0s, and the other has all 999s.

The pretty graph will show how much of a spread of designs there are in the bestNNN file.  As we get closer to the best design possible, I expect the D-values to converge to 0, where all the results in the bestof file are the same design, giving different yields only because of the randomization of the pion decay. 

And as for the beamlines that aren't compatible with the top design, maybe just assign them N/A or a negative number.  It might be useful to know how many design weren't compatible, because they could indicate more diversity.
[DPC]Stephan202
2003-05-29 08:25:57
I'll look into this.  I understand what you mean now, and in theory it shouldn't be too hard to create.  I won't guarantee anything, though.

As for multiple instances of the same configuration in the topNNN file: Currently the script sorts the results by yield, and then checks if the next result has the same config as the current one (it's just a loop that walks through all configs).  This done by string-comparing the first lines of both results, without the #runs=xxx; part.  If two result's configs are identical, the first one will be removed.  This means, that if they are not uploaded by the same person twice (which would make them 100% identical), the one with the highest yield is deleted.

Because of the variation in yields, it's not unlikey that the topNNN still contains identical results (I.E.: the same config simulated twice), since only neighbour-results are compaired.

---
Dutch Power Cow.
MOOH!
Joey2034
2003-05-29 11:52:35
I'm working on my own program now as well.  But I'm not by any means an avid programmer, so it could be slow and painful Roll Eyes
[DPC]Stephan202
2003-05-29 11:55:49
Stephen,

If a variabele is not listed, does this equal var=000;?

Also, I noticed that my client produces results with s28f=xxx.  However, the s28f variabele is not listed in viewresults.exe.  Is there something wrong here?

---
Dutch Power Cow.
MOOH!
Joey2034
2003-05-29 12:13:56
quote:
Originally posted by [DPC]Stephan202:

Also, I noticed that my client produces results with s28f=xxx.  However, the s28f variabele is not listed in viewresults.exe.  Is there something wrong here?




The client can produce up to s30f, while 26 is the minimum.  This corresponds to the 51 to 59 units in the beamline (26 solenoids + 25 drifts, 30 sols + 29 drifts).  Stephen probably left 27 to 30 out of the viewresults because he didn't know how to handle the designs that don't have those values.
[DPC]Stephan202
2003-05-29 12:25:51
Ok, thanx for the explaination.

Did you notice the top results don't have variables beyond s26f?  Also you can see that there is no variable left out in between, which is why I think that 000 means the same as not being listed.

I just made a piece of code that does what you suggested.  But it's very slow.  Not sure how I'm going to solve that.  I can tell you, that when I cross-check between the 10 highest results (that's 9+8+7+6+5+4+3+2+1 comparisons), values range from 827.823 to 32!  So some configs are very similar to eachother.

---
Dutch Power Cow.
MOOH!
Joey2034
2003-05-29 13:22:33
For the variables=000, each variable represents a property of the length, the radius, or the magnetic field strength of the solenoid (or the drift length).  Each of these can take on a range of values.  For the fields, they can take on from -4.0 to +4.0 Teslas, for most of them (I think those are the values anyway).  So a 000 means the minumum value, i.e. -4.0 Teslas.  A 999 represents the maximum value, 4.0 Teslas.  You can find a complete rundown of the ranges in section 3.2 of http://www.stephenbrooks.org/ral/report/
That rundown is for a previous version, so it doesn't completely apply to 4.31, but it gives you an idea of how the system works.

Yes I noticed that there is nothing beyond s26f, which makes the programming a lot easier Wink

I've got a preliminary program done now.  For the procedure outlined above, I'm getting an average of 8264.64 for your best250 file, which I suppose might have been updated since I downloaded it a few hours ago.  That's the average of the top one compared to all the others.  But yeah I'm seeing some low values, like 33, 12, 64, 29, 52, 40. Just to check for errors, what do you get for the calculation between the #1 and #2?  (13.696792% and 13.693488%) My proggy gives 4804, and hand calculations gave 4795 and 4884. Well my hand calculations obviously aren't very accurate.  There are just too many values to keep track of!

And your slow code, mine calculates the entire thing in the blink of an eye.  So if my results turn out to correspond with your resutls, I'll give you the code and you can translate it to PHP (it's in C++ now).  Or maybe it's just that PHP is slower than C++. Is that feasible?  I don't know anything about PHP, so you can enlighten me Smile
[DPC]Stephan202
2003-05-29 13:54:22
4804, that's what my script says too Smile.

I had a bug, so some results I just produced were wrong.  After analysing the first 40 results, the values range from 0 to about 69.000, with one enormous 143642. Can you confirm that?

As for the speed of PHP: it's indeed a lot slower as C++. But not this mach, I think.  You are probably using a much faster code.  I'd be happy to see it.  There's a chance that there are substitute PHP functions.

---
Dutch Power Cow.
MOOH!
[DPC]Stephan202
2003-05-29 14:11:21
I just made some small changes which dramtically increased the speed of the script.  I'll now add some things to make this compatible with results that go beyond the 26-variable (eg, have more variables).  I will therefor assume that the variables are always in the right order, and that there's never a variable left out.
Then I'll need to add default values for variables that are not listed (in order to compare them with results with more variables).  The problem here is, that there's no value between 0 and 999 that exacly corresponds with 0 Tesla... So I need to decide what's default.  499 or 500?  Or perhaps 499.5?

---
Dutch Power Cow.
MOOH!
Joey2034
2003-05-29 14:27:35
I can confirm a few up around 68000, but the highest I get for the entire top 250 is 83434. Maybe you're comparing different ones, so I'll check...

Ok, I just it slightly differently.  I did the D-value calculation for 1&2, 1&3, ..., 1&40, 2&3, 2&4, ...2&40, ...39&40. Doing that, there are a few around 144000, and one of them is 143642. And finally, calculating every possible pair of designs, the highest value is [drum roll] 212928!
Here's the code.  Once you figure out what I've done, vary some of the numbers to get different output.  Right now it's set to calculate for every possible pair of designs, and output only those values which are higher than 212000.



#include "stdafx.h"
#include "stdio.h"
#include "math.h"
#include "iostream.h"
#include "fstream.h"
#include "iomanip.h"
#include "stdlib.h"

int main(int argc, char* argv[])
{
int designs[251][105];
double dvalue=0;
char garbage[20], temp[4];
ifstream dataset("results250n.dat");

for (int i=1 ; i<=250 ; i++)
{
for (int j=1 ; j<=104 ; j++)
{
dataset.get(garbage,500,'=');
dataset.get(garbage,2);
dataset.get(temp,4);
designs[i][j] = atoi(temp);
}
dataset.get(garbage,100);
}
for (int k=1 ; k<=250 ; k++)
{
for (i=k ; i<=250 ; i++)
{
for (int j=1 ; j<=104 ; j++)
{
dvalue += pow( designs[k][j] - designs[i][j] , 2 );
}
if(dvalue>212000)
cout << dvalue << "," ;

dvalue = 0;
}
}
return 0;
}



EDIT: Sorry for the formatting.  I guess the board doesn't recognize spaces and tabs!
Joey2034
2003-05-29 14:38:38
If you want to make it handle more than s26, it might be a little odd.  From the current results file that I've got, the final values are mostly around the top edge, i.e. ~990. If you put 499.5 or anything like that in for the default, you're going to end up with some giant D-values, around 1.000.000. But I looked through the top 1000 file, as well, and they all only go up to 26. When will you even need to factor in the extra values?  I suppose, though, that if you do use 500 for the default, that will make it clear which results come from designs with more than s26. You could then compare those amongst themselves I guess.
[DPC]Stephan202
2003-05-29 14:45:35
It's getting late here, so I'll check your code tomorrow (I need to refresh my C++ knowledge a little, I know nearly no C++ at all.)
What I'd like to know: how many seconds does your program need to check all possible combinations within the top 250?  Mine needs about 27 seconds...

Imagine what it would take to run this test on alle 5000 results I got here

---
Dutch Power Cow.
MOOH!
Joey2034
2003-05-29 15:31:08
All combinations of the top 250 takes about 1 second.  I just tried with the top 1000, and it took 22 seconds, although I was counting in my head because I don't have a stopwatch handy.  It would undoubtedly take longer if I decided to save the output to a file.  I'm running a PIII 600 here, and the computer is quite cluttered, so it's not operating perfectly. 
One combination from the top 1000 gave 731501, which is significantly higher than any of the others (next highest is 578910).  I wonder which combination that was.
Joey2034
2003-05-29 15:46:49
To make this actually worthwhile, I was thinking we might want to do all combinations in the top files, and average all the values together, to get one value per file.  That way it'd be possible to monitor the distributions just from one simple number.  Right now 250 gives 14069.4, if I've done this right.
[DPC]Stephan202
2003-05-30 04:36:14
I see my script is much slower.  Guess there's no use in trying to improve it any further.

I have no time today to do any scripting, but what I may do, is create a script that draws a graph of the data supplied by your program... oh well, I'll see what I can do when I have time.

---
Dutch Power Cow.
MOOH!
Joey2034
2003-05-31 15:45:37
Ok I'm going to try to keep track of this number every day or two, or whenever I can.

Top 10...
31 May 2003, 6743.5
2 June 2003, 11483.0
3 June 2003, 13033.8

Top 250...
29 May 2003, 14069.4
31 May 2003, 8175.8
2 June 2003, 6831.1
3 June 2003, 8105.7

Top 500...
31 May 2003, 12244.0
2 June 2003, 6797.9
3 June 2003, 7035.1

Top 1000...
31 May 2003, 31540.2
2 June 2003, 11582.1
3 June 2003, 11804.4

[This message was edited by Joey2034 on 2003-Jun-03 at 23:57.]
: contact : - - -
E-mail: sbstrudel characterstephenbrooks.orgTwitter: stephenjbrooksMastodon: strudel charactersjbstrudel charactermstdn.io RSS feed

Site has had 25158981 accesses.