stephenbrooks.orgForumMuon1GeneralRejected checksum log analysis
Username: Password:
Search site:
Subscribe to thread via RSS
Stephen Brooks
2004-09-14 10:42:29
Since a few months ago, I've been logging instances of the stats generator rejecting results (because of bad checksums) to a log file.  It's about 3MB long now (but that includes all the results that were rejected), so I wrote a program to graph which users sent the bad results against time.



One pixel means one result rejected, so there haven't been that many.  Most of them seem to come from 'lazykiller' - the files I receive from him often just cut off or go corrupt in the middle.  Fortunately this looks like background errors and no evidence of attempted cheating or anything of that sort.
kitsura
2004-09-14 13:16:58
But doesn't the bad checksum results get rejected before the result file is uploaded?  How is it that there are still more cases of files with bad checksum?
AySz88
2004-09-14 21:52:34
If the file is corrupted or interrupted during a transfer, after the ssender checks them locally, results still can be rejected on the server side.
Stephen Brooks
2004-09-15 02:32:07
quote:
Originally posted by kitsura:
But doesn't the bad checksum results get rejected before the result file is uploaded? 
Good point: they ARE filtered on the client side too, but this could theoretically be hacked around by someone submitting results onto the FTP servers manually, for instance.  Hence the second check.  So in fact what this graph shows are either accidental file corruptions or manual uploads of invalid results.
Stephen Brooks
2004-09-15 04:41:42
Here it is again with the users sorted by frequency and some of the "Bad Format" reinterpreted.
LostinTime
2004-09-15 08:03:38
While this may get some of the results that are missing fixed, this still does not get rid of the server clipping some of the results.

As I stated approx a month ago....

It appears that some people are saying the server is clipping the first result of each type.  I discovered that it could be any result.  (A high result that should make the board and doesnt was the way that I discovered that it could be any.  These were not bad checksums on the client side.  To remedy, I would just resend the high result.  (Not too much a problem for me as I keep a close watch on the results but for most its not a easy thing to do.) And it's not that they are not showing in the stats page, these appeared not to enter the sample list.  So I expect a bad checksum on the server side or worse.. total wipe of the results in question.  While this is really no big deal for me, I'm trying to help my team.

The overall bad thing is that I figure that I'm losing approximately 2000 Mpts (or more) every time I submit snf I submit several times a day.
(Yes, this is pretty accurate.  I'm running a dual Xeon.  Most of the time running w2k pro but on occasions I'll switch to the other hard drive and boot up w2k server.)

Now for a client bug: Unless you click on the sucessful sent button, the results do not get added to the results Sent Log.  In other words, after a successful send, if you just close the Main window, the send log is not updated but on very rare occasions.  (does happen once in a while).  If you do click on the button, the results do always get added to the sent log. 

Hope this helps.
AySz88
2004-09-15 23:20:15
If I recall correctly, I'm fairly sure that the first-result-of-every-file bug was fixed already.... I'd be surprised if it was still in place.
Stephen Brooks
2004-09-16 03:14:24
What's interesting is that if it were rejecting these first results, you would have shown up on that graph above.  It's possible they're somehow just "not being read".

I think I need to test it, as it may be happening with some users but not others.

I think it would be best to do the following diagnostic sequence with the cooperation of LostInTime:
1. He (she?) e-mails me to say he's ready to start the sequence.
2. I add comment tags to the last 5 results in all of his files, labelling them ABCDE or something, so I know whether the missing result is eating into what is already there or the newly-sent stuff.  Then I reply to the e-mail saying I'm ready to continue.
3. He sends a medium-sized (e.g. 100K result file, e.g. via manualsend, but ALSO sends a copy to me via e-mail, so I can compare what arrives.
4. Then I just wait a few hours for it to get through, and see whether the stuff appearing after the ABCDE-marked results is in agreement with what it should be, or whether the first result is disappearing.

At that stage I ought to have the problem file itself so I can test re-sending your file as many times as I need to get it right.

[edit] That "client bug" wasn't hard to fix - I just had to make it log to the file before it produced the dialogue box.  I didn't really intend for the program to be shut down forcefully at that point (why would it, when there's a big button marked "OK" that does the same thing?!) but in all later versions it should behave as expected either way.
LostinTime
2004-09-16 17:28:22
Stephen,

Its a he.

Sure I will help if I can.  It appears to be a random loss on your end (or some where inbetween) not necessarily first , middle , or last type situation.  For your information, all my sends are manual sends (dialup).  The only reason that I caught the ones that I did was that I was expecting to see them in the high results list.  Did notice one other thing, if the total number of results is 3 on a send , it probably doesnt get added even tho the size of the file is big enough to be accepted by the server.  (While most of my sends are in the range of 30kb-150kb occasionally I will send smaller number)

Feel free to let me know what you would like for
me to do.

Jim
LostinTime
2004-09-16 17:37:58
[QUOTE]Originally posted by LostinTime:
Stephen,

Its a he.

Sure I will help if I can.  It appears to be a random loss on your end (or some where inbetween) not necessarily first , middle , or last type situation.  For your information, all my sends are manual sends (dialup).  The only reason that I caught the ones that I did was that I was expecting to see them in the high results list.  Did notice one other thing, if the total number of results is 3 on a send , it probably doesnt get added even tho the size of the file is big enough to be accepted by the server.  (While most of my sends are in the range of 30kb-150kb occasionally I will send smaller number)

Feel free to let me know what you would like for
me to do.

Jim


Might suggest that we just try a few dupkate sends of my results (one normal and one via email--several times) first.
Stephen Brooks
2004-09-17 00:33:31
Someone else who had this said they thought it was the first of _each optimisation_ present in a file, which makes sense as here I pull the files apart and file each optimisation under a different directory.

I'm not sure duplicate sends will help, as that then involves the duplicates checker (which will get rid of them anyway!).  Try sending some new results but e-mailing me the exact results.txt file you send first.  Then hold off sending anything else until I reply and say I've had a look at what effect those results have had (though it should take you a while to generate some more).
Stephen Brooks
2004-09-17 09:05:15
OK, I've got your file and marked the last results of the ends of your files in my database with letters so I can see where they are.  I'm going home for the weekend now, but when back I can see what's happened and compare your file to what actually arrived.
: contact : - - -
E-mail: sbstrudel characterstephenbrooks.orgTwitter: stephenjbrooksMastodon: strudel charactersjbstrudel charactermstdn.io RSS feed

Site has had 25161174 accesses.