stephenbrooks.orgForumMuon1GeneralStats temporarily off
Username: Password:
Page 1 2 3
Search site:
Subscribe to thread via RSS
Stephen Brooks
2007-05-18 16:46:12
...while I'm transferring everything over to a new computer.  The copy operation is taking a while, I'll either get them back up tomorrow or on Monday, depending on when I next get to come in.
Fozzie
2007-05-19 09:46:04
No offence, but it would have been good to give a little prior warning if this was scheduled. 
Stephen Brooks
2007-05-19 15:56:02
I thought it would be a simpler matter than it was (a backup I'd made turned out not to be a complete backup so I had to get the old computer back and manually copy a load of stuff).  So it was going to take longer than just Friday to complete.  That and I only knew my new workstation was ready 2 days ago.
Fozzie
2007-05-19 16:48:10
Thanks Stephen

As you can see there are a lot of people putting in a lot of CPU time.  Especially now that things are getting close at the number 1 slot so any heads up for anything that will effect the project, irrespective of how small it may be, would be welcomed.

Hope it all goes well this weekend and we should see a big change in the numbers come Monday.
[DPC]DanC
2007-05-19 23:56:05
No worries Stephen... It must suck to be a Physicist AND a SysAdm all rolled into one. 
Stephen Brooks
2007-05-20 00:18:54
There is also the issue of Dell workstations being "optimised" to only hold 2 hard disks, whereas my old one had space for about 5. At one point I just wanted to connect all the hard disks to the new machine and copy stuff internally... now I have to transfer everything over the NAS box I (conveniently) had just set up.  Though the first option may not ever have been possible given the new drives are SATA and the old ones IDE.

Yes, whoever it was who in another thread told me Dell workstations sucked was probably right, but I got this one for budget-fudging reasons.  The only way to get the bill into FY2006 (which was in surplus) was to go fast track through our "preferred supplier", which was Dell.  It's still a 2.66GHz quad core 2 Xeon, so I might overclock it until it breaks, then get a non-Dell replacement server it should be good enough for my testing purposes.
[TP]Skatoony
2007-05-20 09:30:26
Wow!  I wish I had a quad core Xeon
Stephen Brooks
2007-05-21 15:22:56
Writing from the new Xeon: OK, I'm copying the stats database over - this is taking a while!  Unfortunately it's all going over a 100Mbit/s network connection and there's 100GB+ to go.

If you think a quad-core Xeon is good, apparently Intel will soon release a quad-socket board, with each one taking a quad-core processor
Pascal
2007-05-21 17:13:43
For 100 GB or more it would have been a good idea to swap the HDD for copying over.  This mass of data needs more than 10 hours via the network for copying.
Stephen Brooks
2007-05-21 18:04:26
As I said above, "the new drives are SATA and the old ones IDE." The new workstation wouldn't take the old drive unless I had an adaptor or an external USB case for it.  Otherwise swapping the drives would have been the plan.

The data all seems to be here now.  I've been recompiling the stats script with altered paths (despite my best attempts to rationalise where I've written literal paths, there are still a lot to change).  It's running now, I'm probably going to leave it overnight but I wouldn't expect any stats because at this early stage it is likely to either (a) crash, (b) do the wrong thing or (c) attempt a full recount that might take days.
Pascal
2007-05-21 18:39:58
It is an intelligent way to choose a mainboard without any IDE interfaces.  Many participants and me just say thanks and go their own way.  Shall the many US overclockers take your project and push it to the way they choose.
[TA]amd.borg
2007-05-22 09:54:08
Please quit whining, Stephen's doin his best.  A few days without stats is not the end of the world.
Stephen Brooks
2007-05-22 11:36:26
--[It is an intelligent way to choose a mainboard without any IDE interfaces.]--

I agree.  Try telling that to Dell, and our silly corporate "preferred supplier" programme that only recommends Dell

Actually, I'm not sure.  It's inevitable one standard will be replaced by another and if you use money efficiently (and so only upgrade every 3+ years) a lot of the components will have become incompatible.  The floppy drive has now been replaced by memory card readers... I also don't know why our all network sockets aren't gigabit ethernet yet.
[TA]Assimilator1
2007-05-22 11:36:28
Does anyone understand what Pascal's saying?? 
He didn't have much choice on the machine.

Stephen
Thanks for your efforts ,btw you're quite sure your new PC doesn't even have one IDE socket?  that'd be unusual.(but wouldn't entirely surprise me with Dell)
In looking for an upgrade for my main rig I found going for a mbrd with 2xIDEs really cut my choice down ,its a right PITA their phasing out IDE so quickly

So when are you getting the quad socket core 2 duo machine then? 
Stephen Brooks
2007-05-22 12:00:40
There's a 8 socket dual-core Opteron machine (which is also 16 cores but in a different topology!) using a Tyan board and built by SCAN, waiting to be installed in a downstairs server room as I speak.  It's not "mine" but for the whole research group to run simulations quickly and test out multithreading in a many-core environment.
Stephen Brooks
2007-05-22 16:02:02
It's counting now - but seems to have LOTS of files to go through.  Both in the number it's counting and the number that have been downloaded and are waiting to be amalagamated into the stats.  It might take another day just to count through all this!  I wonder if I'm going to have to make the stats generator itself multi-threaded to keep up with the results coming from the multi-threaded clients?

I did find an IDE (or what I think is an IDE) port on this Dell, but it's up near the CD-ROM and there's no real hard disk mount there.  It would have been OK just to shove the disk there while copying across, but I've done the copying now anyway.
Pascal
2007-05-22 16:16:14
This is nice to know Stephen.  Perhaps it should be easier to do this with an usb hdd interface?  Or do you think there is any way to minimize the mass of data?  Before I am going to upload any results, I am willing to wait.  You also mentioned that it would be nice to know how the client works without any network connection for longer times.  A week or a month is no problem for me to realize.
laurenu2
2007-05-23 14:32:14
Stephen
Any ETA as to when the stats WILL be back up?

Do you need help to get them > back up <before the long weekend ?

WILL there be any lose of our Data that has not already been recored ?
Stephen Brooks
2007-05-23 15:29:50
Currently, I'm re-running the generator and it's getting stuck on one or two files that don't seem to have copied correctly and have large segments of 0x00 null bytes.  Fortunately the copies of these files on the backup drive are OK, so when the stats script stops on a file, I can reload the correct version.

The FTP download process is still retrieving files OK, so the ones sent while the stats are down shouldn't be lost.
Stephen Brooks
2007-05-23 17:17:52
Hmm it's gone past the corrupt files.  This looks hopeful...
HaloJones
2007-05-23 22:06:02
Hmmm.  not very hopeful at all really.  those stats are around two years old.
laurenu2
2007-05-23 22:39:19
Stats are totaly screwed up I am going backwards
This is what I ment when I said build and teast the server offline
Stephen Brooks
2007-05-24 00:14:39
No idea how it always manages to get itself in such a mess, but I think it's just "adding them up wrong" right now, so I let it continue to receive files overnight.
[dpc]McPhisto
2007-05-24 01:50:08
Good luck, and good night !
laurenu2
2007-05-24 02:46:53
Stephen it is not adding it is subtracting I lost 136,607,001 Mpts And that is not counting the 10,000,000 I have uploaded since you took the stat sever down
I don't thing you ever told us why the server needed to be replaced?
It all seemed to work OK
Pascal
2007-05-24 06:34:34
I am also missing ~200 Gpts and all my simulations from version 4.3
grayint3
2007-05-24 11:26:59
I'm also missing ~11000 Gpts... It's very bad!!!
[TA]BlackMountainCow
2007-05-24 11:34:13
I' down by 3,701,661 points, which is a third of my over all points.  Now I sure hope you got a backup of the stats or something like that! 
Stephen Brooks
2007-05-24 11:57:25
I found out this morning most of DecayRotB hadn't got copied across (explaining the large reduction!), so I've just written a script to try and merge that in.  It ought to come out closer this time.  Might have to do the same on PhaseRotD a bit later.
Stephen Brooks
2007-05-24 12:14:27
--[I don't thing you ever told us why the server needed to be replaced?  It all seemed to work OK]--

The old one made a periodic grinding noise like "NARN NARN NARN" under my desk, and smelled slightly of burning plastic.
[DPC]white_panther
2007-05-24 13:09:47
lol
[TA]amd.borg
2007-05-24 13:38:06
lol NARN NARN
laurenu2
2007-05-24 14:35:34
--[I don't thing you ever told us why the server needed to be replaced?  It all seemed to work OK]--

Oh then all you needed was a new power supply fan . Try it later and use the old server for a backup
[TA]Assimilator1
2007-05-24 15:05:44
Hey Stephen I know your also in the UK (don't know where abouts) ,need any help?  (I'm in Surrey btw)
[XS]riptide
2007-05-25 00:27:06
Leave Stephen alone you animals.!  HAHAHAH!  Stephen, going forward please ask harware questions here so some of us with more than average hardware skills and knowledge can help you.

That 4 way Quad is a Intel demonstration called Tigerton.  The 16 way AMD machine will be a GIANT.  Have a look at www.hwbot.org . See where it does well in wprime.  Currently the most powerful single noded setup out there at the moment.
laurenu2
2007-05-25 01:56:49
((Leave Stephen alone you animals.!  HAHAHAH!  ))

Why there may only be 1 or 2 times "I hope" that he puts him self in a Position
that we can pick on him like this So we have to take full advantage of it when we can
Stephen Brooks
2007-05-25 12:51:27
OK, stats on DecayRotB have been much fixed up apart from a file from LAURENU2 which is very large (of course) and I'm going to have to work out a way of merging it in without freaking out the generator.  I'll work on getting PhaseRotD and any others that have lost stats fixed up today.
[TA]Assimilator1
2007-05-25 13:16:21
lol laurenu2

Btw my Stephen my offer was serious (depending where you are & if PC techie skills are any use to you )

Oh & I'm still about 4million Mpts short
laurenu2
2007-05-25 14:20:29
Others are still missing a lot like Fozzie I think he is still down 150,000,000 so I thing you still have to restore more then just me
Bok said he has backup of all the DPAD stats if the would help
[DPC]ZMT~Mmikie
2007-05-25 14:42:31
Just like me, missing around 2/3 of the points around the 6 mil.
Stephen Brooks
2007-05-25 15:39:23
I found a lot of files had gone on ChicaneLinacB, which I've now replaced from the backup.  I can see the totals on each optimisation (going back in time) so I know where they have to get to to be right.
Stephen Brooks
2007-05-25 16:05:32
...and PhaseRotB and PhaseRotC.  No wonder I was only getting half the score, I was only counting half the database.

Assimilator1 - thanks, but right now it's just a question of copying stuff across, the hardware stuff is done with now.  The real reason I changed server was simply that my new one, to replace the 3 years+ old one was finally ready from the IT people (having been ordered about 2 months earlier).  So I was keen to install it.  Most of the problems we're experiencing seem to have come from dodgy file copying across the network.  I tended to assume it would just do it, whereas now I always check manually I have the same number of bytes and files turning up on one end as I put in the other.
HaloJones
2007-05-25 21:42:40
Rule #1: Never assume anything!

If you're copying from Windows to Windows, I would strongly recommend using something like robocopy rather than Windows Explorer or make a tarball or zip archive and copy that over.  Either way, it sounds like you know where the points have gone, so I hope to be soon back in first place like I was before the outage.
laurenu2
2007-05-26 15:58:38
I always just used a command line with switches like (( xcopy c:*.* d:/c/h/e/k/r ))
Copies about 10 meg a min and you can even boot off the new drive when complete.
It works in the Windows OS in Run . And best of all it is Free
laurenu2
2007-05-26 16:01:20
Sory typo (( xcopy c:*.* d:/c/h/e/k/r ))
RGtx
2007-05-26 18:16:07
86.5G at 10M/min will still take over 6 days to complete.
laurenu2
2007-05-26 20:48:08
((86.5G at 10M/min will still take over 6 days to complete.))

Well see he would have been done 2 or 3 days ago if he had done
the (( xcopy c:*.* d:/c/h/e/k/r )) way by your calculations
RGtx
2007-05-26 21:09:38
It would appear from reading through all the posts in this thread, that most of the delay is in rerunning the stats script itself, not in the data transfer, which incidently was at 100Mbit/s (~750MByte/min).
laurenu2
2007-05-27 23:59:56
Stephen
I found out what made your Sever make that NARN NARN sound
team Anandtech said in our forum http://www.free-dc.org/forum/showpost.php?p=112459&postcount=361 ((We unleashed our never-released-before-ultra-sekrit-tinfoil-encrusted-carbon-fiber-re-inforced-enlarged-caliber Zilla-krunch-bot.  ))

((Unfortunately, it has several side - effects, thusly:

1) An extended weekend suddenly appears before your eyes.  (Not our fault.  Really)
2) Avg.  AC output exceeds avg.  allocation by 2-300%.
3) Several computers in the United Kingdom begin speaking tounges.  The nearest translation to come out is "Narrn.  NARNNn.  NNAARNzN."
))
See Stephen you did not need a new server . It was just those Tean Anandtech boys playing with there Bot's
[TA]BlackMountainCow
2007-05-28 10:00:44
Oh no, our plan has been compromised! 
: contact : - - -
E-mail: sbstrudel characterstephenbrooks.orgTwitter: stephenjbrooksMastodon: strudel charactersjbstrudel charactermstdn.io RSS feed

Site has had 26117829 accesses.