stephenbrooks.orgForumMuon1GeneralCurrent plans
Username: Password:
Search site:
Subscribe to thread via RSS
Stephen Brooks
2002-11-08 12:13:02
It looks now as if the problem on my FTP server is caused by too many users logging in at once.  This was what the 100KB limit was supposed to avoid: by making any one machine wait a week or so between uploads.

The limit may be lowered to 30 or 50KB next version because the simulations take longer.  I don't want it to be user-set because I don't want my server to be overwhelmed by over-nervous stats junkies who set it to 1KB and upload every single result individually wink

As for the FTP thing, I'm working on that.  I want it to be able to have multiple FTP servers that it tries.  If you go here http://stephenbrooks.org/muon1/servers.php you'll see a form that will allow people with their own Muon server to register them.  Everyone can add to this list but only I can delete/modify stuff (so you need to request by e-mail).  I may make adding on-request at some stage also.  There is then a file http://stephenbrooks.org/muon1/servers.csv that holds the login data for all the servers.  I know a command to download a web-file to disk, which I'll add to the sendresults.c module, so that it will attempt to download this servers file when the old one is 7 days old, or if it hasn't got one yet.


"As every 11-year-old kid knows, if you concentrate enough Van-der-Graff generators and expensive special effects in one place, you create a spiral space-time whirly thing, AND an interesting plotline"
Stephen Brooks
2002-11-08 12:23:32
(cross-posted with "Ideas for scaling the project")

I need to replace the line about "parrots" by some guidelines on how the FTP account should be set up for Muon1 done that, but for those of you who already know how the system works, you can signup now.

First thing I'll release next will be a new manualsend.exe that will download the servers.csv file if it can, and then uploads to a random server.  I'll then have to modify my scripts here to flush all servers for results.

The Infopop account of mine is in that list now mostly as a test - if it's still not working when the new manualsend is out, I'll take it off the list.

My local server ought to be up some time soon, but I haven't heard back from the guy who manages the firewall yet...



"As every 11-year-old kid knows, if you concentrate enough Van-der-Graff generators and expensive special effects in one place, you create a spiral space-time whirly thing, AND an interesting plotline"


[This message was edited by Stephen Brooks on 2002-Nov-08 at 20:19.]
[DPC] Jive
2002-11-08 14:02:22
Roger Wilco ...
AySz88
2002-11-08 14:22:22
Suggest a lighter, more-standing-out color for the dotted you use for edit highlighting.
Sorry, thread hijack.
Stephen Brooks
2002-11-08 15:36:23
big grin


"As every 11-year-old kid knows, if you concentrate enough Van-der-Graff generators and expensive special effects in one place, you create a spiral space-time whirly thing, AND an interesting plotline"
Black-X
2002-11-08 16:17:59
Maybe it's an idea to not always get an random server but also can specify which you want to use... So the Clan_Delft users can say they always want to send to me so I can make their stats?

And probably an unique code which is different for every person?  Then I think much people will stay here instead of ecc2 because then I'll make subteam stats generator pages...

If you like...

For an example: http://scriptdev.homeip.net/reader.php
Brian Sogard
2002-11-08 17:13:26
I can provide ftp space for the project, but not with the user specific permissions you ask for.  My domain host does not have provisions in place for this.  I can create users, and I can change the global permissions on any directory, just not user specific.
Stephen Brooks
2002-11-08 18:50:56
quote:
Maybe it's an idea to not always get an random server but also can specify which you want to use... So the Clan_Delft users can say they always want to send to me so I can make their stats?


If you've already got proxy software, then just use that to send the files across before they get to 100KB in size (circumventing the muon1 send), and then when the proxy sends the results on, make it call the manualsend.exe packaged with Muon1 to do this.

quote:
And probably an unique code which is different for every person?  Then I think much people will stay here instead of ecc2 because then I'll make subteam stats generator pages...


That's what user.txt is for...

quote:
I can provide ftp space for the project, but not with the user specific permissions you ask for.  My domain host does not have provisions in place for this.  I can create users, and I can change the global permissions on any directory, just not user specific.


Erm, what's the point of having users if they all have exactly the same access privileges?  If you mean you can only set USER-GROUP-ALL Unix-style rights, then that's fine because you just make the Muon1 user "anonymous" and have the directories owned by the datacenter user, or put the datacenter user in the same user-group as yourself.


"As every 11-year-old kid knows, if you concentrate enough Van-der-Graff generators and expensive special effects in one place, you create a spiral space-time whirly thing, AND an interesting plotline"
Monsoon
2002-11-08 19:29:57
Do you know roughly how much bandwidth a datacenter will use?
Stephen Brooks
2002-11-08 20:16:34
quote:
Originally posted by Monsoon:
Do you know roughly how much bandwidth a datacenter will use?


You mean one of these FTP servers?  (I'm using the word 'datacenter' to refer to something else - i.e. what I am, a computer that stores results and calculates stats).

Well, to put it one way, the total amount of results transferred to here in the history of the project so far is 750MB, and that's over several months.  Webhosts typically give 3GB/month bandwidth as a minimum, so the traffic generated will be moderate, but not very high.  Of course the more people have servers the less this will be per person.


"As every 11-year-old kid knows, if you concentrate enough Van-der-Graff generators and expensive special effects in one place, you create a spiral space-time whirly thing, AND an interesting plotline"
Brian Sogard
2002-11-10 07:00:07
slight change of plan for providing ftp space for muon1...
since my domain host has some strict 'built-in' restrictions on ftp users, planB is setup ftp server here at home on one of my machines running DC projects.  Fine, that's done.. One tiny catch finishing up start-up configuration for serving space to Muon1...

pre-loading signal.dat into the root directory on my server.... signal.dat is a tad difficult to download from another ftp server when the username and password to retrieve signal.dat isn't known, unless I have missed something in discussion (again)
DrHanser
2002-11-10 10:42:44
quote:
Originally posted by Brian Sogard:

pre-loading signal.dat into the root directory on my server.... signal.dat is a tad difficult to download from another ftp server when the username and password to retrieve signal.dat isn't known, unless I have missed something in discussion (again)


I had the same problem last night.

--
The mark of an educated man is one who knows a lot about something, and a little about everything.
ZeonX[OCAU]
2002-11-10 12:57:43
arhh well i just downloaded it from the main ftp server.

But you can login to mine with l/p: muon1/wednesday and download the signal.dat
TheEternalVortex
2002-11-10 13:31:21
Stephen did link to the csv file with all of the logins...
http://stephenbrooks.org/muon1/servers.csv
Brian Sogard
2002-11-10 14:08:06
*chuckle*

I saw the link to the servers.csv, but just didn't put two and two together..
Thanks guys, I thought I must have missed something pretty obvious, turns out I did!.
Stephen Brooks
2002-11-10 14:51:33
Also if you mouseover the server-name on the servers list page, you get the username and password for the public/muon1 user displayed.  You can use that to download signal.dat. 

Thanks all for the cooperation so far.  The list (from my point of view) now looks like this:


...where the original server is now in purgatory until I am confident it will work properly again.  Note that all DynDNS domains (including mine) don't yet appear to work.  Maybe that's just the DNS taking ages to propagate, or maybe it's a flaw in this way of doing things.  Can you check if your domain works if you're using DynDNS (my primitive DNS-detection script may be wrong)?  I'll leave those servers on the list, since the new manualsend will just try random ones until it makes contact.


"As every 11-year-old kid knows, if you concentrate enough Van-der-Graff generators and expensive special effects in one place, you create a spiral space-time whirly thing, AND an interesting plotline"
Brian Sogard
2002-11-10 16:02:02
One way to test both DNS and server being available is to ping it.

I pinged the servers on the list:

zeonx.mega-reviews.com I averaged round trip ping time 611ms
andrey.mine.nu I averaged round trip ping time of 3095ms
muon.serveftp.org round trip ping time averaged 117ms
drhanser.dyndns.org ping timed out (could be a blocked port)
muon.rdsroot.net ping timed out (could be a blocked port, I actually did ftp in to muon.rdsroot.net successfully)
I had a good ping and ftp to sogard.serveftp.net, but it's a local server so it should work, ping did ping to my external IP rather than my internal IP, so it APPEARS that the DNS is good externally.


Hope this is useful info
Stephen Brooks
2002-11-10 16:19:11
OK lookes like DynDNS do their stuff in some edfunced way that baffles PHP's DNS command.  I've just searched the web and found a way to rig up PHP with ping so I'll try using that instead.  I was surprised at the lack of a built-in PHP pinging function.

--[andrey.mine.nu I averaged round trip ping time of 3095ms]--

Where's that one hosted, the Moon?

I've just improved the script to do a proper "gethostbyname" on the addresses, and if they are already in the form of IPs a.b.c.d I do only the FTP port check.  Now it looks much better.  Dr. Hanser's FTP thing is currently down as you say, and so is mine because the network admin hasn't yet unblocked my port.  The others are usually on apart from one that's a bit intermittent but I forget which this was.


"As every 11-year-old kid knows, if you concentrate enough Van-der-Graff generators and expensive special effects in one place, you create a spiral space-time whirly thing, AND an interesting plotline"

[This message was edited by Stephen Brooks on 2002-Nov-11 at 0:04.]
Brian Sogard
2002-11-10 18:39:47
It's interesting to see the speed at which this transition to distributed ftp servers is progressing.  It was only Nov 2 when Stephen posted his concept.  Already there are some servers available and work is moving forward to implement this new system.  I've been doing the DC thing with the different well-known projects for maybe a couple years, it seems the smaller number of participants here lends an atmosphere of co-operation that other projects only see in the alpha and beta test groups.  I guess that sense of community is one of the things I like about this project.
pvs
2002-11-11 04:21:57
How about Security?

Everyone can setup an FTP-Server and get results, so he could delete results, he could take some results and mix them with his own, no one would recognize this.

I think there has to be a closed source programm for sending results, that encrypts every result, gives the results a special key (e.g. [SG]PvS#"number") that is stored with the results in the results.dat and saves a second list on the client containing the following:
results sent "date", "FTP-Server"
first result "key"
last result "key"

If theres nothing like this, the whole "distributed FTP" is nothing than a big joke..

Peter
Stephen Brooks
2002-11-11 06:23:29
Well it's better to have only the FTP admins being able to delete results than it is to have everybody being able to wink
Also the new manualsend program will generate a log to say which server it sent to.  As soon as people complain of their results being lost on an FTP server, I will remove that FTP server from the list for further investigation.

Another thing I could do is make the filename NOT be related to the user, or even better, make the 5 "random" chars include a checksum of the username hashed with the file data.  So if a person tried to rename someone else's file to their own, they would not succeed and also generate an alert of sorts here.

Currently I'm trusting the FTP server admins just like I trusted the users up until now (with a public FTP server).  As it turns out, the "lost" results so far were not malicious deletion, but rather an error caused by Infopop's server and my download script malfunctioning.  This suggests we will be able to get a fair number (>100) of server-admins before a 'bad guy' arrives.

quote:
Originally posted by Brian Sogard:
It's interesting to see the speed at which this transition to distributed ftp servers is progressing.  It was only Nov 2 when Stephen posted his concept.  Already there are some servers available and work is moving forward to implement this new system.


Well it would have been even faster if I hadn't been in bed most of the last week with a cold!
I think the final straw that pushed me from just talking about it as an interesting idea to going ahead with it was my server breaking down.  That sort of thing would _not_ kill a project with distributed servers.
Also a single server is susceptible to DDoS attacks, but the more servers we have, the more bandwidth it would take to DDoS them - in fact, DDoS loses its power when the target becomes distributed too.  With things like GuildFTP giving usage feedback to the admins, they'd soon twig which IPs an attack was coming from and block them, then e-mail the other admins to do the same.
This is not a priori bulletproof security, but we are adaptable and can deal with security threats as and when they happen.  I'm taking a course involving cryptography this year too, so if we get some real bad cheaters appearing, I may eventually have to use a public-key system or something similar.


"As every 11-year-old kid knows, if you concentrate enough Van-der-Graff generators and expensive special effects in one place, you create a spiral space-time whirly thing, AND an interesting plotline"
DukeBox
2002-11-11 06:51:57
Quite offtopic.. could i have your 'team filter' php code ?
Stephen Brooks
2002-11-11 09:59:55
$team=file('http://stephenbrooks.org/muon1/teamids.txt');
foreach ($members as $name)
foreach ($team as $id) if (stristr($name,trim($id)))
echo "User $name is in team $id\n";


Something like that.


"As every 11-year-old kid knows, if you concentrate enough Van-der-Graff generators and expensive special effects in one place, you create a spiral space-time whirly thing, AND an interesting plotline"
TheEternalVortex
2002-11-11 11:40:42
quote:

Where's that one hosted, the Moon?


Well, you might have caught me when I was downloading a bunch of crap smile.
TheEternalVortex
2002-11-11 11:43:36
Oh, and should I hold off uploading results until you finish the new manualsend program?
Stephen Brooks
2002-11-11 12:05:16
quote:
Originally posted by TheEternalVortex:
Oh, and should I hold off uploading results until you finish the new manualsend program?

Well until it's debugged, yes.


"As every 11-year-old kid knows, if you concentrate enough Van-der-Graff generators and expensive special effects in one place, you create a spiral space-time whirly thing, AND an interesting plotline"
cardinal808
2002-11-11 12:43:33
when is that, do you have any ideas?
tommorow?  next week?  next month?
Stephen Brooks
2002-11-11 13:41:59
I got rid of a couple of bugs just now.  You could try the new version http://stephenbrooks.org/muon1/multisend.zip .


"As every 11-year-old kid knows, if you concentrate enough Van-der-Graff generators and expensive special effects in one place, you create a spiral space-time whirly thing, AND an interesting plotline"
DrHanser
2002-11-15 07:54:59
quote:
Originally posted by Brian Sogard:
drhanser.dyndns.org ping timed out (could be a blocked port)



Yes it is.  I contacted Stephen last weekend... my server will be down until I get around to reinstalling Win2K Server this weekend (hopefully -- it's going to be a busy weekend)

--
The mark of an educated man is one who knows a lot about something, and a little about everything.
Stephen Brooks
2002-11-19 16:22:46
Hey, I got my fixed IP working.  big grin big grin Now you're the only one with a red-light on the servers list.


"As every 11-year-old kid knows, if you concentrate enough Van-der-Graff generators and expensive special effects in one place, you create a spiral space-time whirly thing, AND an interesting plotline"
: contact : - - -
E-mail: sbstrudel characterstephenbrooks.orgTwitter: stephenjbrooksMastodon: strudel charactersjbstrudel charactermstdn.io

Site has had 17087076 accesses.