stephenbrooks.org : Away until 24 September

Stephen Brooks 2012-09-14 14:54:00	The project will run "at risk" until the 24th of September because I will be at a conference in Beijing and unable to fix the stats server if it has an error.
Stephen Brooks 2012-09-16 03:57:17	Well here I am posting from the hotel lobby wireless in Beijing. The timestamp will say the middle of the night but it's a sunny day here. Not sure how long the free wireless will last though, I might only have 3 days and it may get clogged when all the other scientists try to check their e-mail simultaneously during the meeting...
runesk 2012-09-16 12:49:52	I guess you jinxed it :/ the stats server seems to down/not working .R
Stephen Brooks 2012-09-19 07:33:21	It looks to me like the stats data (or at least its timestamp) is still updated hourly, however, the whole site has undergone a bork because my webhost has done a migration and I believe the PHP version has increased (probably to PHP5). This changed the conventions for passing variables from the URL to the script, added warnings for opening non-existent files where there was silent behaviour before and changed the times when the script could modify headers (e.g. cookies). So you can see how it caused problems!Â I've probably not found all the bugs yet...
Stephen Brooks 2012-09-24 12:32:08	Well I'm back in the office now and nothing seems to have broken (apart from my webhost PHP thing and occasional weirdnesses with their security software blocking people from the site).

[TN]marvik
2012-09-25 07:39:31

Download from http://stephenbrooks.org/muon1/servers.csv failed, will try later.
Could not get servers.csv file, so cannot send

In addition, it seems like stats.free-dc.org hasn't been able to update its stats since september 18.

http://stats.free-dc.org/stats.php?page=proj&proj=dpad

It may well be that security software. I know a couple of my team mates also have trouble reaching your site.

runesk
2012-09-25 09:53:27

As a host that fails in the way marvik describes, also is blocked from ICMP Echo (aka ping), my guess is that there is some kind of "anti denial of service" software running on the web server.Â

A good suspect is fail2ban (http://www.fail2ban.org/)

It might be something else, but because a host running muon1.exe will try to download linac900ext6xc2_nosample_100.bin and this gives a 404 error, these clients IP addresses will end up in the the error.log of Apache on your web server.Â One of the features for fail2ban, is to monitor these error logs and block IP addresses (using iptables or some kind of firewall script).Â

.R

Stephen Brooks
2012-09-25 11:26:41

Download of servers.csv works fine from here, as does rawstats.txt and simplestats.txt.Â The only thing that should produce a 404 is that nosample file.

Have e-mailed my webhost about the blocking problem, pointing out it might be misdetecting scripted access as DoS attacks.

yoyo
2012-09-25 21:22:10

I'm getting this fault connecting from the yoyo@home server:

++ curl http://www.stephenbrooks.org/muon1/lattices/list.php
curl: (7) couldn't connect to host

Seems that the computer is blocked.
Access works if I connect from my private PC.

yoyo

runesk
2012-09-26 08:52:48

On a computer that is blocked, no traffic seems to be allowed.Â

Usually, you would be able to connect to the following open TCP ports:


Nmap scan report for stephenbrooks.org (77.240.9.17)
Host is up (0.040s latency).
rDNS record for 77.240.9.17: al.netweaver.co.uk
Not shown: 990 filtered ports
PORT    STATE SERVICE
21/tcp  open  ftp
53/tcp  open  domain
80/tcp  open  http
110/tcp open  pop3
143/tcp open  imap
443/tcp open  https
465/tcp open  smtps
587/tcp open  submission
993/tcp open  imaps
995/tcp open  pop3s

Nmap done: 1 IP address (1 host up) scanned in 5.75 seconds

On a blocked host none of these ports are available:


Nmap scan report for stephenbrooks.org (77.240.9.17)
Host is up (0.0033s latency).
rDNS record for 77.240.9.17: al.netweaver.co.uk
All 1000 scanned ports on stephenbrooks.org (77.240.9.17) are filtered

Nmap done: 1 IP address (1 host up) scanned in 9.99 seconds

The last of these host has been running 24 muon1.exe processes for some weeks.Â Sadly, I have no log to pinpoint the time this ban started, but I heard others having this kind of problem after "the migration". Users having multiple hosts behind NAT will also give the same result in the log files.Â

So I still stand by my guess, that some kind of DoS protection mechanism triggers, either by the 404 Errors or by large amounts of request for the same files over and over.

Zerberus
2012-09-26 13:27:49

Want to add my voice. Getting banned regularly (and very quickly). That's not the way it should be.

Stephen Brooks
2012-09-27 19:02:45 They say they've switched mod_security off on my server and lifted various IP blocks.Â Can you try again and see if it works?

[See e-mail]

Thanks, since it will take a while to patch my software that's probably the way to fix it (I think you may only have to disable that one particular rule in mod_security.conf, included from httpd.conf actually). I think the problem is that my program is producing requests that begin GET http://stephenbrooks.org/muon1/samplefiles/linac900ext7xc2_100.bin HTTP/1.1 ...whereas I really only need to do GET /muon1/samplefiles/linac900ext7xc2_100.bin HTTP/1.1 I think "^\\w+:/" breaks down as ^ = start of string \w = "word character class", i.e. [A-Za-z0-9_] + = 1 or more :/ = a colon followed my a backslash ...so this rule is saying it doesn't like the protocol part (http:/ or ftp:/) being specified in the URL request. I'll do some tests so my program looks less like malware in the future! -Stephen ----- Original Message ----- From: "netweaver | help" To: "Stephen Brooks" Sent: Wednesday, September 26, 2012 10:49 PM Subject: [Ticket ID: 106495] Users being blocked from stephenbrooks.org Hello mod_security is now disabled on your account - I've also unblocked a number of IPs.Â Any remaining blocks will time out automatically. Please let us know if we can be of any further assistance, Best Regards Phil netweaver | help

[TN]Ungelovende 2012-09-27 20:38:56	Yes! It works
runesk 2012-09-27 21:17:11	Seems to work. the mod_security and "malformed" request from muon1.exe explains why my squid cache were not banned. .R
Zerberus 2012-09-27 23:43:16	Thanks. First I though the homepage was axed, started one day after your departure. Since my client usually runs at startup I couldn't ever access it.
Stephen Brooks 2012-09-28 16:28:04	I think I've fixed the function of my internet library that was sending the nonstandard requests too. For proxies you have to do GET <whole URL> but for direct requests it's GET <server path only>.
yoyo 2012-09-28 17:58:46	It works now also on my side. The yoyo@home server is able to create new workunits. yoyo
Zerberus 2012-09-29 20:20:53	Classic case of Murphy's law at its best.