I've been dealing with some server issues which have resulted in some outages. What's been happening is that "bing", that wonderful big MS search engine, has had its spiders occasionally pounding the server so hard with page requests that it's brought the server down. I've institute a few changes recently on the server to combat that. The first countermeasure has been a simple load monitor program which shuts the web server down if it gets too busy, and restarts it once it catches up on work and calms down. When it activates, the web site is unavailable, but down time is usually limited between 1-4 minutes. This has kicked in a few times over the past week or two and it's worked well.
Yesterday, however, bing was particularly aggressive and slammed the server so hard/fast the server was down before that counter measure could kick in. Manual reboots were a bit tricky because the bing page requests were coming in continuously such that as soon as the server rebooted, it would get overwhelmed again before I could even intervene. I did get in though and got a list of IP addrs which were the culprits and manually firewalled them off the server. That may well end the matter but I don't know if bing has other IP's they use which I don't yet know about. The down side of this is that bing won't be indexing anything on the server, but that's just the cost of doing business. Until those clowns at bing get their act together and stop overwhelming servers, that's probably how things will have to stay.
I can add another countermeasure that automatically firewalls such "attacks", basically doing automatically what I did manually, and may do so.
