The Butt Ugly Weblog

Oh you bloody RSS bugger

Here's another caveat for RSS and spiders, that Wiki authors probably should be aware of:

Figured out why Grey (the machine that's hosting jspwiki.org, this blog, and suomigo.net has not been doing very well lately: The loads go up to 12, and I am getting lots of errors, as if the machine was under enormous load. However, when looking at the list of top processes, even if the load is up to 12, the CPU usage is about 20%.

I was looking through the load/IP activity logs and realized that there is a four-hour cyclic, massive increase in both the number of simultaneous connections and CPU load. I already had a nagging suspicion that the RCS back end we're using might be the cause of the load-that-does-not-show-up-in-CPU-usage, because it tends to spawn many quick processes very rapidly, so they never show up in top. I went through the Apache log files and realized that MSN bot was hitting all these sites at a very rapid rate.

Then I realized what was going on: since JSPWiki offers an RSS feed for every single page (so that you can follow the changes to any page with your aggregator), the MSN bot wants to download them all, every four hours. So, for jspwiki.org, I get 2000 hits every four hours, at very rapid intervals. Because the MSN bot does not seem to support If-Modified-Since header, I end up sending a HUGE amount of data every day, just to satisfy one bot. Our backend is simply not designed to work well under such conditions: we do cache (because it makes sense for the browser-based interface), but we're not doing memory (or disk) caching of old versions or diffs, so all requests for these go to a back end. That means roughly 50,000 processes created every four hours within about fifteen minutes. And that's just killing the server - amazing it has been up even this much.

So, as a temporary solution I'm going to put RSS feeds of my server to /robots.txt, so that these guys stop indexing them. As a long-term solution I'm going to start to cache the RSS feeds as well.

Update: It's not quite 50,000 processes in 15 minutes, but 200,000 processes/day. Made a script mistake, oops. Still, MSNBot's RSS scanner can cause quite a lot of heavy traffic, if you're not prepared, or you have not designed your back end for such access patterns.