What's That Noise?! [Ian Kallen's Weblog]

All | LAMP | Music | Java | Ruby | The Agilist | Musings | Commute | Ball
« Refactoring: making... | Main | I want free WiFi... »

20040324 Wednesday March 24, 2004

I, Robot I took a look at all of the robots that hit my site. It's really astounding how many of these beasties are running around, crawling up and down links old and new.

Some of these robots are very persistent (or just plain dumb). They try to access pages that haven't been around for a long time getting 404's or redirects elsewhere. I'm presuming that after some number N occurences of non-success responses, that these robots will get a clue and just crawl the links that are there... for some reason, new links that've shown up aren't crawled. Here's a list of crawler URL's:

And while I'm ranting about noise in the logs, what's with all of the Nimda type virus attacks? I would've thougt that that is stuff is done and played out already. But no, everyday the onslaught of IIS vulnerability probe requests come hammering my Apache web server instance. What a buncha BS.

Between the robots and viruses, I probably have as many software entities hitting my site as I do human readers. ( Mar 24 2004, 11:01:06 PM PST ) Permalink


Comments:

Post a Comment:

Comments are closed for this entry.