Spiders, spiders everywhere (but not what you think)

While monitoring my new security configuration in Apache, I've noticed a lot of hits by spiders - automated bots that crawl the Web (see how the spider lingo works?) and catalog your site for various search engines.

Google and MSN Search seem to be the most frequent visitors ("msnbot" and, more cutesy, "googlebot"). In fact, their cataloging power is a little frightening, especially in MSN's case.

Go to http://search.msn.com and enter "Keith, Jenny, Lisa, Andrea's b-day party". Or "Caden discovers my camera strap". It's a little scary how meticulous MSN is about archiving any text on your site.

The spiders account for enough traffic that I've split them off into their own logfile. This way there's less to sort through when I want to monitor traffic by real human beings.

Oh - coincidentally there was a real spider I had to kill near my closet today. But that's hardly newsworthy...