That does it!

I'm blocking search bots from spidering my archives directory. That means the only thing the search bots will be allowed to spider is my main page and my skins directory. The latest "gr4nny s3x" search from an obscene URL did it. I know this sounds draconian, but I feel I have no other choice.

In case you're interested, you can disallow search bots from spidering certain directories using a "robots.txt" file. You can even allow or disallow specific search bots. Here is a site that explains all about robots.txt. Here is another good site on the basics of configuring robots.txt. And here is a robots.txt validator.

It seems that it takes a while for the changes to take effect. Some time ago, I blocked search bots from my cgi-bin directory to prevent searches from serving up my trackback boxes (I know some people are annoyed by that aspect of MT). It took a couple of months for me to quit getting referrals from my trackback boxes.

I also blocked spidering of all my image directories. And I fixed my .htaccess so that people can't hotlink to my images. You can also fix your .htaccess file to forward people coming from certain URLs to other sites. For example, the perv searchers who are coming here from perv URLs I've found in my referrals get sent here instead. Here's a guide to configuring .htaccess.