MemeCode - Blog

Rise Of The Machines

Date: 22/4/2005

After yesterday's spike of website hits I thought I'd look into where that was comming from, and no referer spiked up to explain it. But the number of "bots" hitting the site had gone up to some 30% of the page loads on www.memecode.com, and I don't know about you, but thats er kinda high isn't it?

I've starting tracking the bots by counting their hits per useragent string. And obviously Googlebot and Msnbot are leading the race early on but I suspect a rouge bot 'telnet0.1 noone@example.org' is responsible for yesterday's spike. I have in the past banned IP's due to the shear number of incomming hits for no apparent reason. Then of course Googlebot itself decided to hit my site over 61000 times in a 24 hr period some time ago now.

Is it just me, or does 30% of your site traffic being eaten away by bots just annoy you?

On a somewhat related note, I've also instituted a kill file for porn / scam sites that spam my referer log to help boost their Google ranking. I might add that I've also set my robots.txt file to stop scanning of the stats anyway, so even if you evade the kill file you won't receive any benifit from getting listed as a referer. I suspect that these sites are inserting themselves as the referer by infecting machines with spyware that frigs with IE's outgoing referer field thus littering the web's stats pages with their URL, which in turn makes their Google ranking grow. But I have no conclusive evidence that happens, but it's my current theory.

Comments:

Glenn
01/05/2005 3:48pm

Does anyone know who 'telnet0.1 noone@example.org' really is? Where does this bot live? example.org in not a valid domain.

fret
03/05/2005 12:04am

Well time did tell a tale didn't it?

It seems the most offensive bot is Yahoo! Slurp by a decent margin. I will look into curtailing it's abuse of my site.

From:

Email (optional):

(Will be HTML encoded to evade harvesting)

Message:

Remember username and/or email in a cookie. Notify me of new posts in this thread via email.	BBcode: [q]text[/q] [url=link]description[/url] [img]url_to_image[/img] [pre]some_code[/pre] [b]bold_text[/b]

Sitemap