Mark Pilgrim Battles The Web Robots
26 February 2003 · Last updated: 15 October 2006
No, not a new album by the Flaming Lips, but another killer post on Dive Into Mark: How to block spambots, ban spybots, and tell unwanted robots to go to hell. Mark has taken on the programs that search his website daily, pushing up his bandwidth.
"TurnitinBot hit me a whopping 19,575 times in January... They're sucking down over a gigabyte of data a month for no purpose that even remotely benefits me or any of my readers."
So he's taken them all on, banning each guilty program with typically clever code. He admits it's not a battle he can win, but others too have been tackling this problem. Read the whole story and you too will want to take action.
Useful Info
EMAIL: www.designdetector.com (replace the 1st dot with "@")
NEWSFEED: Subscribe to news of fresh posts and site updates. (RSS 2.0 compatible newsreader required.)
Disclaimer
Some links on this website lead to information provided by external services not under my control, therefore I am not responsible for the content or accuracy of the linked information.
All code examples are not guaranteed 100% free from bugs and/or mistakes. Use them at your own risk. I do not take any blame should a problem occur relating to use of code on this site given as an example for your own use. Such code has been tested and found to work for me, but I cannot vouch for other computer systems (existing now, or in the future) which it may be used on, or changes you introduce yourself based on my code.
This website is © 2008 Christopher Hester, except where separate authors are named. No part of this website may be reproduced or re-used in any way without my prior permission, except content added from separate authors (who retain the copyright on their material), examples of code, and any other content I explicity state is free to copy and make use of.
This page was last updated on 15 October 2006.
