Mark Pilgrim Battles The Web Robots
26th February 2003 · Last updated: 5th October 2016
No, not a new album by the Flaming Lips, but another killer post on Dive Into Mark: How to block spambots, ban spybots, and tell unwanted robots to go to hell. Mark has taken on the programs that search his website daily, pushing up his bandwidth.
"TurnitinBot hit me a whopping 19,575 times in January... They're sucking down over a gigabyte of data a month for no purpose that even remotely benefits me or any of my readers."
So he's taken them all on, banning each guilty program with typically clever code. He admits it's not a battle he can win, but others too have been tackling this problem. Read the whole story and you too will want to take action.