Defeating Bad Web Robots with Apache

Lee Killough has an absolutely great page on tips and tricks for Apache that will help you defeat web robots that don’t respect your robots.txt file.

Seeing the different usages of mod_rewrite may also help me hide some of the cgi scripts I use behind other URLs that are shorter and easier to remember.

Thanks DiveIntoMark for the link.

Friday February 21, 2003   ·   Permalink

---