Skillz wrote:
it seems the bandwidth is being consumed by indexing bots such as the Google Bot and Yahoo Slurp!
Indeed That happened in my redirect too. I denied viewing files, yes I have the same 403 Forbidden, this is for avoid browsing. Later I decided to go on other port. Google Bot sucked for hours my redirect lagging server. Everything posted in a forum send Google to stole files. I don't know if ever a file can be founded in a certain place like a warehouse, but is slowing down server making a delay in each querry.
For my downloads page I'm using an autoindex script, but I think I was bad this time. I put some policies. I BANNED 3 bots: Google, Yahoo, MSN, they just make only troubles. I don't think was so bad ideea but I noticed a relaxed server now.