its exactly what you say....traffic from SE....those bots eat around 200Gb every month. Without them the server would be 20% faster. For a simple page its maybe nice if the bots update every two days. But this site currenlty is 150,000 files and 2Gigs in size. And each and every single bot drops in whenever he pleases to lurk 2Gig archives, because most boardowners have the standard "revisit after 2 day's" in their html. I don't think we need to be afraid the SE overlook us.

Not to speak of the countless SE that ignore robots.txt...or think they rule the net and do a ../slurpconfirm request to see if your site responces with a "valid" 404...if not they kick you to page 25 on their SE.

Its left or right and we can't go both ways. We all want a fast server. But if we all want to squize every possible hit out of SE we can get...and have 100 toplevels online...with 500 replies in each toplevel..?? I walked around the last few days and there are many boards where i didn't even hit the bottom after over 100 page-downs.

Six months archives is still some 75,000 posts. Every possible keyword is hundreds of times present in it. And at least the posts still contain working links. Lurkers are too lazy to surf archives anyway. The poster has nothing to complain eather because his posts remain at least over a month at the boards. And the SE, i almost should say, can kiss my ass :-)

Take care
See ya, Killer

• Time to clean out the archives again..... - Webmaster 04:18:15 07/03/2008 (8)
  • HAHA I was bogging down sff... - Amateur 01:23:14 07/05/2008
  • Done - mrwhippy 10:01:48 07/04/2008
  • Re: Time to clean out the archives again..... - Overload 21:19:59 07/03/2008
    • Re: yes, but - Webmaster 02:56:29 07/04/2008
      • Re: yes, but - Overload 21:21:56 07/07/2008
        • Re: sure - Webmaster 22:20:57 07/07/2008
          • Re: sure - Webmaster 01:45:45 07/08/2008
            • Re: sure - Amateur 01:53:07 07/08/2008