Some friendly advice to operators of robots, spiders and crawlers

Why this site appears to be s-l-o-w

The server which these web pages reside on gets very little real traffic. Combined, all the various domains, email accounts and other services generates a little less than a gigabyte worth of traffic every month. Yup, about a CD and a half is transferred from glide (our server) to the internet as a whole during any given 30-day window.

However, almost 60% of this traffic is robots, spiders and crawlers, generating search indexes for an ever increasing number of competing search engines all over the world. In other words, almost 60% of the bandwidth bill (and an even bigger perecntage is we're talking about page loads) is used up by automated "surfers", looking for content to add to their indices. We have no problem with search engines, only with bandwidth, so we have taken a few steps to ensure there are bandwidth left for those who are actually using the various services here. We don't want to wait for our webmail just because some guy in Japan thinks the pictures in our gallery is worth including in his search engine. He can wait. We probably won't.

Every search engine (and some RSS aggregators) we can identify is limited to a max 2 kbit/s upstream with a 660 ms delay. That is, the bandwidth allotted for sending information back to you. Nothing more, nothing less.

Thank you.