version: desktop | mobile

Dave's Free Press

the Slightly Less Bad And Wrong Blog Edition

 

Recent posts

Less important journals

Search




Subscribe

rss

Journals that link here

Powered by

Bryar 3.0!
geeky silly politics rant weird meta religion music transport london culture drinking perl olympics photography language cooking review hacking books web spam sport media rsnapshot holidays film bastards whisky bryar yapc palm telecoms cars travel maths etiquette clothes lolcats privacy amazon curry security home unix phone go environment bbc latin anglo-saxon linux gps deafness work radio
Thu, 3 Apr 2008

Thanks, Yahoo!

I'd like to express my warm thanks to the lovely people at Yahoo and in particular to their bot-herders. Until quite recently, their web-crawling bots had most irritatingly obeyed robot exclusion rules in the robots.txt file that I have on CPANdeps. But in the last couple of weeks they've got rid of that niggling little exclusion so now they're indexing all of the CPAN's dependencies through my site! And for the benefit of their important customers, they're doing it nice and quickly - a request every few seconds instead of the pedestrian once every few minutes that gentler bots use.

Unfortunately, because generating a dependency tree takes more time than they were allowing between requests, they were filling up my process table, and all my memory, and eating all the CPU, and the only way to get back into the machine was by power-cycling it. So it is with the deepest of regrets that I have had to exclude them.

Cunts.

Posted at 21:27:22 by David Cantrell
keywords: geeky | meta | perl | rant
Permalink | 0 Comments

Archive

2007

Jan Feb Mar Apr
May Jun Jul Aug
Sep Oct Nov Dec
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan

2006

Jan Feb Mar Apr
May Jun Jul Aug
Sep Oct Nov Dec
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan

2005

Jan Feb Mar Apr
May Jun Jul Aug
Sep Oct Nov Dec
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan