Unhandled Perception
From the mind of a developer.

Monday, April 14, 2008

Yahoo! Slurp 3.0

Over the past few weeks, we've [Yahoo!] been preparing for the latest version of the Yahoo! Search crawler with some infrastructure updates, which recently caused a variance in our crawl behavior.

As the new software undergoes a phased rollout to our production crawlers over the next several weeks, you'll see the following changes:

a) The crawlers will start crawling from a different and much smaller set of IP addresses, but it'll still be from the crawl.yahoo.net domain.

b) The crawlers will also publish a new user-agent, 'Yahoo! Slurp/3.0.'

Link:
http://www.ysearchblog.com/archives/000531.html

Labels: ,

0 Comments:

Post a Comment

Links to this post:

Create a Link

<< Home