A web crawler
Every question from this chapter’s lessons, grouped by the lesson that teaches it. Answer each one right to finish; you can try again whenever you like.
0 of 18 answered right
11.1 Requirements, estimates and a crawler's parts
0 of 5 answered rightQuestion 1
The primer’s crawler makes 4 billion crawls a month. Using 400 requests per second ≈ 1 billion a month, how many pages does it crawl per second?
pages/s
Answers within 5% count.
Question 2
At 4 billion crawls a month and 500 KB stored per page, how much page content is stored per month, in PB?
Answers within 5% count, in any unit you choose.
Question 3
At 1,600 pages per second and 500 KB per page, how much download bandwidth does the crawler need, in Gbit/s?
Answers within 5% count, in any unit you choose.
Question 4
Put the steps of the primer’s crawler loop in order, for a page that turns out to be new.
Question 5
How long does RFC 9309 say a crawler should, at most, keep using a cached copy of a site’s
robots.txt?11.2 The URL frontier and politeness
0 of 4 answered rightQuestion 6
Why does Mercator’s frontier have front queues and back queues?
Question 7
Mercator waits 10 times as long as a download took before contacting the same host again. A page from a host took 2 seconds to download. How long until the crawler may contact that host again?
Answers within 2% count, in any unit you choose.
Question 8
A crawler never lets two threads contact the same server at once, yet site administrators still complain about it. Why?
Question 9
A distributed crawler assigns each host to one crawling process by hashing the host name. What is the better way to hash, if processes will be added and removed?
11.3 Duplicate elimination and SimHash
0 of 5 answered rightQuestion 10
Why use SimHash rather than a cryptographic hash such as SHA-1 to find near-duplicate pages?
Question 11
A Bloom filter must hold 1 billion URLs with a 0.1% false-positive rate. Using bits, about how much memory does it need, in decimal GB ( bytes)?
Answers within 5% count, in any unit you choose.
Question 12
To find 64-bit fingerprints within 3 bits of a new one, the bits are split into 4 equal blocks so that at least one block must match exactly. How many bits is each block?
bits
Answers within 1% count.
Question 13
A table of fingerprints is keyed on 16 bits. About how many candidates does one lookup return?
candidates
Answers within 5% count.
Question 14
A crawler uses a Bloom filter for “seen this URL before?”. Some brand-new pages are never crawled. Why?
11.4 Fetching, DNS caching and fault tolerance
0 of 4 answered rightQuestion 15
Mercator’s threads spent 70% of their time on DNS lookups. What fixed it?
Question 16
A site’s calendar has a “next month” link on every page, forever. How do you stop the crawler following it endlessly?
Question 17
How do crawlers usually store billions of downloaded pages?
Question 18
Some pages change every hour and others once a year. How should the crawler schedule re-crawls?