Crawling and inverted indexes
Web search: crawl and index
- One pace per site group
- Bloom filter for seen URLs
- Build the index in parallel
- Shard the index, keep two copies
- Final test
Crawl the web politely, skip URLs you know, build an inverted index in a batch window, and answer 36,000 queries/s from a sharded index that survives the loss of a box.
Final testThe second box fails for 60 s while queries rise x1.25. Your final board runs once more with a new traffic seed.
Stage 4 of 4 · 13 parts given · 137,950 requests/s
Progress on this page stays on this device.