Levels

Daily

Start with the core path. Each chapter opens more worlds.

Start here · Chapter 1, One machine

Overloaded database

Users send 2,000 requests/s.

01 / 41BuildDifficulty

Level 1 of 7 in this chapter

Play

Core path: 0/41
Stars: 0/123
Depth worlds: 0/265

Core path

Chapter 1: One machineFind the bottleneck, read the latency ladder and size a box.0 of 7
1. 01 Overloaded databaseBuildNext
2. 02 The latency ladderOrderLocked
3. 03 Fifty dependent readsEstimateLocked
4. 04 Two boxesBuildLocked
5. 05 How many boxes?EstimateLocked
6. 06 Safe load limitTuneLocked
7. 07 Black FridayBossSurviveLocked
OpensSignal0/10First server, capacity and cost Metal0/12One node, a VPS, racks and the datacenter
One server boardDrag or use arrow keys to change the load
280 req/s · CPU 70% · mean 22 ms
Chapter 2: CachingHit ratio, stale reads, stampedes and the edge.Locked
1. 08 Hit ratioTuneLocked
2. 09 TTL and stale readsTuneLocked
3. 10 StampedeFixLocked
4. 11 Images at the edgeBuildLocked
5. 12 Flash crowdBossSurviveLocked
OpensCacheCaches, CDNs and the edge
Chapter 3: Many machinesScale out behind a load balancer and keep state out of the process.Locked
1. 13 One big or three smallChooseLocked
2. 14 Behind the balancerBuildLocked
3. 15 StatelessBuildLocked
4. 16 URL shortenerBossBuildLocked
OpensScaleLoad balancing, sharding and partitioningWireNetworking and protocols
Chapter 4: QueuesAsync work, requests in flight, backpressure and duplicates.Locked
1. 17 Queue the emailBuildLocked
2. 18 Requests in flightEstimateLocked
3. 19 BackpressureBuildLocked
4. 20 Duplicate deliveryFixLocked
5. 21 Event stormBossSurviveLocked
OpensQueueAsync work, messaging and streamsFlowBatch, streams and analyticsLockSecurity and identityChainPayments and ledgers
Chapter 5: FailureTimeouts, retries, nines and failure domains.Locked
1. 22 No timeoutTuneLocked
2. 23 Retry stormFixLocked
3. 24 Count the ninesEstimateLocked
4. 25 Spread across racksBuildLocked
5. 26 Multi-cause incidentBossFixLocked
OpensGuardTimeouts, retries, breakers and recoveryWatchObservability, deploys and operations
Chapter 6: CopiesReplicas, lag, quorums and network cuts.Locked
1. 27 Read replicasBuildLocked
2. 28 Where is my postFixLocked
3. 29 Sync or async copyBuildLocked
4. 30 Cut the networkBuildLocked
5. 31 Quorum sizesTuneLocked
6. 32 Partition stormBossSurviveLocked
OpensStoreStorage engines and databasesQuorumReplication, consistency and consensus
Chapter 7: PartitionsShard keys, hot keys, adding nodes and tail latency.Locked
1. 33 The shard keyChooseLocked
2. 34 Add a nodeBuildLocked
3. 35 Tail at scaleEstimateLocked
4. 36 Fit the modelChooseLocked
5. 37 Likes at scaleBossBuildLocked
OpensFabricGPU setupsModelMachine learning systemsAgentLLM and agent systems
Chapter 8: Planet scaleLight in glass, paths through space, three continents.Locked
1. 38 Glass or microwaveEstimateLocked
2. 39 Rank long pathsOrderLocked
3. 40 Where space winsTuneLocked
4. 41 Three continentsBossBuildLocked
OpensLightLight in glass, air and vacuumPixelFrontend and mobile systems
End of the core path. Every world opens.

Depth worlds

0/265

Each depth world goes deeper into one topic. A chapter of the core path opens it.

Chapter 1 · One machine

Chapter 2 · CachingLocked

CacheCaches, CDNs and the edge0 of 11
1. 05 Write-through or write-backChooseLocked
2. 06 InvalidateBuildLocked
3. 08 Origin shieldTuneLocked
4. 09 The cache keyFixLocked
5. 10 LRU or LFUChooseLocked
6. 11 The hot keyFixLocked
7. 13 Missing keysFixLocked
8. 15 Personal but cachedBuildLocked
9. 16 Serve staleTuneLocked
10. 17 Where do sessions liveChooseLocked
11. 19 Remove a useless cacheFixLocked
Cache · LockedCaches, CDNs and the edge

Chapter 3 · Many machinesLocked

ScaleLoad balancing, sharding and partitioning0 of 15
1. 02 Uneven workChooseLocked
2. 04 Health checksTuneLocked
3. 05 Sticky troubleFixLocked
4. 06 Boot timeTuneLocked
5. 07 50k readsBuildLocked
6. 09 Range or hashChooseLocked
7. 14 Scatter-gatherFixLocked
8. 16 Hedged requestsTuneLocked
9. 18 Multi-region writesChooseLocked
10. 19 Cell architectureBuildLocked
11. 20 BulkheadsBuildLocked
12. 21 Split the monolith?ChooseLocked
13. 22 Sortable ids without a counterBuildLocked
14. 23 Unique handles across regionsBuildLocked
15. 24 Ten times the trafficSurviveLocked
Scale · LockedLoad balancing, sharding and partitioning
WireNetworking and protocols0 of 21
1. 01 After you press EnterOrderLocked
2. 02 Connection reuseChooseLocked
3. 04 HTTP/2 multiplexingFixLocked
4. 05 HTTP/3 on a lossy linkChooseLocked
5. 06 TLS 1.3 handshakeOrderLocked
6. 07 Zero round tripsChooseLocked
7. 08 Time to liveTuneLocked
8. 09 Closest regionBuildLocked
9. 10 One address, everywhereChooseLocked
10. 11 REST or gRPCChooseLocked
11. 12 Compression levelTuneLocked
12. 13 Polling stormFixLocked
13. 14 One-way newsChooseLocked
14. 15 A million socketsEstimateLocked
15. 16 Live scoresBuildLocked
16. 17 Webhooks that landBuildLocked
17. 18 The long fat pipeEstimateLocked
18. 20 L4 or L7ChooseLocked
19. 21 Loss across the oceanFixLocked
20. 22 N+1 at the gatewayFixLocked
21. 26 200k devicesBuildLocked
Wire · LockedNetworking and protocols

Chapter 4 · QueuesLocked

QueueAsync work, messaging and streams0 of 12
1. 02 Drain the backlogTuneLocked
2. 03 Poison messageFixLocked
3. 05 In orderChooseLocked
4. 07 Lagging consumerFixLocked
5. 08 One event, four servicesBuildLocked
6. 09 Log or queueChooseLocked
7. 10 Log retentionTuneLocked
8. 12 Remind me laterBuildLocked
9. 13 The sagaBuildLocked
10. 14 The outboxBuildLocked
11. 16 Visibility timeoutTuneLocked
12. 17 Priority with fairnessBuildLocked
Queue · LockedAsync work, messaging and streams
FlowBatch, streams and analytics0 of 16
1. 01 ETL or ELTChooseLocked
2. 02 Split the nightly batchBuildLocked
3. 03 Watermark delayTuneLocked
4. 04 Pick the window typeChooseLocked
5. 05 Deduplicate click countsBuildLocked
6. 06 Query the lakeBuildLocked
7. 07 Rows or columnsChooseLocked
8. 08 Partition pruningFixLocked
9. 09 Compact small filesFixLocked
10. 10 Throttle the backfillBuildLocked
11. 11 New fieldFixLocked
12. 12 Trending tags with a sketchBuildLocked
13. 13 Unique visitorsEstimateLocked
14. 14 Bad batchBuildLocked
15. 15 Stream into the analytics storeBuildLocked
16. 16 Year-end data capacitySurviveLocked
Flow · LockedBatch, streams and analytics
LockSecurity and identity0 of 18
1. 02 Stolen tokenFixLocked
2. 03 OAuth code flow with PKCEOrderLocked
3. 04 No more phishingChooseLocked
4. 05 Password hash settingsChooseLocked
5. 06 Limit code guessesBuildLocked
6. 07 Request flood wavesSurviveLocked
7. 08 Bots at the doorBuildLocked
8. 09 Rotate a leaked keyFixLocked
9. 10 Encrypt with data keysBuildLocked
10. 11 Mutual TLS between servicesBuildLocked
11. 12 Scope database rightsFixLocked
12. 13 Tenant data layoutChooseLocked
13. 15 Private downloadsBuildLocked
14. 16 Fetch anythingFixLocked
15. 17 Tamper-evident audit logBuildLocked
16. 18 End-to-end key setupOrderLocked
17. 19 Dependency supply chainChooseLocked
18. 20 Contain a breachFixLocked
Lock · LockedSecurity and identity
ChainPayments and ledgers0 of 16
1. 01 Double entryBuildLocked
2. 03 Card payment statesOrderLocked
3. 04 ReconcileBuildLocked
4. 05 Deposit confirmationsEstimateLocked
5. 06 Fee marketTuneLocked
6. 07 ReorgFixLocked
7. 08 Coins or accountsChooseLocked
8. 09 RPC at scaleBuildLocked
9. 10 IndexerBuildLocked
10. 11 Hot and coldBuildLocked
11. 12 Who signsChooseLocked
12. 13 Stuck noncesFixLocked
13. 15 RollupsChooseLocked
14. 17 Price feedBuildLocked
15. 18 Match engineBuildLocked
16. 20 Market spikeSurviveLocked
Chain · LockedPayments and ledgers

Chapter 5 · FailureLocked

GuardTimeouts, retries, breakers and recovery0 of 15
1. 02 Retry budgetTuneLocked
2. 04 Circuit breakerBuildLocked
3. 05 Token bucket per API keyBuildLocked
4. 06 Shed low-priority loadBuildLocked
5. 07 Last good list as fallbackBuildLocked
6. 08 Single points of failureFixLocked
7. 09 Spread across zonesBuildLocked
8. 10 Warm standby regionBuildLocked
9. 11 Active-active or standbyChooseLocked
10. 12 Deep health checksFixLocked
11. 13 Bulkhead against a cascadeFixLocked
12. 14 Canary with automatic rollbackFixLocked
13. 15 Blue and greenOrderLocked
14. 16 Kill switch deliveryChooseLocked
15. 19 Chaos day: four failure drillsSurviveLocked
Guard · LockedTimeouts, retries, breakers and recovery
WatchObservability, deploys and operations0 of 10
1. 01 Missing metricsFixLocked
2. 02 RED for the service, USE for the diskChooseLocked
3. 03 Metric cardinalityFixLocked
4. 04 Log samplingTuneLocked
5. 05 Find the slow spanBuildLocked
6. 06 Alert fatigueChooseLocked
7. 10 Global config pushFixLocked
8. 11 Next six monthsEstimateLocked
9. 12 Before the usersBuildLocked
10. 14 Checksums and reconciliationFixLocked
Watch · LockedObservability, deploys and operations

Chapter 6 · CopiesLocked

StoreStorage engines and databases0 of 20
1. 01 Pick the storeChooseLocked
2. 02 The missing indexFixLocked
3. 03 Unused indexesTuneLocked
4. 04 B-tree or LSMChooseLocked
5. 05 Compaction debtFixLocked
6. 09 One hot rowFixLocked
7. 11 Page 5,000FixLocked
8. 12 Rename a column liveOrderLocked
9. 13 DenormalizeChooseLocked
10. 14 Any attributeBuildLocked
11. 15 Points per secondBuildLocked
12. 16 Find the postBuildLocked
13. 17 Near meBuildLocked
14. 18 Blob or rowChooseLocked
15. 19 Cold storageEstimateLocked
16. 20 Backups that countBuildLocked
17. 22 Forget meFixLocked
18. 23 The ledger logBuildLocked
19. 24 Follow the changesBuildLocked
20. 26 Wide rowsBuildLocked
Store · LockedStorage engines and databases
QuorumReplication, consistency and consensus0 of 18
1. 02 Promoted too soonFixLocked
2. 03 Two leadersFixLocked
3. 05 Election timeoutTuneLocked
4. 06 Raft electionOrderLocked
5. 10 Cart or balanceChooseLocked
6. 11 Is it linearizableChooseLocked
7. 13 Reads go back in timeFixLocked
8. 14 Skewed clocksFixLocked
9. 15 Lamport timeOrderLocked
10. 16 Lost updateFixLocked
11. 17 Write skewFixLocked
12. 18 Double bookingFixLocked
13. 19 Isolation levelsChooseLocked
14. 20 Two-phase commitOrderLocked
15. 22 The lock with a tokenBuildLocked
16. 24 A counter with no coordinationBuildLocked
17. 25 Locks, OT or CRDTChooseLocked
18. 26 Lease timeoutTuneLocked
Quorum · LockedReplication, consistency and consensus

Chapter 7 · PartitionsLocked

FabricGPU setups0 of 9
1. 01 Weights plus KV cacheEstimateLocked
2. 02 Decode reads every weightEstimateLocked
3. 04 Ring all-reduce timeEstimateLocked
4. 05 Micro-batches fill the pipelineTuneLocked
5. 06 Tensor parallel inside the serverBuildLocked
6. 07 Drain the stragglerFixLocked
7. 08 Rail-aligned leaf switchesBuildLocked
8. 09 Stream the checkpointsFixLocked
9. 10 A week of trainingSurviveLocked
Fabric · LockedGPU setups
ModelMachine learning systems0 of 19
1. 01 Nightly scores or live rankingChooseLocked
2. 02 Online feature storeBuildLocked
3. 03 One path for training and servingFixLocked
4. 04 From the futureFixLocked
5. 05 DriftFixLocked
6. 06 The thresholdTuneLocked
7. 07 Shadow traffic to a new modelBuildLocked
8. 08 How long to testEstimateLocked
9. 09 Canary modelOrderLocked
10. 10 CPU or GPU for boosted treesChooseLocked
11. 11 Batch size inside 50 msTuneLocked
12. 12 Retrieve then rankBuildLocked
13. 13 Recall vs speedTuneLocked
14. 14 Batch embedding on GPUsBuildLocked
15. 15 Cold start for new itemsChooseLocked
16. 16 Feedback loopFixLocked
17. 17 Late labelsChooseLocked
18. 18 Feature refresh intervalTuneLocked
19. 20 Shifting fraud attacksSurviveLocked
Model · LockedMachine learning systems
AgentLLM and agent systems0 of 22
1. 01 Tokens cost moneyEstimateLocked
2. 03 Batch size on one GPUTuneLocked
3. 04 KV memoryEstimateLocked
4. 05 Serve one model on two GPUsBuildLocked
5. 06 Cache the shared prompt prefixTuneLocked
6. 07 Send each request type to its own modelBuildLocked
7. 08 Model provider failoverFixLocked
8. 09 Bounded queue under a token limitBuildLocked
9. 10 Embed on write, retrieve on readBuildLocked
10. 12 Hybrid searchBuildLocked
11. 13 Semantic cache and entry lifetimeTuneLocked
12. 14 Protect each tool by its failure modeBuildLocked
13. 15 Agent loop limitsFixLocked
14. 16 Resume agent tasks after a crashBuildLocked
15. 17 Idempotency key for a tool callFixLocked
16. 18 Warm sandboxes for generated codeBuildLocked
17. 19 Separate page reading from toolsFixLocked
18. 20 Separate queue for human approvalsBuildLocked
19. 21 Run evals off the serving pathBuildLocked
20. 22 One agent, a planner, or a teamChooseLocked
21. 23 What to store as agent memoryChooseLocked
22. 24 Protect a fixed GPU fleet at launchSurviveLocked
Agent · LockedLLM and agent systems

Chapter 8 · Planet scaleLocked

LightLight in glass, air and vacuum0 of 9
1. 03 Fiber route factorChooseLocked
2. 04 The latency floorEstimateLocked
3. 05 Microwave or fiberChooseLocked
4. 06 Up and down legsEstimateLocked
5. 08 Laser path across the AtlanticBuildLocked
6. 09 Hollow-core fiberChooseLocked
7. 10 Leader next to its majorityBuildLocked
8. 11 Subsea cable cutSurviveLocked
9. 12 Lunar relay outboxBuildLocked
Light · LockedLight in glass, air and vacuum
PixelFrontend and mobile systems0 of 12
1. 01 Where the page is renderedChooseLocked
2. 02 JavaScript budgetTuneLocked
3. 03 Heavy imagesBuildLocked
4. 04 Request waterfall on loadFixLocked
5. 05 Backend for frontendBuildLocked
6. 06 Offline edits and mergeBuildLocked
7. 07 Endless feedBuildLocked
8. 10 Optimistic update or waitChooseLocked
9. 12 Old app versionsFixLocked
10. 13 Push campaign under provider limitsBuildLocked
11. 14 Reliable chat sendBuildLocked
12. 16 10k ticksBuildLocked
Pixel · LockedFrontend and mobile systems