Shadow traffic to a new model

  • Build
  • Model 07

Difficulty 3 of 5

A new fraud model must score a copy of every live request for a week before it can replace the live model. It is slower and not yet stable. The API sends each request to every model link. Keep user p99 below 30 ms with no errors, and keep the copies at most 30 s behind.

Play level

HTTP/2gRPCUsers500 req/sAPI server2 vCPU · 2 GBModel (CPU)4 vCPU · 4 GBModel (CPU)12 vCPU · 12 GB
HTTP/2gRPCUsers500 req/sAPI serverModel (CPU)Model (CPU)
The board at the start of the level
A canary before the whole fleetTap to change the deployment method

Canary · 5% of requests fail for 15 seconds · 750 failed requests

Goals and limits

p99 latency
under 30 ms
Error rate
under 0.1%
Queue lag
under 30 s
Monthly cost
under 100 USD/mo
Parts you can add
Message broker, Managed queue, Kafka log, Worker
Part limit
4
Traffic
500 req/s

Concepts

How players did

Player results load from the server.

All levels