Canary model

  • Order
  • Model 09

Difficulty 3 of 5

A new ranking model has completed training. Order its rollout. The held-out test, shadow check, and canary gates all pass. The policy requires each gate before the next stage and keeps the old model ready for rollback.

Play level

Put these steps in order

  • Expand to all traffic and keep monitoring and rollback ready
  • Finish the required canary window and compare all gates
  • Send a small live group to the candidate and retain the baseline
  • Run the candidate on live inputs in shadow and pass its gates
  • Pass the held-out evaluation against the current model
  • Define quality, error, latency, cost, and rollback gates
A canary before the whole fleetTap to change the deployment method

Goals and limits

p99 latency
under 200 ms
Monthly cost
under 10 USD/mo
Traffic
100 req/s

Concepts

How players did

Player results load from the server.

All levels