Canary with automatic rollback

Guard 14
HTTP/2HTTP/2SQLUsers1,000 req/sLoad balanc…1 vCPU · 1 GBNode API2 vCPU · 2 GB × 3Postgres2 vCPU · 4 GBAlerts1 vCPU · 1 GB
HTTP/2HTTP/2SQLUsers1,000 req/sLoad balan…1 vCPU · 1 GBNode APIPostgres2 vCPU · 4 GBAlerts1 vCPU · 1 GB

Fix the board. Turn every goal green.

Traffic
1,000 req/s · 90% reads
Limit
Node API answers about 6,000 req/s
Bill
  • Load balancer (L7)4 USD/mo
  • Node API24 USD/mo
  • Postgres16 USD/mo
  • Alerts4 USD/mo
48 of 60 USD/mo

Goals

Not tested yet

The share of requests that fail or get no answer.99 of 100 requests finish faster than this.What the servers and services cost each month.

At 0:20 a new build reached every API server at once. The API ran 10 times slower and 0.5 percent of requests failed until a person rolled it back at 1:50. A bad build may reach at most 5 percent of traffic and must be undone in 30 s. Errors under 0.1 percent, p99 under 40 ms.

0:20. p99 almost doubles on every API server right after a deploy, and errors rise a little. Graph: p99 by build.

A canary before the whole fleetTap to change the deployment method

Canary · 5% of requests fail for 15 seconds · 750 failed requests