Secret rotation
Fetch secrets at run time, then cache them for less than the rotation overlap. With 20 keys and a 4-minute TTL, the secret store gets about 0.1 reads/s instead of 1,500, and each rotation reaches checkout within 4 minutes.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | Every request reads the secret store | Keys cached for 4 minutes |
|---|---|---|---|
| Error rate | under 0.1% | 74.9% Fail | 0% Pass |
| p99 latency | under 100 ms | 10.1 s Fail | 27 ms Pass |
| Data age | under 300 s | 0 ms Pass | 242 s Pass |
| Monthly cost | 20 USD/mo | 21 USD/mo |
Key numbers
| Number | Value |
|---|---|
| Reads per second, one secret store | 10,000 ops/s |
| Secret read, median | 0.1 ms |
| Default time to live of a local cache | 60 s |
Envelope keys · < 1 key service calls/s · limit 10,000/s
How it works
A leaked key must be replaced. The safe place for a key is a secret store, not the code or a config file. The service fetches the key at run time. After a leak you rotate the key: make a new one, and retire the old one.
Rotation has an overlap. The old key keeps working for a short time after the new key exists, so services that still hold the old key do not fail. In the mini board the old key stops 5 minutes after each rotation.
A service that reads the secret store on every request is a problem. In the mini board checkout reads the key on each request, and the store serves 1,000 reads/s. The reads are many more than the store can serve, so errors rise and p99 passes its limit.
Cache the key in each service for a short time. The cache must live for less than the overlap, so that a rotated key is picked up before the old key dies. With a 4-minute time to live, each service asks again every 4 minutes. With 20 keys the store sees about 0.1 reads/s instead of 1,500, and every rotation reaches checkout within 4 minutes, before the old key stops.
If the secret store is down, the services keep running on cached keys. That is another gain.
In the levels you set the cache life from the overlap and check errors and freshness.