CPU or GPU for boosted trees

Model 10

Choose

A fraud model with 400 boosted trees scores 20,000 payments/s. One score takes about 1 ms on one CPU core. Each score must return within 25 ms at p99. Budget 600 USD/mo. Choose how to serve the model.

Goals

Not tested yet

99 of 100 requests finish faster than this.The share of requests that fail or get no answer.What the servers and services cost each month.
Choose a design
One GPU server with batches of 256
One GPU server, one request at a time
Eight CPU replicas with 4 vCPU each