Overview

Deploying ML models to production is a different discipline from building them. A model that achieves 95% accuracy in a Jupyter notebook can fail in ways that have nothing to do with the model itself, slow inference, memory leaks, model staleness, and concurrency bugs all become real problems at scale.

FastAPI has emerged as the default serving framework for Python ML teams. It's async-native, automatically generates OpenAPI documentation, and integrates naturally with Python's ML ecosystem.

Model Loading Strategy

The most common production mistake is loading the model inside the request handler. This creates a new TensorFlow session on every request, catastrophically slow.

Load the model once at startup using FastAPI's lifespan context manager:

The model lives in application state. Request handlers access it through FastAPI's dependency injection. This pattern supports hot-reloading (swap the model object without restarting the server) and multiple model versions in parallel.

Async Inference with a Thread Pool

TensorFlow's model.predict() is synchronous and CPU/GPU bound. Calling it directly in an async FastAPI handler blocks the event loop, destroying concurrency.

The fix: run inference in a thread pool executor. FastAPI's run_in_executor pattern delegates the blocking call to a thread while the event loop remains free to handle other requests.

Batching for Throughput

Single-sample inference is wasteful, the GPU (or CPU) is underutilised. Production systems batch multiple requests together and run a single inference pass.

Implement a request queue with a background task that collects requests for 10ms, then runs a single batched inference call. At moderate traffic levels, this can multiply throughput 8, 12x.

Monitoring Model Health

Beyond standard API metrics (latency, error rate), ML serving requires domain-specific monitoring:

• Prediction distribution, alert if output distribution shifts significantly (model degradation or data drift)
• Inference latency p99, slow inference often precedes OOM errors
• Input validation failures, unexpected input shapes signal upstream data pipeline issues

We use Prometheus + Grafana for all ML serving dashboards at Code Quore.