MLOps 05 - Model Serving and Deployment Strategies

A Finished Model Is Not a Finished Product

When model training is complete, you're left with a checkpoint file boasting high accuracy. But that file alone creates no business value. Only when a serving system is in place — one that accepts user requests and returns predictions — does the model begin to matter.

Model serving is a far more involved problem than simply loading a model into a Flask server. It demands production-grade answers to latency, throughput, concurrency, model versioning, and zero-downtime deployments. This post walks through the choices you need to make, from inference modes to serving frameworks to deployment strategies.

Real-Time vs. Batch Inference

Model serving broadly falls into two categories: real-time inference and batch inference.

Real-time inference returns predictions immediately as each request arrives. It's used in services where response speed is critical — recommendation engines, fraud detection, chatbots. Requests come in via REST API or gRPC, and responses must go back within tens to hundreds of milliseconds.

Batch inference processes large volumes of data all at once. Calculating churn probabilities for every customer overnight, or refreshing a product recommendation list on a weekly basis, are typical examples. Throughput matters more than response speed, and batch inference is commonly paired with frameworks like Spark or Airflow.

Does every service need real-time inference? Not at all. If the result can arrive hours later without consequence, batch inference is far more cost-effective. On the other hand, if a user is staring at a screen and even a few seconds of delay is unacceptable, real-time inference becomes unavoidable.

REST API Serving Patterns

The most common approach to real-time inference is a REST API. A server loads the model, exposes an HTTP endpoint, and returns predictions in JSON when the client sends input data in JSON.

# Simple serving example with FastAPI
from fastapi import FastAPI
import joblib

app = FastAPI()
model = joblib.load("model.pkl")

@app.post("/predict")
async def predict(data: dict):
    features = preprocess(data)
    prediction = model.predict(features)
    return {"prediction": prediction.tolist()}

This approach is perfectly adequate for prototyping, but it hits walls in production. Model loading time, memory management, concurrent request handling, and GPU utilization all become problems you have to solve yourself. This is precisely why dedicated serving frameworks emerged.

Model Serving Frameworks

Several frameworks exist for production-grade model serving. Understanding the characteristics of each and choosing the right one for the situation is essential.

TensorFlow Serving is the most mature serving solution within the TensorFlow ecosystem. It loads SavedModel format directly and natively supports model versioning and hot-swapping. However, it's not well suited for serving models trained with other frameworks.

NVIDIA Triton Inference Server stands out for its ability to serve models from TensorFlow, PyTorch, ONNX, and other frameworks on a single server. Through dynamic batching, it collects individual requests and processes them on the GPU in a single pass, maximizing throughput. It also supports model ensembles and pipeline configurations, making it a strong choice when complex inference workflows are required.

# Triton model repository structure
model_repository/
├── text_classifier/
│   ├── config.pbtxt
│   ├── 1/
│   │   └── model.onnx
│   └── 2/
│       └── model.onnx

BentoML focuses on lowering the barrier to entry for model serving. With decorator-based API definitions, automatic container builds, and adaptive batching, it enables data scientists to build serving systems without deep infrastructure knowledge.

Latency Optimization

In a serving system, latency directly affects user experience. You need to manage not just the model inference time itself, but the total response time — including data preprocessing, network communication, and postprocessing.

There are several approaches to reducing latency. On the model optimization side, quantization converts FP32 models to INT8 to speed up inference, and inference optimization engines like ONNX Runtime or TensorRT can be leveraged. On the infrastructure side, keeping the model resident in GPU memory eliminates loading overhead, and autoscaling during peak hours maintains consistent response times.

Should you apply every optimization at once? No. The rational approach is to first profile and identify the bottleneck, then optimize the area with the greatest impact.

Containerization

Containerizing a model serving system is effectively a prerequisite for production deployment. Packaging model code, dependencies, and model artifacts into a single Docker image guarantees consistency across environments.

FROM python:3.11-slim

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY model/ /app/model/
COPY serve.py /app/

WORKDIR /app
CMD ["uvicorn", "serve:app", "--host", "0.0.0.0", "--port", "8080"]

Image size directly affects deployment speed. Removing unnecessary dependencies, using multi-stage builds, and loading model files from outside the image when they're large are all considerations. When using GPUs, NVIDIA CUDA base images are necessary, but since these can reach several gigabytes, layer caching strategies become even more important.

Deployment Strategies

Switching all traffic to a new model at once when updating is risky. The new model could have unexpected issues. Safe deployment requires deliberate strategies.

Canary deployment applies the new model to only a fraction of traffic (say 5-10%) first, monitors performance metrics, and gradually increases the traffic share if no problems surface. The advantage is the ability to roll back quickly if the new model's accuracy drops below the existing model or latency spikes.

A/B testing runs two or more models simultaneously and compares business metrics. While canary deployment focuses on technical stability, A/B testing evaluates models based on business outcomes. Comparing click-through rates in a recommendation system, or measuring conversion rates for a search model, are classic examples.

                    ┌──── Model v1 (90%) ────┐
User Request ──→ Load Balancer ──┤                          ├──→ Response
                    └──── Model v2 (10%) ────┘
                         (Canary)

Blue-green deployment prepares the existing environment (blue) and the new environment (green) simultaneously, then switches traffic over all at once. If problems arise, you can immediately revert to the blue environment. It's operationally simpler than canary deployment, but comes with the tradeoff of requiring double the infrastructure cost.

Summary

Model serving is the process of converting trained models into real-world value. The key is selecting the right inference mode — real-time or batch — leveraging production-grade serving frameworks, and applying safe deployment strategies. Containerization ensures environment consistency, and latency optimization guarantees a solid user experience.

In the next post, we'll look at monitoring strategies for deployed models and how to detect data drift.