Some optimizations improve throughput but hurt single-request latency.

Batching, quantization, or graph compilation can introduce overhead that only pays off at scale. In low-traffic scenarios, this overhead dominates. Profile latency at realistic request rates and choose optimizations accordingly.

Common mistakes:

1. Optimizing without workload profiling

2. Using batch inference for real-time APIs

3. Ignoring cold-start costs

Optimize for your actual deployment context.