The batch prediction job used to run in minutes.
As data volume increased, runtime started doubling unexpectedly.
Nothing changed in the model code itself.
Now it’s becoming a bottleneck in the pipeline.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
The batch prediction job used to run in minutes.
As data volume increased, runtime started doubling unexpectedly.
Nothing changed in the model code itself.
Now it’s becoming a bottleneck in the pipeline.
This usually happens when inference is accidentally performed row-by-row instead of in batches.
Many ML frameworks are optimized for vectorized operations. If your inference loop processes one record at a time, performance degrades sharply as data scales. This often sneaks in when inference logic is written similarly to training notebooks.
Check whether predictions are made using batch tensors or DataFrames instead of Python loops. For example, pass entire arrays to
model.predict()rather than iterating over rows.Also verify I/O behavior. Reading data from object storage or databases inside tight loops can be far more expensive than the model computation itself.