I enabled autoscaling to handle traffic spikes.
Instead of improving performance, latency increased.
Cold starts seem frequent.
This feels counterproductive.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I enabled autoscaling to handle traffic spikes.
Instead of improving performance, latency increased.
Cold starts seem frequent.
This feels counterproductive.
Autoscaling can introduce cold start penalties if not tuned correctly.
Model loading and initialization are often expensive. When new instances spin up under load, they may take seconds to become ready, increasing tail latency.
Pre-warm instances or use minimum replica counts to avoid frequent cold starts. Also measure model load time separately from inference time.
For large models, consider keeping them resident in memory or using dedicated inference services.