Ask Sadie McCarthy a questionLost your password? Please enter your email address. You will receive a link and will create a new password via email.
Ask Sadie McCarthy a questionPlease briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
Why do my experiment results look inconsistent across runs?
This is often caused by uncontrolled randomness in the pipeline. Random seeds affect data splits, model initialization, and even parallel execution order. If seeds aren’t fixed consistently, results will vary. Set seeds for all relevant libraries and document them as part of the experiment. Also cheRead more
This is often caused by uncontrolled randomness in the pipeline. Random seeds affect data splits, model initialization, and even parallel execution order. If seeds aren’t fixed consistently, results will vary.
Set seeds for all relevant libraries and document them as part of the experiment. Also check whether data ordering or sampling changes between runs. In distributed environments, nondeterminism can still occur due to hardware or parallelism, so expect small variations.
Common mistakes include: Setting a seed in only one library, Assuming deterministic behavior by default and Comparing runs across different environments
The takeaway is that reproducibility requires intentional control, not assumptions.
See lessWhy does my retrained model perform worse than the previous version?
More recent data does not automatically mean better training data. If the new dataset contains more noise, label errors, or short-term anomalies, the model may learn unstable patterns. Additionally, changes in class balance or feature availability can negatively affect performance. Compare the old aRead more
More recent data does not automatically mean better training data.
If the new dataset contains more noise, label errors, or short-term anomalies, the model may learn unstable patterns. Additionally, changes in class balance or feature availability can negatively affect performance.
Compare the old and new datasets directly. Look at label distributions, missing values, and feature coverage. Evaluate both models on the same fixed holdout dataset to isolate the effect of retraining.
If the model is sensitive to recent trends, consider weighting historical data rather than replacing it entirely. Some systems benefit from gradual updates instead of full retrains. The takeaway is that retraining should be treated as a controlled experiment, not an automatic improvement.
See less