Honest ML: How a Lucky Train Test Split Can Fool Your Model
In a session at the Data Science Innovators Conference 2023, Eric Schles of Johns Hopkins Hospital showed how varying the random seed on a train test split can inflate a model's score. His Honest ML framework tests hundreds of splits, then he built a neural network from the linear algebra up.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- One split can mislead. Changing the random seed on a train test split changes a model's score, so a single number can reflect luck.
- Test hundreds of splits. Schles said 200 to 400 trials give far better confidence than five random splits.
- Show the distribution. Honest ML plots precision and recall across all splits rather than reporting one figure.
- Small data needs it most. With only about 200 data points in medical studies, cross-validation is limited.
- Check error against scale. Metrics like MSE are unitless, so judge them against the units of the use case.
At the Data Science Innovators Conference 2023, Eric Schles, principal data scientist at Johns Hopkins Hospital, argued that many machine learning teams trust accuracy numbers that came from a lucky train test split. In a hands-on session on "Honest ML with TensorFlow and PyTorch," he presented an open-source framework, Honest ML, that tests a model across hundreds of splits, then walked through how neural networks work from linear algebra up.
Why can a lucky train test split fool you?
Schles said the problem starts with something simple: varying the random seed on a scikit-learn train test split, without changing any hyperparameters, changes a model's score. In his words, virtually every model shows that deviation. His question to the audience was how anyone can trust the accuracy reported for a single model.
He described a recent project at work. A linear model looked excellent, and colleagues were excited about publishing results. Skeptical, Schles ran the model through his library. "It turns out we got a lucky train test split," he said, and he called that result terrifying.
He said he ran roughly 250 splits, which made it clear that putting the model into production would cause harm. Teams that skip this kind of testing, he said, ship a model, see lower accuracy in production, and watch the project fail.
How does Honest ML work?
Schles defined Honest ML as a framework for more honest model development that tests a vast number of train test splits, so that a model's accuracy is not the result of a lucky split. It works with any classifier or regressor that follows the scikit-learn API, and he said he has extended it to TensorFlow and PyTorch.
Instead of reporting one number, the framework shows the distribution of precision and recall across all the splits, plotted together with marginal distributions. He told attendees to trim the plot scales to where the data sits. Schles said 200 to 400 trials give a far better sense of confidence than five random splits. He described the approach as borrowing from Bayesian thinking.
Does cross-validation solve the problem?
Not always, according to Schles. Cross-validation scales well when a dataset is huge, because a team can run hundreds or thousands of splits on a billion rows. With small data, that is harder. He works on a study of acute kidney failure in prematurely born babies with only about 200 data points, and said medical datasets of that kind will always be small.
For that audience, he said, Honest ML fills the gap because it runs one train test split at a time. He positioned it as a step after a team has a good enough model through cross-validation and hyperparameter tuning. In the Q&A he said he would recommend getting the distribution after those steps, and he agreed to look at adding rebalancing for imbalanced classes.
Does the problem get worse with more data?
An audience member noted that the demo used a small breast cancer dataset. Schles agreed it was small and said he has not tested larger public datasets because running the tests is expensive. Speaking anecdotally from work on terabytes of data, he said "the bigger the data set the more pronounced this tends to be." He stressed that this was his experience, not a proven result.
He also offered a reassurance. If a model really captures the phenomenon, accuracy varies very little across splits, and the problem matters less. In a paper he wrote with a colleague, he said, the models scored consistently high across splits.
What does it take to build a neural network from scratch?
The second half of the session reviewed the math. Schles covered what a statistical model is, stochastic gradient descent, matrix multiplication and invertibility, and why linear regression can be solved with linear algebra. He noted that matrix multiplication is not commutative, a point he said trips up even junior people.
He then showed how a forward pass is a chain of dot products and how backpropagation works backward through the layers. "Linear algebra is the glue of neural networks," he said.
How do the TensorFlow examples compare?
Schles showed a simple Keras classifier and a regression example, and a custom network built on the low-level TensorFlow API. He said he uses the low-level API at work and prefers TensorFlow to PyTorch for that purpose, while praising PyTorch code as elegant. Time ran out before the PyTorch example, which he said he would post to the project's GitHub page.
On regression error, he cautioned that metrics like MSE are unitless. Saying a score is below one means little: "I did great, well, compared to what?" An error of 40,000 can be excellent on billions of daily rows and poor on a smaller table.
Where to learn more
AI Connect continues this kind of practical programming through its webinars. The 2027 schedule includes 24 free live webinars in four series: the AI x Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and the Plug & Play Series. Details are at /webinars/, and registration for 2027 events is coming soon.
Frequently asked questions
What is Honest ML?
It is a framework from Eric Schles that tests a model across a vast number of train test splits to ensure its accuracy isn't due to a lucky split. It works with scikit-learn style models and has been extended to TensorFlow and PyTorch.
Why does the random seed matter in a train test split?
Schles said that varying the seed, without changing hyperparameters, causes deviation in the score of virtually every model, so one split may overstate performance.
How many train test splits are enough?
Schles said 200 to 400 trials give a much better sense of confidence than five random splits.
Does cross-validation replace Honest ML?
Schles described Honest ML as a next step after a good enough model through cross-validation and hyperparameter tuning, and as especially useful for small datasets.
This article is based on remarks by Eric Schles, Principal Data Scientist, Johns Hopkins Hospital at Data Science Innovators Conference 2023 on April 12, 2023. More coverage: More AI Connect Events.
Filed under: Observability & Data Quality, AI Use Cases & Applications
