The Twin Crises of Science: John Elder on Target Shuffling
In a COLLIDE 2024 keynote, Elder Research's John Elder described two crises: findings that do not replicate and data science projects that never reach production. He showed target shuffling, a simulation that adjusts for cherry-picking, and shared change management lessons.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- Many findings do not replicate. Elder cited estimates that only 10% to 40% of published findings can be reproduced.
- Over search misleads. Trying many combinations and keeping the best inflates apparent significance.
- Shuffling gives honest odds. Randomizing the target and repeating the search showed a result's real chance of being luck, 1 in 5 in his example, not 1 in 300.
- Implementation is the second crisis. He said only a fifth or sixth of lab-successful projects reach production.
- Change management matters. Work with stakeholders from the start and make the work partly theirs.
In a keynote at COLLIDE 2024, the COLLIDE Data & AI Conference in Atlanta in October 2024, John Elder, Founder and Chair of Elder Research, told data scientists that much published research cannot be trusted, and that many data science projects fail for a second reason: nobody puts them into production. In a talk sponsored by Elder Research, he described twin crises of unreliable findings and failed implementation, and he offered a simulation technique, which he called target shuffling, to address the first.
Elder opened by saying his job was to cause the audience "to despair at the state of science" and then show that there is hope.
How reliable is published scientific research?
Elder said peer review is a hard filter. He cited the medical journal The Lancet, which he said publishes about 5% of submissions, yet whose own editors believe "only half of what they publish is actually true." They just do not know which half, he said.
He also cited estimates that only 10% to 40% of published findings can be replicated. The most thorough check he knew of was a 2015 effort in psychology in which dozens of teams tried to replicate 100 of the most popular articles from 2008. The majority failed, he said, and the study went on to be named Science magazine's result of the year.
Elder noted that data scientists do better than most researchers, but "we still have the issue of multiple comparisons." Powerful tools let analysts try millions of ideas, which raises the chance that randomness produces an answer.
Why do so many data science projects never reach production?
The second crisis, Elder said, is that implementing results is hard. He estimated that only about a fifth or a sixth of data science projects that seem to work in the lab reach production, calling it a "horrifying statistic."
Elder Research, now in its 30th year, improved its own record over time, he said. In its first decade it believed it could solve about 90% of the problems it tried, and about 60% of those were implemented. In its second decade, 98% were technically solved and 90% implemented.
What does it take to get AI and analytics results implemented?
Elder described change management as the key. He compared switching a business to a new model with asking people to move a train that has run on the same tracks for 30 years. People will not do that lightly, he said.
His advice was to work with stakeholders from the beginning, help them take ownership, be transparent to build trust, and care about their issues more than your own. He also said teams should deliver professional-grade software, and that Elder Research bought an award-winning software team to improve its deliverables.
"Nobody likes an arrogant consultant," he said, describing a young expert arriving with a black box that overturns how people have worked for 20 years. He urged data scientists to become part of the team and make the work partly theirs. Turning a skeptical technical colleague into an ally takes soft skills, he said, adding that teams need "some people who speak human."
What is target shuffling and how does it fix false discoveries?
Elder said he developed target shuffling early in his company's history, to persuade clients that a result was real before they put money behind it. He described it as a way to adjust for "over search," where trying so many combinations makes a spurious correlation more likely, just as overfitting happens when a model is too complex.
His demonstration used baseball pitch data. Searching a three-dimensional "strike zone" cube broken into 120 chunks turned up a hot spot with a P value of about 1 in 300. Elder explained that the P value is only a valid probability for a single test. After picking the best of many, the number is far too optimistic.
The fix is simulation. Elder shuffled which pitches were hits, so any link between location and hits was random by construction, then ran the same search and recorded the best apparent discovery. After repeating this a thousand times, about 18% of the randomized runs beat the real result. So, he said, the honest odds were closer to one in five than one in 300.
"I don't know anyone who's made a decision based on a P value," Elder said. A simulation is easier to explain to decision makers. Elder said the company is "basically giving it away" and has resources on its website.
What did Elder say about drug development and meta-analysis?
In the question period, Elder described an earlier project with a pharmaceutical company that faced a billion-dollar decision on whether to continue testing a drug. His team took a fresh look at the data, adjusting for survey scales and a stronger placebo effect in Europe. The drug went ahead and became one of three successful drugs from the consortium in a decade, he said.
On meta-analysis, he warned that unpublished negative results, the "file drawer effect," skew the totals. He said only three of the 100 papers in the replication study had reported a negative result.
COLLIDE 2027 returns to Atlanta
COLLIDE 2027, the largest AI conference in the Southern U.S., will bring 1,000+ enterprise data and AI leaders to the Sandy Springs Performing Arts Center in Atlanta on Wednesday, October 13, 2027. Details are on the COLLIDE event page, and registration is coming soon.
Frequently asked questions
What is target shuffling?
A simulation in which the target variable is randomly shuffled and the same search repeated, to see how often chance alone beats the real result.
What are the twin crises of science?
Elder described unreliable research findings and the difficulty of implementing results as the two problems.
Why do many data science projects fail?
Elder said only about a fifth or a sixth of projects that work in the lab reach production, and pointed to weak change management.
What is over search?
Trying so many combinations that a spurious correlation becomes likely, which Elder compared to overfitting.
This article is based on remarks by John Elder, Founder & Chair, Elder Research at COLLIDE Data + AI Conference 2024 on October 10, 2024. More coverage: COLLIDE AI Conference.
Filed under: Observability & Data Quality, People, Culture & Leadership, AI Strategy & ROI
