Scaling Data Models: Lessons From Disney, Meta, Microsoft and Minecraft
In a panel at the Data Science Innovators Conference 2023, data leaders from Disney, Meta, Microsoft and Minecraft discussed why models stall after they are built. They covered buy-in, business alignment, communication, maintenance and scaling.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- Four project phases, four hurdles. Agarwal named buy-in, trust in results, integration and sustained relevance.
- Frame success in business metrics. Duze said accuracy alone is not enough; define the business metric the model should move.
- Simpler can win. Panelists said a simple model or even a spreadsheet can be the right answer, and it builds credibility.
- Communicate without jargon. Gonzalez and Rius urged plain language, and Duze described the data scientist as an interpreter.
- Plan maintenance up front. Assign ownership and thresholds, and track both model and business metrics.
Getting a data model built is the easy part. Getting a business to trust it, use it and keep it current is where projects stall. In a panel at the Data Science Innovators Conference 2023, "Overcoming Challenges in Adopting and Scaling Data Models Across Organizations," four data leaders shared how they handle that gap. The panelists spoke for themselves and not their employers.
Mary Stimson, a big data analyst and data application developer at Southwest Airlines, moderated. The panelists were Francisco Rius, Head of Data Science at Minecraft; Anshul Agarwal, Director of Decision Science at The Walt Disney Company; Charles Duze, an analytics leader at Meta; and Leondra Gonzalez, Senior Data and Applied Scientist at Microsoft.
What challenges do teams face in adopting data models?
Agarwal described four phases of a project, each with its own obstacle. In the pre-sales phase, teams must win business buy-in and persuade people to change how they work. In development, the challenge is trust in the results, especially when a prediction differs from what stakeholders expect. Integration with core business systems follows, and then sustained relevance as business rules and external conditions change.
Rius added a perspective from creative industries, where decisions can lean on gut feel and educating partners matters. He noted that data science teams are typically three to four percent of a large organization, so they cannot staff every project and must build people who understand context. Once a model is running, quality problems can erode trust, so teams need to diagnose and interpret odd results fast.
How do you align a model with business goals?
Duze outlined a sequence: understand how data-minded your organization is, understand the business objective and frame the problem in its terms, confirm a model is the right solution at the right time, and express success as a business metric and not just accuracy or precision. Then set realistic stages and "measure and iterate."
A simpler option is sometimes better, he said, and choosing it builds credibility for later models. Gonzalez agreed that the solution should come from business strategy and not from applying the fanciest model available, which she said saves time, money and resources. She also urged teams to involve IT early, since the data may not even be collected yet.
How should data teams communicate value to stakeholders?
Gonzalez, who runs a data and AI literacy side business, said technical communication can be a habit in the data science community. Insights should be digestible, she said, using terms stakeholders know. She keeps a list of business acronyms and offers plain-language explanations for terms such as p-value.
Answering an audience question from a data scientist who is the only one in their company, Duze described the role as "the interpreter": understand the other side's problem, translate it into a data problem, and translate the answer back. He urged building trust in small steps, like a Costco sample, so that stakeholders come asking for more.
Rius described a weekly rhythm of business where analysts and data scientists share insights with a few hundred people, which he said creates a non-threatening relationship with creative teams. Agarwal said one team ran an A/B test with some entities using model recommendations and others using the existing process, which helped clients see the model was at least as good as their current approach.
How do you keep models flexible and current?
Duze said teams often neglect maintenance. Plan from the start who owns it, which metrics and thresholds matter and how to adjust. Track both model metrics and business metrics, since models almost inevitably drift as data and inputs change.
Rius advocated staying lean: easy-to-use tech stacks and code analysts can read pay off many times over compared with something sophisticated that one person understands. "Don't overlook the value of simplicity," he said, noting that heuristics often stand the test of time.
What did the panel say about scale, cloud and examples?
For large data sets, Agarwal recommended starting small with a proof of concept, then a pilot, before scaling, and using cloud platforms for compute. On cloud versus in-house, he cited serverless scaling of compute as a guiding factor. Rius added that serverless limits the risk of one heavy job taking down the whole ecosystem and that virtualizing data sources is easier in the cloud.
Duze gave an example from his time at Shopify, where app developers paid for fraudulent store reviews and a manual review team could not scale. The team started with a simple logistic regression and built its own training data alongside it. That first phase built the infrastructure and process, so the model could be upgraded later.
Where to hear from data and AI leaders
For practitioner panels like this one, see the 2027 webinar schedule: 24 free live sessions in four series, including the AI × Enterprise Panel Series. Details are at /webinars/; registration for 2027 events is coming soon.
Frequently asked questions
Why do data models fail to get adopted?
Panelists pointed to missing buy-in, lack of trust in results, difficulty integrating with business systems and models that fall out of date as the business changes.
How do you align a data model with business goals?
Duze recommended understanding the business objective, confirming a model is the right solution, defining success in business metrics and iterating in stages.
How can a lone data scientist win support?
Panelists advised communicating without jargon, acting as an interpreter between data and business, and building trust with small early wins.
How should teams scale models to large data sets?
Agarwal suggested a proof of concept, then a pilot, before moving to larger data, and using cloud compute for scale.
This article is based on remarks by Francisco Rius, Head of Data Science, Minecraft; Anshul Agarwal, Director of Decision Science, The Walt Disney Company; Charles Duze, Analytics Leadership, Meta; Leondra Gonzalez, Senior Data & Applied Scientist, Microsoft; moderator Mary Stimson, Big Data Analyst and Data Application Developer, Southwest Airlines at Data Science Innovators Conference 2023 on April 12, 2023. More coverage: More AI Connect Events.
Filed under: People, Culture & Leadership, AI Strategy & ROI, Observability & Data Quality
