Cloud Migration and the AI Data Foundation: WhereScape Lessons
In a COLLIDE 2024 session, WhereScape's Patrick O'Halloran compared small, mid-sized and large companies building data foundations in the cloud. He recommended narrow scope, rapid prototyping and treating the work as a business process, with automation to cut testing and rework.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- Foundation first. O'Halloran said AI results depend on the quality of the data feeding them.
- Warehouse projects are risky. He cited a Gartner report that about half did not deliver or were not used.
- Keep scope small. Pick one business area at a time.
- Make it a business process. Involve analysts early and prototype often.
- Automation shifts effort. Generated code cuts testing, rework and documentation work, he said.
At COLLIDE 2024, the COLLIDE Data & AI Conference in Atlanta in October 2024, Patrick O'Halloran, Solutions Architect at WhereScape, used three customer case studies to argue that the data warehouse projects behind AI efforts fail for organizational reasons as often as technical ones. In a lunchtime session sponsored by WhereScape, he said the fix is a smaller scope, early prototypes and treating the build as a business process, with automation to speed the work.
What is a data foundation, and why is it risky to build?
O'Halloran said "data warehouse" is "a limiting term." Whether a company runs a lake, lakehouse or Data Vault, he said, the pattern is the same. Multiple source systems feed a large database, the data is curated and aggregated, and a presentation layer serves analysts, data scientists and AI systems. He called that a data foundation, which can be in the cloud or on premises.
He tied it to AI through quality: "garbage in, garbage out." The quality of the data put into a predictive or generative AI system determines the quality of the results, he said.
O'Halloran cited a Gartner report, which he believed was from 2022, that called building a data warehouse the riskiest project an IT team can take on. By his recollection, about half were either not finished or did not deliver and went unused, because analysts did not understand or trust the contents. He also named growing numbers of source systems, compliance demands, scarce staff and manual coding as common problems.
How does data warehouse automation work?
WhereScape began about 22 years ago as a New Zealand consultancy that built data warehouses by hand, O'Halloran said. It built tools for the repetitive work and customers asked to buy them, so it became a software company. He said it now has about 1,200 customers.
He argued that automation brings consistency, which makes it easier to apply rules for governance, personal data and data quality. "A car is more than just a fast horse," he said. He described a builder tool that generates the code to move data from a source such as SQL Server into a target such as Snowflake, handling type conversions and loading, and said it has produced tens of thousands of lines of Python code he did not write.
He added that the same speed allows rapid prototypes. An analyst can see a dimensional model within minutes and give feedback, he said, which matters more than rigid requirements.
What do small, medium and large companies get wrong?
O'Halloran described a pattern by company size.
- Small company: A specialty fabrics maker with roughly 200 to 300 employees had to merge US and European SAP analytics into one cloud system. Two engineers who had seen a 20-person team spend a year on a Data Vault project and abandon it built the whole warehouse in nine months using WhereScape, he said. He tied this to Frederick Brooks's "The Mythical Man-Month," noting that communication lines grow with team size.
- Mid-sized company: Firms of roughly 200 to 2,000 people have the most failures, he said. Informal communication is gone, but they lack a program management office or dedicated QA group. Requests from six or seven groups can make scope balloon until the project collapses.
- Large enterprise: Slow change boards and compliance reviews frustrate people, he said, but structure and controls exist.
What lessons did he draw for AI-ready data projects?
O'Halloran's first lesson was scope. "Don't try and drink the whole ocean," he said, recommending one area, such as accounting or sales, at a time.
His second was to treat the build as a business process, not an IT project. He urged teams to involve analysts and source-system owners early and to hold regular reviews with end users. He also quoted Brooks's advice to "plan on throwing one away": build a prototype and discard it, since sunk effort should not decide the architecture.
Can automation cut cloud costs and testing time?
Cloud bills were a concern he said he hears often because pricing is consumption based. At a Snowflake Summit two years earlier, he said, more than half of the vendors pitched ways to cut Snowflake costs. His argument was that doing things right the first time reduces testing, rework and maintenance.
He gave one example. A QA department allowed objects generated by WhereScape to skip unit testing. With 400 objects at about four hours each, he calculated that saved 1,600 hours. He also pointed to generated documentation and data lineage, which can answer "where did this number come from?" in moments, not days. Because generated code comes from templates, a change in rules can be regenerated across hundreds of objects, he said. These figures were O'Halloran's own examples and have not been independently verified.
COLLIDE 2027 returns to Atlanta
COLLIDE 2027, the largest AI conference in the Southern U.S., will bring 1,000+ enterprise data and AI leaders to the Sandy Springs Performing Arts Center in Atlanta on Wednesday, October 13, 2027. Details are on the COLLIDE event page, and registration is coming soon.
Frequently asked questions
Why do data warehouse projects fail?
O'Halloran cited lack of trust, ballooning scope, many source systems and manual coding, and said analysts are often involved too late.
What is a data foundation?
His term for source systems feeding a curated, aggregated database with a presentation layer for analysts, data scientists and AI.
What is data warehouse automation?
Generating the design, code and documentation for a data warehouse from templates and metadata rather than hand coding.
How can teams manage cloud costs?
He said getting designs right the first time reduces testing and rework, which lowers consumption.
This article is based on remarks by Patrick O'Halloran, Solutions Architect, WhereScape at COLLIDE Data + AI Conference 2024 on October 10, 2024. More coverage: COLLIDE AI Conference.
Filed under: Data & AI Infrastructure, AI Strategy & ROI
