Should enterprises host their own LLMs, use cloud models or run small models on laptops? In a Data Science Connect webinar, engineers from SandboxAQ, Alteryx and Plotly weighed the tradeoffs and explained how governed access, testing and open frameworks help future-proof AI infrastructure.
Key takeaways
- Start with the major providers. Lounsbury advised proving use cases on hosted models before investing in self-hosting, which brings GPU, cost and scaling burdens.
- Watch AI costs. Leichenauer said cloud AI makes it easy to rack up large bills by accident, so usage observability matters.
- Make governed access easy. A central, audited access point to approved LLMs keeps employees from going around policy.
- Test your data pipeline. Test cases show whether chunking, embedding and prompt changes still produce correct RAG answers.
- Avoid lock-in. Open frameworks and multi-vendor proofs of concept make it easier to switch as the market shifts.
Generative AI is changing how companies prepare, share and process data, and the infrastructure behind it has to keep up. In a Data Science Connect expert panel webinar on December 4, 2024, sponsored by Alteryx, Plotly, Zenlytic and SandboxAQ, three technical leaders discussed how to scale AI across an enterprise without losing control of cost, security or data quality.
The panel featured Stefan Leichenauer, VP of Engineering at SandboxAQ; Jonathan Lounsbury, Principal Engineer at Alteryx; and Nathan Drezner, Technical Product Manager at Plotly. Brian Mink, co-founder and president of Data Science Connect, moderated.
How is generative AI changing data infrastructure?
Lounsbury said generative AI helps teams figure out how to clean and transform data and gets them to an "80% solution" quickly, even if that first workflow isn't production ready. Large language models also open new ways to process data, such as reading PDF images that once required OCR, summarizing text and running sentiment analysis.
Drezner pointed to unstructured data. Teams no longer need to spend hours cleaning a data set before they can work with it, he said, and "rusty" data sets that were nearly impossible to analyze can get another look.
Leichenauer added that foundation models increasingly combine documents, audio, video, images and sensor data at the same time. That means data pipelines must be able to handle and feed all of those streams.
Should enterprises self-host LLMs or use cloud models?
This question came up throughout the hour. Drezner said self-hosted models running inside a company's own network are becoming more compelling, and the choice depends on data privacy needs, comfort with model accuracy and willingness to manage the hosting work.
Lounsbury urged caution. "Hosting your own models is a really big investment," he said, citing developer time, cost and scarce cloud GPU instances. His advice: prove out use cases with the major providers first, then move specific, well-defined use cases to a self-hosted open source model where it saves money.
He also described how quickly self-hosting grows complicated. A GPU that is too costly to leave running can lead to autoscaling and warm pools with a preloaded foundation model. Unless hosting LLMs is your core business, he said, rely on the companies solving those problems.
Leichenauer framed it as a spectrum that mirrors the decade-old cloud debate, weighing risk, privacy and maintenance against economies of scale. He warned that it is easy to "rack up huge costs accidentally" with cloud AI, so observability into what is used, and how much, matters.
Drezner described a third path: running small models on a laptop. He said a small model of about three billion parameters can run on most modern laptops with 8 or 16 gigabytes of RAM.
How do organizations balance AI innovation with governance and compliance?
Leichenauer, whose company works with customers subject to HIPAA and financial regulations, said the key word is adaptability. Regulations, models and use cases keep changing, so organizations need a clear owner for governance and compliance. "You have to balance going fast versus overstepping," he said.
Drezner noted that anyone can upload a CSV of protected data to a public chatbot, so clear policies on what staff can send to hosted models are essential. Teams often don't need to send full data sets at all, he said. Sending just the headers and data types can be enough to generate useful code.
Lounsbury argued that policies only work if compliance is easy. He recommended a central, governed access point to approved LLMs that also audits usage, which he said is the approach Alteryx is taking with its own generative AI tools.
How can teams ensure data quality for AI systems?
For retrieval-augmented generation and fine-tuning, Lounsbury recommended a set of test cases. As teams adjust chunking, embedding models and prompts, the tests show whether the model is still answering correctly.
Drezner said ETL and data pipelines matter even more now, along with a human in the loop. "You can't just hand things to a model and hope for the best," he said.
Leichenauer suggested treating data as a product, with an owner and customers across the business, and gathering their feedback as with any product.
How can companies avoid vendor lock-in with AI?
Silos and cultural resistance also slow enterprise AI. Leichenauer recommended ongoing training and an AI center of excellence to guide teams. Lounsbury favored a platform where that center can build use cases and share them across the organization.
On future-proofing, Leichenauer warned that a chosen vendor "may not even exist two years from now." He advised using open frameworks for APIs, testing several vendors in proofs of concept and asking how painful a later switch would be. Lounsbury agreed, recommending platforms that let teams plug in new LLMs as they emerge.
In a lightning round, Leichenauer stressed flexibility. Drezner said to "always use verifiable analytics in tandem with AI strategies," because LLMs can make things up. Lounsbury's advice: don't build it yourself, but choose a partner that offers governed access to LLMs so teams can move from proof of concept to production quickly.
Join the conversation at Data Science Connect webinars
Expert panels like this one are a regular part of Data Science Connect's programming. The 2027 schedule includes 24 free live webinars across four series: the AI × Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and the Plug & Play Series. Registration details are coming soon.
Frequently asked questions
Should companies self-host large language models?
The panel said self-hosting can make sense where privacy is paramount, but it is a large investment, so most organizations should prove use cases with major providers first and self-host only well-defined use cases.
How can employees use AI without leaking sensitive data?
Panelists recommended clear policies on what can go to hosted models, a central governed access point to approved LLMs, and sending only summaries such as column headers and data types rather than full data sets.
How do you avoid AI vendor lock-in?
Leichenauer advised using open frameworks for APIs, testing several vendors in proofs of concept and weighing how hard it would be to switch later.
Can you run an LLM on a laptop?
Drezner said a small model of about three billion parameters can run on most modern laptops with 8 or 16 GB of RAM.
Watch: Future-Proofing Infrastructure: Democratizing Data for the AI Generation on YouTube
Event: Webinars · More coverage: Webinars insights
