Leaderboard scores rarely predict how a model will handle a company's own data. At COLLIDE 2025, HumanSignal ML Evangelist Micaela Kaplan explained how enterprises build custom AI benchmarks with rubric-based scoring, start small, and bring domain experts into the process.
Key takeaways
- Evals are the process, benchmarks the tool. A benchmark pairs a task set with a scoring method.
- Leaderboards use public data. Generic benchmarks can't show how a model handles private, messy enterprise data.
- Score with rubrics. Generative AI rarely has one right answer, so break quality into criteria such as risk, compliance and next steps.
- Start with 20 to 50 tasks. Grow and document the benchmark as the model matures so versions stay comparable.
- Bring in domain experts. Data scientists, subject experts, compliance and business owners should define quality together.
Public leaderboard scores say little about how a model will perform on a company's own data, HumanSignal ML Evangelist Micaela Kaplan told attendees at COLLIDE Data + AI Conference 2025 in Atlanta, in a session sponsored by HumanSignal. Kaplan, who described herself as a data scientist who now talks a lot about data science, explained how enterprises can build custom benchmarks with rubric-based scoring to measure AI quality before and after models reach production.
What is the difference between AI evals and benchmarks?
Kaplan drew a clear line between the two. "Evals are a process and benchmarks are a tool within that process," she said. AI evaluation includes classic statistical metrics such as precision, recall, F1 and accuracy, human-in-the-loop review where people mark outputs right or wrong, and LLM-as-a-judge techniques that use a model to make those judgments.
A benchmark, in her definition, is a standardized, repeatable test for an AI system with two parts. The first is a task set tailored to the use case, covering common requests, challenging cases, areas that draw bad feedback and adversarial inputs. The second is a scoring method. That can be a simple right-or-wrong check for classification tasks, judgment-based scoring with a rubric, or a composite of both.
Some in the audience might see this as a held-out test set from traditional machine learning, Kaplan acknowledged. The difference, she said, is that benchmarks must be standardized and repeatable so teams can compare model versions and different base models against each other.
Why do generic AI leaderboards fall short?
Leaderboards rank models on generic benchmarks that test general knowledge, math, logic and reasoning, usually with public or semi-public data sets. Kaplan showed how two models answered a question from a 2024 high school math exam, which is useful for comparing reasoning. She also mentioned HealthBench and said HumanSignal worked closely with the group behind a recently released legal benchmark.
The problem is relevance. "Most of the time your data isn't public," she said, so a benchmark built by a large model provider can't tell a company how a model will handle its own work.
How do custom AI benchmarks work?
Custom benchmarks test what a business cares about, according to Kaplan. They use domain-specific knowledge and jargon, give teams a safe place to use messy internal data that can't be made public, and can measure KPIs and business outcomes rather than only correctness.
She showed a sample question from a financial institution scored with a rubric rather than a binary answer. The rubric looked for risk identification, regulatory and compliance considerations, practical next steps, and a balance between service and compliance. Breaking scoring down this way shows where a model performs well and where it "might need some love," she said.
Teams don't need thousands of samples to start. "Benchmarks should evolve as your models evolve," Kaplan said. At the proof-of-concept stage, 20 to 50 key tasks are enough to show whether a model warrants more investment. As understanding deepens, teams add tasks, study failure patterns and adapt questions from off-the-shelf benchmarks to see where they stand against the industry.
Documentation is essential, she said, because a model scored on an old benchmark can't be compared with one scored on a new version.
Who should be involved in building an AI benchmark?
Before writing questions, Kaplan said, teams should define what quality means to them: factual accuracy, fluency, alignment with brand policies and company values, compliance and risk, or specific KPIs. She warned against building rubrics in a spreadsheet, calling that her nightmare.
Data scientists should be involved because they build the model, but so should domain experts. A medical use case, for example, needs doctors to say what correct means medically. Compliance staff and business owners who can define KPIs also belong in the process. "Building a benchmark is a collaborative process," she said.
A good benchmark answers practical questions, Kaplan said: whether a model has reached the threshold to be trusted in production, whether a new model beats the last one, and whether a team can adopt a cheaper model without losing quality.
Should humans or models score AI benchmarks?
Kaplan demonstrated HumanSignal's Label Studio platform, where reviewers attach rubric criteria to a specific question, such as requiring an answer to disclose that the AI is not a medical professional, worth 10 points. Scoring can be done by a person or by a model, depending on the stage of the project.
Even when a model does the scoring, she recommended periodic human checks, because "models are models and they don't always perform the way we think they're going to perform." She said Label Studio is available as an open source version, an enterprise version and a middle tier called Starter Cloud.
Learn how enterprises test AI at COLLIDE 2027
COLLIDE, the largest AI conference in the Southern U.S., brings more than 1,000 enterprise data and AI leaders to Atlanta for practical sessions on evaluating and deploying AI. The 2027 conference takes place Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center. Learn more at COLLIDE, or join one of Data Science Connect's free live webinars, including the AI Demo Days series.
Frequently asked questions
What is an AI benchmark?
Kaplan defined it as a standardized, repeatable test for an AI system made up of a task set tailored to a use case and a scoring method.
Why aren't public leaderboards enough to choose a model?
They test general knowledge on public or semi-public data, while most enterprise data isn't public, so the results may not reflect performance on a company's own tasks.
How many examples does a custom AI benchmark need?
Kaplan said 20 to 50 key tasks are enough at the proof-of-concept stage, with more added as the model and the team's understanding mature.
Should LLM-as-a-judge scoring be checked by humans?
Yes. Kaplan recommended having humans check model-based scoring at least some of the time, since models don't always perform as expected.
Watch: How Successful Enterprises Measure AI Quality, brought to you by HumanSignal on YouTube
Event: COLLIDE AI Conference · More coverage: COLLIDE AI Conference insights
