Skip to main contentMain content

Boardroom Answers · AI & Data · AI, Data & Analytics

Show me the eval. Not the architecture slide — the actual evaluation methodology, coverage, and baselines. If you can’t, everything else you’ve said is decoration?

The question a Chief Data Scientist / VP of AI & Analytics asks.

The short answer

Three real layers — a deterministic golden-fixture gate and an offline quality baseline on every build, and live production quality monitoring — plus an LLM-judge harness that is built but not yet running, so there is no judge baseline to quote. The gaps are stated before you find them.

The full executive answer

Fair — here is exactly what exists, and exactly how deep it goes. Layer one, on every single build with zero AI calls: a golden-fixture regression harness. Each covered module has known-good output fixtures that must pass its schema and its content rules — checks a schema alone cannot express, like framework citations resolving against our real 74-entry catalogue, roadmap milestone totals reconciling, cost ranges being ordered — plus deliberate mutations that the gate must catch. That last property is the point: if anyone loosens a schema or rule far enough that a known-bad output slips through, the build fails. It is deterministic, free, and it guards the output contracts. The same build runs a standing red-team corpus against the injection and PII defences.

Layer two, an offline quality baseline, also on every build and also with zero AI calls: 36 reference generations across 12 modules, scored by seven deterministic scorers — structure, specificity, use of the caller’s own figures, numeral grounding, filler, accountability, citation integrity — against a pinned floor, with planted-defect negative controls the scorers must catch. The reference outputs are authored, not sampled from a live model, and we say so. Model-quality judgment by an LLM is built but not yet running: the harness scores generated outputs one-to-five on six rubric dimensions — specificity, quantification, actionability, register, honesty, evidence grounding — and must quote the weakest sentence in the output, but it needs a model key we have not yet enabled in CI, so there is no judge baseline today and no build fails on a judge score. Layer three, in production: every insight gets a deterministic quality score persisted with it, aggregated into a live health view — average quality, worst decile, fallback rate, latency — with scheduled alerting when it degrades.

Now what it is NOT, before you ask: there are no large human-labelled gold datasets yet — the fixtures and rubric are engineering-built; there is no LLM-judge baseline yet, and when the judge runs its scoring will inherit LLM-judge limitations and stay anchored by the deterministic layers, not the other way round; not every module has equal fixture depth — the harness itself reports covered versus uncovered modules rather than hiding it; and there is no production score history because we are pre-launch. The designed path to depth is the feedback loop: down-rated, edited and flagged outputs are explicitly the candidate pool for new gold cases, so evaluation deepens with real usage. You have seen vendors claim "99% accuracy" with no methodology; I would rather show you a real, honest harness than a fake number.

Grounded in: NIST AI RMF MEASURE function (TEVV — test, evaluation, verification, validation); LLM-as-judge literature and its known limitations; ISO/IEC 42001 §9 performance evaluation.

Want this answered live, on your data?