Skip to main content

Boardroom Answers · AI & Data · AI, Data & Analytics

Show me the eval. Not the architecture slide — the actual evaluation methodology, coverage, and baselines. If you can’t, everything else you’ve said is decoration.?

The question a Chief Data Scientist / VP of AI & Analytics asks.

The short answer

Three real layers — a deterministic golden-fixture gate on every build, an LLM judge scoring six rubric dimensions against a baseline, and live production quality monitoring — with the gaps stated before you find them.

The full executive answer

Fair — here is exactly what exists, and exactly how deep it goes. Layer one, on every single build with zero AI calls: a golden-fixture regression harness. Each covered module has known-good output fixtures that must pass its schema and its content rules — checks a schema alone cannot express, like framework citations resolving against our real 74-entry catalogue, roadmap milestone totals reconciling, cost ranges being ordered — plus deliberate mutations that the gate must catch. That last property is the point: if anyone loosens a schema or rule far enough that a known-bad output slips through, the build fails. It is deterministic, free, and it guards the output contracts. The same build runs a standing red-team corpus against the injection and PII defences.

Layer two, model-quality judgment: an LLM-judge harness scores generated outputs one-to-five on six defined rubric dimensions — specificity, quantification, actionability, register, honesty, evidence grounding — against a committed baseline, failing on material regression, and it must quote the weakest sentence in the output, which keeps the judge honest. Layer three, in production: every insight gets a deterministic quality score persisted with it, aggregated into a live health view — average quality, worst decile, fallback rate, latency — with scheduled alerting when it degrades.

Now what it is NOT, before you ask: there are no large human-labelled gold datasets yet — the fixtures and rubric are engineering-built; judge scoring inherits LLM-judge limitations and is anchored by the deterministic layers, not the other way round; not every module has equal fixture depth — the harness itself reports covered versus uncovered modules rather than hiding it; and there is no production score history because we are pre-launch. The designed path to depth is the feedback loop: down-rated, edited and flagged outputs are explicitly the candidate pool for new gold cases, so evaluation deepens with real usage. You have seen vendors claim "99% accuracy" with no methodology; I would rather show you a real, honest harness than a fake number.

Grounded in: NIST AI RMF MEASURE function (TEVV — test, evaluation, verification, validation); LLM-as-judge literature and its known limitations; ISO/IEC 42001 §9 performance evaluation.

Want this answered live, on your data?