APT Evidence Engine — 10-Week Build

Database Architecture

7-Table Schema

Evidence flows from source → evidence_item → composition/benchmark_run. Right-sizing profiles encode deployment constraints for each pattern.

Figure 1 — Entity-Relationship diagram. Solid arrows show FK constraints; dashed arrow shows evidence_item → benchmark_run (routed via left margin to avoid crossings). right_sizing_profile has no FK to any other table — it is queried separately by the engine against composition patterns.

Table	Rows	Purpose
`source`	11	Origin of every evidence item (benchmark, paper, vendor, deployment)
`evidence_item`	31	Individual evidence records with type (metric / claim / deployment_record)
`component`	88	Atomic AI components: LLMs, retrievers, tools, rerankers, agents
`composition`	175	Pipelines built from components; each has a pattern (RAG, web_nav, …)
`composition_component`	0	Bridge table: which components play which role in each composition
`benchmark_run`	502	One row per (composition × benchmark task × hardware tier) measurement
`right_sizing_profile`	11	Deployment scenarios with quality/latency/cost constraints per industry

Schema + Seed Data

The question: Can one database hold benchmark evidence from completely different worlds — public leaderboards, research papers, hardware-performance runs, vendor marketing claims — side by side, without the data corrupting or silently contradicting itself?

What the result shows: Yes. Seven linked tables hold 818 rows. Every categorical field (source type, evidence type, component type, composition pattern, hardware tier) is checked against a fixed vocabulary at insert time — so a typo like hardware_tier='H1000' is rejected by the database, not quietly stored. The bar chart below is the actual row count per table after seeding.

$ make populate && make show-w1

Figure 2 — Row counts per table after full seeding. benchmark_run (502) dominates — one row per (composition × benchmark task × hardware tier).

HELM Lite + Queries + Interface

The question: Once the data is loaded, can we actually ask useful things of it — "which model setups meet my budget?", "which are impossible to satisfy?" — and is the engine guaranteed to only read the data, never silently change it underneath us?

What the result shows: Four queries run against the 502 rows. Of all compositions, 68 are feasible under a typical budget (quality ≥ 0.7, latency ≤ 800ms, cost ≤ $0.02/1k); 175 have at least one constraint they can never meet (a "binding" constraint). And after every read, the database is byte-for-byte unchanged — the engine cannot mutate the evidence it reasons over.

$ make show-w2

Click a query — each asks ONE yes/no question of every composition, using its worst measured value per axis (so a "pass" is a guarantee, not luck):

Q2 — Feasible (fits a typical budget)

rule: worst quality ≥ 0.70  AND  worst lat ≤ 800ms  AND  worst cost ≤ $0.02
68 of 175 compositions pass.

✓ RETURNED — e.g.:
  HELM/GPT-4o             q≥0.88  lat≤320ms   cost≤$0.0150  all 3 pass
  HELM/Claude-3.5-Sonnet  q≥0.85  lat≤280ms   cost≤$0.0120  all 3 pass

✗ NOT returned — and exactly why:
  MLPerf/Mixtral-8x7B     q≥0.79  lat≤1200ms  cost≤$0.0006  good+cheap, too SLOW
  HELM/Llama-3-8B         q≥0.69  lat≤80ms    cost≤$0.0004  fast+cheap, quality short

How Evidence Is Stored

Most benchmarks give you one number per model — "GPT-4o ROUGE-L: 0.885". The problem: a single point erases whether the model is consistently 0.885 or varies between 0.83 and 0.94 depending on the task variant. If your deployment requires quality ≥ 0.85, the answer is "sometimes yes, sometimes no" — but a point estimate silently says YES.

The engine stores evidence as a Belief interval [lo, hi]. For each composition, lo is the worst measured quality across benchmark tasks and hi is the best. This turns every threshold query into a three-valued answer instead of a blind ranking:

Model	Interval	Query: quality ≥ 0.80?	Result
GPT-4o (HELM)	[0.83, 0.94]	lo = 0.83 ≥ 0.80 → always above	decidable YES
Llama-3-70B (BEIR)	[0.72, 0.86]	lo < 0.80 < hi → sometimes above, sometimes below	underdetermined
BioMedLM (MedHELM)	[0.59, 0.65]	hi = 0.65 < 0.80 → never above	decidable NO
WebArena / memory_GB	⊥	axis was never measured by any source	blocks query

⊥ (bottom) is not zero and not NULL. A NULL means "we forgot to fill this in." ⊥ means no public benchmark has ever reported this axis for this model — it is a structural gap that cannot be filled by re-running the existing benchmarks. This is why memory_GB (80.9% ⊥) makes certain deployment decisions permanently underdetermined: the data simply doesn't exist.

There are three confidence tiers controlling which evidence the engine uses by default:

Tier	What qualifies	Example sources
H — high	Peer-reviewed benchmark with public reproducibility, code, and held-out test sets	HELM Lite, MLPerf v4.0, BEIR, MedHELM
M — medium	Published leaderboard with methodology described but not fully reproducible	RouterBench, HAL, BFCL v3, WebArena
L — low	Vendor claims, auto-extracted from paper tables, single-run numbers with no variance	ZenML case studies

Figure 3 — Each model's measured quality interval [lo, hi] is plotted against the 0.80 threshold. GPT-4o sits entirely above → decidable YES. Llama-3-70B straddles the line → UNCERTAIN (the evidence can't yet decide). BioMedLM is entirely below → decidable NO. memory_GB is ⊥ — no benchmark ever measured it, so any query needing it is blocked, not guessed.

Dense Benchmark Coverage

The question: Is there actually enough real, varied benchmark data to exercise every kind of decision — or is this a toy with a handful of cherry-picked rows?

What the result shows: 11 independent public benchmarks were loaded, giving 502 measured runs that span LLM reasoning (HELM), retrieval (BEIR/KILT), function-calling (BFCL), medical NLP (MedHELM), hardware performance (MLPerf/MLEnergy), and web agents (WebArena/HAL). 478 of the 502 rows carry at least 4 of the 5 core measurement columns — dense enough that the decision engine has something real to chew on in every scenario. The chart is the actual per-source row count.

$ make show-w4

Figure 4 — Source coverage. 11 public benchmarks span LLM reasoning (HELM), retrieval (BEIR/KILT), function calling (BFCL), medical NLP (MedHELM), hardware (MLPerf), agent tasks (HAL/WebArena), and deployment context (ZenML).

What the Engine Decides — Three Outcomes

When you query the engine ("find me a RAG pipeline with quality ≥ 0.85, latency ≤ 800ms, cost ≤ $0.015/1k"), it doesn't always return a winner. It returns one of three outcomes depending on what the evidence actually supports.

Outcome	What it means in plain English	What the engine returns
decidable	At least one option is provably above the bar. Every axis you asked about has high-confidence evidence and all constraints are satisfied.	A ranked list. The primary pick is the cheapest cost/task; if several tie within 10% of that minimum cost, the smallest model wins (smaller proxies for lower energy and risk) — not the absolute best, the minimum-sufficient safe choice.
underdetermined	Some options look promising but the evidence is too sparse or too noisy to commit. For example, a model passes quality but has no latency measurement at all (⊥), or its quality interval straddles the threshold.	The axis that is blocking the decision + the cheapest benchmark that would measure it. This tells you exactly what experiment to run next rather than guessing.
infeasible	The constraints are impossible to satisfy simultaneously — not because of missing data, but because no model in the database achieves all of them at once even at their best measured values.	The binding axes (which constraints fail) and how far the best candidate falls short. E.g. "best quality is 0.94, you asked for 0.99 — relax by 0.05 or remove the latency constraint".

Why not just always return the highest scorer? Because "highest scorer" on benchmark X may still fail your deployment threshold — and you wouldn't know until it fails in production. The engine only commits when it can prove the answer, and tells you what's missing when it can't.

Concrete examples from the W10 demo:

Scenario	Query	Outcome	Why
Clinical HIPAA	ROUGE-L ≥ 0.30, lat ≤ 2000ms, cost ≤ $0.05	decidable	BioMedLM covers all axes at kappa=H from MedHELM
Open-Domain RAG	quality ≥ 0.75, pattern=rag, cost ≤ $0.02	decidable	Contriever+Llama3-70B from BEIR satisfies all three
Impossible	quality ≥ 0.99, lat ≤ 10ms, cost ≤ $0.00001	infeasible	No model in 502 rows achieves all three simultaneously

The same engine, three asks — click one

Each is a real ask; the panel shows the literal result. Notice the engine never just shrugs — even when it can't decide, it hands you the next move.

outcome: DECIDABLE — the evidence proves an answer

ask: summarization, quality ≥ 0.30, lat ≤ 2000ms, cost ≤ $0.05

  → COMMITS: MedHELM/BioMedLM   quality 0.612 · 420ms · $0.0003/1k

  why: 3 candidates clear the bar; 2 tie at the minimum cost, so the
       smallest model (BioMedLM-2.7B) wins. Provenance: MedHELM, kappa=H.
       Not the highest scorer — the minimum-sufficient safe pick.

What Is a "Composition"?

A composition is a named AI pipeline assembled from one or more components (LLM + retriever, or LLM + tool, etc.) and tagged with a pattern that describes its structure. The pattern determines what constraints make sense: a RAG pipeline has latency dominated by retrieval, a bare LLM doesn't. 175 compositions are registered across 7 patterns.

Pattern	Count	What it is	Benchmark source
`bare_llm`	91	A single LLM called directly — no retrieval, no tools. The simplest deployment: send a prompt, get an answer. Used for summarization, QA, coding, math tasks where the model's parametric knowledge is sufficient.	HELM Lite, RouterBench, BFCL v3, MLPerf
`web_nav`	38	An LLM that drives a browser: clicks buttons, fills forms, navigates pages to complete a multi-step task (e.g. "find the cheapest flight departing tomorrow"). Measured by task success rate, not quality score.	WebArena (6 sites × 5 agents)
`rag`	24	Retrieval-Augmented Generation: a retriever (BM25, DPR, Contriever, E5) first finds relevant documents from a corpus, then an LLM generates the answer from those documents. Quality depends on retriever precision as much as LLM capability.	BEIR (18 datasets), KILT (6 tasks)
`tool_agent`	12	An LLM that calls external tools (Python interpreter, API endpoints, calculators) and uses their outputs as part of its response. Measured on Berkeley Function Calling Leaderboard (BFCL) — does the model call the right function with the right arguments?	BFCL v3
`single_agent`	6	A single LLM operating autonomously on long-horizon tasks (coding, research, planning). Measured by the Holistic Agent Leaderboard (HAL) on standardized agent benchmarks.	HAL leaderboard
`multi_agent`	4	Multiple LLMs collaborating: one proposes, another critiques, a third synthesizes. HAL measures whether multi-agent setups outperform single models on the same task — often at 3–5× the cost.	HAL leaderboard
`rag_reasoning`	0	RAG + chain-of-thought reasoning before answering. Seeded in profiles but no benchmark run rows yet — an axis that is ⊥ for most queries.	(profile only)

Each composition also lists its components. For example, the BEIR RAG pipeline "Contriever + Llama3-70B" has two components: a retriever (Contriever) and an llm (Llama-3-70B, 70B parameters, open weights, Apache-2.0 license). This decomposition lets the engine answer constraint queries like "only open-weight models" or "no proprietary APIs" — enforced at the component level, not just the pipeline level.

How each pattern is actually wired

The pattern isn't just a label — it's the wiring that data flows through. Here is each pattern as a pipeline, with a real registered composition as the example:

bare_llm prompt→ LLM →answer e.g. HELM/GPT-4o · 91 of these

rag query→ retriever →top docs→ LLM →answer e.g. BEIR/BM25+GPT-4o · 24 of these

tool_agent request→ LLM → calls a function →result folded into answer e.g. BFCL/GPT-4o · 12 of these

web_nav task→ LLM ⇄ browser (click, type, read — repeat)→done? e.g. WebArena/GPT-4o · 38 of these

single_agent goal→ LLM: plan → act → observe ↻ →done e.g. HAL/GPT-4o/single · 6 of these

multi_agent goal→ LLM proposes ⇄ LLM reviews →agreed answer e.g. HAL/GPT-4o/multi · 4 of these

$ make show-w6

Figure 5 — Composition counts by pattern. bare_llm dominates because HELM, RouterBench, and MLPerf each test single-model performance across many tasks. RAG and agent patterns have fewer rows because their benchmark sources are smaller and more specialized.

What happens when you query the composition registry

Here is the complete flow for a concrete RAG query — what goes in, what the engine does at each step, what comes out:

Step	What the engine does	Concrete result
① Input	`right_size(pattern="rag", quality≥0.75, cost≤$0.02/1k)` — user needs a retrieval-augmented pipeline meeting these constraints	query received
② Filter compositions	SELECT from `composition` WHERE `pattern = 'rag'` → each row names a specific retriever+LLM pair and cites an evidence_item that backs it	24 RAG compositions (Contriever, BM25, E5, DPR + various LLMs)
③ Fetch measurements	JOIN `benchmark_run` on `composition_id` — read quality, latency_p95_ms, cost_per_1k_tokens. Apply kappa filter: keep only H or M confidence (drop vendor claims).	Measurements from BEIR (18 retrieval datasets) + KILT (6 knowledge tasks)
④ Apply constraints	Keep only rows where quality ≥ 0.75 AND cost ≤ $0.02/1k. Three compositions survive. Quality spread = (0.805 − 0.758) / 0.805 = 5.8%	3 compositions survive. Spread ≤ 10% → branch tiebreak_10pct
⑤ Output	Return the cheapest composition within 10% of the top quality score, with provenance (source, evidence_id, license)	BEIR/Contriever+Llama3-70B — quality=0.758, lat=305ms, cost=$0.001/1k — 15× cheaper than BM25+GPT-4o at the same quality tier

$ make show-w6

Agent Performance + Cost-Quality Frontier

This week adds agent benchmark data from two sources and computes the cost-quality Pareto frontier — the set of models where no other model is both cheaper and higher quality simultaneously. The frontier is what the right-sizing engine looks at first when you query for a summarization task.

Agent data: HAL + WebArena

HAL (Holistic Agent Leaderboard) measures LLMs on long-horizon autonomous tasks: the model must plan, act, observe, and repeat over many steps. Each model is tested in two modes: single-agent (the model works alone) and multi-agent (multiple instances collaborate, with one proposing and another reviewing).

Model	Pattern	Task success	Latency	Cost/task
GPT-4o	multi_agent	0.845	2400ms	$0.062
Claude-3.5-Sonnet	multi_agent	0.831	2200ms	$0.048
GPT-4o	single_agent	0.782	1800ms	$0.038
Llama-3-70B	single_agent	0.654	2600ms	$0.008
Llama-3-8B	single_agent	0.521	1400ms	$0.002

WebArena puts LLMs in a realistic web browser environment with 6 sites (shopping, admin portal, Reddit, GitLab, Maps, Wikipedia) and measures whether they can complete multi-step web tasks (e.g. "order the cheapest shirt in size M" on the shopping site). The metric is task success rate — 0.0 means always fails, 1.0 means always succeeds.

Agent	Mode	Overall success	Best site	Worst site	Cost/task
GPT-4o	full-agent	44.5%	Wikipedia (+5.5pp)	GitLab (−8pp)	$0.095
Claude-3.5-Sonnet	agent	42.8%	Wikipedia (+5.5pp)	GitLab (−8pp)	$0.055
GPT-4o	multimodal	41.2%	Reddit (+2pp)	Admin (−5pp)	$0.078
Claude-3.5-Sonnet	text-only	39.5%	Reddit (+2pp)	Admin (−5pp)	$0.038
Llama-3-70B	text-only	24.8%	Reddit (+2pp)	GitLab (−8pp)	$0.012

Cost-Quality Pareto Frontier (summarization)

For summarization tasks, the engine computes which models are on the efficiency frontier — meaning no other model is simultaneously cheaper AND better. Models below the frontier are dominated: you can always find something better for the same price or cheaper for the same quality.

$ make show-w7

Figure 6 — Cost-quality comparison for summarization. ★ BERT-Large is Pareto-optimal: quality 0.875 at just $0.0001/1k. GPT-4o achieves higher quality (0.92) but at 150× the cost ($0.015/1k). The right-sizing engine selects the cheapest option within 10% of the top score — not the absolute best.

show_w7.py apt_engine.db

What is "agent data"? Unlike a one-shot QA score, an agent must take
MANY steps (plan → act → observe → repeat) to finish ONE task. So the
metric is success rate = fraction of whole tasks completed, and cost
is per task (a task burns many LLM calls), not per 1k tokens.

HAL — autonomous long-horizon tasks (e.g. "debug this repo and make tests pass")
  one task = a full episode; success = did it actually finish?
  [multi_agent ] HAL/GPT-4o/multi-agent      success=0.845 cost=$0.062/task ← 2 LLMs: propose+review
  [multi_agent ] HAL/Claude-3.5/multi        success=0.831 cost=$0.048/task
  [single_agent] HAL/GPT-4o/single-agent     success=0.782 cost=$0.038/task ← 1 LLM, ~2× cheaper, ~6pt worse
  [single_agent] HAL/Llama-3-70B/single      success=0.654 cost=$0.008/task
  [single_agent] HAL/Llama-3-8B/single       success=0.521 cost=$0.002/task ← cheapest, fails ~half

WebArena — drive a real browser across 6 sites (shop, Reddit, GitLab, …)
  one task = "order the cheapest size-M shirt"; success = did the order go through?
  [web_nav] WebArena/GPT-4o/full-agent       success=44.5% cost=$0.095/task ← best, still fails the majority
  [web_nav] WebArena/Claude-3.5/text-only    success=39.5% cost=$0.038/task ← 2.5× cheaper, ~5pt worse
  ... (web agents are HARD — nobody clears 50%)

── Now the frontier (separate, for summarization) ───────────
Cost-Quality Frontier — cheapest model at each quality tier:
  Name                 Quality  Lat    Cost/1k
  --------------------------------------------------
  MLPerf/BERT-Large      0.875    45ms   $0.0001 ★ Pareto-optimal
  MLPerf/Llama-3-8B      0.720    262ms  $0.0002
  HELM/GPT-4o            0.920    320ms  $0.0150 ← +5pt quality, 150× the cost

------------------------------------------------------------
→ Answer: agent success tops out below 85% (web below 50%), and
  spending 150× more buys only +5 quality points. "Best" ≠ "right-sized".

Deployment Context + Missingness Report

The question: Real enterprises don't just want "the best model" — they have hard rules: GDPR, who handles PII, whether a human must sign off. When those rules are encoded, does the database actually have the measurements to answer them — or are there axes it can never speak to?

What the result shows: Five real ZenML enterprise profiles are loaded with their regulatory regime, PII handling, and human-review flags (table below). The coverage report then reveals exactly which measurement axes exist and which don't: quality, latency and cost are 100% covered, energy is sparse, and memory is an 80.9% structural blind spot — no benchmark reports it, so any rule depending on it stays undecidable rather than being guessed.

5 Enterprise Deployment Profiles

Each profile encodes a real company's deployment constraints. When the right-sizing engine receives a query tagged with an industry, it checks candidate models against that industry's regulatory and quality requirements before returning a recommendation.

Profile	Industry	Regulation	PII data handled	Human review
`rsp-accenture`	consulting	GDPR	✓ yes	✓ required before deployment
`rsp-doordash`	food delivery	—	✓ yes	✗ fully automated
`rsp-uber`	rideshare	—	✓ yes	✗ fully automated
`rsp-linkedin`	professional network	GDPR	✓ yes	✓ required before deployment
`rsp-telekom`	telecommunications	GDPR	✗ no	✓ required before deployment

A GDPR profile with human_review=True restricts the engine to sources with auditable provenance — models benchmarked on third-party leaderboards, not vendor self-reports. The missingness report below checks whether the DB actually has the measurement data to answer those constraints.

$ make show-w8

Figure 7 — Column missingness. Core columns (quality, latency, cost) are fully populated. energy_J has sparse coverage (MLEnergy only). memory_GB is the structural blind spot — 80.9% missing across all benchmark sources.

How right_size() Makes a Decision

right_size(query, db) takes a deployment query (task type, quality floor, latency ceiling, cost ceiling, regulatory flags) and runs the deterministic decision rule from brief §4.3: filter to candidates that satisfy every hard constraint → rank survivors by cost per task (cheapest is the primary pick) → if several tie within 10% of the minimum cost, break the tie by smallest model size (a proxy for lower energy and operational risk) → mark everything within 20% of the minimum cost as comparable alternatives → if nothing qualifies, name the binding constraint. The engine stops at the first branch that fires.

Figure 8 — right_size() 5-branch decision tree, implementing brief §4.3. The engine stops at the first branch that fires (top-to-bottom). Hard-constraint filter → rank survivors by cost/task → if several tie within 10% of the minimum cost, the smallest model wins (smaller proxies for lower energy and operational risk). HIPAA example: 2 MedHELM rows sit within 10% of min cost ($0.0003), both BioMedLM-2.7B → tiebreak_10pct returns BioMedLM summarization.

Walk-through: HIPAA clinical summarization

Step (§4.3)	Question	Answer	Branch?
1 · filter	Any rows with ROUGE-L ≥ 0.30, latency ≤ 2000ms, cost ≤ $0.05, HIPAA-safe?	Yes — 3 MedHELM rows qualify	continue
2 · confidence	Do they survive the H/M kappa filter?	Yes — MedHELM is tier M (published leaderboard)	continue
3 · rank by cost	Cheapest cost/task? Which others within 10% of it?	min = $0.0003. Two BioMedLM rows tie at $0.0003; meditron-70b ($0.0008) is >10% out	tiebreak_10pct fires
4 · model-size tiebreak	Of the 2 within 10% of min cost, which is the smallest model?	Both are BioMedLM-2.7B; summarization variant (q=0.612) wins	← return this

$ make show-w9

W10

Four-Scenario End-to-End Demo

The question: Put it all together — four completely different real-world asks (a HIPAA clinical tool, a web-navigation agent, an open-domain RAG system, and a deliberately impossible request). For each one, what does the engine actually return, and does it correctly say "no" when the ask is impossible instead of inventing an answer?

What the result shows: Together the scenarios exercise three branches of the §4.3 rule — tiebreak_10pct (HIPAA, web-nav), feasible (RAG), and infeasible_binding (impossible). The three realistic asks each get a concrete recommendation with provenance; the impossible ask is correctly refused — the engine names the binding constraint rather than hallucinating a model that doesn't exist. Click each to see the real output.

$ make demo

Click a scenario — each shows what the engine returns for that ask:

How do we know it's real?

Every answer above is checked automatically

Each claim this page makes is backed by an automated guarantee that re-runs on every change. These aren't "did the code run" checks — each one pins down a specific promise the engine makes.

The guarantee	What's actually checked
The data can't be corrupted	Inserting a row with an out-of-vocabulary value, or a foreign key pointing nowhere, is rejected by the database — not silently stored.
The engine never changes the evidence	After every read operation (`cell()`, `candidates()`, `required_fields()`) the database is byte-for-byte identical to before.
The four queries return the right sets	Q1 (missing), Q2 (feasible), Q3 (borderline), Q4 (binding) each return exactly the compositions they should for known inputs.
The decision rule routes correctly	All 5 branches fire on the right inputs: feasible, tiebreak_10pct, comparable_20pct, infeasible_binding, confidence_filtered.
The whole thing runs offline, fast	The full 4-scenario demo completes end-to-end with no API key in under 5 minutes.

Project brief, mapped

Every appendix of the brief — in the engine

The ten weeks above are the build. This section closes the loop with the rest of the project brief: the canonical 13-table architecture the 7-table MVP is a subset of, the controlled vocabularies, the derived prudence profiles, the full 5-tier source plan, the weekly checkpoints and scope-reduction order, the flat-extraction scratchpad, and the risk register. Click a tab.

App A + C. The canonical long-term architecture is 13 tables that separate components (one table per system type) from compositions. The 10-week MVP implements a 7-table simplification: the seven per-type component tables (base_llm, retriever, rag_system, reasoning_method, agent_system, tool_use_system, optimization_augmented_system, serving_stack) collapse into one generic component table, and composition_pattern becomes a controlled-vocab field. Availability: P1 = dense, P2 = sparse/paper-specific, P3 = rare / needs custom APT eval.

#	Canonical table	Purpose	Avail.	In 7-table MVP
1	`base_llm`	One row per model/version/config	P1 q/cost · P2 energy	→ `component`
2	`retriever`	Dense/sparse/late-interaction retrievers	P1 q · P2 cost	→ `component`
3	`rag_system`	Config-level RAG record	P1/P2	→ `composition`
4	`reasoning_method`	Prompting / deliberation / verification	P1 q · P3 failure	→ `composition`
5	`agent_system`	Single/multi-agent benchmarked system	P1 HAL	→ `composition`
6	`tool_use_system`	Function-calling / tool-use system	P1 BFCL	→ `composition`
7	`optimization_augmented_system`	LLM + solver / code-exec / verifier	P2/P3	(deferred)
8	`serving_stack`	Runtime / inference-serving stack	P2	→ `component`
9	`deployment_context`	Operational & regulatory setting	P2/P3	→ `right_sizing_profile`
10	`composition`	A benchmarked/deployed AI Box — the heart	P1/P2	✓ kept
11	`benchmark_run`	One row per composition × benchmark × seed	P1	✓ kept
12	`composition_pattern`	Controlled-vocab AI Box archetypes	derived	→ field on `composition`
13	`source`	Source / evidence metadata	required	✓ kept

7-table MVP = source · evidence_item · component · composition · composition_component · benchmark_run · right_sizing_profile — see the ER diagram at the top of this page.

App D. Every categorical field is constrained to a fixed vocabulary, enforced by the database at insert time (the W1 check). These are the canonical archetype lists.

Field	Allowed values
`base_llm.archetype`	dense-decoder · MoE · state-space · hybrid · reasoning-tuned · instruction-tuned-only · RLHF · RLVR
`retriever.archetype`	lexical-bm25 · dense-bi-encoder · sparse-learned · late-interaction · cross-encoder-reranker · generative-retriever
`rag_system.archetype`	standard · multi-hop · iterative/self-RAG · corrective-RAG · agentic-RAG · graph-RAG · multimodal-RAG · realm-pretrained · react-style
`reasoning_method.archetype`	direct · cot · cot-self-consistency · tot · react · reflexion · debate · mcts · best-of-N · genrm-verify · budget-forced · latent-steering
`agent_system.archetype`	single-agent-reactive · single-agent-planner-executor · supervisor-worker · multi-agent-debate · role-play · pipeline-of-roles · dynamic-team · parallel-sampling+selector
`tool_use_system.archetype`	single-turn-function-calling · parallel-function-calling · nested · multi-turn-stateful · MCP-server-orchestration
`optimization_augmented.archetype`	formulate-and-solve · generate-and-check · proof-loop · code-exec-verify · heuristic-synthesis
`composition.task_archetype`	closed-domain QA · open-domain QA · summarization · classification · extraction · reasoning-math · code-generation · code-repair · web-navigation · desktop-automation · customer-service · scientific-research · optimization · verification · data-analysis
`deployment.regulatory_regime`	HIPAA · GDPR · EU AI Act high-risk · EU AI Act limited-risk · SOC 2 · FedRAMP · PCI-DSS · none-public · internal-only

App B / week-9 deliverable. The right_sizing_profile derives a prudence envelope per composition pattern from the real benchmark_run rows: minimum & median quality, median cost / latency / energy. The prudence_score normalizes each axis to [0,1] across patterns (utility higher-better; cost, latency, energy inverted so lower is better) and averages them — a single "minimum-sufficient" ranking. Numbers below are computed live from the 502 rows.

Pattern	n	q min	q median	cost med	lat med	energy med	prudence
`bare_llm`	265	0.620	0.800	$0.0010	240ms	75J	0.97
`tool_agent`	62	0.710	0.851	$0.0070	320ms	78J	0.93
`rag`	124	0.589	0.789	$0.0018	317ms	85J	0.71
`single_agent`	6	0.445	0.594	$0.0700	5350ms	83J	0.51
`web_nav`	38	0.168	0.395	$0.0550	12000ms	85J	0.27
`multi_agent`	4	0.621	0.681	$0.2050	17750ms	83J	0.22

Reading: bare_llm and tool_agent are the most prudent — high quality at the lowest cost/latency. multi_agent is least prudent: it buys a few quality points at 200× the cost and 70× the latency of a bare LLM. This is the quantitative form of "right-sized ≠ biggest." (The MVP's right_sizing_profile table also stores the 11 enterprise deployment profiles seen in W8 — the constraints a query is matched against.)

App F. The brief prioritizes ~50 sources across 5 tiers, densest first. The MVP loaded 11 (highlighted); the rest are reference material to extract after the Tier-1 set passes the week-4 checkpoint. Counts are carried from the consolidated schema artifact and flagged for verification before external use.

Tier	Sources (✓ = loaded in this MVP)
1 — dense leaderboards	HELM Classic · HELM Capabilities · ✓ HELM Lite · ✓ MLPerf Inference v5.0/5.1/6.0 · ✓ HAL (Holistic Agent Leaderboard) · AI Agents That Matter · ✓ BFCL · ✓ RouterBench
2 — RAG & retrieval	✓ BEIR · ✓ KILT · RAGAS · RAGBench · RAGTruth · MIRAGE · BERGEN · ARES · MEMERAG · RAG-RewardBench
3 — agents / tools / reasoning	AgentBench · SWE-bench Verified · GAIA · τ-bench / τ²-Bench · AppWorld · ✓ WebArena / VisualWebArena · OSWorld · CORE-Bench · ToolBench / StableToolBench · HumanEval / MBPP / LiveCodeBench · GSM8K / MATH / AIME / GPQA
4 — serving / energy / optimization	vLLM · Orca · SGLang · TensorRT-LLM · ✓ ML.ENERGY · NL4Opt · OptiBench · ConstraintBench · DafnyBench / FVEval · Code2Inv
5 — composition / production	Compound AI Systems (BAIR) · DSPy · FrugalGPT · LLMSelector · Cascadia · Routing-and-Cascading · Enterprise Blueprint · ✓ ZenML LLMOps DB (Accenture, DoorDash, Uber, LinkedIn, Deutsche Telekom…) · ✓ MedHELM · Beyond-Accuracy / AgentPrune

Canonical extraction-yield estimate: benchmark_run 1,500–3,000 rows · composition 300–600. The MVP's conservative target was 500–1,000 benchmark rows / 50–150 compositions — this build reached 502 / 175.

§6.1 — weekly checkpoints. Each Friday is a binary gate; a miss triggers scope reduction, not failure. This build met every one (final tallies in parentheses).

Week	Checkpoint criterion	This build
W1	7 tables + FKs, data dictionary, ≥20 seed rows across 5 tables	818 rows
W2	CSV/JSON loader, ≥1 real source loaded, Q1–Q4 run on seed data	HELM Lite + Q1–Q4
W4	≥200 benchmark_run with 4/5 core cols; ≥50 MLPerf; ≥50 HELM	502 · 54 · 120
W6	≥400 rows; retriever/RAG/tool components; ≥30 compositions linked	502 · 175 comps
W8	≥500 rows; ≥50 compositions; ≥5 ZenML deployment_context rows	502 · 175 · 5
W9	right_sizing_profile for ≥4 patterns; §4.3 rule as one function; demo sane	6 patterns · ✓
W10	Demo runs clean-checkout end-to-end; reproducible in <30 min	<5 min, no API key

§6.2 — scope-reduction order (if behind, in this fixed order, supervisor-approved): ① drop optimization-augmented sources → ② drop production case studies → ③ cut composition patterns 8 → 4 → ④ drop right_sizing_profile, ship benchmark_run only.

Non-negotiable (never reduced): the 7-table MVP schema · the evidence-linkage workflow · the §4.3 right-sizing decision rule · the week-10 demo.

App G. A single denormalized table, apt_experiment_observation, exists only as a manual-QA scratchpad for auditing rows in a spreadsheet before they enter the normalized schema. It is explicitly not a valid deliverable — it can't support the §4.3 rule or the demo because component reuse, composition lineage, and evidence linkage all need the normalized structure. Rows captured here in weeks 3–5 must be migrated by end of week 6.

Column group	Representative columns
Identity / provenance	record_id · paper_id · doi_or_arxiv_url · evidence_locator · extraction_confidence · manual_review_status
Task & dataset	domain · task_family · downstream_task_name · dataset_name · evaluation_setting · distribution_shift_tested
Model / system identity	system_type · model_name · base_model · parameter_count_m · num_agents · num_tools
Right-sizing knobs	quantization_bits · pruning_used · distillation_used · retrieval_type · retrieval_top_k · cache_used
Hardware / deployment	hardware_tier · gpu_or_accelerator_type · vram_gb · power_limit_w · offline_capable
Effectiveness	primary_metric_name · primary_metric_value · metric_direction · robustness/calibration/abstention_json
Efficiency	latency_p50/p95_ms · throughput_per_sec · peak_memory_gb · energy_j_per_task · inference_cost_usd_per_1k
Governance / burden	explainability_level · traceability_level · privacy_mode · human_review_required · reviewer_time_sec · escalation_rate
Derived right-sizing	utility/cost/latency/energy/governance/burden_score_norm · prudence_score · pareto_rank · recommended_min_tier
Missingness / QC	missing_critical_fields · normalization_notes · unit_conversion_notes · needs_manual_check

§8 — risk register. The design choices throughout this engine trace back to these risks.

Risk	Mitigation (visible in the build)
Noisy paper extraction (PDFs omit hardware/cost/energy)	Structured leaderboards first; missing fields kept as ⊥, never invented (W4, W8)
Over-large schema starves the timeline	7-table MVP first; 13-table kept as canonical target (this section)
Benchmark incomparability	Preserve metric name, direction, units, dataset, split, hardware (benchmark_run columns)
Vendor case-study bias	`value_source_type` + confidence tiers; reported / estimated / measured kept separate (W3, W8)
Sparse energy & human-burden data	Treated as P3; caveated ⊥ unless locally measured (W8 missingness)
Outdated cost numbers	Store `price_as_of_date` / `value_source_type`; refresh before deployment use

§7.3 — time allocation. ~30–40% of the 10 weeks is engineering (schema migrations, loaders, unit normalization, query + demo packaging), concentrated in weeks 1–2 and 9–10 — so row-count is the wrong progress metric there.

§4.4 — snapshot policy. All sources frozen at their week-1 versions (recorded in source.version / access_date); no re-extraction on upstream updates without supervisor sign-off, so HELM/MLPerf/HAL re-runs don't move the target.

§4.5 — experience assumption. Brief assumes an intern with ≥1 prior data-eng/ML project, SQL + Python, paper-reading basics; for a more junior intern, weeks 1–2 stretch to 3 and source/row targets drop to the lower bound.

Reproduce

From Scratch — Offline, No API Key

Clone + populate + run it yourself

bash

git clone https://github.com/triquang26/prudent-ai
cd prudent-ai
git checkout feat/apt-evidence-engine
cd apt-engine

pip install -r requirements.txt
make populate    # seeds DB (~2s, 502 rows)

make show-w1     # W1: schema + seed data
make show-w2     # W2: HELM Lite + queries + interface
make show-w4     # W4: dense benchmark coverage
make show-w6     # W6: RAG / retriever / tool-use
make show-w7     # W7: agents + cost-quality frontier
make show-w8     # W8: deployment context + missingness
make show-w9     # W9: right-sizing decision rule
make demo        # W10: 4-scenario end-to-end demo
make test        # re-run every guarantee

Right-Sizing AI Deploymentsfrom Evidence

7-Table Schema

Schema + Seed Data

HELM Lite + Queries + Interface

How Evidence Is Stored

Dense Benchmark Coverage

What the Engine Decides — Three Outcomes

The same engine, three asks — click one

What Is a "Composition"?

How each pattern is actually wired

What happens when you query the composition registry

Agent Performance + Cost-Quality Frontier

Agent data: HAL + WebArena

Cost-Quality Pareto Frontier (summarization)

Deployment Context + Missingness Report

5 Enterprise Deployment Profiles

How right_size() Makes a Decision

Walk-through: HIPAA clinical summarization

Four-Scenario End-to-End Demo

Every answer above is checked automatically

Every appendix of the brief — in the engine

From Scratch — Offline, No API Key

Clone + populate + run it yourself

Right-Sizing AI Deployments
from Evidence