Evidence-First
Harness
Deterministic assurance for AI-generated code. Treats every AI patch as an untrusted proposal — routes outputs through deterministic policy gates that produce evidence bundles.
"No agent-generated change may be accepted unless every material claim introduced by the change is linked to sufficient, reproducible, risk-adjusted evidence."
Repository metrics reflect the current public repository state at the time of publication and may change.
Smoke Test
[ Quick Start ]
git clone https://github.com/rmax-ai/evidence-first-harness.gitcd evidence-first-harness && uv sync --extra devuv run pytest tests/ -quv run efh run --repo .[ Representative Run ]
$ efh run --repo .
workflow_started run_id=run_3da28c6662d9
worktree_created base_commit=9007ebb1
specification_agent_call opus-4-6 in=294 out=4096
planner_agent_call sonnet-5 in=430 out=770
implementation_agent_call deepseek in=276 out=34
evidence_executed formatting ruff fail
evidence_executed lint ruff fail
evidence_executed type_check pyright fail
evidence_executed secret_scan secrets pass
evidence_executed tests pytest fail
decision_rendered decision=repair_required
mandatory_failed=4 passed=1 tier=3
│ Agent Model In Out │
├──────────────────┼──────────────────┼──────┼──────┤
│ specification claude-opus-4-6 294 4096 │
│ planner claude-sonnet-5 430 770 │
│ implementation deepseek-chat 276 34 │
├──────────────────┼──────────────────┼──────┼──────┤
│ TOTAL 1000 4900 │The transcript below is a representative smoke-test run from the alpha harness. It shows the intended separation of roles: LLM agents produce implementation and review artifacts, while deterministic checks and policy decide the workflow outcome. Exact runtime, token counts, and cost vary with provider pricing, API behavior, cache state, model routing, and repository state. Treat the displayed cost as a run-specific estimate, not a guaranteed current API price.
Estimated run cost: approximately $0.1125 under current pricing assumptions as of 2026-07-13, using Sonnet 5 introductory pricing ($2/$10 per 1M tokens). Under standard Sonnet 5 pricing ($3/$15 from 2026-09-01), the estimate is $0.1170. These estimates exclude hidden/tool overhead and provider-side accounting differences. The cost in the transcript reflects the project's original pinned pricing assumptions.
Architecture
Illustrative architecture — not a benchmark or production deployment. In the shown configuration, LLM agents produce implementation and review artifacts; deterministic checks and policy decide the workflow outcome.
| # | Component | Type | Controls |
|---|---|---|---|
| [01] | Specification agent | LLM AGENT | Interprets tasks, derives requirements |
| [02] | Planner agent | LLM AGENT | Proposes implementation plan |
| [03] | Implementation agent | LLM AGENT | Generates code in sandbox |
| [04] | Independent test agent | LLM AGENT | Generates additional tests (stubbed in alpha) |
| [05] | Adversarial review agent | LLM AGENT | Identifies unsupported claims (stubbed in alpha) |
| [06] | Explanation agent | LLM AGENT | Converts evidence to report (stubbed in alpha) |
| [07] | Policy engine | DETERMINISTIC | Required evidence, thresholds, approval roles |
| [08] | Decision engine | DETERMINISTIC | Accept / reject / repair decision |
| [09] | Sandbox manager | DETERMINISTIC | Isolation, permissions, timeouts |
| [10] | Evidence executors | DETERMINISTIC | Run checks, record EvidenceRecords |
| [11] | AST analyzer | DETERMINISTIC | Impact analysis, test selection |
| [12] | Provenance recorder | DETERMINISTIC | Hash-chained event stream |
Model Routing & Pricing
[ Agent Routing ]
| Agent | Model | Provider | Live | In | Out |
|---|---|---|---|---|---|
| Specification | claude-opus-4-6 | Anthropic | 294 | 4096 | |
| Planner | claude-sonnet-5 | Anthropic | 430 | 770 | |
| Implementation | deepseek-chat‡ | DeepSeek | 276 | 34 | |
| Independent Test | claude-haiku-4-5 | Anthropic | 0 | 0 | |
| Adversarial Review | gemini-3.5-flash | 0 | 0 | ||
| Explanation | gemini-3.5-flash | 0 | 0 |
‡ deepseek-chat is a compatibility alias for deepseek-v4-flash. Alias deprecated 2026-07-24.
[ Pricing — USD per 1M tokens, as of 2026-07-13 ]
| Model | Input $ | Output $ |
|---|---|---|
| claude-opus-4-6 | $5.00 | $25.00 |
| claude-sonnet-5 | $2.00 | $10.00 |
| Introductory pricing through 2026-08-31. $3/$15 from 2026-09-01. | ||
| claude-haiku-4-5 | $1.00 | $5.00 |
| deepseek-v4-flash | $0.14 | $0.28 |
| Cache-miss input; cache-hit is $0.0028. Alias: deepseek-chat until 2026-07-24. | ||
| gemini-3.5-flash | $1.50 | $9.00 |
| Standard tier visible pricing. Actual cost may differ by tier/region. | ||
The harness routes implementation and evaluation through distinct roles. In this alpha, 3 of 6 agent roles are live LLM agents and the remaining evaluator roles are stubbed or deterministic placeholders.
Evidence Tiers
formatting, lint, type check, targeted tests, secret scan
+ integration tests, security scan, mutation testing
+ contract tests, dependency scan, performance, rollback