Benchmarks
Offline pipeline measurements for the cold-and-warm-offline-v1 protocol. The protocol measures
deterministic generation, policy verification, deduplication, quota selection, and artifact writing.
It makes no model calls, so no provider latency or cost is included.
Protocol
- Start from the shipped fictional example and replicate its four seed scenarios across
Nsynthetic scenario families. - Build twice into a fresh output directory: the first run is a cold cache, the second reuses it.
- Assert that both runs produce the same dataset digest before reporting timings, so a speedup can never come from a silently different dataset.
- Record the engine fingerprint, dependency versions, host platform, accepted episode count, artifact size, and specification size with the timings.
uv run python scripts/benchmark_pipeline.py \
--workspace runs/benchmark-workspace \
--report runs/benchmark.json \
--families 25 100 400
Reference run
Measured on 2026-11-01. The unedited JSON report records the engine fingerprint and dataset digests. MB and KB below use decimal units.
| Component | Value |
|---|---|
| Host | macOS 26.6, arm64 |
| Python | 3.12.12 |
| Decision Forge | 0.3.0 |
| pydantic / filelock | 2.14.2 / 4.2.4 |
| Families | Scenarios | Accepted episodes | Cold (s) | Warm (s) | Episodes/s (cold) | Run artifacts | Specification |
|---|---|---|---|---|---|---|---|
| 25 | 100 | 700 | 0.221 | 0.140 | 3,168 | 1.18 MB | 43.9 KB |
| 100 | 400 | 2,800 | 0.835 | 0.570 | 3,352 | 4.60 MB | 130.8 KB |
| 400 | 1,600 | 11,200 | 3.340 | 2.380 | 3,354 | 18.30 MB | 481.8 KB |
Cold throughput ranged from 3,168 to 3,354 accepted episodes per second on these inputs. Each size has one cold and one warm measurement; these results characterize the recorded fixture and host.
Reading these numbers
- Timings are single-machine and single-process. They describe the offline compiler, not a service.
- Model-backed steps (
compile-corpus,expand) are excluded: their latency and cost are dominated by the endpoint and are recorded per run incalls.jsonandexpansion.jsoninstead. - The dataset digest is asserted equal across cold and warm runs; a benchmark that changes the dataset is a failure, not a speedup.
- Streaming, parallel workers, and multi-node execution are not implemented, so no such numbers are claimed.
Cost
Generation is free of model calls. The two archived model experiments report their own costs:
successful document-compilation runs at $0.293428 (Claude Opus 5.5) and $0.1505675 (GPT-6.1 Sol),
and multi-step workflows at $0.036412 for 22 calls. The compilation ledger totals $1.08587197 in
known costs across attempts plus five attempts with unknown cost; total billed cost is unknown. See experiments/
for the preserved journals, including failed attempts whose cost is unknown rather than zero.