Generation planning, verified synthesis, and deduplication
This page describes the three stages that turn a reviewed specification into a dataset whose distribution is deliberate rather than incidental:
- Plan exact per-cell quotas from the deterministic pool, offline.
- Synthesize the missing situations with a model, verifying every proposal by policy replay.
- Deduplicate the merged pool globally, including held-out leak detection.
The offline core still runs without credentials. Stage 2 requires an explicit expand command with
a model endpoint; nothing calls a model implicitly.
1. Plan the recipe
uv run decision-forge plan-recipe examples/customer_support/project.yaml \
--output runs/plan --per-cell 5
The planner is an agent with a typed, discoverable tool registry. Each call is appended to a ledger so a plan can be audited and resumed instead of being re-derived in memory:
| Tool | Purpose |
|---|---|
inspect_project |
Nodes, candidates, dimensions, families, strata, declared quotas |
measure_pool |
Generate the deterministic pool in memory, remove duplicates as a build would, and count examples per exact cell |
propose_quotas |
Derive quotas from the objective, optionally scaled by a model advisor |
check_feasibility |
Compare requested quotas with measured supply |
plan_expansion |
Turn the shortfall into verified-synthesis requests plus a call estimate |
commit_plan |
Persist plan.json, quotas.md, and expansion-requests.json |
A cell is (node_id, stratum, split, valid_choices). The plan then separates three kinds of gap:
expansion_requests: cells a model can plausibly fill.unfillablewithheld_out_split: held-out cells. Synthesizing them would reveal the target answer for the split that measures generalization, so they are never requested.unfillablewith an "already exist" reason: the node's declared state space is fully enumerated, so a model could only restate situations the pool already holds.
Derived quotas clamp held-out cells to what generation actually produced, and the clamp is reported in
held_out_clamped. An explicit --mode explicit --targets quota is never changed silently: the
shortfall is reported and the build refuses the plan.
Pass --base-url and --model to let a model re-weight the measured cells. Weights outside the
measured cell set are rejected; the advisor can shape the distribution but not invent cells.
2. Synthesize the missing situations
export FORGE_API_KEY=... # explicit, per command
uv run decision-forge expand runs/demo runs/plan/expansion-requests.json \
--output runs/expansion \
--base-url https://api.deepseek.com/v1 --model deepseek-chat \
--states-per-call 8 --max-calls 20 --max-output-tokens 8000
The model proposes state bindings only, never labels:
{"states": [{"bindings": [{"path": "state.tier", "value": "gold"}], "rationale": "..."}]}
Every proposal is then replayed through the declared policy:
| Gate | Rejection reason recorded in quarantine.json |
|---|---|
| Path must be decision-relevant | Proposal sets an undeclared state path: ... |
| Dimension values are the value space | Proposal value outside the declared dimension domain: ... |
| The policy must produce the requested answer set | Policy replay produced [...] instead of the requested [...] |
| A state with no permitted candidate is useless | Synthesized state leaves no permitted candidate |
| Inputs must be new | exact duplicate / near duplicate of an existing example |
Published rows carry label_basis: model_proposed_policy_verified and a provenance record in
intervention: the request digest, the base episode, and the applied bindings.
Operational properties:
- Held-out data is never synthesized.
ExpansionRequest.splitonly acceptstrainanddevelopment; the build refuses any synthesized row intest. - Provider failure cannot duplicate publication. The journal reserves a call before the request,
accepted rows are checkpointed to
accepted.jsonlafter each call, andexpansion.jsonis written once at the end. A resumed run continues from the next call index and reloads the verified rows. - Budgets are explicit and resumable.
max_callsandmax_input_bytesare excluded from the fingerprint so raising them resumes the same artifact.max_empty_roundsstops a model that keeps repeating itself; verified work is kept and the remaining gap is reported inshortfalls. - Cost is reported, or reported as unknown.
usage.cost_usdisnullwhen the endpoint returns no price. The build reportscost_basis: provider_reported | unreported_by_endpoint, never0.
3. Deduplicate globally and detect leaks
The build in step 4 deduplicates the merged pool before applying quotas. To inspect the result for a completed run:
uv run decision-forge dedup runs/final
Scenario-local identity cannot see that two scenarios describe the same decision problem, because it includes the scenario ID. Deduplication therefore works on the model-visible input:
- A structural key (organization, policy version/context, state, and candidate identities/attributes) groups examples that pose the same problem. Different visible policies remain separate. Wording can only be compared inside a group.
- MinHash over normalized word tokens estimates presentation similarity inside a group, so a cosmetic edit collapses. Word tokens are used instead of character n-grams because a one-word edit in a short presentation changes most character n-grams.
Keeper policy: the most held-out member survives (test > development > train, then lowest ID).
A cross-split duplicate is a split leak: the training copy is dropped and listed in split_leaks,
because a near-identical training row inflates held-out scores. Identical inputs that disagree about
stratum or the label set are reported in metadata_conflicts; keeping one copy would otherwise
silently empty a quota cell.
4. Build with the plan and the synthesis
uv run decision-forge build examples/customer_support/project.yaml \
--output runs/final \
--recipe-plan runs/plan/plan.json \
--expansion runs/expansion
The build re-verifies every synthesized row (current scenario/family/node lineage, family split,
source hashes, policy replay, and policy fingerprint) before merging it, so a hand-edited expansion
artifact cannot inject a wrong label. The merged pool is deduplicated, then quotas are applied. report.json gains synthesized, dedup, and a
generation_cost block with real provider calls and cost basis.
5. Measure whether the data helps
scripts/finetune_evaluation.py closes the loop with a fixed budget:
uv run python scripts/finetune_evaluation.py prepare runs/final --output runs/finetune --budget-examples 2000
# use a candidate-readout trainer with the exported rows, then serve the checkpoint
uv run python scripts/finetune_evaluation.py evaluate runs/final --output runs/base.json \
--arm base_model --base-url https://api.deepseek.com/v1 --model deepseek-chat
uv run python scripts/finetune_evaluation.py evaluate runs/final --output runs/tuned.json \
--arm fine_tuned --base-url http://localhost:8000/v1 --model local-lora
uv run python scripts/finetune_evaluation.py compare runs/base.json runs/tuned.json --output runs/compare.json
prepare exports candidate-probability decision rows and writes a pinned handoff plan. Its example
budget limits the actual training file; selected IDs and file digests make that selection auditable.
It preserves held-out family assignments and refuses a dataset without eligible training or test rows.
Training requires a compatible candidate-readout trainer; see the decision training contract.
The endpoint commands measure generated-choice accuracy separately from candidate NLL. Both arms must use the same held-out cases and fixed prompt. Dataset identity alone does not establish that a training run occurred or improved decision quality.
What this does not claim
Model-written synthesis currently proposes decision-time state, not new policy text. The model cannot
introduce rules, candidates, or evidence, so the label space stays closed and every published label is
traceable to declared policy. Free-text presentation rewrites are available through
decision-forge rewrite, which requires deterministic invariants (unchanged option set and unchanged
numeric values) plus an independent equivalence decision before publication.