DocForge2/DEVELOPMENT_NOTES.md

DocForge2 development notes

This is the running implementation record for DocForge2. It records what is active, what was measured, what changed, what failed, why architectural decisions were made, and which ideas were deferred. Stable user and compatibility contracts still belong in dedicated documentation.

Working rules

• Only one milestone is active at a time.
• main remains the last fully verified milestone.
• Active implementation occurs on dev.
• Every milestone begins from direct repository evidence and ends with focused tests, the complete repository gate, updated measurements, documentation closeout, and a clean pushed state.
• WorldForge, ScrapeStation, legacy DocForge, and production MCP bindings remain out of scope.
• DocForge2 does not self-host during this program.
• Release tags and Forgejo releases require Rob's explicit approval.

Milestone 0 — complete

Milestone 0 established the public successor, preserved the complete lineage and v1 tag, integrated the no-AST and adapter-lifecycle work, froze compatibility guarantees, added repository-native quality and contract gates, and recorded cold/warm performance, memory, rendering, and response sizes.

The central measurement was decisive: a 1,000-node warm exact lookup took about 286 ms while the generation-pinned SQLite query path took about 0.4–1.4 ms. Repeated whole-source loading and validation, not SQLite, is the first optimization target.

Milestone 1 — complete: fast, observable core

Outcome

Warm retrieval should disappear into normal tool overhead. Routine reads must not parse project sources. Status must not render or rebuild hidden work. Results must remain bounded independently of project size.

Starting evidence

• Generic Project.load() walks, captures, parses, rereads, validates, hashes, and checks Git for the complete source set.
• Exact retrieval validates twice around one bounded SQLite query.
• Context compilation performs three full project loads.
• Render status recompiles the complete manual.
• Incremental adapters already prove that manifest attestation can make no-change synchronization and exact retrieval sub-millisecond on a tiny fixture.
• Pinned viewer queries prove the current SQLite schema can serve bounded reads quickly.

Final outcome

Routine generic reads now parse zero canonical source files, use one generation-pinned SQLite snapshot, and return bounded results. Status checks perform no hidden rendering or rebuilding. Structured counters prove those invariants independently of machine timing. Complete loading and deep validation remain recovery and equivalence oracles.

Work log

Linear dependency validation

The inherited dependency-cycle preparation scanned every edge once for every node. The graph validator now constructs dependency adjacency in one edge pass and sorts each adjacency list before an iterative deterministic depth-first cycle check. The iterative stack also removes recursion depth as a failure mode on large valid graphs. The same edge pass now rejects missing sources as well as missing targets.

A 10,000-node regression test counts complete edge-collection iteration passes and caps them at four. The focused correctness and bounded-pass tests pass, and the configured strict source type gate is clean.

One validation command initially included tests/test_core.py in a direct Pyright invocation. Repository Pyright intentionally covers src and tools, so that command reported existing untyped test-result indexing rather than a source defect. Rerunning the repository-configured type gate produced zero diagnostics.

Persistent generic source generations

Generic projects now persist a version-1 source-generation receipt only after complete source loading and fully verified index publication. The receipt binds the explicit generic source contract, project and root identity, adapter, revision, source hash, every canonical/authority/ descriptor regular-file identity, and every source-membership directory identity.

A normal warm check reads no canonical source bytes. It validates the known directories and files directly using device, inode, mode, size, nanosecond modification time, and nanosecond change time. Directory identities detect add, delete, and rename operations without an rglob. Any missing, malformed, incompatible, foreign, or dirty receipt becomes a cache miss and falls back to the full canonical load and row-verification oracle. Successful fallback verification repairs the disposable receipt.

The final 1,000-node evidence run initially exposed a repeatable 52 ms exact-read maximum against the 50 ms target. Profiling showed no source parsing or SQLite cost; each request rebuilt and revalidated 1,000 Path objects from the unchanged JSON receipt twice. The project binding now caches only the strictly validated receipt structure behind its device, inode, size, modification time, and change-time signature. Canonical file and directory identities are still recaptured before and after every query. Receipt replacement or mutation invalidates the cache and fails closed. The same ordered exact-read profile fell from about 40–52 ms to about 18 ms without weakening stale-read refusal.

The receipt is deliberately generic-project behavior. Incremental adapter manifests retain authority over generated or specialist source identities. A one-method legacy adapter continues to work even when it cannot provide a cheap generation.

Request-scoped immutable reads

Index reads now use one read-only SQLite transaction pinned to one verified file signature and one source identity. Existence checks and queries share that connection. Before returning, the request rechecks the index signature and current cheap source generation. A concurrent source or index change fails closed.

Context compilation hydrates nodes and edges from the pinned derived snapshot while retaining profiles from the immutable descriptor. It no longer loads or parses canonical sources. The public full Project.load() and deep ProjectIndex.check() behavior remains the recovery and equivalence oracle.

Focused tests prove that fresh-process-style generic reads can run exact, search, filter, backlinks, dependency, impact, context, and no-change synchronization operations while Project.load() is forbidden. They also prove a final source-generation change is rejected before return and missing/corrupt receipts fall back and repair.

The final clean 1,000-file run recorded:

Operation	Milestone 0 median	Milestone 1 median	Milestone 1 p95
Warm no-change synchronize	142.479 ms	9.192 ms	9.324 ms
Exact node	286.306 ms	17.887 ms	18.577 ms
Search, limit 20	288.793 ms	19.518 ms	19.884 ms
Dependencies, depth 8	287.791 ms	18.117 ms	19.061 ms
Context, 32k, page 20	436.897 ms	25.395 ms	25.867 ms
Render status	150.591 ms	18.969 ms	19.613 ms
Visualization status	—	9.875 ms	10.836 ms

The recorded run came from clean commit 6253c45a5eca01efa8c73ea3dfe4d85c55878ada. Every measured operation passed its p95 threshold and work-counter contract.

Read-only audit reconciliation

The three Milestone 1 audits agreed on the main architecture:

• Keep complete loading and deep checking as independent truth oracles.
• Trust only versioned, identity-bound disposable generation receipts.
• Use one pinned read transaction and retain a final dirty check.
• Hydrate context from the current index.
• Replace full-edge traversal scans with bounded indexed frontier reads.
• Add compact success receipts before allowing large mutations to report post-write size errors.
• Replace hidden render-status rendering with a receipt comparison.
• Add algorithmic counters and parse-count gates alongside wall-clock thresholds.

One audit identified a correctness risk beyond latency: a large mutating MCP operation can commit successfully and then be replaced by result_too_large. This must be fixed in Milestone 1 so exactly-once operations never report a false failure after mutation.

Mutation success receipts

Proposal, preview, and canonical-application MCP mutations now declare an internal response policy. Before runtime validation or mutation, the service proves that a minimum receipt containing the actual input identity and fixed-length hash fields fits the configured output limit. If it cannot, the operation returns a preflight size error with mutation_committed = false and does not call the mutation.

Small results retain the existing full payload. Oversized successful results become a version-1 compact receipt that preserves exact changeset identity, hash, workflow scalars, and lifecycle state while omitting full operations. Application receipts also preserve changed-source counts and derived-refresh status/counts. If the compact form is still too large, the service returns the minimum receipt proven by preflight. It never converts committed success into a post-write size failure.

End-to-end MCP tests exercise two large hash-chained appends followed by canonical application. Each response stays within 1,600 compact JSON characters, exposes the new exact hash, and reports committed success. A separate 700-character preflight test proves the callback and changeset file are never created. Changeset lifecycle receipts now obey the configured changeset byte limit on both write and read.

Receipt-based render status

Successful declared renders now publish a bounded, atomic version-1 receipt below the disposable cache. It binds project/root/adapter/source identity, the normalized view configuration, renderer identity, template and output hashes, byte size, and safe regular-file identities. Generic renders also publish the verified source generation used by cheap status.

Normal status compares only source-generation, descriptor/view, template-file, output-file, and receipt identities. It does not call Project.load(), prepare the renderer, construct HTML, read the full output, rebuild the index, or repair missing state. Missing and corrupt receipts are unverified; source, template, or output changes are stale. An explicit deep option on the Python, CLI, and MCP status surfaces preserves the old side-effect-free full-render equivalence oracle.

Receipt failure after atomic output replacement is reported as degraded publication success, not a false render failure. Canonical application converts the same condition into a degraded derived-refresh report while retaining canonical success. Focused tests forbid source loading and renderer preparation during warm status and cover output, template, missing-receipt, corrupt- receipt, and post-publication receipt-failure behavior.

After race hardening, a 50-sample three-node receipt-status check measured a 3.202 ms median and 3.509 ms p95, compared with the 1.941 ms Milestone 0 three-node full-render status. The small fixture does not show the scaling benefit; the 1,000-node Milestone 0 status baseline was 150.591 ms and will be rerun in the final Milestone 1 evidence pass.

Bounded indexed retrieval

Search, metadata filtering, backlinks, dependency traversal, and impact traversal now query one extra row beyond the requested bound and report limit plus truncated. Backlinks, dependency, and impact APIs accept the same additive limit option through Python, CLI, and MCP surfaces. Omitted limits are capped by the project max_results policy.

Traversal no longer loads the complete edge table and repeatedly scans it. It performs deterministically ordered frontier queries through the existing source primary key or target index. Each request also has a deterministic edge-examination budget derived from its result limit. The response includes candidate_edges_consumed, candidate_edges_limit, and truncation_reason counters so algorithmic work can be asserted independently of machine timing. truncated is true when either another unique result exists or the work budget prevents proving completeness.

The read-only query-plan audit found that source-ordered unfiltered incoming traversal required a temporary SQLite sort with the version-2 (target_id, relation, source_id) index. Direct EXPLAIN QUERY PLAN evidence showed USE TEMP B-TREE FOR ORDER BY. A measured additive (target_id, source_id, relation) index removes that sort. The disposable index schema is now version 3, so existing version-2 indexes rebuild without changing canonical source or proposals. Frontier cursors are streamed and stop immediately on the first omitted unique result. A focused core, CLI, MCP, Ruff, and Pyright gate passes for this work-in-progress slice.

Structured profiling and zero-work gates

DocForge now has an opt-in, request-local diagnostics collector backed by ContextVar. It emits one bounded version-1 aggregate with a fixed operation name, outcome, total elapsed nanoseconds, fixed stage timing keys, and fixed integer counters. It never records paths, node IDs, queries, source text, or SQL. Disabled mode reads no clock and adds no response field, preserving the existing CLI and MCP payloads.

The generic loader, adapter projection and extraction paths, source-generation checks, index checks/synchronization/build/read transactions, render status/preparation/output hashing, MCP runtime validation, and viewer-manager requests now expose direct proof counters. A warm incremental adapter cache hit still counts the enclosing project load, so the counters cannot hide full adapter assembly merely because extraction was reused.

MCP servers and the CLI accept the additive --diagnostics startup option. Diagnostics are attached to structured successes and errors only when the complete MCP response still fits its configured output budget; they are discarded before any primary result or compact mutation receipt. Warm generic error decoration now reads the persisted source generation before falling back to complete loading. Render- and visualization-status error paths explicitly disable both recovery synchronization and complete identity loading.

Context isolation tests cover threads, concurrent async tasks, repeated stages, nested collectors, exceptions, and disabled collection. Repository tests assert that warm success and error reads, render status, and visualization status perform zero project loads, source parses, adapter projection/extraction, index builds, render preparation, output construction, and output hashing. The result JSON schema contains the same closed operation, stage, and counter sets as the implementation.

The maintained tools/milestone1_benchmark.py harness adds hard counter and p95 latency gates to a disposable generic project. The smoke target is part of make gate; the final 1,000-node evidence is recorded in benchmarks/milestone1-2026-07-29.json. The historical Milestone 0 harness remains behaviorally unchanged as comparison evidence; it only exposes shared fixture and measurement helpers to the Milestone 1 harness.

Visualization snapshot freshness

Visualization workers now receive a version-1 snapshot specification containing the exact validated index publication signature: device, inode, size, modification time, and change time. Both the manager and worker reject a launch if that publication changes before startup. The transmitted project root, root fingerprint, source identity, adapter, counts, limits, and confined index path are strictly validated before the worker may serve source or graph data.

Worker health reports index freshness through stat-only comparison. It does not open SQLite and does not renew the browser activity lease. The version-2 viewer-manager protocol validates the complete worker identity and distinguishes an unreachable worker from a live stale worker. A stale worker stays lifecycle running for accurate diagnosis, but the next visualize request stops it and launches a newly validated snapshot instead of reusing it.

Client status separately compares the worker's pinned source identity with IncrementalStateProject.incremental_state(). The composite snapshot is stale if either proof is stale, current only when both proofs are current, and unknown otherwise. A stopped worker has unknown snapshot identity. MCP preserves this state at the top-level staleness field and disables recovery synchronization and full-load error decoration.

Tests cover signature mutation before worker startup, malformed identity, missing and symlinked indexes, stat-only health, unchanged activity, current/unknown/stale source states, live stale workers, non-reuse, and zero-load status. The Milestone 1 benchmark now measures current, stale, not-running, and unavailable visualization status separately with the same zero-work and 50 ms p95 gates as other receipt status operations.

Bounded pagination and exact large-result review

The final Milestone 1 contract audit found that count limits and the global MCP output ceiling were not sufficient. A 1,000-node context response already exceeded the normal 200,000-character tool limit, and one allowed changeset operation can be larger than that limit. Returning result_too_large kept transport bounded but stranded useful evidence.

Version-1 pagination now uses canonical, base64url cursors with a domain-separated SHA-256 corruption checksum. Cursors bind the project, adapter, canonical generation, semantic query, collection hash, and position. They are deliberately unkeyed read tokens rather than authorization credentials. Corrupt tokens fail as invalid_cursor; changed generations or collections fail as stale_cursor with explicit pagination-restart remediation.

Context transport flattens the compiler's deterministic selected entries followed by all explicit omissions, then partitions each page back into the existing arrays. Both item count and exact compact-JSON response size constrain packing. An individually oversized entry becomes a bounded, hash-identified omission and advances the cursor, avoiding an infinite retry while preserving the fact that evidence was excluded.

Changeset list, inspection, validation, and diff reads preserve direct full-result defaults while MCP uses bounded pages. Pages retain exact changeset identity and hash. Large operation pages compact content-bearing fields into hashes and character counts. A single oversized structured diff is serialized once as canonical ASCII JSON and returned through hash-bound chunks that reconstruct the exact legacy operations and changes arrays. This solves transport growth without lowering canonical changeset limits or adding cursor storage.

The benchmark now validates zero-work and operation-specific counters for every warmup and measured sample, records bounded semantic response summaries, covers filter, backlinks, outgoing and incoming traversal, and measures current/stale/missing/corrupt render receipts plus all visualization lifecycle states. A maintained query-plan test prevents the incoming traversal temporary sort from returning.

Milestone closeout

The complete repository gate passed with 138 tests and 77 subtests, zero Pyright diagnostics, warning-strict execution, package builds, contract checks, and both benchmark smoke gates. The clean 1,000-node benchmark passed all maintained thresholds and is interpreted in docs/MILESTONE_1_BASELINE.md. Compatibility, measured decisions, limitations, and scope evidence are frozen in docs/MILESTONE_1_CLOSEOUT.md.

Milestone 1 made no storage rewrite, self-hosting change, production integration change, tag, or release.

Initial design constraints

• Full rebuild remains the recovery and equivalence oracle.
• Canonical content remains authoritative.
• Existing one-method load_projection() adapters remain unchanged.
• No-AST adapters remain first-class.
• Indexes, source-generation receipts, and caches remain disposable.
• Cheap reads may trust only identity-bound, versioned, corruption-checked receipts.
• Any optimization must fail closed on source mutation and must preserve stale-read refusal.

Future ideas and suggestions

These are notes, not commitments:

• A stable source-generation provider may deserve a public adapter capability only after both the generic project and one incremental adapter prove the same boundary.
• Profiling receipts could eventually feed the human-facing project control panel, but Milestone 1 should expose structured data before adding UI.
• A durable telemetry exporter remains deliberately deferred. Request-local bounded aggregates are enough to prove compiler work in Milestone 1 without adding persistence, cardinality, or privacy risks.
• The stat identity is a cheap publication proof, not a cryptographic integrity scan. Full index validation remains the launch and query oracle.
• Cursor authentication remains deliberately absent. If read cursors ever carry authority rather than bounded positions, they will need a different versioned security contract and persisted key lifecycle.

Milestone 2 — complete: agent retrieval and MCP experience

Audit reconciliation

Three independent read-only audits covered effective policy and bootstrap, task-shaped retrieval and context capsules, and generation diffs plus client configuration and doctor checks.

They agreed on these boundaries:

• Keep the project descriptor at schema version 1. Process capability and client configuration are machine-specific bindings, not canonical project content.
• Preserve the legacy adapter-policy payload, no-AST shorthand, tool names, default tool ordering, one-method adapters, and custom context provider.
• Add one versioned effective-policy authority and derive bootstrap, contract, instructions, and access reporting from it.
• Add one task-context operation with a closed task-kind vocabulary and one immutable, generation-pinned retrieval plan. Do not create a tool for every task kind.
• Produce evidence gaps only from declared plan requirements and completed bounded checks. Never infer missing facts from arbitrary project naming.
• Record only the latest bounded generation transition as disposable evidence. Do not add a history database.
• Preview client configuration by default. Any write must be explicit, atomic, merge-preserving, and backed by a verified client-format driver.
• Keep doctor strictly read-only. It must not bootstrap, synchronize, build, render, start a viewer, or rewrite client configuration.

Versioned effective policy and session contract

The binding now composes an immutable version-1 policy containing capability mode, adapter evolution, AST and Logic behavior, synchronization and integrity levels, render and viewer behavior, profiling, blocked tools, prohibitions, and explicit precedence. --no-ast is a restrictive override. The exact legacy adapter_policy response remains a projection of the new object.

Bootstrap reuses the identity already proven by synchronization and no longer reloads the complete project. Its additive version-1 session contract reports binding, generation, effective policy, actual registered surfaces and mutation access, render policies, first operation, filtered workflow, and prohibitions. Read mode does not recommend proposals. Proposal mode recommends registration and review only with writer access. Application is recommended only when the exact-hash applier is enabled.

Existing factory defaults and tool order remain unchanged. Explicit application mode fails closed without an applier. Operator mode is reserved and currently adds no tools.

Versioned task retrieval and context capsules

The first Milestone 2 retrieval slice adds one docforge_get_task_context read tool rather than a family of task-specific tools. Its closed task kinds are change, implementation, failure, ownership, test, operation, and release. One immutable RetrievalPlanV1 derives exact or lexical focus, bounded bidirectional graph traversal, metadata hydration, required evidence categories, and fixed work budgets from the project descriptor and effective policy.

The public executor re-derives every submitted plan before opening SQLite. It rejects modified steps, task identity, requirements, category order, bounds, policy identity, or hashes as invalid_retrieval_plan. Traversal binds the project relation set by canonical hash and queries the already-validated edge table by endpoint, avoiding relation-sized SQL parameter lists. Version-1 internal ceilings are 1,000 evidence items, 100,000 candidate edges, and 10,000 task query characters.

The executor uses one immutable SQLite read generation. It rejects missing explicit focus, blocks unresolved or tied lexical focus, stops at deterministic evidence and candidate-edge limits, and checks source identity again when the transaction closes. ContextCapsuleV1 binds the generation, policy, request, plan, evidence collection, and complete capsule with canonical hashes.

Project relation names remain authoritative. The core recognizes only a versioned alias map for structure, implementation, dependency, execution, data, evidence, and context. Unknown allowed relations stay visible under their raw names as unclassified. Required evidence diagnostics distinguish categories the project never declared, completed bounded checks with no selected evidence, and incomplete checks caused by a result, work, token, or response limit.

Each evidence item carries a stable content hash, confined source identity, shortest selected graph path, every additional qualifying relationship reason observed during traversal, and explicit limitations where the current graph cannot prove evidence type, extractor identity, relationship source provenance, or observation time. The planner contains no Logic operation, so no-AST bindings can use task context without weakening their existing Logic prohibition.

Path direction is relative to the preceding traversal node. Additional relationship reasons use the evidence node as their direction subject. Candidate-edge and unclassified-relation ceilings produce explicit omissions and bounded summaries.

MCP pagination preserves the complete plan, collection, and capsule hashes while returning bounded pages. Its cursor additionally binds the effective policy and task request. An individually oversized item advances once as a hash-identified omission. A later generation or policy change fails closed as stale_cursor.

The legacy profile-context contract remains intact. A custom context provider does not silently gain core task planning. Version 1 defines no custom task-planner extension, so the additive tool returns task_context_unavailable without synchronization or a complete projection load.

Two independent pre-commit audits reproduced and closed plan-forgery, relation-sized SQL, SQLite-parameter portability, ambiguous relationship-direction, missing work-limit evidence, schema/runtime drift, incomplete page hashing, and custom-provider hidden-load defects. Regression coverage includes 33,005 valid relation names, fixed extreme project limits, tampered plans, evidence-relative directions, edge and unclassified limits, schema-valid pages, changed cursor semantics, oversized evidence advancement, no-AST retrieval, and legacy complete-projection adapters.

The complete repository gate passes with 158 tests and 101 subtests, zero Pyright diagnostics, warning-strict execution, package builds, public-contract validation, and the maintained Milestone 0 and Milestone 1 smoke benchmarks. Gitleaks 8.30.1 reports no secret findings in the working tree.

Latest-generation diff receipt

Three read-only audits reconciled the index publication, public transport, compatibility, and no-AST boundaries before implementation. The selected design stores one disposable generation-diff.json receipt. It does not add a history database, arbitrary generation selectors, source text, rendered content, or Logic details.

Before a build loads current source, it accepts an existing index only when its exact main-file inode has a matching stable whole-file attestation and no WAL, journal, or shared-memory sidecar. It then captures that predecessor through an immutable main-file transaction. The capture validates the SQLite application and schema IDs, project/root/adapter binding, integrity, complete node and edge rows, Logic aggregate identity, FTS count, metadata hashes and counts, and final file signature. It never calls normal check or synchronization and never repairs predecessor evidence.

The final source revalidation now compares exact nodes and edges in addition to source hash, revision, and Logic. A verified predecessor that maps the same source identity to different graph content fails before publication as generation_collision. This closes a pre-existing adapter determinism gap found during the generation-diff audit.

SQLite replacement is now the explicit derived mutation commit point. Whole-file attestation, cheap source-generation, and generation-diff receipts publish independently afterward. Any post-commit receipt failure returns status = ok, index = published, a bounded degraded publication record, and receipt-stage names. It never rolls back the new index or reports a false failed mutation. Attestation hashing checks the exact index signature before, during, and immediately before receipt publication.

Version-1 diff semantics compare every core Node field by stable node ID and exact edge triples. Node renames are removal plus addition. Edge changes are removal plus addition. Exact summary counts and a full ordered item-hash collection cover every change. Retained details are deterministically ordered and independently capped at 1,000 items and 1 MiB with explicit item- or byte-limit evidence. A first build or untrusted predecessor is a baseline with no fabricated all-added result. A same-generation reindex republishes the existing meaningful transition against the new index file identity instead of erasing it with an empty diff.

The additive public surfaces are:

• CLI generation-diff [--limit N] [--cursor OPAQUE].
• MCP docforge_get_generation_diff(limit=None, cursor=None).
• Telemetry operations cli.generation-diff and mcp.generation_diff.

Public reads do not open SQLite, call project.load(), extract an adapter projection, parse source, check, synchronize, build, or repair. They strictly validate the bounded receipt, compare stable receipt and index file identities, require two matching cheap source-generation checks, and report unknown for legacy adapters without that capability. Missing, corrupt, foreign, oversized, symlinked, stale, or concurrently changed evidence remains a read-only status outcome.

Generation-diff pagination binds the complete stored receipt hash and effective policy. That hash already covers project, generation, graph, collection, and committed-index identity. Page size may change. A replaced receipt returns stale_cursor. One top-level pagination object owns the only cursor. The nested version-1 page uses receipt_header.stored_receipt_hash so it never misrepresents the complete receipt hash as the hash of a partial header. The summary distinguishes additional retained pages from details permanently omitted by the fixed publication limits.

Adversarial coverage now includes strict runtime/schema rejection, predecessor attestation and generation identity, live and synthetic SQLite sidecars, cache-root symlink substitution, source/sidecar changes during diff preparation, independent receipt failures, and degraded post-commit identity and durability failures. Focused verification passes the direct, CLI, MCP, schema, pagination, legacy, incremental no-AST, and zero-work suites. The complete repository gate passes with 176 tests and 113 subtests, zero Pyright diagnostics, package builds, web checks, and the maintained Milestone 0 and Milestone 1 smoke benchmarks. Final independent re-audit is in progress before this slice is committed.

The final dense benchmark uses a 1,000-node transition with 1,000 changed details. Its receipt is 775,663 bytes. Direct status is 37.659 ms median and 39.108 ms p95. A maximum-size MCP request returns 307 items in 199,754 bytes at 54.037 ms median and 56.617 ms p95. Four pages reconstruct all 1,000 retained details in 652,798 bytes at 201.55 ms median. Peak RSS is 79,096 KiB. Every hidden-work counter remains zero; the read performs exactly two cheap source-generation checks.

Measurement found and removed two avoidable costs before commit. Receipt loading had repeated the complete 1,000-item validator solely to check project identity; it now validates once and compares the three binding fields directly. Page fitting had encoded every growing prefix; it now uses an exact logarithmic search and retains the hash-only oversized-item omission path. The maximum page fell from 272.06 ms p95 to 56.617 ms p95, while full traversal fell from roughly 859 ms to 203.55 ms p95. Regression tests require one receipt validation and at most 15 response encodes for 1,000 page candidates. Final independent publication, contract, and performance audits approve the slice for commit.

Deterministic client configuration and read-only doctor

Client integration remains an explicit machine-local boundary rather than canonical project content. docforge configure {codex,claude,openclaw} --project PATH previews a deterministic version-1 fragment by default. An optional output path publishes only a standalone fragment into an existing real directory. Publication is create-only, private-mode, no-follow, bounded, and conflict-aware. Existing differing client configuration is never merged, replaced, or silently overwritten.

Generated commands use the exact current virtual-environment Python executable with isolated module startup. The binding records explicit read, proposal, or application mode, no-AST policy, render policy, empty environment, and bounded timeouts. Proposal and application generation fail closed unless the descriptor declares the required writer and matching applier identity. A generic CLI cannot reconstruct project-owned adapter composition, so custom adapters return an explicit unavailable result instead of generating a misleading command.

Codex and OpenClaw fragments include their verified timeout fields. Claude JSON fragment syntax is supported, while its timeout representation remains an explicit warning. The configuration result has a strict JSON schema and canonical plan hash. Diagnostics are additive and remain disabled by default.

docforge doctor --client CLIENT performs bounded, non-mutating inspection only. It reads the project descriptor and selected client file through stable, directory-bound, no-follow handles; parses at most 1 MiB and 256 server entries; selects at most one exact project binding; validates the closed server argument set; checks executable, capability, declared authority, no-AST, timeouts, environment-key names, and tool-filter presence; and performs only a stat-level index presence check. It never loads canonical sources, opens SQLite, starts MCP, executes the configured command, synchronizes, builds, renders, starts a viewer, or writes client configuration.

Doctor reports healthy, degraded, or unhealthy with stable process exit codes 0, 1, and 2. Secret environment values are parsed only to enforce bounded string limits and are never returned. Unknown or unverified client tool filtering, Claude timeout representation, implicit legacy capability mode, shadowed authority, and missing disposable indexes are warnings. Unsafe paths, malformed matching entries, unexpected executables, wrong project roots, invalid authorities, and missing configuration are failures.

The first benchmark smoke failed for the correct product reason: its disposable doctor fragment used the shared path /tmp/doctor-codex.toml, where a previous run had left different content. The harness now creates a project subdirectory inside one unique temporary root and places the client fragment beside it. This preserves create-only conflict safety and makes every run disposable.

The final pre-commit 1,000-node audit sample passes every provisional Milestone 2 gate. Task context reconstructs 1,000 candidates as 108 cited evidence records and 892 explicit bounded omissions across 11 pages in 703.808 ms. Generation diff reconstructs 1,000 changed details across 10 pages in 427.450 ms. Maximum pages remain below the 200,000-byte MCP budget; generation diff uses 199,566 bytes and proves that diagnostics are discarded before the primary result. Isolated peak RSS is 86,168 KiB.

Configuration preview now includes a bounded real import probe of the exact isolated interpreter, so its provisional single-sample latency is about 315 ms rather than the earlier sub-millisecond derivation-only figure. Doctor remains below 1 ms on generated disposable configurations. Every configuration and doctor hidden-work counter is zero.

The aggregate make gate includes the Milestone 2 smoke benchmark. The frozen candidate passes 205 tests and 120 schema subtests, strict warnings, Ruff, formatting, Pyright, web checks, compilation, lock and dependency checks, package builds, and all three milestone smoke benchmarks. Three independent final audits approve client publication and policy binding, doctor fail-closed behavior, and benchmark/contract coverage. Clean-revision benchmark evidence is still required before closeout.

Milestone 2 closeout

Candidate commit fb0df5e4a1c591c2a84788fd4814d98550f11863 passed the clean ten-sample Milestone 2 benchmark. Task-context complete traversal measured 703.561 ms median and 721.847 ms p95 across 11 bounded pages. It reconstructed the exact 1,000-candidate collection from 108 cited evidence records, 891 original token-budget omissions, and one hash-attested response-limit surrogate. Generation-diff complete traversal measured 418.607 ms median and 425.315 ms p95 across 10 pages.

Read and no-AST bootstrap remained below 10 ms p95. The maximum generation page used 199,566 bytes of the 200,000-byte budget and correctly discarded diagnostics before primary evidence. Configuration preview measured about 314 ms median and 365 ms p95 because it proves the real isolated interpreter import on every invocation. Codex and OpenClaw doctor checks remained below 0.6 ms p95; Claude remained explicitly degraded because its timeout format is unverified. Isolated-process peak RSS was 86,448 KiB against the 262,144 KiB gate.

All measured configuration and doctor counters were zero. Task-context pages performed one index check and two cheap generation checks with no loads, parses, synchronization, builds, extraction, rendering, or viewer work. Generation-diff pages performed two cheap generation checks and no index check. The canonical machine-readable result is benchmarks/milestone2-2026-07-29.json.

Milestone 2 is complete. Follow-up ideas stay explicitly later-scope: avoid recomputing the task-shaped capsule for every continuation page, add authenticated continuation when the threat model requires it, verify a native Claude timeout representation, and introduce adapter-owned launcher metadata before generating configurations for custom adapters.

Milestone 3 — complete: independent projections

Milestone 3 began only after main and dev were aligned at the verified Milestone 2 closeout. Three read-only audits ran before source changes:

• Manual planning, immutable packages, renderer isolation, receipts, preview/application integration, and full/incremental equivalence.
• Portable graph planning, static artifacts, the live viewer boundary, worker protocol, and static plus interactive accessibility.
• Packaging, optional dependencies, public contracts, projection policies, performance, incremental fragments, and maintained gates.

The active design constraints are unchanged: renderers consume one validated immutable generation; manual and graph plans remain separate; the live viewer is not retrieval authority; core remains usable without rendering; status performs no hidden rendering; full rendering remains the recovery and equivalence oracle; no storage rewrite is assumed.

Milestone 3 architecture decision

The three audits converged on one compatibility-first boundary:

• The existing docforge.render_contract names, GenericHtmlRenderer.prepare() signature, generic_html renderer identity, and byte output remain the version-1 compatibility surface. They become adapters over the new manual-planning path rather than being changed in place.
• New ManualRenderPlanV1, GraphViewPlanV1, ProjectionPackageV1, and ProjectionReceiptV1 contracts use strict canonical JSON, deterministic ordering, independent item and byte bounds, exact generation and policy binding, and content-derived identities.
• Plans and packages contain selected graph facts and bounded content. They never contain a project object, SQLite handle, absolute project or index path, arbitrary query, command, or project-provided executable code.
• The planner owns graph selection and meaning. A renderer may transform only a validated package into declared artifacts and cannot select nodes, invent relationships, crawl the project, choose publication paths, or mutate canonical sources.
• Manual and portable graph renderers live behind independent import boundaries. Renderer dependencies load lazily. Default installation behavior remains compatible during the initial migration; optional dependency changes require their own verified packaging decision.
• Portable graph rendering is additive. It does not replace or silently change docforge_visualize, graph-browser@17, the viewer-manager protocol, or the query-backed live viewer.
• Effective policy version 1 remains frozen. Milestone 3 introduces a version-2 projection-policy view for manual auto|explicit|disabled, portable graph explicit|disabled, and live viewer on-demand|disabled enforcement, while retaining the version-1 projection for existing clients.
• Publication commits content-addressed artifacts first, renderer evidence second, and a bounded generation/view manifest last. Status remains receipt-only. Failures after artifact replacement report committed degraded success rather than an ordinary failed mutation.
• Full planning and rendering remain the recovery and equivalence oracle. Incremental fragments are disposable, keyed from complete plan semantics, and may be reused only when byte-exact artifact equivalence is proven.
• The live source endpoint must stop reading mutable canonical files behind a pinned graph snapshot. Portable artifacts never inherit that path-bearing behavior.

The first implementation slice freezes existing golden output, adds the four versioned contracts and validators, introduces pure manual and graph planners, and makes the legacy manual renderer a compatibility wrapper. Publication hardening, detached rendering, incremental fragments, portable graph publication, independent policy enforcement, accessibility, and maintained performance gates follow on top of that frozen boundary.

Milestone 3 contract slice

The first slice now implements:

• Strict Draft 2020-12 schemas and runtime canonical-hash validation for manual plans, graph plans, projection packages, and projection receipts.
• A deterministic manual planner that owns page selection, navigation, cross-references, backlinks, search documents, component assignments, orphan diagnostics, and cycle diagnostics.
• A deterministic graph planner with exact-root or metadata-only lexical scope, closed filters, explicit node/edge/work bounds, deterministic omissions, path/source-body exclusion, and no-AST Logic exclusion.
• A separate docforge_renderers.manual package. Its renderer accepts only a validated package and has no project, SQLite, publication-path, or filesystem-write API.
• The frozen GenericHtmlRenderer compatibility shim over the new planner/package/renderer pipeline. The alpha artifact remains exactly 2,043 bytes with output SHA-256 81656bb89debc7ad1fbe8bc290e9a3ba90664442b17a6d57e908d30d20c47f77 and legacy render identity 1c0a49c28ba3b0dabf94be36e75def197dee1be3cb73ac405b09875383c8dc5f.
• Rejection of project-template scripts, inline event handlers, javascript: URLs, embedded browsing contexts, and refresh redirects.
• Wheel inclusion for both typed packages and every published JSON schema. Importing docforge no longer imports markdown_it or the manual renderer package.
• A live-viewer correction: source evidence now comes from the pinned index generation. The viewer no longer reopens mutable canonical files behind an older graph snapshot.

The new repository-native contract target passed 91 tests and 120 subtests at the slice boundary. The combined projection, rendering, and live-viewer focus passed with byte-exact compatibility and no hidden source/path authority.

Durable portable graph publication

The portable graph path now has its own declared graph_render views, pure plans, fixed portable_graph_html renderer, content-addressed artifact store, renderer receipts, and one bounded generation/view manifest as the publication commit. It supports Nodes, Flow, and Web without including Logic. Static HTML contains the complete pre-rendered graph and treats JavaScript as progressive enhancement.

Publication revalidates source, view, artifact, receipt, and output identities across replacement. Status reads only bounded manifest and receipt evidence. It never plans or renders. Repair may restore a declared output from its content-addressed artifact. A post-artifact failure that cannot be rolled back returns explicit degraded committed evidence rather than reporting an ordinary failed mutation.

Detached workers and incremental fragments

Manual and portable graph packages execute through one fixed one-request child protocol. The parent launches isolated Python from a trusted working directory with a sanitized environment, spools stdout to disk, reads one bounded canonical response, and validates the complete artifact and receipt identity. The worker accepts only the two built-in renderer identities. Requests are bounded by the 24,000,000-byte package contract, actual artifact transfer by 20,000,000 bytes, and execution by a 30-second timeout.

Manual fragment records are semantic, versioned, canonical, hash-bound, and stored below a dedicated confined cache. The worker independently recomputes the expected page fragment before using a record. Corrupt, forged, oversized, stale, or aggregate-oversized records fall back to the full detached render. Cold fragment creation is compared byte-for-byte with that full oracle before cache publication. The cache retains only the current inventory and is capped at 10,000 entries and 64,000,000 bytes.

Independent policies and accessibility

Projection policy version 2 independently composes manual auto|explicit|disabled, portable graph explicit|disabled, and live viewer on-demand|disabled. CLI, MCP, generated client configuration, doctor, render services, canonical application, onboarding, and viewer-manager entry points enforce their relevant policy. Status remains available when an active operation is disabled.

Generated client evidence binds the projection policy, its hash, projection availability, and the current descriptor hash into the configuration hash. Validation cross-checks omitted default selectors against the bound descriptor so coordinated policy and availability drift fails closed. The version-1 effective-policy payload remains unchanged for existing clients.

Pinned Playwright 1.62.0 and axe-core 4.12.1 gates exercise the frozen manual, portable graph, and live viewer with selected WCAG A/AA axe tags and keyboard interaction flows. Portable and live graph presentation received only the minimal contrast and nested-role corrections needed by those gates.

Scale and runtime hardening

The first 1,000-node full benchmark exposed recursive strongly connected-component traversal in manual planning. Cycle detection now uses an iterative two-pass traversal. A regression covers the descriptor maximum of 10,000 nodes as both a deep acyclic chain and one strongly connected component.

The isolated wheel proof also exposed a Python runpy warning when the worker module was imported during package initialization before -m execution. A private fixed module entrypoint now owns child startup. Malformed child input returns code 2 with empty stdout and stderr.

Configured render ceilings above 20,000,000 bytes remain accepted for compatibility, and small actual artifacts render normally. The detached protocol still rejects an actual transfer beyond its fixed 20,000,000-byte boundary.

Milestone 3 closeout

Candidate f5dccb5e1c312121f1af63780162f593d9363b98 passed the complete repository gate: formatting, Python and web lint, strict types, compilation, 281 tests and 272 subtests, three accessibility flows, lock and dependency checks, package builds, and all milestone smoke benchmarks. The maintained projection contract subset passed 142 tests and 236 subtests.

The clean ten-sample 1,000-node benchmark passed every latency, memory, response-size, no-work, and equivalence gate. Manual full rendering measured 810.490 ms p95, portable graph full rendering 323.690 ms p95, and receipt-only status 111.381 ms and 59.331 ms p95 respectively. Direct detached worker peaks were 88,580,096 and 89,583,616 bytes. The separately gated production manual worker peak was 104,771,584 bytes. Production cold, warm, forced-full, add, change, delete, and reorder outputs were byte-identical.

Production warm fragment rendering measured 2,206.540 ms p95 versus 978.870 ms for forced full. Milestone 3 therefore closes the fragment isolation, invalidation, equivalence, and recovery contract without claiming a throughput win. Later optimization must begin from that evidence.

The exact method and measurements are recorded in docs/MILESTONE_3_BASELINE.md and benchmarks/milestone3-2026-07-29.json. The candidate passed an isolated wheel CLI/MCP/worker proof. Gitleaks 8.30.1 found no findings across the six Milestone 3 commits or candidate tree.

Milestone 3 is complete. No tag, release, production integration repointing, WorldForge change, ScrapeStation change, storage rewrite, or self-hosting dependency was introduced. Milestone 4 remains directional and has not started.

Milestone 4 — complete: adapter SDK and product documentation

Milestone 4 activated from the verified Milestone 3 closeout. Its scope was deliberately additive: freeze an adapter-authoring boundary, prove narrow language references, make heavy frontends optional, generate command and client integration evidence, and document adoption. WorldForge, ScrapeStation, the legacy repository, production bindings, self-hosting, storage replacement, and release publication remained excluded.

Adapter SDK and conformance

docforge.adapter_sdk is now the public authoring facade for typed projections, manifests, source contributions, complete assemblies, incremental loaders, project bindings, core graph models, and conformance reports. Complete evidence includes primary graph and function Logic. An incremental adapter that publishes Logic must provide an independent load_complete_assembly() oracle.

The conformance helper repeats complete loading for determinism, compares the complete assembly with load_projection(), and proves exact complete/incremental graph-plus-Logic parity. It does not pretend to replace separate confinement, restart, no-AST, corrupt-cache, or retrieval tests.

An adversarial review found that extraction caches and adapter assemblies lacked aggregate boundaries. Version-1 cache reads and writes are now regular-file-only, identity-checked, capped at 64,000,000 bytes and 10,000 sources, and fail safely to a miss. Adapter projects enforce primary node limits and deterministic edge and Logic multipliers before publication.

Reference language evidence

The Python reference uses the standard-library AST and publishes files, modules, classes, functions, arguments, local static imports, and function Logic. Its manifest uses tokenization rather than AST parsing, so unchanged warm builds perform no syntax parse.

JavaScript and TypeScript use separate pinned optional Tree-sitter grammars. They publish files, modules, classes, functions/methods, static project-relative imports/re-exports, and function Logic. Their manifests use a closed comment/string-aware ESM scan and do not load Tree-sitter.

C++ uses explicit non-overlapping source roots and one confined compile_commands.json as translation-unit inventory and fingerprint evidence. It validates entries but never executes a command, compiler, response file, or project program. It publishes syntax and directly resolvable project-local quoted includes. Its manifest currently parses for include discovery, so no zero-warm-parser claim is made.

None of the references claim compiler-resolved calls, inheritance, types, symbol references, compiler include semantics, macro semantics, runtime behavior, or semantic ownership.

The first packaging proof exposed that language parsers were inherited as mandatory dependencies. The base wheel now depends only on Markdown and MCP packages. Python works in the base wheel; JavaScript, TypeScript, C++, and all-language extras install their respective Tree-sitter frontends. Missing extras return a closed error with the exact install target.

Fixed reference MCP and launchers

A strict .docforge/reference-adapter.toml selects one fixed reference language, project identity, and explicit source roots. C++ additionally requires the compilation database. The fixed python -m docforge.reference_mcp binding constructs only repository-owned providers and exposes the 21 read tools.

The initial launcher design generated python -I -m for arbitrary project module names without proving that isolated Python could resolve them. The corrected AdapterLauncherV1 accepts only an installed project-confined top-level module or the exact trusted docforge.reference_mcp exception. A fixed bounded find_spec probe runs under isolated Python without importing project code. Tests then launch both a real project module and the real reference MCP process.

Generated Codex, Claude, and OpenClaw fragments bind the exact launcher, descriptor, source availability, effective and projection policy, interpreter, canonical arguments, empty environment, artifact bytes, and hashes. There is no arbitrary command, argument list, working directory, environment, discovery, or callable selector.

Generated reference and documentation gates

CLI tables are derived from the real argparse subcommands. MCP tables are derived from a real application-enabled server's list_tools() registrations, including capability surface, arguments, descriptions, and input-schema hashes. The generated artifact contains 28 CLI rows and 36 MCP rows.

An adversarial review found a target-change race between the generator's identity check and replacement. Publication now locks cooperating generators, uses no-clobber publication for a missing target, and uses Linux atomic exchange plus displaced-byte and file-identity verification for an existing target. Raced data is restored or retained for recovery rather than discarded.

docs-check validates generated drift, local files and anchors, one H1 per page, README-rooted reachability, required Milestone 4 inventory, the generated notice, and every documented reference-adapter TOML example against the packaged schema.

Candidate, adoption, and scale evidence

Frozen executable candidate 95271dcf2e48045b9d3aed9b9ea09c7fc155692c passed formatting, Python and web lint, strict Pyright, compilation, 142 contract tests and 268 subtests, 347 total tests and 402 subtests, three accessibility flows, lock and dependency checks, package builds, offline wheel adoption, and every maintained smoke benchmark.

The offline adoption proof installed the base wheel from the lock without network access. It proved no Tree-sitter package or module was installed, built and checked a real Python reference project, performed isolated MCP bootstrap/search/get-node over the exact 21 read tools, and proved that base-only C++ fails with optional_dependency_missing and docforge[cpp] remediation.

The clean 334-source benchmark produced 1,002 primary nodes, 1,001 primary edges, 334 Logic projections, 2,338 Logic nodes, and 2,338 Logic edges. Complete and incremental output matched exactly. Warm build p95 was 1,265.387 ms with zero ast.parse and zero extract_source calls. Complete equivalence, corrupt cache recovery, and corrupt index recovery were all below 1.9 seconds. The largest per-operation traced peak was 69,997,166 bytes and process high-water was 78,798,848 bytes. Exact hashes and method are recorded in benchmarks/milestone4-2026-07-29.json and docs/MILESTONE_4_BASELINE.md.

Milestone 4 is complete. No tag or release was created. Milestone 5 remains unstarted until its own active-slice contract is frozen.