diff --git a/ACTIVE_SLICE.md b/ACTIVE_SLICE.md index 59bc152..4d6b119 100644 --- a/ACTIVE_SLICE.md +++ b/ACTIVE_SLICE.md @@ -1,16 +1,13 @@ -# Active milestone +# Milestone state ```text -Milestone: 3 — independent projections -Goal: Make manual output, portable graph artifacts, and the live viewer independent generation-pinned consumers of the validated graph. -In scope: Versioned ManualRenderPlan and GraphViewPlan; immutable projection packages and receipts; independent manual and graph renderers; projection policies; incremental fragments; equivalence, recovery, accessibility, response-size, performance, and memory gates. -Out of scope: Adapter SDK expansion; remote render services; shared render farms; third-party renderer ecosystems; self-hosting; storage replacement; WorldForge or ScrapeStation changes; production MCP repointing; tags and releases. -Done when: Manual and graph plans are versioned and bounded; renderers cannot crawl project state or mutate canonical facts; portable and live graph modes remain separate; policies are enforced independently; status is receipt-only; full/incremental output is equivalent; accessibility and maintained scale gates pass. -Status: Active implementation. Three independent audits were reconciled before source changes. The -versioned plan/package/receipt contracts, pure manual and graph planners, isolated manual renderer, -legacy byte-compatibility shim, packaged schemas, and pinned live-source correction are implemented -and focused-green. Durable publication, portable graph artifacts, detached workers, fragment -equivalence, independent policy enforcement, accessibility, and maintained scale gates remain. +Last completed milestone: 3 — independent projections +Outcome: Manual output, portable graph artifacts, and the live viewer are independent generation-pinned consumers of the validated graph. +Evidence: Clean candidate f5dccb5e1c312121f1af63780162f593d9363b98; 281 tests and 272 subtests; 3 accessibility flows; clean 1,000-node ten-sample benchmark; isolated wheel proof; no secret-scan findings. +Active milestone: None. +Next directional milestone: 4 — adapter SDK and product documentation. +Status: Milestone 3 is closed. Milestone 4 has not started. ``` -Milestones 4–5 remain directional context and are not active. +Milestones 4–5 remain directional context. Do not begin Milestone 4 without a new active-slice +contract. diff --git a/DEVELOPMENT_NOTES.md b/DEVELOPMENT_NOTES.md index f00abff..2548012 100644 --- a/DEVELOPMENT_NOTES.md +++ b/DEVELOPMENT_NOTES.md @@ -345,7 +345,7 @@ These are notes, not commitments: than bounded positions, they will need a different versioned security contract and persisted key lifecycle. -## Milestone 2 — active: agent retrieval and MCP experience +## Milestone 2 — complete: agent retrieval and MCP experience ### Audit reconciliation @@ -611,10 +611,10 @@ task-shaped capsule for every continuation page, add authenticated continuation model requires it, verify a native Claude timeout representation, and introduce adapter-owned launcher metadata before generating configurations for custom adapters. -## Milestone 3 — active: independent projections +## Milestone 3 — complete: independent projections Milestone 3 began only after `main` and `dev` were aligned at the verified Milestone 2 closeout. -Three read-only audits are running before source changes: +Three read-only audits ran before source changes: - Manual planning, immutable packages, renderer isolation, receipts, preview/application integration, and full/incremental equivalence. @@ -692,8 +692,95 @@ The first slice now implements: - A live-viewer correction: source evidence now comes from the pinned index generation. The viewer no longer reopens mutable canonical files behind an older graph snapshot. -The new repository-native contract target passes 91 tests and 120 subtests. The combined -projection, rendering, and live-viewer focus passes with byte-exact compatibility and no hidden -source/path authority. This is not Milestone 3 closeout: durable multi-artifact publication, -portable graph rendering, detached workers, fragment reuse/equivalence, policy version 2, -accessibility, and maintained scale evidence remain active work. +The new repository-native contract target passed 91 tests and 120 subtests at the slice boundary. +The combined projection, rendering, and live-viewer focus passed with byte-exact compatibility and +no hidden source/path authority. + +### Durable portable graph publication + +The portable graph path now has its own declared `graph_render` views, pure plans, fixed +`portable_graph_html` renderer, content-addressed artifact store, renderer receipts, and one bounded +generation/view manifest as the publication commit. It supports Nodes, Flow, and Web without +including Logic. Static HTML contains the complete pre-rendered graph and treats JavaScript as +progressive enhancement. + +Publication revalidates source, view, artifact, receipt, and output identities across replacement. +Status reads only bounded manifest and receipt evidence. It never plans or renders. Repair may +restore a declared output from its content-addressed artifact. A post-artifact failure that cannot +be rolled back returns explicit degraded committed evidence rather than reporting an ordinary +failed mutation. + +### Detached workers and incremental fragments + +Manual and portable graph packages execute through one fixed one-request child protocol. The +parent launches isolated Python from a trusted working directory with a sanitized environment, +spools stdout to disk, reads one bounded canonical response, and validates the complete artifact +and receipt identity. The worker accepts only the two built-in renderer identities. Requests are +bounded by the 24,000,000-byte package contract, actual artifact transfer by 20,000,000 bytes, and +execution by a 30-second timeout. + +Manual fragment records are semantic, versioned, canonical, hash-bound, and stored below a +dedicated confined cache. The worker independently recomputes the expected page fragment before +using a record. Corrupt, forged, oversized, stale, or aggregate-oversized records fall back to the +full detached render. Cold fragment creation is compared byte-for-byte with that full oracle before +cache publication. The cache retains only the current inventory and is capped at 10,000 entries +and 64,000,000 bytes. + +### Independent policies and accessibility + +Projection policy version 2 independently composes manual `auto|explicit|disabled`, portable graph +`explicit|disabled`, and live viewer `on-demand|disabled`. CLI, MCP, generated client +configuration, doctor, render services, canonical application, onboarding, and viewer-manager +entry points enforce their relevant policy. Status remains available when an active operation is +disabled. + +Generated client evidence binds the projection policy, its hash, projection availability, and the +current descriptor hash into the configuration hash. Validation cross-checks omitted default +selectors against the bound descriptor so coordinated policy and availability drift fails closed. +The version-1 effective-policy payload remains unchanged for existing clients. + +Pinned Playwright 1.62.0 and axe-core 4.12.1 gates exercise the frozen manual, portable graph, and +live viewer with selected WCAG A/AA axe tags and keyboard interaction flows. Portable and live +graph presentation received only the minimal contrast and nested-role corrections needed by those +gates. + +### Scale and runtime hardening + +The first 1,000-node full benchmark exposed recursive strongly connected-component traversal in +manual planning. Cycle detection now uses an iterative two-pass traversal. A regression covers the +descriptor maximum of 10,000 nodes as both a deep acyclic chain and one strongly connected +component. + +The isolated wheel proof also exposed a Python `runpy` warning when the worker module was imported +during package initialization before `-m` execution. A private fixed module entrypoint now owns +child startup. Malformed child input returns code 2 with empty stdout and stderr. + +Configured render ceilings above 20,000,000 bytes remain accepted for compatibility, and small +actual artifacts render normally. The detached protocol still rejects an actual transfer beyond +its fixed 20,000,000-byte boundary. + +### Milestone 3 closeout + +Candidate `f5dccb5e1c312121f1af63780162f593d9363b98` passed the complete repository gate: formatting, +Python and web lint, strict types, compilation, 281 tests and 272 subtests, three accessibility +flows, lock and dependency checks, package builds, and all milestone smoke benchmarks. The +maintained projection contract subset passed 142 tests and 236 subtests. + +The clean ten-sample 1,000-node benchmark passed every latency, memory, response-size, no-work, and +equivalence gate. Manual full rendering measured 810.490 ms p95, portable graph full rendering +323.690 ms p95, and receipt-only status 111.381 ms and 59.331 ms p95 respectively. Direct detached +worker peaks were 88,580,096 and 89,583,616 bytes. The separately gated production manual worker +peak was 104,771,584 bytes. Production cold, warm, forced-full, add, change, delete, and reorder +outputs were byte-identical. + +Production warm fragment rendering measured 2,206.540 ms p95 versus 978.870 ms for forced full. +Milestone 3 therefore closes the fragment isolation, invalidation, equivalence, and recovery +contract without claiming a throughput win. Later optimization must begin from that evidence. + +The exact method and measurements are recorded in `docs/MILESTONE_3_BASELINE.md` and +`benchmarks/milestone3-2026-07-29.json`. The candidate passed an isolated wheel CLI/MCP/worker +proof. Gitleaks 8.30.1 found no findings across the six Milestone 3 commits or candidate tree. + +Milestone 3 is complete. No tag, release, production integration repointing, WorldForge change, +ScrapeStation change, storage rewrite, or self-hosting dependency was introduced. Milestone 4 +remains directional and has not started. diff --git a/README.md b/README.md index 16a03a6..20ba862 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,12 @@ declared manuals, visualizes project structure, and manages reviewable documenta - Detects project-local adapter implementation and configuration changes and requires a fresh project-bound process before any further MCP work. - Keeps function-scoped control-flow projections separate from the primary architecture graph. +- Compiles manuals and portable graph artifacts from separate versioned, generation-pinned plans + and immutable packages. +- Runs built-in manual and graph renderers in fixed detached workers with validated receipts, + bounded transfer, and no project-path authority. +- Enforces independent manual, portable-graph, and live-viewer policy while keeping status + receipt-only. - Runs a managed loopback graph browser with neighborhood, semantic Flow, convergence Web, function-scoped Logic, source inspection, and branch-aware node hiding. - Supports generic documentation projects and project-owned source adapters. @@ -73,6 +79,63 @@ details, and explicit truncation. Paged results use one top-level cursor and a v `receipt_header`; `stored_receipt_hash` identifies the complete persisted receipt. The read never exposes Logic details, loads canonical source, repairs derived state, or invents history. +## Independent projections + +Manual compilation, portable graph rendering, and the live viewer consume the same validated graph +generation through separate boundaries: + +```text +validated generation + ├── ManualRenderPlanV1 → immutable package → detached manual renderer + ├── GraphViewPlanV1 → immutable package → detached portable graph renderer + └── pinned index → managed read-only live viewer +``` + +Plans, packages, and receipts are canonical, versioned, hash-identified, bounded, and contain no +project object, SQLite handle, absolute project path, command, or caller-selected renderer module. +Renderers cannot select graph facts, crawl canonical sources, choose publication paths, or mutate +project state. + +Declare portable graph output separately from manual views: + +```toml +[graph_render] +output_root = ".docforge/portable-graph" + +[[graph_render.views]] +id = "architecture" +renderer = "portable_graph_html" +output = "architecture.html" +title = "Architecture" +root = "architecture.overview" +initial_mode = "web" +depth = 3 +max_nodes = 250 +max_edges = 1000 +max_work = 100000 +include_logic = false +``` + +Plan, publish, and inspect it explicitly: + +```bash +.venv/bin/docforge --project-root "$PROJECT" graph-plan architecture +.venv/bin/docforge --project-root "$PROJECT" graph-render architecture +.venv/bin/docforge --project-root "$PROJECT" graph-render-status architecture +``` + +The version-2 projection policy independently selects manual +`auto|explicit|disabled`, portable graph `explicit|disabled`, and live viewer +`on-demand|disabled`. Use `--manual-render-policy`, `--portable-graph-policy`, and +`--live-viewer-policy` on CLI/MCP startup or generated client configuration. Status remains +available when the corresponding active operation is disabled. + +Manual fragment reuse is disposable. Cold record publication is guarded by byte-exact comparison +with a full detached render; warm records are independently recomputed and validated inside the +worker. Full rendering remains the recovery and equivalence oracle. Portable artifacts commit +content-addressed output and renderer evidence before one bounded generation/view manifest; status +does not plan or render. + ## Graph views The browser presents the primary architecture graph through three complementary views and loads a @@ -170,10 +233,18 @@ DocForge describes them as a source graph. performance, memory, rendering and response sizes, bottlenecks, and missing coverage. - [Milestone 0 closeout](docs/MILESTONE_0_CLOSEOUT.md) — lineage, migration, security scan, repository state, and fresh-clone proof. +- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) — warm operation latency, structured work, + status, retrieval, and memory measurements. +- [Milestone 1 closeout](docs/MILESTONE_1_CLOSEOUT.md) — fast-core contracts, adversarial + validation, compatibility boundaries, and exact candidate evidence. - [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) — task context, generation diff, client configuration, doctor, response-size, counter, and memory measurements. - [Milestone 2 closeout](docs/MILESTONE_2_CLOSEOUT.md) — implemented contracts, adversarial validation, exclusions, and exact candidate evidence. +- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) — manual, fragment, worker, portable graph, + status, equivalence, response-size, and memory measurements. +- [Milestone 3 closeout](docs/MILESTONE_3_CLOSEOUT.md) — independent projection contracts, + adversarial validation, compatibility boundaries, and exact candidate evidence. - [MCP contract](docs/MCP_CONTRACT.md) — exact tool and process boundary. - [Viewer manager](docs/VIEWER_MANAGER.md) — native service setup and lifecycle. - [Adapter decision](docs/APPLICATION_DECISION.md) — why custom adapters own canonical @@ -197,6 +268,8 @@ make gate Focused entry points are available as `make contract`, `make test`, `make type`, `make benchmark-smoke`, `make benchmark`, `make benchmark-m1-smoke`, and `make benchmark-m1`. Milestone 2 adds `make benchmark-m2-smoke` and `make benchmark-m2`. +Milestone 3 adds `make accessibility`, `make benchmark-m3-smoke`, `make benchmark-m3`, and +`make benchmark-m3-full`. The committed 1,000-node baseline and its measurement method are under `benchmarks/`. diff --git a/SLICE_HISTORY.md b/SLICE_HISTORY.md index fa7bf77..5b4cd75 100644 --- a/SLICE_HISTORY.md +++ b/SLICE_HISTORY.md @@ -1,5 +1,51 @@ # Completed slices +## DocForge2 Milestone 3 independent projections + +### Changed + +- Added strict version-1 manual plan, graph plan, projection package, and projection receipt + contracts with canonical identities and packaged schemas. +- Kept the legacy manual API and exact alpha bytes as a compatibility wrapper over a pure planner, + immutable package, and independent renderer. +- Added generation-pinned portable graph planning, detached rendering, content-addressed durable + publication, repair, and receipt-only status. +- Added a fixed isolated worker protocol with bounded request, response, artifact, timeout, + environment, and renderer inventory. +- Added semantic manual fragments with independent worker validation, corruption recovery, + full-render equivalence, and bounded cache retention. +- Added version-2 independent manual, portable-graph, and live-viewer policy with descriptor-bound + generated-client evidence while retaining effective policy version 1. +- Corrected live source reads to use the pinned index generation. +- Added static and interactive accessibility gates for the manual, portable graph, and live viewer. +- Replaced recursive cycle planning with an iterative traversal proven at 10,000 nodes. + +### Verification + +- The complete gate passed 281 tests, 272 subtests, three accessibility flows, dependency checks, + builds, and all milestone smoke benchmarks. +- The maintained projection contract subset passed 142 tests and 236 subtests. +- The clean ten-sample 1,000-node benchmark passed every latency, memory, response-size, no-work, + deterministic-output, and full/incremental equivalence threshold. +- An isolated wheel passed CLI, MCP, detached manual rendering, and malformed worker startup. +- Three adversarial review tracks were reconciled, and Gitleaks found no findings in the six + milestone commits or candidate tree. + +### Limits + +- Production fragment reuse is correct and recoverable but is slower than forced full rendering at + the maintained 1,000-page fixture. No speedup is claimed. +- Actual detached artifact transfer is capped at 20,000,000 bytes. +- Portable graph publication is CLI-owned. MCP exposes read-only plan and status tools. +- Remote render services, render farms, third-party renderer ecosystems, storage replacement, + self-hosting, adapter SDK expansion, and production integration changes remain out of scope. +- No tag or release was created. + +### Next gate + +Milestone 4 remains directional and is not active. Create a new active-slice contract before +starting adapter SDK or product-documentation implementation. + ## DocForge2 Milestone 0 successor foundation ### Changed diff --git a/benchmarks/README.md b/benchmarks/README.md index 9d78eda..2141f1a 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -34,6 +34,14 @@ make benchmark-m2-smoke make benchmark-m2 ``` +Run the Milestone 3 independent-projection smoke and maintained full gates: + +```bash +make benchmark-m3-smoke +make benchmark-m3 +make benchmark-m3-full +``` + The benchmark creates canonical sources, derived state, changesets, rendered output, and caches only in a disposable temporary directory. It does not read another project, self-host DocForge, or mutate repository content. @@ -56,9 +64,16 @@ records whether diagnostics were dropped for response budget, checks all hidden- and measures isolated-process peak RSS. Its interpretation is in [`docs/MILESTONE_2_BASELINE.md`](../docs/MILESTONE_2_BASELINE.md). -The generic fixture exposes whole-source scaling. It does not replace the incremental adapter -equivalence tests and does not claim to measure a portable graph renderer, because Milestone 0 has -no portable graph-planning or graph-rendering contract. +`milestone3-2026-07-29.json` is the clean-tree independent-projection baseline captured from commit +`f5dccb5e1c312121f1af63780162f593d9363b98`. It measures versioned manual and graph planning, +in-process and detached rendering, production cold/warm/forced-full behavior, fragment cache +sweeps, add/change/delete/reorder equivalence, portable publication, receipt-only status, traced +memory, detached worker peak memory, and response size. Its interpretation is in +[`docs/MILESTONE_3_BASELINE.md`](../docs/MILESTONE_3_BASELINE.md). + +The generic fixtures expose whole-source and projection scaling. They do not replace incremental +adapter equivalence tests. Milestone 3's portable fixture contains 1,000 Nodes/Flow/Web nodes and +999 edges; portable version 1 deliberately excludes Logic. The Milestone 1 harness treats wall time and structured work counters as separate gates. Warm operations fail if they load a complete project, parse source files, reconstruct an adapter @@ -73,4 +88,5 @@ status. The 1,000-node run records bounded semantic summaries for exact errors, backlinks, both traversal directions, paged context, render receipt states, and visualization freshness. The reported p95 uses the nearest-rank method; with ten samples it is the maximum. `process_peak_rss_kib` is the cumulative main-process `RUSAGE_SELF` high-water mark and excludes the -detached viewer worker. +detached viewer worker. It is diagnostic and not operation-local. Milestone 3 memory gates use +per-operation `tracemalloc` peaks and detached worker receipt peaks instead. diff --git a/benchmarks/milestone3-2026-07-29.json b/benchmarks/milestone3-2026-07-29.json new file mode 100644 index 0000000..3386a8d --- /dev/null +++ b/benchmarks/milestone3-2026-07-29.json @@ -0,0 +1,442 @@ +{ + "benchmark": "docforge2_milestone3", + "environment": { + "implementation": "CPython", + "machine": "x86_64", + "platform": "Linux-7.1.3-200.nobara.fc44.x86_64-x86_64-with-glibc2.43", + "python": "3.14.6" + }, + "equivalence": { + "manual_full_vs_fragment_assisted": true, + "manual_in_process_vs_detached": true, + "manual_production_cold_warm_full": true, + "manual_production_variants": { + "add": true, + "change": true, + "delete": true, + "reorder": true + }, + "portable_graph_in_process_vs_detached": true + }, + "fixture": { + "edge_count": 999, + "full_coverage": true, + "kind": "synthetic_generic_projection", + "manual_page_count": 1000, + "node_count": 1000, + "portable_graph_edge_count": 999, + "portable_graph_node_count": 1000 + }, + "memory": { + "manual_production_worker_peak_bytes": 104771584, + "manual_worker_peak_bytes": 88580096, + "portable_graph_worker_peak_bytes": 89583616, + "process_peak_rss_kib": 102692 + }, + "method": { + "clock": "time.perf_counter_ns", + "detached_peak_memory": "worker receipt resource peak RSS", + "determinism": "stable semantic summaries must match across samples; report JSON uses sorted keys", + "full_mode_node_requirement": 1000, + "in_process_peak_memory": "tracemalloc per measured invocation", + "maximum_worker_artifact_bytes": 20000000, + "process_peak_memory": "resource.getrusage(RUSAGE_SELF).ru_maxrss", + "response_size": "UTF-8 bytes of canonical compact sorted JSON", + "samples": 10 + }, + "mode": "full", + "operations": { + "fragment_assisted_equivalence": { + "max_ms": 666.119, + "maximum_response_bytes": 664, + "maximum_traced_peak_bytes": 5332792, + "median_ms": 638.999, + "min_ms": 633.157, + "p95_limit_ms": 15000, + "p95_ms": 666.119, + "response_limit_bytes": 128000, + "samples": 10, + "stable_result": { + "artifact_bytes": 583149, + "artifact_sha256": "278ff8cd6b092e7d8114527dba88051917aa827b22d4aed61e57acf004e5870c", + "package_bytes": 2364302, + "package_id": "ecc21e58106c420a9f78ffbba997778ca98b61586ecfdb23a15b08f431a52c42", + "plan_bytes": 1006393, + "plan_id": "47008f5177230e3ad5abbfee8bccaff0498963106cb33ed7cdd6f8bbc3c495de" + }, + "traced_peak_limit_bytes": 268435456 + }, + "fragment_cache_hit_sweep": { + "max_ms": 514.324, + "maximum_response_bytes": 145, + "maximum_traced_peak_bytes": 6941174, + "median_ms": 498.911, + "min_ms": 495.447, + "p95_limit_ms": 5000, + "p95_ms": 514.324, + "response_limit_bytes": 32768, + "samples": 10, + "stable_result": { + "aggregate_content_bytes": 516921, + "fragment_count": 1000, + "ordered_record_hash": "99a95d575504238233a6367c884fa1d50231855f2f2ffe3dae90fa18c4136f72" + }, + "traced_peak_limit_bytes": 268435456 + }, + "fragment_cache_miss_sweep": { + "max_ms": 106.888, + "maximum_response_bytes": 145, + "maximum_traced_peak_bytes": 146245, + "median_ms": 106.203, + "min_ms": 105.383, + "p95_limit_ms": 5000, + "p95_ms": 106.888, + "response_limit_bytes": 32768, + "samples": 10, + "stable_result": { + "aggregate_content_bytes": 516921, + "fragment_count": 1000, + "ordered_record_hash": "99a95d575504238233a6367c884fa1d50231855f2f2ffe3dae90fa18c4136f72" + }, + "traced_peak_limit_bytes": 268435456 + }, + "fragment_cache_put_sweep": { + "max_ms": 539.212, + "maximum_response_bytes": 145, + "maximum_traced_peak_bytes": 509548, + "median_ms": 539.212, + "min_ms": 539.212, + "p95_limit_ms": 10000, + "p95_ms": 539.212, + "response_limit_bytes": 32768, + "samples": 1, + "stable_result": { + "aggregate_content_bytes": 516921, + "fragment_count": 1000, + "ordered_record_hash": "99a95d575504238233a6367c884fa1d50231855f2f2ffe3dae90fa18c4136f72" + }, + "traced_peak_limit_bytes": 268435456 + }, + "manual_detached_worker": { + "child_peak_limit_bytes": 268435456, + "max_ms": 599.394, + "maximum_child_peak_bytes": 88580096, + "maximum_response_bytes": 667, + "maximum_traced_peak_bytes": 28203708, + "median_ms": 591.317, + "min_ms": 585.123, + "p95_limit_ms": 20000, + "p95_ms": 599.394, + "response_limit_bytes": 128000, + "samples": 10, + "stable_result": { + "artifact_bytes": 583149, + "artifact_sha256": "278ff8cd6b092e7d8114527dba88051917aa827b22d4aed61e57acf004e5870c", + "package_bytes": 1007297, + "package_id": "5fec20ace40b6db7e89083704af371245a5b5b73247f71edb6eab0cb2bd5e2c1", + "plan_bytes": 1006393, + "plan_id": "47008f5177230e3ad5abbfee8bccaff0498963106cb33ed7cdd6f8bbc3c495de" + }, + "traced_peak_limit_bytes": 268435456 + }, + "manual_forced_full": { + "child_peak_limit_bytes": 268435456, + "max_ms": 978.87, + "maximum_child_peak_bytes": 104771584, + "maximum_response_bytes": 668, + "maximum_traced_peak_bytes": 30425714, + "median_ms": 944.135, + "min_ms": 934.773, + "p95_limit_ms": 20000, + "p95_ms": 978.87, + "response_limit_bytes": 128000, + "samples": 10, + "stable_result": { + "output_bytes": 583149, + "output_sha256": "278ff8cd6b092e7d8114527dba88051917aa827b22d4aed61e57acf004e5870c", + "render_identity": "73f9a175afd7bfea5af70d4c7df902c982f042c413d533817dc2bade9e54d4b0" + }, + "traced_peak_limit_bytes": 268435456 + }, + "manual_full_render": { + "max_ms": 810.49, + "maximum_response_bytes": 664, + "maximum_traced_peak_bytes": 4412754, + "median_ms": 801.948, + "min_ms": 773.7, + "p95_limit_ms": 15000, + "p95_ms": 810.49, + "response_limit_bytes": 128000, + "samples": 10, + "stable_result": { + "artifact_bytes": 583149, + "artifact_sha256": "278ff8cd6b092e7d8114527dba88051917aa827b22d4aed61e57acf004e5870c", + "package_bytes": 1007297, + "package_id": "5fec20ace40b6db7e89083704af371245a5b5b73247f71edb6eab0cb2bd5e2c1", + "plan_bytes": 1006393, + "plan_id": "47008f5177230e3ad5abbfee8bccaff0498963106cb33ed7cdd6f8bbc3c495de" + }, + "traced_peak_limit_bytes": 268435456 + }, + "manual_incremental_cold": { + "child_peak_limit_bytes": 268435456, + "max_ms": 2827.152, + "maximum_child_peak_bytes": 99454976, + "maximum_response_bytes": 668, + "maximum_traced_peak_bytes": 35160716, + "median_ms": 2827.152, + "min_ms": 2827.152, + "p95_limit_ms": 20000, + "p95_ms": 2827.152, + "response_limit_bytes": 128000, + "samples": 1, + "stable_result": { + "output_bytes": 583149, + "output_sha256": "278ff8cd6b092e7d8114527dba88051917aa827b22d4aed61e57acf004e5870c", + "render_identity": "73f9a175afd7bfea5af70d4c7df902c982f042c413d533817dc2bade9e54d4b0" + }, + "traced_peak_limit_bytes": 268435456 + }, + "manual_incremental_warm": { + "child_peak_limit_bytes": 268435456, + "max_ms": 2206.54, + "maximum_child_peak_bytes": 104767488, + "maximum_response_bytes": 669, + "maximum_traced_peak_bytes": 34676043, + "median_ms": 2143.388, + "min_ms": 2125.466, + "p95_limit_ms": 20000, + "p95_ms": 2206.54, + "response_limit_bytes": 128000, + "samples": 10, + "stable_result": { + "output_bytes": 583149, + "output_sha256": "278ff8cd6b092e7d8114527dba88051917aa827b22d4aed61e57acf004e5870c", + "render_identity": "73f9a175afd7bfea5af70d4c7df902c982f042c413d533817dc2bade9e54d4b0" + }, + "traced_peak_limit_bytes": 268435456 + }, + "manual_status_no_work": { + "max_ms": 111.381, + "maximum_response_bytes": 1542, + "maximum_traced_peak_bytes": 1792025, + "median_ms": 62.881, + "min_ms": 60.111, + "p95_limit_ms": 500, + "p95_ms": 111.381, + "response_limit_bytes": 256000, + "samples": 10, + "stable_result": { + "counters": { + "adapter_projection_loads": 0, + "adapter_source_extractions": 0, + "index_builds": 0, + "index_checks": 0, + "index_synchronizations": 0, + "project_loads": 0, + "render_output_bytes_built": 0, + "render_output_bytes_hashed": 0, + "render_prepare_calls": 0, + "source_bytes_parsed": 0, + "source_files_parsed": 0, + "source_generation_checks": 2, + "viewer_manager_requests": 0 + }, + "response": { + "adapter": "generic", + "configured": true, + "outputs": [ + { + "actual_output_hash": "278ff8cd6b092e7d8114527dba88051917aa827b22d4aed61e57acf004e5870c", + "expected_output_hash": "278ff8cd6b092e7d8114527dba88051917aa827b22d4aed61e57acf004e5870c", + "path": ".docforge/rendered/manual.html", + "projection_receipt": { + "artifacts": [ + { + "artifact_id": "manual.html", + "bytes": 583149, + "media_type": "text/html; charset=utf-8", + "sha256": "278ff8cd6b092e7d8114527dba88051917aa827b22d4aed61e57acf004e5870c" + } + ], + "contract": "docforge.projection-receipt", + "diagnostics": { + "warnings": [] + }, + "kind": "manual", + "package_id": "b30c65390570537a16ce03afe6a992599c6ca0c4ef5ea2f06d14b33e248bc9aa", + "peak_memory_bytes": 104771584, + "plan_id": "47008f5177230e3ad5abbfee8bccaff0498963106cb33ed7cdd6f8bbc3c495de", + "receipt_id": "ead4c25b34ada8df674903a592e25dd6bbf9817df2b04e5ebc66cde9e41c1ed1", + "renderer": { + "renderer_id": "generic_html", + "renderer_version": "1+markdown-it-py-4.2.0" + }, + "schema_version": 1, + "timing": { + "elapsed_ns": 114544865 + } + }, + "reason": null, + "receipt_schema_version": 1, + "render_identity": "73f9a175afd7bfea5af70d4c7df902c982f042c413d533817dc2bade9e54d4b0", + "renderer": "generic_html", + "renderer_version": "1+markdown-it-py-4.2.0", + "state": "current", + "template_hash": "ae0ebb3eadeb530e9d033c8fbd21321d406e29d04c0ae7d07a9a0f929f0ffa9f", + "verification": "receipt", + "view_id": "manual" + } + ], + "project_id": "synthetic-1000", + "project_root_fingerprint": "0747acfd975703c0", + "revision": "unversioned", + "source_hash": "a314da4ffa8fcf291ef7a7b0fc87737caeaa89fa3c558ea442afeb0e5ae49c2d", + "state": "current", + "status": "ok", + "verification": "receipt" + } + }, + "traced_peak_limit_bytes": 268435456 + }, + "portable_graph_detached_worker": { + "child_peak_limit_bytes": 268435456, + "max_ms": 268.428, + "maximum_child_peak_bytes": 89583616, + "maximum_response_bytes": 660, + "maximum_traced_peak_bytes": 27587227, + "median_ms": 265.448, + "min_ms": 263.761, + "p95_limit_ms": 20000, + "p95_ms": 268.428, + "response_limit_bytes": 128000, + "samples": 10, + "stable_result": { + "artifact_bytes": 718383, + "artifact_sha256": "91dd5bb31679225e5af40a1f94a9e75f72921e36416e3d8b00c45f46a37a3a30", + "package_bytes": 398715, + "package_id": "72f96d7ab92c1d6b59f4f32539558232ecf35deda099fc4887c77890da1d4309", + "plan_bytes": 398158, + "plan_id": "6dac35937d855f72c6670f87aace3c0cb1dbff67bb842c55370429584064f76b" + }, + "traced_peak_limit_bytes": 268435456 + }, + "portable_graph_full_render": { + "max_ms": 323.69, + "maximum_response_bytes": 657, + "maximum_traced_peak_bytes": 3490976, + "median_ms": 303.736, + "min_ms": 301.215, + "p95_limit_ms": 10000, + "p95_ms": 323.69, + "response_limit_bytes": 128000, + "samples": 10, + "stable_result": { + "artifact_bytes": 718383, + "artifact_sha256": "91dd5bb31679225e5af40a1f94a9e75f72921e36416e3d8b00c45f46a37a3a30", + "package_bytes": 398715, + "package_id": "72f96d7ab92c1d6b59f4f32539558232ecf35deda099fc4887c77890da1d4309", + "plan_bytes": 398158, + "plan_id": "6dac35937d855f72c6670f87aace3c0cb1dbff67bb842c55370429584064f76b" + }, + "traced_peak_limit_bytes": 268435456 + }, + "portable_graph_status_no_work": { + "max_ms": 59.331, + "maximum_response_bytes": 879, + "maximum_traced_peak_bytes": 685012, + "median_ms": 58.467, + "min_ms": 58.052, + "p95_limit_ms": 500, + "p95_ms": 59.331, + "response_limit_bytes": 256000, + "samples": 10, + "stable_result": { + "counters": { + "adapter_projection_loads": 0, + "adapter_source_extractions": 0, + "index_builds": 0, + "index_checks": 0, + "index_synchronizations": 0, + "project_loads": 0, + "render_output_bytes_built": 0, + "render_output_bytes_hashed": 0, + "render_prepare_calls": 0, + "source_bytes_parsed": 0, + "source_files_parsed": 0, + "source_generation_checks": 2, + "viewer_manager_requests": 0 + }, + "response": { + "adapter": "generic", + "configured": true, + "outputs": [ + { + "artifact": { + "artifact_id": "portable-graph.html", + "bytes": 718383, + "media_type": "text/html; charset=utf-8", + "sha256": "91dd5bb31679225e5af40a1f94a9e75f72921e36416e3d8b00c45f46a37a3a30" + }, + "package_id": "72f96d7ab92c1d6b59f4f32539558232ecf35deda099fc4887c77890da1d4309", + "path": ".docforge/portable-graph/architecture.html", + "plan_id": "6dac35937d855f72c6670f87aace3c0cb1dbff67bb842c55370429584064f76b", + "publication_id": "2620ce106593d5499931b145a42fc1eccd31f5e356b19bf1d76687f8d45bcad2", + "reason": null, + "renderer": "portable_graph_html", + "renderer_version": "1", + "state": "current", + "verification": "manifest", + "view_id": "architecture" + } + ], + "project_id": "synthetic-1000", + "project_root_fingerprint": "0747acfd975703c0", + "revision": "unversioned", + "source_hash": "a314da4ffa8fcf291ef7a7b0fc87737caeaa89fa3c558ea442afeb0e5ae49c2d", + "state": "current", + "status": "ok" + } + }, + "traced_peak_limit_bytes": 268435456 + } + }, + "schema_version": 1, + "sizes": { + "fragment_assisted_manual": { + "aggregate_fragment_content_bytes": 516921, + "artifact_bytes": 583149, + "artifact_sha256": "278ff8cd6b092e7d8114527dba88051917aa827b22d4aed61e57acf004e5870c", + "fragment_count": 1000, + "package_bytes": 2364302, + "package_id": "ecc21e58106c420a9f78ffbba997778ca98b61586ecfdb23a15b08f431a52c42", + "plan_bytes": 1006393, + "plan_id": "47008f5177230e3ad5abbfee8bccaff0498963106cb33ed7cdd6f8bbc3c495de", + "receipt_bytes": 664 + }, + "manual": { + "artifact_bytes": 583149, + "artifact_sha256": "278ff8cd6b092e7d8114527dba88051917aa827b22d4aed61e57acf004e5870c", + "package_bytes": 1007297, + "package_id": "5fec20ace40b6db7e89083704af371245a5b5b73247f71edb6eab0cb2bd5e2c1", + "plan_bytes": 1006393, + "plan_id": "47008f5177230e3ad5abbfee8bccaff0498963106cb33ed7cdd6f8bbc3c495de", + "receipt_bytes": 664 + }, + "manual_status_response_bytes": 1542, + "portable_graph": { + "artifact_bytes": 718383, + "artifact_sha256": "91dd5bb31679225e5af40a1f94a9e75f72921e36416e3d8b00c45f46a37a3a30", + "package_bytes": 398715, + "package_id": "72f96d7ab92c1d6b59f4f32539558232ecf35deda099fc4887c77890da1d4309", + "plan_bytes": 398158, + "plan_id": "6dac35937d855f72c6670f87aace3c0cb1dbff67bb842c55370429584064f76b", + "receipt_bytes": 657 + }, + "portable_graph_status_response_bytes": 879 + }, + "source": { + "dirty": false, + "revision": "f5dccb5e1c312121f1af63780162f593d9363b98" + } +} diff --git a/docs/COMPATIBILITY.md b/docs/COMPATIBILITY.md index 12c6247..3925f59 100644 --- a/docs/COMPATIBILITY.md +++ b/docs/COMPATIBILITY.md @@ -71,6 +71,19 @@ Existing hand-written client configurations remain valid and are never rewritten Doctor is inspection-only and does not become a hidden bootstrap, synchronization, or migration path. +The following Milestone 3 CLI additions are also additive: + +- `docforge graph-plan VIEW_ID` +- `docforge graph-render VIEW_ID` +- `docforge graph-render-status [VIEW_ID]` +- `--manual-render-policy auto|explicit|disabled` +- `--portable-graph-policy explicit|disabled` +- `--live-viewer-policy on-demand|disabled` + +MCP adds the read-only `docforge_graph_plan` and `docforge_graph_render_status` tools. Portable +graph publication remains an explicit local CLI integration action. Existing manual render, +preview, visualization, and status names remain supported. + ## Versioned data contracts Milestone 0 preserves: @@ -96,6 +109,11 @@ Milestone 0 preserves: - Latest-generation-diff page schema version 1. Pages use one top-level pagination object and a nested `receipt_header`. `stored_receipt_hash` names the complete stored receipt. Opaque cursors may be restarted after a server or receipt change and are not durable public identifiers. +- Manual render-plan schema version 1. +- Graph view-plan schema version 1. +- Projection-package schema version 1. +- Projection-receipt schema version 1. +- Independent projection-policy schema version 2. Effective policy version 1 remains frozen. Indexes, attestations, extraction caches, previews, and rendered artifacts are disposable. A schema change may rebuild them. Canonical project content and stored proposals may not be silently @@ -161,12 +179,24 @@ The following guarantees remain stable: ## Rendering and visualization -The `generic_html` renderer remains the supported version-1 manual projection. It retains confined -paths, raw-HTML suppression, fixed template tokens, deterministic identities, atomic replacement, -and side-effect-free status. +The `generic_html` renderer remains the supported version-1 manual projection. Its public +`GenericHtmlRenderer.prepare()` signature, renderer identity, frozen alpha bytes, confined paths, +raw-HTML suppression, fixed template tokens, deterministic identities, atomic replacement, and +side-effect-free status remain compatible. It now delegates through a versioned manual plan, +immutable package, and independent renderer. + +The `portable_graph_html` renderer and `graph_render` descriptor table are additive. Manual and +portable graph declarations, plans, policies, publication receipts, and status remain separate. +The portable renderer does not replace the existing live viewer or `docforge_visualize`. + +Existing project descriptors may retain any positive `max_render_bytes` accepted by schema version +1. A value above 20,000,000 bytes does not make the descriptor invalid, and a smaller actual +artifact still renders. Actual detached worker transfer is a separate fixed 20,000,000-byte +runtime boundary. The live graph viewer remains a read-only consumer of a generation-pinned validated index. It does -not become project authority or MCP retrieval authority. +not become project authority or MCP retrieval authority. Source reads use the pinned index +generation instead of reopening mutable canonical files behind that generation. ## Task-context compatibility @@ -199,8 +229,9 @@ The exact version-1 relation aliases are frozen by the MCP contract and reposito Changing an alias category requires a new planner version; it is not a silent implementation detail. -`ManualRenderPlan`, `GraphViewPlan`, a portable graph renderer, and independently packaged -renderers are later-milestone direction. Milestone 0 does not claim that those contracts exist. +Milestone 3 adds `ManualRenderPlanV1`, `GraphViewPlanV1`, projection package and receipt version 1, +and projection policy version 2. These are additive submodule and schema contracts. They do not +change the legacy task-context, adapter, changeset, or effective-policy contracts described above. ## Safety boundary @@ -221,6 +252,8 @@ Milestone 0 records rather than redesigns these areas: callers use targeted retrieval for that node. - One individually oversized changeset diff is transported as reconstructable canonical-JSON chunks. Cursors are corruption-detecting read tokens, not authenticated authorization tokens. -- Manual planning is not separated from rendering. -- There is no portable graph-planning or graph-rendering contract. +- Production fragment validation is currently slower than forced-full rendering at the maintained + 1,000-page fixture. Full rendering remains the equivalence and recovery oracle. +- Remote render services, render farms, third-party renderer ecosystems, and a separate render MCP + remain deferred. - DocForge2 does not self-host its bootstrap documentation. diff --git a/docs/CONTRACT.md b/docs/CONTRACT.md index ec2b062..cfbaf30 100644 --- a/docs/CONTRACT.md +++ b/docs/CONTRACT.md @@ -24,6 +24,11 @@ commit when Git is available; it cannot change repository state. - Latest generation-diff page: `schemas/generation-diff-page.schema.json`, version 1. - Generated client configuration: `schemas/client-configuration.schema.json`, version 1. - Client doctor result: `schemas/doctor-result.schema.json`, version 1. +- Manual render plan: `schemas/manual-render-plan.schema.json`, version 1. +- Graph view plan: `schemas/graph-view-plan.schema.json`, version 1. +- Projection package: `schemas/projection-package.schema.json`, version 1. +- Projection receipt: `schemas/projection-receipt.schema.json`, version 1. +- Independent projection policy: `schemas/projection-policy.schema.json`, version 2. - Index schema: version 3, disposable and reproducible. - Index attestation: schema version 1, disposable and reproducible. - Core, CLI, and MCP server: version 1.3.0.dev0. @@ -82,7 +87,9 @@ the header alone. One top-level pagination object carries the only continuation Generated Codex, Claude, and OpenClaw fragments are machine-local projections. They are not canonical project content. Version 1 binds the selected project, exact isolated Python interpreter, canonical argument layout, effective policy, no-AST projection, render policy, -timeouts, artifact bytes, and configuration hash. +timeouts, artifact bytes, and configuration hash. Milestone 3 adds the version-2 projection policy, +its hash, projection availability, and the exact descriptor hash to that attested configuration +evidence. Omitted default selectors are recomposed against the bound descriptor. Preview is side-effect free. Explicit publication creates only one new private standalone fragment in an existing real directory. It never merges or replaces different content. Descriptor, @@ -135,10 +142,12 @@ preview root, and one or more stable view IDs. Each view names a built-in render derived output file, title, and optional family filter. Paths are resolved under the project root and may not overlap canonical content, authority files, changesets, templates, or previews. -The initial `generic_html` renderer uses pinned CommonMark parsing with raw HTML disabled. Templates -are UTF-8 files with a fixed token vocabulary; they cannot name commands, modules, or executable +The `generic_html` renderer uses pinned CommonMark parsing with raw HTML disabled. Templates are +UTF-8 files with a fixed token vocabulary; they cannot name commands, modules, or executable renderers. Render identity covers the canonical source hash, optional changeset hash, selected node -and edge identities, view configuration, template hash, renderer contract, and exact parser version. +and edge identities, view configuration, template hash, renderer contract, and exact parser +version. The frozen version-1 API and alpha bytes are preserved by a compatibility wrapper over the +manual plan/package/renderer path. An explicit CLI render atomically replaces one declared derived output. MCP can render a validated changeset only to its isolated preview path. Normal status verifies bounded source, configuration, @@ -146,6 +155,56 @@ template, output, renderer, and publication-receipt identities without reconstru Explicit deep status remains the side-effect-free full-render oracle. Input changes detected before atomic replacement fail without publishing a current receipt for stale output. +## Independent projection boundary + +Manual and portable graph plans are separate version-1 contracts over one immutable validated +generation. They use canonical JSON, deterministic ordering, fixed structural and serialized-size +bounds, and content-derived identities. Plans contain selected graph facts and bounded content. +They contain no live project object, database handle, absolute project or index path, arbitrary +query, command, executable path, or caller-selected module. + +Projection packages bind one plan to inert assets, a closed built-in renderer identity, declared +component versions, and an artifact inventory with a byte allowance. Receipts bind the exact +package, plan, renderer, artifact hashes and sizes, diagnostics, timing, and detached peak memory. +Manual and graph renderer modules accept only validated packages. They cannot select nodes, invent +relationships, read project state, choose publication paths, or write canonical files. + +Detached execution uses one fixed private Python module, isolated mode, a trusted working +directory, a sanitized environment, exactly one canonical newline-terminated JSON request and +response, a closed renderer allowlist, a 30-second timeout, disk-spooled stdout, and bounded reads. The package +contract is capped at 24,000,000 bytes and actual detached artifact transfer at 20,000,000 bytes. +Project descriptors may retain a larger `max_render_bytes` compatibility allowance, but an actual +detached transfer above the fixed worker boundary fails closed. + +Portable graph configuration is independent of manual render configuration. One view selects +either an exact root or a bounded metadata-only lexical query plus closed filters and node, edge, +depth, and work limits. Logic is excluded. The renderer emits a complete static Nodes, Flow, or Web +artifact and uses JavaScript only as progressive enhancement. + +Portable publication commits a content-addressed artifact, renderer receipt, and one bounded +generation/view manifest in that order. The manifest is the publication commit. Status reads only +bounded manifest and receipt evidence and never plans or renders. Repair restores declared output +only from validated content-addressed evidence. A failure after a replacement that cannot be +proven rolled back returns explicit degraded committed evidence. + +Manual fragment records are disposable semantic cache entries. Their keys bind the renderer, +component version, and complete page semantics. The detached renderer recomputes the expected page +fragment before accepting cached bytes. Cold creation is compared with a full detached render +before cache publication. Invalid, corrupt, forged, stale, individually oversized, or +aggregate-oversized records fall back to the full oracle. The dedicated cache retains only current +keys and is capped at 10,000 entries and 64,000,000 bytes. + +Projection policy version 2 composes manual `auto|explicit|disabled`, portable graph +`explicit|disabled`, and live viewer `on-demand|disabled` independently. Active plan, render, +application, onboarding, and viewer-start operations enforce the relevant policy before hidden +work. Receipt-only status and explicit viewer stop remain available. Effective policy version 1 +and its legacy projections remain unchanged. + +The live viewer remains separate from portable graph publication. It consumes one +generation-pinned validated index through the viewer manager. Source reads come from that pinned +generation and do not reopen mutable canonical files behind an older snapshot. Neither live nor +portable visualization is retrieval or canonical authority. + Normal MCP access does not expose canonical application. An explicitly configured canonical applier registers one hash-bound application tool. No MCP mode exposes arbitrary renderer execution, arbitrary file writes, shell commands, Git mutation, build commands, deployment, or diff --git a/docs/MILESTONE_3_BASELINE.md b/docs/MILESTONE_3_BASELINE.md new file mode 100644 index 0000000..6c95702 --- /dev/null +++ b/docs/MILESTONE_3_BASELINE.md @@ -0,0 +1,90 @@ +# Milestone 3 baseline + +## Scope and method + +This baseline records the independent-projection behavior completed in Milestone 3. It was +captured on 2026-07-29 from clean candidate commit +`f5dccb5e1c312121f1af63780162f593d9363b98`. + +The maintained command was: + +```bash +.venv/bin/python tools/milestone3_benchmark.py \ + --mode full \ + --nodes 1000 \ + --samples 10 \ + --output benchmarks/milestone3-2026-07-29.json +``` + +The synthetic generic fixture contains 1,000 manual pages, 1,000 portable-graph nodes, and 999 +edges. Durations use `time.perf_counter_ns()` and nearest-rank p95. In-process peak memory uses +`tracemalloc`; detached worker peak memory comes from the worker receipt and `RUSAGE_SELF`. +Every measured result is checked for deterministic semantic identity and bounded response size. + +Environment: + +- Linux 7.1.3-200.nobara.fc44.x86_64. +- CPython 3.14.6. +- x86_64. +- Ten samples except the one-time production cold render and fragment-cache population. +- In-process and detached-worker memory ceiling: 268,435,456 bytes. +- Detached artifact-transfer ceiling: 20,000,000 bytes. + +The complete machine-readable result is +[`benchmarks/milestone3-2026-07-29.json`](../benchmarks/milestone3-2026-07-29.json). + +## Results + +| Operation | Median | p95 | Limit | Maximum response | +|---|---:|---:|---:|---:| +| Manual full plan/package/render | 801.948 ms | 810.490 ms | 15,000 ms | 664 B | +| Manual detached worker | 591.317 ms | 599.394 ms | 20,000 ms | 667 B | +| Fragment-assisted equivalence | 638.999 ms | 666.119 ms | 15,000 ms | 664 B | +| Production incremental cold | 2,827.152 ms | 2,827.152 ms | 20,000 ms | 668 B | +| Production incremental warm | 2,143.388 ms | 2,206.540 ms | 20,000 ms | 669 B | +| Production forced full | 944.135 ms | 978.870 ms | 20,000 ms | 668 B | +| Fragment cache miss sweep | 106.203 ms | 106.888 ms | 5,000 ms | 145 B | +| Fragment cache hit sweep | 498.911 ms | 514.324 ms | 5,000 ms | 145 B | +| Portable graph full plan/package/render | 303.736 ms | 323.690 ms | 10,000 ms | 657 B | +| Portable graph detached worker | 265.448 ms | 268.428 ms | 20,000 ms | 660 B | +| Manual receipt-only status | 62.881 ms | 111.381 ms | 500 ms | 1,542 B | +| Portable graph receipt-only status | 58.467 ms | 59.331 ms | 500 ms | 879 B | + +The largest traced in-process peak was 35,160,716 bytes. The direct manual worker track peaked at +88,580,096 bytes and the portable graph worker at 89,583,616 bytes. The production manual paths, +including cold, warm, forced-full, and mutation variants, peaked at 104,771,584 bytes. Every child +peak was validated from its projection receipt against the 268,435,456-byte gate. + +The manual artifact was 583,149 bytes. The portable graph artifact was 718,383 bytes. The manual +plan was 1,006,393 bytes and its ordinary package was 1,007,297 bytes. The graph plan was 398,158 +bytes and its package was 398,715 bytes. + +## Equivalence and no-work gates + +The benchmark proved exact output equivalence for: + +- Manual in-process and detached rendering. +- Manual full and fragment-assisted rendering. +- Production cold, warm, and forced-full rendering. +- Production add, change, delete, and reorder variants. +- Portable graph in-process and detached rendering. + +Manual and portable-graph status each performed zero project loads, source parses, adapter +projection loads, adapter extraction, index checks, synchronization, index builds, render +preparation, output construction, output hashing, and viewer-manager requests. Each status path +performed only two cheap source-generation checks and verified committed receipt or manifest +evidence. + +## Measured limits and future notes + +- Fragment reuse is a correctness, isolation, and recovery boundary in this milestone. At 1,000 + pages, production warm fragment validation is slower than the forced-full path. Later + optimization must start from this measurement and preserve byte equivalence. +- Full rendering remains the oracle and recovery path. Invalid, corrupt, oversized, stale, or + mismatched fragment records fall back without changing canonical facts. +- The benchmark main-process `ru_maxrss` value was 102,692 KiB. It is cumulative across all + main-process operations and is recorded only as diagnostic context. Detached child peaks are + measured separately. Per-operation traced peaks and every detached receipt peak own the memory + gates. +- The results do not justify a storage rewrite, render farm, remote renderer, or separate render + MCP. diff --git a/docs/MILESTONE_3_CLOSEOUT.md b/docs/MILESTONE_3_CLOSEOUT.md new file mode 100644 index 0000000..c46c193 --- /dev/null +++ b/docs/MILESTONE_3_CLOSEOUT.md @@ -0,0 +1,89 @@ +# Milestone 3 closeout + +## Outcome + +Milestone 3 is complete. Manual compilation, portable graph rendering, and the live viewer are +separate generation-pinned consumers of the validated graph. They cannot become canonical or +retrieval authority. + +Implemented contracts: + +- Version-1 `ManualRenderPlan`, `GraphViewPlan`, projection package, and projection receipt. +- Strict canonical JSON identities and packaged Draft 2020-12 schemas. +- Independent manual and portable-graph renderer import boundaries. +- One isolated, fixed, one-request detached worker protocol with bounded request, response, + artifact, timeout, environment, and renderer inventory. +- Content-addressed portable graph artifacts, renderer receipts, generation/view manifests, + receipt-only status, repair, and degraded committed-publication evidence. +- Disposable semantic fragment records with bounded cache inventory, corruption recovery, and + full-render equivalence. +- Version-2 independent projection policy while preserving version-1 effective-policy behavior. +- Generation-pinned live source reads and a separate read-only viewer-manager lifecycle. +- Automated axe-tag and keyboard gates for the manual, portable graph, and live viewer. +- Repository-native contract, smoke, scale, response-size, memory, and equivalence gates. + +## Candidate evidence + +The frozen implementation candidate is +`f5dccb5e1c312121f1af63780162f593d9363b98`. + +The complete repository gate passed: + +- Ruff formatting and lint. +- HTML, rendered-manual HTML, portable-graph HTML, CSS, and JavaScript checks. +- Pyright with zero diagnostics. +- Warning-strict compilation and tests. +- 281 tests and 272 subtests. +- Three Playwright and axe accessibility flows. The alpha manual is checked with WCAG 2.0/2.1 + A/AA axe tags; portable and live graph flows add WCAG 2.2 A/AA tags and keyboard interaction. +- Lock and npm dependency-tree checks. +- Wheel and source-distribution builds. +- Milestone 0, 1, 2, and 3 smoke benchmarks. + +The maintained projection contract subset passed 142 tests and 236 subtests. A 10,000-node deep +chain and one 10,000-node strongly connected component prove that manual cycle planning has no +recursion-depth failure. + +An isolated wheel installation passed CLI and MCP startup, a real detached manual render, and the +closed malformed-worker-request contract. Six Milestone 3 commits and the complete candidate tree +passed Gitleaks 8.30.1 with no findings. + +Three independent adversarial review tracks covered manual isolation and fragment integrity, +portable publication and policy binding, and worker/accessibility/benchmark gates. Reproduced +project import, hostile environment, unbounded stdout, fragment forgery, cache growth, aggregate +overflow, coordinated policy drift, render-limit compatibility, deep-graph, and module-startup +defects were fixed and regression-tested before closeout. + +The clean ten-sample 1,000-node benchmark passed every threshold. Exact measurements, equivalence +results, memory peaks, and response sizes are recorded in +[`MILESTONE_3_BASELINE.md`](MILESTONE_3_BASELINE.md) and +[`benchmarks/milestone3-2026-07-29.json`](../benchmarks/milestone3-2026-07-29.json). + +## Preserved boundaries + +- The `docforge` distribution, package, CLI, MCP executable, and existing tool names remain. +- The frozen alpha manual remains exactly 2,043 bytes with its legacy output hash and render + identity. +- Legacy one-method `load_projection()` adapters remain supported. +- Effective policy version 1, no-AST behavior, and existing client bindings remain compatible. +- Project descriptor schema version 1 and SQLite index schema version 3 remain unchanged. +- Configured `max_render_bytes` values above the detached transfer ceiling still load; a small + actual artifact renders normally. Actual detached transfer remains capped at 20,000,000 bytes. +- No storage replacement or self-hosting dependency was introduced. +- WorldForge and ScrapeStation were not touched. +- No production MCP integration was repointed. +- The legacy Forgejo repository and `legacy` remote were not changed. +- No tag, release, release announcement, or visibility change was created. + +## Known follow-up work + +Milestone 4 remains directional and is not active. Its adapter SDK and product-documentation work +must not silently absorb these separate future ideas: + +- Optimize production fragment reuse only from measured profiles while preserving the forced-full + oracle. +- Add authenticated cursors only if a stronger threat model requires them. +- Verify Claude's native timeout representation. +- Add versioned adapter-owned launcher metadata before generating custom-adapter configurations. +- Keep remote render services, shared render farms, third-party renderers, storage replacement, + and self-hosting deferred until their own evidence justifies them. diff --git a/docs/USER_MANUAL.md b/docs/USER_MANUAL.md index a92ed1b..640f9ae 100644 --- a/docs/USER_MANUAL.md +++ b/docs/USER_MANUAL.md @@ -27,6 +27,10 @@ incremental methods while retaining the full loader as a fallback. - Opt-in incremental adapter extraction with reverse-dependency invalidation. - Lazy function-scoped logic projections that do not densify the primary graph. - Declared HTML render views. Arbitrary templates, render commands, and output paths are rejected. +- Separate versioned manual and portable graph plans, immutable packages, detached built-in + renderers, and validated receipts. +- Content-addressed portable Nodes/Flow/Web artifacts with receipt-only status and repair. +- Independent manual, portable-graph, and live-viewer policy. - A loopback-only graph browser with Nodes, semantic Flow, and convergence Web views, relationship keys, source inspection, branch-aware node hiding, panel resizing, zooming, and managed idle shutdown. @@ -47,7 +51,11 @@ Canonical files own facts: canonical Markdown/TOML or adapter sources ↓ validate disposable SQLite graph - ↓ query / visualize / compile context + ├── query / compile context + ├── ManualRenderPlanV1 → detached manual renderer → declared manual + ├── GraphViewPlanV1 → detached graph renderer → portable Nodes/Flow/Web artifact + └── pinned index → managed live Nodes/Flow/Web/Logic viewer + ↓ people and agents ↓ propose isolated changeset + preview @@ -77,8 +85,8 @@ source format. Clone and verify DocForge: ```bash -git clone forgejo@repo.andraxion.net:administrator/DocForge.git /absolute/path/DocForge -cd /absolute/path/DocForge +git clone forgejo@repo.andraxion.net:administrator/DocForge2.git /absolute/path/DocForge2 +cd /absolute/path/DocForge2 uv sync --group dev npm ci @@ -159,6 +167,27 @@ output = "Docs/Rendered/Manual.html" title = "My Project Manual" families = ["architecture", "system", "operations", "roadmap"] +[graph_render] +output_root = ".docforge/portable-graph" + +[[graph_render.views]] +id = "architecture" +renderer = "portable_graph_html" +output = "architecture.html" +title = "Architecture" +root = "architecture.overview" +initial_mode = "web" +depth = 3 +max_nodes = 250 +max_edges = 1000 +max_work = 100000 +families = ["architecture", "system"] +relations = ["depends_on", "owns", "calls", "reads", "writes", "tested_by", "relates_to"] +authorities = [] +statuses = ["current", "active", "verified"] +tags = [] +include_logic = false + [graph] allowed_relations = ["depends_on", "owns", "calls", "reads", "writes", "tested_by", "relates_to"] @@ -492,13 +521,25 @@ builds or repairs the index. ```text render-status [VIEW_ID] [--deep] render VIEW_ID +graph-plan VIEW_ID +graph-render VIEW_ID +graph-render-status [VIEW_ID] preview CHANGESET_ID VIEW_ID apply CHANGESET_ID --changeset-hash SHA256 --applier WRITER_ID ``` +`graph-plan` validates and returns one declared `GraphViewPlanV1` without publishing. A portable +view must select exactly one stable `root` or metadata-only lexical `query`. It may use only Nodes, +Flow, or Web as `initial_mode`; portable version 1 excludes function-scoped Logic. + +`graph-render` explicitly publishes the declared static artifact, content-addressed renderer +evidence, and generation/view manifest. `graph-render-status` verifies only bounded committed +evidence and never plans or renders. Portable publication is a local CLI action. + The CLI apply command supports the generic adapter. It verifies that the configured writer owns the changeset, applies the exact reviewed hash, rebuilds the index, checks it, and regenerates every -declared render. It does not commit or push the result. +declared manual render when manual policy is `auto`. It does not publish portable graphs, commit, +or push the result. ### Viewer commands @@ -549,6 +590,20 @@ declares the named writer, and application requires the same writer/applier iden `--no-ast` to preserve the no-AST binding. Generic CLI generation refuses project-owned adapters because it cannot safely reconstruct their composition. +Select projection behavior independently: + +```bash +docforge configure codex \ + --project /absolute/path/MyProject \ + --manual-render-policy explicit \ + --portable-graph-policy disabled \ + --live-viewer-policy on-demand +``` + +The generated version-1 configuration result carries an additive version-2 `projection_policy`, +its hash, projection availability, and the exact descriptor hash. Omitted default selectors are +validated against that descriptor rather than trusted as self-reported output. + Inspect one configured client binding: ```bash @@ -569,6 +624,54 @@ starts MCP, executes the configured command, synchronizes, builds, renders, star writes configuration. Claude timeout representation and client filtering that cannot be proved locally remain explicit warnings. +## Independent projection behavior + +Manual and portable graph renderers consume immutable, path-free packages. A package binds one +generation-pinned plan, inert assets, fixed component versions, a built-in renderer identity, and +an exact artifact inventory. The detached child cannot select nodes, open the project or index, +choose a publication path, execute project code, or mutate canonical facts. + +Child startup is fixed to isolated Python, a private module entrypoint, a trusted working +directory, and a sanitized environment. One request and response use canonical newline-terminated +JSON. The request, response, receipt, execution time, and disk-spooled stdout are bounded. Actual +artifact transfer is capped at 20,000,000 bytes even when the descriptor retains a larger +`max_render_bytes` compatibility value. + +Manual fragment records are disposable semantic cache entries. On a cold miss, DocForge performs a +trusted full detached render, extracts candidate page fragments, and compares fragment-assisted +output byte-for-byte before publishing records. On a warm hit, the worker recomputes each expected +page fragment before accepting cached bytes. Corrupt, forged, stale, individually oversized, or +aggregate-oversized records fall back to the full oracle. Fragment reuse is currently a correctness +and recovery boundary, not a promised speedup. + +Projection policy version 2 is: + +```text +manual: auto | explicit | disabled +portable_graph: explicit | disabled +live_viewer: on-demand | disabled +``` + +For ordinary CLI commands, place the corresponding global flag before the subcommand: + +```bash +docforge --project-root "$PROJECT" --manual-render-policy disabled render manual +docforge --project-root "$PROJECT" --portable-graph-policy disabled graph-plan architecture +docforge --project-root "$PROJECT" --live-viewer-policy disabled visualize +``` + +An active operation blocked by policy returns `projection_policy_forbids_operation` before hidden +work. Manual and portable receipt-only status remain available. Viewer status and explicit stop +remain available when viewer start is disabled. + +A non-disabled projection also requires its declared configuration or runtime. Manual `explicit` +requires manual render configuration. Manual `auto` additionally requires canonical application in +the current operation or server capability. Portable graph `explicit` requires portable graph +render configuration, and live viewer `on-demand` requires its runtime. An unavailable selection +returns `projection_policy_unavailable` before work begins. In particular, ordinary CLI `render` +operations cannot select manual `auto`; use `explicit`, or let a configured canonical `apply` +operation own automatic regeneration. + ## MCP usage Run one MCP server per project with absolute paths: @@ -586,7 +689,10 @@ Select the session's declared surface explicitly when useful: ```bash docforge-mcp \ --project-root /absolute/path/MyProject \ - --capability-mode read + --capability-mode read \ + --manual-render-policy explicit \ + --portable-graph-policy explicit \ + --live-viewer-policy on-demand ``` Supported modes are `read`, `proposal`, `application`, and `operator`. Existing startup defaults @@ -613,8 +719,9 @@ identity, not a command. The changeset creator, configured writer, and canonical Call `docforge_bootstrap` first. Its version-1 `session_contract` contains the fixed binding, current graph generation, effective policy, actual capabilities, render policies, prohibitions, -and a recommended first operation. Workflow guidance does not recommend registration or -application when those startup capabilities are unavailable. +and a recommended first operation. The result also carries the independently composed version-2 +`projection_policy` and hash. Workflow guidance does not recommend registration or application +when those startup capabilities are unavailable. Example MCP client configuration: @@ -653,11 +760,16 @@ Example MCP client configuration: - `docforge_get_task_context` - `docforge_validate_project` - `docforge_render_status` +- `docforge_graph_plan` +- `docforge_graph_render_status` - `docforge_visualize` - `docforge_visualization_status` - `docforge_stop_visualization` - `docforge_get_generation_diff` +MCP graph plan and status are read-only. MCP does not expose portable graph publication; use the +explicit local `graph-render` CLI command. + ### Proposal tools - `docforge_create_changeset` @@ -976,8 +1088,29 @@ docforge --project-root "$PROJECT" render-status docforge --project-root "$PROJECT" render VIEW_ID ``` -Successful canonical apply regenerates all declared views automatically. A manual canonical edit -requires reindexing and rendering. +Successful canonical apply regenerates declared manual views only when manual policy is `auto`. +A manual canonical edit requires reindexing and explicit rendering. Portable graph publication +always remains a separate explicit CLI action. + +Portable graph publication has separate status and policy: + +```bash +docforge --project-root "$PROJECT" graph-render-status +docforge --project-root "$PROJECT" graph-render architecture +``` + +### `projection_policy_forbids_operation` + +The process was deliberately started with the relevant manual, portable-graph, or live-viewer +operation disabled. Restart with an allowed selector after confirming that the integration should +receive that capability. Status and explicit stop operations remain available as described above. + +### `projection_policy_unavailable` + +The selected non-disabled projection has no matching project configuration or runtime. Add the +declared manual or portable graph render configuration, or make the live viewer runtime available, +before selecting that mode. Manual `auto` also requires an operation or MCP server with canonical +application enabled. Use manual `explicit` for a standalone CLI render. ### Descriptor changed after startup @@ -992,8 +1125,11 @@ Run the complete release gate from the DocForge repository: make gate ``` -Use `make benchmark` for the historical Milestone 0 baseline and `make benchmark-m1` for the -counter-gated 1,000-node warm-operation benchmark. +Use `make benchmark` for the historical Milestone 0 baseline, `make benchmark-m1` for the +counter-gated warm-operation benchmark, `make benchmark-m2` for agent workflow gates, and +`make benchmark-m3-full` for the ten-sample 1,000-node projection, worker, fragment, status, +equivalence, response-size, and memory gates. `make accessibility` runs the generated manual, +portable graph, and live viewer axe and keyboard flows. Project-specific vocabulary, extraction rules, and serialization belong in the project adapter. Generic core behavior must remain deterministic, project-bound, and recoverable.