diff --git a/ACTIVE_SLICE.md b/ACTIVE_SLICE.md index bd86996..22b7caa 100644 --- a/ACTIVE_SLICE.md +++ b/ACTIVE_SLICE.md @@ -1,72 +1,15 @@ # Milestone state ```text -Last completed milestone: 3 — independent projections -Evidence: d6d9f47672c513c4927c5c42ad97a175ed651a74 is merged and pushed on main and dev. -Active milestone: 4 — adapter SDK and product documentation. +Last completed milestone: 4 — adapter SDK and product documentation. +Evidence: Frozen executable candidate 95271dcf2e48045b9d3aed9b9ea09c7fc155692c; 347 tests and 402 subtests; 3 accessibility flows; offline fresh-wheel adoption; clean 1,002-node graph-plus-Logic benchmark; no secret-scan findings. +Active milestone: None. Outcome: New projects can adopt DocForge incrementally, and users can understand the CLI, MCP server, policies, adapters, renderers, and agent integration without reading implementation code. -Status: Active. The first gate freezes and proves the adapter SDK before language references. +Status: Milestone 4 is closed. Milestone 5 has not started. Next directional milestone: 5 — stabilization and first DocForge2 release. ``` -## Milestone 4 contract +Milestone 5 remains directional context. Do not begin release stabilization, versioning, tagging, +or publication without a new active-slice contract based on the Milestone 4 evidence. -Milestone 4 closes only when all of the following are true: - -1. A typed public adapter SDK and repository-native conformance kit make complete, incremental, - Logic, confinement, restart, cache-recovery, no-AST, and retrieval contracts usable without - reading core implementation. -2. Repository-owned runnable reference integrations exist for Python, JavaScript/TypeScript, and - C++. They declare their evidence authorities and unsupported facts. Syntax-only evidence never - claims compiler-resolved calls, inheritance, types, or ownership. -3. Heavy language frontends are optional. A base wheel and no-AST operation work without them. - Missing optional features fail with closed, actionable errors. -4. CLI and MCP reference tables are generated deterministically from implementation-owned - metadata and a repository gate rejects drift. -5. Custom adapters can declare one versioned, immutable, project-bound launcher. Codex, Claude, - and OpenClaw client configuration is generated from that launcher without arbitrary discovery - or execution. -6. Dedicated onboarding, authority, descriptor, policy, adapter, language, legacy/no-AST, - rendering, agent-integration, security, recovery/performance, and DocForge v1 migration - guidance passes a fresh-wheel adoption proof. -7. Existing package, import, CLI, MCP, legacy-adapter, generic-project, no-AST, projection, - application, and recovery behavior remains additively compatible. - -## Fixed decisions - -- Milestone 4 provides stable in-repository SDK APIs and reference integrations. It does not - promise separately published language-adapter distributions. -- Complete adapter evidence includes primary graph and function Logic. An incremental adapter that - publishes Logic must expose an independent complete Logic oracle before equivalence can pass. -- Python reference evidence may use the standard-library AST. JavaScript/TypeScript and C++ - reference evidence uses optional pinned parsing frontends for syntax plus declared build - inventory. It omits semantic facts that require a compiler, language server, or SCIP evidence. -- C++ inventory is bound to a project-confined compilation database. WorldForge is not used or - changed. -- JavaScript and TypeScript are both exercised. TypeScript support requires its own optional - pinned grammar; it is not silently parsed as JavaScript. -- Adapter extraction caches remain disposable and may become safely ahead of the last published - SQLite index. The index remains the active validated generation. Recovery must prove this - behavior instead of claiming one joint cache-and-index transaction. -- Generated references and templates are derived artifacts. Implementation metadata is their - authority. - -## Exclusions - -- No WorldForge, ScrapeStation, legacy-repository, or production-binding change. -- No remote adapter execution, arbitrary adapter commands, renderer ecosystem, render farm, - storage rewrite, or cross-project graph. -- No compiler or language-server installation by DocForge. -- No self-hosting requirement. -- No version, release tag, or Forgejo release before Milestone 5. - -## Slice order - -1. Freeze complete graph-plus-Logic equivalence, the SDK facade, conformance protocol, schemas, and - baseline measurements. -2. Add deterministic command metadata, generated CLI/MCP references, and a drift gate. -3. Add the Python reference integration and scaled complete/incremental benchmark. -4. Add optional JavaScript/TypeScript and C++ reference integrations with bounded evidence. -5. Add versioned adapter launcher metadata and generated agent configurations. -6. Freeze the candidate, perform one documentation closeout, and run focused, complete, - fresh-wheel, adoption, benchmark, accessibility, build, and secret-scan gates. +The completed Milestone 4 contract and its exclusions are preserved in `SLICE_HISTORY.md`. diff --git a/DEVELOPMENT_NOTES.md b/DEVELOPMENT_NOTES.md index 2548012..78f6be8 100644 --- a/DEVELOPMENT_NOTES.md +++ b/DEVELOPMENT_NOTES.md @@ -784,3 +784,108 @@ proof. Gitleaks 8.30.1 found no findings across the six Milestone 3 commits or c Milestone 3 is complete. No tag, release, production integration repointing, WorldForge change, ScrapeStation change, storage rewrite, or self-hosting dependency was introduced. Milestone 4 remains directional and has not started. + +## Milestone 4 — complete: adapter SDK and product documentation + +Milestone 4 activated from the verified Milestone 3 closeout. Its scope was deliberately additive: +freeze an adapter-authoring boundary, prove narrow language references, make heavy frontends +optional, generate command and client integration evidence, and document adoption. WorldForge, +ScrapeStation, the legacy repository, production bindings, self-hosting, storage replacement, and +release publication remained excluded. + +### Adapter SDK and conformance + +`docforge.adapter_sdk` is now the public authoring facade for typed projections, manifests, source +contributions, complete assemblies, incremental loaders, project bindings, core graph models, and +conformance reports. Complete evidence includes primary graph and function Logic. An incremental +adapter that publishes Logic must provide an independent `load_complete_assembly()` oracle. + +The conformance helper repeats complete loading for determinism, compares the complete assembly +with `load_projection()`, and proves exact complete/incremental graph-plus-Logic parity. It does not +pretend to replace separate confinement, restart, no-AST, corrupt-cache, or retrieval tests. + +An adversarial review found that extraction caches and adapter assemblies lacked aggregate +boundaries. Version-1 cache reads and writes are now regular-file-only, identity-checked, capped at +64,000,000 bytes and 10,000 sources, and fail safely to a miss. Adapter projects enforce primary +node limits and deterministic edge and Logic multipliers before publication. + +### Reference language evidence + +The Python reference uses the standard-library AST and publishes files, modules, classes, +functions, arguments, local static imports, and function Logic. Its manifest uses tokenization +rather than AST parsing, so unchanged warm builds perform no syntax parse. + +JavaScript and TypeScript use separate pinned optional Tree-sitter grammars. They publish files, +modules, classes, functions/methods, static project-relative imports/re-exports, and function +Logic. Their manifests use a closed comment/string-aware ESM scan and do not load Tree-sitter. + +C++ uses explicit non-overlapping source roots and one confined `compile_commands.json` as +translation-unit inventory and fingerprint evidence. It validates entries but never executes a +command, compiler, response file, or project program. It publishes syntax and directly resolvable +project-local quoted includes. Its manifest currently parses for include discovery, so no +zero-warm-parser claim is made. + +None of the references claim compiler-resolved calls, inheritance, types, symbol references, +compiler include semantics, macro semantics, runtime behavior, or semantic ownership. + +The first packaging proof exposed that language parsers were inherited as mandatory dependencies. +The base wheel now depends only on Markdown and MCP packages. Python works in the base wheel; +JavaScript, TypeScript, C++, and all-language extras install their respective Tree-sitter +frontends. Missing extras return a closed error with the exact install target. + +### Fixed reference MCP and launchers + +A strict `.docforge/reference-adapter.toml` selects one fixed reference language, project +identity, and explicit source roots. C++ additionally requires the compilation database. The fixed +`python -m docforge.reference_mcp` binding constructs only repository-owned providers and exposes +the 21 read tools. + +The initial launcher design generated `python -I -m` for arbitrary project module names without +proving that isolated Python could resolve them. The corrected `AdapterLauncherV1` accepts only an +installed project-confined top-level module or the exact trusted `docforge.reference_mcp` +exception. A fixed bounded `find_spec` probe runs under isolated Python without importing project +code. Tests then launch both a real project module and the real reference MCP process. + +Generated Codex, Claude, and OpenClaw fragments bind the exact launcher, descriptor, source +availability, effective and projection policy, interpreter, canonical arguments, empty +environment, artifact bytes, and hashes. There is no arbitrary command, argument list, working +directory, environment, discovery, or callable selector. + +### Generated reference and documentation gates + +CLI tables are derived from the real argparse subcommands. MCP tables are derived from a real +application-enabled server's `list_tools()` registrations, including capability surface, +arguments, descriptions, and input-schema hashes. The generated artifact contains 28 CLI rows and +36 MCP rows. + +An adversarial review found a target-change race between the generator's identity check and +replacement. Publication now locks cooperating generators, uses no-clobber publication for a +missing target, and uses Linux atomic exchange plus displaced-byte and file-identity verification +for an existing target. Raced data is restored or retained for recovery rather than discarded. + +`docs-check` validates generated drift, local files and anchors, one H1 per page, README-rooted +reachability, required Milestone 4 inventory, the generated notice, and every documented +reference-adapter TOML example against the packaged schema. + +### Candidate, adoption, and scale evidence + +Frozen executable candidate `95271dcf2e48045b9d3aed9b9ea09c7fc155692c` passed formatting, +Python and web lint, strict Pyright, compilation, 142 contract tests and 268 subtests, 347 total +tests and 402 subtests, three accessibility flows, lock and dependency checks, package builds, +offline wheel adoption, and every maintained smoke benchmark. + +The offline adoption proof installed the base wheel from the lock without network access. It +proved no Tree-sitter package or module was installed, built and checked a real Python reference +project, performed isolated MCP bootstrap/search/get-node over the exact 21 read tools, and proved +that base-only C++ fails with `optional_dependency_missing` and `docforge[cpp]` remediation. + +The clean 334-source benchmark produced 1,002 primary nodes, 1,001 primary edges, 334 Logic +projections, 2,338 Logic nodes, and 2,338 Logic edges. Complete and incremental output matched +exactly. Warm build p95 was 1,265.387 ms with zero `ast.parse` and zero `extract_source` calls. +Complete equivalence, corrupt cache recovery, and corrupt index recovery were all below 1.9 +seconds. The largest per-operation traced peak was 69,997,166 bytes and process high-water was +78,798,848 bytes. Exact hashes and method are recorded in +`benchmarks/milestone4-2026-07-29.json` and `docs/MILESTONE_4_BASELINE.md`. + +Milestone 4 is complete. No tag or release was created. Milestone 5 remains unstarted until its own +active-slice contract is frozen. diff --git a/README.md b/README.md index 20ba862..6e14013 100644 --- a/README.md +++ b/README.md @@ -1,88 +1,107 @@ # DocForge -DocForge is a project-scoped documentation graph for people and AI agents. It validates canonical -documentation, builds a disposable search and relationship index, compiles bounded context, renders -declared manuals, visualizes project structure, and manages reviewable documentation changesets. +DocForge is a project-scoped documentation and source graph for people and AI agents. Canonical +project files remain authoritative; DocForge validates them, builds disposable search and +relationship indexes, compiles bounded context, manages reviewable changesets, renders declared +manuals and portable graph views, and serves one project-bound MCP surface. -## What it does +DocForge never treats indexed text as instructions. It does not run project build commands, +compilers, Git operations, deployments, or arbitrary renderers, and it does not select projects +globally. -- Validates stable Markdown/TOML nodes and typed relationships. -- Builds a deterministic SQLite search and graph index. -- Exposes project-bound CLI and MCP query surfaces. -- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, and bounded - continuation. -- Records one bounded, versioned latest-generation graph transition without creating a history - database. -- Generates deterministic project-bound Codex, Claude, and OpenClaw client fragments without - copying ambient secrets. -- Diagnoses one client binding through bounded read-only checks without starting MCP or rebuilding - project state. -- Automatically synchronizes disposable indexes before MCP work. -- Creates, validates, diffs, and previews isolated changesets. -- Registers complete proposals atomically without caller-managed hash chaining. -- Applies one explicitly approved changeset hash through CLI or gated MCP. -- Supports opt-in incremental adapters with reverse-dependency invalidation and full-build - equivalence checks. -- Detects project-local adapter implementation and configuration changes and requires a fresh - project-bound process before any further MCP work. -- Keeps function-scoped control-flow projections separate from the primary architecture graph. -- Compiles manuals and portable graph artifacts from separate versioned, generation-pinned plans - and immutable packages. -- Runs built-in manual and graph renderers in fixed detached workers with validated receipts, - bounded transfer, and no project-path authority. -- Enforces independent manual, portable-graph, and live-viewer policy while keeping status - receipt-only. -- Runs a managed loopback graph browser with neighborhood, semantic Flow, convergence Web, - function-scoped Logic, source inspection, and branch-aware node hiding. -- Supports generic documentation projects and project-owned source adapters. +DocForge 1.0.0 remains the first stable release in this product line. This checkout is the +post-1.0 development line (`1.3.0.dev0`) and contains the Milestone 4 adapter-platform work. +Milestone 4 is not a new product release, and Milestone 5 release and publication work remains +separate. -DocForge never treats indexed text as instructions. It does not run shell commands, mutate Git, -build applications, deploy, publish, or select projects globally. +## Start here -For implementation projects, the recommended cadence is to read canonical documentation during -intake, keep it read-only through implementation and focused testing, freeze and validate a release -candidate, then perform one atomic documentation closeout before the final commit and tag. This -keeps the manual authoritative without using it as an implementation notebook. +- New installation or first project: [New-project quickstart](docs/NEW_PROJECT_QUICKSTART.md) +- Mental model and authority: [Core concepts and authority](docs/CORE_CONCEPTS_AND_AUTHORITY.md) +- Complete configuration shape: [Project descriptor](docs/PROJECT_DESCRIPTOR.md) +- Fixed runnable examples: [Reference adapters](docs/REFERENCE_ADAPTERS.md) +- Agent and client setup: [Agent integration](docs/AGENT_INTEGRATION.md) +- Exact live command and tool inventory: [Generated command reference](docs/COMMAND_REFERENCE.md) +- Task-oriented operating guide: [User manual](docs/USER_MANUAL.md) -## Release 1 +## Current capabilities -DocForge 1.0.0 is the first stable product release. It combines the project-scoped graph, CLI and -MCP query surfaces, reviewable hash-approved changesets, generic and project-owned adapters, -declared rendering, and the complete Nodes/Flow/Web visualization model in one supported release. +- Validates Markdown and TOML nodes, stable IDs, typed relationships, project limits, and confined + paths. +- Builds a deterministic, disposable SQLite graph and search index with integrity and generation + evidence. +- Exposes project-bound CLI and MCP read, proposal, application, rendering, and visualization + surfaces according to the startup policy. +- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, bounded + output, and deterministic continuation. +- Creates isolated documentation changesets, validates complete projected graphs, and applies only + one explicitly approved changeset hash through a separately bound canonical applier. +- Supports complete-projection adapters and opt-in incremental adapters with reverse-dependency + invalidation, bounded extraction caches, and a clean full-build equivalence oracle. +- Publishes function-scoped Logic separately from the primary graph for Python, JavaScript, + TypeScript, and C++ integrations that provide it. +- Compiles manual and portable graph plans into immutable packages for fixed detached renderers, + then records bounded receipts. +- Runs a managed loopback graph viewer with Nodes, Flow, Web, lazy Logic, source inspection, and + branch-aware hiding. +- Generates deterministic Codex, Claude, and OpenClaw client fragments without copying ambient + environment values or secrets. -The post-1.0 incremental compiler is a backward-compatible, optional enhancement. Existing Release -1 adapters that implement only `load_projection()` continue to use the original complete-projection -path without modification. Adapters gain incremental performance only when they additionally -implement the source manifest and extraction methods. Incremental adapters must retain -`load_projection()` as their clean-rebuild fallback and equivalence oracle. +## Adapter platform -MCP bindings can additionally select `--no-ast` when an owner wants to preserve an existing -non-AST adapter. The binding advertises that policy to clients, forbids adapter rewrites that add -AST, Tree-sitter, compiler-AST, or function-Logic extraction, blocks the Logic tool, and rejects -nonempty Logic publication. Complete-projection adapters continue unchanged, and non-AST -incremental fingerprinting and caching remain allowed. +Adapter authors use the public `docforge.adapter_sdk` surface for graph types, complete and +incremental contracts, validation helpers, and `verify_adapter_conformance()`. Existing adapters +that implement only `load_projection()` remain supported. Incremental adapters add a manifest and +source extraction while retaining `load_projection()` as the complete graph oracle. An adapter +that publishes Logic incrementally also supplies `load_complete_assembly()` so the complete oracle +covers both graph and Logic. -DocForge2 bindings may also declare -`--capability-mode read|proposal|application|operator`. Bootstrap returns one versioned effective -policy and the actual startup-gated capabilities. Existing tool surfaces and the legacy no-AST -payload remain compatible. +The repository includes bounded reference integrations for: -`docforge_get_task_context` is an additive read tool for `change`, `implementation`, `failure`, -`ownership`, `test`, `operation`, and `release` work. It derives a closed version-1 retrieval plan, -executes it against one immutable index generation, and returns a hash-bound context capsule. -Project relation names remain authoritative. DocForge classifies only its versioned alias set and -preserves every unknown relation as `unclassified` instead of guessing semantics. +- Python, using the standard-library AST and publishing only project-local imports; +- JavaScript and TypeScript, using their distinct optional Tree-sitter grammars and publishing only + project-local static relative imports and re-exports; +- C++, using `compile_commands.json` as translation-unit inventory and fingerprint evidence, without + executing its commands or a compiler, and publishing only directly resolvable project-local + quoted includes. -`docforge_get_generation_diff` reports the latest verified primary-graph transition through one -bounded disposable receipt. It includes exact node and edge change counts, hash-bound retained -details, and explicit truncation. Paged results use one top-level cursor and a versioned -`receipt_header`; `stored_receipt_hash` identifies the complete persisted receipt. The read never -exposes Logic details, loads canonical source, repairs derived state, or invents history. +These integrations demonstrate the adapter contract; they do not claim resolved calls, +inheritance, types, runtime behavior, macro expansion, compiler include semantics, or semantic +ownership. Python works from the base wheel. Install the `javascript`, `typescript`, or `cpp` +extra for the corresponding grammar, or `languages` for all three: -## Independent projections +```bash +uv pip install "/absolute/path/DocForge[javascript]" +uv pip install "/absolute/path/DocForge[typescript]" +uv pip install "/absolute/path/DocForge[cpp]" +uv pip install "/absolute/path/DocForge[languages]" +``` -Manual compilation, portable graph rendering, and the live viewer consume the same validated graph -generation through separate boundaries: +Reference projects use the fixed `.docforge/reference-adapter.toml` descriptor and the installed +`docforge.reference_mcp` module. That server is read-only and exposes no proposal or application +surface. + +Project-owned adapters use `AdapterLauncherV1` and +`generate_adapter_client_configuration()` to produce a client fragment from an explicitly +constructed adapter project. The launcher is immutable and contains no command, caller arguments, +working directory, environment, discovery rule, or callable selector. It resolves through isolated +Python to one installed, project-owned top-level module; the only trusted dotted exception is the +fixed `docforge.reference_mcp` binding. Generic `docforge configure` intentionally refuses custom +adapters because it cannot safely reconstruct project-owned composition. + +See the [Adapter authoring guide](docs/ADAPTER_AUTHORING_GUIDE.md), [Reference +adapters](docs/REFERENCE_ADAPTERS.md), and [Agent integration](docs/AGENT_INTEGRATION.md) for the +supported routes. + +## Authority and projections + +Canonical Markdown, TOML, adapter-declared sources, and descriptor files own project facts. The +SQLite index, extraction cache, render packages, previews, portable artifacts, receipts, client +fragments, and viewer processes are derived and replaceable. + +The primary graph contains project nodes and relationships. Logic is a separate, lazy, +function-scoped control-flow projection. Manual output, portable graph output, and the live viewer +are independent consumers of one validated generation: ```text validated generation @@ -91,190 +110,142 @@ validated generation └── pinned index → managed read-only live viewer ``` -Plans, packages, and receipts are canonical, versioned, hash-identified, bounded, and contain no -project object, SQLite handle, absolute project path, command, or caller-selected renderer module. -Renderers cannot select graph facts, crawl canonical sources, choose publication paths, or mutate -project state. +The version-2 projection policy selects each consumer independently: -Declare portable graph output separately from manual views: - -```toml -[graph_render] -output_root = ".docforge/portable-graph" - -[[graph_render.views]] -id = "architecture" -renderer = "portable_graph_html" -output = "architecture.html" -title = "Architecture" -root = "architecture.overview" -initial_mode = "web" -depth = 3 -max_nodes = 250 -max_edges = 1000 -max_work = 100000 -include_logic = false +```text +manual: auto | explicit | disabled +portable_graph: explicit | disabled +live_viewer: on-demand | disabled ``` -Plan, publish, and inspect it explicitly: +The process capability policy is separate. Capability mode controls the registered read, +proposal, application, or reserved operator surface; descriptor writers and the startup-bound +canonical applier determine whether mutations are actually authorized. `--no-ast` is a restrictive +binding policy over adapter evolution, Logic publication, and Logic retrieval. It is not a parser +inspection mechanism or a filesystem sandbox, and the AST/Tree-sitter reference integrations +should not be presented as no-AST adapters. + +Read [Policy precedence](docs/POLICY_PRECEDENCE.md), [Rendering and +visualization](docs/RENDERING_AND_VISUALIZATION.md), and [Legacy and no-AST +operation](docs/LEGACY_AND_NO_AST.md) before changing those boundaries. + +## Five-minute generic project + +Requirements are Python 3.12 or newer and an installed DocForge environment. From a development +checkout, `uv sync --group dev` creates `.venv`: ```bash -.venv/bin/docforge --project-root "$PROJECT" graph-plan architecture -.venv/bin/docforge --project-root "$PROJECT" graph-render architecture -.venv/bin/docforge --project-root "$PROJECT" graph-render-status architecture -``` - -The version-2 projection policy independently selects manual -`auto|explicit|disabled`, portable graph `explicit|disabled`, and live viewer -`on-demand|disabled`. Use `--manual-render-policy`, `--portable-graph-policy`, and -`--live-viewer-policy` on CLI/MCP startup or generated client configuration. Status remains -available when the corresponding active operation is disabled. - -Manual fragment reuse is disposable. Cold record publication is guarded by byte-exact comparison -with a full detached render; warm records are independently recomputed and validated inside the -worker. Full rendering remains the recovery and equivalence oracle. Portable artifacts commit -content-addressed output and renderer evidence before one bounded generation/view manifest; status -does not plan or render. - -## Graph views - -The browser presents the primary architecture graph through three complementary views and loads a -fourth function-scoped view only when requested: - -- **Nodes** shows a bounded, relation-neutral neighborhood around the focus. It is the broad - inspection view for seeing stored incoming and outgoing relationships without changing their - direction. Semantic cards distinguish structure, behavior, dependencies, execution, data, - evidence, context, and other relationships. -- **Flow** shows semantic origin-to-destination paths that terminate at the focus. DocForge - reverses prerequisite-style relationships for presentation, so imports, dependencies, reads, - inheritance, definitions, and tests flow toward the thing they help create or exercise. -- **Web** shows the larger convergence picture: Flow contributors plus contextual relationships, - callers, containers, and direct members or execution dependencies owned by the focus. -- **Logic** shows the possible static control paths inside a focused Python, JavaScript, or C++ - function or method. Entry, decisions, actions, loops, convergence points, returns, and - exceptions connect through explicit - `TRUE`, `FALSE`, `NEXT`, `CASE`, `LOOP`, `RETURN`, and `RAISE` paths. Logic is stored separately - and does not add statement-level noise to Nodes, Flow, Web, or search. - -Graph cards show the node's readable leaf name and kind without clipping either value. The full -qualified identity remains available in the tooltip, compact descriptor, and full inspector. -The left browser panel can combine text, family, node-kind, language, and capability filters. -Quick presets expose Logic-ready nodes, Python callables, tests, routes, and documentation without -requiring users to know stable IDs. Selecting a canvas node emphasizes its directly connected -neighbors and edges while muting unrelated paths. - -**Hide node** removes noise without changing the index. In Flow and Web, hiding a contributor also -removes upstream ancestors that no longer have a path to the focus. Nodes between the hidden -contributor and the focus stay visible, and alternate ancestor paths remain intact. In Logic, -hiding a step inserts an explicit omitted-path bridge so downstream control flow remains readable. -**Restore hidden** restores the presentation. - -## Five-minute start - -Requirements are Python 3.12+, `uv`, and Node.js/npm. - -```bash -git clone forgejo@repo.andraxion.net:administrator/DocForge2.git /absolute/path/DocForge2 -cd /absolute/path/DocForge2 +git clone /absolute/path/DocForge +cd /absolute/path/DocForge uv sync --group dev -npm ci PROJECT=/absolute/path/MyProject +.venv/bin/docforge --project-root "$PROJECT" onboard +.venv/bin/docforge --project-root "$PROJECT" onboard \ + --scaffold \ + --project-id my-project \ + --title "My Project" +``` + +The first command is read-only. Scaffolding is explicit and create-only: it writes a generic +descriptor, one canonical overview node, and a built-in manual template, then indexes and renders +them. Detected source languages remain `adapter_required` until a validated frontend is selected. + +Operate the configured project: + +```bash .venv/bin/docforge --project-root "$PROJECT" validate .venv/bin/docforge --project-root "$PROJECT" reindex +.venv/bin/docforge --project-root "$PROJECT" search architecture .venv/bin/docforge --project-root "$PROJECT" visualize ``` -Install the persistent per-user graph viewer once: +Install the persistent per-user graph viewer manager once when using the live viewer: ```bash .venv/bin/docforge-viewer-manager install-user-service ``` -Start an MCP server for one project: +Start one generic, read-only MCP server: ```bash .venv/bin/docforge-mcp \ --project-root "$PROJECT" \ - --proposal-writer project-editor + --capability-mode read ``` -Add `--canonical-applier project-editor` only when that MCP integration should expose the -hash-bound `docforge_apply_changeset` tool. - -Preview a read-only Codex fragment and diagnose an installed binding: - -```bash -.venv/bin/docforge configure codex --project "$PROJECT" -.venv/bin/docforge doctor --client codex --project "$PROJECT" -``` - -Pass `--output /absolute/path/docforge.toml` only when creating a standalone fragment. DocForge -never replaces or merges an existing different client file. - -For an unconfigured codebase, begin with a read-only language and documentation assessment: - -```bash -.venv/bin/docforge --project-root /absolute/path/MyProject onboard -``` - -Add `--scaffold`, a stable project ID, and a title to create, index, and render a generic starter -manual. Source files are reported separately and require a validated language frontend before -DocForge describes them as a source graph. +Add a descriptor-authorized proposal writer only when the client should create proposals. Add a +matching `--canonical-applier` only when that integration should expose exact-hash application. +See the [quickstart](docs/NEW_PROJECT_QUICKSTART.md) for the reference-adapter and generated-client +paths. ## Documentation -- [User manual](docs/USER_MANUAL.md) — features, setup, visualization, CLI, MCP, apply, adapters, - and troubleshooting. -- [Core contract](docs/CONTRACT.md) — invariants and security boundary. -- [Milestone 0 compatibility](docs/COMPATIBILITY.md) — preserved package, CLI, MCP, adapter, - schema, changeset, rendering, and no-AST guarantees. -- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) — validation evidence, cold and warm - performance, memory, rendering and response sizes, bottlenecks, and missing coverage. -- [Milestone 0 closeout](docs/MILESTONE_0_CLOSEOUT.md) — lineage, migration, security scan, - repository state, and fresh-clone proof. -- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) — warm operation latency, structured work, - status, retrieval, and memory measurements. -- [Milestone 1 closeout](docs/MILESTONE_1_CLOSEOUT.md) — fast-core contracts, adversarial - validation, compatibility boundaries, and exact candidate evidence. -- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) — task context, generation diff, client - configuration, doctor, response-size, counter, and memory measurements. -- [Milestone 2 closeout](docs/MILESTONE_2_CLOSEOUT.md) — implemented contracts, adversarial - validation, exclusions, and exact candidate evidence. -- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) — manual, fragment, worker, portable graph, - status, equivalence, response-size, and memory measurements. -- [Milestone 3 closeout](docs/MILESTONE_3_CLOSEOUT.md) — independent projection contracts, - adversarial validation, compatibility boundaries, and exact candidate evidence. -- [MCP contract](docs/MCP_CONTRACT.md) — exact tool and process boundary. -- [Viewer manager](docs/VIEWER_MANAGER.md) — native service setup and lifecycle. -- [Adapter decision](docs/APPLICATION_DECISION.md) — why custom adapters own canonical - serialization. -- [Incremental adapter indexing](docs/INCREMENTAL_INDEXING.md) — source-scoped extraction, - invalidation, equivalence, relationship changes, and the lazy Logic boundary. -- [Project onboarding](docs/PROJECT_ONBOARDING.md) — repository assessment, safe manual - scaffolding, language frontends, source/manual integration, proof, and MCP activation. -- [Language adapter authoring](docs/ADAPTER_AUTHORING_GUIDE.md) — implementation sequence, - stable identities, overlap ownership, normalization, incremental equivalence, troubleshooting, - and the complete adapter proof matrix. +### Learn and operate + +- [New-project quickstart](docs/NEW_PROJECT_QUICKSTART.md) +- [Core concepts and authority](docs/CORE_CONCEPTS_AND_AUTHORITY.md) +- [Project descriptor](docs/PROJECT_DESCRIPTOR.md) +- [Policy precedence](docs/POLICY_PRECEDENCE.md) +- [Reference adapters](docs/REFERENCE_ADAPTERS.md) +- [Agent integration](docs/AGENT_INTEGRATION.md) +- [User manual](docs/USER_MANUAL.md) +- [Generated command reference](docs/COMMAND_REFERENCE.md) +- [Project onboarding](docs/PROJECT_ONBOARDING.md) +- [Rendering and visualization](docs/RENDERING_AND_VISUALIZATION.md) +- [Recovery and performance](docs/RECOVERY_AND_PERFORMANCE.md) +- [Security](docs/SECURITY.md) + +### Contracts and compatibility + +- [Core contract](docs/CONTRACT.md) +- [MCP contract](docs/MCP_CONTRACT.md) +- [Compatibility contract](docs/COMPATIBILITY.md) +- [Adapter authoring guide](docs/ADAPTER_AUTHORING_GUIDE.md) +- [Incremental indexing](docs/INCREMENTAL_INDEXING.md) +- [Adapter application decision](docs/APPLICATION_DECISION.md) +- [Legacy and no-AST operation](docs/LEGACY_AND_NO_AST.md) +- [Migrating from version 1](docs/MIGRATING_FROM_V1.md) +- [Viewer manager](docs/VIEWER_MANAGER.md) + +### Milestone evidence + +- [Milestone 4 baseline](docs/MILESTONE_4_BASELINE.md) +- [Milestone 4 closeout](docs/MILESTONE_4_CLOSEOUT.md) +- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) and [closeout](docs/MILESTONE_3_CLOSEOUT.md) +- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) and [closeout](docs/MILESTONE_2_CLOSEOUT.md) +- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) and [closeout](docs/MILESTONE_1_CLOSEOUT.md) +- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) and [closeout](docs/MILESTONE_0_CLOSEOUT.md) + +Historical milestone records preserve the facts and dependency observations of their frozen +candidates. Use the current guides and contracts for present behavior. ## Development -Run the complete repository-native gate: +Run the repository-native gate: ```bash make gate ``` -Focused entry points are available as `make contract`, `make test`, `make type`, -`make benchmark-smoke`, `make benchmark`, `make benchmark-m1-smoke`, and -`make benchmark-m1`. Milestone 2 adds `make benchmark-m2-smoke` and `make benchmark-m2`. -Milestone 3 adds `make accessibility`, `make benchmark-m3-smoke`, `make benchmark-m3`, and -`make benchmark-m3-full`. +Milestone 4 maintenance entry points include: -The committed 1,000-node baseline and its measurement method are under `benchmarks/`. +```bash +make adoption-m4 +make benchmark-m4-smoke +make benchmark-m4 +make benchmark-m4-full +make command-reference-check +make docs-check +``` -Pass `--diagnostics` to `docforge` or `docforge-mcp` to attach bounded request-local stage timings -and compiler-work counters. Diagnostics are disabled by default and are dropped before primary MCP -results when the configured output budget is tight. +`adoption-m4` builds and exercises a fresh base wheel without Tree-sitter packages and proves the +Python reference plus a real isolated read-only MCP retrieval. The optional-language integrations +have their own focused tests and extras. `benchmark-m4` runs the maintained full adapter workload; +the smoke target is for routine gate coverage, not final performance evidence. -See [AGENTS.md](AGENTS.md) before changing core boundaries. +Pass `--diagnostics` to `docforge` or `docforge-mcp` for bounded request-local timings and compiler +work counters. Diagnostics are disabled by default and do not displace a primary result when the +configured output budget is tight. + +Read [AGENTS.md](AGENTS.md) before changing core boundaries. diff --git a/SLICE_HISTORY.md b/SLICE_HISTORY.md index 5b4cd75..0caab02 100644 --- a/SLICE_HISTORY.md +++ b/SLICE_HISTORY.md @@ -1,5 +1,50 @@ # Completed slices +## DocForge2 Milestone 4 adapter SDK and product documentation + +### Changed + +- Added stable `docforge.adapter_sdk` authoring imports and a conformance helper for deterministic + complete and exact complete/incremental primary graph plus Logic equivalence. +- Added bounded Python, JavaScript, TypeScript, and C++ reference integrations with explicit + unsupported-fact reports and optional heavy frontends. +- Added a closed `.docforge/reference-adapter.toml` contract and fixed read-only reference MCP + server. +- Added immutable project-bound launcher declarations and deterministic Codex, Claude, and OpenClaw + configuration generation for custom adapters. +- Generated CLI and MCP references from live implementation metadata and added race-safe + publication plus repository drift checks. +- Added strict documentation-graph and fresh-wheel adoption gates and dedicated onboarding, + authority, descriptor, policy, adapter, legacy/no-AST, rendering, agent-integration, security, + recovery/performance, and v1 migration guides. + +### Verification + +- The executable candidate passed 347 tests and 402 subtests, 142 contract tests and 268 subtests, + three accessibility flows, lint, strict types, compilation, dependencies, builds, adoption, and + every maintained smoke benchmark. +- The offline base wheel contained no Tree-sitter distribution and ran a real Python reference + build/check and isolated 21-tool MCP retrieval flow. Missing C++ support failed with the exact + `docforge[cpp]` remediation. +- The clean 334-source benchmark produced 1,002 nodes and exact full/incremental graph and Logic + evidence, warm zero Python parsing/extraction, and exact corrupt-cache and corrupt-index + recovery below every latency, memory, and response gate. +- Gitleaks found no findings in the Milestone 4 commit range or candidate tree. + +### Limits + +- The reference integrations publish syntax and local static relationships. They do not claim + compiler-resolved calls, types, inheritance, macros, runtime behavior, or semantic ownership. +- C++ compilation-database commands are inert and never executed. +- C++ manifest include discovery uses Tree-sitter and makes no zero-warm-parser claim. +- No WorldForge, ScrapeStation, legacy repository, production binding, language distribution, + self-hosting, tag, or release was changed. + +### Next gate + +Milestone 5 remains directional. Create a new active-slice contract before stabilization, +versioning, tagging, or publication. + ## DocForge2 Milestone 3 independent projections ### Changed diff --git a/benchmarks/README.md b/benchmarks/README.md index 2141f1a..acfedd2 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -42,6 +42,14 @@ make benchmark-m3 make benchmark-m3-full ``` +Run the Milestone 4 adapter SDK smoke, maintained full, and fresh-wheel adoption gates: + +```bash +make benchmark-m4-smoke +make benchmark-m4-full +make adoption-m4 +``` + The benchmark creates canonical sources, derived state, changesets, rendered output, and caches only in a disposable temporary directory. It does not read another project, self-host DocForge, or mutate repository content. @@ -71,6 +79,13 @@ sweeps, add/change/delete/reorder equivalence, portable publication, receipt-onl memory, detached worker peak memory, and response size. Its interpretation is in [`docs/MILESTONE_3_BASELINE.md`](../docs/MILESTONE_3_BASELINE.md). +`milestone4-2026-07-29.json` is the clean-tree adapter SDK and reference-Python baseline captured +from commit `95271dcf2e48045b9d3aed9b9ea09c7fc155692c`. It measures cold and warm +incremental builds, complete/incremental graph-plus-Logic equivalence, corrupt extraction-cache +recovery, corrupt-index recovery, parser and extraction counters, traced and process memory, and +bounded response size. Its interpretation is in +[`docs/MILESTONE_4_BASELINE.md`](../docs/MILESTONE_4_BASELINE.md). + The generic fixtures expose whole-source and projection scaling. They do not replace incremental adapter equivalence tests. Milestone 3's portable fixture contains 1,000 Nodes/Flow/Web nodes and 999 edges; portable version 1 deliberately excludes Logic. @@ -89,4 +104,5 @@ backlinks, both traversal directions, paged context, render receipt states, and freshness. The reported p95 uses the nearest-rank method; with ten samples it is the maximum. `process_peak_rss_kib` is the cumulative main-process `RUSAGE_SELF` high-water mark and excludes the detached viewer worker. It is diagnostic and not operation-local. Milestone 3 memory gates use -per-operation `tracemalloc` peaks and detached worker receipt peaks instead. +per-operation `tracemalloc` peaks and detached worker receipt peaks instead. Milestone 4 uses +per-operation `tracemalloc` peaks and a cumulative benchmark-process high-water gate. diff --git a/benchmarks/milestone4-2026-07-29.json b/benchmarks/milestone4-2026-07-29.json new file mode 100644 index 0000000..2f993d3 --- /dev/null +++ b/benchmarks/milestone4-2026-07-29.json @@ -0,0 +1,205 @@ +{ + "benchmark": "docforge2_milestone4", + "environment": { + "implementation": "CPython", + "machine": "x86_64", + "platform": "Linux-7.1.3-200.nobara.fc44.x86_64-x86_64-with-glibc2.43", + "python": "3.14.6" + }, + "evidence": { + "cache_recovery_exact": true, + "complete_assembly_hash": "2888182ab765fbffe3ba873c1613345640e8e6d89be74ddfca7452c0a5056345", + "edge_count": 1001, + "edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35", + "full_incremental_graph_and_logic_exact": true, + "index_recovery_exact": true, + "logic_edge_count": 2338, + "logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c", + "logic_node_count": 2338, + "logic_projection_count": 334, + "node_count": 1002, + "node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f", + "source_count": 334, + "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", + "warm_zero_ast_parse": true, + "warm_zero_source_extraction": true + }, + "evidence_sha256": "4a8db461a311b5df4abd5aa00063e9a347d8b9dba19b30e35684f561d5271549", + "fixture": { + "expected_node_count": 1002, + "kind": "synthetic_python_reference_adapter", + "nodes_per_source": 3, + "source_count": 334 + }, + "memory": { + "process_peak_bytes": 78798848, + "process_peak_limit_bytes": 536870912 + }, + "method": { + "ast_instrumentation": "temporary counter around stdlib ast.parse", + "clock": "time.perf_counter_ns", + "extraction_instrumentation": "reference adapter extract_source entry counter", + "full_mode_source_requirement": 334, + "in_process_peak_memory": "tracemalloc per measured invocation", + "process_peak_memory": "resource.getrusage(RUSAGE_SELF).ru_maxrss", + "response_size": "UTF-8 bytes of compact sorted JSON", + "samples": 3, + "threshold_basis": "Regression tripwires leave several times the Milestone 3 1,000-node allowances for AST, Logic, extraction-cache, and SQLite work." + }, + "mode": "full", + "operations": { + "cold_incremental_build": { + "max_ms": 1599.76, + "maximum_response_bytes": 14580, + "maximum_traced_peak_bytes": 68573540, + "median_ms": 1599.76, + "min_ms": 1599.76, + "p95_limit_ms": 30000.0, + "p95_ms": 1599.76, + "response_limit_bytes": 262144, + "samples": 1, + "stable_result": { + "ast_parse_calls": 668, + "cache_hits": 0, + "deleted_sources": 0, + "edge_count": 1001, + "edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35", + "extraction_calls": 334, + "invalidated_sources": 334, + "logic_edge_count": 2338, + "logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c", + "logic_node_count": 2338, + "logic_projection_count": 334, + "node_count": 1002, + "node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f", + "reparsed_sources": 334, + "revision": "30ef23aa4106", + "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", + "status": "ok", + "total_sources": 334 + }, + "traced_peak_limit_bytes": 268435456 + }, + "corrupt_extraction_cache_recovery": { + "max_ms": 1695.406, + "maximum_response_bytes": 14580, + "maximum_traced_peak_bytes": 69776923, + "median_ms": 1695.406, + "min_ms": 1695.406, + "p95_limit_ms": 30000.0, + "p95_ms": 1695.406, + "response_limit_bytes": 262144, + "samples": 1, + "stable_result": { + "ast_parse_calls": 668, + "cache_hits": 0, + "deleted_sources": 0, + "edge_count": 1001, + "edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35", + "extraction_calls": 334, + "invalidated_sources": 334, + "logic_edge_count": 2338, + "logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c", + "logic_node_count": 2338, + "logic_projection_count": 334, + "node_count": 1002, + "node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f", + "reparsed_sources": 334, + "revision": "30ef23aa4106", + "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", + "status": "ok", + "total_sources": 334 + }, + "traced_peak_limit_bytes": 268435456 + }, + "corrupt_index_recovery": { + "max_ms": 1707.879, + "maximum_response_bytes": 14779, + "maximum_traced_peak_bytes": 68561924, + "median_ms": 1707.879, + "min_ms": 1707.879, + "p95_limit_ms": 30000.0, + "p95_ms": 1707.879, + "response_limit_bytes": 262144, + "samples": 1, + "stable_result": { + "action": "rebuilt", + "ast_parse_calls": 0, + "cache_hits": 334, + "edge_count": 1001, + "edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35", + "extraction_calls": 0, + "initial_error_code": "invalid_index", + "logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c", + "logic_projection_count": 334, + "node_count": 1002, + "node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f", + "reparsed_sources": 0, + "revision": "30ef23aa4106", + "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", + "status": "ok" + }, + "traced_peak_limit_bytes": 268435456 + }, + "full_incremental_equivalence": { + "max_ms": 1897.919, + "maximum_response_bytes": 230, + "maximum_traced_peak_bytes": 64305919, + "median_ms": 1897.919, + "min_ms": 1897.919, + "p95_limit_ms": 30000.0, + "p95_ms": 1897.919, + "response_limit_bytes": 262144, + "samples": 1, + "stable_result": { + "ast_parse_calls": 1336, + "edge_count": 1001, + "extraction_calls": 668, + "logic_projection_count": 334, + "node_count": 1002, + "project_id": "milestone4-python-benchmark", + "revision": "30ef23aa4106", + "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", + "status": "ok" + }, + "traced_peak_limit_bytes": 268435456 + }, + "warm_incremental_build": { + "max_ms": 1265.387, + "maximum_response_bytes": 14578, + "maximum_traced_peak_bytes": 69997166, + "median_ms": 1244.427, + "min_ms": 1239.941, + "p95_limit_ms": 20000.0, + "p95_ms": 1265.387, + "response_limit_bytes": 262144, + "samples": 3, + "stable_result": { + "ast_parse_calls": 0, + "cache_hits": 334, + "deleted_sources": 0, + "edge_count": 1001, + "edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35", + "extraction_calls": 0, + "invalidated_sources": 0, + "logic_edge_count": 2338, + "logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c", + "logic_node_count": 2338, + "logic_projection_count": 334, + "node_count": 1002, + "node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f", + "reparsed_sources": 0, + "revision": "30ef23aa4106", + "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", + "status": "ok", + "total_sources": 334 + }, + "traced_peak_limit_bytes": 268435456 + } + }, + "schema_version": 1, + "source": { + "dirty": false, + "revision": "95271dcf2e48045b9d3aed9b9ea09c7fc155692c" + } +} diff --git a/docs/ADAPTER_AUTHORING_GUIDE.md b/docs/ADAPTER_AUTHORING_GUIDE.md index f77c3fe..df37df7 100644 --- a/docs/ADAPTER_AUTHORING_GUIDE.md +++ b/docs/ADAPTER_AUTHORING_GUIDE.md @@ -8,6 +8,30 @@ Use this guide after the repository assessment in [Project Onboarding](PROJECT_ONBOARDING.md). The onboarding checklist decides whether an adapter is needed. This guide defines how to build and prove one. +## Public authoring surface + +Adapter authors should import typed contracts, validation helpers, graph models, and the +conformance helper from `docforge.adapter_sdk`: + +```python +from docforge.adapter_sdk import ( + AdapterAssembly, + AdapterImplementation, + AdapterManifest, + AdapterProject, + AdapterProjectSettings, + AdapterProjection, + AdapterSource, + AdapterSourceProjection, + LogicProjection, + verify_adapter_conformance, +) +``` + +Language-specific modules under `docforge.adapters` are repository reference implementations, not +the general authoring namespace. Their exact, deliberately narrow behavior is documented in +[Reference Adapters](REFERENCE_ADAPTERS.md). + ## Required outcome A production adapter must provide one reproducible public graph from authoritative project @@ -51,11 +75,39 @@ class MyAdapter: manifest: AdapterManifest, contributions: tuple[AdapterSourceProjection, ...], ) -> AdapterAssembly: ... + def load_complete_assembly(self) -> AdapterAssembly: ... def load_projection(self) -> AdapterProjection: ... ``` `assemble_projection()` is optional only when extraction units already have disjoint ownership. -`load_projection()` is always required. It is the clean rebuild and equivalence oracle. +`load_projection()` is always required. It is the clean primary-graph compatibility oracle. +`load_complete_assembly()` supplies the cache-independent complete graph-plus-Logic oracle. It is +required when incremental contributions publish Logic; without it, DocForge cannot prove Logic +parity. + +## What the conformance helper proves + +Run the public helper against a fresh confined cache root: + +```python +report = verify_adapter_conformance( + MyAdapter(project_root), + cache_root=project_root / ".docforge" / "cache" / "adapter-conformance", +) +``` + +`verify_adapter_conformance()` proves: + +- two repeated complete assemblies are exactly deterministic, including Logic; +- when `load_complete_assembly()` exists, its primary graph exactly matches + `load_projection()`; and +- when the incremental methods exist, the assembled incremental graph and Logic exactly match the + independent complete oracle. + +The report records stable identity, counts, and an assembly hash. This helper does not by itself +prove process confinement, implementation restart behavior, corrupt-cache recovery, warm zero +parsing, retrieval, or every case in the proof matrix below. Keep those as separate focused and +integration tests. ## Step 1: define authority before parsing @@ -324,6 +376,25 @@ The manifest remains a current source snapshot, not a Git-index snapshot. A Git- omit a deleted source whether its deletion is unstaged or staged. Staging is never a required DocForge synchronization step. +### Current aggregate bounds + +DocForge limits one extraction-cache generation to 10,000 source contributions and 64,000,000 +encoded bytes. Malformed, incompatible, missing, oversized, or unsafe cache data is treated as a +cache miss. + +The assembled graph is bounded by the effective project `Limits.max_nodes`. When adapter settings +do not override limits, DocForge selects at least 10,000 nodes and raises that bound to the +manifest's `estimated_nodes` when larger. Aggregate assembly ceilings are: + +- nodes: `max_nodes`; +- relationships: `max_nodes * 32`; +- Logic nodes across all functions: `max_nodes * 32`; and +- Logic edges across all functions: `max_nodes * 64`. + +Adapters should set realistic limits and fail before retaining unbounded frontend evidence. The +cache and assembly bounds do not replace each adapter's own bounded source, command, parser, or +per-function limits. + ## Step 9: keep the complete path independent The full rebuild must not read the incremental extraction cache. Otherwise equivalence compares @@ -470,7 +541,8 @@ For every such change: - [ ] Two independent complete builds match exactly. - [ ] Incremental extraction uses authoritative dependencies. - [ ] The complete oracle is independent of the cache. -- [ ] Complete and incremental public projections match exactly. +- [ ] Complete and incremental public graph and Logic projections match exactly. +- [ ] `verify_adapter_conformance()` passes, and the separate proof-matrix cases also pass. - [ ] Corrupt, missing, and interrupted cache cases fail safely. - [ ] Session composition and family isolation are proven. - [ ] Viewer, query, context, and Logic retrieval are proven. diff --git a/docs/AGENT_INTEGRATION.md b/docs/AGENT_INTEGRATION.md new file mode 100644 index 0000000..0f5ae3b --- /dev/null +++ b/docs/AGENT_INTEGRATION.md @@ -0,0 +1,105 @@ +# Agent integration + +DocForge generates deterministic, project-bound MCP client fragments for Codex, Claude, and +OpenClaw. Generic projects and custom adapters use different configuration APIs, but both produce +fixed standard-input/output bindings with empty generated environments and explicit capability +policy. + +## Fixed reference binding + +Configure one of the in-repository adapters with +`.docforge/reference-adapter.toml` as described in +[Reference Adapters](REFERENCE_ADAPTERS.md), then start: + +```bash +python -I -m docforge.reference_mcp \ + --project-root /absolute/path/to/project \ + --capability-mode read +``` + +`docforge.reference_mcp` is a fixed trusted module in the installed DocForge distribution. It +selects a provider only from the validated reference configuration and reports binding metadata +containing: + +- `server_module = "docforge.reference_mcp"`; +- `adapter_mode = "reference"`; +- the selected `reference_language`; and +- the exact `reference_config_hash`. + +The reference binding registers exactly the 21 read tools listed in +[MCP Boundary](MCP_CONTRACT.md#read-tools). It has no proposal or canonical-application surface. + +## Generate a reference client fragment + +The public launcher and generator APIs are: + +```python +from pathlib import Path + +from docforge.adapter_launcher import AdapterLauncherV1 +from docforge.client_config import generate_adapter_client_configuration +from docforge.reference_mcp import REFERENCE_MCP_MODULE, create_reference_project + +root = Path("/absolute/path/to/project") +project = create_reference_project(root) +launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE) + +plan = generate_adapter_client_configuration( + project, + launcher, + "codex", # "codex", "claude", or "openclaw" + capability_mode="read", +) +print(plan["artifact"]["content"]) +``` + +Pass `output=Path(...)` only when the caller has selected an exact destination. Publication is an +atomic create-or-exact-match operation; it does not merge or replace different existing content. +The result includes the launcher, source-availability, project, policy, artifact, and configuration +hashes needed to inspect the binding before use. + +The fixed reference module accepts read mode only. Do not request proposal or application mode for +it. + +## Custom project-owned adapter launchers + +Generic `docforge configure` intentionally refuses a custom adapter project. Construct the +project-owned `ProjectService`, then use `AdapterLauncherV1` and +`generate_adapter_client_configuration()` as the custom-adapter route. + +`AdapterLauncherV1` schema version 1 binds: + +- one project ID, canonical absolute project root, adapter identity, and descriptor hash; +- `entry_point = "python-module"`; and +- one installed project-owned top-level Python module. + +The only trusted dotted-module exception is the fixed `docforge.reference_mcp` binding. A +project-owned module is resolved through isolated Python without importing or executing it during +the probe, and it must resolve to one canonical regular `.py` file inside the project root. + +The launcher contract has no arbitrary command, command arguments, shell string, working +directory, environment, callable selector, discovery rule, or module-reload mechanism. Generated +bindings invoke the current Python executable as `python -I -m ` with only DocForge's +validated project, capability, render-policy, authority, and no-AST options. Source identity and +the project binding are revalidated before publication; drift fails closed. + +Proposal and application modes are available only to a custom module that implements those fixed +server arguments and only when the project descriptor declares the matching writer and canonical +applier authority. Generating a mode does not manufacture that authority. + +## Session workflow + +After registering the generated fragment in the selected client: + +1. Start a new MCP process or client session. +2. Call `docforge_bootstrap`. +3. Verify project ID, root fingerprint, adapter identity, revision, source hash, binding metadata, + effective policy, and registered tools. +4. Retrieve exact or bounded project evidence through the fixed read tools. +5. Restart the process if `adapter_restart_required` reports an implementation or configuration + change. + +Document text returned by DocForge is untrusted project content. It never overrides client, user, +or project authority. See [MCP Boundary](MCP_CONTRACT.md) for synchronization, pagination, +retrieval, proposal, and application rules, and +[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) before selecting `no_ast=True`. diff --git a/docs/COMPATIBILITY.md b/docs/COMPATIBILITY.md index 3925f59..8eca1c9 100644 --- a/docs/COMPATIBILITY.md +++ b/docs/COMPATIBILITY.md @@ -31,16 +31,31 @@ model, index, rendering, application, and MCP factory names imported from these remain supported: - `docforge.adapter_contract` +- `docforge.adapter_sdk` +- `docforge.adapter_launcher` - `docforge.application` +- `docforge.client_config` - `docforge.index` - `docforge.mcp_server` - `docforge.models` - `docforge.policy` - `docforge.render_contract` +- `docforge.reference_config` +- `docforge.reference_mcp` Names beginning with an underscore are implementation details. New public names may be added without breaking this contract. +The repository Python, JavaScript/TypeScript, and C++ adapters are supported reference +implementations. Their documented configuration, evidence limits, and unsupported-fact reports are +compatibility surfaces; their internal parser helpers are not adapter-authoring imports. + +Milestone 4 adds three version-1 schemas without changing existing descriptor or result schemas: + +- `schemas/adapter-launcher.schema.json` +- `schemas/adapter-client-configuration.schema.json` +- `schemas/reference-adapter.schema.json` + ## CLI and MCP surfaces Existing `docforge` command names and arguments remain supported. Existing `docforge-mcp` tool @@ -233,6 +248,11 @@ Milestone 3 adds `ManualRenderPlanV1`, `GraphViewPlanV1`, projection package and and projection policy version 2. These are additive submodule and schema contracts. They do not change the legacy task-context, adapter, changeset, or effective-policy contracts described above. +Milestone 4 adds `docforge.adapter_sdk`, the fixed reference configuration and read-only MCP +binding, immutable adapter launchers, and generated adapter client fragments. A legacy adapter with +only `load_projection()` remains first-class and need not adopt incremental extraction, Logic, a +reference configuration, or launcher metadata. + ## Safety boundary DocForge remains bound to one explicit project root. It rejects absolute paths, root escapes, and @@ -245,8 +265,9 @@ publication, or project switching. Milestone 0 records rather than redesigns these areas: - Generic warm reads still repeat whole-project discovery, parsing, and validation. -- Tree-sitter and the JavaScript and C++ grammars remain mandatory installation dependencies even - when their runtime modules are unused. +- The base wheel intentionally omits Tree-sitter. JavaScript, TypeScript, and C++ syntax evidence + requires the matching `docforge[javascript]`, `docforge[typescript]`, or `docforge[cpp]` extra. + Python reference evidence uses the standard library and remains available in the base wheel. - Several version strings and defaults remain duplicated. - One individually oversized context entry is represented as explicit bounded omission evidence; callers use targeted retrieval for that node. diff --git a/docs/CONTRACT.md b/docs/CONTRACT.md index cfbaf30..177a62d 100644 --- a/docs/CONTRACT.md +++ b/docs/CONTRACT.md @@ -29,6 +29,9 @@ commit when Git is available; it cannot change repository state. - Projection package: `schemas/projection-package.schema.json`, version 1. - Projection receipt: `schemas/projection-receipt.schema.json`, version 1. - Independent projection policy: `schemas/projection-policy.schema.json`, version 2. +- Adapter launcher: `schemas/adapter-launcher.schema.json`, version 1. +- Adapter client configuration: `schemas/adapter-client-configuration.schema.json`, version 1. +- Reference adapter configuration: `schemas/reference-adapter.schema.json`, version 1. - Index schema: version 3, disposable and reproducible. - Index attestation: schema version 1, disposable and reproducible. - Core, CLI, and MCP server: version 1.3.0.dev0. @@ -103,6 +106,42 @@ complete projection, opens SQLite, starts MCP, executes the configured command, builds, renders, starts a viewer, or writes configuration. Unprovable client behavior is a warning, not an invented success. +## Public adapter SDK and reference binding + +`docforge.adapter_sdk` is the stable adapter-authoring import boundary. It exposes the typed +projection, manifest, source contribution, complete assembly, project wrapper, graph model, and +conformance contracts needed by an adapter without requiring authors to import core implementation +modules. + +Complete evidence includes the primary graph and function Logic. An incremental adapter that +publishes Logic must implement `load_complete_assembly()` as an independent complete oracle. +`verify_adapter_conformance()` proves repeated complete determinism, equality between the complete +assembly and `load_projection()`, and exact complete/incremental graph-plus-Logic parity. Separate +tests remain responsible for confinement, restart behavior, no-AST behavior, cache recovery, and +retrieval. + +Adapter assemblies are bounded before publication. Primary nodes use the descriptor `max_nodes` +limit. Primary edges, Logic nodes, and Logic edges use fixed deterministic multipliers over that +limit. Version-1 extraction caches are regular-file-only, bounded to 10,000 sources and +64,000,000 bytes, and are treated as misses when corrupt, oversized, foreign, or incompatible. + +`.docforge/reference-adapter.toml` is a closed version-1 selection among `python`, `javascript`, +`typescript`, and `cpp`. It declares one project identity and explicit non-overlapping source +roots. C++ additionally requires a confined `compile_commands.json`. It cannot declare a command, +module, environment, writer, applier, or remote endpoint. + +`python -m docforge.reference_mcp --project-root ROOT` constructs only the selected fixed +repository reference adapter and exposes the read surface. It never registers proposal or +application tools. + +`AdapterLauncherV1` is an immutable project-bound Python-module declaration. It accepts no +arbitrary command, arguments, working directory, environment, discovery, callable selector, or +module reload. Custom launchers resolve one installed top-level module through isolated Python and +require its origin inside the project root. The fixed `docforge.reference_mcp` module is the only +trusted dotted exception. `generate_adapter_client_configuration()` binds generated Codex, +Claude, and OpenClaw fragments to that launcher, current source availability, effective policy, +descriptor, interpreter, and exact artifact bytes. + ## Isolated proposal model Create, update, move, and delete are ordered node operations inside an isolated changeset. Every diff --git a/docs/CORE_CONCEPTS_AND_AUTHORITY.md b/docs/CORE_CONCEPTS_AND_AUTHORITY.md new file mode 100644 index 0000000..4fc1c9d --- /dev/null +++ b/docs/CORE_CONCEPTS_AND_AUTHORITY.md @@ -0,0 +1,165 @@ +# Core concepts and authority + +DocForge is a project-bound knowledge compiler. It turns explicit canonical project facts into +validated graphs and bounded derived views without transferring authority to the index, an agent, +or a renderer. + +## One project, one explicit root + +Every operation is bound to one canonical real project directory. Paths in descriptors and adapter +configuration are project-relative and confined beneath that root. A CLI or MCP process does not +discover or switch projects after startup. + +The project root determines: + +- which descriptor and canonical sources may be read; +- where derived cache and changeset roots may exist; +- which project identity, revision, source hash, and generation appear in results; +- which writer, applier, rendering, and viewer policies can be selected. + +Generated client fragments preserve that binding. They are machine-local configuration +projections, not portable project authority. + +## Canonical facts and derived evidence + +Canonical inputs own the facts: + +- generic Markdown or TOML node sources; +- authority files named by a generic descriptor; +- an adapter's declared canonical sources and implementation boundary; +- `.docforge/project.toml` for a generic project; +- `.docforge/reference-adapter.toml` for a fixed reference integration. + +Everything DocForge builds from those inputs is derived: + +- SQLite indexes, attestations, generation receipts, and generation diffs; +- incremental extraction caches; +- task-context capsules and query responses; +- changeset previews; +- render plans, immutable packages, fragments, artifacts, and receipts; +- portable graph publications and live-viewer processes; +- generated Codex, Claude, and OpenClaw fragments. + +Derived state may be discarded and rebuilt. A derived artifact can prove what it was bound to, but +it cannot override current canonical content. + +## Nodes, relationships, and Logic + +The primary graph contains nodes and directed relationships. + +A node has a stable project-wide ID, title, family, authority, status, tags, summary, content, +source identity, and content hash. A generic Markdown file contains one node beneath a TOML +metadata block; a generic TOML source may contain multiple nodes. Adapter nodes use the same public +graph contract. + +A relationship is an exact `(source, relation, target)` triple. The project descriptor defines the +allowed relation names. DocForge gives `depends_on` special acyclic validation, but it does not +invent domain meaning for a project's other relation names. Retrieval recognizes only a versioned +alias set for task planning and reports unknown allowed relations as `unclassified`. + +Logic is deliberately separate. It is a lazy function-scoped control-flow projection owned by one +primary node. Decisions, actions, loops, convergence points, returns, and exceptions connect +through explicit branch edges. Logic does not add statement-level nodes to ordinary Nodes, Flow, +Web, search, or generation-diff results. Python, JavaScript, TypeScript, and C++ integrations may +publish Logic when their adapter contract supports it. + +## Authority, status, family, and tags + +These fields answer different questions: + +- `authority` describes the role of the content. The generic vocabulary is `authoritative`, + `approved_plan`, `derived`, `proposal`, and `historical`. +- `status` is project-defined lifecycle state such as `current`, `active`, or `verified`. +- `family` is a project-defined content grouping used for filtering, profiles, rendering, and + writer permissions. +- `tags` are exact project labels for retrieval and presentation. + +An `authoritative` node can still become stale; authority is not a freshness claim. A `derived` +node is still canonical if it is stored in a declared canonical source; the label describes its +role, not whether DocForge may silently regenerate it. Status and authority never grant an MCP +writer permission. + +## Validation, synchronization, and generations + +Validation loads the complete canonical graph and rejects unsafe paths, invalid source formats, +duplicate IDs, unresolved relationships, prohibited cycles, violated project limits, and +adapter-specific contract failures. + +The disposable index is published atomically only after the complete graph and SQLite integrity +checks pass. Its attestation binds the whole index file. A successful replacement is the derived +publication commit point; later receipt-writing trouble is reported as degraded evidence rather +than as a false claim that the replacement failed. + +MCP operations automatically synchronize derived state under a project lock before normal work. +A graph generation identifies one validated indexed snapshot. Generation-pinned retrieval and +rendering do not reopen mutable sources behind an older snapshot. + +The latest generation diff is one bounded primary-graph transition, not a history database. It +contains no Logic details or source text. + +## Complete and incremental adapters + +`load_projection()` is the compatibility baseline and clean graph oracle. An incremental adapter +adds: + +- `load_manifest()` for cheap project identity, source inventory, fingerprints, and dependencies; +- `extract_source()` for one cacheable source contribution; +- optionally `assemble_projection()` to normalize overlapping contributions. + +When incremental contributions contain Logic, `load_complete_assembly()` supplies a +cache-independent complete graph-plus-Logic oracle. Warm cache behavior is an optimization, never a +different authority path. Corrupt or incompatible extraction caches are treated as misses, and a +clean complete build remains the equivalence and recovery boundary. + +The public types, validators, and conformance helper are exported from `docforge.adapter_sdk`. See +the [Adapter authoring guide](ADAPTER_AUTHORING_GUIDE.md) and [Incremental +indexing](INCREMENTAL_INDEXING.md). + +## Proposals are not canonical changes + +A changeset is an isolated, ordered proposal over an exact canonical base. Each operation names +preconditions, and the final changeset has a content-derived hash. Validation projects the complete +resulting graph before application. + +Canonical application requires: + +1. a writer declared in the descriptor; +2. a process started with the matching proposal and application authority; +3. one explicit final changeset hash; +4. unchanged source, relationship, permission, and graph preconditions. + +Application does not perform Git mutation, build, deployment, or publication. Until exact-hash +application succeeds, canonical project files remain unchanged. + +## Three independent output projections + +Manual rendering, portable graph publication, and the live viewer are separate: + +- a manual is a declared derived HTML view over selected nodes; +- a portable graph is a content-addressed static Nodes, Flow, or Web artifact; +- the live viewer is a managed loopback process pinned to one validated index generation and can + request lazy Logic. + +Their policies compose independently. Disabling one does not transfer its authority to another. +None is canonical documentation or a retrieval authority. See [Rendering and +visualization](RENDERING_AND_VISUALIZATION.md). + +## Binding policy is not project truth + +Capability mode, no-AST preservation, diagnostics, and projection modes describe one running +process or generated client binding. They do not rewrite the descriptor or canonical graph. + +`--no-ast` forbids AST-family adapter evolution and Logic publication/retrieval for that binding. +It does not inspect parser implementation, sandbox the filesystem, or convert an existing +AST/Tree-sitter adapter into a no-AST adapter. Read [Policy precedence](POLICY_PRECEDENCE.md) and +[Legacy and no-AST operation](LEGACY_AND_NO_AST.md). + +## Trust the narrowest evidence + +DocForge reports stable IDs, hashes, generations, omissions, truncation, and provenance limits so a +consumer can distinguish proof from inference. A source inventory is not a semantic graph; a +syntax-level relationship is not compiler resolution; a configuration doctor is not a connection +test; a viewer snapshot is not continuous monitoring; and a benchmark is not a release. + +Continue with the [Project descriptor](PROJECT_DESCRIPTOR.md), [New-project +quickstart](NEW_PROJECT_QUICKSTART.md), or [Core contract](CONTRACT.md). diff --git a/docs/INCREMENTAL_INDEXING.md b/docs/INCREMENTAL_INDEXING.md index 4abb941..8af16da 100644 --- a/docs/INCREMENTAL_INDEXING.md +++ b/docs/INCREMENTAL_INDEXING.md @@ -4,6 +4,10 @@ DocForge Release 1 adapters return one complete immutable projection. That contr supported. The incremental compiler adds an opt-in source-scoped contract that avoids reparsing unchanged files while preserving the same validated, atomically published graph. +Import these contracts from the public `docforge.adapter_sdk` facade. See +[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for the preserved one-method contract and +[Reference Adapters](REFERENCE_ADAPTERS.md) for the four maintained implementations. + ## Release 1 compatibility The incremental interface is additive: @@ -34,18 +38,26 @@ Incremental indexing is an extraction optimization. It does not weaken publicati An interrupted extraction never replaces the last validated SQLite index. A malformed, incompatible, or missing cache is a cache miss, not a partial graph. +One cache generation is bounded to 10,000 source contributions and 64,000,000 encoded bytes. +Aggregate graph and Logic assembly limits are described in the +[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md#current-aggregate-bounds). + ## Adapter contract -An incremental loader implements all three methods: +An incremental loader implements the first, second, and fourth methods. It implements +`load_complete_assembly()` as well when it publishes Logic: ```python class MyAdapter: def load_manifest(self) -> AdapterManifest: ... def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ... + def load_complete_assembly(self) -> AdapterAssembly: ... def load_projection(self) -> AdapterProjection: ... ``` -`load_projection()` remains the deterministic full-rebuild fallback and equivalence oracle. +`load_projection()` remains the deterministic full-rebuild primary-graph oracle. An incremental +adapter that publishes Logic must additionally implement `load_complete_assembly()` as the +cache-independent complete graph-plus-Logic oracle. Each `AdapterSource` declares: @@ -137,7 +149,8 @@ fresh process must import and validate the current adapter. ``` Adapters can call `AdapterProject.verify_incremental_equivalence()` in release and contract tests. -The check compares project identity, revision, source hash, nodes, and relationships against +The check compares project identity, revision, source hash, nodes, relationships, and Logic against +the independent complete assembly. It also requires the complete assembly's primary graph to match `load_projection()`. ## Manual changes and relationships @@ -170,12 +183,29 @@ logic empty until they implement a language analyzer. This boundary prevents thousands of boolean expressions and basic blocks from polluting Nodes, Flow, Web, ordinary search, or architectural traversal. The Logic tab and `docforge_get_logic` request one function-scoped projection on demand. The built-in analyzers cover Python, -JavaScript, and C++. Python uses the standard-library AST. JavaScript and C++ share pinned -Tree-sitter infrastructure with thin language-aware control-flow profiles. Parsers run only while -extracting a changed source contribution; ordinary graph reads do not load or execute them. A -grammar alone supplies syntax, not control-flow meaning, so each new language still needs a small -semantic profile for its branch, loop, case, exception, and termination constructs. All analyzers -report possible static paths; they do not claim runtime branch outcomes. +JavaScript, TypeScript, and C++. Python uses the standard-library AST. JavaScript, TypeScript, and +C++ use distinct optional Tree-sitter grammars with thin language-aware control-flow profiles. +Ordinary graph reads use stored projections and do not load or execute these parsers. A grammar +alone supplies syntax, not control-flow meaning, so each new language still needs a profile for +its branch, loop, case, exception, and termination constructs. All analyzers report possible +static paths; they do not claim runtime branch outcomes. + +## Manifest and warm-parser scope + +Parser work is language-specific and must be measured at the correct boundary: + +- Python manifest construction fingerprints source and tokenizes local imports without calling + `ast.parse`. +- JavaScript and TypeScript manifest construction lexes static relative module specifiers without + invoking their distinct Tree-sitter extraction parsers. Focused tests prove this behavior on an + unchanged warm build. +- The C++ reference manifest parses inventoried sources with `tree-sitter-cpp` to discover quoted + include dependencies. A warm C++ extraction-cache hit is not a zero-parser claim. + +The maintained Python benchmark instruments the unchanged warm path and requires zero +`ast.parse` calls and zero `extract_source` calls. That exact zero-parser benchmark claim is Python +only. JavaScript and TypeScript retain focused parser-free-manifest tests; C++ deliberately does +not. ## Full rebuilds diff --git a/docs/LEGACY_AND_NO_AST.md b/docs/LEGACY_AND_NO_AST.md new file mode 100644 index 0000000..c1125c5 --- /dev/null +++ b/docs/LEGACY_AND_NO_AST.md @@ -0,0 +1,80 @@ +# Legacy adapters and no-AST policy + +DocForge preserves the original one-method adapter contract while offering incremental extraction, +complete graph-plus-Logic assemblies, and an independently selectable no-AST MCP policy. These are +separate compatibility boundaries. + +## One-method adapters remain valid + +An existing adapter that implements only: + +```python +def load_projection(self) -> AdapterProjection: ... +``` + +continues to work. It receives the same projection validation, indexing, querying, visualization, +and MCP behavior as before. It does not receive extraction-cache speedups and publishes no Logic +through the complete assembly contract. + +Incremental adoption is additive. Implement `load_manifest()` and `extract_source()` while keeping +`load_projection()` as the independent complete graph oracle. If incremental contributions publish +Logic, also implement `load_complete_assembly()` so complete and incremental graph-plus-Logic +output can be compared exactly. The public authoring surface is `docforge.adapter_sdk`; see the +[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md). + +## What no-AST means + +`--no-ast` is an immutable policy on one MCP process binding. It preserves the configured adapter +strategy but forbids AST, Tree-sitter, compiler-AST, and function-Logic evolution for that binding. +It also rejects nonempty Logic before index publication and rejects a pre-existing index that +contains Logic before reads or live visualization. + +Start the generic server with: + +```bash +docforge-mcp --project-root /absolute/path/to/project --no-ast +``` + +Project-owned integrations select the same policy with +`create_project_server(..., no_ast=True)` or `create_read_only_server(..., no_ast=True)`. +`docforge_bootstrap` and `docforge_get_contract` report the effective policy, and +`docforge_get_logic` returns `adapter_policy_forbids_logic`. + +Changing this policy requires a new process. It does not rewrite or reload an adapter in place. + +## What no-AST does not mean + +No-AST is not: + +- an inspection mechanism that proves which parser an arbitrary adapter uses; +- a filesystem, Python-import, or process sandbox; +- a promise that the adapter performs no non-AST fingerprinting, dependency discovery, caching, or + complete-projection loading; or +- permission to change project files, adapter code, or the configured capability surface. + +Repository permissions and project instructions remain responsible for processes that have direct +filesystem access. The MCP boundary still excludes shell execution, arbitrary file operations, +Git mutation, builds, deployment, publication, project switching, and cross-project retrieval. + +## Reference adapters are not no-AST adapters + +The built-in Python reference extracts syntax and Logic with the standard-library AST. The +JavaScript, TypeScript, and C++ references extract syntax and Logic with their optional +Tree-sitter grammars. Their normal projections contain Logic, so they must not be presented or +configured as no-AST adapters. + +The narrower performance statements in [Reference Adapters](REFERENCE_ADAPTERS.md) concern +specific manifest and warm-cache paths. A parser-free manifest does not turn an adapter that +publishes AST-derived Logic into a no-AST adapter. + +## Migration choices + +Keep a one-method adapter when its complete rebuild is acceptable and no function Logic is needed. +Adopt incremental extraction when measured source parsing dominates synchronization. Adopt the +complete assembly contract when Logic is published or overlapping raw evidence needs deterministic +ownership. + +Choose no-AST only when the project binding must forbid AST-derived publication and future adapter +upgrades of that kind. It is a binding policy, not a substitute for adapter confinement, +conformance tests, restart detection, cache recovery, or the full proof matrix in +[Incremental Adapter Indexing](INCREMENTAL_INDEXING.md). diff --git a/docs/MCP_CONTRACT.md b/docs/MCP_CONTRACT.md index b710bb8..d1c345d 100644 --- a/docs/MCP_CONTRACT.md +++ b/docs/MCP_CONTRACT.md @@ -28,6 +28,30 @@ Bootstrap and contract results include `effective_policy` schema version 1 plus registered surface and startup-bound proposal/application access. The project descriptor remains schema version 1 and does not silently acquire machine-specific process policy. +## Fixed reference binding + +`docforge.reference_mcp` is the fixed runnable binding for the in-repository Python, JavaScript, +TypeScript, and C++ reference adapters: + +```bash +python -I -m docforge.reference_mcp \ + --project-root /absolute/project \ + --capability-mode read +``` + +It loads only `.docforge/reference-adapter.toml`, selects only the fixed provider for the declared +language, and constructs the same project-bound read surface through `create_read_only_server()`. +Bootstrap binding metadata includes `server_module = "docforge.reference_mcp"`, +`adapter_mode = "reference"`, the selected `reference_language`, and the exact +`reference_config_hash`. + +The reference command accepts read capability mode only. It registers all 21 read tools below and +no isolated proposal or canonical-application tools. The configuration cannot select a provider, +module, command, arguments, working directory, environment, or arbitrary discovery behavior. See +[Reference Adapters](REFERENCE_ADAPTERS.md) for language scope and +[Agent Integration](AGENT_INTEGRATION.md) for the immutable launcher and generated client +fragments. + ## Read tools - `docforge_bootstrap` @@ -45,6 +69,8 @@ schema version 1 and does not silently acquire machine-specific process policy. - `docforge_get_task_context` - `docforge_validate_project` - `docforge_render_status` +- `docforge_graph_plan` +- `docforge_graph_render_status` - `docforge_visualize` - `docforge_stop_visualization` - `docforge_visualization_status` @@ -349,3 +375,6 @@ permissions and project instructions remain responsible for that broader boundar The legacy `adapter_policy` object remains byte-compatible. It is now a projection of the versioned `effective_policy`; `--no-ast` restrictively overrides adapter evolution, AST analysis, and Logic indexing without widening any other capability. + +See [Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for the compatibility boundary and +why the AST and Tree-sitter reference adapters are not no-AST adapters. diff --git a/docs/MIGRATING_FROM_V1.md b/docs/MIGRATING_FROM_V1.md new file mode 100644 index 0000000..dab9669 --- /dev/null +++ b/docs/MIGRATING_FROM_V1.md @@ -0,0 +1,79 @@ +# Migrating from DocForge v1 + +DocForge2 preserves the `docforge` distribution, Python package, CLI command, MCP tool prefix, +generic project descriptor version, and legacy adapter entry point. Migration is an additive +validation exercise, not a canonical-content rewrite. + +Read [compatibility](COMPATIBILITY.md), [legacy and no-AST operation](LEGACY_AND_NO_AST.md), and +[recovery](RECOVERY_AND_PERFORMANCE.md) before changing a production binding. + +## What remains compatible + +- Generic Markdown and TOML projects keep `.docforge/project.toml` schema version 1. +- A custom adapter implementing only `load_projection()` remains valid. +- Existing canonical nodes, stable IDs, source paths, relationship vocabulary, changesets, and + reviewed hashes are not silently rewritten. +- Existing CLI and MCP names remain available. New fields, commands, tools, plans, policies, and + reference integrations are additive. +- `docforge-mcp --no-ast` remains the shorthand for the preserved no-AST binding. + +Disposable index and cache schemas may change. Rebuild them rather than copying them as authority. + +## Recommended migration + +1. Record the v1 package version, adapter identity, project descriptor, canonical source hash, + active changesets, generated client configuration, and current rendered outputs. +2. Back up canonical sources and active proposal files. Derived `.docforge/cache`, preview, + portable-graph, and viewer state need not be authoritative backups. +3. Install the DocForge2 candidate in a separate environment. Do not repoint the production MCP + binding yet. +4. Run the existing adapter through complete loading and `ProjectIndex.check()`. For incremental + adapters, verify complete/incremental equivalence. Logic-producing incremental adapters must + implement the independent complete assembly oracle. +5. Compare project ID, adapter, revision, source hash, primary node and edge hashes, Logic policy, + and retrieval results with the v1 evidence. +6. Recompose effective capability and projection policy. Do not assume that a new default grants + proposal, application, rendering, or viewer access. +7. Rebuild disposable indexes, extraction caches, fragments, receipts, previews, and portable + artifacts from the validated candidate. +8. Generate a new client fragment. Generic projects use `docforge configure`; custom adapters use + `AdapterLauncherV1` with `generate_adapter_client_configuration()`. +9. Run doctor, real MCP bootstrap, representative retrieval, rendering status, recovery, and + exact-hash proposal/application checks in a disposable or shadow environment. +10. Repoint one production binding only after the candidate and rollback procedure pass. + +## Adopting reference adapters + +The repository reference adapters are narrow syntax evidence, not automatic semantic replacements +for mature v1 integrations. Python publishes modules, classes, functions, and local imports. +JavaScript and TypeScript publish syntax plus static project-relative imports and re-exports. +C++ uses a confined `compile_commands.json` as inert translation-unit inventory and publishes +directly resolvable project-local quoted includes. + +Do not replace a compiler-, language-server-, or project-owned semantic adapter if the task depends +on resolved calls, types, inheritance, macros, compiler include semantics, runtime facts, or +ownership. Use [reference adapters](REFERENCE_ADAPTERS.md) as a measured starting point. + +## Changesets + +Existing changesets remain bound to their original project, root, base revision, source hash, +writer, operations, and exact hash. Do not edit a changeset to make it look current. Retrieve and +validate it through the candidate. Rebase only when every precondition remains true; otherwise +create and review a new proposal. + +Before enabling canonical application, prove the exact serializer round trip and recovery behavior +for the selected adapter. The fixed reference MCP server is intentionally read-only. + +## Rollback + +Keep the v1 environment, configuration fragment, and service definition until the DocForge2 +binding has passed live validation. To roll back: + +1. Stop the DocForge2 MCP/viewer processes. +2. Restore the previous client or service binding. +3. Restore canonical sources only if a verified application receipt says they changed and the + adapter-specific rollback requires it. +4. Discard DocForge2-derived indexes, receipts, fragments, previews, and portable artifacts. +5. Restart v1 and verify its recorded project/source identity. + +Never move a published tag or reuse a version identity for a corrected release. diff --git a/docs/MILESTONE_4_BASELINE.md b/docs/MILESTONE_4_BASELINE.md new file mode 100644 index 0000000..8aa1abb --- /dev/null +++ b/docs/MILESTONE_4_BASELINE.md @@ -0,0 +1,95 @@ +# Milestone 4 baseline + +## Scope and method + +This baseline records the adapter SDK, Python reference adapter, incremental equivalence, and +recovery behavior completed in Milestone 4. It was captured on 2026-07-29 from clean executable +candidate `95271dcf2e48045b9d3aed9b9ea09c7fc155692c`. + +The maintained command was: + +```bash +.venv/bin/python tools/milestone4_benchmark.py \ + --mode full \ + --output benchmarks/milestone4-2026-07-29.json +``` + +The synthetic project contains 334 Python files. Each file contributes one module, one function, +and one argument node, for 1,002 primary nodes. Imports form a deterministic chain. Durations use +`time.perf_counter_ns()` and nearest-rank p95. Each ordinary result is serialized as compact sorted +JSON for its response-size gate. Per-operation memory uses `tracemalloc`; cumulative process +high-water uses `RUSAGE_SELF`. + +Environment: + +- Linux 7.1.3-200.nobara.fc44.x86_64 with glibc 2.43. +- CPython 3.14.6. +- x86_64. +- Three warm samples; cold, equivalence, and recovery operations run once. +- Per-operation traced-memory ceiling: 268,435,456 bytes. +- Process high-water ceiling: 536,870,912 bytes. +- Response ceiling: 524,288 bytes. + +The complete machine-readable result is +[`benchmarks/milestone4-2026-07-29.json`](../benchmarks/milestone4-2026-07-29.json). +Its SHA-256 is `b6a871dde730a119fc0a138c47bd25f2c533ed3171a9a3d098075aef08173600`. +The stable evidence payload SHA-256 is +`4a8db461a311b5df4abd5aa00063e9a347d8b9dba19b30e35684f561d5271549`. + +## Results + +| Operation | Median | p95 | Limit | Traced peak | Response | +|---|---:|---:|---:|---:|---:| +| Cold incremental build | 1,599.760 ms | 1,599.760 ms | 20,000 ms | 68,573,540 B | 14,580 B | +| Warm incremental build | 1,244.427 ms | 1,265.387 ms | 20,000 ms | 69,997,166 B | 14,578 B | +| Complete/incremental equivalence | 1,897.919 ms | 1,897.919 ms | 30,000 ms | 64,305,919 B | 230 B | +| Corrupt extraction-cache recovery | 1,695.406 ms | 1,695.406 ms | 30,000 ms | 69,776,923 B | 14,580 B | +| Corrupt index recovery | 1,707.879 ms | 1,707.879 ms | 30,000 ms | 68,561,924 B | 14,779 B | + +Process high-water was 78,798,848 bytes. The regression limits intentionally leave multiple times +the measured headroom; they are tripwires, not performance promises. + +## Deterministic graph and Logic evidence + +The candidate produced: + +- 1,002 primary nodes with hash + `f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f`. +- 1,001 primary edges with hash + `00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35`. +- 334 Logic projections containing 2,338 Logic nodes and 2,338 Logic edges, with hash + `c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c`. +- Complete assembly hash + `2888182ab765fbffe3ba873c1613345640e8e6d89be74ddfca7452c0a5056345`. +- Source hash + `30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2`. + +The independently loaded complete assembly and the incremental assembly matched exactly across +project identity, revision, source, nodes, edges, and Logic. + +## Work and recovery gates + +The warm build recorded all 334 sources as cache hits, zero reparsed sources, zero `ast.parse` +calls, and zero `extract_source` calls. Corrupting the extraction cache forced all sources through +extraction and reproduced the same graph and Logic hashes. Corrupting SQLite rebuilt the index +entirely from cache hits with zero parsing or extraction and reproduced those hashes. + +This zero-parser evidence applies to the Python benchmark. Focused JavaScript and TypeScript tests +prove parser-free manifests. The C++ reference manifest uses Tree-sitter for bounded quoted-include +discovery and does not make a zero-warm-parser claim. + +## Fresh-wheel and reference evidence + +The offline adoption proof built and installed the base wheel without Tree-sitter packages, +built and checked a real Python reference project, started the isolated reference MCP server, and +performed bootstrap, search, and exact retrieval over its 21 read tools. Selecting C++ without its +extra failed with `optional_dependency_missing` and `docforge[cpp]` remediation. + +Focused fixtures produced: + +- Python: 14 nodes, 13 edges, 5 Logic projections. +- JavaScript: 14 nodes, 14 edges, 5 Logic projections. +- TypeScript: 13 nodes, 14 edges, 4 Logic projections. +- C++: 17 nodes, 16 edges, 5 Logic projections. + +These fixture counts verify implementations; they are not language-wide completeness claims. diff --git a/docs/MILESTONE_4_CLOSEOUT.md b/docs/MILESTONE_4_CLOSEOUT.md new file mode 100644 index 0000000..b5eda38 --- /dev/null +++ b/docs/MILESTONE_4_CLOSEOUT.md @@ -0,0 +1,83 @@ +# Milestone 4 closeout + +## Outcome + +Milestone 4 is complete. New projects can adopt a public adapter SDK or one of four narrow +repository reference integrations, attach a fixed read-only MCP server, and follow maintained +product documentation without reading core implementation. + +Implemented contracts: + +- Stable `docforge.adapter_sdk` authoring imports. +- Independent complete primary-graph-plus-Logic oracle and exact incremental equivalence. +- Bounded adapter assemblies and version-1 extraction caches. +- Base Python, optional JavaScript, optional TypeScript, and optional C++ reference integrations. +- Closed `.docforge/reference-adapter.toml` and fixed `docforge.reference_mcp` read-only binding. +- Immutable, project-bound, launchable `AdapterLauncherV1` declarations and generated Codex, + Claude, and OpenClaw fragments for custom adapters. +- Live implementation-derived CLI and MCP reference tables with race-safe publication and drift + checking. +- Strict documentation graph, link, anchor, H1, reachability, required-page, generated-notice, and + documented-reference-config checks. +- Offline fresh-wheel adoption and maintained 1,002-node scale/recovery gates. + +## Candidate evidence + +The frozen executable candidate is +`95271dcf2e48045b9d3aed9b9ea09c7fc155692c`. + +Its complete executable gate passed: + +- Ruff formatting and Python lint. +- HTML, rendered-manual HTML, portable-graph HTML, CSS, and JavaScript checks. +- Pyright with zero diagnostics. +- Warning-strict compilation. +- 142 contract tests and 268 subtests. +- 347 complete tests and 402 subtests. +- Three Playwright and axe accessibility flows for the manual, portable graph, and live viewer. +- Lock and npm dependency-tree checks. +- Wheel and source-distribution builds. +- Offline fresh-wheel adoption. +- Milestone 0, 1, 2, 3, and 4 smoke benchmarks. + +The clean full benchmark passed exact graph-plus-Logic equivalence, warm zero Python parser and +extraction work, corrupt-cache recovery, corrupt-index recovery, response, memory, and latency +gates. Exact results are in [the Milestone 4 baseline](MILESTONE_4_BASELINE.md) and +[`benchmarks/milestone4-2026-07-29.json`](../benchmarks/milestone4-2026-07-29.json). + +Gitleaks 8.30.1 scanned the Milestone 4 commit range and candidate tree with no findings. The SSH +remote syntax prevents Gitleaks from constructing finding hyperlinks; it does not affect scanning. + +## Reference scope + +- Python uses the standard-library AST and publishes syntax plus local imports. +- JavaScript and TypeScript use distinct optional Tree-sitter grammars and publish syntax plus + project-local static relative imports and re-exports. +- C++ uses a confined `compile_commands.json` as inert translation-unit inventory and publishes + syntax plus directly resolvable project-local quoted includes. +- The references do not claim resolved calls, inheritance, types, symbol references, compiler + include semantics, macro semantics, runtime behavior, or semantic ownership. + +The C++ reference never executes a compiler or compilation-database command. It is not a Clang +semantic adapter. + +## Preserved boundaries + +- The `docforge` distribution, Python package, CLI, MCP executable, and tool names remain. +- Generic projects and one-method `load_projection()` adapters remain supported. +- Descriptor schema version 1, index schema version 3, effective policy version 1, and no-AST + behavior remain. +- Heavy language frontends are optional. Base generic and Python operation installs no + Tree-sitter distribution. +- Reference MCP is read-only. Proposal and application remain explicit project-owned gates. +- No language adapter was separately published. +- No WorldForge, ScrapeStation, legacy-repository, production-binding, storage, or self-hosting + change was made. +- No tag or Forgejo release was created for Milestone 4. + +## Later work + +Milestone 5 owns stabilization and the first DocForge2 release. Release identity, compatibility +matrix, migration and recovery proofs, comparative real-task evidence, versioning, tagging, and +publication must be validated there. Remote adapters, render farms, third-party renderers, +cross-project graphs, storage replacement, and self-hosting remain deferred without measured need. diff --git a/docs/NEW_PROJECT_QUICKSTART.md b/docs/NEW_PROJECT_QUICKSTART.md index a3460ba..808fb3e 100644 --- a/docs/NEW_PROJECT_QUICKSTART.md +++ b/docs/NEW_PROJECT_QUICKSTART.md @@ -1,6 +1,276 @@ -# DocForge setup moved to the user manual +# New-project quickstart -The complete installation, project setup, visualization, CLI, MCP, application, adapter, and -troubleshooting reference now lives in the [DocForge user manual](USER_MANUAL.md). +This guide takes a new installation from an empty project binding to a validated generic manual or +one of DocForge's fixed reference source graphs. For the complete operating reference, see the +[user manual](USER_MANUAL.md). -This file remains only so existing bookmarks and links continue to resolve. +## 1. Install DocForge + +DocForge requires Python 3.12 or newer. A base installation includes the generic project service, +the public adapter SDK, the Python reference integration, CLI, MCP server, renderers, and viewer +assets. It does not install Tree-sitter. + +From a source checkout: + +```bash +git clone /absolute/path/DocForge +cd /absolute/path/DocForge +uv sync --group dev + +DOCFORGE=/absolute/path/DocForge/.venv/bin/docforge +DOCFORGE_MCP=/absolute/path/DocForge/.venv/bin/docforge-mcp +DOCFORGE_PYTHON=/absolute/path/DocForge/.venv/bin/python +``` + +For an isolated consumer environment, install the checkout or a built wheel with `uv pip install`. +Add only the language extras that project needs: + +```bash +uv venv /absolute/path/docforge-env --python 3.12 +uv pip install --python /absolute/path/docforge-env/bin/python /absolute/path/DocForge +uv pip install --python /absolute/path/docforge-env/bin/python \ + "/absolute/path/DocForge[javascript]" +uv pip install --python /absolute/path/docforge-env/bin/python \ + "/absolute/path/DocForge[typescript]" +uv pip install --python /absolute/path/docforge-env/bin/python \ + "/absolute/path/DocForge[cpp]" +``` + +The `languages` extra installs all three optional grammar families. JavaScript and TypeScript are +separate extras because they use distinct grammars. The C++ extra supplies a syntax grammar, not a +compiler or Clang semantic frontend. + +Confirm the installation: + +```bash +"$DOCFORGE" --help +"$DOCFORGE_MCP" --help +``` + +## 2. Choose a project route + +Use a generic project when Markdown or TOML documentation is canonical. Use a fixed reference +adapter when you want a bounded source inventory and syntax-level graph for Python, JavaScript, +TypeScript, or C++. Use a project-owned adapter when the production contract must supply richer +semantics. + +- Generic manual: continue with [Create a generic project](#create-a-generic-project). +- Fixed source example: continue with [Use a reference adapter](#use-a-reference-adapter). +- Production language frontend: follow the [Adapter authoring guide](ADAPTER_AUTHORING_GUIDE.md). + +## Create a generic project + +Set the absolute project root and assess it without writing: + +```bash +PROJECT=/absolute/path/MyProject +"$DOCFORGE" --project-root "$PROJECT" onboard +``` + +The assessment reports detected languages, build evidence, documentation candidates, current +configuration, and available capabilities. Detection never invents a source graph. + +Create a generic starter explicitly: + +```bash +"$DOCFORGE" --project-root "$PROJECT" onboard \ + --scaffold \ + --project-id my-project \ + --title "My Project" +``` + +Scaffolding is create-only and refuses existing target files. It creates: + +- `.docforge/project.toml`, the generic project descriptor; +- `docs/docforge/content/architecture-overview.md`, one canonical node; +- `.docforge/templates/manual.html`, one built-in-renderer template; +- derived index, receipt, and rendered output below `.docforge`. + +If a language was detected, source graph status remains `adapter_required`. The generic starter +does not claim source semantics. + +Validate and inspect it: + +```bash +"$DOCFORGE" --project-root "$PROJECT" validate +"$DOCFORGE" --project-root "$PROJECT" reindex +"$DOCFORGE" --project-root "$PROJECT" search architecture +"$DOCFORGE" --project-root "$PROJECT" show architecture.overview +"$DOCFORGE" --project-root "$PROJECT" render-status +``` + +The descriptor is explained field by field in [Project descriptor](PROJECT_DESCRIPTOR.md). Add +canonical nodes only after choosing their authority, stable IDs, families, statuses, and allowed +relationships; [Core concepts and authority](CORE_CONCEPTS_AND_AUTHORITY.md) defines those terms. + +### Start a generic MCP binding + +Start read-only first: + +```bash +"$DOCFORGE_MCP" \ + --project-root "$PROJECT" \ + --capability-mode read +``` + +Call `docforge_bootstrap` before other tools. It reports the exact project identity, generation, +effective policy, projection policy, available capabilities, and recommended first read. + +To enable proposals, the descriptor must declare the writer and the process must select it: + +```bash +"$DOCFORGE_MCP" \ + --project-root "$PROJECT" \ + --capability-mode proposal \ + --proposal-writer project-editor +``` + +Canonical application is a separate startup gate. Do not add it to a read-only client: + +```bash +"$DOCFORGE_MCP" \ + --project-root "$PROJECT" \ + --capability-mode application \ + --proposal-writer project-editor \ + --canonical-applier project-editor +``` + +Application accepts one exact reviewed changeset hash. It does not commit, push, build, deploy, or +publish the project. + +### Generate a generic client fragment + +Preview is side-effect free: + +```bash +"$DOCFORGE" configure codex --project "$PROJECT" +"$DOCFORGE" configure claude --project "$PROJECT" +"$DOCFORGE" configure openclaw --project "$PROJECT" +``` + +Add `--output /absolute/path/new-fragment` to create one new private standalone file. Generation +does not merge with or replace a different existing file. Diagnose an installed binding with: + +```bash +"$DOCFORGE" doctor --client codex --project "$PROJECT" +``` + +Doctor is a bounded configuration inspector, not a connection test. See [Agent +integration](AGENT_INTEGRATION.md) for client-specific layouts and limitations. + +## Use a reference adapter + +Reference adapters read one fixed descriptor: +`.docforge/reference-adapter.toml`. The path is not selectable. + +Create a Python project configuration: + +```toml reference-adapter +schema_version = 1 +project_id = "my-python-project" +title = "My Python Project" +language = "python" +source_roots = ["src"] +``` + +JavaScript uses `language = "javascript"` and the `javascript` extra. TypeScript uses +`language = "typescript"` and the `typescript` extra. + +C++ additionally requires a confined compilation database: + +```toml reference-adapter +schema_version = 1 +project_id = "my-cpp-project" +title = "My C++ Project" +language = "cpp" +source_roots = ["src", "include"] +compilation_database = "compile_commands.json" +``` + +The C++ integration reads `compile_commands.json` only as bounded translation-unit inventory and +fingerprint evidence. It never executes a recorded command or compiler. + +Run the fixed read-only server through the same installed Python interpreter: + +```bash +"$DOCFORGE_PYTHON" -I -m docforge.reference_mcp \ + --project-root "$PROJECT" \ + --capability-mode read +``` + +The binding chooses one in-package provider from the descriptor language. It accepts no provider +module, command, argument list, working directory, environment, discovery rule, proposal writer, +or canonical applier. Its MCP surface is the read subset documented in the [generated command +reference](COMMAND_REFERENCE.md). + +### Generate a reference-adapter client fragment + +Generic `docforge configure` intentionally refuses custom adapters. Construct the adapter project +and immutable launcher, then call the custom-adapter generator: + +```python +from pathlib import Path + +from docforge.adapter_launcher import AdapterLauncherV1 +from docforge.client_config import generate_adapter_client_configuration +from docforge.reference_mcp import REFERENCE_MCP_MODULE, create_reference_project + +root = Path("/absolute/path/MyProject").resolve(strict=True) +project = create_reference_project(root) +launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE) + +preview = generate_adapter_client_configuration( + project, + launcher, + "codex", + capability_mode="read", +) +print(preview["artifact"]["content"]) +``` + +Use `client="claude"` or `client="openclaw"` for those formats. Pass an absolute `output` path only +when creating a new standalone private fragment. The generated launch is bound to the selected +project, descriptor hash, adapter identity, installed module, isolated interpreter, policy, and +source availability evidence. + +Read [Reference adapters](REFERENCE_ADAPTERS.md) before relying on the graph. The Python example +publishes local imports, the JavaScript and TypeScript examples publish project-local static +relative imports and re-exports, and the C++ example publishes directly resolvable project-local +quoted includes. None is a complete semantic compiler frontend. + +## 3. Add visualization only when needed + +Install the per-user viewer manager once: + +```bash +/absolute/path/DocForge/.venv/bin/docforge-viewer-manager install-user-service +``` + +Then start a project-bound snapshot: + +```bash +"$DOCFORGE" --project-root "$PROJECT" visualize +"$DOCFORGE" --project-root "$PROJECT" visualization-status +``` + +The listener is loopback-only and tokenized. The viewer is a derived snapshot, not canonical +authority and not a continuously monitored filesystem view. Read [Rendering and +visualization](RENDERING_AND_VISUALIZATION.md) for manual, portable, and live-viewer differences. + +## 4. Verify the maintained checkout + +Contributors can run: + +```bash +make command-reference-check +make docs-check +make adoption-m4 +make benchmark-m4-smoke +``` + +Run `make gate` before a release candidate. `benchmark-m4-smoke` is routine coverage; +`benchmark-m4` is the maintained full adapter workload. + +Continue with [Project onboarding](PROJECT_ONBOARDING.md) for production integration, +[Policy precedence](POLICY_PRECEDENCE.md) before widening a binding, and [Security](SECURITY.md) +before exposing any MCP process. diff --git a/docs/POLICY_PRECEDENCE.md b/docs/POLICY_PRECEDENCE.md new file mode 100644 index 0000000..2b275b1 --- /dev/null +++ b/docs/POLICY_PRECEDENCE.md @@ -0,0 +1,202 @@ +# Policy precedence + +DocForge composes an immutable effective policy for each project-bound process. It resolves +restrictions in a fixed order: + +```text +core safety + > explicit binding + > no-AST shorthand + > resource availability +``` + +A lower layer can make a requested operation unavailable; it cannot override a higher-layer +prohibition. Bootstrap and generated client evidence report the composed result and hashes so a +client does not need to infer policy from command-line arguments. + +## 1. Core safety + +Core safety is unconditional. No capability mode exposes: + +- arbitrary file access or project switching; +- arbitrary renderer, module, command, shell, argument, working-directory, or environment + selection; +- Git mutation; +- project builds or compiler execution; +- deployment or publication. + +Canonical application is limited to DocForge's validated serializer boundary. Adapter launch is +limited to the immutable launcher contract. Rendering is limited to declared views and fixed +built-in workers. An indexed instruction cannot alter any of these rules. + +## 2. Explicit binding + +A process is bound at startup to one project root, descriptor, adapter, capability mode, writer and +applier identities when present, no-AST selection, diagnostics selection, and projection modes. +The binding does not change during the process lifetime. + +Capability modes are: + +- `read`: register only the read surface; +- `proposal`: add proposal tools when a valid selected writer is available; +- `application`: require a startup-bound canonical applier and expose exact-hash application; +- `operator`: reserved; it currently adds no tools. + +Mode describes the maximum registered surface. Actual authority can be narrower. A descriptor must +declare the selected writer, including allowed families and operation types. Application requires +the matching configured writer, changeset creator, and canonical-applier identity. A mode name +cannot create a missing descriptor grant. + +Generic generated client fragments default to read mode. Other construction paths preserve their +documented compatible factory defaults. Treat `docforge_bootstrap.session_contract` and its actual +capabilities as authoritative for a running server. + +## 3. No-AST shorthand + +`--no-ast` is a restrictive compatibility shorthand. It composes: + +- adapter evolution `preserve`; +- AST analysis `forbidden`; +- Logic indexing `off`; +- `docforge_get_logic` blocked; +- prohibitions on AST, Tree-sitter, compiler-AST, and function-Logic upgrades. + +Non-AST source fingerprinting and incremental caching remain allowed. Existing one-method adapters +continue to use `load_projection()`. + +The shorthand does not inspect how an existing adapter was implemented, sandbox its filesystem +reads, or transform an AST-based adapter into a no-AST adapter. Do not run the Python, +JavaScript/TypeScript, or C++ syntax reference integrations and then describe the binding as a +proved no-AST integration. See [Legacy and no-AST operation](LEGACY_AND_NO_AST.md). + +## 4. Resource availability + +Even an allowed policy cannot create a missing resource: + +- application mode requires a startup-bound canonical applier; +- manual `explicit` or `auto` requires declared manual render configuration; +- manual `auto` also requires canonical application in the current operation or server; +- portable graph `explicit` requires declared portable graph configuration; +- live viewer `on-demand` requires the viewer runtime; +- an optional reference language requires its installed extra; +- a project-owned launcher module must be installed, isolated, project-owned, and unchanged. + +Unavailable requested modes fail before hidden work. Missing optional language grammars return an +actionable install target such as `docforge[typescript]` or `docforge[cpp]`; DocForge does not +silently downgrade to a different frontend. + +## Effective process policy + +The version-1 effective policy reports: + +- capability mode and whether it came from a factory default or explicit selection; +- adapter evolution, AST analysis, and Logic indexing; +- automatic synchronization and validated integrity; +- the compatible manual-render projection; +- profiling state; +- blocked tools and prohibitions; +- the exact precedence list. + +This version-1 projection preserves compatibility. It is not the complete version-2 rendering +policy; portable graph and live-viewer choices are reported separately. + +## Independent projection policy + +Manual rendering, portable graph publication, and live visualization use a separate immutable +version-2 policy: + +```text +manual: auto | explicit | disabled +portable_graph: explicit | disabled +live_viewer: on-demand | disabled +``` + +When a selector is omitted, composition uses availability-aware defaults: + +- manual is `auto` only when manual configuration and canonical application are both available; + otherwise it is `explicit` when configured, or `disabled`; +- portable graph is `explicit` when configured, otherwise `disabled`; +- live viewer is `on-demand` when the runtime is available, otherwise `disabled`. + +An explicit non-disabled selection for a missing resource returns +`projection_policy_unavailable`. An active operation prohibited by the selected policy returns +`projection_policy_forbids_operation` before planning, rendering, or viewer startup. + +Status remains intentionally narrower than active work. Manual and portable receipt-only status +are available when their active operations are disabled. Viewer status and explicit stop remain +available when viewer start is disabled. + +Ordinary standalone CLI rendering uses manual `explicit`. Manual `auto` belongs to a canonical +application operation that owns automatic regeneration. + +## Diagnostics do not grant authority + +`--diagnostics` enables bounded request-local stage timings and compiler-work counters. It does not +enable tools, broaden paths, retain project content, or displace a primary MCP result that already +needs the response budget. + +## Descriptor policy and process policy + +The project descriptor is canonical project configuration. The process policy is a runtime +restriction. They compose by intersection: + +```text +operation is available + only if core permits it + and the startup binding registers it + and no-AST permits it + and required resources exist + and the descriptor grants the requested project authority + and current graph/hash preconditions validate +``` + +Changing a descriptor does not retarget a running project-owned process. Descriptor, adapter +implementation, or launcher drift requires a fresh process. + +## Common decisions + +For an agent that only reads documentation: + +```bash +docforge-mcp \ + --project-root /absolute/path/MyProject \ + --capability-mode read +``` + +For an agent that may prepare reviewable proposals: + +```bash +docforge-mcp \ + --project-root /absolute/path/MyProject \ + --capability-mode proposal \ + --proposal-writer project-editor +``` + +For a tightly controlled application process: + +```bash +docforge-mcp \ + --project-root /absolute/path/MyProject \ + --capability-mode application \ + --proposal-writer project-editor \ + --canonical-applier project-editor +``` + +For a read binding with every active output projection disabled: + +```bash +docforge-mcp \ + --project-root /absolute/path/MyProject \ + --capability-mode read \ + --manual-render-policy disabled \ + --portable-graph-policy disabled \ + --live-viewer-policy disabled +``` + +Prefer the narrowest binding that completes the workflow. Call `docforge_bootstrap` first and use +the returned effective policy, projection policy, actual capabilities, and prohibitions rather +than assumptions based on client configuration. + +See the [MCP contract](MCP_CONTRACT.md), [Security](SECURITY.md), [Project +descriptor](PROJECT_DESCRIPTOR.md), and [Rendering and +visualization](RENDERING_AND_VISUALIZATION.md). diff --git a/docs/PROJECT_DESCRIPTOR.md b/docs/PROJECT_DESCRIPTOR.md new file mode 100644 index 0000000..6e1be88 --- /dev/null +++ b/docs/PROJECT_DESCRIPTOR.md @@ -0,0 +1,279 @@ +# Project descriptor + +A generic DocForge project is selected by one fixed file: +`.docforge/project.toml` beneath an explicit project root. The descriptor is schema version 1. +DocForge validates both the JSON-schema shape in `schemas/project.schema.json` and runtime +invariants that schema alone cannot prove. + +Reference integrations use a different fixed descriptor, +`.docforge/reference-adapter.toml`; see [Reference adapters](REFERENCE_ADAPTERS.md). +Project-owned adapters construct the same runtime `ProjectDescriptor` contract through the public +adapter SDK. + +## Complete generic example + +```toml +schema_version = 1 +project_id = "my-project" +title = "My Project" +adapter = "generic" + +[sources] +content_roots = ["docs/docforge/content"] +authority_files = [] + +[derived] +cache_root = ".docforge/cache" +index = ".docforge/cache/index.sqlite3" + +[changesets] +root = ".docforge/changesets" + +[[changesets.writers]] +id = "project-editor" +families = ["architecture", "operations", "system"] +operations = ["create", "update", "move", "delete"] + +[render] +template_root = ".docforge/templates" +preview_root = ".docforge/previews" + +[[render.views]] +id = "manual" +renderer = "generic_html" +template = "manual.html" +output = ".docforge/rendered/manual.html" +title = "My Project Manual" +families = ["architecture", "operations", "system"] + +[graph_render] +output_root = ".docforge/portable-graph" + +[[graph_render.views]] +id = "architecture" +renderer = "portable_graph_html" +output = "architecture.html" +title = "Architecture" +root = "architecture.overview" +initial_mode = "web" +depth = 3 +max_nodes = 250 +max_edges = 1000 +max_work = 100000 +families = ["architecture", "system"] +relations = ["depends_on", "owns", "calls", "reads", "writes", "tested_by", "relates_to"] +authorities = [] +statuses = ["current", "active", "verified"] +tags = [] +include_logic = false + +[graph] +allowed_relations = [ + "calls", + "depends_on", + "owns", + "reads", + "relates_to", + "tested_by", + "writes", +] + +[limits] +max_source_bytes = 500000 +max_nodes = 10000 +max_query_chars = 500 +max_results = 100 +max_traversal_depth = 6 +max_context_tokens = 12000 +max_tool_output_chars = 200000 +max_changesets = 100 +max_changeset_operations = 100 +max_changeset_bytes = 1000000 +max_render_views = 20 +max_template_bytes = 1000000 +max_render_bytes = 1000000 + +[[profiles]] +id = "development" +families = ["architecture", "operations", "system"] +statuses = ["current", "active", "verified"] +required_nodes = ["architecture.overview"] +token_budget = 8000 +dependency_depth = 3 +``` + +Rendering sections are optional. Profiles and limits may also be omitted; runtime defaults then +apply. The required top-level fields are `schema_version`, `project_id`, `title`, `adapter`, +`sources`, `derived`, `changesets`, and `graph`. + +## Identity fields + +`schema_version` must be `1`. + +`project_id` is the stable machine identity. It is lowercase and may contain digits, dots, +underscores, and hyphens after its first character. Do not derive it from a mutable display title. + +`title` is the human-readable project name. + +`adapter` is `generic` for this file format. Project-owned adapters publish a validated +`adapter_id@adapter_version` identity through their runtime descriptor; changing adapter identity +invalidates incompatible derived state. + +The descriptor's byte content contributes to a descriptor hash. Client fragments, launchers, index +evidence, and policy results use that hash to detect drift. + +## Canonical sources + +`sources.content_roots` lists the project-relative directories containing generic Markdown and TOML +nodes. Each path must resolve beneath the project root. Canonical content roots may not overlap the +derived cache. + +`sources.authority_files` lists additional project-relative regular files whose content belongs to +the canonical project identity. They are not automatically parsed as nodes. + +A Markdown node contains one TOML metadata block followed by Markdown content: + +```markdown ++++ +schema_version = 1 +id = "architecture.overview" +title = "Architecture overview" +family = "architecture" +authority = "authoritative" +status = "current" +tags = ["architecture"] +summary = "Defines the top-level architecture and ownership." ++++ + +# Architecture overview + +Describe systems, ownership, runtime flow, failure behavior, and proof. +``` + +Every node ID is project-wide and stable. Every relationship target must resolve. A TOML source may +contain multiple `[[nodes]]` records; proposal-enabled multi-node files need stable +`source_anchor` values where creation or movement requires an exact record boundary. + +## Derived state + +`derived.cache_root` owns disposable indexes, attestations, extraction caches, render receipts, +projection artifacts, and viewer registry state. + +`derived.index` must be inside `derived.cache_root`. Canonical content and cache paths must not +overlap. + +Derived state is not a backup. If it is deleted or rejected as corrupt, DocForge rebuilds it from +validated canonical sources. + +## Changesets and writers + +`changesets.root` is the confined proposal store. It must not overlap canonical content or the +derived cache. + +Each `changesets.writers` entry declares: + +- a stable writer `id`; +- the node `families` that writer may change; +- allowed `operations`: `create`, `update`, `move`, and/or `delete`. + +The descriptor grant is necessary but not sufficient. A process must also select that writer at +startup, and canonical application requires a separately bound matching applier. Capability mode +does not broaden the descriptor grant. See [Policy precedence](POLICY_PRECEDENCE.md). + +## Allowed relationships + +`graph.allowed_relations` is the exact project vocabulary accepted on edges. It must be nonempty. +Relationship names are stable IDs. DocForge rejects relationships that are not declared and gives +`depends_on` additional cycle validation. + +The core does not reinterpret a custom relationship just because its spelling resembles a known +term. Task-context retrieval classifies only the documented versioned aliases and preserves +unknown allowed relationships as `unclassified`. + +## Context profiles + +Each `profiles` entry defines one bounded context compilation: + +- `id` selects the profile; +- `families` and `statuses` filter eligible nodes; +- `required_nodes` names stable nodes that must be present; +- `token_budget` limits compiled content; +- `dependency_depth` bounds relationship expansion. + +Profiles choose derived retrieval scope. They do not change node authority or writer permissions. + +## Limits + +Positive limits bound input, graph, retrieval, proposal, and rendering work. Current defaults are: + +- `max_source_bytes = 1000000` +- `max_nodes = 10000` +- `max_query_chars = 500` +- `max_results = 100` +- `max_traversal_depth = 8` +- `max_context_tokens = 32000` +- `max_tool_output_chars = 200000` +- `max_changesets = 1000` +- `max_changeset_operations = 100` +- `max_changeset_bytes = 1000000` +- `max_render_views = 100` +- `max_template_bytes = 1000000` +- `max_render_bytes = 1000000` + +Smaller project limits are useful policy. They cannot widen fixed internal worker, package, +response, or cache ceilings. In particular, detached renderer transfer has its own fixed boundary +even if a compatibility descriptor retains a larger `max_render_bytes`. + +## Manual rendering + +The optional `render` section declares: + +- one confined `template_root`; +- one isolated `preview_root`; +- one or more stable views. + +Each view uses the built-in `generic_html` renderer, a template beneath `template_root`, one +declared output path, a title, and a family filter. Paths may not overlap canonical content, +authority files, changesets, cache, templates, or previews in unsafe ways. + +Templates are inert UTF-8 files with a fixed token vocabulary. They cannot select executable +renderers or commands. See [Rendering and visualization](RENDERING_AND_VISUALIZATION.md). + +## Portable graph rendering + +The optional `graph_render` section declares an output root and one or more +`portable_graph_html` views. Each view selects exactly one: + +- `root`, an exact stable node ID; or +- `query`, a bounded metadata-only lexical seed. + +It may then restrict families, relations, authorities, statuses, and tags, plus depth, node, edge, +and work limits. `initial_mode` is `nodes`, `flow`, or `web`. Portable graph contract version 1 +requires `include_logic = false`. + +The declared `output` is relative to `graph_render.output_root`. + +## Paths and confinement + +Descriptor paths are project-relative. Absolute paths and parent traversal are rejected. Runtime +validation also rejects symlink escapes, unexpected file types, unsafe overlap, changing path +identity during sensitive reads or publication, and derived outputs outside their declared roots. + +The explicit CLI `--project-root` is the only project selector. The MCP process binds it at startup +and exposes no project-switching tool. + +## Validate changes safely + +After editing the descriptor: + +```bash +docforge --project-root /absolute/path/MyProject validate +docforge --project-root /absolute/path/MyProject reindex +docforge --project-root /absolute/path/MyProject check +``` + +Descriptor or adapter implementation drift makes a project-owned running process fail closed; start +a fresh process after changing those boundaries. + +For first-time creation, prefer the create-only [New-project +quickstart](NEW_PROJECT_QUICKSTART.md). For full invariants, read the [Core contract](CONTRACT.md). diff --git a/docs/PROJECT_ONBOARDING.md b/docs/PROJECT_ONBOARDING.md index 90a75ef..a16efea 100644 --- a/docs/PROJECT_ONBOARDING.md +++ b/docs/PROJECT_ONBOARDING.md @@ -5,6 +5,11 @@ DocForge onboarding has two separate outcomes: 1. A generic manual can be configured, indexed, rendered, visualized, and exposed through the MCP. 2. A source graph additionally requires one validated language frontend per source language. +DocForge ships narrow fixed references for Python, JavaScript, TypeScript, and C++. They are useful +for syntax-scoped projects and adoption proof, but production semantic requirements may still +require a project-owned compiler or language-service adapter. Review +[Reference Adapters](REFERENCE_ADAPTERS.md) before selecting a frontend. + The onboarding command never claims that source semantics exist merely because it found source files. It reports each detected language as `adapter_required` until a project integration supplies and proves that frontend. @@ -64,6 +69,32 @@ immediately usable through the generic CLI, viewer, and MCP. The starter overview records detected languages and states that the source graph is unavailable until a language frontend passes the adapter proof. That limitation is deliberate. +## Configure a fixed reference adapter + +The generic onboarding scaffold and fixed reference configuration are separate project routes. +For a syntax-scoped reference project, create `.docforge/reference-adapter.toml`: + +```toml +schema_version = 1 +project_id = "my-python-project" +title = "My Python project" +language = "python" +source_roots = ["src"] +``` + +Then start the fixed read-only binding: + +```bash +python -I -m docforge.reference_mcp \ + --project-root /absolute/path/MyProject \ + --capability-mode read +``` + +The configuration selects only a fixed in-repository provider and cannot name a command or custom +module. JavaScript, TypeScript, and C++ require their respective optional extras; C++ also requires +`compilation_database = "compile_commands.json"`. The complete configuration and supported-fact +contract are in [Reference Adapters](REFERENCE_ADAPTERS.md). + ## Complete onboarding checklist ### 1. Repository assessment @@ -118,6 +149,13 @@ For every source language: - [ ] Define dependency discovery. - [ ] State unsupported semantic facts explicitly. +Decide whether the project needs production semantic evidence or the narrower syntax-only +reference scope. The Python reference publishes only project-local imports. JavaScript and +TypeScript publish only project-local static relative imports and re-exports. The C++ reference +publishes only directly resolvable project-local quoted includes and does not run a compiler. None +of those references resolves calls, inheritance, types, symbols, runtime behavior, or semantic +ownership. + All frontends emit the same DocForge contracts: - `AdapterManifest` inventories fingerprinted extraction units and dependencies. @@ -151,6 +189,11 @@ this checklist summarizes. - [ ] Assign shared symbols to one deterministic source contribution. - [ ] Record compiler-derived project include dependencies. +These are production semantic-adapter expectations. The built-in C++ reference uses +`compile_commands.json` only as bounded translation-unit inventory and fingerprint evidence. It +parses commands and arguments as inert data, executes no command or compiler, and does not claim +compiler include semantics, symbol ownership, or a Clang-derived graph. + #### Rust - [ ] Read the Cargo workspace and package graph. @@ -240,6 +283,10 @@ from filenames. Done when a new session can identify and retrieve the correct project without being told its file layout. +The fixed reference server is read-only. Generic and custom-adapter client generation, including +the immutable custom launcher boundary, is documented in +[Agent Integration](AGENT_INTEGRATION.md). + ### 10. Operating guide and maintenance - [ ] Record the authority and progressive-reading order. @@ -286,14 +333,18 @@ to one configured project root. DocForge does not let an MCP call install dependencies, run project builds, modify Git, deploy, or publish. A project integration may use its own normal development workflow for those actions. -## Frontend packaging direction +## Reference and production frontend boundaries -Reusable language frontends should be separate packages or project-owned adapters over the public -DocForge contracts. They must not put language-specific rules into the graph, index, viewer, or MCP -core. +The public authoring namespace is `docforge.adapter_sdk`. Project-owned and separately distributed +production frontends should build on that contract without putting language-specific rules into +the graph, index, viewer, or MCP core. -Worldforge is the first complete C++ reference integration. A reusable C++ package should be -extracted only after that integration proves stable ownership, compiler dependency invalidation, -and complete/incremental equivalence. Rust and Java frontends should then implement the same -contract using their authoritative build and language tooling rather than copying C++ extraction -rules. +The in-repository Python, JavaScript, TypeScript, and C++ adapters are reference implementations. +Their syntax-scoped behavior is useful without becoming a claim that every project in those +languages has complete semantic coverage. A compiler-backed production C++ adapter may resolve +build flags, calls, types, inheritance, include semantics, and ownership when it can prove those +facts. The reference C++ adapter does none of that and is not a Clang semantic adapter. + +New Rust, Java, or other frontends should use their authoritative build and language tooling and +must pass the graph-plus-Logic complete/incremental contract in the +[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md). diff --git a/docs/RECOVERY_AND_PERFORMANCE.md b/docs/RECOVERY_AND_PERFORMANCE.md new file mode 100644 index 0000000..62f6499 --- /dev/null +++ b/docs/RECOVERY_AND_PERFORMANCE.md @@ -0,0 +1,86 @@ +# Recovery and performance + +DocForge keeps canonical project evidence separate from disposable indexes, caches, receipts, and +rendered projections. Recovery rebuilds derived state from current authority; it does not rewrite +canonical content to make a cache look valid. + +For the underlying boundaries, see [core authority](CORE_CONCEPTS_AND_AUTHORITY.md) and the +[security model](SECURITY.md). + +## Recovery order + +Use the narrowest verified recovery: + +1. Run `docforge check` or the corresponding MCP status operation. +2. Run `docforge sync` to repair a missing or stale disposable index when safe. +3. Run `docforge reindex` for an explicit complete rebuild. +4. Restart a long-running MCP or viewer binding after descriptor or adapter implementation drift. +5. Regenerate a derived client fragment, command reference, preview, manual, or portable graph + from current project evidence. + +Never delete or rewrite canonical sources, active changesets, or reviewed hashes as cache cleanup. + +## Extraction-cache recovery + +Incremental adapter contributions are stored in a version-1 disposable extraction cache. Reads are +no-follow, regular-file-only, identity-checked, bounded to 64,000,000 bytes and 10,000 source +records, and fail to a cache miss on malformed or incompatible data. Publication is atomic. + +After a miss, current manifest sources are extracted again and the complete assembly is validated. +The active SQLite generation remains authoritative for reads until a new verified index is +published. An extraction cache may therefore be safely ahead of the last index; the two files do +not pretend to be one transaction. + +## Index and receipt recovery + +A validated SQLite index is a generation-pinned derived snapshot. Missing, corrupt, unattested, or +stale indexes rebuild from the complete project or adapter oracle. Generation receipts and render +receipts are post-publication evidence. Failure to write a receipt after a committed artifact is +reported as degraded committed success, not as permission to repeat a mutation. + +The live viewer pins one validated index identity. Index replacement makes the running snapshot +stale and causes a later visualize request to start a fresh worker. + +## Proposal and application recovery + +Hash or base conflicts are not cache failures. Retrieve the current changeset and diff, then +review the new exact hash. Rebase is allowed only when every touched node, relationship, source, +permission, and graph invariant still matches. A content conflict requires a new proposal. + +If a canonical serializer fails its round-trip check, use its reported rollback state. Do not +reapply a changeset whose application may already have committed. See +[migrating from v1](MIGRATING_FROM_V1.md) for rollback planning. + +## Milestone 4 scale evidence + +The maintained Python reference benchmark creates 334 Python source files and produces: + +- 1,002 primary nodes and 1,001 primary edges. +- 334 Logic projections with 2,338 Logic nodes and 2,338 Logic edges. +- Exact complete/incremental primary graph and Logic equality. +- Exact output after corrupt extraction-cache recovery and corrupt-index recovery. +- Zero `ast.parse` calls and zero `extract_source` calls during a warm build. + +On the frozen Milestone 4 candidate, operation p95 values were 1.266 to 1.898 seconds. Per-operation +traced peaks were about 61 to 67 MiB, and process high-water was 78,798,848 bytes. These are +regression measurements from one machine, not universal latency promises. The machine-readable +record is `../benchmarks/milestone4-2026-07-29.json`; detailed method and hashes are in +[the Milestone 4 baseline](MILESTONE_4_BASELINE.md). + +The zero-parser claim is deliberately Python-only. Focused JavaScript and TypeScript tests prove +their manifests avoid Tree-sitter. The C++ reference manifest currently uses Tree-sitter while +discovering bounded quoted includes, so a warm C++ cache hit is not evidence of zero parser work. + +## Maintained gates + +Run: + +```bash +make gate +make adoption-m4 +make benchmark-m4-full +make docs-check +``` + +The main gate includes smoke benchmarks. Full milestone evidence is recorded separately from a +clean candidate so smoke or dirty-tree results cannot become release claims. diff --git a/docs/REFERENCE_ADAPTERS.md b/docs/REFERENCE_ADAPTERS.md new file mode 100644 index 0000000..4287651 --- /dev/null +++ b/docs/REFERENCE_ADAPTERS.md @@ -0,0 +1,149 @@ +# Reference adapters + +DocForge includes fixed, syntax-scoped reference adapters for Python, JavaScript, TypeScript, and +C++. They demonstrate the public adapter contract, deterministic complete and incremental +publication, function Logic, cache recovery, and a runnable read-only MCP binding. They are not +compiler or language-service replacements. + +Use a reference adapter when its deliberately narrow graph is sufficient or when proving a fresh +DocForge integration. Use the [Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md) for a +production adapter that needs resolved symbols, calls, inheritance, types, build semantics, or +semantic ownership. + +## Installation + +The Python reference adapter uses the standard library and is available in the base wheel. The +other frontends are separate optional extras: + +```bash +python -m pip install docforge +python -m pip install "docforge[javascript]" +python -m pip install "docforge[typescript]" +python -m pip install "docforge[cpp]" +``` + +Install `docforge[languages]` only when one environment intentionally needs all three Tree-sitter +frontends. JavaScript and TypeScript use distinct grammar packages and distinct extras. + +## Fixed project configuration + +The runnable reference binding reads exactly +`.docforge/reference-adapter.toml` below the selected project root. The configuration is data only: +it cannot name a provider, Python module, command, arguments, working directory, environment, or +discovery rule. + +Reference adapter configuration `.docforge/reference-adapter.toml`: + +```toml +schema_version = 1 +project_id = "example-python" +title = "Example Python project" +language = "python" +source_roots = ["src"] +``` + +`language` is one of `python`, `javascript`, `typescript`, or `cpp`. Source roots must be sorted, +unique, non-overlapping project-relative directories outside `.docforge`. The configuration, +roots, and inventoried source files must be regular confined paths without symlink traversal. The +configuration is limited to 65,536 bytes, at most 64 source roots, and 65,536 examined inventory +entries. + +C++ additionally requires one explicit project-confined compilation database: + +```toml +schema_version = 1 +project_id = "example-cpp" +title = "Example C++ project" +language = "cpp" +source_roots = ["include", "src"] +compilation_database = "compile_commands.json" +``` + +`compilation_database` is forbidden for the other three languages. + +## Published scope + +All four adapters publish deterministic file, module or translation-unit, class or struct, and +function nodes where their syntax supports those categories. They publish lexical `contains` +relationships, a narrow set of project-local `depends_on` relationships, and function-scoped +Logic. Calls that appear inside Logic are syntax steps, not resolved symbol relationships. + +### Python + +Python uses `ast` from the standard library for source extraction and Logic. Manifest construction +does not build a Python AST; it tokenizes imports only far enough to publish dependencies that +resolve to another module in the declared source inventory. + +It does not import or execute project code. It does not resolve dynamic imports, calls, +inheritance, imported symbols, types, overloads, re-exports, decorators, metaclasses, descriptors, +or runtime-generated behavior. + +### JavaScript and TypeScript + +JavaScript uses the optional `tree-sitter-javascript` grammar. TypeScript uses the distinct +optional `tree-sitter-typescript` grammar. Their focused incremental tests prove that manifest +construction and an unchanged warm build do not invoke the Tree-sitter extraction parser. + +Only static relative imports and re-exports that resolve to another inventoried source file become +dependencies. Dynamic `import()`, `require()`, bare package specifiers, aliases, `tsconfig` paths, +loader hooks, types, interfaces, overloads, calls, inheritance, symbols, and runtime behavior are +not resolved. The adapters never import, compile, transpile, or execute project code. + +### C++ + +C++ treats `compile_commands.json` as the authoritative bounded translation-unit inventory and as +fingerprint evidence. Commands, arguments, directories, and output fields are parsed as inert +data. The adapter executes no compiler, build tool, command, project binary, or project code. + +The reference C++ manifest uses the optional `tree-sitter-cpp` grammar to parse inventoried sources +and discover quoted includes. It publishes a dependency only when that quoted include resolves +directly to a real project-local header in the declared roots. Angle-bracket includes, compiler +include paths, frameworks, generated headers, compiler-provided headers, conditional compilation, +and macro expansion are omitted. + +This is not a Clang semantic adapter. It does not claim resolved calls, types, templates, aliases, +concepts, references, inheritance, overload ownership, out-of-line semantic ownership, macro +semantics, or compiler include semantics. A warm C++ extraction-cache hit is therefore not evidence +that manifest construction performed zero parser work. + +## Complete and incremental proof + +Each reference adapter implements: + +- `load_projection()` for the complete primary graph compatibility oracle; +- `load_complete_assembly()` for the complete graph-plus-Logic oracle; +- `load_manifest()` and `extract_source()` for incremental extraction; and +- deterministic assembly of the complete current contribution set. + +The maintained fixtures prove complete determinism, exact complete/incremental graph and Logic +parity, reverse-dependency invalidation, additions and deletions, corrupt extraction-cache +recovery, confinement, and exact unsupported-fact inventories. See +[Incremental Adapter Indexing](INCREMENTAL_INDEXING.md) for the cache contract and +[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for compatibility and policy limits. + +Current fixture evidence is: + +| Language | Primary nodes | Relationships | Logic projections | +|---|---:|---:|---:| +| Python | 14 | 13 | 5 | +| JavaScript | 14 | 14 | 5 | +| TypeScript | 13 | 14 | 4 | +| C++ | 17 | 16 | 5 | + +These are regression-fixture shapes, not promises for arbitrary repositories. + +## Read-only reference server + +Start the fixed server with: + +```bash +python -I -m docforge.reference_mcp \ + --project-root /absolute/path/to/project \ + --capability-mode read +``` + +The server selects one of the four in-repository providers solely from the validated fixed +configuration. Its cache stays below `.docforge/cache/reference-adapter/`. It registers +the 21-tool read surface and no proposal or application tools. See +[Agent Integration](AGENT_INTEGRATION.md) for generated client fragments and +[MCP Boundary](MCP_CONTRACT.md) for the exact tool contract. diff --git a/docs/RENDERING_AND_VISUALIZATION.md b/docs/RENDERING_AND_VISUALIZATION.md new file mode 100644 index 0000000..90bbeb4 --- /dev/null +++ b/docs/RENDERING_AND_VISUALIZATION.md @@ -0,0 +1,233 @@ +# Rendering and visualization + +DocForge has three independent ways to present one validated graph generation: + +```text +validated generation + ├── manual plan → immutable package → detached HTML renderer → declared manual output + ├── graph plan → immutable package → detached graph renderer → portable static artifact + └── pinned index → managed loopback viewer → interactive Nodes, Flow, Web, and lazy Logic +``` + +They share validated facts but not authority, publication, or lifecycle. A manual render does not +publish a portable graph. A portable graph does not start the live viewer. None is canonical +project content. + +## Manual rendering + +A generic descriptor may declare a manual template root, isolated preview root, and stable views: + +```toml +[render] +template_root = ".docforge/templates" +preview_root = ".docforge/previews" + +[[render.views]] +id = "manual" +renderer = "generic_html" +template = "manual.html" +output = ".docforge/rendered/manual.html" +title = "My Project Manual" +families = ["architecture", "operations", "system"] +``` + +The renderer name is fixed to `generic_html`. Templates are confined UTF-8 assets with a fixed +token vocabulary. They cannot name commands, Python modules, executable renderers, or arbitrary +publication paths. Raw HTML in canonical Markdown is disabled by the pinned CommonMark path. + +Plan and render identity covers the canonical source hash, optional changeset hash, selected nodes +and relationships, view configuration, template hash, renderer contract, and parser version. + +Use: + +```bash +docforge --project-root "$PROJECT" render-status +docforge --project-root "$PROJECT" render-status manual --deep +docforge --project-root "$PROJECT" render manual +docforge --project-root "$PROJECT" preview CHANGESET_ID manual +``` + +Normal status verifies bounded source, configuration, template, output, renderer, and receipt +identities without reconstructing output. `--deep` explicitly runs the side-effect-free full-render +oracle. A changeset preview writes only to the isolated preview root. + +Manual `auto` means that a successful canonical application owns regeneration of every declared +manual view. It does not mean background rendering, and ordinary standalone CLI render cannot +select `auto`; use `explicit`. + +## Portable graph rendering + +Portable graph configuration is separate: + +```toml +[graph_render] +output_root = ".docforge/portable-graph" + +[[graph_render.views]] +id = "architecture" +renderer = "portable_graph_html" +output = "architecture.html" +title = "Architecture" +root = "architecture.overview" +initial_mode = "web" +depth = 3 +max_nodes = 250 +max_edges = 1000 +max_work = 100000 +families = ["architecture", "system"] +relations = ["depends_on", "owns", "calls", "reads", "writes", "tested_by", "relates_to"] +authorities = [] +statuses = ["current", "active", "verified"] +tags = [] +include_logic = false +``` + +A view selects exactly one exact `root` node or bounded metadata-only lexical `query`. Exact +filters and node, edge, depth, and work limits close the selection. The initial mode is `nodes`, +`flow`, or `web`. Portable graph contract version 1 excludes function-scoped Logic. + +Use: + +```bash +docforge --project-root "$PROJECT" graph-plan architecture +docforge --project-root "$PROJECT" graph-render architecture +docforge --project-root "$PROJECT" graph-render-status architecture +``` + +`graph-plan` validates and returns the generation-pinned plan without publication. `graph-render` +is an explicit local CLI publication action. It commits a content-addressed artifact, renderer +receipt, and bounded generation/view manifest in that order; the manifest is the publication +commit. `graph-render-status` checks only bounded committed evidence and never plans or renders. + +MCP may plan and inspect portable graph evidence, but it does not publish the portable artifact. +Use the explicit project-local CLI command for publication. + +Portable output is a complete static artifact. JavaScript is progressive enhancement, not a +requirement for the graph facts to be present. + +## Live viewer + +The live viewer is a managed, read-only browser over one generation-pinned validated SQLite file. +Install the per-user manager once: + +```bash +docforge-viewer-manager install-user-service +``` + +Then operate a project viewer: + +```bash +docforge --project-root "$PROJECT" visualize +docforge --project-root "$PROJECT" visualize --node architecture.overview +docforge --project-root "$PROJECT" visualize --query persistence +docforge --project-root "$PROJECT" visualization-status +docforge --project-root "$PROJECT" visualization-stop +``` + +`visualize` accepts only an optional stable node ID or lexical query, bounded traversal depth, and +the local `--no-open` presentation choice. It does not accept a project root override, database +path, SQL, template, command, renderer, bind address, or module. + +The HTTP listener binds to `127.0.0.1` on an operating-system-selected port. A random token is part +of every accepted path. The server supports only `GET` and `HEAD`, sets no-store and restrictive +browser security headers, and has no write, project-selection, arbitrary-query, or static +filesystem endpoint. + +The viewer is a snapshot. Index replacement or alteration makes that snapshot fail closed; start a +new visualization to use a new validated generation. Source inspection reads only project-confined +source evidence bound to the pinned generation. + +## Nodes, Flow, Web, and Logic + +The interactive viewer offers four complementary projections: + +- **Nodes** shows a bounded relation-neutral incoming and outgoing neighborhood. +- **Flow** presents semantic contributors toward the focus. Prerequisite-style stored + relationships may be reversed for presentation without changing stored direction. +- **Web** expands the convergence picture with contributors, callers, containers, members, and + contextual relationships. +- **Logic** loads a function-scoped control-flow projection only when requested. + +Logic supports the explicit control paths published by Python, JavaScript, TypeScript, and C++ +integrations. It stays outside the primary graph, portable graph version 1, search, and +generation-diff receipts. + +Hiding a node is presentation-only. In Flow and Web, ancestors without another path to the focus +are removed. In Logic, an omitted-path bridge preserves downstream readability. Restore reverses +the presentation change; neither action mutates the index. + +## Immutable plan and worker boundary + +Manual and graph plans are versioned, canonical JSON with deterministic ordering and fixed +structural and byte limits. They contain selected graph facts and bounded content, but no live +project object, SQLite handle, absolute project or index path, arbitrary query, command, +executable path, or caller-selected renderer module. + +A projection package binds one plan to inert assets, fixed component versions, a closed built-in +renderer identity, and an exact artifact inventory. The detached worker: + +- runs one fixed private module through isolated Python; +- uses a trusted working directory and sanitized environment; +- accepts one canonical newline-terminated JSON request; +- returns one bounded canonical JSON response; +- has a fixed renderer allowlist and timeout; +- cannot select graph facts, read project state, choose output paths, or mutate canonical files. + +The package contract is bounded, and actual artifact transfer has a fixed 20,000,000-byte ceiling. +A larger descriptor `max_render_bytes` compatibility value does not widen that worker boundary. + +## Manual fragment reuse + +Manual fragments are disposable semantic cache records, not publication authority. A cold record +is accepted only after byte-exact comparison against a full detached render. On a warm hit, the +worker independently recomputes the expected fragment before reuse. + +Corrupt, forged, stale, incompatible, individually oversized, or aggregate-oversized records fall +back to the complete full-render oracle. The current cache is bounded to 10,000 records and +64,000,000 bytes. + +Fragment reuse is a correctness and recovery boundary. Do not promise a speedup without current +measurements. + +## Projection policy + +The version-2 projection policy is independent of capability mode: + +```text +manual: auto | explicit | disabled +portable_graph: explicit | disabled +live_viewer: on-demand | disabled +``` + +Set global CLI flags before the subcommand: + +```bash +docforge --project-root "$PROJECT" --manual-render-policy disabled render manual +docforge --project-root "$PROJECT" --portable-graph-policy disabled graph-plan architecture +docforge --project-root "$PROJECT" --live-viewer-policy disabled visualize +``` + +An active operation blocked by policy fails before hidden work. Receipt-only manual and portable +status remain available. Viewer status and explicit stop remain available when viewer startup is +disabled. + +A non-disabled selection also requires its resource: declared manual configuration, declared +portable graph configuration, or viewer runtime. Manual `auto` additionally requires canonical +application. Read [Policy precedence](POLICY_PRECEDENCE.md) for exact composition. + +## Failure and recovery + +Render input drift detected before replacement fails without publishing a current receipt for stale +output. Portable graph publication uses content-addressed evidence and a final manifest commit. +Status never repairs implicitly. + +Recovery is explicit: + +- rerun a manual render from validated canonical state; +- use deep manual status for the full, side-effect-free equivalence oracle; +- rerun portable graph publication, or repair only from validated content-addressed evidence; +- stop and restart a stale live viewer against a current validated index. + +Do not recover a derived output by editing its receipt or treating it as canonical. See [Recovery +and performance](RECOVERY_AND_PERFORMANCE.md), [Security](SECURITY.md), and the [Viewer +manager](VIEWER_MANAGER.md). diff --git a/docs/SECURITY.md b/docs/SECURITY.md new file mode 100644 index 0000000..4c8569e --- /dev/null +++ b/docs/SECURITY.md @@ -0,0 +1,101 @@ +# Security model + +DocForge is a project-bound knowledge compiler. Its security boundary is an explicit project root, +closed configuration, bounded data, and exact identities. It is not a general process sandbox. + +Start with [core authority](CORE_CONCEPTS_AND_AUTHORITY.md), then use +[policy precedence](POLICY_PRECEDENCE.md) to decide which capabilities a server should expose. + +## Project and path confinement + +Descriptors, reference-adapter configurations, canonical sources, authority files, templates, +changesets, caches, indexes, previews, and declared outputs are resolved against one project root. +DocForge rejects absolute paths where only project-relative paths are allowed, parent traversal, +symbolic-link escapes, unsafe file types, and protected-root overlap. Important reads use +no-follow file descriptors and compare file identity before and after reading. + +Confinement protects DocForge operations. It does not stop another process with repository access +from changing files. Long-running bindings revalidate descriptor and adapter implementation +identity and require a restart after drift. + +## Untrusted project content + +Documentation, source text, templates, adapter metadata, compiler-database entries, and changeset +content are data. They cannot redefine policy or instruct DocForge to execute a command. Generic +manual rendering disables raw HTML, accepts a fixed template vocabulary, and rejects script-like +content. Portable graph and manual workers accept validated inert packages and fixed built-in +renderer identities. + +The C++ reference adapter reads `compile_commands.json` only as bounded translation-unit inventory +and fingerprint evidence. It never executes the recorded command, compiler, response file, or +project program. + +## Adapter launcher boundary + +`AdapterLauncherV1` contains one Python module name and project identity. It contains no command, +shell string, arbitrary argument list, working directory, environment, discovery rule, or callable +selector. + +Custom modules must be installed top-level modules. An isolated `python -I` probe resolves the +module without importing it and requires its regular-file origin to remain inside the bound +project. The sole trusted dotted exception is the packaged `docforge.reference_mcp` module. Client +fragments use the exact validated interpreter and canonical fixed arguments with an empty +environment. + +This proves that the declared module is resolvable and project-bound. It does not make arbitrary +module code safe. Project owners remain responsible for the implementation they install. + +## Mutation boundary + +Normal MCP and the fixed reference server are read-only. Proposal tools exist only when a +startup-bound writer is authorized by the descriptor. Canonical application exists only when a +matching applier is explicitly configured. + +Every proposal append, rebase, abandonment, and application is hash-bound. Application requires +the exact changeset hash that was reviewed. Source identity, content hashes, permissions, +conflicts, graph validity, and serializer round trips are checked before success. DocForge never +turns prose approval into a fuzzy merge. + +## Derived state and publication + +SQLite indexes, source-generation receipts, extraction caches, render fragments, previews, and +portable artifacts are disposable. Corrupt, stale, foreign, oversized, or mismatched derived +state is rejected or rebuilt from current project evidence. + +Generated command-reference publication serializes cooperating writers and uses no-clobber or +compare-and-swap publication. A raced target is restored or retained for recovery instead of being +silently discarded. Projection publication records when an artifact was committed but later +receipt verification degraded, so a completed mutation is never reported as an ordinary failure. + +## Limits and denial-of-service resistance + +Inputs, results, traversal, context, changesets, renders, worker protocols, manifests, adapter +assemblies, and extraction caches have explicit count and byte limits. Incremental extraction +caches are capped at 10,000 sources and 64,000,000 bytes. Adapter primary nodes use the project +`max_nodes` limit; edges and Logic have deterministic multipliers over that limit. + +Limits reduce accidental and adversarial amplification. An in-process adapter can still allocate +memory before returning data, so only trusted project-owned adapter code should run in the server +process. + +## Secrets and network behavior + +DocForge does not copy the parent environment into generated client fragments or detached +projection workers. Doctor checks never return environment values. The live viewer binds to +loopback, uses an unguessable URL token, supports read-only methods, and serves no arbitrary +filesystem tree. + +Project secrets must not be placed in canonical documentation, adapter configuration, compiler +databases, templates, or changesets. Repository release gates include secret scanning, but that +scan is not a substitute for credential hygiene. + +## No-AST boundary + +`--no-ast` is a binding policy that preserves the selected adapter and prohibits Logic publication +and retrieval. It is not a parser detector, filesystem sandbox, or promise that unrelated +processes cannot parse source. See [legacy and no-AST operation](LEGACY_AND_NO_AST.md). + +## Reporting and recovery + +Do not bypass a confinement, identity, policy, hash, or limit error. Preserve the failing evidence, +stop the affected binding, and follow [recovery and performance](RECOVERY_AND_PERFORMANCE.md). diff --git a/docs/USER_MANUAL.md b/docs/USER_MANUAL.md index 640f9ae..21f3007 100644 --- a/docs/USER_MANUAL.md +++ b/docs/USER_MANUAL.md @@ -35,6 +35,8 @@ incremental methods while retaining the full loader as a fallback. relationship keys, source inspection, branch-aware node hiding, panel resizing, zooming, and managed idle shutdown. - A generic Markdown/TOML adapter plus contracts for deterministic project-owned adapters. +- A public adapter SDK, optional Python/JavaScript/TypeScript/C++ reference integrations, and one + fixed read-only reference MCP binding. DocForge does not run shell commands from documentation, mutate Git, build an application, deploy, publish, choose a project globally, or cross project boundaries. @@ -100,6 +102,49 @@ uv run pytest -q Use the executables under `/absolute/path/DocForge/.venv/bin/` when DocForge is not installed into the active shell environment. +For an installed distribution, choose only the language extras the project needs: + +```bash +python -m pip install docforge +python -m pip install 'docforge[javascript]' +python -m pip install 'docforge[typescript]' +python -m pip install 'docforge[cpp]' +``` + +The base wheel contains the Python reference adapter and no Tree-sitter distribution. JavaScript, +TypeScript, and C++ require their matching optional extras. `docforge[languages]` installs all +three optional frontend groups. + +### Configure a reference source project + +Reference adapters are a narrow alternative to the generic documentation descriptor. Create +`.docforge/reference-adapter.toml`: + +```toml reference-adapter +schema_version = 1 +project_id = "my-python-project" +title = "My Python Project" +language = "python" +source_roots = ["src"] +``` + +Start the fixed read-only server: + +```bash +python -I -m docforge.reference_mcp \ + --project-root /absolute/path/MyProject \ + --capability-mode read +``` + +For C++, set `language = "cpp"` and add a project-relative +`compilation_database = "compile_commands.json"`. The database is inert bounded inventory; the +reference adapter does not execute its commands or compiler. + +The reference integrations publish syntax and local static relationships only. They do not claim +resolved calls, types, inheritance, macro behavior, compiler include semantics, runtime behavior, +or semantic ownership. See [reference adapters](REFERENCE_ADAPTERS.md) for exact evidence and +limitations. + ### Assess and onboard an unconfigured project Run a read-only assessment before writing configuration: @@ -478,6 +523,10 @@ Every command emits deterministic JSON: docforge --project-root /absolute/path/MyProject ``` +The sections below group common workflows. The implementation-derived list of all 28 current +commands, exact invocations, 36 generic MCP tools, arguments, and input-schema hashes is the +[generated command reference](COMMAND_REFERENCE.md). + ### Project and index commands ```text @@ -590,6 +639,31 @@ declares the named writer, and application requires the same writer/applier iden `--no-ast` to preserve the no-AST binding. Generic CLI generation refuses project-owned adapters because it cannot safely reconstruct their composition. +A custom adapter owner supplies the already constructed project and immutable launcher through the +Python API: + +```python +from docforge.adapter_launcher import AdapterLauncherV1 +from docforge.client_config import generate_adapter_client_configuration + +launcher = AdapterLauncherV1.for_project( + project, + module="my_project_docforge", +) +fragment = generate_adapter_client_configuration( + project, + launcher, + "codex", + capability_mode="read", +) +``` + +The top-level module must be installed for the exact isolated Python environment and resolve to a +regular file inside the project root. The fixed `docforge.reference_mcp` module is the only trusted +dotted exception. Generation probes resolution without importing the custom module, binds current +source availability and policy, and emits no arbitrary command, arguments, working directory, or +environment. + Select projection behavior independently: ```bash @@ -684,6 +758,10 @@ docforge-mcp \ Omit `--proposal-writer` when the MCP client should not create or append proposals. +For `.docforge/reference-adapter.toml`, use the fixed `docforge.reference_mcp` command shown in +[setup](#configure-a-reference-source-project). It exposes exactly the 21 read tools and never +registers proposal or application tools. + Select the session's declared surface explicitly when useful: ```bash @@ -986,6 +1064,28 @@ changes outside this process boundary. ## Troubleshooting +### `optional_dependency_missing` + +Install the exact extra named in the error into the same Python environment that starts DocForge: + +```bash +python -m pip install 'docforge[javascript]' +python -m pip install 'docforge[typescript]' +python -m pip install 'docforge[cpp]' +``` + +Do not install every frontend merely to suppress the check. A missing optional parser is a closed, +actionable capability error and does not affect base generic or Python reference operation. + +### `adapter_launcher_unavailable` or `invalid_adapter_launcher` + +Use one installed top-level Python module whose resolved regular-file origin is inside the project +root, or use the fixed `docforge.reference_mcp` binding. Arbitrary dotted modules, packages, +stdlib modules, missing modules, commands, argument strings, working directories, and environment +injection are rejected. Test the exact generated fragment rather than editing its command by hand. + +See [agent integration](AGENT_INTEGRATION.md) and the [security model](SECURITY.md). + ### `adapter_restart_required` The project-local adapter code, its declared descriptor, or another implementation file changed @@ -1128,8 +1228,11 @@ make gate Use `make benchmark` for the historical Milestone 0 baseline, `make benchmark-m1` for the counter-gated warm-operation benchmark, `make benchmark-m2` for agent workflow gates, and `make benchmark-m3-full` for the ten-sample 1,000-node projection, worker, fragment, status, -equivalence, response-size, and memory gates. `make accessibility` runs the generated manual, -portable graph, and live viewer axe and keyboard flows. +equivalence, response-size, and memory gates. `make benchmark-m4-full` runs the 1,002-node adapter +and recovery benchmark. `make adoption-m4` performs the offline fresh-wheel proof. +`make command-reference-check` rejects command-reference drift, and `make docs-check` validates +the maintained documentation graph. `make accessibility` runs the generated manual, portable +graph, and live viewer axe and keyboard flows. Project-specific vocabulary, extraction rules, and serialization belong in the project adapter. Generic core behavior must remain deterministic, project-bound, and recoverable. diff --git a/tools/docs_pending_m4_pages.txt b/tools/docs_pending_m4_pages.txt index 852d59b..6b179f8 100644 --- a/tools/docs_pending_m4_pages.txt +++ b/tools/docs_pending_m4_pages.txt @@ -1,13 +1 @@ -# Remove each entry in the same commit that authors the required page. -docs/AGENT_INTEGRATION.md -docs/CORE_CONCEPTS_AND_AUTHORITY.md -docs/LEGACY_AND_NO_AST.md -docs/MIGRATING_FROM_V1.md -docs/MILESTONE_4_BASELINE.md -docs/MILESTONE_4_CLOSEOUT.md -docs/POLICY_PRECEDENCE.md -docs/PROJECT_DESCRIPTOR.md -docs/RECOVERY_AND_PERFORMANCE.md -docs/REFERENCE_ADAPTERS.md -docs/RENDERING_AND_VISUALIZATION.md -docs/SECURITY.md +# Milestone 4 required-page inventory is complete.