administrator/DocForge2
1
0
Fork 0
Code Issues Pull requests Projects Releases 2 Packages Wiki Activity Actions Pages

Close Milestone 4 with adapter adoption evidence

Browse source
This commit is contained in:
Andraxion 2026-07-29 15:34:25 -04:00
parent 95271dcf2e
commit 6d06195950
27 changed files with 2870 additions and 325 deletions
Download patch file Download diff file Expand all files Collapse all files

71
ACTIVE_SLICE.md
View file

@ -1,72 +1,15 @@
# Milestone state
```text
Last completed milestone: 3 — independent projections
Evidence: d6d9f47672c513c4927c5c42ad97a175ed651a74 is merged and pushed on main and dev.
Active milestone: 4 — adapter SDK and product documentation.
Last completed milestone: 4 — adapter SDK and product documentation.
Evidence: Frozen executable candidate 95271dcf2e48045b9d3aed9b9ea09c7fc155692c; 347 tests and 402 subtests; 3 accessibility flows; offline fresh-wheel adoption; clean 1,002-node graph-plus-Logic benchmark; no secret-scan findings.
Active milestone: None.
Outcome: New projects can adopt DocForge incrementally, and users can understand the CLI, MCP server, policies, adapters, renderers, and agent integration without reading implementation code.
Status: Active. The first gate freezes and proves the adapter SDK before language references.
Status: Milestone 4 is closed. Milestone 5 has not started.
Next directional milestone: 5 — stabilization and first DocForge2 release.
```
## Milestone 4 contract
Milestone 5 remains directional context. Do not begin release stabilization, versioning, tagging,
or publication without a new active-slice contract based on the Milestone 4 evidence.
Milestone 4 closes only when all of the following are true:
1. A typed public adapter SDK and repository-native conformance kit make complete, incremental,
Logic, confinement, restart, cache-recovery, no-AST, and retrieval contracts usable without
reading core implementation.
2. Repository-owned runnable reference integrations exist for Python, JavaScript/TypeScript, and
C++. They declare their evidence authorities and unsupported facts. Syntax-only evidence never
claims compiler-resolved calls, inheritance, types, or ownership.
3. Heavy language frontends are optional. A base wheel and no-AST operation work without them.
Missing optional features fail with closed, actionable errors.
4. CLI and MCP reference tables are generated deterministically from implementation-owned
metadata and a repository gate rejects drift.
5. Custom adapters can declare one versioned, immutable, project-bound launcher. Codex, Claude,
and OpenClaw client configuration is generated from that launcher without arbitrary discovery
or execution.
6. Dedicated onboarding, authority, descriptor, policy, adapter, language, legacy/no-AST,
rendering, agent-integration, security, recovery/performance, and DocForge v1 migration
guidance passes a fresh-wheel adoption proof.
7. Existing package, import, CLI, MCP, legacy-adapter, generic-project, no-AST, projection,
application, and recovery behavior remains additively compatible.
## Fixed decisions
- Milestone 4 provides stable in-repository SDK APIs and reference integrations. It does not
promise separately published language-adapter distributions.
- Complete adapter evidence includes primary graph and function Logic. An incremental adapter that
publishes Logic must expose an independent complete Logic oracle before equivalence can pass.
- Python reference evidence may use the standard-library AST. JavaScript/TypeScript and C++
reference evidence uses optional pinned parsing frontends for syntax plus declared build
inventory. It omits semantic facts that require a compiler, language server, or SCIP evidence.
- C++ inventory is bound to a project-confined compilation database. WorldForge is not used or
changed.
- JavaScript and TypeScript are both exercised. TypeScript support requires its own optional
pinned grammar; it is not silently parsed as JavaScript.
- Adapter extraction caches remain disposable and may become safely ahead of the last published
SQLite index. The index remains the active validated generation. Recovery must prove this
behavior instead of claiming one joint cache-and-index transaction.
- Generated references and templates are derived artifacts. Implementation metadata is their
authority.
## Exclusions
- No WorldForge, ScrapeStation, legacy-repository, or production-binding change.
- No remote adapter execution, arbitrary adapter commands, renderer ecosystem, render farm,
storage rewrite, or cross-project graph.
- No compiler or language-server installation by DocForge.
- No self-hosting requirement.
- No version, release tag, or Forgejo release before Milestone 5.
## Slice order
1. Freeze complete graph-plus-Logic equivalence, the SDK facade, conformance protocol, schemas, and
baseline measurements.
2. Add deterministic command metadata, generated CLI/MCP references, and a drift gate.
3. Add the Python reference integration and scaled complete/incremental benchmark.
4. Add optional JavaScript/TypeScript and C++ reference integrations with bounded evidence.
5. Add versioned adapter launcher metadata and generated agent configurations.
6. Freeze the candidate, perform one documentation closeout, and run focused, complete,
fresh-wheel, adoption, benchmark, accessibility, build, and secret-scan gates.
The completed Milestone 4 contract and its exclusions are preserved in `SLICE_HISTORY.md`.

105
DEVELOPMENT_NOTES.md
View file

@ -784,3 +784,108 @@ proof. Gitleaks 8.30.1 found no findings across the six Milestone 3 commits or c
Milestone 3 is complete. No tag, release, production integration repointing, WorldForge change,
ScrapeStation change, storage rewrite, or self-hosting dependency was introduced. Milestone 4
remains directional and has not started.
## Milestone 4 — complete: adapter SDK and product documentation
Milestone 4 activated from the verified Milestone 3 closeout. Its scope was deliberately additive:
freeze an adapter-authoring boundary, prove narrow language references, make heavy frontends
optional, generate command and client integration evidence, and document adoption. WorldForge,
ScrapeStation, the legacy repository, production bindings, self-hosting, storage replacement, and
release publication remained excluded.
### Adapter SDK and conformance
`docforge.adapter_sdk` is now the public authoring facade for typed projections, manifests, source
contributions, complete assemblies, incremental loaders, project bindings, core graph models, and
conformance reports. Complete evidence includes primary graph and function Logic. An incremental
adapter that publishes Logic must provide an independent `load_complete_assembly()` oracle.
The conformance helper repeats complete loading for determinism, compares the complete assembly
with `load_projection()`, and proves exact complete/incremental graph-plus-Logic parity. It does not
pretend to replace separate confinement, restart, no-AST, corrupt-cache, or retrieval tests.
An adversarial review found that extraction caches and adapter assemblies lacked aggregate
boundaries. Version-1 cache reads and writes are now regular-file-only, identity-checked, capped at
64,000,000 bytes and 10,000 sources, and fail safely to a miss. Adapter projects enforce primary
node limits and deterministic edge and Logic multipliers before publication.
### Reference language evidence
The Python reference uses the standard-library AST and publishes files, modules, classes,
functions, arguments, local static imports, and function Logic. Its manifest uses tokenization
rather than AST parsing, so unchanged warm builds perform no syntax parse.
JavaScript and TypeScript use separate pinned optional Tree-sitter grammars. They publish files,
modules, classes, functions/methods, static project-relative imports/re-exports, and function
Logic. Their manifests use a closed comment/string-aware ESM scan and do not load Tree-sitter.
C++ uses explicit non-overlapping source roots and one confined `compile_commands.json` as
translation-unit inventory and fingerprint evidence. It validates entries but never executes a
command, compiler, response file, or project program. It publishes syntax and directly resolvable
project-local quoted includes. Its manifest currently parses for include discovery, so no
zero-warm-parser claim is made.
None of the references claim compiler-resolved calls, inheritance, types, symbol references,
compiler include semantics, macro semantics, runtime behavior, or semantic ownership.
The first packaging proof exposed that language parsers were inherited as mandatory dependencies.
The base wheel now depends only on Markdown and MCP packages. Python works in the base wheel;
JavaScript, TypeScript, C++, and all-language extras install their respective Tree-sitter
frontends. Missing extras return a closed error with the exact install target.
### Fixed reference MCP and launchers
A strict `.docforge/reference-adapter.toml` selects one fixed reference language, project
identity, and explicit source roots. C++ additionally requires the compilation database. The fixed
`python -m docforge.reference_mcp` binding constructs only repository-owned providers and exposes
the 21 read tools.
The initial launcher design generated `python -I -m` for arbitrary project module names without
proving that isolated Python could resolve them. The corrected `AdapterLauncherV1` accepts only an
installed project-confined top-level module or the exact trusted `docforge.reference_mcp`
exception. A fixed bounded `find_spec` probe runs under isolated Python without importing project
code. Tests then launch both a real project module and the real reference MCP process.
Generated Codex, Claude, and OpenClaw fragments bind the exact launcher, descriptor, source
availability, effective and projection policy, interpreter, canonical arguments, empty
environment, artifact bytes, and hashes. There is no arbitrary command, argument list, working
directory, environment, discovery, or callable selector.
### Generated reference and documentation gates
CLI tables are derived from the real argparse subcommands. MCP tables are derived from a real
application-enabled server's `list_tools()` registrations, including capability surface,
arguments, descriptions, and input-schema hashes. The generated artifact contains 28 CLI rows and
36 MCP rows.
An adversarial review found a target-change race between the generator's identity check and
replacement. Publication now locks cooperating generators, uses no-clobber publication for a
missing target, and uses Linux atomic exchange plus displaced-byte and file-identity verification
for an existing target. Raced data is restored or retained for recovery rather than discarded.
`docs-check` validates generated drift, local files and anchors, one H1 per page, README-rooted
reachability, required Milestone 4 inventory, the generated notice, and every documented
reference-adapter TOML example against the packaged schema.
### Candidate, adoption, and scale evidence
Frozen executable candidate `95271dcf2e48045b9d3aed9b9ea09c7fc155692c` passed formatting,
Python and web lint, strict Pyright, compilation, 142 contract tests and 268 subtests, 347 total
tests and 402 subtests, three accessibility flows, lock and dependency checks, package builds,
offline wheel adoption, and every maintained smoke benchmark.
The offline adoption proof installed the base wheel from the lock without network access. It
proved no Tree-sitter package or module was installed, built and checked a real Python reference
project, performed isolated MCP bootstrap/search/get-node over the exact 21 read tools, and proved
that base-only C++ fails with `optional_dependency_missing` and `docforge[cpp]` remediation.
The clean 334-source benchmark produced 1,002 primary nodes, 1,001 primary edges, 334 Logic
projections, 2,338 Logic nodes, and 2,338 Logic edges. Complete and incremental output matched
exactly. Warm build p95 was 1,265.387 ms with zero `ast.parse` and zero `extract_source` calls.
Complete equivalence, corrupt cache recovery, and corrupt index recovery were all below 1.9
seconds. The largest per-operation traced peak was 69,997,166 bytes and process high-water was
78,798,848 bytes. Exact hashes and method are recorded in
`benchmarks/milestone4-2026-07-29.json` and `docs/MILESTONE_4_BASELINE.md`.
Milestone 4 is complete. No tag or release was created. Milestone 5 remains unstarted until its own
active-slice contract is frozen.

409
README.md
View file

@ -1,88 +1,107 @@
# DocForge
DocForge is a project-scoped documentation graph for people and AI agents. It validates canonical
documentation, builds a disposable search and relationship index, compiles bounded context, renders
declared manuals, visualizes project structure, and manages reviewable documentation changesets.
DocForge is a project-scoped documentation and source graph for people and AI agents. Canonical
project files remain authoritative; DocForge validates them, builds disposable search and
relationship indexes, compiles bounded context, manages reviewable changesets, renders declared
manuals and portable graph views, and serves one project-bound MCP surface.
## What it does
DocForge never treats indexed text as instructions. It does not run project build commands,
compilers, Git operations, deployments, or arbitrary renderers, and it does not select projects
globally.
- Validates stable Markdown/TOML nodes and typed relationships.
- Builds a deterministic SQLite search and graph index.
- Exposes project-bound CLI and MCP query surfaces.
- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, and bounded
continuation.
- Records one bounded, versioned latest-generation graph transition without creating a history
database.
- Generates deterministic project-bound Codex, Claude, and OpenClaw client fragments without
copying ambient secrets.
- Diagnoses one client binding through bounded read-only checks without starting MCP or rebuilding
project state.
- Automatically synchronizes disposable indexes before MCP work.
- Creates, validates, diffs, and previews isolated changesets.
- Registers complete proposals atomically without caller-managed hash chaining.
- Applies one explicitly approved changeset hash through CLI or gated MCP.
- Supports opt-in incremental adapters with reverse-dependency invalidation and full-build
equivalence checks.
- Detects project-local adapter implementation and configuration changes and requires a fresh
project-bound process before any further MCP work.
- Keeps function-scoped control-flow projections separate from the primary architecture graph.
- Compiles manuals and portable graph artifacts from separate versioned, generation-pinned plans
and immutable packages.
- Runs built-in manual and graph renderers in fixed detached workers with validated receipts,
bounded transfer, and no project-path authority.
- Enforces independent manual, portable-graph, and live-viewer policy while keeping status
receipt-only.
- Runs a managed loopback graph browser with neighborhood, semantic Flow, convergence Web,
function-scoped Logic, source inspection, and branch-aware node hiding.
- Supports generic documentation projects and project-owned source adapters.
DocForge 1.0.0 remains the first stable release in this product line. This checkout is the
post-1.0 development line (`1.3.0.dev0`) and contains the Milestone 4 adapter-platform work.
Milestone 4 is not a new product release, and Milestone 5 release and publication work remains
separate.
DocForge never treats indexed text as instructions. It does not run shell commands, mutate Git,
build applications, deploy, publish, or select projects globally.
## Start here
For implementation projects, the recommended cadence is to read canonical documentation during
intake, keep it read-only through implementation and focused testing, freeze and validate a release
candidate, then perform one atomic documentation closeout before the final commit and tag. This
keeps the manual authoritative without using it as an implementation notebook.
- New installation or first project: [New-project quickstart](docs/NEW_PROJECT_QUICKSTART.md)
- Mental model and authority: [Core concepts and authority](docs/CORE_CONCEPTS_AND_AUTHORITY.md)
- Complete configuration shape: [Project descriptor](docs/PROJECT_DESCRIPTOR.md)
- Fixed runnable examples: [Reference adapters](docs/REFERENCE_ADAPTERS.md)
- Agent and client setup: [Agent integration](docs/AGENT_INTEGRATION.md)
- Exact live command and tool inventory: [Generated command reference](docs/COMMAND_REFERENCE.md)
- Task-oriented operating guide: [User manual](docs/USER_MANUAL.md)
## Release 1
## Current capabilities
DocForge 1.0.0 is the first stable product release. It combines the project-scoped graph, CLI and
MCP query surfaces, reviewable hash-approved changesets, generic and project-owned adapters,
declared rendering, and the complete Nodes/Flow/Web visualization model in one supported release.
- Validates Markdown and TOML nodes, stable IDs, typed relationships, project limits, and confined
paths.
- Builds a deterministic, disposable SQLite graph and search index with integrity and generation
evidence.
- Exposes project-bound CLI and MCP read, proposal, application, rendering, and visualization
surfaces according to the startup policy.
- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, bounded
output, and deterministic continuation.
- Creates isolated documentation changesets, validates complete projected graphs, and applies only
one explicitly approved changeset hash through a separately bound canonical applier.
- Supports complete-projection adapters and opt-in incremental adapters with reverse-dependency
invalidation, bounded extraction caches, and a clean full-build equivalence oracle.
- Publishes function-scoped Logic separately from the primary graph for Python, JavaScript,
TypeScript, and C++ integrations that provide it.
- Compiles manual and portable graph plans into immutable packages for fixed detached renderers,
then records bounded receipts.
- Runs a managed loopback graph viewer with Nodes, Flow, Web, lazy Logic, source inspection, and
branch-aware hiding.
- Generates deterministic Codex, Claude, and OpenClaw client fragments without copying ambient
environment values or secrets.
The post-1.0 incremental compiler is a backward-compatible, optional enhancement. Existing Release
1 adapters that implement only `load_projection()` continue to use the original complete-projection
path without modification. Adapters gain incremental performance only when they additionally
implement the source manifest and extraction methods. Incremental adapters must retain
`load_projection()` as their clean-rebuild fallback and equivalence oracle.
## Adapter platform
MCP bindings can additionally select `--no-ast` when an owner wants to preserve an existing
non-AST adapter. The binding advertises that policy to clients, forbids adapter rewrites that add
AST, Tree-sitter, compiler-AST, or function-Logic extraction, blocks the Logic tool, and rejects
nonempty Logic publication. Complete-projection adapters continue unchanged, and non-AST
incremental fingerprinting and caching remain allowed.
Adapter authors use the public `docforge.adapter_sdk` surface for graph types, complete and
incremental contracts, validation helpers, and `verify_adapter_conformance()`. Existing adapters
that implement only `load_projection()` remain supported. Incremental adapters add a manifest and
source extraction while retaining `load_projection()` as the complete graph oracle. An adapter
that publishes Logic incrementally also supplies `load_complete_assembly()` so the complete oracle
covers both graph and Logic.
DocForge2 bindings may also declare
`--capability-mode read|proposal|application|operator`. Bootstrap returns one versioned effective
policy and the actual startup-gated capabilities. Existing tool surfaces and the legacy no-AST
payload remain compatible.
The repository includes bounded reference integrations for:
`docforge_get_task_context` is an additive read tool for `change`, `implementation`, `failure`,
`ownership`, `test`, `operation`, and `release` work. It derives a closed version-1 retrieval plan,
executes it against one immutable index generation, and returns a hash-bound context capsule.
Project relation names remain authoritative. DocForge classifies only its versioned alias set and
preserves every unknown relation as `unclassified` instead of guessing semantics.
- Python, using the standard-library AST and publishing only project-local imports;
- JavaScript and TypeScript, using their distinct optional Tree-sitter grammars and publishing only
project-local static relative imports and re-exports;
- C++, using `compile_commands.json` as translation-unit inventory and fingerprint evidence, without
executing its commands or a compiler, and publishing only directly resolvable project-local
quoted includes.
`docforge_get_generation_diff` reports the latest verified primary-graph transition through one
bounded disposable receipt. It includes exact node and edge change counts, hash-bound retained
details, and explicit truncation. Paged results use one top-level cursor and a versioned
`receipt_header`; `stored_receipt_hash` identifies the complete persisted receipt. The read never
exposes Logic details, loads canonical source, repairs derived state, or invents history.
These integrations demonstrate the adapter contract; they do not claim resolved calls,
inheritance, types, runtime behavior, macro expansion, compiler include semantics, or semantic
ownership. Python works from the base wheel. Install the `javascript`, `typescript`, or `cpp`
extra for the corresponding grammar, or `languages` for all three:
## Independent projections
```bash
uv pip install "/absolute/path/DocForge[javascript]"
uv pip install "/absolute/path/DocForge[typescript]"
uv pip install "/absolute/path/DocForge[cpp]"
uv pip install "/absolute/path/DocForge[languages]"
```
Manual compilation, portable graph rendering, and the live viewer consume the same validated graph
generation through separate boundaries:
Reference projects use the fixed `.docforge/reference-adapter.toml` descriptor and the installed
`docforge.reference_mcp` module. That server is read-only and exposes no proposal or application
surface.
Project-owned adapters use `AdapterLauncherV1` and
`generate_adapter_client_configuration()` to produce a client fragment from an explicitly
constructed adapter project. The launcher is immutable and contains no command, caller arguments,
working directory, environment, discovery rule, or callable selector. It resolves through isolated
Python to one installed, project-owned top-level module; the only trusted dotted exception is the
fixed `docforge.reference_mcp` binding. Generic `docforge configure` intentionally refuses custom
adapters because it cannot safely reconstruct project-owned composition.
See the [Adapter authoring guide](docs/ADAPTER_AUTHORING_GUIDE.md), [Reference
adapters](docs/REFERENCE_ADAPTERS.md), and [Agent integration](docs/AGENT_INTEGRATION.md) for the
supported routes.
## Authority and projections
Canonical Markdown, TOML, adapter-declared sources, and descriptor files own project facts. The
SQLite index, extraction cache, render packages, previews, portable artifacts, receipts, client
fragments, and viewer processes are derived and replaceable.
The primary graph contains project nodes and relationships. Logic is a separate, lazy,
function-scoped control-flow projection. Manual output, portable graph output, and the live viewer
are independent consumers of one validated generation:
```text
validated generation
@ -91,190 +110,142 @@ validated generation
└── pinned index → managed read-only live viewer
```
Plans, packages, and receipts are canonical, versioned, hash-identified, bounded, and contain no
project object, SQLite handle, absolute project path, command, or caller-selected renderer module.
Renderers cannot select graph facts, crawl canonical sources, choose publication paths, or mutate
project state.
The version-2 projection policy selects each consumer independently:
Declare portable graph output separately from manual views:
```toml
[graph_render]
output_root = ".docforge/portable-graph"
[[graph_render.views]]
id = "architecture"
renderer = "portable_graph_html"
output = "architecture.html"
title = "Architecture"
root = "architecture.overview"
initial_mode = "web"
depth = 3
max_nodes = 250
max_edges = 1000
max_work = 100000
include_logic = false
```text
manual: auto | explicit | disabled
portable_graph: explicit | disabled
live_viewer: on-demand | disabled
```
Plan, publish, and inspect it explicitly:
The process capability policy is separate. Capability mode controls the registered read,
proposal, application, or reserved operator surface; descriptor writers and the startup-bound
canonical applier determine whether mutations are actually authorized. `--no-ast` is a restrictive
binding policy over adapter evolution, Logic publication, and Logic retrieval. It is not a parser
inspection mechanism or a filesystem sandbox, and the AST/Tree-sitter reference integrations
should not be presented as no-AST adapters.
Read [Policy precedence](docs/POLICY_PRECEDENCE.md), [Rendering and
visualization](docs/RENDERING_AND_VISUALIZATION.md), and [Legacy and no-AST
operation](docs/LEGACY_AND_NO_AST.md) before changing those boundaries.
## Five-minute generic project
Requirements are Python 3.12 or newer and an installed DocForge environment. From a development
checkout, `uv sync --group dev` creates `.venv`:
```bash
.venv/bin/docforge --project-root "$PROJECT" graph-plan architecture
.venv/bin/docforge --project-root "$PROJECT" graph-render architecture
.venv/bin/docforge --project-root "$PROJECT" graph-render-status architecture
```
The version-2 projection policy independently selects manual
`auto|explicit|disabled`, portable graph `explicit|disabled`, and live viewer
`on-demand|disabled`. Use `--manual-render-policy`, `--portable-graph-policy`, and
`--live-viewer-policy` on CLI/MCP startup or generated client configuration. Status remains
available when the corresponding active operation is disabled.
Manual fragment reuse is disposable. Cold record publication is guarded by byte-exact comparison
with a full detached render; warm records are independently recomputed and validated inside the
worker. Full rendering remains the recovery and equivalence oracle. Portable artifacts commit
content-addressed output and renderer evidence before one bounded generation/view manifest; status
does not plan or render.
## Graph views
The browser presents the primary architecture graph through three complementary views and loads a
fourth function-scoped view only when requested:
- **Nodes** shows a bounded, relation-neutral neighborhood around the focus. It is the broad
inspection view for seeing stored incoming and outgoing relationships without changing their
direction. Semantic cards distinguish structure, behavior, dependencies, execution, data,
evidence, context, and other relationships.
- **Flow** shows semantic origin-to-destination paths that terminate at the focus. DocForge
reverses prerequisite-style relationships for presentation, so imports, dependencies, reads,
inheritance, definitions, and tests flow toward the thing they help create or exercise.
- **Web** shows the larger convergence picture: Flow contributors plus contextual relationships,
callers, containers, and direct members or execution dependencies owned by the focus.
- **Logic** shows the possible static control paths inside a focused Python, JavaScript, or C++
function or method. Entry, decisions, actions, loops, convergence points, returns, and
exceptions connect through explicit
`TRUE`, `FALSE`, `NEXT`, `CASE`, `LOOP`, `RETURN`, and `RAISE` paths. Logic is stored separately
and does not add statement-level noise to Nodes, Flow, Web, or search.
Graph cards show the node's readable leaf name and kind without clipping either value. The full
qualified identity remains available in the tooltip, compact descriptor, and full inspector.
The left browser panel can combine text, family, node-kind, language, and capability filters.
Quick presets expose Logic-ready nodes, Python callables, tests, routes, and documentation without
requiring users to know stable IDs. Selecting a canvas node emphasizes its directly connected
neighbors and edges while muting unrelated paths.
**Hide node** removes noise without changing the index. In Flow and Web, hiding a contributor also
removes upstream ancestors that no longer have a path to the focus. Nodes between the hidden
contributor and the focus stay visible, and alternate ancestor paths remain intact. In Logic,
hiding a step inserts an explicit omitted-path bridge so downstream control flow remains readable.
**Restore hidden** restores the presentation.
## Five-minute start
Requirements are Python 3.12+, `uv`, and Node.js/npm.
```bash
git clone forgejo@repo.andraxion.net:administrator/DocForge2.git /absolute/path/DocForge2
cd /absolute/path/DocForge2
git clone <repository-url> /absolute/path/DocForge
cd /absolute/path/DocForge
uv sync --group dev
npm ci
PROJECT=/absolute/path/MyProject
.venv/bin/docforge --project-root "$PROJECT" onboard
.venv/bin/docforge --project-root "$PROJECT" onboard \
--scaffold \
--project-id my-project \
--title "My Project"
```
The first command is read-only. Scaffolding is explicit and create-only: it writes a generic
descriptor, one canonical overview node, and a built-in manual template, then indexes and renders
them. Detected source languages remain `adapter_required` until a validated frontend is selected.
Operate the configured project:
```bash
.venv/bin/docforge --project-root "$PROJECT" validate
.venv/bin/docforge --project-root "$PROJECT" reindex
.venv/bin/docforge --project-root "$PROJECT" search architecture
.venv/bin/docforge --project-root "$PROJECT" visualize
```
Install the persistent per-user graph viewer once:
Install the persistent per-user graph viewer manager once when using the live viewer:
```bash
.venv/bin/docforge-viewer-manager install-user-service
```
Start an MCP server for one project:
Start one generic, read-only MCP server:
```bash
.venv/bin/docforge-mcp \
--project-root "$PROJECT" \
--proposal-writer project-editor
--capability-mode read
```
Add `--canonical-applier project-editor` only when that MCP integration should expose the
hash-bound `docforge_apply_changeset` tool.
Preview a read-only Codex fragment and diagnose an installed binding:
```bash
.venv/bin/docforge configure codex --project "$PROJECT"
.venv/bin/docforge doctor --client codex --project "$PROJECT"
```
Pass `--output /absolute/path/docforge.toml` only when creating a standalone fragment. DocForge
never replaces or merges an existing different client file.
For an unconfigured codebase, begin with a read-only language and documentation assessment:
```bash
.venv/bin/docforge --project-root /absolute/path/MyProject onboard
```
Add `--scaffold`, a stable project ID, and a title to create, index, and render a generic starter
manual. Source files are reported separately and require a validated language frontend before
DocForge describes them as a source graph.
Add a descriptor-authorized proposal writer only when the client should create proposals. Add a
matching `--canonical-applier` only when that integration should expose exact-hash application.
See the [quickstart](docs/NEW_PROJECT_QUICKSTART.md) for the reference-adapter and generated-client
paths.
## Documentation
- [User manual](docs/USER_MANUAL.md) — features, setup, visualization, CLI, MCP, apply, adapters,
and troubleshooting.
- [Core contract](docs/CONTRACT.md) — invariants and security boundary.
- [Milestone 0 compatibility](docs/COMPATIBILITY.md) — preserved package, CLI, MCP, adapter,
schema, changeset, rendering, and no-AST guarantees.
- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) — validation evidence, cold and warm
performance, memory, rendering and response sizes, bottlenecks, and missing coverage.
- [Milestone 0 closeout](docs/MILESTONE_0_CLOSEOUT.md) — lineage, migration, security scan,
repository state, and fresh-clone proof.
- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) — warm operation latency, structured work,
status, retrieval, and memory measurements.
- [Milestone 1 closeout](docs/MILESTONE_1_CLOSEOUT.md) — fast-core contracts, adversarial
validation, compatibility boundaries, and exact candidate evidence.
- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) — task context, generation diff, client
configuration, doctor, response-size, counter, and memory measurements.
- [Milestone 2 closeout](docs/MILESTONE_2_CLOSEOUT.md) — implemented contracts, adversarial
validation, exclusions, and exact candidate evidence.
- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) — manual, fragment, worker, portable graph,
status, equivalence, response-size, and memory measurements.
- [Milestone 3 closeout](docs/MILESTONE_3_CLOSEOUT.md) — independent projection contracts,
adversarial validation, compatibility boundaries, and exact candidate evidence.
- [MCP contract](docs/MCP_CONTRACT.md) — exact tool and process boundary.
- [Viewer manager](docs/VIEWER_MANAGER.md) — native service setup and lifecycle.
- [Adapter decision](docs/APPLICATION_DECISION.md) — why custom adapters own canonical
serialization.
- [Incremental adapter indexing](docs/INCREMENTAL_INDEXING.md) — source-scoped extraction,
invalidation, equivalence, relationship changes, and the lazy Logic boundary.
- [Project onboarding](docs/PROJECT_ONBOARDING.md) — repository assessment, safe manual
scaffolding, language frontends, source/manual integration, proof, and MCP activation.
- [Language adapter authoring](docs/ADAPTER_AUTHORING_GUIDE.md) — implementation sequence,
stable identities, overlap ownership, normalization, incremental equivalence, troubleshooting,
and the complete adapter proof matrix.
### Learn and operate
- [New-project quickstart](docs/NEW_PROJECT_QUICKSTART.md)
- [Core concepts and authority](docs/CORE_CONCEPTS_AND_AUTHORITY.md)
- [Project descriptor](docs/PROJECT_DESCRIPTOR.md)
- [Policy precedence](docs/POLICY_PRECEDENCE.md)
- [Reference adapters](docs/REFERENCE_ADAPTERS.md)
- [Agent integration](docs/AGENT_INTEGRATION.md)
- [User manual](docs/USER_MANUAL.md)
- [Generated command reference](docs/COMMAND_REFERENCE.md)
- [Project onboarding](docs/PROJECT_ONBOARDING.md)
- [Rendering and visualization](docs/RENDERING_AND_VISUALIZATION.md)
- [Recovery and performance](docs/RECOVERY_AND_PERFORMANCE.md)
- [Security](docs/SECURITY.md)
### Contracts and compatibility
- [Core contract](docs/CONTRACT.md)
- [MCP contract](docs/MCP_CONTRACT.md)
- [Compatibility contract](docs/COMPATIBILITY.md)
- [Adapter authoring guide](docs/ADAPTER_AUTHORING_GUIDE.md)
- [Incremental indexing](docs/INCREMENTAL_INDEXING.md)
- [Adapter application decision](docs/APPLICATION_DECISION.md)
- [Legacy and no-AST operation](docs/LEGACY_AND_NO_AST.md)
- [Migrating from version 1](docs/MIGRATING_FROM_V1.md)
- [Viewer manager](docs/VIEWER_MANAGER.md)
### Milestone evidence
- [Milestone 4 baseline](docs/MILESTONE_4_BASELINE.md)
- [Milestone 4 closeout](docs/MILESTONE_4_CLOSEOUT.md)
- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) and [closeout](docs/MILESTONE_3_CLOSEOUT.md)
- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) and [closeout](docs/MILESTONE_2_CLOSEOUT.md)
- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) and [closeout](docs/MILESTONE_1_CLOSEOUT.md)
- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) and [closeout](docs/MILESTONE_0_CLOSEOUT.md)
Historical milestone records preserve the facts and dependency observations of their frozen
candidates. Use the current guides and contracts for present behavior.
## Development
Run the complete repository-native gate:
Run the repository-native gate:
```bash
make gate
```
Focused entry points are available as `make contract`, `make test`, `make type`,
`make benchmark-smoke`, `make benchmark`, `make benchmark-m1-smoke`, and
`make benchmark-m1`. Milestone 2 adds `make benchmark-m2-smoke` and `make benchmark-m2`.
Milestone 3 adds `make accessibility`, `make benchmark-m3-smoke`, `make benchmark-m3`, and
`make benchmark-m3-full`.
Milestone 4 maintenance entry points include:
The committed 1,000-node baseline and its measurement method are under `benchmarks/`.
```bash
make adoption-m4
make benchmark-m4-smoke
make benchmark-m4
make benchmark-m4-full
make command-reference-check
make docs-check
```
Pass `--diagnostics` to `docforge` or `docforge-mcp` to attach bounded request-local stage timings
and compiler-work counters. Diagnostics are disabled by default and are dropped before primary MCP
results when the configured output budget is tight.
`adoption-m4` builds and exercises a fresh base wheel without Tree-sitter packages and proves the
Python reference plus a real isolated read-only MCP retrieval. The optional-language integrations
have their own focused tests and extras. `benchmark-m4` runs the maintained full adapter workload;
the smoke target is for routine gate coverage, not final performance evidence.
See [AGENTS.md](AGENTS.md) before changing core boundaries.
Pass `--diagnostics` to `docforge` or `docforge-mcp` for bounded request-local timings and compiler
work counters. Diagnostics are disabled by default and do not displace a primary result when the
configured output budget is tight.
Read [AGENTS.md](AGENTS.md) before changing core boundaries.

45
SLICE_HISTORY.md
View file

@ -1,5 +1,50 @@
# Completed slices
## DocForge2 Milestone 4 adapter SDK and product documentation
### Changed
- Added stable `docforge.adapter_sdk` authoring imports and a conformance helper for deterministic
complete and exact complete/incremental primary graph plus Logic equivalence.
- Added bounded Python, JavaScript, TypeScript, and C++ reference integrations with explicit
unsupported-fact reports and optional heavy frontends.
- Added a closed `.docforge/reference-adapter.toml` contract and fixed read-only reference MCP
server.
- Added immutable project-bound launcher declarations and deterministic Codex, Claude, and OpenClaw
configuration generation for custom adapters.
- Generated CLI and MCP references from live implementation metadata and added race-safe
publication plus repository drift checks.
- Added strict documentation-graph and fresh-wheel adoption gates and dedicated onboarding,
authority, descriptor, policy, adapter, legacy/no-AST, rendering, agent-integration, security,
recovery/performance, and v1 migration guides.
### Verification
- The executable candidate passed 347 tests and 402 subtests, 142 contract tests and 268 subtests,
three accessibility flows, lint, strict types, compilation, dependencies, builds, adoption, and
every maintained smoke benchmark.
- The offline base wheel contained no Tree-sitter distribution and ran a real Python reference
build/check and isolated 21-tool MCP retrieval flow. Missing C++ support failed with the exact
`docforge[cpp]` remediation.
- The clean 334-source benchmark produced 1,002 nodes and exact full/incremental graph and Logic
evidence, warm zero Python parsing/extraction, and exact corrupt-cache and corrupt-index
recovery below every latency, memory, and response gate.
- Gitleaks found no findings in the Milestone 4 commit range or candidate tree.
### Limits
- The reference integrations publish syntax and local static relationships. They do not claim
compiler-resolved calls, types, inheritance, macros, runtime behavior, or semantic ownership.
- C++ compilation-database commands are inert and never executed.
- C++ manifest include discovery uses Tree-sitter and makes no zero-warm-parser claim.
- No WorldForge, ScrapeStation, legacy repository, production binding, language distribution,
self-hosting, tag, or release was changed.
### Next gate
Milestone 5 remains directional. Create a new active-slice contract before stabilization,
versioning, tagging, or publication.
## DocForge2 Milestone 3 independent projections
### Changed

18
benchmarks/README.md
View file

@ -42,6 +42,14 @@ make benchmark-m3
make benchmark-m3-full
```
Run the Milestone 4 adapter SDK smoke, maintained full, and fresh-wheel adoption gates:
```bash
make benchmark-m4-smoke
make benchmark-m4-full
make adoption-m4
```
The benchmark creates canonical sources, derived state, changesets, rendered output, and caches
only in a disposable temporary directory. It does not read another project, self-host DocForge, or
mutate repository content.
@ -71,6 +79,13 @@ sweeps, add/change/delete/reorder equivalence, portable publication, receipt-onl
memory, detached worker peak memory, and response size. Its interpretation is in
[`docs/MILESTONE_3_BASELINE.md`](../docs/MILESTONE_3_BASELINE.md).
`milestone4-2026-07-29.json` is the clean-tree adapter SDK and reference-Python baseline captured
from commit `95271dcf2e48045b9d3aed9b9ea09c7fc155692c`. It measures cold and warm
incremental builds, complete/incremental graph-plus-Logic equivalence, corrupt extraction-cache
recovery, corrupt-index recovery, parser and extraction counters, traced and process memory, and
bounded response size. Its interpretation is in
[`docs/MILESTONE_4_BASELINE.md`](../docs/MILESTONE_4_BASELINE.md).
The generic fixtures expose whole-source and projection scaling. They do not replace incremental
adapter equivalence tests. Milestone 3's portable fixture contains 1,000 Nodes/Flow/Web nodes and
999 edges; portable version 1 deliberately excludes Logic.
@ -89,4 +104,5 @@ backlinks, both traversal directions, paged context, render receipt states, and
freshness. The reported p95 uses the nearest-rank method; with ten samples it is the maximum.
`process_peak_rss_kib` is the cumulative main-process `RUSAGE_SELF` high-water mark and excludes the
detached viewer worker. It is diagnostic and not operation-local. Milestone 3 memory gates use
per-operation `tracemalloc` peaks and detached worker receipt peaks instead.
per-operation `tracemalloc` peaks and detached worker receipt peaks instead. Milestone 4 uses
per-operation `tracemalloc` peaks and a cumulative benchmark-process high-water gate.

205
benchmarks/milestone4-2026-07-29.json Normal file
View file

@ -0,0 +1,205 @@
{
"benchmark": "docforge2_milestone4",
"environment": {
"implementation": "CPython",
"machine": "x86_64",
"platform": "Linux-7.1.3-200.nobara.fc44.x86_64-x86_64-with-glibc2.43",
"python": "3.14.6"
},
"evidence": {
"cache_recovery_exact": true,
"complete_assembly_hash": "2888182ab765fbffe3ba873c1613345640e8e6d89be74ddfca7452c0a5056345",
"edge_count": 1001,
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
"full_incremental_graph_and_logic_exact": true,
"index_recovery_exact": true,
"logic_edge_count": 2338,
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
"logic_node_count": 2338,
"logic_projection_count": 334,
"node_count": 1002,
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
"source_count": 334,
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
"warm_zero_ast_parse": true,
"warm_zero_source_extraction": true
},
"evidence_sha256": "4a8db461a311b5df4abd5aa00063e9a347d8b9dba19b30e35684f561d5271549",
"fixture": {
"expected_node_count": 1002,
"kind": "synthetic_python_reference_adapter",
"nodes_per_source": 3,
"source_count": 334
},
"memory": {
"process_peak_bytes": 78798848,
"process_peak_limit_bytes": 536870912
},
"method": {
"ast_instrumentation": "temporary counter around stdlib ast.parse",
"clock": "time.perf_counter_ns",
"extraction_instrumentation": "reference adapter extract_source entry counter",
"full_mode_source_requirement": 334,
"in_process_peak_memory": "tracemalloc per measured invocation",
"process_peak_memory": "resource.getrusage(RUSAGE_SELF).ru_maxrss",
"response_size": "UTF-8 bytes of compact sorted JSON",
"samples": 3,
"threshold_basis": "Regression tripwires leave several times the Milestone 3 1,000-node allowances for AST, Logic, extraction-cache, and SQLite work."
},
"mode": "full",
"operations": {
"cold_incremental_build": {
"max_ms": 1599.76,
"maximum_response_bytes": 14580,
"maximum_traced_peak_bytes": 68573540,
"median_ms": 1599.76,
"min_ms": 1599.76,
"p95_limit_ms": 30000.0,
"p95_ms": 1599.76,
"response_limit_bytes": 262144,
"samples": 1,
"stable_result": {
"ast_parse_calls": 668,
"cache_hits": 0,
"deleted_sources": 0,
"edge_count": 1001,
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
"extraction_calls": 334,
"invalidated_sources": 334,
"logic_edge_count": 2338,
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
"logic_node_count": 2338,
"logic_projection_count": 334,
"node_count": 1002,
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
"reparsed_sources": 334,
"revision": "30ef23aa4106",
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
"status": "ok",
"total_sources": 334
},
"traced_peak_limit_bytes": 268435456
},
"corrupt_extraction_cache_recovery": {
"max_ms": 1695.406,
"maximum_response_bytes": 14580,
"maximum_traced_peak_bytes": 69776923,
"median_ms": 1695.406,
"min_ms": 1695.406,
"p95_limit_ms": 30000.0,
"p95_ms": 1695.406,
"response_limit_bytes": 262144,
"samples": 1,
"stable_result": {
"ast_parse_calls": 668,
"cache_hits": 0,
"deleted_sources": 0,
"edge_count": 1001,
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
"extraction_calls": 334,
"invalidated_sources": 334,
"logic_edge_count": 2338,
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
"logic_node_count": 2338,
"logic_projection_count": 334,
"node_count": 1002,
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
"reparsed_sources": 334,
"revision": "30ef23aa4106",
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
"status": "ok",
"total_sources": 334
},
"traced_peak_limit_bytes": 268435456
},
"corrupt_index_recovery": {
"max_ms": 1707.879,
"maximum_response_bytes": 14779,
"maximum_traced_peak_bytes": 68561924,
"median_ms": 1707.879,
"min_ms": 1707.879,
"p95_limit_ms": 30000.0,
"p95_ms": 1707.879,
"response_limit_bytes": 262144,
"samples": 1,
"stable_result": {
"action": "rebuilt",
"ast_parse_calls": 0,
"cache_hits": 334,
"edge_count": 1001,
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
"extraction_calls": 0,
"initial_error_code": "invalid_index",
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
"logic_projection_count": 334,
"node_count": 1002,
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
"reparsed_sources": 0,
"revision": "30ef23aa4106",
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
"status": "ok"
},
"traced_peak_limit_bytes": 268435456
},
"full_incremental_equivalence": {
"max_ms": 1897.919,
"maximum_response_bytes": 230,
"maximum_traced_peak_bytes": 64305919,
"median_ms": 1897.919,
"min_ms": 1897.919,
"p95_limit_ms": 30000.0,
"p95_ms": 1897.919,
"response_limit_bytes": 262144,
"samples": 1,
"stable_result": {
"ast_parse_calls": 1336,
"edge_count": 1001,
"extraction_calls": 668,
"logic_projection_count": 334,
"node_count": 1002,
"project_id": "milestone4-python-benchmark",
"revision": "30ef23aa4106",
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
"status": "ok"
},
"traced_peak_limit_bytes": 268435456
},
"warm_incremental_build": {
"max_ms": 1265.387,
"maximum_response_bytes": 14578,
"maximum_traced_peak_bytes": 69997166,
"median_ms": 1244.427,
"min_ms": 1239.941,
"p95_limit_ms": 20000.0,
"p95_ms": 1265.387,
"response_limit_bytes": 262144,
"samples": 3,
"stable_result": {
"ast_parse_calls": 0,
"cache_hits": 334,
"deleted_sources": 0,
"edge_count": 1001,
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
"extraction_calls": 0,
"invalidated_sources": 0,
"logic_edge_count": 2338,
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
"logic_node_count": 2338,
"logic_projection_count": 334,
"node_count": 1002,
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
"reparsed_sources": 0,
"revision": "30ef23aa4106",
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
"status": "ok",
"total_sources": 334
},
"traced_peak_limit_bytes": 268435456
}
},
"schema_version": 1,
"source": {
"dirty": false,
"revision": "95271dcf2e48045b9d3aed9b9ea09c7fc155692c"
}
}

76
docs/ADAPTER_AUTHORING_GUIDE.md
View file

@ -8,6 +8,30 @@ Use this guide after the repository assessment in
[Project Onboarding](PROJECT_ONBOARDING.md). The onboarding checklist decides whether an adapter
is needed. This guide defines how to build and prove one.
## Public authoring surface
Adapter authors should import typed contracts, validation helpers, graph models, and the
conformance helper from `docforge.adapter_sdk`:
```python
from docforge.adapter_sdk import (
AdapterAssembly,
AdapterImplementation,
AdapterManifest,
AdapterProject,
AdapterProjectSettings,
AdapterProjection,
AdapterSource,
AdapterSourceProjection,
LogicProjection,
verify_adapter_conformance,
)
```
Language-specific modules under `docforge.adapters` are repository reference implementations, not
the general authoring namespace. Their exact, deliberately narrow behavior is documented in
[Reference Adapters](REFERENCE_ADAPTERS.md).
## Required outcome
A production adapter must provide one reproducible public graph from authoritative project
@ -51,11 +75,39 @@ class MyAdapter:
manifest: AdapterManifest,
contributions: tuple[AdapterSourceProjection, ...],
) -> AdapterAssembly: ...
def load_complete_assembly(self) -> AdapterAssembly: ...
def load_projection(self) -> AdapterProjection: ...
```
`assemble_projection()` is optional only when extraction units already have disjoint ownership.
`load_projection()` is always required. It is the clean rebuild and equivalence oracle.
`load_projection()` is always required. It is the clean primary-graph compatibility oracle.
`load_complete_assembly()` supplies the cache-independent complete graph-plus-Logic oracle. It is
required when incremental contributions publish Logic; without it, DocForge cannot prove Logic
parity.
## What the conformance helper proves
Run the public helper against a fresh confined cache root:
```python
report = verify_adapter_conformance(
MyAdapter(project_root),
cache_root=project_root / ".docforge" / "cache" / "adapter-conformance",
)
```
`verify_adapter_conformance()` proves:
- two repeated complete assemblies are exactly deterministic, including Logic;
- when `load_complete_assembly()` exists, its primary graph exactly matches
`load_projection()`; and
- when the incremental methods exist, the assembled incremental graph and Logic exactly match the
independent complete oracle.
The report records stable identity, counts, and an assembly hash. This helper does not by itself
prove process confinement, implementation restart behavior, corrupt-cache recovery, warm zero
parsing, retrieval, or every case in the proof matrix below. Keep those as separate focused and
integration tests.
## Step 1: define authority before parsing
@ -324,6 +376,25 @@ The manifest remains a current source snapshot, not a Git-index snapshot. A Git-
omit a deleted source whether its deletion is unstaged or staged. Staging is never a required
DocForge synchronization step.
### Current aggregate bounds
DocForge limits one extraction-cache generation to 10,000 source contributions and 64,000,000
encoded bytes. Malformed, incompatible, missing, oversized, or unsafe cache data is treated as a
cache miss.
The assembled graph is bounded by the effective project `Limits.max_nodes`. When adapter settings
do not override limits, DocForge selects at least 10,000 nodes and raises that bound to the
manifest's `estimated_nodes` when larger. Aggregate assembly ceilings are:
- nodes: `max_nodes`;
- relationships: `max_nodes * 32`;
- Logic nodes across all functions: `max_nodes * 32`; and
- Logic edges across all functions: `max_nodes * 64`.
Adapters should set realistic limits and fail before retaining unbounded frontend evidence. The
cache and assembly bounds do not replace each adapter's own bounded source, command, parser, or
per-function limits.
## Step 9: keep the complete path independent
The full rebuild must not read the incremental extraction cache. Otherwise equivalence compares
@ -470,7 +541,8 @@ For every such change:
- [ ] Two independent complete builds match exactly.
- [ ] Incremental extraction uses authoritative dependencies.
- [ ] The complete oracle is independent of the cache.
- [ ] Complete and incremental public projections match exactly.
- [ ] Complete and incremental public graph and Logic projections match exactly.
- [ ] `verify_adapter_conformance()` passes, and the separate proof-matrix cases also pass.
- [ ] Corrupt, missing, and interrupted cache cases fail safely.
- [ ] Session composition and family isolation are proven.
- [ ] Viewer, query, context, and Logic retrieval are proven.

105
docs/AGENT_INTEGRATION.md Normal file
View file

@ -0,0 +1,105 @@
# Agent integration
DocForge generates deterministic, project-bound MCP client fragments for Codex, Claude, and
OpenClaw. Generic projects and custom adapters use different configuration APIs, but both produce
fixed standard-input/output bindings with empty generated environments and explicit capability
policy.
## Fixed reference binding
Configure one of the in-repository adapters with
`.docforge/reference-adapter.toml` as described in
[Reference Adapters](REFERENCE_ADAPTERS.md), then start:
```bash
python -I -m docforge.reference_mcp \
--project-root /absolute/path/to/project \
--capability-mode read
```
`docforge.reference_mcp` is a fixed trusted module in the installed DocForge distribution. It
selects a provider only from the validated reference configuration and reports binding metadata
containing:
- `server_module = "docforge.reference_mcp"`;
- `adapter_mode = "reference"`;
- the selected `reference_language`; and
- the exact `reference_config_hash`.
The reference binding registers exactly the 21 read tools listed in
[MCP Boundary](MCP_CONTRACT.md#read-tools). It has no proposal or canonical-application surface.
## Generate a reference client fragment
The public launcher and generator APIs are:
```python
from pathlib import Path
from docforge.adapter_launcher import AdapterLauncherV1
from docforge.client_config import generate_adapter_client_configuration
from docforge.reference_mcp import REFERENCE_MCP_MODULE, create_reference_project
root = Path("/absolute/path/to/project")
project = create_reference_project(root)
launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE)
plan = generate_adapter_client_configuration(
project,
launcher,
"codex", # "codex", "claude", or "openclaw"
capability_mode="read",
)
print(plan["artifact"]["content"])
```
Pass `output=Path(...)` only when the caller has selected an exact destination. Publication is an
atomic create-or-exact-match operation; it does not merge or replace different existing content.
The result includes the launcher, source-availability, project, policy, artifact, and configuration
hashes needed to inspect the binding before use.
The fixed reference module accepts read mode only. Do not request proposal or application mode for
it.
## Custom project-owned adapter launchers
Generic `docforge configure` intentionally refuses a custom adapter project. Construct the
project-owned `ProjectService`, then use `AdapterLauncherV1` and
`generate_adapter_client_configuration()` as the custom-adapter route.
`AdapterLauncherV1` schema version 1 binds:
- one project ID, canonical absolute project root, adapter identity, and descriptor hash;
- `entry_point = "python-module"`; and
- one installed project-owned top-level Python module.
The only trusted dotted-module exception is the fixed `docforge.reference_mcp` binding. A
project-owned module is resolved through isolated Python without importing or executing it during
the probe, and it must resolve to one canonical regular `.py` file inside the project root.
The launcher contract has no arbitrary command, command arguments, shell string, working
directory, environment, callable selector, discovery rule, or module-reload mechanism. Generated
bindings invoke the current Python executable as `python -I -m <module>` with only DocForge's
validated project, capability, render-policy, authority, and no-AST options. Source identity and
the project binding are revalidated before publication; drift fails closed.
Proposal and application modes are available only to a custom module that implements those fixed
server arguments and only when the project descriptor declares the matching writer and canonical
applier authority. Generating a mode does not manufacture that authority.
## Session workflow
After registering the generated fragment in the selected client:
1. Start a new MCP process or client session.
2. Call `docforge_bootstrap`.
3. Verify project ID, root fingerprint, adapter identity, revision, source hash, binding metadata,
effective policy, and registered tools.
4. Retrieve exact or bounded project evidence through the fixed read tools.
5. Restart the process if `adapter_restart_required` reports an implementation or configuration
change.
Document text returned by DocForge is untrusted project content. It never overrides client, user,
or project authority. See [MCP Boundary](MCP_CONTRACT.md) for synchronization, pagination,
retrieval, proposal, and application rules, and
[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) before selecting `no_ast=True`.

25
docs/COMPATIBILITY.md
View file

@ -31,16 +31,31 @@ model, index, rendering, application, and MCP factory names imported from these
remain supported:
- `docforge.adapter_contract`
- `docforge.adapter_sdk`
- `docforge.adapter_launcher`
- `docforge.application`
- `docforge.client_config`
- `docforge.index`
- `docforge.mcp_server`
- `docforge.models`
- `docforge.policy`
- `docforge.render_contract`
- `docforge.reference_config`
- `docforge.reference_mcp`
Names beginning with an underscore are implementation details. New public names may be added
without breaking this contract.
The repository Python, JavaScript/TypeScript, and C++ adapters are supported reference
implementations. Their documented configuration, evidence limits, and unsupported-fact reports are
compatibility surfaces; their internal parser helpers are not adapter-authoring imports.
Milestone 4 adds three version-1 schemas without changing existing descriptor or result schemas:
- `schemas/adapter-launcher.schema.json`
- `schemas/adapter-client-configuration.schema.json`
- `schemas/reference-adapter.schema.json`
## CLI and MCP surfaces
Existing `docforge` command names and arguments remain supported. Existing `docforge-mcp` tool
@ -233,6 +248,11 @@ Milestone 3 adds `ManualRenderPlanV1`, `GraphViewPlanV1`, projection package and
and projection policy version 2. These are additive submodule and schema contracts. They do not
change the legacy task-context, adapter, changeset, or effective-policy contracts described above.
Milestone 4 adds `docforge.adapter_sdk`, the fixed reference configuration and read-only MCP
binding, immutable adapter launchers, and generated adapter client fragments. A legacy adapter with
only `load_projection()` remains first-class and need not adopt incremental extraction, Logic, a
reference configuration, or launcher metadata.
## Safety boundary
DocForge remains bound to one explicit project root. It rejects absolute paths, root escapes, and
@ -245,8 +265,9 @@ publication, or project switching.
Milestone 0 records rather than redesigns these areas:
- Generic warm reads still repeat whole-project discovery, parsing, and validation.
- Tree-sitter and the JavaScript and C++ grammars remain mandatory installation dependencies even
when their runtime modules are unused.
- The base wheel intentionally omits Tree-sitter. JavaScript, TypeScript, and C++ syntax evidence
requires the matching `docforge[javascript]`, `docforge[typescript]`, or `docforge[cpp]` extra.
Python reference evidence uses the standard library and remains available in the base wheel.
- Several version strings and defaults remain duplicated.
- One individually oversized context entry is represented as explicit bounded omission evidence;
callers use targeted retrieval for that node.

39
docs/CONTRACT.md
View file

@ -29,6 +29,9 @@ commit when Git is available; it cannot change repository state.
- Projection package: `schemas/projection-package.schema.json`, version 1.
- Projection receipt: `schemas/projection-receipt.schema.json`, version 1.
- Independent projection policy: `schemas/projection-policy.schema.json`, version 2.
- Adapter launcher: `schemas/adapter-launcher.schema.json`, version 1.
- Adapter client configuration: `schemas/adapter-client-configuration.schema.json`, version 1.
- Reference adapter configuration: `schemas/reference-adapter.schema.json`, version 1.
- Index schema: version 3, disposable and reproducible.
- Index attestation: schema version 1, disposable and reproducible.
- Core, CLI, and MCP server: version 1.3.0.dev0.
@ -103,6 +106,42 @@ complete projection, opens SQLite, starts MCP, executes the configured command,
builds, renders, starts a viewer, or writes configuration. Unprovable client behavior is a warning,
not an invented success.
## Public adapter SDK and reference binding
`docforge.adapter_sdk` is the stable adapter-authoring import boundary. It exposes the typed
projection, manifest, source contribution, complete assembly, project wrapper, graph model, and
conformance contracts needed by an adapter without requiring authors to import core implementation
modules.
Complete evidence includes the primary graph and function Logic. An incremental adapter that
publishes Logic must implement `load_complete_assembly()` as an independent complete oracle.
`verify_adapter_conformance()` proves repeated complete determinism, equality between the complete
assembly and `load_projection()`, and exact complete/incremental graph-plus-Logic parity. Separate
tests remain responsible for confinement, restart behavior, no-AST behavior, cache recovery, and
retrieval.
Adapter assemblies are bounded before publication. Primary nodes use the descriptor `max_nodes`
limit. Primary edges, Logic nodes, and Logic edges use fixed deterministic multipliers over that
limit. Version-1 extraction caches are regular-file-only, bounded to 10,000 sources and
64,000,000 bytes, and are treated as misses when corrupt, oversized, foreign, or incompatible.
`.docforge/reference-adapter.toml` is a closed version-1 selection among `python`, `javascript`,
`typescript`, and `cpp`. It declares one project identity and explicit non-overlapping source
roots. C++ additionally requires a confined `compile_commands.json`. It cannot declare a command,
module, environment, writer, applier, or remote endpoint.
`python -m docforge.reference_mcp --project-root ROOT` constructs only the selected fixed
repository reference adapter and exposes the read surface. It never registers proposal or
application tools.
`AdapterLauncherV1` is an immutable project-bound Python-module declaration. It accepts no
arbitrary command, arguments, working directory, environment, discovery, callable selector, or
module reload. Custom launchers resolve one installed top-level module through isolated Python and
require its origin inside the project root. The fixed `docforge.reference_mcp` module is the only
trusted dotted exception. `generate_adapter_client_configuration()` binds generated Codex,
Claude, and OpenClaw fragments to that launcher, current source availability, effective policy,
descriptor, interpreter, and exact artifact bytes.
## Isolated proposal model
Create, update, move, and delete are ordered node operations inside an isolated changeset. Every

165
docs/CORE_CONCEPTS_AND_AUTHORITY.md Normal file
View file

@ -0,0 +1,165 @@
# Core concepts and authority
DocForge is a project-bound knowledge compiler. It turns explicit canonical project facts into
validated graphs and bounded derived views without transferring authority to the index, an agent,
or a renderer.
## One project, one explicit root
Every operation is bound to one canonical real project directory. Paths in descriptors and adapter
configuration are project-relative and confined beneath that root. A CLI or MCP process does not
discover or switch projects after startup.
The project root determines:
- which descriptor and canonical sources may be read;
- where derived cache and changeset roots may exist;
- which project identity, revision, source hash, and generation appear in results;
- which writer, applier, rendering, and viewer policies can be selected.
Generated client fragments preserve that binding. They are machine-local configuration
projections, not portable project authority.
## Canonical facts and derived evidence
Canonical inputs own the facts:
- generic Markdown or TOML node sources;
- authority files named by a generic descriptor;
- an adapter's declared canonical sources and implementation boundary;
- `.docforge/project.toml` for a generic project;
- `.docforge/reference-adapter.toml` for a fixed reference integration.
Everything DocForge builds from those inputs is derived:
- SQLite indexes, attestations, generation receipts, and generation diffs;
- incremental extraction caches;
- task-context capsules and query responses;
- changeset previews;
- render plans, immutable packages, fragments, artifacts, and receipts;
- portable graph publications and live-viewer processes;
- generated Codex, Claude, and OpenClaw fragments.
Derived state may be discarded and rebuilt. A derived artifact can prove what it was bound to, but
it cannot override current canonical content.
## Nodes, relationships, and Logic
The primary graph contains nodes and directed relationships.
A node has a stable project-wide ID, title, family, authority, status, tags, summary, content,
source identity, and content hash. A generic Markdown file contains one node beneath a TOML
metadata block; a generic TOML source may contain multiple nodes. Adapter nodes use the same public
graph contract.
A relationship is an exact `(source, relation, target)` triple. The project descriptor defines the
allowed relation names. DocForge gives `depends_on` special acyclic validation, but it does not
invent domain meaning for a project's other relation names. Retrieval recognizes only a versioned
alias set for task planning and reports unknown allowed relations as `unclassified`.
Logic is deliberately separate. It is a lazy function-scoped control-flow projection owned by one
primary node. Decisions, actions, loops, convergence points, returns, and exceptions connect
through explicit branch edges. Logic does not add statement-level nodes to ordinary Nodes, Flow,
Web, search, or generation-diff results. Python, JavaScript, TypeScript, and C++ integrations may
publish Logic when their adapter contract supports it.
## Authority, status, family, and tags
These fields answer different questions:
- `authority` describes the role of the content. The generic vocabulary is `authoritative`,
`approved_plan`, `derived`, `proposal`, and `historical`.
- `status` is project-defined lifecycle state such as `current`, `active`, or `verified`.
- `family` is a project-defined content grouping used for filtering, profiles, rendering, and
writer permissions.
- `tags` are exact project labels for retrieval and presentation.
An `authoritative` node can still become stale; authority is not a freshness claim. A `derived`
node is still canonical if it is stored in a declared canonical source; the label describes its
role, not whether DocForge may silently regenerate it. Status and authority never grant an MCP
writer permission.
## Validation, synchronization, and generations
Validation loads the complete canonical graph and rejects unsafe paths, invalid source formats,
duplicate IDs, unresolved relationships, prohibited cycles, violated project limits, and
adapter-specific contract failures.
The disposable index is published atomically only after the complete graph and SQLite integrity
checks pass. Its attestation binds the whole index file. A successful replacement is the derived
publication commit point; later receipt-writing trouble is reported as degraded evidence rather
than as a false claim that the replacement failed.
MCP operations automatically synchronize derived state under a project lock before normal work.
A graph generation identifies one validated indexed snapshot. Generation-pinned retrieval and
rendering do not reopen mutable sources behind an older snapshot.
The latest generation diff is one bounded primary-graph transition, not a history database. It
contains no Logic details or source text.
## Complete and incremental adapters
`load_projection()` is the compatibility baseline and clean graph oracle. An incremental adapter
adds:
- `load_manifest()` for cheap project identity, source inventory, fingerprints, and dependencies;
- `extract_source()` for one cacheable source contribution;
- optionally `assemble_projection()` to normalize overlapping contributions.
When incremental contributions contain Logic, `load_complete_assembly()` supplies a
cache-independent complete graph-plus-Logic oracle. Warm cache behavior is an optimization, never a
different authority path. Corrupt or incompatible extraction caches are treated as misses, and a
clean complete build remains the equivalence and recovery boundary.
The public types, validators, and conformance helper are exported from `docforge.adapter_sdk`. See
the [Adapter authoring guide](ADAPTER_AUTHORING_GUIDE.md) and [Incremental
indexing](INCREMENTAL_INDEXING.md).
## Proposals are not canonical changes
A changeset is an isolated, ordered proposal over an exact canonical base. Each operation names
preconditions, and the final changeset has a content-derived hash. Validation projects the complete
resulting graph before application.
Canonical application requires:
1. a writer declared in the descriptor;
2. a process started with the matching proposal and application authority;
3. one explicit final changeset hash;
4. unchanged source, relationship, permission, and graph preconditions.
Application does not perform Git mutation, build, deployment, or publication. Until exact-hash
application succeeds, canonical project files remain unchanged.
## Three independent output projections
Manual rendering, portable graph publication, and the live viewer are separate:
- a manual is a declared derived HTML view over selected nodes;
- a portable graph is a content-addressed static Nodes, Flow, or Web artifact;
- the live viewer is a managed loopback process pinned to one validated index generation and can
request lazy Logic.
Their policies compose independently. Disabling one does not transfer its authority to another.
None is canonical documentation or a retrieval authority. See [Rendering and
visualization](RENDERING_AND_VISUALIZATION.md).
## Binding policy is not project truth
Capability mode, no-AST preservation, diagnostics, and projection modes describe one running
process or generated client binding. They do not rewrite the descriptor or canonical graph.
`--no-ast` forbids AST-family adapter evolution and Logic publication/retrieval for that binding.
It does not inspect parser implementation, sandbox the filesystem, or convert an existing
AST/Tree-sitter adapter into a no-AST adapter. Read [Policy precedence](POLICY_PRECEDENCE.md) and
[Legacy and no-AST operation](LEGACY_AND_NO_AST.md).
## Trust the narrowest evidence
DocForge reports stable IDs, hashes, generations, omissions, truncation, and provenance limits so a
consumer can distinguish proof from inference. A source inventory is not a semantic graph; a
syntax-level relationship is not compiler resolution; a configuration doctor is not a connection
test; a viewer snapshot is not continuous monitoring; and a benchmark is not a release.
Continue with the [Project descriptor](PROJECT_DESCRIPTOR.md), [New-project
quickstart](NEW_PROJECT_QUICKSTART.md), or [Core contract](CONTRACT.md).

48
docs/INCREMENTAL_INDEXING.md
View file

@ -4,6 +4,10 @@ DocForge Release 1 adapters return one complete immutable projection. That contr
supported. The incremental compiler adds an opt-in source-scoped contract that avoids reparsing
unchanged files while preserving the same validated, atomically published graph.
Import these contracts from the public `docforge.adapter_sdk` facade. See
[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for the preserved one-method contract and
[Reference Adapters](REFERENCE_ADAPTERS.md) for the four maintained implementations.
## Release 1 compatibility
The incremental interface is additive:
@ -34,18 +38,26 @@ Incremental indexing is an extraction optimization. It does not weaken publicati
An interrupted extraction never replaces the last validated SQLite index. A malformed,
incompatible, or missing cache is a cache miss, not a partial graph.
One cache generation is bounded to 10,000 source contributions and 64,000,000 encoded bytes.
Aggregate graph and Logic assembly limits are described in the
[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md#current-aggregate-bounds).
## Adapter contract
An incremental loader implements all three methods:
An incremental loader implements the first, second, and fourth methods. It implements
`load_complete_assembly()` as well when it publishes Logic:
```python
class MyAdapter:
def load_manifest(self) -> AdapterManifest: ...
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ...
def load_complete_assembly(self) -> AdapterAssembly: ...
def load_projection(self) -> AdapterProjection: ...
```
`load_projection()` remains the deterministic full-rebuild fallback and equivalence oracle.
`load_projection()` remains the deterministic full-rebuild primary-graph oracle. An incremental
adapter that publishes Logic must additionally implement `load_complete_assembly()` as the
cache-independent complete graph-plus-Logic oracle.
Each `AdapterSource` declares:
@ -137,7 +149,8 @@ fresh process must import and validate the current adapter.
```
Adapters can call `AdapterProject.verify_incremental_equivalence()` in release and contract tests.
The check compares project identity, revision, source hash, nodes, and relationships against
The check compares project identity, revision, source hash, nodes, relationships, and Logic against
the independent complete assembly. It also requires the complete assembly's primary graph to match
`load_projection()`.
## Manual changes and relationships
@ -170,12 +183,29 @@ logic empty until they implement a language analyzer.
This boundary prevents thousands of boolean expressions and basic blocks from polluting Nodes,
Flow, Web, ordinary search, or architectural traversal. The Logic tab and `docforge_get_logic`
request one function-scoped projection on demand. The built-in analyzers cover Python,
JavaScript, and C++. Python uses the standard-library AST. JavaScript and C++ share pinned
Tree-sitter infrastructure with thin language-aware control-flow profiles. Parsers run only while
extracting a changed source contribution; ordinary graph reads do not load or execute them. A
grammar alone supplies syntax, not control-flow meaning, so each new language still needs a small
semantic profile for its branch, loop, case, exception, and termination constructs. All analyzers
report possible static paths; they do not claim runtime branch outcomes.
JavaScript, TypeScript, and C++. Python uses the standard-library AST. JavaScript, TypeScript, and
C++ use distinct optional Tree-sitter grammars with thin language-aware control-flow profiles.
Ordinary graph reads use stored projections and do not load or execute these parsers. A grammar
alone supplies syntax, not control-flow meaning, so each new language still needs a profile for
its branch, loop, case, exception, and termination constructs. All analyzers report possible
static paths; they do not claim runtime branch outcomes.
## Manifest and warm-parser scope
Parser work is language-specific and must be measured at the correct boundary:
- Python manifest construction fingerprints source and tokenizes local imports without calling
`ast.parse`.
- JavaScript and TypeScript manifest construction lexes static relative module specifiers without
invoking their distinct Tree-sitter extraction parsers. Focused tests prove this behavior on an
unchanged warm build.
- The C++ reference manifest parses inventoried sources with `tree-sitter-cpp` to discover quoted
include dependencies. A warm C++ extraction-cache hit is not a zero-parser claim.
The maintained Python benchmark instruments the unchanged warm path and requires zero
`ast.parse` calls and zero `extract_source` calls. That exact zero-parser benchmark claim is Python
only. JavaScript and TypeScript retain focused parser-free-manifest tests; C++ deliberately does
not.
## Full rebuilds

80
docs/LEGACY_AND_NO_AST.md Normal file
View file

@ -0,0 +1,80 @@
# Legacy adapters and no-AST policy
DocForge preserves the original one-method adapter contract while offering incremental extraction,
complete graph-plus-Logic assemblies, and an independently selectable no-AST MCP policy. These are
separate compatibility boundaries.
## One-method adapters remain valid
An existing adapter that implements only:
```python
def load_projection(self) -> AdapterProjection: ...
```
continues to work. It receives the same projection validation, indexing, querying, visualization,
and MCP behavior as before. It does not receive extraction-cache speedups and publishes no Logic
through the complete assembly contract.
Incremental adoption is additive. Implement `load_manifest()` and `extract_source()` while keeping
`load_projection()` as the independent complete graph oracle. If incremental contributions publish
Logic, also implement `load_complete_assembly()` so complete and incremental graph-plus-Logic
output can be compared exactly. The public authoring surface is `docforge.adapter_sdk`; see the
[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md).
## What no-AST means
`--no-ast` is an immutable policy on one MCP process binding. It preserves the configured adapter
strategy but forbids AST, Tree-sitter, compiler-AST, and function-Logic evolution for that binding.
It also rejects nonempty Logic before index publication and rejects a pre-existing index that
contains Logic before reads or live visualization.
Start the generic server with:
```bash
docforge-mcp --project-root /absolute/path/to/project --no-ast
```
Project-owned integrations select the same policy with
`create_project_server(..., no_ast=True)` or `create_read_only_server(..., no_ast=True)`.
`docforge_bootstrap` and `docforge_get_contract` report the effective policy, and
`docforge_get_logic` returns `adapter_policy_forbids_logic`.
Changing this policy requires a new process. It does not rewrite or reload an adapter in place.
## What no-AST does not mean
No-AST is not:
- an inspection mechanism that proves which parser an arbitrary adapter uses;
- a filesystem, Python-import, or process sandbox;
- a promise that the adapter performs no non-AST fingerprinting, dependency discovery, caching, or
complete-projection loading; or
- permission to change project files, adapter code, or the configured capability surface.
Repository permissions and project instructions remain responsible for processes that have direct
filesystem access. The MCP boundary still excludes shell execution, arbitrary file operations,
Git mutation, builds, deployment, publication, project switching, and cross-project retrieval.
## Reference adapters are not no-AST adapters
The built-in Python reference extracts syntax and Logic with the standard-library AST. The
JavaScript, TypeScript, and C++ references extract syntax and Logic with their optional
Tree-sitter grammars. Their normal projections contain Logic, so they must not be presented or
configured as no-AST adapters.
The narrower performance statements in [Reference Adapters](REFERENCE_ADAPTERS.md) concern
specific manifest and warm-cache paths. A parser-free manifest does not turn an adapter that
publishes AST-derived Logic into a no-AST adapter.
## Migration choices
Keep a one-method adapter when its complete rebuild is acceptable and no function Logic is needed.
Adopt incremental extraction when measured source parsing dominates synchronization. Adopt the
complete assembly contract when Logic is published or overlapping raw evidence needs deterministic
ownership.
Choose no-AST only when the project binding must forbid AST-derived publication and future adapter
upgrades of that kind. It is a binding policy, not a substitute for adapter confinement,
conformance tests, restart detection, cache recovery, or the full proof matrix in
[Incremental Adapter Indexing](INCREMENTAL_INDEXING.md).

29
docs/MCP_CONTRACT.md
View file

@ -28,6 +28,30 @@ Bootstrap and contract results include `effective_policy` schema version 1 plus
registered surface and startup-bound proposal/application access. The project descriptor remains
schema version 1 and does not silently acquire machine-specific process policy.
## Fixed reference binding
`docforge.reference_mcp` is the fixed runnable binding for the in-repository Python, JavaScript,
TypeScript, and C++ reference adapters:
```bash
python -I -m docforge.reference_mcp \
--project-root /absolute/project \
--capability-mode read
```
It loads only `.docforge/reference-adapter.toml`, selects only the fixed provider for the declared
language, and constructs the same project-bound read surface through `create_read_only_server()`.
Bootstrap binding metadata includes `server_module = "docforge.reference_mcp"`,
`adapter_mode = "reference"`, the selected `reference_language`, and the exact
`reference_config_hash`.
The reference command accepts read capability mode only. It registers all 21 read tools below and
no isolated proposal or canonical-application tools. The configuration cannot select a provider,
module, command, arguments, working directory, environment, or arbitrary discovery behavior. See
[Reference Adapters](REFERENCE_ADAPTERS.md) for language scope and
[Agent Integration](AGENT_INTEGRATION.md) for the immutable launcher and generated client
fragments.
## Read tools
- `docforge_bootstrap`
@ -45,6 +69,8 @@ schema version 1 and does not silently acquire machine-specific process policy.
- `docforge_get_task_context`
- `docforge_validate_project`
- `docforge_render_status`
- `docforge_graph_plan`
- `docforge_graph_render_status`
- `docforge_visualize`
- `docforge_stop_visualization`
- `docforge_visualization_status`
@ -349,3 +375,6 @@ permissions and project instructions remain responsible for that broader boundar
The legacy `adapter_policy` object remains byte-compatible. It is now a projection of the
versioned `effective_policy`; `--no-ast` restrictively overrides adapter evolution, AST analysis,
and Logic indexing without widening any other capability.
See [Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for the compatibility boundary and
why the AST and Tree-sitter reference adapters are not no-AST adapters.

79
docs/MIGRATING_FROM_V1.md Normal file
View file

@ -0,0 +1,79 @@
# Migrating from DocForge v1
DocForge2 preserves the `docforge` distribution, Python package, CLI command, MCP tool prefix,
generic project descriptor version, and legacy adapter entry point. Migration is an additive
validation exercise, not a canonical-content rewrite.
Read [compatibility](COMPATIBILITY.md), [legacy and no-AST operation](LEGACY_AND_NO_AST.md), and
[recovery](RECOVERY_AND_PERFORMANCE.md) before changing a production binding.
## What remains compatible
- Generic Markdown and TOML projects keep `.docforge/project.toml` schema version 1.
- A custom adapter implementing only `load_projection()` remains valid.
- Existing canonical nodes, stable IDs, source paths, relationship vocabulary, changesets, and
reviewed hashes are not silently rewritten.
- Existing CLI and MCP names remain available. New fields, commands, tools, plans, policies, and
reference integrations are additive.
- `docforge-mcp --no-ast` remains the shorthand for the preserved no-AST binding.
Disposable index and cache schemas may change. Rebuild them rather than copying them as authority.
## Recommended migration
1. Record the v1 package version, adapter identity, project descriptor, canonical source hash,
active changesets, generated client configuration, and current rendered outputs.
2. Back up canonical sources and active proposal files. Derived `.docforge/cache`, preview,
portable-graph, and viewer state need not be authoritative backups.
3. Install the DocForge2 candidate in a separate environment. Do not repoint the production MCP
binding yet.
4. Run the existing adapter through complete loading and `ProjectIndex.check()`. For incremental
adapters, verify complete/incremental equivalence. Logic-producing incremental adapters must
implement the independent complete assembly oracle.
5. Compare project ID, adapter, revision, source hash, primary node and edge hashes, Logic policy,
and retrieval results with the v1 evidence.
6. Recompose effective capability and projection policy. Do not assume that a new default grants
proposal, application, rendering, or viewer access.
7. Rebuild disposable indexes, extraction caches, fragments, receipts, previews, and portable
artifacts from the validated candidate.
8. Generate a new client fragment. Generic projects use `docforge configure`; custom adapters use
`AdapterLauncherV1` with `generate_adapter_client_configuration()`.
9. Run doctor, real MCP bootstrap, representative retrieval, rendering status, recovery, and
exact-hash proposal/application checks in a disposable or shadow environment.
10. Repoint one production binding only after the candidate and rollback procedure pass.
## Adopting reference adapters
The repository reference adapters are narrow syntax evidence, not automatic semantic replacements
for mature v1 integrations. Python publishes modules, classes, functions, and local imports.
JavaScript and TypeScript publish syntax plus static project-relative imports and re-exports.
C++ uses a confined `compile_commands.json` as inert translation-unit inventory and publishes
directly resolvable project-local quoted includes.
Do not replace a compiler-, language-server-, or project-owned semantic adapter if the task depends
on resolved calls, types, inheritance, macros, compiler include semantics, runtime facts, or
ownership. Use [reference adapters](REFERENCE_ADAPTERS.md) as a measured starting point.
## Changesets
Existing changesets remain bound to their original project, root, base revision, source hash,
writer, operations, and exact hash. Do not edit a changeset to make it look current. Retrieve and
validate it through the candidate. Rebase only when every precondition remains true; otherwise
create and review a new proposal.
Before enabling canonical application, prove the exact serializer round trip and recovery behavior
for the selected adapter. The fixed reference MCP server is intentionally read-only.
## Rollback
Keep the v1 environment, configuration fragment, and service definition until the DocForge2
binding has passed live validation. To roll back:
1. Stop the DocForge2 MCP/viewer processes.
2. Restore the previous client or service binding.
3. Restore canonical sources only if a verified application receipt says they changed and the
adapter-specific rollback requires it.
4. Discard DocForge2-derived indexes, receipts, fragments, previews, and portable artifacts.
5. Restart v1 and verify its recorded project/source identity.
Never move a published tag or reuse a version identity for a corrected release.

95
docs/MILESTONE_4_BASELINE.md Normal file
View file

@ -0,0 +1,95 @@
# Milestone 4 baseline
## Scope and method
This baseline records the adapter SDK, Python reference adapter, incremental equivalence, and
recovery behavior completed in Milestone 4. It was captured on 2026-07-29 from clean executable
candidate `95271dcf2e48045b9d3aed9b9ea09c7fc155692c`.
The maintained command was:
```bash
.venv/bin/python tools/milestone4_benchmark.py \
--mode full \
--output benchmarks/milestone4-2026-07-29.json
```
The synthetic project contains 334 Python files. Each file contributes one module, one function,
and one argument node, for 1,002 primary nodes. Imports form a deterministic chain. Durations use
`time.perf_counter_ns()` and nearest-rank p95. Each ordinary result is serialized as compact sorted
JSON for its response-size gate. Per-operation memory uses `tracemalloc`; cumulative process
high-water uses `RUSAGE_SELF`.
Environment:
- Linux 7.1.3-200.nobara.fc44.x86_64 with glibc 2.43.
- CPython 3.14.6.
- x86_64.
- Three warm samples; cold, equivalence, and recovery operations run once.
- Per-operation traced-memory ceiling: 268,435,456 bytes.
- Process high-water ceiling: 536,870,912 bytes.
- Response ceiling: 524,288 bytes.
The complete machine-readable result is
[`benchmarks/milestone4-2026-07-29.json`](../benchmarks/milestone4-2026-07-29.json).
Its SHA-256 is `b6a871dde730a119fc0a138c47bd25f2c533ed3171a9a3d098075aef08173600`.
The stable evidence payload SHA-256 is
`4a8db461a311b5df4abd5aa00063e9a347d8b9dba19b30e35684f561d5271549`.
## Results
| Operation | Median | p95 | Limit | Traced peak | Response |
|---|---:|---:|---:|---:|---:|
| Cold incremental build | 1,599.760 ms | 1,599.760 ms | 20,000 ms | 68,573,540 B | 14,580 B |
| Warm incremental build | 1,244.427 ms | 1,265.387 ms | 20,000 ms | 69,997,166 B | 14,578 B |
| Complete/incremental equivalence | 1,897.919 ms | 1,897.919 ms | 30,000 ms | 64,305,919 B | 230 B |
| Corrupt extraction-cache recovery | 1,695.406 ms | 1,695.406 ms | 30,000 ms | 69,776,923 B | 14,580 B |
| Corrupt index recovery | 1,707.879 ms | 1,707.879 ms | 30,000 ms | 68,561,924 B | 14,779 B |
Process high-water was 78,798,848 bytes. The regression limits intentionally leave multiple times
the measured headroom; they are tripwires, not performance promises.
## Deterministic graph and Logic evidence
The candidate produced:
- 1,002 primary nodes with hash
`f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f`.
- 1,001 primary edges with hash
`00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35`.
- 334 Logic projections containing 2,338 Logic nodes and 2,338 Logic edges, with hash
`c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c`.
- Complete assembly hash
`2888182ab765fbffe3ba873c1613345640e8e6d89be74ddfca7452c0a5056345`.
- Source hash
`30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2`.
The independently loaded complete assembly and the incremental assembly matched exactly across
project identity, revision, source, nodes, edges, and Logic.
## Work and recovery gates
The warm build recorded all 334 sources as cache hits, zero reparsed sources, zero `ast.parse`
calls, and zero `extract_source` calls. Corrupting the extraction cache forced all sources through
extraction and reproduced the same graph and Logic hashes. Corrupting SQLite rebuilt the index
entirely from cache hits with zero parsing or extraction and reproduced those hashes.
This zero-parser evidence applies to the Python benchmark. Focused JavaScript and TypeScript tests
prove parser-free manifests. The C++ reference manifest uses Tree-sitter for bounded quoted-include
discovery and does not make a zero-warm-parser claim.
## Fresh-wheel and reference evidence
The offline adoption proof built and installed the base wheel without Tree-sitter packages,
built and checked a real Python reference project, started the isolated reference MCP server, and
performed bootstrap, search, and exact retrieval over its 21 read tools. Selecting C++ without its
extra failed with `optional_dependency_missing` and `docforge[cpp]` remediation.
Focused fixtures produced:
- Python: 14 nodes, 13 edges, 5 Logic projections.
- JavaScript: 14 nodes, 14 edges, 5 Logic projections.
- TypeScript: 13 nodes, 14 edges, 4 Logic projections.
- C++: 17 nodes, 16 edges, 5 Logic projections.
These fixture counts verify implementations; they are not language-wide completeness claims.

83
docs/MILESTONE_4_CLOSEOUT.md Normal file
View file

@ -0,0 +1,83 @@
# Milestone 4 closeout
## Outcome
Milestone 4 is complete. New projects can adopt a public adapter SDK or one of four narrow
repository reference integrations, attach a fixed read-only MCP server, and follow maintained
product documentation without reading core implementation.
Implemented contracts:
- Stable `docforge.adapter_sdk` authoring imports.
- Independent complete primary-graph-plus-Logic oracle and exact incremental equivalence.
- Bounded adapter assemblies and version-1 extraction caches.
- Base Python, optional JavaScript, optional TypeScript, and optional C++ reference integrations.
- Closed `.docforge/reference-adapter.toml` and fixed `docforge.reference_mcp` read-only binding.
- Immutable, project-bound, launchable `AdapterLauncherV1` declarations and generated Codex,
Claude, and OpenClaw fragments for custom adapters.
- Live implementation-derived CLI and MCP reference tables with race-safe publication and drift
checking.
- Strict documentation graph, link, anchor, H1, reachability, required-page, generated-notice, and
documented-reference-config checks.
- Offline fresh-wheel adoption and maintained 1,002-node scale/recovery gates.
## Candidate evidence
The frozen executable candidate is
`95271dcf2e48045b9d3aed9b9ea09c7fc155692c`.
Its complete executable gate passed:
- Ruff formatting and Python lint.
- HTML, rendered-manual HTML, portable-graph HTML, CSS, and JavaScript checks.
- Pyright with zero diagnostics.
- Warning-strict compilation.
- 142 contract tests and 268 subtests.
- 347 complete tests and 402 subtests.
- Three Playwright and axe accessibility flows for the manual, portable graph, and live viewer.
- Lock and npm dependency-tree checks.
- Wheel and source-distribution builds.
- Offline fresh-wheel adoption.
- Milestone 0, 1, 2, 3, and 4 smoke benchmarks.
The clean full benchmark passed exact graph-plus-Logic equivalence, warm zero Python parser and
extraction work, corrupt-cache recovery, corrupt-index recovery, response, memory, and latency
gates. Exact results are in [the Milestone 4 baseline](MILESTONE_4_BASELINE.md) and
[`benchmarks/milestone4-2026-07-29.json`](../benchmarks/milestone4-2026-07-29.json).
Gitleaks 8.30.1 scanned the Milestone 4 commit range and candidate tree with no findings. The SSH
remote syntax prevents Gitleaks from constructing finding hyperlinks; it does not affect scanning.
## Reference scope
- Python uses the standard-library AST and publishes syntax plus local imports.
- JavaScript and TypeScript use distinct optional Tree-sitter grammars and publish syntax plus
project-local static relative imports and re-exports.
- C++ uses a confined `compile_commands.json` as inert translation-unit inventory and publishes
syntax plus directly resolvable project-local quoted includes.
- The references do not claim resolved calls, inheritance, types, symbol references, compiler
include semantics, macro semantics, runtime behavior, or semantic ownership.
The C++ reference never executes a compiler or compilation-database command. It is not a Clang
semantic adapter.
## Preserved boundaries
- The `docforge` distribution, Python package, CLI, MCP executable, and tool names remain.
- Generic projects and one-method `load_projection()` adapters remain supported.
- Descriptor schema version 1, index schema version 3, effective policy version 1, and no-AST
behavior remain.
- Heavy language frontends are optional. Base generic and Python operation installs no
Tree-sitter distribution.
- Reference MCP is read-only. Proposal and application remain explicit project-owned gates.
- No language adapter was separately published.
- No WorldForge, ScrapeStation, legacy-repository, production-binding, storage, or self-hosting
change was made.
- No tag or Forgejo release was created for Milestone 4.
## Later work
Milestone 5 owns stabilization and the first DocForge2 release. Release identity, compatibility
matrix, migration and recovery proofs, comparative real-task evidence, versioning, tagging, and
publication must be validated there. Remote adapters, render farms, third-party renderers,
cross-project graphs, storage replacement, and self-hosting remain deferred without measured need.

278
docs/NEW_PROJECT_QUICKSTART.md
View file

@ -1,6 +1,276 @@
# DocForge setup moved to the user manual
# New-project quickstart
The complete installation, project setup, visualization, CLI, MCP, application, adapter, and
troubleshooting reference now lives in the [DocForge user manual](USER_MANUAL.md).
This guide takes a new installation from an empty project binding to a validated generic manual or
one of DocForge's fixed reference source graphs. For the complete operating reference, see the
[user manual](USER_MANUAL.md).
This file remains only so existing bookmarks and links continue to resolve.
## 1. Install DocForge
DocForge requires Python 3.12 or newer. A base installation includes the generic project service,
the public adapter SDK, the Python reference integration, CLI, MCP server, renderers, and viewer
assets. It does not install Tree-sitter.
From a source checkout:
```bash
git clone <repository-url> /absolute/path/DocForge
cd /absolute/path/DocForge
uv sync --group dev
DOCFORGE=/absolute/path/DocForge/.venv/bin/docforge
DOCFORGE_MCP=/absolute/path/DocForge/.venv/bin/docforge-mcp
DOCFORGE_PYTHON=/absolute/path/DocForge/.venv/bin/python
```
For an isolated consumer environment, install the checkout or a built wheel with `uv pip install`.
Add only the language extras that project needs:
```bash
uv venv /absolute/path/docforge-env --python 3.12
uv pip install --python /absolute/path/docforge-env/bin/python /absolute/path/DocForge
uv pip install --python /absolute/path/docforge-env/bin/python \
"/absolute/path/DocForge[javascript]"
uv pip install --python /absolute/path/docforge-env/bin/python \
"/absolute/path/DocForge[typescript]"
uv pip install --python /absolute/path/docforge-env/bin/python \
"/absolute/path/DocForge[cpp]"
```
The `languages` extra installs all three optional grammar families. JavaScript and TypeScript are
separate extras because they use distinct grammars. The C++ extra supplies a syntax grammar, not a
compiler or Clang semantic frontend.
Confirm the installation:
```bash
"$DOCFORGE" --help
"$DOCFORGE_MCP" --help
```
## 2. Choose a project route
Use a generic project when Markdown or TOML documentation is canonical. Use a fixed reference
adapter when you want a bounded source inventory and syntax-level graph for Python, JavaScript,
TypeScript, or C++. Use a project-owned adapter when the production contract must supply richer
semantics.
- Generic manual: continue with [Create a generic project](#create-a-generic-project).
- Fixed source example: continue with [Use a reference adapter](#use-a-reference-adapter).
- Production language frontend: follow the [Adapter authoring guide](ADAPTER_AUTHORING_GUIDE.md).
## Create a generic project
Set the absolute project root and assess it without writing:
```bash
PROJECT=/absolute/path/MyProject
"$DOCFORGE" --project-root "$PROJECT" onboard
```
The assessment reports detected languages, build evidence, documentation candidates, current
configuration, and available capabilities. Detection never invents a source graph.
Create a generic starter explicitly:
```bash
"$DOCFORGE" --project-root "$PROJECT" onboard \
--scaffold \
--project-id my-project \
--title "My Project"
```
Scaffolding is create-only and refuses existing target files. It creates:
- `.docforge/project.toml`, the generic project descriptor;
- `docs/docforge/content/architecture-overview.md`, one canonical node;
- `.docforge/templates/manual.html`, one built-in-renderer template;
- derived index, receipt, and rendered output below `.docforge`.
If a language was detected, source graph status remains `adapter_required`. The generic starter
does not claim source semantics.
Validate and inspect it:
```bash
"$DOCFORGE" --project-root "$PROJECT" validate
"$DOCFORGE" --project-root "$PROJECT" reindex
"$DOCFORGE" --project-root "$PROJECT" search architecture
"$DOCFORGE" --project-root "$PROJECT" show architecture.overview
"$DOCFORGE" --project-root "$PROJECT" render-status
```
The descriptor is explained field by field in [Project descriptor](PROJECT_DESCRIPTOR.md). Add
canonical nodes only after choosing their authority, stable IDs, families, statuses, and allowed
relationships; [Core concepts and authority](CORE_CONCEPTS_AND_AUTHORITY.md) defines those terms.
### Start a generic MCP binding
Start read-only first:
```bash
"$DOCFORGE_MCP" \
--project-root "$PROJECT" \
--capability-mode read
```
Call `docforge_bootstrap` before other tools. It reports the exact project identity, generation,
effective policy, projection policy, available capabilities, and recommended first read.
To enable proposals, the descriptor must declare the writer and the process must select it:
```bash
"$DOCFORGE_MCP" \
--project-root "$PROJECT" \
--capability-mode proposal \
--proposal-writer project-editor
```
Canonical application is a separate startup gate. Do not add it to a read-only client:
```bash
"$DOCFORGE_MCP" \
--project-root "$PROJECT" \
--capability-mode application \
--proposal-writer project-editor \
--canonical-applier project-editor
```
Application accepts one exact reviewed changeset hash. It does not commit, push, build, deploy, or
publish the project.
### Generate a generic client fragment
Preview is side-effect free:
```bash
"$DOCFORGE" configure codex --project "$PROJECT"
"$DOCFORGE" configure claude --project "$PROJECT"
"$DOCFORGE" configure openclaw --project "$PROJECT"
```
Add `--output /absolute/path/new-fragment` to create one new private standalone file. Generation
does not merge with or replace a different existing file. Diagnose an installed binding with:
```bash
"$DOCFORGE" doctor --client codex --project "$PROJECT"
```
Doctor is a bounded configuration inspector, not a connection test. See [Agent
integration](AGENT_INTEGRATION.md) for client-specific layouts and limitations.
## Use a reference adapter
Reference adapters read one fixed descriptor:
`.docforge/reference-adapter.toml`. The path is not selectable.
Create a Python project configuration:
```toml reference-adapter
schema_version = 1
project_id = "my-python-project"
title = "My Python Project"
language = "python"
source_roots = ["src"]
```
JavaScript uses `language = "javascript"` and the `javascript` extra. TypeScript uses
`language = "typescript"` and the `typescript` extra.
C++ additionally requires a confined compilation database:
```toml reference-adapter
schema_version = 1
project_id = "my-cpp-project"
title = "My C++ Project"
language = "cpp"
source_roots = ["src", "include"]
compilation_database = "compile_commands.json"
```
The C++ integration reads `compile_commands.json` only as bounded translation-unit inventory and
fingerprint evidence. It never executes a recorded command or compiler.
Run the fixed read-only server through the same installed Python interpreter:
```bash
"$DOCFORGE_PYTHON" -I -m docforge.reference_mcp \
--project-root "$PROJECT" \
--capability-mode read
```
The binding chooses one in-package provider from the descriptor language. It accepts no provider
module, command, argument list, working directory, environment, discovery rule, proposal writer,
or canonical applier. Its MCP surface is the read subset documented in the [generated command
reference](COMMAND_REFERENCE.md).
### Generate a reference-adapter client fragment
Generic `docforge configure` intentionally refuses custom adapters. Construct the adapter project
and immutable launcher, then call the custom-adapter generator:
```python
from pathlib import Path
from docforge.adapter_launcher import AdapterLauncherV1
from docforge.client_config import generate_adapter_client_configuration
from docforge.reference_mcp import REFERENCE_MCP_MODULE, create_reference_project
root = Path("/absolute/path/MyProject").resolve(strict=True)
project = create_reference_project(root)
launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE)
preview = generate_adapter_client_configuration(
project,
launcher,
"codex",
capability_mode="read",
)
print(preview["artifact"]["content"])
```
Use `client="claude"` or `client="openclaw"` for those formats. Pass an absolute `output` path only
when creating a new standalone private fragment. The generated launch is bound to the selected
project, descriptor hash, adapter identity, installed module, isolated interpreter, policy, and
source availability evidence.
Read [Reference adapters](REFERENCE_ADAPTERS.md) before relying on the graph. The Python example
publishes local imports, the JavaScript and TypeScript examples publish project-local static
relative imports and re-exports, and the C++ example publishes directly resolvable project-local
quoted includes. None is a complete semantic compiler frontend.
## 3. Add visualization only when needed
Install the per-user viewer manager once:
```bash
/absolute/path/DocForge/.venv/bin/docforge-viewer-manager install-user-service
```
Then start a project-bound snapshot:
```bash
"$DOCFORGE" --project-root "$PROJECT" visualize
"$DOCFORGE" --project-root "$PROJECT" visualization-status
```
The listener is loopback-only and tokenized. The viewer is a derived snapshot, not canonical
authority and not a continuously monitored filesystem view. Read [Rendering and
visualization](RENDERING_AND_VISUALIZATION.md) for manual, portable, and live-viewer differences.
## 4. Verify the maintained checkout
Contributors can run:
```bash
make command-reference-check
make docs-check
make adoption-m4
make benchmark-m4-smoke
```
Run `make gate` before a release candidate. `benchmark-m4-smoke` is routine coverage;
`benchmark-m4` is the maintained full adapter workload.
Continue with [Project onboarding](PROJECT_ONBOARDING.md) for production integration,
[Policy precedence](POLICY_PRECEDENCE.md) before widening a binding, and [Security](SECURITY.md)
before exposing any MCP process.

202
docs/POLICY_PRECEDENCE.md Normal file
View file

@ -0,0 +1,202 @@
# Policy precedence
DocForge composes an immutable effective policy for each project-bound process. It resolves
restrictions in a fixed order:
```text
core safety
> explicit binding
> no-AST shorthand
> resource availability
```
A lower layer can make a requested operation unavailable; it cannot override a higher-layer
prohibition. Bootstrap and generated client evidence report the composed result and hashes so a
client does not need to infer policy from command-line arguments.
## 1. Core safety
Core safety is unconditional. No capability mode exposes:
- arbitrary file access or project switching;
- arbitrary renderer, module, command, shell, argument, working-directory, or environment
selection;
- Git mutation;
- project builds or compiler execution;
- deployment or publication.
Canonical application is limited to DocForge's validated serializer boundary. Adapter launch is
limited to the immutable launcher contract. Rendering is limited to declared views and fixed
built-in workers. An indexed instruction cannot alter any of these rules.
## 2. Explicit binding
A process is bound at startup to one project root, descriptor, adapter, capability mode, writer and
applier identities when present, no-AST selection, diagnostics selection, and projection modes.
The binding does not change during the process lifetime.
Capability modes are:
- `read`: register only the read surface;
- `proposal`: add proposal tools when a valid selected writer is available;
- `application`: require a startup-bound canonical applier and expose exact-hash application;
- `operator`: reserved; it currently adds no tools.
Mode describes the maximum registered surface. Actual authority can be narrower. A descriptor must
declare the selected writer, including allowed families and operation types. Application requires
the matching configured writer, changeset creator, and canonical-applier identity. A mode name
cannot create a missing descriptor grant.
Generic generated client fragments default to read mode. Other construction paths preserve their
documented compatible factory defaults. Treat `docforge_bootstrap.session_contract` and its actual
capabilities as authoritative for a running server.
## 3. No-AST shorthand
`--no-ast` is a restrictive compatibility shorthand. It composes:
- adapter evolution `preserve`;
- AST analysis `forbidden`;
- Logic indexing `off`;
- `docforge_get_logic` blocked;
- prohibitions on AST, Tree-sitter, compiler-AST, and function-Logic upgrades.
Non-AST source fingerprinting and incremental caching remain allowed. Existing one-method adapters
continue to use `load_projection()`.
The shorthand does not inspect how an existing adapter was implemented, sandbox its filesystem
reads, or transform an AST-based adapter into a no-AST adapter. Do not run the Python,
JavaScript/TypeScript, or C++ syntax reference integrations and then describe the binding as a
proved no-AST integration. See [Legacy and no-AST operation](LEGACY_AND_NO_AST.md).
## 4. Resource availability
Even an allowed policy cannot create a missing resource:
- application mode requires a startup-bound canonical applier;
- manual `explicit` or `auto` requires declared manual render configuration;
- manual `auto` also requires canonical application in the current operation or server;
- portable graph `explicit` requires declared portable graph configuration;
- live viewer `on-demand` requires the viewer runtime;
- an optional reference language requires its installed extra;
- a project-owned launcher module must be installed, isolated, project-owned, and unchanged.
Unavailable requested modes fail before hidden work. Missing optional language grammars return an
actionable install target such as `docforge[typescript]` or `docforge[cpp]`; DocForge does not
silently downgrade to a different frontend.
## Effective process policy
The version-1 effective policy reports:
- capability mode and whether it came from a factory default or explicit selection;
- adapter evolution, AST analysis, and Logic indexing;
- automatic synchronization and validated integrity;
- the compatible manual-render projection;
- profiling state;
- blocked tools and prohibitions;
- the exact precedence list.
This version-1 projection preserves compatibility. It is not the complete version-2 rendering
policy; portable graph and live-viewer choices are reported separately.
## Independent projection policy
Manual rendering, portable graph publication, and live visualization use a separate immutable
version-2 policy:
```text
manual: auto | explicit | disabled
portable_graph: explicit | disabled
live_viewer: on-demand | disabled
```
When a selector is omitted, composition uses availability-aware defaults:
- manual is `auto` only when manual configuration and canonical application are both available;
otherwise it is `explicit` when configured, or `disabled`;
- portable graph is `explicit` when configured, otherwise `disabled`;
- live viewer is `on-demand` when the runtime is available, otherwise `disabled`.
An explicit non-disabled selection for a missing resource returns
`projection_policy_unavailable`. An active operation prohibited by the selected policy returns
`projection_policy_forbids_operation` before planning, rendering, or viewer startup.
Status remains intentionally narrower than active work. Manual and portable receipt-only status
are available when their active operations are disabled. Viewer status and explicit stop remain
available when viewer start is disabled.
Ordinary standalone CLI rendering uses manual `explicit`. Manual `auto` belongs to a canonical
application operation that owns automatic regeneration.
## Diagnostics do not grant authority
`--diagnostics` enables bounded request-local stage timings and compiler-work counters. It does not
enable tools, broaden paths, retain project content, or displace a primary MCP result that already
needs the response budget.
## Descriptor policy and process policy
The project descriptor is canonical project configuration. The process policy is a runtime
restriction. They compose by intersection:
```text
operation is available
only if core permits it
and the startup binding registers it
and no-AST permits it
and required resources exist
and the descriptor grants the requested project authority
and current graph/hash preconditions validate
```
Changing a descriptor does not retarget a running project-owned process. Descriptor, adapter
implementation, or launcher drift requires a fresh process.
## Common decisions
For an agent that only reads documentation:
```bash
docforge-mcp \
--project-root /absolute/path/MyProject \
--capability-mode read
```
For an agent that may prepare reviewable proposals:
```bash
docforge-mcp \
--project-root /absolute/path/MyProject \
--capability-mode proposal \
--proposal-writer project-editor
```
For a tightly controlled application process:
```bash
docforge-mcp \
--project-root /absolute/path/MyProject \
--capability-mode application \
--proposal-writer project-editor \
--canonical-applier project-editor
```
For a read binding with every active output projection disabled:
```bash
docforge-mcp \
--project-root /absolute/path/MyProject \
--capability-mode read \
--manual-render-policy disabled \
--portable-graph-policy disabled \
--live-viewer-policy disabled
```
Prefer the narrowest binding that completes the workflow. Call `docforge_bootstrap` first and use
the returned effective policy, projection policy, actual capabilities, and prohibitions rather
than assumptions based on client configuration.
See the [MCP contract](MCP_CONTRACT.md), [Security](SECURITY.md), [Project
descriptor](PROJECT_DESCRIPTOR.md), and [Rendering and
visualization](RENDERING_AND_VISUALIZATION.md).

279
docs/PROJECT_DESCRIPTOR.md Normal file
View file

@ -0,0 +1,279 @@
# Project descriptor
A generic DocForge project is selected by one fixed file:
`.docforge/project.toml` beneath an explicit project root. The descriptor is schema version 1.
DocForge validates both the JSON-schema shape in `schemas/project.schema.json` and runtime
invariants that schema alone cannot prove.
Reference integrations use a different fixed descriptor,
`.docforge/reference-adapter.toml`; see [Reference adapters](REFERENCE_ADAPTERS.md).
Project-owned adapters construct the same runtime `ProjectDescriptor` contract through the public
adapter SDK.
## Complete generic example
```toml
schema_version = 1
project_id = "my-project"
title = "My Project"
adapter = "generic"
[sources]
content_roots = ["docs/docforge/content"]
authority_files = []
[derived]
cache_root = ".docforge/cache"
index = ".docforge/cache/index.sqlite3"
[changesets]
root = ".docforge/changesets"
[[changesets.writers]]
id = "project-editor"
families = ["architecture", "operations", "system"]
operations = ["create", "update", "move", "delete"]
[render]
template_root = ".docforge/templates"
preview_root = ".docforge/previews"
[[render.views]]
id = "manual"
renderer = "generic_html"
template = "manual.html"
output = ".docforge/rendered/manual.html"
title = "My Project Manual"
families = ["architecture", "operations", "system"]
[graph_render]
output_root = ".docforge/portable-graph"
[[graph_render.views]]
id = "architecture"
renderer = "portable_graph_html"
output = "architecture.html"
title = "Architecture"
root = "architecture.overview"
initial_mode = "web"
depth = 3
max_nodes = 250
max_edges = 1000
max_work = 100000
families = ["architecture", "system"]
relations = ["depends_on", "owns", "calls", "reads", "writes", "tested_by", "relates_to"]
authorities = []
statuses = ["current", "active", "verified"]
tags = []
include_logic = false
[graph]
allowed_relations = [
"calls",
"depends_on",
"owns",
"reads",
"relates_to",
"tested_by",
"writes",
]
[limits]
max_source_bytes = 500000
max_nodes = 10000
max_query_chars = 500
max_results = 100
max_traversal_depth = 6
max_context_tokens = 12000
max_tool_output_chars = 200000
max_changesets = 100
max_changeset_operations = 100
max_changeset_bytes = 1000000
max_render_views = 20
max_template_bytes = 1000000
max_render_bytes = 1000000
[[profiles]]
id = "development"
families = ["architecture", "operations", "system"]
statuses = ["current", "active", "verified"]
required_nodes = ["architecture.overview"]
token_budget = 8000
dependency_depth = 3
```
Rendering sections are optional. Profiles and limits may also be omitted; runtime defaults then
apply. The required top-level fields are `schema_version`, `project_id`, `title`, `adapter`,
`sources`, `derived`, `changesets`, and `graph`.
## Identity fields
`schema_version` must be `1`.
`project_id` is the stable machine identity. It is lowercase and may contain digits, dots,
underscores, and hyphens after its first character. Do not derive it from a mutable display title.
`title` is the human-readable project name.
`adapter` is `generic` for this file format. Project-owned adapters publish a validated
`adapter_id@adapter_version` identity through their runtime descriptor; changing adapter identity
invalidates incompatible derived state.
The descriptor's byte content contributes to a descriptor hash. Client fragments, launchers, index
evidence, and policy results use that hash to detect drift.
## Canonical sources
`sources.content_roots` lists the project-relative directories containing generic Markdown and TOML
nodes. Each path must resolve beneath the project root. Canonical content roots may not overlap the
derived cache.
`sources.authority_files` lists additional project-relative regular files whose content belongs to
the canonical project identity. They are not automatically parsed as nodes.
A Markdown node contains one TOML metadata block followed by Markdown content:
```markdown
+++
schema_version = 1
id = "architecture.overview"
title = "Architecture overview"
family = "architecture"
authority = "authoritative"
status = "current"
tags = ["architecture"]
summary = "Defines the top-level architecture and ownership."
+++
# Architecture overview
Describe systems, ownership, runtime flow, failure behavior, and proof.
```
Every node ID is project-wide and stable. Every relationship target must resolve. A TOML source may
contain multiple `[[nodes]]` records; proposal-enabled multi-node files need stable
`source_anchor` values where creation or movement requires an exact record boundary.
## Derived state
`derived.cache_root` owns disposable indexes, attestations, extraction caches, render receipts,
projection artifacts, and viewer registry state.
`derived.index` must be inside `derived.cache_root`. Canonical content and cache paths must not
overlap.
Derived state is not a backup. If it is deleted or rejected as corrupt, DocForge rebuilds it from
validated canonical sources.
## Changesets and writers
`changesets.root` is the confined proposal store. It must not overlap canonical content or the
derived cache.
Each `changesets.writers` entry declares:
- a stable writer `id`;
- the node `families` that writer may change;
- allowed `operations`: `create`, `update`, `move`, and/or `delete`.
The descriptor grant is necessary but not sufficient. A process must also select that writer at
startup, and canonical application requires a separately bound matching applier. Capability mode
does not broaden the descriptor grant. See [Policy precedence](POLICY_PRECEDENCE.md).
## Allowed relationships
`graph.allowed_relations` is the exact project vocabulary accepted on edges. It must be nonempty.
Relationship names are stable IDs. DocForge rejects relationships that are not declared and gives
`depends_on` additional cycle validation.
The core does not reinterpret a custom relationship just because its spelling resembles a known
term. Task-context retrieval classifies only the documented versioned aliases and preserves
unknown allowed relationships as `unclassified`.
## Context profiles
Each `profiles` entry defines one bounded context compilation:
- `id` selects the profile;
- `families` and `statuses` filter eligible nodes;
- `required_nodes` names stable nodes that must be present;
- `token_budget` limits compiled content;
- `dependency_depth` bounds relationship expansion.
Profiles choose derived retrieval scope. They do not change node authority or writer permissions.
## Limits
Positive limits bound input, graph, retrieval, proposal, and rendering work. Current defaults are:
- `max_source_bytes = 1000000`
- `max_nodes = 10000`
- `max_query_chars = 500`
- `max_results = 100`
- `max_traversal_depth = 8`
- `max_context_tokens = 32000`
- `max_tool_output_chars = 200000`
- `max_changesets = 1000`
- `max_changeset_operations = 100`
- `max_changeset_bytes = 1000000`
- `max_render_views = 100`
- `max_template_bytes = 1000000`
- `max_render_bytes = 1000000`
Smaller project limits are useful policy. They cannot widen fixed internal worker, package,
response, or cache ceilings. In particular, detached renderer transfer has its own fixed boundary
even if a compatibility descriptor retains a larger `max_render_bytes`.
## Manual rendering
The optional `render` section declares:
- one confined `template_root`;
- one isolated `preview_root`;
- one or more stable views.
Each view uses the built-in `generic_html` renderer, a template beneath `template_root`, one
declared output path, a title, and a family filter. Paths may not overlap canonical content,
authority files, changesets, cache, templates, or previews in unsafe ways.
Templates are inert UTF-8 files with a fixed token vocabulary. They cannot select executable
renderers or commands. See [Rendering and visualization](RENDERING_AND_VISUALIZATION.md).
## Portable graph rendering
The optional `graph_render` section declares an output root and one or more
`portable_graph_html` views. Each view selects exactly one:
- `root`, an exact stable node ID; or
- `query`, a bounded metadata-only lexical seed.
It may then restrict families, relations, authorities, statuses, and tags, plus depth, node, edge,
and work limits. `initial_mode` is `nodes`, `flow`, or `web`. Portable graph contract version 1
requires `include_logic = false`.
The declared `output` is relative to `graph_render.output_root`.
## Paths and confinement
Descriptor paths are project-relative. Absolute paths and parent traversal are rejected. Runtime
validation also rejects symlink escapes, unexpected file types, unsafe overlap, changing path
identity during sensitive reads or publication, and derived outputs outside their declared roots.
The explicit CLI `--project-root` is the only project selector. The MCP process binds it at startup
and exposes no project-switching tool.
## Validate changes safely
After editing the descriptor:
```bash
docforge --project-root /absolute/path/MyProject validate
docforge --project-root /absolute/path/MyProject reindex
docforge --project-root /absolute/path/MyProject check
```
Descriptor or adapter implementation drift makes a project-owned running process fail closed; start
a fresh process after changing those boundaries.
For first-time creation, prefer the create-only [New-project
quickstart](NEW_PROJECT_QUICKSTART.md). For full invariants, read the [Core contract](CONTRACT.md).

69
docs/PROJECT_ONBOARDING.md
View file

@ -5,6 +5,11 @@ DocForge onboarding has two separate outcomes:
1. A generic manual can be configured, indexed, rendered, visualized, and exposed through the MCP.
2. A source graph additionally requires one validated language frontend per source language.
DocForge ships narrow fixed references for Python, JavaScript, TypeScript, and C++. They are useful
for syntax-scoped projects and adoption proof, but production semantic requirements may still
require a project-owned compiler or language-service adapter. Review
[Reference Adapters](REFERENCE_ADAPTERS.md) before selecting a frontend.
The onboarding command never claims that source semantics exist merely because it found source
files. It reports each detected language as `adapter_required` until a project integration supplies
and proves that frontend.
@ -64,6 +69,32 @@ immediately usable through the generic CLI, viewer, and MCP.
The starter overview records detected languages and states that the source graph is unavailable
until a language frontend passes the adapter proof. That limitation is deliberate.
## Configure a fixed reference adapter
The generic onboarding scaffold and fixed reference configuration are separate project routes.
For a syntax-scoped reference project, create `.docforge/reference-adapter.toml`:
```toml
schema_version = 1
project_id = "my-python-project"
title = "My Python project"
language = "python"
source_roots = ["src"]
```
Then start the fixed read-only binding:
```bash
python -I -m docforge.reference_mcp \
--project-root /absolute/path/MyProject \
--capability-mode read
```
The configuration selects only a fixed in-repository provider and cannot name a command or custom
module. JavaScript, TypeScript, and C++ require their respective optional extras; C++ also requires
`compilation_database = "compile_commands.json"`. The complete configuration and supported-fact
contract are in [Reference Adapters](REFERENCE_ADAPTERS.md).
## Complete onboarding checklist
### 1. Repository assessment
@ -118,6 +149,13 @@ For every source language:
- [ ] Define dependency discovery.
- [ ] State unsupported semantic facts explicitly.
Decide whether the project needs production semantic evidence or the narrower syntax-only
reference scope. The Python reference publishes only project-local imports. JavaScript and
TypeScript publish only project-local static relative imports and re-exports. The C++ reference
publishes only directly resolvable project-local quoted includes and does not run a compiler. None
of those references resolves calls, inheritance, types, symbols, runtime behavior, or semantic
ownership.
All frontends emit the same DocForge contracts:
- `AdapterManifest` inventories fingerprinted extraction units and dependencies.
@ -151,6 +189,11 @@ this checklist summarizes.
- [ ] Assign shared symbols to one deterministic source contribution.
- [ ] Record compiler-derived project include dependencies.
These are production semantic-adapter expectations. The built-in C++ reference uses
`compile_commands.json` only as bounded translation-unit inventory and fingerprint evidence. It
parses commands and arguments as inert data, executes no command or compiler, and does not claim
compiler include semantics, symbol ownership, or a Clang-derived graph.
#### Rust
- [ ] Read the Cargo workspace and package graph.
@ -240,6 +283,10 @@ from filenames.
Done when a new session can identify and retrieve the correct project without being told its file
layout.
The fixed reference server is read-only. Generic and custom-adapter client generation, including
the immutable custom launcher boundary, is documented in
[Agent Integration](AGENT_INTEGRATION.md).
### 10. Operating guide and maintenance
- [ ] Record the authority and progressive-reading order.
@ -286,14 +333,18 @@ to one configured project root.
DocForge does not let an MCP call install dependencies, run project builds, modify Git, deploy, or
publish. A project integration may use its own normal development workflow for those actions.
## Frontend packaging direction
## Reference and production frontend boundaries
Reusable language frontends should be separate packages or project-owned adapters over the public
DocForge contracts. They must not put language-specific rules into the graph, index, viewer, or MCP
core.
The public authoring namespace is `docforge.adapter_sdk`. Project-owned and separately distributed
production frontends should build on that contract without putting language-specific rules into
the graph, index, viewer, or MCP core.
Worldforge is the first complete C++ reference integration. A reusable C++ package should be
extracted only after that integration proves stable ownership, compiler dependency invalidation,
and complete/incremental equivalence. Rust and Java frontends should then implement the same
contract using their authoritative build and language tooling rather than copying C++ extraction
rules.
The in-repository Python, JavaScript, TypeScript, and C++ adapters are reference implementations.
Their syntax-scoped behavior is useful without becoming a claim that every project in those
languages has complete semantic coverage. A compiler-backed production C++ adapter may resolve
build flags, calls, types, inheritance, include semantics, and ownership when it can prove those
facts. The reference C++ adapter does none of that and is not a Clang semantic adapter.
New Rust, Java, or other frontends should use their authoritative build and language tooling and
must pass the graph-plus-Logic complete/incremental contract in the
[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md).

86
docs/RECOVERY_AND_PERFORMANCE.md Normal file
View file

@ -0,0 +1,86 @@
# Recovery and performance
DocForge keeps canonical project evidence separate from disposable indexes, caches, receipts, and
rendered projections. Recovery rebuilds derived state from current authority; it does not rewrite
canonical content to make a cache look valid.
For the underlying boundaries, see [core authority](CORE_CONCEPTS_AND_AUTHORITY.md) and the
[security model](SECURITY.md).
## Recovery order
Use the narrowest verified recovery:
1. Run `docforge check` or the corresponding MCP status operation.
2. Run `docforge sync` to repair a missing or stale disposable index when safe.
3. Run `docforge reindex` for an explicit complete rebuild.
4. Restart a long-running MCP or viewer binding after descriptor or adapter implementation drift.
5. Regenerate a derived client fragment, command reference, preview, manual, or portable graph
from current project evidence.
Never delete or rewrite canonical sources, active changesets, or reviewed hashes as cache cleanup.
## Extraction-cache recovery
Incremental adapter contributions are stored in a version-1 disposable extraction cache. Reads are
no-follow, regular-file-only, identity-checked, bounded to 64,000,000 bytes and 10,000 source
records, and fail to a cache miss on malformed or incompatible data. Publication is atomic.
After a miss, current manifest sources are extracted again and the complete assembly is validated.
The active SQLite generation remains authoritative for reads until a new verified index is
published. An extraction cache may therefore be safely ahead of the last index; the two files do
not pretend to be one transaction.
## Index and receipt recovery
A validated SQLite index is a generation-pinned derived snapshot. Missing, corrupt, unattested, or
stale indexes rebuild from the complete project or adapter oracle. Generation receipts and render
receipts are post-publication evidence. Failure to write a receipt after a committed artifact is
reported as degraded committed success, not as permission to repeat a mutation.
The live viewer pins one validated index identity. Index replacement makes the running snapshot
stale and causes a later visualize request to start a fresh worker.
## Proposal and application recovery
Hash or base conflicts are not cache failures. Retrieve the current changeset and diff, then
review the new exact hash. Rebase is allowed only when every touched node, relationship, source,
permission, and graph invariant still matches. A content conflict requires a new proposal.
If a canonical serializer fails its round-trip check, use its reported rollback state. Do not
reapply a changeset whose application may already have committed. See
[migrating from v1](MIGRATING_FROM_V1.md) for rollback planning.
## Milestone 4 scale evidence
The maintained Python reference benchmark creates 334 Python source files and produces:
- 1,002 primary nodes and 1,001 primary edges.
- 334 Logic projections with 2,338 Logic nodes and 2,338 Logic edges.
- Exact complete/incremental primary graph and Logic equality.
- Exact output after corrupt extraction-cache recovery and corrupt-index recovery.
- Zero `ast.parse` calls and zero `extract_source` calls during a warm build.
On the frozen Milestone 4 candidate, operation p95 values were 1.266 to 1.898 seconds. Per-operation
traced peaks were about 61 to 67 MiB, and process high-water was 78,798,848 bytes. These are
regression measurements from one machine, not universal latency promises. The machine-readable
record is `../benchmarks/milestone4-2026-07-29.json`; detailed method and hashes are in
[the Milestone 4 baseline](MILESTONE_4_BASELINE.md).
The zero-parser claim is deliberately Python-only. Focused JavaScript and TypeScript tests prove
their manifests avoid Tree-sitter. The C++ reference manifest currently uses Tree-sitter while
discovering bounded quoted includes, so a warm C++ cache hit is not evidence of zero parser work.
## Maintained gates
Run:
```bash
make gate
make adoption-m4
make benchmark-m4-full
make docs-check
```
The main gate includes smoke benchmarks. Full milestone evidence is recorded separately from a
clean candidate so smoke or dirty-tree results cannot become release claims.

149
docs/REFERENCE_ADAPTERS.md Normal file
View file

@ -0,0 +1,149 @@
# Reference adapters
DocForge includes fixed, syntax-scoped reference adapters for Python, JavaScript, TypeScript, and
C++. They demonstrate the public adapter contract, deterministic complete and incremental
publication, function Logic, cache recovery, and a runnable read-only MCP binding. They are not
compiler or language-service replacements.
Use a reference adapter when its deliberately narrow graph is sufficient or when proving a fresh
DocForge integration. Use the [Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md) for a
production adapter that needs resolved symbols, calls, inheritance, types, build semantics, or
semantic ownership.
## Installation
The Python reference adapter uses the standard library and is available in the base wheel. The
other frontends are separate optional extras:
```bash
python -m pip install docforge
python -m pip install "docforge[javascript]"
python -m pip install "docforge[typescript]"
python -m pip install "docforge[cpp]"
```
Install `docforge[languages]` only when one environment intentionally needs all three Tree-sitter
frontends. JavaScript and TypeScript use distinct grammar packages and distinct extras.
## Fixed project configuration
The runnable reference binding reads exactly
`.docforge/reference-adapter.toml` below the selected project root. The configuration is data only:
it cannot name a provider, Python module, command, arguments, working directory, environment, or
discovery rule.
Reference adapter configuration `.docforge/reference-adapter.toml`:
```toml
schema_version = 1
project_id = "example-python"
title = "Example Python project"
language = "python"
source_roots = ["src"]
```
`language` is one of `python`, `javascript`, `typescript`, or `cpp`. Source roots must be sorted,
unique, non-overlapping project-relative directories outside `.docforge`. The configuration,
roots, and inventoried source files must be regular confined paths without symlink traversal. The
configuration is limited to 65,536 bytes, at most 64 source roots, and 65,536 examined inventory
entries.
C++ additionally requires one explicit project-confined compilation database:
```toml
schema_version = 1
project_id = "example-cpp"
title = "Example C++ project"
language = "cpp"
source_roots = ["include", "src"]
compilation_database = "compile_commands.json"
```
`compilation_database` is forbidden for the other three languages.
## Published scope
All four adapters publish deterministic file, module or translation-unit, class or struct, and
function nodes where their syntax supports those categories. They publish lexical `contains`
relationships, a narrow set of project-local `depends_on` relationships, and function-scoped
Logic. Calls that appear inside Logic are syntax steps, not resolved symbol relationships.
### Python
Python uses `ast` from the standard library for source extraction and Logic. Manifest construction
does not build a Python AST; it tokenizes imports only far enough to publish dependencies that
resolve to another module in the declared source inventory.
It does not import or execute project code. It does not resolve dynamic imports, calls,
inheritance, imported symbols, types, overloads, re-exports, decorators, metaclasses, descriptors,
or runtime-generated behavior.
### JavaScript and TypeScript
JavaScript uses the optional `tree-sitter-javascript` grammar. TypeScript uses the distinct
optional `tree-sitter-typescript` grammar. Their focused incremental tests prove that manifest
construction and an unchanged warm build do not invoke the Tree-sitter extraction parser.
Only static relative imports and re-exports that resolve to another inventoried source file become
dependencies. Dynamic `import()`, `require()`, bare package specifiers, aliases, `tsconfig` paths,
loader hooks, types, interfaces, overloads, calls, inheritance, symbols, and runtime behavior are
not resolved. The adapters never import, compile, transpile, or execute project code.
### C++
C++ treats `compile_commands.json` as the authoritative bounded translation-unit inventory and as
fingerprint evidence. Commands, arguments, directories, and output fields are parsed as inert
data. The adapter executes no compiler, build tool, command, project binary, or project code.
The reference C++ manifest uses the optional `tree-sitter-cpp` grammar to parse inventoried sources
and discover quoted includes. It publishes a dependency only when that quoted include resolves
directly to a real project-local header in the declared roots. Angle-bracket includes, compiler
include paths, frameworks, generated headers, compiler-provided headers, conditional compilation,
and macro expansion are omitted.
This is not a Clang semantic adapter. It does not claim resolved calls, types, templates, aliases,
concepts, references, inheritance, overload ownership, out-of-line semantic ownership, macro
semantics, or compiler include semantics. A warm C++ extraction-cache hit is therefore not evidence
that manifest construction performed zero parser work.
## Complete and incremental proof
Each reference adapter implements:
- `load_projection()` for the complete primary graph compatibility oracle;
- `load_complete_assembly()` for the complete graph-plus-Logic oracle;
- `load_manifest()` and `extract_source()` for incremental extraction; and
- deterministic assembly of the complete current contribution set.
The maintained fixtures prove complete determinism, exact complete/incremental graph and Logic
parity, reverse-dependency invalidation, additions and deletions, corrupt extraction-cache
recovery, confinement, and exact unsupported-fact inventories. See
[Incremental Adapter Indexing](INCREMENTAL_INDEXING.md) for the cache contract and
[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for compatibility and policy limits.
Current fixture evidence is:
| Language | Primary nodes | Relationships | Logic projections |
|---|---:|---:|---:|
| Python | 14 | 13 | 5 |
| JavaScript | 14 | 14 | 5 |
| TypeScript | 13 | 14 | 4 |
| C++ | 17 | 16 | 5 |
These are regression-fixture shapes, not promises for arbitrary repositories.
## Read-only reference server
Start the fixed server with:
```bash
python -I -m docforge.reference_mcp \
--project-root /absolute/path/to/project \
--capability-mode read
```
The server selects one of the four in-repository providers solely from the validated fixed
configuration. Its cache stays below `.docforge/cache/reference-adapter/<language>`. It registers
the 21-tool read surface and no proposal or application tools. See
[Agent Integration](AGENT_INTEGRATION.md) for generated client fragments and
[MCP Boundary](MCP_CONTRACT.md) for the exact tool contract.

233
docs/RENDERING_AND_VISUALIZATION.md Normal file
View file

@ -0,0 +1,233 @@
# Rendering and visualization
DocForge has three independent ways to present one validated graph generation:
```text
validated generation
├── manual plan → immutable package → detached HTML renderer → declared manual output
├── graph plan → immutable package → detached graph renderer → portable static artifact
└── pinned index → managed loopback viewer → interactive Nodes, Flow, Web, and lazy Logic
```
They share validated facts but not authority, publication, or lifecycle. A manual render does not
publish a portable graph. A portable graph does not start the live viewer. None is canonical
project content.
## Manual rendering
A generic descriptor may declare a manual template root, isolated preview root, and stable views:
```toml
[render]
template_root = ".docforge/templates"
preview_root = ".docforge/previews"
[[render.views]]
id = "manual"
renderer = "generic_html"
template = "manual.html"
output = ".docforge/rendered/manual.html"
title = "My Project Manual"
families = ["architecture", "operations", "system"]
```
The renderer name is fixed to `generic_html`. Templates are confined UTF-8 assets with a fixed
token vocabulary. They cannot name commands, Python modules, executable renderers, or arbitrary
publication paths. Raw HTML in canonical Markdown is disabled by the pinned CommonMark path.
Plan and render identity covers the canonical source hash, optional changeset hash, selected nodes
and relationships, view configuration, template hash, renderer contract, and parser version.
Use:
```bash
docforge --project-root "$PROJECT" render-status
docforge --project-root "$PROJECT" render-status manual --deep
docforge --project-root "$PROJECT" render manual
docforge --project-root "$PROJECT" preview CHANGESET_ID manual
```
Normal status verifies bounded source, configuration, template, output, renderer, and receipt
identities without reconstructing output. `--deep` explicitly runs the side-effect-free full-render
oracle. A changeset preview writes only to the isolated preview root.
Manual `auto` means that a successful canonical application owns regeneration of every declared
manual view. It does not mean background rendering, and ordinary standalone CLI render cannot
select `auto`; use `explicit`.
## Portable graph rendering
Portable graph configuration is separate:
```toml
[graph_render]
output_root = ".docforge/portable-graph"
[[graph_render.views]]
id = "architecture"
renderer = "portable_graph_html"
output = "architecture.html"
title = "Architecture"
root = "architecture.overview"
initial_mode = "web"
depth = 3
max_nodes = 250
max_edges = 1000
max_work = 100000
families = ["architecture", "system"]
relations = ["depends_on", "owns", "calls", "reads", "writes", "tested_by", "relates_to"]
authorities = []
statuses = ["current", "active", "verified"]
tags = []
include_logic = false
```
A view selects exactly one exact `root` node or bounded metadata-only lexical `query`. Exact
filters and node, edge, depth, and work limits close the selection. The initial mode is `nodes`,
`flow`, or `web`. Portable graph contract version 1 excludes function-scoped Logic.
Use:
```bash
docforge --project-root "$PROJECT" graph-plan architecture
docforge --project-root "$PROJECT" graph-render architecture
docforge --project-root "$PROJECT" graph-render-status architecture
```
`graph-plan` validates and returns the generation-pinned plan without publication. `graph-render`
is an explicit local CLI publication action. It commits a content-addressed artifact, renderer
receipt, and bounded generation/view manifest in that order; the manifest is the publication
commit. `graph-render-status` checks only bounded committed evidence and never plans or renders.
MCP may plan and inspect portable graph evidence, but it does not publish the portable artifact.
Use the explicit project-local CLI command for publication.
Portable output is a complete static artifact. JavaScript is progressive enhancement, not a
requirement for the graph facts to be present.
## Live viewer
The live viewer is a managed, read-only browser over one generation-pinned validated SQLite file.
Install the per-user manager once:
```bash
docforge-viewer-manager install-user-service
```
Then operate a project viewer:
```bash
docforge --project-root "$PROJECT" visualize
docforge --project-root "$PROJECT" visualize --node architecture.overview
docforge --project-root "$PROJECT" visualize --query persistence
docforge --project-root "$PROJECT" visualization-status
docforge --project-root "$PROJECT" visualization-stop
```
`visualize` accepts only an optional stable node ID or lexical query, bounded traversal depth, and
the local `--no-open` presentation choice. It does not accept a project root override, database
path, SQL, template, command, renderer, bind address, or module.
The HTTP listener binds to `127.0.0.1` on an operating-system-selected port. A random token is part
of every accepted path. The server supports only `GET` and `HEAD`, sets no-store and restrictive
browser security headers, and has no write, project-selection, arbitrary-query, or static
filesystem endpoint.
The viewer is a snapshot. Index replacement or alteration makes that snapshot fail closed; start a
new visualization to use a new validated generation. Source inspection reads only project-confined
source evidence bound to the pinned generation.
## Nodes, Flow, Web, and Logic
The interactive viewer offers four complementary projections:
- **Nodes** shows a bounded relation-neutral incoming and outgoing neighborhood.
- **Flow** presents semantic contributors toward the focus. Prerequisite-style stored
relationships may be reversed for presentation without changing stored direction.
- **Web** expands the convergence picture with contributors, callers, containers, members, and
contextual relationships.
- **Logic** loads a function-scoped control-flow projection only when requested.
Logic supports the explicit control paths published by Python, JavaScript, TypeScript, and C++
integrations. It stays outside the primary graph, portable graph version 1, search, and
generation-diff receipts.
Hiding a node is presentation-only. In Flow and Web, ancestors without another path to the focus
are removed. In Logic, an omitted-path bridge preserves downstream readability. Restore reverses
the presentation change; neither action mutates the index.
## Immutable plan and worker boundary
Manual and graph plans are versioned, canonical JSON with deterministic ordering and fixed
structural and byte limits. They contain selected graph facts and bounded content, but no live
project object, SQLite handle, absolute project or index path, arbitrary query, command,
executable path, or caller-selected renderer module.
A projection package binds one plan to inert assets, fixed component versions, a closed built-in
renderer identity, and an exact artifact inventory. The detached worker:
- runs one fixed private module through isolated Python;
- uses a trusted working directory and sanitized environment;
- accepts one canonical newline-terminated JSON request;
- returns one bounded canonical JSON response;
- has a fixed renderer allowlist and timeout;
- cannot select graph facts, read project state, choose output paths, or mutate canonical files.
The package contract is bounded, and actual artifact transfer has a fixed 20,000,000-byte ceiling.
A larger descriptor `max_render_bytes` compatibility value does not widen that worker boundary.
## Manual fragment reuse
Manual fragments are disposable semantic cache records, not publication authority. A cold record
is accepted only after byte-exact comparison against a full detached render. On a warm hit, the
worker independently recomputes the expected fragment before reuse.
Corrupt, forged, stale, incompatible, individually oversized, or aggregate-oversized records fall
back to the complete full-render oracle. The current cache is bounded to 10,000 records and
64,000,000 bytes.
Fragment reuse is a correctness and recovery boundary. Do not promise a speedup without current
measurements.
## Projection policy
The version-2 projection policy is independent of capability mode:
```text
manual: auto | explicit | disabled
portable_graph: explicit | disabled
live_viewer: on-demand | disabled
```
Set global CLI flags before the subcommand:
```bash
docforge --project-root "$PROJECT" --manual-render-policy disabled render manual
docforge --project-root "$PROJECT" --portable-graph-policy disabled graph-plan architecture
docforge --project-root "$PROJECT" --live-viewer-policy disabled visualize
```
An active operation blocked by policy fails before hidden work. Receipt-only manual and portable
status remain available. Viewer status and explicit stop remain available when viewer startup is
disabled.
A non-disabled selection also requires its resource: declared manual configuration, declared
portable graph configuration, or viewer runtime. Manual `auto` additionally requires canonical
application. Read [Policy precedence](POLICY_PRECEDENCE.md) for exact composition.
## Failure and recovery
Render input drift detected before replacement fails without publishing a current receipt for stale
output. Portable graph publication uses content-addressed evidence and a final manifest commit.
Status never repairs implicitly.
Recovery is explicit:
- rerun a manual render from validated canonical state;
- use deep manual status for the full, side-effect-free equivalence oracle;
- rerun portable graph publication, or repair only from validated content-addressed evidence;
- stop and restart a stale live viewer against a current validated index.
Do not recover a derived output by editing its receipt or treating it as canonical. See [Recovery
and performance](RECOVERY_AND_PERFORMANCE.md), [Security](SECURITY.md), and the [Viewer
manager](VIEWER_MANAGER.md).

101
docs/SECURITY.md Normal file
View file

@ -0,0 +1,101 @@
# Security model
DocForge is a project-bound knowledge compiler. Its security boundary is an explicit project root,
closed configuration, bounded data, and exact identities. It is not a general process sandbox.
Start with [core authority](CORE_CONCEPTS_AND_AUTHORITY.md), then use
[policy precedence](POLICY_PRECEDENCE.md) to decide which capabilities a server should expose.
## Project and path confinement
Descriptors, reference-adapter configurations, canonical sources, authority files, templates,
changesets, caches, indexes, previews, and declared outputs are resolved against one project root.
DocForge rejects absolute paths where only project-relative paths are allowed, parent traversal,
symbolic-link escapes, unsafe file types, and protected-root overlap. Important reads use
no-follow file descriptors and compare file identity before and after reading.
Confinement protects DocForge operations. It does not stop another process with repository access
from changing files. Long-running bindings revalidate descriptor and adapter implementation
identity and require a restart after drift.
## Untrusted project content
Documentation, source text, templates, adapter metadata, compiler-database entries, and changeset
content are data. They cannot redefine policy or instruct DocForge to execute a command. Generic
manual rendering disables raw HTML, accepts a fixed template vocabulary, and rejects script-like
content. Portable graph and manual workers accept validated inert packages and fixed built-in
renderer identities.
The C++ reference adapter reads `compile_commands.json` only as bounded translation-unit inventory
and fingerprint evidence. It never executes the recorded command, compiler, response file, or
project program.
## Adapter launcher boundary
`AdapterLauncherV1` contains one Python module name and project identity. It contains no command,
shell string, arbitrary argument list, working directory, environment, discovery rule, or callable
selector.
Custom modules must be installed top-level modules. An isolated `python -I` probe resolves the
module without importing it and requires its regular-file origin to remain inside the bound
project. The sole trusted dotted exception is the packaged `docforge.reference_mcp` module. Client
fragments use the exact validated interpreter and canonical fixed arguments with an empty
environment.
This proves that the declared module is resolvable and project-bound. It does not make arbitrary
module code safe. Project owners remain responsible for the implementation they install.
## Mutation boundary
Normal MCP and the fixed reference server are read-only. Proposal tools exist only when a
startup-bound writer is authorized by the descriptor. Canonical application exists only when a
matching applier is explicitly configured.
Every proposal append, rebase, abandonment, and application is hash-bound. Application requires
the exact changeset hash that was reviewed. Source identity, content hashes, permissions,
conflicts, graph validity, and serializer round trips are checked before success. DocForge never
turns prose approval into a fuzzy merge.
## Derived state and publication
SQLite indexes, source-generation receipts, extraction caches, render fragments, previews, and
portable artifacts are disposable. Corrupt, stale, foreign, oversized, or mismatched derived
state is rejected or rebuilt from current project evidence.
Generated command-reference publication serializes cooperating writers and uses no-clobber or
compare-and-swap publication. A raced target is restored or retained for recovery instead of being
silently discarded. Projection publication records when an artifact was committed but later
receipt verification degraded, so a completed mutation is never reported as an ordinary failure.
## Limits and denial-of-service resistance
Inputs, results, traversal, context, changesets, renders, worker protocols, manifests, adapter
assemblies, and extraction caches have explicit count and byte limits. Incremental extraction
caches are capped at 10,000 sources and 64,000,000 bytes. Adapter primary nodes use the project
`max_nodes` limit; edges and Logic have deterministic multipliers over that limit.
Limits reduce accidental and adversarial amplification. An in-process adapter can still allocate
memory before returning data, so only trusted project-owned adapter code should run in the server
process.
## Secrets and network behavior
DocForge does not copy the parent environment into generated client fragments or detached
projection workers. Doctor checks never return environment values. The live viewer binds to
loopback, uses an unguessable URL token, supports read-only methods, and serves no arbitrary
filesystem tree.
Project secrets must not be placed in canonical documentation, adapter configuration, compiler
databases, templates, or changesets. Repository release gates include secret scanning, but that
scan is not a substitute for credential hygiene.
## No-AST boundary
`--no-ast` is a binding policy that preserves the selected adapter and prohibits Logic publication
and retrieval. It is not a parser detector, filesystem sandbox, or promise that unrelated
processes cannot parse source. See [legacy and no-AST operation](LEGACY_AND_NO_AST.md).
## Reporting and recovery
Do not bypass a confinement, identity, policy, hash, or limit error. Preserve the failing evidence,
stop the affected binding, and follow [recovery and performance](RECOVERY_AND_PERFORMANCE.md).

107
docs/USER_MANUAL.md
View file

@ -35,6 +35,8 @@ incremental methods while retaining the full loader as a fallback.
relationship keys, source inspection, branch-aware node hiding, panel resizing, zooming, and
managed idle shutdown.
- A generic Markdown/TOML adapter plus contracts for deterministic project-owned adapters.
- A public adapter SDK, optional Python/JavaScript/TypeScript/C++ reference integrations, and one
fixed read-only reference MCP binding.
DocForge does not run shell commands from documentation, mutate Git, build an application, deploy,
publish, choose a project globally, or cross project boundaries.
@ -100,6 +102,49 @@ uv run pytest -q
Use the executables under `/absolute/path/DocForge/.venv/bin/` when DocForge is not installed into
the active shell environment.
For an installed distribution, choose only the language extras the project needs:
```bash
python -m pip install docforge
python -m pip install 'docforge[javascript]'
python -m pip install 'docforge[typescript]'
python -m pip install 'docforge[cpp]'
```
The base wheel contains the Python reference adapter and no Tree-sitter distribution. JavaScript,
TypeScript, and C++ require their matching optional extras. `docforge[languages]` installs all
three optional frontend groups.
### Configure a reference source project
Reference adapters are a narrow alternative to the generic documentation descriptor. Create
`.docforge/reference-adapter.toml`:
```toml reference-adapter
schema_version = 1
project_id = "my-python-project"
title = "My Python Project"
language = "python"
source_roots = ["src"]
```
Start the fixed read-only server:
```bash
python -I -m docforge.reference_mcp \
--project-root /absolute/path/MyProject \
--capability-mode read
```
For C++, set `language = "cpp"` and add a project-relative
`compilation_database = "compile_commands.json"`. The database is inert bounded inventory; the
reference adapter does not execute its commands or compiler.
The reference integrations publish syntax and local static relationships only. They do not claim
resolved calls, types, inheritance, macro behavior, compiler include semantics, runtime behavior,
or semantic ownership. See [reference adapters](REFERENCE_ADAPTERS.md) for exact evidence and
limitations.
### Assess and onboard an unconfigured project
Run a read-only assessment before writing configuration:
@ -478,6 +523,10 @@ Every command emits deterministic JSON:
docforge --project-root /absolute/path/MyProject <command>
```
The sections below group common workflows. The implementation-derived list of all 28 current
commands, exact invocations, 36 generic MCP tools, arguments, and input-schema hashes is the
[generated command reference](COMMAND_REFERENCE.md).
### Project and index commands
```text
@ -590,6 +639,31 @@ declares the named writer, and application requires the same writer/applier iden
`--no-ast` to preserve the no-AST binding. Generic CLI generation refuses project-owned adapters
because it cannot safely reconstruct their composition.
A custom adapter owner supplies the already constructed project and immutable launcher through the
Python API:
```python
from docforge.adapter_launcher import AdapterLauncherV1
from docforge.client_config import generate_adapter_client_configuration
launcher = AdapterLauncherV1.for_project(
project,
module="my_project_docforge",
)
fragment = generate_adapter_client_configuration(
project,
launcher,
"codex",
capability_mode="read",
)
```
The top-level module must be installed for the exact isolated Python environment and resolve to a
regular file inside the project root. The fixed `docforge.reference_mcp` module is the only trusted
dotted exception. Generation probes resolution without importing the custom module, binds current
source availability and policy, and emits no arbitrary command, arguments, working directory, or
environment.
Select projection behavior independently:
```bash
@ -684,6 +758,10 @@ docforge-mcp \
Omit `--proposal-writer` when the MCP client should not create or append proposals.
For `.docforge/reference-adapter.toml`, use the fixed `docforge.reference_mcp` command shown in
[setup](#configure-a-reference-source-project). It exposes exactly the 21 read tools and never
registers proposal or application tools.
Select the session's declared surface explicitly when useful:
```bash
@ -986,6 +1064,28 @@ changes outside this process boundary.
## Troubleshooting
### `optional_dependency_missing`
Install the exact extra named in the error into the same Python environment that starts DocForge:
```bash
python -m pip install 'docforge[javascript]'
python -m pip install 'docforge[typescript]'
python -m pip install 'docforge[cpp]'
```
Do not install every frontend merely to suppress the check. A missing optional parser is a closed,
actionable capability error and does not affect base generic or Python reference operation.
### `adapter_launcher_unavailable` or `invalid_adapter_launcher`
Use one installed top-level Python module whose resolved regular-file origin is inside the project
root, or use the fixed `docforge.reference_mcp` binding. Arbitrary dotted modules, packages,
stdlib modules, missing modules, commands, argument strings, working directories, and environment
injection are rejected. Test the exact generated fragment rather than editing its command by hand.
See [agent integration](AGENT_INTEGRATION.md) and the [security model](SECURITY.md).
### `adapter_restart_required`
The project-local adapter code, its declared descriptor, or another implementation file changed
@ -1128,8 +1228,11 @@ make gate
Use `make benchmark` for the historical Milestone 0 baseline, `make benchmark-m1` for the
counter-gated warm-operation benchmark, `make benchmark-m2` for agent workflow gates, and
`make benchmark-m3-full` for the ten-sample 1,000-node projection, worker, fragment, status,
equivalence, response-size, and memory gates. `make accessibility` runs the generated manual,
portable graph, and live viewer axe and keyboard flows.
equivalence, response-size, and memory gates. `make benchmark-m4-full` runs the 1,002-node adapter
and recovery benchmark. `make adoption-m4` performs the offline fresh-wheel proof.
`make command-reference-check` rejects command-reference drift, and `make docs-check` validates
the maintained documentation graph. `make accessibility` runs the generated manual, portable
graph, and live viewer axe and keyboard flows.
Project-specific vocabulary, extraction rules, and serialization belong in the project adapter.
Generic core behavior must remain deterministic, project-bound, and recoverable.

14
tools/docs_pending_m4_pages.txt
View file

@ -1,13 +1 @@
# Remove each entry in the same commit that authors the required page.
docs/AGENT_INTEGRATION.md
docs/CORE_CONCEPTS_AND_AUTHORITY.md
docs/LEGACY_AND_NO_AST.md
docs/MIGRATING_FROM_V1.md
docs/MILESTONE_4_BASELINE.md
docs/MILESTONE_4_CLOSEOUT.md
docs/POLICY_PRECEDENCE.md
docs/PROJECT_DESCRIPTOR.md
docs/RECOVERY_AND_PERFORMANCE.md
docs/REFERENCE_ADAPTERS.md
docs/RENDERING_AND_VISUALIZATION.md
docs/SECURITY.md
# Milestone 4 required-page inventory is complete.