1
0
Fork 0
Code Issues Pull requests Projects Releases 2 Packages Wiki Activity Actions Pages

Compare commits

..

23 commits

Author SHA1 Message Date
6d06195950 Close Milestone 4 with adapter adoption evidence 2026-07-29 15:34:25 -04:00
95271dcf2e Expand documentation gate coverage 2026-07-29 15:21:54 -04:00
9023f28cc0 Add strict documentation graph gate 2026-07-29 15:21:29 -04:00
983a55f156 Add fresh-wheel adoption proof 2026-07-29 15:11:56 -04:00
f0dea0cedc Document generated command surface 2026-07-29 15:10:47 -04:00
cd3c761988 Freeze adapter launcher public imports 2026-07-29 15:10:07 -04:00
663811b1d3 Format Milestone 4 hardening code 2026-07-29 15:05:11 -04:00
cca374bf99 Gate generated command reference 2026-07-29 15:04:12 -04:00
e525efd2fc Add Milestone 4 adapter benchmark gate 2026-07-29 15:03:31 -04:00
ecbf94d14a Prove adapter launcher modules are runnable 2026-07-29 15:01:11 -04:00
3bf3836ac3 Make command reference publication race-safe 2026-07-29 15:00:28 -04:00
efe4a443b7 Add strict reference project MCP binding 2026-07-29 14:54:10 -04:00
7bee0220c2 Add deterministic C++ reference adapter 2026-07-29 14:53:21 -04:00
f00aa65a73 Bound adapter assemblies and extraction caches 2026-07-29 14:52:38 -04:00
9cd7c4e424 Keep reference manifests parser-free 2026-07-29 14:52:05 -04:00
58c0196be5 Add JavaScript and TypeScript reference adapter 2026-07-29 14:39:53 -04:00
cb52bf8ae6 Add project-owned adapter client launchers 2026-07-29 14:33:42 -04:00
0d498fc385 Add deterministic command reference tool 2026-07-29 14:31:15 -04:00
8956c5c5f5 Add deterministic Python reference adapter 2026-07-29 14:30:05 -04:00
e6523c0c00 Add adapter SDK conformance proofs 2026-07-29 14:23:29 -04:00
85f6629cb1 Generate command references from live registrations 2026-07-29 14:20:43 -04:00
395d732348 Make language frontends optional 2026-07-29 14:18:31 -04:00
fee2bb0080 Activate Milestone 4 adapter SDK contract 2026-07-29 14:14:52 -04:00
83 changed files with 14959 additions and 312 deletions

View file

@ -1,13 +1,15 @@
# Milestone state
```text
Last completed milestone: 3 — independent projections
Outcome: Manual output, portable graph artifacts, and the live viewer are independent generation-pinned consumers of the validated graph.
Evidence: Clean candidate f5dccb5e1c312121f1af63780162f593d9363b98; 281 tests and 272 subtests; 3 accessibility flows; clean 1,000-node ten-sample benchmark; isolated wheel proof; no secret-scan findings.
Last completed milestone: 4 — adapter SDK and product documentation.
Evidence: Frozen executable candidate 95271dcf2e48045b9d3aed9b9ea09c7fc155692c; 347 tests and 402 subtests; 3 accessibility flows; offline fresh-wheel adoption; clean 1,002-node graph-plus-Logic benchmark; no secret-scan findings.
Active milestone: None.
Next directional milestone: 4 — adapter SDK and product documentation.
Status: Milestone 3 is closed. Milestone 4 has not started.
Outcome: New projects can adopt DocForge incrementally, and users can understand the CLI, MCP server, policies, adapters, renderers, and agent integration without reading implementation code.
Status: Milestone 4 is closed. Milestone 5 has not started.
Next directional milestone: 5 — stabilization and first DocForge2 release.
```
Milestones 45 remain directional context. Do not begin Milestone 4 without a new active-slice
contract.
Milestone 5 remains directional context. Do not begin release stabilization, versioning, tagging,
or publication without a new active-slice contract based on the Milestone 4 evidence.
The completed Milestone 4 contract and its exclusions are preserved in `SLICE_HISTORY.md`.

View file

@ -784,3 +784,108 @@ proof. Gitleaks 8.30.1 found no findings across the six Milestone 3 commits or c
Milestone 3 is complete. No tag, release, production integration repointing, WorldForge change,
ScrapeStation change, storage rewrite, or self-hosting dependency was introduced. Milestone 4
remains directional and has not started.
## Milestone 4 — complete: adapter SDK and product documentation
Milestone 4 activated from the verified Milestone 3 closeout. Its scope was deliberately additive:
freeze an adapter-authoring boundary, prove narrow language references, make heavy frontends
optional, generate command and client integration evidence, and document adoption. WorldForge,
ScrapeStation, the legacy repository, production bindings, self-hosting, storage replacement, and
release publication remained excluded.
### Adapter SDK and conformance
`docforge.adapter_sdk` is now the public authoring facade for typed projections, manifests, source
contributions, complete assemblies, incremental loaders, project bindings, core graph models, and
conformance reports. Complete evidence includes primary graph and function Logic. An incremental
adapter that publishes Logic must provide an independent `load_complete_assembly()` oracle.
The conformance helper repeats complete loading for determinism, compares the complete assembly
with `load_projection()`, and proves exact complete/incremental graph-plus-Logic parity. It does not
pretend to replace separate confinement, restart, no-AST, corrupt-cache, or retrieval tests.
An adversarial review found that extraction caches and adapter assemblies lacked aggregate
boundaries. Version-1 cache reads and writes are now regular-file-only, identity-checked, capped at
64,000,000 bytes and 10,000 sources, and fail safely to a miss. Adapter projects enforce primary
node limits and deterministic edge and Logic multipliers before publication.
### Reference language evidence
The Python reference uses the standard-library AST and publishes files, modules, classes,
functions, arguments, local static imports, and function Logic. Its manifest uses tokenization
rather than AST parsing, so unchanged warm builds perform no syntax parse.
JavaScript and TypeScript use separate pinned optional Tree-sitter grammars. They publish files,
modules, classes, functions/methods, static project-relative imports/re-exports, and function
Logic. Their manifests use a closed comment/string-aware ESM scan and do not load Tree-sitter.
C++ uses explicit non-overlapping source roots and one confined `compile_commands.json` as
translation-unit inventory and fingerprint evidence. It validates entries but never executes a
command, compiler, response file, or project program. It publishes syntax and directly resolvable
project-local quoted includes. Its manifest currently parses for include discovery, so no
zero-warm-parser claim is made.
None of the references claim compiler-resolved calls, inheritance, types, symbol references,
compiler include semantics, macro semantics, runtime behavior, or semantic ownership.
The first packaging proof exposed that language parsers were inherited as mandatory dependencies.
The base wheel now depends only on Markdown and MCP packages. Python works in the base wheel;
JavaScript, TypeScript, C++, and all-language extras install their respective Tree-sitter
frontends. Missing extras return a closed error with the exact install target.
### Fixed reference MCP and launchers
A strict `.docforge/reference-adapter.toml` selects one fixed reference language, project
identity, and explicit source roots. C++ additionally requires the compilation database. The fixed
`python -m docforge.reference_mcp` binding constructs only repository-owned providers and exposes
the 21 read tools.
The initial launcher design generated `python -I -m` for arbitrary project module names without
proving that isolated Python could resolve them. The corrected `AdapterLauncherV1` accepts only an
installed project-confined top-level module or the exact trusted `docforge.reference_mcp`
exception. A fixed bounded `find_spec` probe runs under isolated Python without importing project
code. Tests then launch both a real project module and the real reference MCP process.
Generated Codex, Claude, and OpenClaw fragments bind the exact launcher, descriptor, source
availability, effective and projection policy, interpreter, canonical arguments, empty
environment, artifact bytes, and hashes. There is no arbitrary command, argument list, working
directory, environment, discovery, or callable selector.
### Generated reference and documentation gates
CLI tables are derived from the real argparse subcommands. MCP tables are derived from a real
application-enabled server's `list_tools()` registrations, including capability surface,
arguments, descriptions, and input-schema hashes. The generated artifact contains 28 CLI rows and
36 MCP rows.
An adversarial review found a target-change race between the generator's identity check and
replacement. Publication now locks cooperating generators, uses no-clobber publication for a
missing target, and uses Linux atomic exchange plus displaced-byte and file-identity verification
for an existing target. Raced data is restored or retained for recovery rather than discarded.
`docs-check` validates generated drift, local files and anchors, one H1 per page, README-rooted
reachability, required Milestone 4 inventory, the generated notice, and every documented
reference-adapter TOML example against the packaged schema.
### Candidate, adoption, and scale evidence
Frozen executable candidate `95271dcf2e48045b9d3aed9b9ea09c7fc155692c` passed formatting,
Python and web lint, strict Pyright, compilation, 142 contract tests and 268 subtests, 347 total
tests and 402 subtests, three accessibility flows, lock and dependency checks, package builds,
offline wheel adoption, and every maintained smoke benchmark.
The offline adoption proof installed the base wheel from the lock without network access. It
proved no Tree-sitter package or module was installed, built and checked a real Python reference
project, performed isolated MCP bootstrap/search/get-node over the exact 21 read tools, and proved
that base-only C++ fails with `optional_dependency_missing` and `docforge[cpp]` remediation.
The clean 334-source benchmark produced 1,002 primary nodes, 1,001 primary edges, 334 Logic
projections, 2,338 Logic nodes, and 2,338 Logic edges. Complete and incremental output matched
exactly. Warm build p95 was 1,265.387 ms with zero `ast.parse` and zero `extract_source` calls.
Complete equivalence, corrupt cache recovery, and corrupt index recovery were all below 1.9
seconds. The largest per-operation traced peak was 69,997,166 bytes and process high-water was
78,798,848 bytes. Exact hashes and method are recorded in
`benchmarks/milestone4-2026-07-29.json` and `docs/MILESTONE_4_BASELINE.md`.
Milestone 4 is complete. No tag or release was created. Milestone 5 remains unstarted until its own
active-slice contract is frozen.

View file

@ -5,7 +5,7 @@ NPM := npm
PYTHONPYCACHEPREFIX := /tmp/docforge-quality-pycache
PYTEST_BASETEMP := /tmp/docforge-quality-pytest
.PHONY: accessibility benchmark benchmark-m1 benchmark-m1-smoke benchmark-m2 benchmark-m2-smoke benchmark-m3 benchmark-m3-full benchmark-m3-smoke benchmark-smoke build compile contract dependencies format-check gate lint lock test type
.PHONY: accessibility adoption-m4 benchmark benchmark-m1 benchmark-m1-smoke benchmark-m2 benchmark-m2-smoke benchmark-m3 benchmark-m3-full benchmark-m3-smoke benchmark-m4 benchmark-m4-full benchmark-m4-smoke benchmark-smoke build command-reference-check compile contract dependencies docs-check format-check gate lint lock test type
accessibility:
$(NPM) run test:accessibility
@ -59,6 +59,17 @@ dependencies:
build:
$(UV) build
adoption-m4:
$(PYTHON) tools/milestone4_adoption.py
command-reference-check:
$(PYTHON) tools/generate_command_reference.py \
--output docs/COMMAND_REFERENCE.md --check > /dev/null
docs-check: command-reference-check
$(PYTHON) tools/check_documentation.py \
--pending-inventory tools/docs_pending_m4_pages.txt
benchmark-smoke:
$(PYTHON) tools/milestone0_baseline.py --nodes 25 --samples 1 --cold-samples 1 \
--output /tmp/docforge-milestone0-smoke.json > /dev/null
@ -89,4 +100,13 @@ benchmark-m3:
benchmark-m3-full: benchmark-m3
gate: format-check lint type compile contract test accessibility lock dependencies build benchmark-smoke benchmark-m1-smoke benchmark-m2-smoke benchmark-m3-smoke
benchmark-m4-smoke:
$(PYTHON) tools/milestone4_benchmark.py --mode smoke \
--output /tmp/docforge-milestone4-smoke.json > /dev/null
benchmark-m4:
$(PYTHON) tools/milestone4_benchmark.py --mode full
benchmark-m4-full: benchmark-m4
gate: format-check lint type compile contract test accessibility lock dependencies build docs-check benchmark-smoke benchmark-m1-smoke benchmark-m2-smoke benchmark-m3-smoke benchmark-m4-smoke

409
README.md
View file

@ -1,88 +1,107 @@
# DocForge
DocForge is a project-scoped documentation graph for people and AI agents. It validates canonical
documentation, builds a disposable search and relationship index, compiles bounded context, renders
declared manuals, visualizes project structure, and manages reviewable documentation changesets.
DocForge is a project-scoped documentation and source graph for people and AI agents. Canonical
project files remain authoritative; DocForge validates them, builds disposable search and
relationship indexes, compiles bounded context, manages reviewable changesets, renders declared
manuals and portable graph views, and serves one project-bound MCP surface.
## What it does
DocForge never treats indexed text as instructions. It does not run project build commands,
compilers, Git operations, deployments, or arbitrary renderers, and it does not select projects
globally.
- Validates stable Markdown/TOML nodes and typed relationships.
- Builds a deterministic SQLite search and graph index.
- Exposes project-bound CLI and MCP query surfaces.
- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, and bounded
continuation.
- Records one bounded, versioned latest-generation graph transition without creating a history
database.
- Generates deterministic project-bound Codex, Claude, and OpenClaw client fragments without
copying ambient secrets.
- Diagnoses one client binding through bounded read-only checks without starting MCP or rebuilding
project state.
- Automatically synchronizes disposable indexes before MCP work.
- Creates, validates, diffs, and previews isolated changesets.
- Registers complete proposals atomically without caller-managed hash chaining.
- Applies one explicitly approved changeset hash through CLI or gated MCP.
- Supports opt-in incremental adapters with reverse-dependency invalidation and full-build
equivalence checks.
- Detects project-local adapter implementation and configuration changes and requires a fresh
project-bound process before any further MCP work.
- Keeps function-scoped control-flow projections separate from the primary architecture graph.
- Compiles manuals and portable graph artifacts from separate versioned, generation-pinned plans
and immutable packages.
- Runs built-in manual and graph renderers in fixed detached workers with validated receipts,
bounded transfer, and no project-path authority.
- Enforces independent manual, portable-graph, and live-viewer policy while keeping status
receipt-only.
- Runs a managed loopback graph browser with neighborhood, semantic Flow, convergence Web,
function-scoped Logic, source inspection, and branch-aware node hiding.
- Supports generic documentation projects and project-owned source adapters.
DocForge 1.0.0 remains the first stable release in this product line. This checkout is the
post-1.0 development line (`1.3.0.dev0`) and contains the Milestone 4 adapter-platform work.
Milestone 4 is not a new product release, and Milestone 5 release and publication work remains
separate.
DocForge never treats indexed text as instructions. It does not run shell commands, mutate Git,
build applications, deploy, publish, or select projects globally.
## Start here
For implementation projects, the recommended cadence is to read canonical documentation during
intake, keep it read-only through implementation and focused testing, freeze and validate a release
candidate, then perform one atomic documentation closeout before the final commit and tag. This
keeps the manual authoritative without using it as an implementation notebook.
- New installation or first project: [New-project quickstart](docs/NEW_PROJECT_QUICKSTART.md)
- Mental model and authority: [Core concepts and authority](docs/CORE_CONCEPTS_AND_AUTHORITY.md)
- Complete configuration shape: [Project descriptor](docs/PROJECT_DESCRIPTOR.md)
- Fixed runnable examples: [Reference adapters](docs/REFERENCE_ADAPTERS.md)
- Agent and client setup: [Agent integration](docs/AGENT_INTEGRATION.md)
- Exact live command and tool inventory: [Generated command reference](docs/COMMAND_REFERENCE.md)
- Task-oriented operating guide: [User manual](docs/USER_MANUAL.md)
## Release 1
## Current capabilities
DocForge 1.0.0 is the first stable product release. It combines the project-scoped graph, CLI and
MCP query surfaces, reviewable hash-approved changesets, generic and project-owned adapters,
declared rendering, and the complete Nodes/Flow/Web visualization model in one supported release.
- Validates Markdown and TOML nodes, stable IDs, typed relationships, project limits, and confined
paths.
- Builds a deterministic, disposable SQLite graph and search index with integrity and generation
evidence.
- Exposes project-bound CLI and MCP read, proposal, application, rendering, and visualization
surfaces according to the startup policy.
- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, bounded
output, and deterministic continuation.
- Creates isolated documentation changesets, validates complete projected graphs, and applies only
one explicitly approved changeset hash through a separately bound canonical applier.
- Supports complete-projection adapters and opt-in incremental adapters with reverse-dependency
invalidation, bounded extraction caches, and a clean full-build equivalence oracle.
- Publishes function-scoped Logic separately from the primary graph for Python, JavaScript,
TypeScript, and C++ integrations that provide it.
- Compiles manual and portable graph plans into immutable packages for fixed detached renderers,
then records bounded receipts.
- Runs a managed loopback graph viewer with Nodes, Flow, Web, lazy Logic, source inspection, and
branch-aware hiding.
- Generates deterministic Codex, Claude, and OpenClaw client fragments without copying ambient
environment values or secrets.
The post-1.0 incremental compiler is a backward-compatible, optional enhancement. Existing Release
1 adapters that implement only `load_projection()` continue to use the original complete-projection
path without modification. Adapters gain incremental performance only when they additionally
implement the source manifest and extraction methods. Incremental adapters must retain
`load_projection()` as their clean-rebuild fallback and equivalence oracle.
## Adapter platform
MCP bindings can additionally select `--no-ast` when an owner wants to preserve an existing
non-AST adapter. The binding advertises that policy to clients, forbids adapter rewrites that add
AST, Tree-sitter, compiler-AST, or function-Logic extraction, blocks the Logic tool, and rejects
nonempty Logic publication. Complete-projection adapters continue unchanged, and non-AST
incremental fingerprinting and caching remain allowed.
Adapter authors use the public `docforge.adapter_sdk` surface for graph types, complete and
incremental contracts, validation helpers, and `verify_adapter_conformance()`. Existing adapters
that implement only `load_projection()` remain supported. Incremental adapters add a manifest and
source extraction while retaining `load_projection()` as the complete graph oracle. An adapter
that publishes Logic incrementally also supplies `load_complete_assembly()` so the complete oracle
covers both graph and Logic.
DocForge2 bindings may also declare
`--capability-mode read|proposal|application|operator`. Bootstrap returns one versioned effective
policy and the actual startup-gated capabilities. Existing tool surfaces and the legacy no-AST
payload remain compatible.
The repository includes bounded reference integrations for:
`docforge_get_task_context` is an additive read tool for `change`, `implementation`, `failure`,
`ownership`, `test`, `operation`, and `release` work. It derives a closed version-1 retrieval plan,
executes it against one immutable index generation, and returns a hash-bound context capsule.
Project relation names remain authoritative. DocForge classifies only its versioned alias set and
preserves every unknown relation as `unclassified` instead of guessing semantics.
- Python, using the standard-library AST and publishing only project-local imports;
- JavaScript and TypeScript, using their distinct optional Tree-sitter grammars and publishing only
project-local static relative imports and re-exports;
- C++, using `compile_commands.json` as translation-unit inventory and fingerprint evidence, without
executing its commands or a compiler, and publishing only directly resolvable project-local
quoted includes.
`docforge_get_generation_diff` reports the latest verified primary-graph transition through one
bounded disposable receipt. It includes exact node and edge change counts, hash-bound retained
details, and explicit truncation. Paged results use one top-level cursor and a versioned
`receipt_header`; `stored_receipt_hash` identifies the complete persisted receipt. The read never
exposes Logic details, loads canonical source, repairs derived state, or invents history.
These integrations demonstrate the adapter contract; they do not claim resolved calls,
inheritance, types, runtime behavior, macro expansion, compiler include semantics, or semantic
ownership. Python works from the base wheel. Install the `javascript`, `typescript`, or `cpp`
extra for the corresponding grammar, or `languages` for all three:
## Independent projections
```bash
uv pip install "/absolute/path/DocForge[javascript]"
uv pip install "/absolute/path/DocForge[typescript]"
uv pip install "/absolute/path/DocForge[cpp]"
uv pip install "/absolute/path/DocForge[languages]"
```
Manual compilation, portable graph rendering, and the live viewer consume the same validated graph
generation through separate boundaries:
Reference projects use the fixed `.docforge/reference-adapter.toml` descriptor and the installed
`docforge.reference_mcp` module. That server is read-only and exposes no proposal or application
surface.
Project-owned adapters use `AdapterLauncherV1` and
`generate_adapter_client_configuration()` to produce a client fragment from an explicitly
constructed adapter project. The launcher is immutable and contains no command, caller arguments,
working directory, environment, discovery rule, or callable selector. It resolves through isolated
Python to one installed, project-owned top-level module; the only trusted dotted exception is the
fixed `docforge.reference_mcp` binding. Generic `docforge configure` intentionally refuses custom
adapters because it cannot safely reconstruct project-owned composition.
See the [Adapter authoring guide](docs/ADAPTER_AUTHORING_GUIDE.md), [Reference
adapters](docs/REFERENCE_ADAPTERS.md), and [Agent integration](docs/AGENT_INTEGRATION.md) for the
supported routes.
## Authority and projections
Canonical Markdown, TOML, adapter-declared sources, and descriptor files own project facts. The
SQLite index, extraction cache, render packages, previews, portable artifacts, receipts, client
fragments, and viewer processes are derived and replaceable.
The primary graph contains project nodes and relationships. Logic is a separate, lazy,
function-scoped control-flow projection. Manual output, portable graph output, and the live viewer
are independent consumers of one validated generation:
```text
validated generation
@ -91,190 +110,142 @@ validated generation
└── pinned index → managed read-only live viewer
```
Plans, packages, and receipts are canonical, versioned, hash-identified, bounded, and contain no
project object, SQLite handle, absolute project path, command, or caller-selected renderer module.
Renderers cannot select graph facts, crawl canonical sources, choose publication paths, or mutate
project state.
The version-2 projection policy selects each consumer independently:
Declare portable graph output separately from manual views:
```toml
[graph_render]
output_root = ".docforge/portable-graph"
[[graph_render.views]]
id = "architecture"
renderer = "portable_graph_html"
output = "architecture.html"
title = "Architecture"
root = "architecture.overview"
initial_mode = "web"
depth = 3
max_nodes = 250
max_edges = 1000
max_work = 100000
include_logic = false
```text
manual: auto | explicit | disabled
portable_graph: explicit | disabled
live_viewer: on-demand | disabled
```
Plan, publish, and inspect it explicitly:
The process capability policy is separate. Capability mode controls the registered read,
proposal, application, or reserved operator surface; descriptor writers and the startup-bound
canonical applier determine whether mutations are actually authorized. `--no-ast` is a restrictive
binding policy over adapter evolution, Logic publication, and Logic retrieval. It is not a parser
inspection mechanism or a filesystem sandbox, and the AST/Tree-sitter reference integrations
should not be presented as no-AST adapters.
Read [Policy precedence](docs/POLICY_PRECEDENCE.md), [Rendering and
visualization](docs/RENDERING_AND_VISUALIZATION.md), and [Legacy and no-AST
operation](docs/LEGACY_AND_NO_AST.md) before changing those boundaries.
## Five-minute generic project
Requirements are Python 3.12 or newer and an installed DocForge environment. From a development
checkout, `uv sync --group dev` creates `.venv`:
```bash
.venv/bin/docforge --project-root "$PROJECT" graph-plan architecture
.venv/bin/docforge --project-root "$PROJECT" graph-render architecture
.venv/bin/docforge --project-root "$PROJECT" graph-render-status architecture
```
The version-2 projection policy independently selects manual
`auto|explicit|disabled`, portable graph `explicit|disabled`, and live viewer
`on-demand|disabled`. Use `--manual-render-policy`, `--portable-graph-policy`, and
`--live-viewer-policy` on CLI/MCP startup or generated client configuration. Status remains
available when the corresponding active operation is disabled.
Manual fragment reuse is disposable. Cold record publication is guarded by byte-exact comparison
with a full detached render; warm records are independently recomputed and validated inside the
worker. Full rendering remains the recovery and equivalence oracle. Portable artifacts commit
content-addressed output and renderer evidence before one bounded generation/view manifest; status
does not plan or render.
## Graph views
The browser presents the primary architecture graph through three complementary views and loads a
fourth function-scoped view only when requested:
- **Nodes** shows a bounded, relation-neutral neighborhood around the focus. It is the broad
inspection view for seeing stored incoming and outgoing relationships without changing their
direction. Semantic cards distinguish structure, behavior, dependencies, execution, data,
evidence, context, and other relationships.
- **Flow** shows semantic origin-to-destination paths that terminate at the focus. DocForge
reverses prerequisite-style relationships for presentation, so imports, dependencies, reads,
inheritance, definitions, and tests flow toward the thing they help create or exercise.
- **Web** shows the larger convergence picture: Flow contributors plus contextual relationships,
callers, containers, and direct members or execution dependencies owned by the focus.
- **Logic** shows the possible static control paths inside a focused Python, JavaScript, or C++
function or method. Entry, decisions, actions, loops, convergence points, returns, and
exceptions connect through explicit
`TRUE`, `FALSE`, `NEXT`, `CASE`, `LOOP`, `RETURN`, and `RAISE` paths. Logic is stored separately
and does not add statement-level noise to Nodes, Flow, Web, or search.
Graph cards show the node's readable leaf name and kind without clipping either value. The full
qualified identity remains available in the tooltip, compact descriptor, and full inspector.
The left browser panel can combine text, family, node-kind, language, and capability filters.
Quick presets expose Logic-ready nodes, Python callables, tests, routes, and documentation without
requiring users to know stable IDs. Selecting a canvas node emphasizes its directly connected
neighbors and edges while muting unrelated paths.
**Hide node** removes noise without changing the index. In Flow and Web, hiding a contributor also
removes upstream ancestors that no longer have a path to the focus. Nodes between the hidden
contributor and the focus stay visible, and alternate ancestor paths remain intact. In Logic,
hiding a step inserts an explicit omitted-path bridge so downstream control flow remains readable.
**Restore hidden** restores the presentation.
## Five-minute start
Requirements are Python 3.12+, `uv`, and Node.js/npm.
```bash
git clone forgejo@repo.andraxion.net:administrator/DocForge2.git /absolute/path/DocForge2
cd /absolute/path/DocForge2
git clone <repository-url> /absolute/path/DocForge
cd /absolute/path/DocForge
uv sync --group dev
npm ci
PROJECT=/absolute/path/MyProject
.venv/bin/docforge --project-root "$PROJECT" onboard
.venv/bin/docforge --project-root "$PROJECT" onboard \
--scaffold \
--project-id my-project \
--title "My Project"
```
The first command is read-only. Scaffolding is explicit and create-only: it writes a generic
descriptor, one canonical overview node, and a built-in manual template, then indexes and renders
them. Detected source languages remain `adapter_required` until a validated frontend is selected.
Operate the configured project:
```bash
.venv/bin/docforge --project-root "$PROJECT" validate
.venv/bin/docforge --project-root "$PROJECT" reindex
.venv/bin/docforge --project-root "$PROJECT" search architecture
.venv/bin/docforge --project-root "$PROJECT" visualize
```
Install the persistent per-user graph viewer once:
Install the persistent per-user graph viewer manager once when using the live viewer:
```bash
.venv/bin/docforge-viewer-manager install-user-service
```
Start an MCP server for one project:
Start one generic, read-only MCP server:
```bash
.venv/bin/docforge-mcp \
--project-root "$PROJECT" \
--proposal-writer project-editor
--capability-mode read
```
Add `--canonical-applier project-editor` only when that MCP integration should expose the
hash-bound `docforge_apply_changeset` tool.
Preview a read-only Codex fragment and diagnose an installed binding:
```bash
.venv/bin/docforge configure codex --project "$PROJECT"
.venv/bin/docforge doctor --client codex --project "$PROJECT"
```
Pass `--output /absolute/path/docforge.toml` only when creating a standalone fragment. DocForge
never replaces or merges an existing different client file.
For an unconfigured codebase, begin with a read-only language and documentation assessment:
```bash
.venv/bin/docforge --project-root /absolute/path/MyProject onboard
```
Add `--scaffold`, a stable project ID, and a title to create, index, and render a generic starter
manual. Source files are reported separately and require a validated language frontend before
DocForge describes them as a source graph.
Add a descriptor-authorized proposal writer only when the client should create proposals. Add a
matching `--canonical-applier` only when that integration should expose exact-hash application.
See the [quickstart](docs/NEW_PROJECT_QUICKSTART.md) for the reference-adapter and generated-client
paths.
## Documentation
- [User manual](docs/USER_MANUAL.md) — features, setup, visualization, CLI, MCP, apply, adapters,
and troubleshooting.
- [Core contract](docs/CONTRACT.md) — invariants and security boundary.
- [Milestone 0 compatibility](docs/COMPATIBILITY.md) — preserved package, CLI, MCP, adapter,
schema, changeset, rendering, and no-AST guarantees.
- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) — validation evidence, cold and warm
performance, memory, rendering and response sizes, bottlenecks, and missing coverage.
- [Milestone 0 closeout](docs/MILESTONE_0_CLOSEOUT.md) — lineage, migration, security scan,
repository state, and fresh-clone proof.
- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) — warm operation latency, structured work,
status, retrieval, and memory measurements.
- [Milestone 1 closeout](docs/MILESTONE_1_CLOSEOUT.md) — fast-core contracts, adversarial
validation, compatibility boundaries, and exact candidate evidence.
- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) — task context, generation diff, client
configuration, doctor, response-size, counter, and memory measurements.
- [Milestone 2 closeout](docs/MILESTONE_2_CLOSEOUT.md) — implemented contracts, adversarial
validation, exclusions, and exact candidate evidence.
- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) — manual, fragment, worker, portable graph,
status, equivalence, response-size, and memory measurements.
- [Milestone 3 closeout](docs/MILESTONE_3_CLOSEOUT.md) — independent projection contracts,
adversarial validation, compatibility boundaries, and exact candidate evidence.
- [MCP contract](docs/MCP_CONTRACT.md) — exact tool and process boundary.
- [Viewer manager](docs/VIEWER_MANAGER.md) — native service setup and lifecycle.
- [Adapter decision](docs/APPLICATION_DECISION.md) — why custom adapters own canonical
serialization.
- [Incremental adapter indexing](docs/INCREMENTAL_INDEXING.md) — source-scoped extraction,
invalidation, equivalence, relationship changes, and the lazy Logic boundary.
- [Project onboarding](docs/PROJECT_ONBOARDING.md) — repository assessment, safe manual
scaffolding, language frontends, source/manual integration, proof, and MCP activation.
- [Language adapter authoring](docs/ADAPTER_AUTHORING_GUIDE.md) — implementation sequence,
stable identities, overlap ownership, normalization, incremental equivalence, troubleshooting,
and the complete adapter proof matrix.
### Learn and operate
- [New-project quickstart](docs/NEW_PROJECT_QUICKSTART.md)
- [Core concepts and authority](docs/CORE_CONCEPTS_AND_AUTHORITY.md)
- [Project descriptor](docs/PROJECT_DESCRIPTOR.md)
- [Policy precedence](docs/POLICY_PRECEDENCE.md)
- [Reference adapters](docs/REFERENCE_ADAPTERS.md)
- [Agent integration](docs/AGENT_INTEGRATION.md)
- [User manual](docs/USER_MANUAL.md)
- [Generated command reference](docs/COMMAND_REFERENCE.md)
- [Project onboarding](docs/PROJECT_ONBOARDING.md)
- [Rendering and visualization](docs/RENDERING_AND_VISUALIZATION.md)
- [Recovery and performance](docs/RECOVERY_AND_PERFORMANCE.md)
- [Security](docs/SECURITY.md)
### Contracts and compatibility
- [Core contract](docs/CONTRACT.md)
- [MCP contract](docs/MCP_CONTRACT.md)
- [Compatibility contract](docs/COMPATIBILITY.md)
- [Adapter authoring guide](docs/ADAPTER_AUTHORING_GUIDE.md)
- [Incremental indexing](docs/INCREMENTAL_INDEXING.md)
- [Adapter application decision](docs/APPLICATION_DECISION.md)
- [Legacy and no-AST operation](docs/LEGACY_AND_NO_AST.md)
- [Migrating from version 1](docs/MIGRATING_FROM_V1.md)
- [Viewer manager](docs/VIEWER_MANAGER.md)
### Milestone evidence
- [Milestone 4 baseline](docs/MILESTONE_4_BASELINE.md)
- [Milestone 4 closeout](docs/MILESTONE_4_CLOSEOUT.md)
- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) and [closeout](docs/MILESTONE_3_CLOSEOUT.md)
- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) and [closeout](docs/MILESTONE_2_CLOSEOUT.md)
- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) and [closeout](docs/MILESTONE_1_CLOSEOUT.md)
- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) and [closeout](docs/MILESTONE_0_CLOSEOUT.md)
Historical milestone records preserve the facts and dependency observations of their frozen
candidates. Use the current guides and contracts for present behavior.
## Development
Run the complete repository-native gate:
Run the repository-native gate:
```bash
make gate
```
Focused entry points are available as `make contract`, `make test`, `make type`,
`make benchmark-smoke`, `make benchmark`, `make benchmark-m1-smoke`, and
`make benchmark-m1`. Milestone 2 adds `make benchmark-m2-smoke` and `make benchmark-m2`.
Milestone 3 adds `make accessibility`, `make benchmark-m3-smoke`, `make benchmark-m3`, and
`make benchmark-m3-full`.
Milestone 4 maintenance entry points include:
The committed 1,000-node baseline and its measurement method are under `benchmarks/`.
```bash
make adoption-m4
make benchmark-m4-smoke
make benchmark-m4
make benchmark-m4-full
make command-reference-check
make docs-check
```
Pass `--diagnostics` to `docforge` or `docforge-mcp` to attach bounded request-local stage timings
and compiler-work counters. Diagnostics are disabled by default and are dropped before primary MCP
results when the configured output budget is tight.
`adoption-m4` builds and exercises a fresh base wheel without Tree-sitter packages and proves the
Python reference plus a real isolated read-only MCP retrieval. The optional-language integrations
have their own focused tests and extras. `benchmark-m4` runs the maintained full adapter workload;
the smoke target is for routine gate coverage, not final performance evidence.
See [AGENTS.md](AGENTS.md) before changing core boundaries.
Pass `--diagnostics` to `docforge` or `docforge-mcp` for bounded request-local timings and compiler
work counters. Diagnostics are disabled by default and do not displace a primary result when the
configured output budget is tight.
Read [AGENTS.md](AGENTS.md) before changing core boundaries.

View file

@ -1,5 +1,50 @@
# Completed slices
## DocForge2 Milestone 4 adapter SDK and product documentation
### Changed
- Added stable `docforge.adapter_sdk` authoring imports and a conformance helper for deterministic
complete and exact complete/incremental primary graph plus Logic equivalence.
- Added bounded Python, JavaScript, TypeScript, and C++ reference integrations with explicit
unsupported-fact reports and optional heavy frontends.
- Added a closed `.docforge/reference-adapter.toml` contract and fixed read-only reference MCP
server.
- Added immutable project-bound launcher declarations and deterministic Codex, Claude, and OpenClaw
configuration generation for custom adapters.
- Generated CLI and MCP references from live implementation metadata and added race-safe
publication plus repository drift checks.
- Added strict documentation-graph and fresh-wheel adoption gates and dedicated onboarding,
authority, descriptor, policy, adapter, legacy/no-AST, rendering, agent-integration, security,
recovery/performance, and v1 migration guides.
### Verification
- The executable candidate passed 347 tests and 402 subtests, 142 contract tests and 268 subtests,
three accessibility flows, lint, strict types, compilation, dependencies, builds, adoption, and
every maintained smoke benchmark.
- The offline base wheel contained no Tree-sitter distribution and ran a real Python reference
build/check and isolated 21-tool MCP retrieval flow. Missing C++ support failed with the exact
`docforge[cpp]` remediation.
- The clean 334-source benchmark produced 1,002 nodes and exact full/incremental graph and Logic
evidence, warm zero Python parsing/extraction, and exact corrupt-cache and corrupt-index
recovery below every latency, memory, and response gate.
- Gitleaks found no findings in the Milestone 4 commit range or candidate tree.
### Limits
- The reference integrations publish syntax and local static relationships. They do not claim
compiler-resolved calls, types, inheritance, macros, runtime behavior, or semantic ownership.
- C++ compilation-database commands are inert and never executed.
- C++ manifest include discovery uses Tree-sitter and makes no zero-warm-parser claim.
- No WorldForge, ScrapeStation, legacy repository, production binding, language distribution,
self-hosting, tag, or release was changed.
### Next gate
Milestone 5 remains directional. Create a new active-slice contract before stabilization,
versioning, tagging, or publication.
## DocForge2 Milestone 3 independent projections
### Changed

View file

@ -42,6 +42,14 @@ make benchmark-m3
make benchmark-m3-full
```
Run the Milestone 4 adapter SDK smoke, maintained full, and fresh-wheel adoption gates:
```bash
make benchmark-m4-smoke
make benchmark-m4-full
make adoption-m4
```
The benchmark creates canonical sources, derived state, changesets, rendered output, and caches
only in a disposable temporary directory. It does not read another project, self-host DocForge, or
mutate repository content.
@ -71,6 +79,13 @@ sweeps, add/change/delete/reorder equivalence, portable publication, receipt-onl
memory, detached worker peak memory, and response size. Its interpretation is in
[`docs/MILESTONE_3_BASELINE.md`](../docs/MILESTONE_3_BASELINE.md).
`milestone4-2026-07-29.json` is the clean-tree adapter SDK and reference-Python baseline captured
from commit `95271dcf2e48045b9d3aed9b9ea09c7fc155692c`. It measures cold and warm
incremental builds, complete/incremental graph-plus-Logic equivalence, corrupt extraction-cache
recovery, corrupt-index recovery, parser and extraction counters, traced and process memory, and
bounded response size. Its interpretation is in
[`docs/MILESTONE_4_BASELINE.md`](../docs/MILESTONE_4_BASELINE.md).
The generic fixtures expose whole-source and projection scaling. They do not replace incremental
adapter equivalence tests. Milestone 3's portable fixture contains 1,000 Nodes/Flow/Web nodes and
999 edges; portable version 1 deliberately excludes Logic.
@ -89,4 +104,5 @@ backlinks, both traversal directions, paged context, render receipt states, and
freshness. The reported p95 uses the nearest-rank method; with ten samples it is the maximum.
`process_peak_rss_kib` is the cumulative main-process `RUSAGE_SELF` high-water mark and excludes the
detached viewer worker. It is diagnostic and not operation-local. Milestone 3 memory gates use
per-operation `tracemalloc` peaks and detached worker receipt peaks instead.
per-operation `tracemalloc` peaks and detached worker receipt peaks instead. Milestone 4 uses
per-operation `tracemalloc` peaks and a cumulative benchmark-process high-water gate.

View file

@ -0,0 +1,205 @@
{
"benchmark": "docforge2_milestone4",
"environment": {
"implementation": "CPython",
"machine": "x86_64",
"platform": "Linux-7.1.3-200.nobara.fc44.x86_64-x86_64-with-glibc2.43",
"python": "3.14.6"
},
"evidence": {
"cache_recovery_exact": true,
"complete_assembly_hash": "2888182ab765fbffe3ba873c1613345640e8e6d89be74ddfca7452c0a5056345",
"edge_count": 1001,
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
"full_incremental_graph_and_logic_exact": true,
"index_recovery_exact": true,
"logic_edge_count": 2338,
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
"logic_node_count": 2338,
"logic_projection_count": 334,
"node_count": 1002,
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
"source_count": 334,
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
"warm_zero_ast_parse": true,
"warm_zero_source_extraction": true
},
"evidence_sha256": "4a8db461a311b5df4abd5aa00063e9a347d8b9dba19b30e35684f561d5271549",
"fixture": {
"expected_node_count": 1002,
"kind": "synthetic_python_reference_adapter",
"nodes_per_source": 3,
"source_count": 334
},
"memory": {
"process_peak_bytes": 78798848,
"process_peak_limit_bytes": 536870912
},
"method": {
"ast_instrumentation": "temporary counter around stdlib ast.parse",
"clock": "time.perf_counter_ns",
"extraction_instrumentation": "reference adapter extract_source entry counter",
"full_mode_source_requirement": 334,
"in_process_peak_memory": "tracemalloc per measured invocation",
"process_peak_memory": "resource.getrusage(RUSAGE_SELF).ru_maxrss",
"response_size": "UTF-8 bytes of compact sorted JSON",
"samples": 3,
"threshold_basis": "Regression tripwires leave several times the Milestone 3 1,000-node allowances for AST, Logic, extraction-cache, and SQLite work."
},
"mode": "full",
"operations": {
"cold_incremental_build": {
"max_ms": 1599.76,
"maximum_response_bytes": 14580,
"maximum_traced_peak_bytes": 68573540,
"median_ms": 1599.76,
"min_ms": 1599.76,
"p95_limit_ms": 30000.0,
"p95_ms": 1599.76,
"response_limit_bytes": 262144,
"samples": 1,
"stable_result": {
"ast_parse_calls": 668,
"cache_hits": 0,
"deleted_sources": 0,
"edge_count": 1001,
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
"extraction_calls": 334,
"invalidated_sources": 334,
"logic_edge_count": 2338,
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
"logic_node_count": 2338,
"logic_projection_count": 334,
"node_count": 1002,
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
"reparsed_sources": 334,
"revision": "30ef23aa4106",
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
"status": "ok",
"total_sources": 334
},
"traced_peak_limit_bytes": 268435456
},
"corrupt_extraction_cache_recovery": {
"max_ms": 1695.406,
"maximum_response_bytes": 14580,
"maximum_traced_peak_bytes": 69776923,
"median_ms": 1695.406,
"min_ms": 1695.406,
"p95_limit_ms": 30000.0,
"p95_ms": 1695.406,
"response_limit_bytes": 262144,
"samples": 1,
"stable_result": {
"ast_parse_calls": 668,
"cache_hits": 0,
"deleted_sources": 0,
"edge_count": 1001,
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
"extraction_calls": 334,
"invalidated_sources": 334,
"logic_edge_count": 2338,
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
"logic_node_count": 2338,
"logic_projection_count": 334,
"node_count": 1002,
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
"reparsed_sources": 334,
"revision": "30ef23aa4106",
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
"status": "ok",
"total_sources": 334
},
"traced_peak_limit_bytes": 268435456
},
"corrupt_index_recovery": {
"max_ms": 1707.879,
"maximum_response_bytes": 14779,
"maximum_traced_peak_bytes": 68561924,
"median_ms": 1707.879,
"min_ms": 1707.879,
"p95_limit_ms": 30000.0,
"p95_ms": 1707.879,
"response_limit_bytes": 262144,
"samples": 1,
"stable_result": {
"action": "rebuilt",
"ast_parse_calls": 0,
"cache_hits": 334,
"edge_count": 1001,
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
"extraction_calls": 0,
"initial_error_code": "invalid_index",
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
"logic_projection_count": 334,
"node_count": 1002,
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
"reparsed_sources": 0,
"revision": "30ef23aa4106",
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
"status": "ok"
},
"traced_peak_limit_bytes": 268435456
},
"full_incremental_equivalence": {
"max_ms": 1897.919,
"maximum_response_bytes": 230,
"maximum_traced_peak_bytes": 64305919,
"median_ms": 1897.919,
"min_ms": 1897.919,
"p95_limit_ms": 30000.0,
"p95_ms": 1897.919,
"response_limit_bytes": 262144,
"samples": 1,
"stable_result": {
"ast_parse_calls": 1336,
"edge_count": 1001,
"extraction_calls": 668,
"logic_projection_count": 334,
"node_count": 1002,
"project_id": "milestone4-python-benchmark",
"revision": "30ef23aa4106",
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
"status": "ok"
},
"traced_peak_limit_bytes": 268435456
},
"warm_incremental_build": {
"max_ms": 1265.387,
"maximum_response_bytes": 14578,
"maximum_traced_peak_bytes": 69997166,
"median_ms": 1244.427,
"min_ms": 1239.941,
"p95_limit_ms": 20000.0,
"p95_ms": 1265.387,
"response_limit_bytes": 262144,
"samples": 3,
"stable_result": {
"ast_parse_calls": 0,
"cache_hits": 334,
"deleted_sources": 0,
"edge_count": 1001,
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
"extraction_calls": 0,
"invalidated_sources": 0,
"logic_edge_count": 2338,
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
"logic_node_count": 2338,
"logic_projection_count": 334,
"node_count": 1002,
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
"reparsed_sources": 0,
"revision": "30ef23aa4106",
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
"status": "ok",
"total_sources": 334
},
"traced_peak_limit_bytes": 268435456
}
},
"schema_version": 1,
"source": {
"dirty": false,
"revision": "95271dcf2e48045b9d3aed9b9ea09c7fc155692c"
}
}

View file

@ -8,6 +8,30 @@ Use this guide after the repository assessment in
[Project Onboarding](PROJECT_ONBOARDING.md). The onboarding checklist decides whether an adapter
is needed. This guide defines how to build and prove one.
## Public authoring surface
Adapter authors should import typed contracts, validation helpers, graph models, and the
conformance helper from `docforge.adapter_sdk`:
```python
from docforge.adapter_sdk import (
AdapterAssembly,
AdapterImplementation,
AdapterManifest,
AdapterProject,
AdapterProjectSettings,
AdapterProjection,
AdapterSource,
AdapterSourceProjection,
LogicProjection,
verify_adapter_conformance,
)
```
Language-specific modules under `docforge.adapters` are repository reference implementations, not
the general authoring namespace. Their exact, deliberately narrow behavior is documented in
[Reference Adapters](REFERENCE_ADAPTERS.md).
## Required outcome
A production adapter must provide one reproducible public graph from authoritative project
@ -51,11 +75,39 @@ class MyAdapter:
manifest: AdapterManifest,
contributions: tuple[AdapterSourceProjection, ...],
) -> AdapterAssembly: ...
def load_complete_assembly(self) -> AdapterAssembly: ...
def load_projection(self) -> AdapterProjection: ...
```
`assemble_projection()` is optional only when extraction units already have disjoint ownership.
`load_projection()` is always required. It is the clean rebuild and equivalence oracle.
`load_projection()` is always required. It is the clean primary-graph compatibility oracle.
`load_complete_assembly()` supplies the cache-independent complete graph-plus-Logic oracle. It is
required when incremental contributions publish Logic; without it, DocForge cannot prove Logic
parity.
## What the conformance helper proves
Run the public helper against a fresh confined cache root:
```python
report = verify_adapter_conformance(
MyAdapter(project_root),
cache_root=project_root / ".docforge" / "cache" / "adapter-conformance",
)
```
`verify_adapter_conformance()` proves:
- two repeated complete assemblies are exactly deterministic, including Logic;
- when `load_complete_assembly()` exists, its primary graph exactly matches
`load_projection()`; and
- when the incremental methods exist, the assembled incremental graph and Logic exactly match the
independent complete oracle.
The report records stable identity, counts, and an assembly hash. This helper does not by itself
prove process confinement, implementation restart behavior, corrupt-cache recovery, warm zero
parsing, retrieval, or every case in the proof matrix below. Keep those as separate focused and
integration tests.
## Step 1: define authority before parsing
@ -324,6 +376,25 @@ The manifest remains a current source snapshot, not a Git-index snapshot. A Git-
omit a deleted source whether its deletion is unstaged or staged. Staging is never a required
DocForge synchronization step.
### Current aggregate bounds
DocForge limits one extraction-cache generation to 10,000 source contributions and 64,000,000
encoded bytes. Malformed, incompatible, missing, oversized, or unsafe cache data is treated as a
cache miss.
The assembled graph is bounded by the effective project `Limits.max_nodes`. When adapter settings
do not override limits, DocForge selects at least 10,000 nodes and raises that bound to the
manifest's `estimated_nodes` when larger. Aggregate assembly ceilings are:
- nodes: `max_nodes`;
- relationships: `max_nodes * 32`;
- Logic nodes across all functions: `max_nodes * 32`; and
- Logic edges across all functions: `max_nodes * 64`.
Adapters should set realistic limits and fail before retaining unbounded frontend evidence. The
cache and assembly bounds do not replace each adapter's own bounded source, command, parser, or
per-function limits.
## Step 9: keep the complete path independent
The full rebuild must not read the incremental extraction cache. Otherwise equivalence compares
@ -470,7 +541,8 @@ For every such change:
- [ ] Two independent complete builds match exactly.
- [ ] Incremental extraction uses authoritative dependencies.
- [ ] The complete oracle is independent of the cache.
- [ ] Complete and incremental public projections match exactly.
- [ ] Complete and incremental public graph and Logic projections match exactly.
- [ ] `verify_adapter_conformance()` passes, and the separate proof-matrix cases also pass.
- [ ] Corrupt, missing, and interrupted cache cases fail safely.
- [ ] Session composition and family isolation are proven.
- [ ] Viewer, query, context, and Logic retrieval are proven.

105
docs/AGENT_INTEGRATION.md Normal file
View file

@ -0,0 +1,105 @@
# Agent integration
DocForge generates deterministic, project-bound MCP client fragments for Codex, Claude, and
OpenClaw. Generic projects and custom adapters use different configuration APIs, but both produce
fixed standard-input/output bindings with empty generated environments and explicit capability
policy.
## Fixed reference binding
Configure one of the in-repository adapters with
`.docforge/reference-adapter.toml` as described in
[Reference Adapters](REFERENCE_ADAPTERS.md), then start:
```bash
python -I -m docforge.reference_mcp \
--project-root /absolute/path/to/project \
--capability-mode read
```
`docforge.reference_mcp` is a fixed trusted module in the installed DocForge distribution. It
selects a provider only from the validated reference configuration and reports binding metadata
containing:
- `server_module = "docforge.reference_mcp"`;
- `adapter_mode = "reference"`;
- the selected `reference_language`; and
- the exact `reference_config_hash`.
The reference binding registers exactly the 21 read tools listed in
[MCP Boundary](MCP_CONTRACT.md#read-tools). It has no proposal or canonical-application surface.
## Generate a reference client fragment
The public launcher and generator APIs are:
```python
from pathlib import Path
from docforge.adapter_launcher import AdapterLauncherV1
from docforge.client_config import generate_adapter_client_configuration
from docforge.reference_mcp import REFERENCE_MCP_MODULE, create_reference_project
root = Path("/absolute/path/to/project")
project = create_reference_project(root)
launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE)
plan = generate_adapter_client_configuration(
project,
launcher,
"codex", # "codex", "claude", or "openclaw"
capability_mode="read",
)
print(plan["artifact"]["content"])
```
Pass `output=Path(...)` only when the caller has selected an exact destination. Publication is an
atomic create-or-exact-match operation; it does not merge or replace different existing content.
The result includes the launcher, source-availability, project, policy, artifact, and configuration
hashes needed to inspect the binding before use.
The fixed reference module accepts read mode only. Do not request proposal or application mode for
it.
## Custom project-owned adapter launchers
Generic `docforge configure` intentionally refuses a custom adapter project. Construct the
project-owned `ProjectService`, then use `AdapterLauncherV1` and
`generate_adapter_client_configuration()` as the custom-adapter route.
`AdapterLauncherV1` schema version 1 binds:
- one project ID, canonical absolute project root, adapter identity, and descriptor hash;
- `entry_point = "python-module"`; and
- one installed project-owned top-level Python module.
The only trusted dotted-module exception is the fixed `docforge.reference_mcp` binding. A
project-owned module is resolved through isolated Python without importing or executing it during
the probe, and it must resolve to one canonical regular `.py` file inside the project root.
The launcher contract has no arbitrary command, command arguments, shell string, working
directory, environment, callable selector, discovery rule, or module-reload mechanism. Generated
bindings invoke the current Python executable as `python -I -m <module>` with only DocForge's
validated project, capability, render-policy, authority, and no-AST options. Source identity and
the project binding are revalidated before publication; drift fails closed.
Proposal and application modes are available only to a custom module that implements those fixed
server arguments and only when the project descriptor declares the matching writer and canonical
applier authority. Generating a mode does not manufacture that authority.
## Session workflow
After registering the generated fragment in the selected client:
1. Start a new MCP process or client session.
2. Call `docforge_bootstrap`.
3. Verify project ID, root fingerprint, adapter identity, revision, source hash, binding metadata,
effective policy, and registered tools.
4. Retrieve exact or bounded project evidence through the fixed read tools.
5. Restart the process if `adapter_restart_required` reports an implementation or configuration
change.
Document text returned by DocForge is untrusted project content. It never overrides client, user,
or project authority. See [MCP Boundary](MCP_CONTRACT.md) for synchronization, pagination,
retrieval, proposal, and application rules, and
[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) before selecting `no_ast=True`.

79
docs/COMMAND_REFERENCE.md Normal file
View file

@ -0,0 +1,79 @@
# DocForge command reference
> Generated from the live CLI parser and MCP registrations. Do not edit this file by hand.
Global CLI options are documented in `docforge --help` and are not repeated in each command row. The MCP table is the complete generic application-enabled surface; the fixed reference-adapter server exposes only its read rows.
## CLI commands
| Command | Invocation |
|---|---|
| `apply` | `docforge apply [-h] --changeset-hash CHANGESET_HASH --applier APPLIER changeset_id` |
| `backlinks` | `docforge backlinks [-h] [--relation RELATION] [--limit LIMIT] node_id` |
| `build` | `docforge build [-h]` |
| `check` | `docforge check [-h]` |
| `configure` | `docforge configure [-h] --project PROJECT [--name NAME] [--capability-mode {read,proposal,application}] [--proposal-writer PROPOSAL_WRITER] [--canonical-applier CANONICAL_APPLIER] [--no-ast] [--manual-render-policy {auto,explicit,disabled}] [--portable-graph-policy {explicit,disabled}] [--live-viewer-policy {on-demand,disabled}] [--startup-timeout STARTUP_TIMEOUT] [--tool-timeout TOOL_TIMEOUT] [--output OUTPUT] {codex,claude,openclaw}` |
| `context` | `docforge context [-h] [--budget BUDGET] [--limit LIMIT] [--cursor CURSOR] profile` |
| `dependencies` | `docforge dependencies [-h] [--depth DEPTH] [--limit LIMIT] node_id` |
| `doctor` | `docforge doctor [-h] --client {codex,claude,openclaw} [--project PROJECT] [--config CONFIG] [--server-name SERVER_NAME]` |
| `filter` | `docforge filter [-h] [--family FAMILY] [--authority AUTHORITY] [--status STATUS] [--tag TAG] [--limit LIMIT]` |
| `generation-diff` | `docforge generation-diff [-h] [--limit LIMIT] [--cursor CURSOR]` |
| `graph-plan` | `docforge graph-plan [-h] view_id` |
| `graph-render` | `docforge graph-render [-h] view_id` |
| `graph-render-status` | `docforge graph-render-status [-h] [view_id]` |
| `impact` | `docforge impact [-h] [--depth DEPTH] [--limit LIMIT] node_id` |
| `info` | `docforge info [-h]` |
| `onboard` | `docforge onboard [-h] [--language LANGUAGE] [--scaffold] [--project-id PROJECT_ID] [--title TITLE] [--content-root CONTENT_ROOT]` |
| `preview` | `docforge preview [-h] changeset_id view_id` |
| `reindex` | `docforge reindex [-h]` |
| `render` | `docforge render [-h] view_id` |
| `render-status` | `docforge render-status [-h] [--deep] [view_id]` |
| `search` | `docforge search [-h] [--limit LIMIT] query` |
| `show` | `docforge show [-h] node_id` |
| `sync` | `docforge sync [-h]` |
| `validate` | `docforge validate [-h]` |
| `validate-index` | `docforge validate-index [-h]` |
| `visualization-status` | `docforge visualization-status [-h]` |
| `visualization-stop` | `docforge visualization-stop [-h]` |
| `visualize` | `docforge visualize [-h] [--node NODE \| --query QUERY] [--depth DEPTH] [--no-open]` |
## MCP tools
| Surface | Tool | Required arguments | Optional arguments | Input schema SHA-256 | Description |
|---|---|---|---|---|---|
| read | `docforge_backlinks` | `node_id` | `limit`, `relation` | `bf9a5936e2d1ee70897dc0795ec9d716be63c9447e6fdcd3890d5dd92f45f690` | Return bounded incoming relationships for one exact stable node. |
| read | `docforge_bootstrap` | — | — | `af6c17a59a7713cfb4497f0f66b05efc091abbfe109573bb86d673e39b9b7413` | Synchronize and report the complete fixed project binding and workflow. |
| read | `docforge_dependencies` | `node_id` | `depth`, `limit` | `1283a8a030cf9369493a13e2aa87317d371470904a397910bc0b52997b3ec3b8` | Traverse declared depends_on relationships within the configured depth limit. |
| read | `docforge_filter_nodes` | — | `authority`, `family`, `limit`, `status`, `tag` | `2a51c4d8f4d76d89a2e7f90b3b06d14c7e16303b9c81e62af6747286b58cf68b` | Filter current nodes deterministically by validated metadata. |
| read | `docforge_get_context` | `profile` | `budget`, `cursor`, `limit` | `2f164828c2891a88d0ea7173790a72cc6f3da1279f14da2ca495e8989880b298` | Compile bounded cited context from one configured profile with explicit omissions. |
| read | `docforge_get_contract` | — | — | `19eb6785523ed9db645d17f61df2b8b584ac10ad308199e9a96e6e176a87169e` | Report canonical and derived boundaries plus allowed and excluded operations. |
| read | `docforge_get_generation_diff` | — | `cursor`, `limit` | `0a2fccfea4bf5d78481d40b49e7c812ded9fbe9c93cb79296ede78f22b259fcf` | Return the latest bounded primary-graph generation transition. |
| read | `docforge_get_logic` | `owner_node_id` | — | `56909ee4d72143b470528ea2b3544325e1e1920dbcc758f3caee4a4924f904a1` | Return the lazy control-flow projection owned by one function or method. |
| read | `docforge_get_node` | `node_id` | — | `52408b0c3a297e7b5cf96b293fd6d4625f05c4b090c02bfe806d25ce6c2d1c68` | Return one exact stable node from the current validated project index. |
| read | `docforge_get_task_context` | `task`, `task_kind` | `budget`, `cursor`, `focus_node_id`, `limit` | `0ffbc030bbe5cca2ae24920ec812ac976be9090da4fa2887d87526aba64a21f7` | Return one bounded task-shaped context capsule with explicit evidence gaps. |
| read | `docforge_graph_plan` | `view_id` | — | `7a014ba692f57badca9ca2098b88f82f4749ada6d71f5a3b8bc141abc86645f9` | Plan one declared portable graph without publishing derived output. |
| read | `docforge_graph_render_status` | — | `view_id` | `1d2f3604b0151ba2e228cc6ce1829d7f4244193b53d85df46eb6a3e608c59564` | Report portable-graph publication state without rendering. |
| read | `docforge_impact` | `node_id` | `depth`, `limit` | `1c71db5bb4e1afe180d98dd5000a757b7e3fd1a048b5f68af13d9b7ac56e2ce2` | Traverse bounded incoming relationships and report exact paths. |
| read | `docforge_project_info` | — | — | `a42973543d5e6f4147953d175f6aec43d228cd6bd1a63f3ca4e84a51ad7e8a64` | Report the fixed project identity, revision, source hash, and index health. |
| read | `docforge_render_status` | — | `deep`, `view_id` | `5ea5683bd74e685d4cabb1c960a66eff23a001bd2c68e1af350c6e46e936b9fd` | Report receipt state, or explicitly recompute the side-effect-free render oracle. |
| read | `docforge_search` | `query` | `limit` | `735fff8bd82b20730b0907db8bf61eb0de9fff240ef4fa239ce555779ae49047` | Run bounded lexical search over the current validated project index. |
| read | `docforge_stop_visualization` | — | — | `066a6e34019f4abcd3523f24d9e447775e1479adbea1134341f434a6bb5f8d35` | Explicitly stop this project's persistent read-only graph browser. |
| read | `docforge_sync` | — | — | `a2ac50c03b9b0e08b271216f76920279563ed264bef06d6791695c68dcb3b2a2` | Ensure the disposable project index matches current canonical sources. |
| read | `docforge_validate_project` | — | — | `3192ba97a6d7fff13008c5ed3bfa549d1eeea95332579592d96b8fd326ad06f7` | Validate current canonical sources and graph without writing any project file. |
| read | `docforge_visualization_status` | — | — | `03751f891e0a9318355a6b28be551eec6009c0f96f05ff99536e2f401b76f87f` | Report this project's managed graph browser lifecycle state. |
| read | `docforge_visualize` | — | `depth`, `node_id`, `query` | `cdc9669ba934e591379ea684934d99e2e80ba3f181d6c58867774a8064c32f4b` | Start the fixed read-only graph browser for this configured project. |
| proposal | `docforge_abandon_changeset` | `changeset_id`, `expected_changeset_hash`, `reason` | — | `8813e40ffc12b1ae3d9abe73db8de214085a4a8042323146283fffa8f7476a62` | Mark one proposal abandoned while preserving its audit record. |
| proposal | `docforge_create_changeset` | `changeset_id` | — | `0de93025b9caaa9e33b4b2b3bcfabdde087c944daba48dd77ae3c8f757c52ab8` | Create an empty hash-bound proposal under the configured isolated changeset root. |
| proposal | `docforge_get_changeset` | `changeset_id` | `cursor`, `limit` | `919737a8e732cf95305a2964cddf98f1fdce9ce2cfb52ff659436c244a38495d` | Inspect a stored proposal even when its canonical base has become stale. |
| proposal | `docforge_get_changeset_diff` | `changeset_id` | `cursor`, `limit` | `5cc1d2b035f37e3866766cfa0c3541b144762b764bf455a641ab459df03f8a59` | Return a deterministic structured and textual diff without applying the proposal. |
| proposal | `docforge_list_changesets` | — | `cursor`, `include_history`, `limit`, `status` | `b96519477781cb20acbfc9502a4b7d64a9119e0bd8d71e27290e90677aaf9ced` | List active proposals by default, with optional lifecycle history. |
| proposal | `docforge_preview_changeset` | `changeset_id`, `view_id` | — | `d84e1710a5cac52607037a7e581fcb221c42d3776f510eefe42fb6779fee7fb3` | Render one validated changeset through a declared view into its isolated preview path. |
| proposal | `docforge_propose_node_create` | `changeset_id`, `content`, `expected_changeset_hash`, `metadata`, `node_id`, `rationale`, `relationship_changes`, `target_source` | — | `8953b502b9d8d9c097935bae51fce4e830286eace9df3cb6c37b7e0542be5a4e` | Append one validated node creation without writing its canonical target. |
| proposal | `docforge_propose_node_delete` | `changeset_id`, `expected_changeset_hash`, `expected_content_hash`, `node_id`, `rationale`, `relationship_changes` | — | `8fb6a22e5c1ead3793384652e4b61e75cc23c2725906542533a121401f7faaee` | Append one validated deletion with explicit incident relationship removals. |
| proposal | `docforge_propose_node_move` | `changeset_id`, `expected_changeset_hash`, `expected_content_hash`, `node_id`, `rationale`, `target_source` | — | `32ef6fc4b9795a6a8e30f4f81f63d84d7350e983773f796524382d71cb870a67` | Append one validated same-format node move without moving a canonical file. |
| proposal | `docforge_propose_node_update` | `changeset_id`, `content`, `expected_changeset_hash`, `expected_content_hash`, `metadata`, `node_id`, `rationale`, `relationship_changes` | — | `6a29a73b9e50c9fe22cb6d056bbcdacef30c93c0822c8a9276ef33f9e51e7728` | Append one validated node update without changing canonical content. |
| proposal | `docforge_propose_relationship_update` | `changeset_id`, `expected_changeset_hash`, `expected_content_hash`, `node_id`, `rationale`, `relationship_changes` | — | `8e21fc7ba35289940dd7f12294b14e8553163ab901002dcd4a3998cdb33556c3` | Queue hash-bound relationship changes without rewriting node content. |
| proposal | `docforge_rebase_changeset` | `changeset_id`, `expected_changeset_hash` | — | `505434b2f9f611649221b95d0251db347c44752d1c73e5d69e1abfd916966788` | Safely rebase a proposal when every touched fact remains unchanged. |
| proposal | `docforge_register_changes` | `changeset_id`, `operations` | — | `c5a7d6ac07d3d3b95c13b0c57ad9b69360f2f063b1c8e22dd5f64f6d3a83dc5b` | Atomically register and validate a complete hash-bound proposal. |
| proposal | `docforge_validate_changeset` | `changeset_id` | `cursor`, `limit` | `cb9df5104dbefc3382900ec812ea581a43612d1577619071f1f03674041652c7` | Validate a proposal against its exact canonical base and other active proposals. |
| application | `docforge_apply_changeset` | `changeset_id`, `expected_changeset_hash` | — | `80259f5001f36b14f960c10bd94751d1431077b72e081fb5a5c806b79073b085` | Apply one exact validated changeset and refresh declared derived state. |

View file

@ -31,16 +31,31 @@ model, index, rendering, application, and MCP factory names imported from these
remain supported:
- `docforge.adapter_contract`
- `docforge.adapter_sdk`
- `docforge.adapter_launcher`
- `docforge.application`
- `docforge.client_config`
- `docforge.index`
- `docforge.mcp_server`
- `docforge.models`
- `docforge.policy`
- `docforge.render_contract`
- `docforge.reference_config`
- `docforge.reference_mcp`
Names beginning with an underscore are implementation details. New public names may be added
without breaking this contract.
The repository Python, JavaScript/TypeScript, and C++ adapters are supported reference
implementations. Their documented configuration, evidence limits, and unsupported-fact reports are
compatibility surfaces; their internal parser helpers are not adapter-authoring imports.
Milestone 4 adds three version-1 schemas without changing existing descriptor or result schemas:
- `schemas/adapter-launcher.schema.json`
- `schemas/adapter-client-configuration.schema.json`
- `schemas/reference-adapter.schema.json`
## CLI and MCP surfaces
Existing `docforge` command names and arguments remain supported. Existing `docforge-mcp` tool
@ -233,6 +248,11 @@ Milestone 3 adds `ManualRenderPlanV1`, `GraphViewPlanV1`, projection package and
and projection policy version 2. These are additive submodule and schema contracts. They do not
change the legacy task-context, adapter, changeset, or effective-policy contracts described above.
Milestone 4 adds `docforge.adapter_sdk`, the fixed reference configuration and read-only MCP
binding, immutable adapter launchers, and generated adapter client fragments. A legacy adapter with
only `load_projection()` remains first-class and need not adopt incremental extraction, Logic, a
reference configuration, or launcher metadata.
## Safety boundary
DocForge remains bound to one explicit project root. It rejects absolute paths, root escapes, and
@ -245,8 +265,9 @@ publication, or project switching.
Milestone 0 records rather than redesigns these areas:
- Generic warm reads still repeat whole-project discovery, parsing, and validation.
- Tree-sitter and the JavaScript and C++ grammars remain mandatory installation dependencies even
when their runtime modules are unused.
- The base wheel intentionally omits Tree-sitter. JavaScript, TypeScript, and C++ syntax evidence
requires the matching `docforge[javascript]`, `docforge[typescript]`, or `docforge[cpp]` extra.
Python reference evidence uses the standard library and remains available in the base wheel.
- Several version strings and defaults remain duplicated.
- One individually oversized context entry is represented as explicit bounded omission evidence;
callers use targeted retrieval for that node.

View file

@ -29,6 +29,9 @@ commit when Git is available; it cannot change repository state.
- Projection package: `schemas/projection-package.schema.json`, version 1.
- Projection receipt: `schemas/projection-receipt.schema.json`, version 1.
- Independent projection policy: `schemas/projection-policy.schema.json`, version 2.
- Adapter launcher: `schemas/adapter-launcher.schema.json`, version 1.
- Adapter client configuration: `schemas/adapter-client-configuration.schema.json`, version 1.
- Reference adapter configuration: `schemas/reference-adapter.schema.json`, version 1.
- Index schema: version 3, disposable and reproducible.
- Index attestation: schema version 1, disposable and reproducible.
- Core, CLI, and MCP server: version 1.3.0.dev0.
@ -103,6 +106,42 @@ complete projection, opens SQLite, starts MCP, executes the configured command,
builds, renders, starts a viewer, or writes configuration. Unprovable client behavior is a warning,
not an invented success.
## Public adapter SDK and reference binding
`docforge.adapter_sdk` is the stable adapter-authoring import boundary. It exposes the typed
projection, manifest, source contribution, complete assembly, project wrapper, graph model, and
conformance contracts needed by an adapter without requiring authors to import core implementation
modules.
Complete evidence includes the primary graph and function Logic. An incremental adapter that
publishes Logic must implement `load_complete_assembly()` as an independent complete oracle.
`verify_adapter_conformance()` proves repeated complete determinism, equality between the complete
assembly and `load_projection()`, and exact complete/incremental graph-plus-Logic parity. Separate
tests remain responsible for confinement, restart behavior, no-AST behavior, cache recovery, and
retrieval.
Adapter assemblies are bounded before publication. Primary nodes use the descriptor `max_nodes`
limit. Primary edges, Logic nodes, and Logic edges use fixed deterministic multipliers over that
limit. Version-1 extraction caches are regular-file-only, bounded to 10,000 sources and
64,000,000 bytes, and are treated as misses when corrupt, oversized, foreign, or incompatible.
`.docforge/reference-adapter.toml` is a closed version-1 selection among `python`, `javascript`,
`typescript`, and `cpp`. It declares one project identity and explicit non-overlapping source
roots. C++ additionally requires a confined `compile_commands.json`. It cannot declare a command,
module, environment, writer, applier, or remote endpoint.
`python -m docforge.reference_mcp --project-root ROOT` constructs only the selected fixed
repository reference adapter and exposes the read surface. It never registers proposal or
application tools.
`AdapterLauncherV1` is an immutable project-bound Python-module declaration. It accepts no
arbitrary command, arguments, working directory, environment, discovery, callable selector, or
module reload. Custom launchers resolve one installed top-level module through isolated Python and
require its origin inside the project root. The fixed `docforge.reference_mcp` module is the only
trusted dotted exception. `generate_adapter_client_configuration()` binds generated Codex,
Claude, and OpenClaw fragments to that launcher, current source availability, effective policy,
descriptor, interpreter, and exact artifact bytes.
## Isolated proposal model
Create, update, move, and delete are ordered node operations inside an isolated changeset. Every

View file

@ -0,0 +1,165 @@
# Core concepts and authority
DocForge is a project-bound knowledge compiler. It turns explicit canonical project facts into
validated graphs and bounded derived views without transferring authority to the index, an agent,
or a renderer.
## One project, one explicit root
Every operation is bound to one canonical real project directory. Paths in descriptors and adapter
configuration are project-relative and confined beneath that root. A CLI or MCP process does not
discover or switch projects after startup.
The project root determines:
- which descriptor and canonical sources may be read;
- where derived cache and changeset roots may exist;
- which project identity, revision, source hash, and generation appear in results;
- which writer, applier, rendering, and viewer policies can be selected.
Generated client fragments preserve that binding. They are machine-local configuration
projections, not portable project authority.
## Canonical facts and derived evidence
Canonical inputs own the facts:
- generic Markdown or TOML node sources;
- authority files named by a generic descriptor;
- an adapter's declared canonical sources and implementation boundary;
- `.docforge/project.toml` for a generic project;
- `.docforge/reference-adapter.toml` for a fixed reference integration.
Everything DocForge builds from those inputs is derived:
- SQLite indexes, attestations, generation receipts, and generation diffs;
- incremental extraction caches;
- task-context capsules and query responses;
- changeset previews;
- render plans, immutable packages, fragments, artifacts, and receipts;
- portable graph publications and live-viewer processes;
- generated Codex, Claude, and OpenClaw fragments.
Derived state may be discarded and rebuilt. A derived artifact can prove what it was bound to, but
it cannot override current canonical content.
## Nodes, relationships, and Logic
The primary graph contains nodes and directed relationships.
A node has a stable project-wide ID, title, family, authority, status, tags, summary, content,
source identity, and content hash. A generic Markdown file contains one node beneath a TOML
metadata block; a generic TOML source may contain multiple nodes. Adapter nodes use the same public
graph contract.
A relationship is an exact `(source, relation, target)` triple. The project descriptor defines the
allowed relation names. DocForge gives `depends_on` special acyclic validation, but it does not
invent domain meaning for a project's other relation names. Retrieval recognizes only a versioned
alias set for task planning and reports unknown allowed relations as `unclassified`.
Logic is deliberately separate. It is a lazy function-scoped control-flow projection owned by one
primary node. Decisions, actions, loops, convergence points, returns, and exceptions connect
through explicit branch edges. Logic does not add statement-level nodes to ordinary Nodes, Flow,
Web, search, or generation-diff results. Python, JavaScript, TypeScript, and C++ integrations may
publish Logic when their adapter contract supports it.
## Authority, status, family, and tags
These fields answer different questions:
- `authority` describes the role of the content. The generic vocabulary is `authoritative`,
`approved_plan`, `derived`, `proposal`, and `historical`.
- `status` is project-defined lifecycle state such as `current`, `active`, or `verified`.
- `family` is a project-defined content grouping used for filtering, profiles, rendering, and
writer permissions.
- `tags` are exact project labels for retrieval and presentation.
An `authoritative` node can still become stale; authority is not a freshness claim. A `derived`
node is still canonical if it is stored in a declared canonical source; the label describes its
role, not whether DocForge may silently regenerate it. Status and authority never grant an MCP
writer permission.
## Validation, synchronization, and generations
Validation loads the complete canonical graph and rejects unsafe paths, invalid source formats,
duplicate IDs, unresolved relationships, prohibited cycles, violated project limits, and
adapter-specific contract failures.
The disposable index is published atomically only after the complete graph and SQLite integrity
checks pass. Its attestation binds the whole index file. A successful replacement is the derived
publication commit point; later receipt-writing trouble is reported as degraded evidence rather
than as a false claim that the replacement failed.
MCP operations automatically synchronize derived state under a project lock before normal work.
A graph generation identifies one validated indexed snapshot. Generation-pinned retrieval and
rendering do not reopen mutable sources behind an older snapshot.
The latest generation diff is one bounded primary-graph transition, not a history database. It
contains no Logic details or source text.
## Complete and incremental adapters
`load_projection()` is the compatibility baseline and clean graph oracle. An incremental adapter
adds:
- `load_manifest()` for cheap project identity, source inventory, fingerprints, and dependencies;
- `extract_source()` for one cacheable source contribution;
- optionally `assemble_projection()` to normalize overlapping contributions.
When incremental contributions contain Logic, `load_complete_assembly()` supplies a
cache-independent complete graph-plus-Logic oracle. Warm cache behavior is an optimization, never a
different authority path. Corrupt or incompatible extraction caches are treated as misses, and a
clean complete build remains the equivalence and recovery boundary.
The public types, validators, and conformance helper are exported from `docforge.adapter_sdk`. See
the [Adapter authoring guide](ADAPTER_AUTHORING_GUIDE.md) and [Incremental
indexing](INCREMENTAL_INDEXING.md).
## Proposals are not canonical changes
A changeset is an isolated, ordered proposal over an exact canonical base. Each operation names
preconditions, and the final changeset has a content-derived hash. Validation projects the complete
resulting graph before application.
Canonical application requires:
1. a writer declared in the descriptor;
2. a process started with the matching proposal and application authority;
3. one explicit final changeset hash;
4. unchanged source, relationship, permission, and graph preconditions.
Application does not perform Git mutation, build, deployment, or publication. Until exact-hash
application succeeds, canonical project files remain unchanged.
## Three independent output projections
Manual rendering, portable graph publication, and the live viewer are separate:
- a manual is a declared derived HTML view over selected nodes;
- a portable graph is a content-addressed static Nodes, Flow, or Web artifact;
- the live viewer is a managed loopback process pinned to one validated index generation and can
request lazy Logic.
Their policies compose independently. Disabling one does not transfer its authority to another.
None is canonical documentation or a retrieval authority. See [Rendering and
visualization](RENDERING_AND_VISUALIZATION.md).
## Binding policy is not project truth
Capability mode, no-AST preservation, diagnostics, and projection modes describe one running
process or generated client binding. They do not rewrite the descriptor or canonical graph.
`--no-ast` forbids AST-family adapter evolution and Logic publication/retrieval for that binding.
It does not inspect parser implementation, sandbox the filesystem, or convert an existing
AST/Tree-sitter adapter into a no-AST adapter. Read [Policy precedence](POLICY_PRECEDENCE.md) and
[Legacy and no-AST operation](LEGACY_AND_NO_AST.md).
## Trust the narrowest evidence
DocForge reports stable IDs, hashes, generations, omissions, truncation, and provenance limits so a
consumer can distinguish proof from inference. A source inventory is not a semantic graph; a
syntax-level relationship is not compiler resolution; a configuration doctor is not a connection
test; a viewer snapshot is not continuous monitoring; and a benchmark is not a release.
Continue with the [Project descriptor](PROJECT_DESCRIPTOR.md), [New-project
quickstart](NEW_PROJECT_QUICKSTART.md), or [Core contract](CONTRACT.md).

View file

@ -4,6 +4,10 @@ DocForge Release 1 adapters return one complete immutable projection. That contr
supported. The incremental compiler adds an opt-in source-scoped contract that avoids reparsing
unchanged files while preserving the same validated, atomically published graph.
Import these contracts from the public `docforge.adapter_sdk` facade. See
[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for the preserved one-method contract and
[Reference Adapters](REFERENCE_ADAPTERS.md) for the four maintained implementations.
## Release 1 compatibility
The incremental interface is additive:
@ -34,18 +38,26 @@ Incremental indexing is an extraction optimization. It does not weaken publicati
An interrupted extraction never replaces the last validated SQLite index. A malformed,
incompatible, or missing cache is a cache miss, not a partial graph.
One cache generation is bounded to 10,000 source contributions and 64,000,000 encoded bytes.
Aggregate graph and Logic assembly limits are described in the
[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md#current-aggregate-bounds).
## Adapter contract
An incremental loader implements all three methods:
An incremental loader implements the first, second, and fourth methods. It implements
`load_complete_assembly()` as well when it publishes Logic:
```python
class MyAdapter:
def load_manifest(self) -> AdapterManifest: ...
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ...
def load_complete_assembly(self) -> AdapterAssembly: ...
def load_projection(self) -> AdapterProjection: ...
```
`load_projection()` remains the deterministic full-rebuild fallback and equivalence oracle.
`load_projection()` remains the deterministic full-rebuild primary-graph oracle. An incremental
adapter that publishes Logic must additionally implement `load_complete_assembly()` as the
cache-independent complete graph-plus-Logic oracle.
Each `AdapterSource` declares:
@ -137,7 +149,8 @@ fresh process must import and validate the current adapter.
```
Adapters can call `AdapterProject.verify_incremental_equivalence()` in release and contract tests.
The check compares project identity, revision, source hash, nodes, and relationships against
The check compares project identity, revision, source hash, nodes, relationships, and Logic against
the independent complete assembly. It also requires the complete assembly's primary graph to match
`load_projection()`.
## Manual changes and relationships
@ -170,12 +183,29 @@ logic empty until they implement a language analyzer.
This boundary prevents thousands of boolean expressions and basic blocks from polluting Nodes,
Flow, Web, ordinary search, or architectural traversal. The Logic tab and `docforge_get_logic`
request one function-scoped projection on demand. The built-in analyzers cover Python,
JavaScript, and C++. Python uses the standard-library AST. JavaScript and C++ share pinned
Tree-sitter infrastructure with thin language-aware control-flow profiles. Parsers run only while
extracting a changed source contribution; ordinary graph reads do not load or execute them. A
grammar alone supplies syntax, not control-flow meaning, so each new language still needs a small
semantic profile for its branch, loop, case, exception, and termination constructs. All analyzers
report possible static paths; they do not claim runtime branch outcomes.
JavaScript, TypeScript, and C++. Python uses the standard-library AST. JavaScript, TypeScript, and
C++ use distinct optional Tree-sitter grammars with thin language-aware control-flow profiles.
Ordinary graph reads use stored projections and do not load or execute these parsers. A grammar
alone supplies syntax, not control-flow meaning, so each new language still needs a profile for
its branch, loop, case, exception, and termination constructs. All analyzers report possible
static paths; they do not claim runtime branch outcomes.
## Manifest and warm-parser scope
Parser work is language-specific and must be measured at the correct boundary:
- Python manifest construction fingerprints source and tokenizes local imports without calling
`ast.parse`.
- JavaScript and TypeScript manifest construction lexes static relative module specifiers without
invoking their distinct Tree-sitter extraction parsers. Focused tests prove this behavior on an
unchanged warm build.
- The C++ reference manifest parses inventoried sources with `tree-sitter-cpp` to discover quoted
include dependencies. A warm C++ extraction-cache hit is not a zero-parser claim.
The maintained Python benchmark instruments the unchanged warm path and requires zero
`ast.parse` calls and zero `extract_source` calls. That exact zero-parser benchmark claim is Python
only. JavaScript and TypeScript retain focused parser-free-manifest tests; C++ deliberately does
not.
## Full rebuilds

80
docs/LEGACY_AND_NO_AST.md Normal file
View file

@ -0,0 +1,80 @@
# Legacy adapters and no-AST policy
DocForge preserves the original one-method adapter contract while offering incremental extraction,
complete graph-plus-Logic assemblies, and an independently selectable no-AST MCP policy. These are
separate compatibility boundaries.
## One-method adapters remain valid
An existing adapter that implements only:
```python
def load_projection(self) -> AdapterProjection: ...
```
continues to work. It receives the same projection validation, indexing, querying, visualization,
and MCP behavior as before. It does not receive extraction-cache speedups and publishes no Logic
through the complete assembly contract.
Incremental adoption is additive. Implement `load_manifest()` and `extract_source()` while keeping
`load_projection()` as the independent complete graph oracle. If incremental contributions publish
Logic, also implement `load_complete_assembly()` so complete and incremental graph-plus-Logic
output can be compared exactly. The public authoring surface is `docforge.adapter_sdk`; see the
[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md).
## What no-AST means
`--no-ast` is an immutable policy on one MCP process binding. It preserves the configured adapter
strategy but forbids AST, Tree-sitter, compiler-AST, and function-Logic evolution for that binding.
It also rejects nonempty Logic before index publication and rejects a pre-existing index that
contains Logic before reads or live visualization.
Start the generic server with:
```bash
docforge-mcp --project-root /absolute/path/to/project --no-ast
```
Project-owned integrations select the same policy with
`create_project_server(..., no_ast=True)` or `create_read_only_server(..., no_ast=True)`.
`docforge_bootstrap` and `docforge_get_contract` report the effective policy, and
`docforge_get_logic` returns `adapter_policy_forbids_logic`.
Changing this policy requires a new process. It does not rewrite or reload an adapter in place.
## What no-AST does not mean
No-AST is not:
- an inspection mechanism that proves which parser an arbitrary adapter uses;
- a filesystem, Python-import, or process sandbox;
- a promise that the adapter performs no non-AST fingerprinting, dependency discovery, caching, or
complete-projection loading; or
- permission to change project files, adapter code, or the configured capability surface.
Repository permissions and project instructions remain responsible for processes that have direct
filesystem access. The MCP boundary still excludes shell execution, arbitrary file operations,
Git mutation, builds, deployment, publication, project switching, and cross-project retrieval.
## Reference adapters are not no-AST adapters
The built-in Python reference extracts syntax and Logic with the standard-library AST. The
JavaScript, TypeScript, and C++ references extract syntax and Logic with their optional
Tree-sitter grammars. Their normal projections contain Logic, so they must not be presented or
configured as no-AST adapters.
The narrower performance statements in [Reference Adapters](REFERENCE_ADAPTERS.md) concern
specific manifest and warm-cache paths. A parser-free manifest does not turn an adapter that
publishes AST-derived Logic into a no-AST adapter.
## Migration choices
Keep a one-method adapter when its complete rebuild is acceptable and no function Logic is needed.
Adopt incremental extraction when measured source parsing dominates synchronization. Adopt the
complete assembly contract when Logic is published or overlapping raw evidence needs deterministic
ownership.
Choose no-AST only when the project binding must forbid AST-derived publication and future adapter
upgrades of that kind. It is a binding policy, not a substitute for adapter confinement,
conformance tests, restart detection, cache recovery, or the full proof matrix in
[Incremental Adapter Indexing](INCREMENTAL_INDEXING.md).

View file

@ -28,6 +28,30 @@ Bootstrap and contract results include `effective_policy` schema version 1 plus
registered surface and startup-bound proposal/application access. The project descriptor remains
schema version 1 and does not silently acquire machine-specific process policy.
## Fixed reference binding
`docforge.reference_mcp` is the fixed runnable binding for the in-repository Python, JavaScript,
TypeScript, and C++ reference adapters:
```bash
python -I -m docforge.reference_mcp \
--project-root /absolute/project \
--capability-mode read
```
It loads only `.docforge/reference-adapter.toml`, selects only the fixed provider for the declared
language, and constructs the same project-bound read surface through `create_read_only_server()`.
Bootstrap binding metadata includes `server_module = "docforge.reference_mcp"`,
`adapter_mode = "reference"`, the selected `reference_language`, and the exact
`reference_config_hash`.
The reference command accepts read capability mode only. It registers all 21 read tools below and
no isolated proposal or canonical-application tools. The configuration cannot select a provider,
module, command, arguments, working directory, environment, or arbitrary discovery behavior. See
[Reference Adapters](REFERENCE_ADAPTERS.md) for language scope and
[Agent Integration](AGENT_INTEGRATION.md) for the immutable launcher and generated client
fragments.
## Read tools
- `docforge_bootstrap`
@ -45,6 +69,8 @@ schema version 1 and does not silently acquire machine-specific process policy.
- `docforge_get_task_context`
- `docforge_validate_project`
- `docforge_render_status`
- `docforge_graph_plan`
- `docforge_graph_render_status`
- `docforge_visualize`
- `docforge_stop_visualization`
- `docforge_visualization_status`
@ -349,3 +375,6 @@ permissions and project instructions remain responsible for that broader boundar
The legacy `adapter_policy` object remains byte-compatible. It is now a projection of the
versioned `effective_policy`; `--no-ast` restrictively overrides adapter evolution, AST analysis,
and Logic indexing without widening any other capability.
See [Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for the compatibility boundary and
why the AST and Tree-sitter reference adapters are not no-AST adapters.

79
docs/MIGRATING_FROM_V1.md Normal file
View file

@ -0,0 +1,79 @@
# Migrating from DocForge v1
DocForge2 preserves the `docforge` distribution, Python package, CLI command, MCP tool prefix,
generic project descriptor version, and legacy adapter entry point. Migration is an additive
validation exercise, not a canonical-content rewrite.
Read [compatibility](COMPATIBILITY.md), [legacy and no-AST operation](LEGACY_AND_NO_AST.md), and
[recovery](RECOVERY_AND_PERFORMANCE.md) before changing a production binding.
## What remains compatible
- Generic Markdown and TOML projects keep `.docforge/project.toml` schema version 1.
- A custom adapter implementing only `load_projection()` remains valid.
- Existing canonical nodes, stable IDs, source paths, relationship vocabulary, changesets, and
reviewed hashes are not silently rewritten.
- Existing CLI and MCP names remain available. New fields, commands, tools, plans, policies, and
reference integrations are additive.
- `docforge-mcp --no-ast` remains the shorthand for the preserved no-AST binding.
Disposable index and cache schemas may change. Rebuild them rather than copying them as authority.
## Recommended migration
1. Record the v1 package version, adapter identity, project descriptor, canonical source hash,
active changesets, generated client configuration, and current rendered outputs.
2. Back up canonical sources and active proposal files. Derived `.docforge/cache`, preview,
portable-graph, and viewer state need not be authoritative backups.
3. Install the DocForge2 candidate in a separate environment. Do not repoint the production MCP
binding yet.
4. Run the existing adapter through complete loading and `ProjectIndex.check()`. For incremental
adapters, verify complete/incremental equivalence. Logic-producing incremental adapters must
implement the independent complete assembly oracle.
5. Compare project ID, adapter, revision, source hash, primary node and edge hashes, Logic policy,
and retrieval results with the v1 evidence.
6. Recompose effective capability and projection policy. Do not assume that a new default grants
proposal, application, rendering, or viewer access.
7. Rebuild disposable indexes, extraction caches, fragments, receipts, previews, and portable
artifacts from the validated candidate.
8. Generate a new client fragment. Generic projects use `docforge configure`; custom adapters use
`AdapterLauncherV1` with `generate_adapter_client_configuration()`.
9. Run doctor, real MCP bootstrap, representative retrieval, rendering status, recovery, and
exact-hash proposal/application checks in a disposable or shadow environment.
10. Repoint one production binding only after the candidate and rollback procedure pass.
## Adopting reference adapters
The repository reference adapters are narrow syntax evidence, not automatic semantic replacements
for mature v1 integrations. Python publishes modules, classes, functions, and local imports.
JavaScript and TypeScript publish syntax plus static project-relative imports and re-exports.
C++ uses a confined `compile_commands.json` as inert translation-unit inventory and publishes
directly resolvable project-local quoted includes.
Do not replace a compiler-, language-server-, or project-owned semantic adapter if the task depends
on resolved calls, types, inheritance, macros, compiler include semantics, runtime facts, or
ownership. Use [reference adapters](REFERENCE_ADAPTERS.md) as a measured starting point.
## Changesets
Existing changesets remain bound to their original project, root, base revision, source hash,
writer, operations, and exact hash. Do not edit a changeset to make it look current. Retrieve and
validate it through the candidate. Rebase only when every precondition remains true; otherwise
create and review a new proposal.
Before enabling canonical application, prove the exact serializer round trip and recovery behavior
for the selected adapter. The fixed reference MCP server is intentionally read-only.
## Rollback
Keep the v1 environment, configuration fragment, and service definition until the DocForge2
binding has passed live validation. To roll back:
1. Stop the DocForge2 MCP/viewer processes.
2. Restore the previous client or service binding.
3. Restore canonical sources only if a verified application receipt says they changed and the
adapter-specific rollback requires it.
4. Discard DocForge2-derived indexes, receipts, fragments, previews, and portable artifacts.
5. Restart v1 and verify its recorded project/source identity.
Never move a published tag or reuse a version identity for a corrected release.

View file

@ -0,0 +1,95 @@
# Milestone 4 baseline
## Scope and method
This baseline records the adapter SDK, Python reference adapter, incremental equivalence, and
recovery behavior completed in Milestone 4. It was captured on 2026-07-29 from clean executable
candidate `95271dcf2e48045b9d3aed9b9ea09c7fc155692c`.
The maintained command was:
```bash
.venv/bin/python tools/milestone4_benchmark.py \
--mode full \
--output benchmarks/milestone4-2026-07-29.json
```
The synthetic project contains 334 Python files. Each file contributes one module, one function,
and one argument node, for 1,002 primary nodes. Imports form a deterministic chain. Durations use
`time.perf_counter_ns()` and nearest-rank p95. Each ordinary result is serialized as compact sorted
JSON for its response-size gate. Per-operation memory uses `tracemalloc`; cumulative process
high-water uses `RUSAGE_SELF`.
Environment:
- Linux 7.1.3-200.nobara.fc44.x86_64 with glibc 2.43.
- CPython 3.14.6.
- x86_64.
- Three warm samples; cold, equivalence, and recovery operations run once.
- Per-operation traced-memory ceiling: 268,435,456 bytes.
- Process high-water ceiling: 536,870,912 bytes.
- Response ceiling: 524,288 bytes.
The complete machine-readable result is
[`benchmarks/milestone4-2026-07-29.json`](../benchmarks/milestone4-2026-07-29.json).
Its SHA-256 is `b6a871dde730a119fc0a138c47bd25f2c533ed3171a9a3d098075aef08173600`.
The stable evidence payload SHA-256 is
`4a8db461a311b5df4abd5aa00063e9a347d8b9dba19b30e35684f561d5271549`.
## Results
| Operation | Median | p95 | Limit | Traced peak | Response |
|---|---:|---:|---:|---:|---:|
| Cold incremental build | 1,599.760 ms | 1,599.760 ms | 20,000 ms | 68,573,540 B | 14,580 B |
| Warm incremental build | 1,244.427 ms | 1,265.387 ms | 20,000 ms | 69,997,166 B | 14,578 B |
| Complete/incremental equivalence | 1,897.919 ms | 1,897.919 ms | 30,000 ms | 64,305,919 B | 230 B |
| Corrupt extraction-cache recovery | 1,695.406 ms | 1,695.406 ms | 30,000 ms | 69,776,923 B | 14,580 B |
| Corrupt index recovery | 1,707.879 ms | 1,707.879 ms | 30,000 ms | 68,561,924 B | 14,779 B |
Process high-water was 78,798,848 bytes. The regression limits intentionally leave multiple times
the measured headroom; they are tripwires, not performance promises.
## Deterministic graph and Logic evidence
The candidate produced:
- 1,002 primary nodes with hash
`f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f`.
- 1,001 primary edges with hash
`00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35`.
- 334 Logic projections containing 2,338 Logic nodes and 2,338 Logic edges, with hash
`c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c`.
- Complete assembly hash
`2888182ab765fbffe3ba873c1613345640e8e6d89be74ddfca7452c0a5056345`.
- Source hash
`30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2`.
The independently loaded complete assembly and the incremental assembly matched exactly across
project identity, revision, source, nodes, edges, and Logic.
## Work and recovery gates
The warm build recorded all 334 sources as cache hits, zero reparsed sources, zero `ast.parse`
calls, and zero `extract_source` calls. Corrupting the extraction cache forced all sources through
extraction and reproduced the same graph and Logic hashes. Corrupting SQLite rebuilt the index
entirely from cache hits with zero parsing or extraction and reproduced those hashes.
This zero-parser evidence applies to the Python benchmark. Focused JavaScript and TypeScript tests
prove parser-free manifests. The C++ reference manifest uses Tree-sitter for bounded quoted-include
discovery and does not make a zero-warm-parser claim.
## Fresh-wheel and reference evidence
The offline adoption proof built and installed the base wheel without Tree-sitter packages,
built and checked a real Python reference project, started the isolated reference MCP server, and
performed bootstrap, search, and exact retrieval over its 21 read tools. Selecting C++ without its
extra failed with `optional_dependency_missing` and `docforge[cpp]` remediation.
Focused fixtures produced:
- Python: 14 nodes, 13 edges, 5 Logic projections.
- JavaScript: 14 nodes, 14 edges, 5 Logic projections.
- TypeScript: 13 nodes, 14 edges, 4 Logic projections.
- C++: 17 nodes, 16 edges, 5 Logic projections.
These fixture counts verify implementations; they are not language-wide completeness claims.

View file

@ -0,0 +1,83 @@
# Milestone 4 closeout
## Outcome
Milestone 4 is complete. New projects can adopt a public adapter SDK or one of four narrow
repository reference integrations, attach a fixed read-only MCP server, and follow maintained
product documentation without reading core implementation.
Implemented contracts:
- Stable `docforge.adapter_sdk` authoring imports.
- Independent complete primary-graph-plus-Logic oracle and exact incremental equivalence.
- Bounded adapter assemblies and version-1 extraction caches.
- Base Python, optional JavaScript, optional TypeScript, and optional C++ reference integrations.
- Closed `.docforge/reference-adapter.toml` and fixed `docforge.reference_mcp` read-only binding.
- Immutable, project-bound, launchable `AdapterLauncherV1` declarations and generated Codex,
Claude, and OpenClaw fragments for custom adapters.
- Live implementation-derived CLI and MCP reference tables with race-safe publication and drift
checking.
- Strict documentation graph, link, anchor, H1, reachability, required-page, generated-notice, and
documented-reference-config checks.
- Offline fresh-wheel adoption and maintained 1,002-node scale/recovery gates.
## Candidate evidence
The frozen executable candidate is
`95271dcf2e48045b9d3aed9b9ea09c7fc155692c`.
Its complete executable gate passed:
- Ruff formatting and Python lint.
- HTML, rendered-manual HTML, portable-graph HTML, CSS, and JavaScript checks.
- Pyright with zero diagnostics.
- Warning-strict compilation.
- 142 contract tests and 268 subtests.
- 347 complete tests and 402 subtests.
- Three Playwright and axe accessibility flows for the manual, portable graph, and live viewer.
- Lock and npm dependency-tree checks.
- Wheel and source-distribution builds.
- Offline fresh-wheel adoption.
- Milestone 0, 1, 2, 3, and 4 smoke benchmarks.
The clean full benchmark passed exact graph-plus-Logic equivalence, warm zero Python parser and
extraction work, corrupt-cache recovery, corrupt-index recovery, response, memory, and latency
gates. Exact results are in [the Milestone 4 baseline](MILESTONE_4_BASELINE.md) and
[`benchmarks/milestone4-2026-07-29.json`](../benchmarks/milestone4-2026-07-29.json).
Gitleaks 8.30.1 scanned the Milestone 4 commit range and candidate tree with no findings. The SSH
remote syntax prevents Gitleaks from constructing finding hyperlinks; it does not affect scanning.
## Reference scope
- Python uses the standard-library AST and publishes syntax plus local imports.
- JavaScript and TypeScript use distinct optional Tree-sitter grammars and publish syntax plus
project-local static relative imports and re-exports.
- C++ uses a confined `compile_commands.json` as inert translation-unit inventory and publishes
syntax plus directly resolvable project-local quoted includes.
- The references do not claim resolved calls, inheritance, types, symbol references, compiler
include semantics, macro semantics, runtime behavior, or semantic ownership.
The C++ reference never executes a compiler or compilation-database command. It is not a Clang
semantic adapter.
## Preserved boundaries
- The `docforge` distribution, Python package, CLI, MCP executable, and tool names remain.
- Generic projects and one-method `load_projection()` adapters remain supported.
- Descriptor schema version 1, index schema version 3, effective policy version 1, and no-AST
behavior remain.
- Heavy language frontends are optional. Base generic and Python operation installs no
Tree-sitter distribution.
- Reference MCP is read-only. Proposal and application remain explicit project-owned gates.
- No language adapter was separately published.
- No WorldForge, ScrapeStation, legacy-repository, production-binding, storage, or self-hosting
change was made.
- No tag or Forgejo release was created for Milestone 4.
## Later work
Milestone 5 owns stabilization and the first DocForge2 release. Release identity, compatibility
matrix, migration and recovery proofs, comparative real-task evidence, versioning, tagging, and
publication must be validated there. Remote adapters, render farms, third-party renderers,
cross-project graphs, storage replacement, and self-hosting remain deferred without measured need.

View file

@ -1,6 +1,276 @@
# DocForge setup moved to the user manual
# New-project quickstart
The complete installation, project setup, visualization, CLI, MCP, application, adapter, and
troubleshooting reference now lives in the [DocForge user manual](USER_MANUAL.md).
This guide takes a new installation from an empty project binding to a validated generic manual or
one of DocForge's fixed reference source graphs. For the complete operating reference, see the
[user manual](USER_MANUAL.md).
This file remains only so existing bookmarks and links continue to resolve.
## 1. Install DocForge
DocForge requires Python 3.12 or newer. A base installation includes the generic project service,
the public adapter SDK, the Python reference integration, CLI, MCP server, renderers, and viewer
assets. It does not install Tree-sitter.
From a source checkout:
```bash
git clone <repository-url> /absolute/path/DocForge
cd /absolute/path/DocForge
uv sync --group dev
DOCFORGE=/absolute/path/DocForge/.venv/bin/docforge
DOCFORGE_MCP=/absolute/path/DocForge/.venv/bin/docforge-mcp
DOCFORGE_PYTHON=/absolute/path/DocForge/.venv/bin/python
```
For an isolated consumer environment, install the checkout or a built wheel with `uv pip install`.
Add only the language extras that project needs:
```bash
uv venv /absolute/path/docforge-env --python 3.12
uv pip install --python /absolute/path/docforge-env/bin/python /absolute/path/DocForge
uv pip install --python /absolute/path/docforge-env/bin/python \
"/absolute/path/DocForge[javascript]"
uv pip install --python /absolute/path/docforge-env/bin/python \
"/absolute/path/DocForge[typescript]"
uv pip install --python /absolute/path/docforge-env/bin/python \
"/absolute/path/DocForge[cpp]"
```
The `languages` extra installs all three optional grammar families. JavaScript and TypeScript are
separate extras because they use distinct grammars. The C++ extra supplies a syntax grammar, not a
compiler or Clang semantic frontend.
Confirm the installation:
```bash
"$DOCFORGE" --help
"$DOCFORGE_MCP" --help
```
## 2. Choose a project route
Use a generic project when Markdown or TOML documentation is canonical. Use a fixed reference
adapter when you want a bounded source inventory and syntax-level graph for Python, JavaScript,
TypeScript, or C++. Use a project-owned adapter when the production contract must supply richer
semantics.
- Generic manual: continue with [Create a generic project](#create-a-generic-project).
- Fixed source example: continue with [Use a reference adapter](#use-a-reference-adapter).
- Production language frontend: follow the [Adapter authoring guide](ADAPTER_AUTHORING_GUIDE.md).
## Create a generic project
Set the absolute project root and assess it without writing:
```bash
PROJECT=/absolute/path/MyProject
"$DOCFORGE" --project-root "$PROJECT" onboard
```
The assessment reports detected languages, build evidence, documentation candidates, current
configuration, and available capabilities. Detection never invents a source graph.
Create a generic starter explicitly:
```bash
"$DOCFORGE" --project-root "$PROJECT" onboard \
--scaffold \
--project-id my-project \
--title "My Project"
```
Scaffolding is create-only and refuses existing target files. It creates:
- `.docforge/project.toml`, the generic project descriptor;
- `docs/docforge/content/architecture-overview.md`, one canonical node;
- `.docforge/templates/manual.html`, one built-in-renderer template;
- derived index, receipt, and rendered output below `.docforge`.
If a language was detected, source graph status remains `adapter_required`. The generic starter
does not claim source semantics.
Validate and inspect it:
```bash
"$DOCFORGE" --project-root "$PROJECT" validate
"$DOCFORGE" --project-root "$PROJECT" reindex
"$DOCFORGE" --project-root "$PROJECT" search architecture
"$DOCFORGE" --project-root "$PROJECT" show architecture.overview
"$DOCFORGE" --project-root "$PROJECT" render-status
```
The descriptor is explained field by field in [Project descriptor](PROJECT_DESCRIPTOR.md). Add
canonical nodes only after choosing their authority, stable IDs, families, statuses, and allowed
relationships; [Core concepts and authority](CORE_CONCEPTS_AND_AUTHORITY.md) defines those terms.
### Start a generic MCP binding
Start read-only first:
```bash
"$DOCFORGE_MCP" \
--project-root "$PROJECT" \
--capability-mode read
```
Call `docforge_bootstrap` before other tools. It reports the exact project identity, generation,
effective policy, projection policy, available capabilities, and recommended first read.
To enable proposals, the descriptor must declare the writer and the process must select it:
```bash
"$DOCFORGE_MCP" \
--project-root "$PROJECT" \
--capability-mode proposal \
--proposal-writer project-editor
```
Canonical application is a separate startup gate. Do not add it to a read-only client:
```bash
"$DOCFORGE_MCP" \
--project-root "$PROJECT" \
--capability-mode application \
--proposal-writer project-editor \
--canonical-applier project-editor
```
Application accepts one exact reviewed changeset hash. It does not commit, push, build, deploy, or
publish the project.
### Generate a generic client fragment
Preview is side-effect free:
```bash
"$DOCFORGE" configure codex --project "$PROJECT"
"$DOCFORGE" configure claude --project "$PROJECT"
"$DOCFORGE" configure openclaw --project "$PROJECT"
```
Add `--output /absolute/path/new-fragment` to create one new private standalone file. Generation
does not merge with or replace a different existing file. Diagnose an installed binding with:
```bash
"$DOCFORGE" doctor --client codex --project "$PROJECT"
```
Doctor is a bounded configuration inspector, not a connection test. See [Agent
integration](AGENT_INTEGRATION.md) for client-specific layouts and limitations.
## Use a reference adapter
Reference adapters read one fixed descriptor:
`.docforge/reference-adapter.toml`. The path is not selectable.
Create a Python project configuration:
```toml reference-adapter
schema_version = 1
project_id = "my-python-project"
title = "My Python Project"
language = "python"
source_roots = ["src"]
```
JavaScript uses `language = "javascript"` and the `javascript` extra. TypeScript uses
`language = "typescript"` and the `typescript` extra.
C++ additionally requires a confined compilation database:
```toml reference-adapter
schema_version = 1
project_id = "my-cpp-project"
title = "My C++ Project"
language = "cpp"
source_roots = ["src", "include"]
compilation_database = "compile_commands.json"
```
The C++ integration reads `compile_commands.json` only as bounded translation-unit inventory and
fingerprint evidence. It never executes a recorded command or compiler.
Run the fixed read-only server through the same installed Python interpreter:
```bash
"$DOCFORGE_PYTHON" -I -m docforge.reference_mcp \
--project-root "$PROJECT" \
--capability-mode read
```
The binding chooses one in-package provider from the descriptor language. It accepts no provider
module, command, argument list, working directory, environment, discovery rule, proposal writer,
or canonical applier. Its MCP surface is the read subset documented in the [generated command
reference](COMMAND_REFERENCE.md).
### Generate a reference-adapter client fragment
Generic `docforge configure` intentionally refuses custom adapters. Construct the adapter project
and immutable launcher, then call the custom-adapter generator:
```python
from pathlib import Path
from docforge.adapter_launcher import AdapterLauncherV1
from docforge.client_config import generate_adapter_client_configuration
from docforge.reference_mcp import REFERENCE_MCP_MODULE, create_reference_project
root = Path("/absolute/path/MyProject").resolve(strict=True)
project = create_reference_project(root)
launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE)
preview = generate_adapter_client_configuration(
project,
launcher,
"codex",
capability_mode="read",
)
print(preview["artifact"]["content"])
```
Use `client="claude"` or `client="openclaw"` for those formats. Pass an absolute `output` path only
when creating a new standalone private fragment. The generated launch is bound to the selected
project, descriptor hash, adapter identity, installed module, isolated interpreter, policy, and
source availability evidence.
Read [Reference adapters](REFERENCE_ADAPTERS.md) before relying on the graph. The Python example
publishes local imports, the JavaScript and TypeScript examples publish project-local static
relative imports and re-exports, and the C++ example publishes directly resolvable project-local
quoted includes. None is a complete semantic compiler frontend.
## 3. Add visualization only when needed
Install the per-user viewer manager once:
```bash
/absolute/path/DocForge/.venv/bin/docforge-viewer-manager install-user-service
```
Then start a project-bound snapshot:
```bash
"$DOCFORGE" --project-root "$PROJECT" visualize
"$DOCFORGE" --project-root "$PROJECT" visualization-status
```
The listener is loopback-only and tokenized. The viewer is a derived snapshot, not canonical
authority and not a continuously monitored filesystem view. Read [Rendering and
visualization](RENDERING_AND_VISUALIZATION.md) for manual, portable, and live-viewer differences.
## 4. Verify the maintained checkout
Contributors can run:
```bash
make command-reference-check
make docs-check
make adoption-m4
make benchmark-m4-smoke
```
Run `make gate` before a release candidate. `benchmark-m4-smoke` is routine coverage;
`benchmark-m4` is the maintained full adapter workload.
Continue with [Project onboarding](PROJECT_ONBOARDING.md) for production integration,
[Policy precedence](POLICY_PRECEDENCE.md) before widening a binding, and [Security](SECURITY.md)
before exposing any MCP process.

202
docs/POLICY_PRECEDENCE.md Normal file
View file

@ -0,0 +1,202 @@
# Policy precedence
DocForge composes an immutable effective policy for each project-bound process. It resolves
restrictions in a fixed order:
```text
core safety
> explicit binding
> no-AST shorthand
> resource availability
```
A lower layer can make a requested operation unavailable; it cannot override a higher-layer
prohibition. Bootstrap and generated client evidence report the composed result and hashes so a
client does not need to infer policy from command-line arguments.
## 1. Core safety
Core safety is unconditional. No capability mode exposes:
- arbitrary file access or project switching;
- arbitrary renderer, module, command, shell, argument, working-directory, or environment
selection;
- Git mutation;
- project builds or compiler execution;
- deployment or publication.
Canonical application is limited to DocForge's validated serializer boundary. Adapter launch is
limited to the immutable launcher contract. Rendering is limited to declared views and fixed
built-in workers. An indexed instruction cannot alter any of these rules.
## 2. Explicit binding
A process is bound at startup to one project root, descriptor, adapter, capability mode, writer and
applier identities when present, no-AST selection, diagnostics selection, and projection modes.
The binding does not change during the process lifetime.
Capability modes are:
- `read`: register only the read surface;
- `proposal`: add proposal tools when a valid selected writer is available;
- `application`: require a startup-bound canonical applier and expose exact-hash application;
- `operator`: reserved; it currently adds no tools.
Mode describes the maximum registered surface. Actual authority can be narrower. A descriptor must
declare the selected writer, including allowed families and operation types. Application requires
the matching configured writer, changeset creator, and canonical-applier identity. A mode name
cannot create a missing descriptor grant.
Generic generated client fragments default to read mode. Other construction paths preserve their
documented compatible factory defaults. Treat `docforge_bootstrap.session_contract` and its actual
capabilities as authoritative for a running server.
## 3. No-AST shorthand
`--no-ast` is a restrictive compatibility shorthand. It composes:
- adapter evolution `preserve`;
- AST analysis `forbidden`;
- Logic indexing `off`;
- `docforge_get_logic` blocked;
- prohibitions on AST, Tree-sitter, compiler-AST, and function-Logic upgrades.
Non-AST source fingerprinting and incremental caching remain allowed. Existing one-method adapters
continue to use `load_projection()`.
The shorthand does not inspect how an existing adapter was implemented, sandbox its filesystem
reads, or transform an AST-based adapter into a no-AST adapter. Do not run the Python,
JavaScript/TypeScript, or C++ syntax reference integrations and then describe the binding as a
proved no-AST integration. See [Legacy and no-AST operation](LEGACY_AND_NO_AST.md).
## 4. Resource availability
Even an allowed policy cannot create a missing resource:
- application mode requires a startup-bound canonical applier;
- manual `explicit` or `auto` requires declared manual render configuration;
- manual `auto` also requires canonical application in the current operation or server;
- portable graph `explicit` requires declared portable graph configuration;
- live viewer `on-demand` requires the viewer runtime;
- an optional reference language requires its installed extra;
- a project-owned launcher module must be installed, isolated, project-owned, and unchanged.
Unavailable requested modes fail before hidden work. Missing optional language grammars return an
actionable install target such as `docforge[typescript]` or `docforge[cpp]`; DocForge does not
silently downgrade to a different frontend.
## Effective process policy
The version-1 effective policy reports:
- capability mode and whether it came from a factory default or explicit selection;
- adapter evolution, AST analysis, and Logic indexing;
- automatic synchronization and validated integrity;
- the compatible manual-render projection;
- profiling state;
- blocked tools and prohibitions;
- the exact precedence list.
This version-1 projection preserves compatibility. It is not the complete version-2 rendering
policy; portable graph and live-viewer choices are reported separately.
## Independent projection policy
Manual rendering, portable graph publication, and live visualization use a separate immutable
version-2 policy:
```text
manual: auto | explicit | disabled
portable_graph: explicit | disabled
live_viewer: on-demand | disabled
```
When a selector is omitted, composition uses availability-aware defaults:
- manual is `auto` only when manual configuration and canonical application are both available;
otherwise it is `explicit` when configured, or `disabled`;
- portable graph is `explicit` when configured, otherwise `disabled`;
- live viewer is `on-demand` when the runtime is available, otherwise `disabled`.
An explicit non-disabled selection for a missing resource returns
`projection_policy_unavailable`. An active operation prohibited by the selected policy returns
`projection_policy_forbids_operation` before planning, rendering, or viewer startup.
Status remains intentionally narrower than active work. Manual and portable receipt-only status
are available when their active operations are disabled. Viewer status and explicit stop remain
available when viewer start is disabled.
Ordinary standalone CLI rendering uses manual `explicit`. Manual `auto` belongs to a canonical
application operation that owns automatic regeneration.
## Diagnostics do not grant authority
`--diagnostics` enables bounded request-local stage timings and compiler-work counters. It does not
enable tools, broaden paths, retain project content, or displace a primary MCP result that already
needs the response budget.
## Descriptor policy and process policy
The project descriptor is canonical project configuration. The process policy is a runtime
restriction. They compose by intersection:
```text
operation is available
only if core permits it
and the startup binding registers it
and no-AST permits it
and required resources exist
and the descriptor grants the requested project authority
and current graph/hash preconditions validate
```
Changing a descriptor does not retarget a running project-owned process. Descriptor, adapter
implementation, or launcher drift requires a fresh process.
## Common decisions
For an agent that only reads documentation:
```bash
docforge-mcp \
--project-root /absolute/path/MyProject \
--capability-mode read
```
For an agent that may prepare reviewable proposals:
```bash
docforge-mcp \
--project-root /absolute/path/MyProject \
--capability-mode proposal \
--proposal-writer project-editor
```
For a tightly controlled application process:
```bash
docforge-mcp \
--project-root /absolute/path/MyProject \
--capability-mode application \
--proposal-writer project-editor \
--canonical-applier project-editor
```
For a read binding with every active output projection disabled:
```bash
docforge-mcp \
--project-root /absolute/path/MyProject \
--capability-mode read \
--manual-render-policy disabled \
--portable-graph-policy disabled \
--live-viewer-policy disabled
```
Prefer the narrowest binding that completes the workflow. Call `docforge_bootstrap` first and use
the returned effective policy, projection policy, actual capabilities, and prohibitions rather
than assumptions based on client configuration.
See the [MCP contract](MCP_CONTRACT.md), [Security](SECURITY.md), [Project
descriptor](PROJECT_DESCRIPTOR.md), and [Rendering and
visualization](RENDERING_AND_VISUALIZATION.md).

279
docs/PROJECT_DESCRIPTOR.md Normal file
View file

@ -0,0 +1,279 @@
# Project descriptor
A generic DocForge project is selected by one fixed file:
`.docforge/project.toml` beneath an explicit project root. The descriptor is schema version 1.
DocForge validates both the JSON-schema shape in `schemas/project.schema.json` and runtime
invariants that schema alone cannot prove.
Reference integrations use a different fixed descriptor,
`.docforge/reference-adapter.toml`; see [Reference adapters](REFERENCE_ADAPTERS.md).
Project-owned adapters construct the same runtime `ProjectDescriptor` contract through the public
adapter SDK.
## Complete generic example
```toml
schema_version = 1
project_id = "my-project"
title = "My Project"
adapter = "generic"
[sources]
content_roots = ["docs/docforge/content"]
authority_files = []
[derived]
cache_root = ".docforge/cache"
index = ".docforge/cache/index.sqlite3"
[changesets]
root = ".docforge/changesets"
[[changesets.writers]]
id = "project-editor"
families = ["architecture", "operations", "system"]
operations = ["create", "update", "move", "delete"]
[render]
template_root = ".docforge/templates"
preview_root = ".docforge/previews"
[[render.views]]
id = "manual"
renderer = "generic_html"
template = "manual.html"
output = ".docforge/rendered/manual.html"
title = "My Project Manual"
families = ["architecture", "operations", "system"]
[graph_render]
output_root = ".docforge/portable-graph"
[[graph_render.views]]
id = "architecture"
renderer = "portable_graph_html"
output = "architecture.html"
title = "Architecture"
root = "architecture.overview"
initial_mode = "web"
depth = 3
max_nodes = 250
max_edges = 1000
max_work = 100000
families = ["architecture", "system"]
relations = ["depends_on", "owns", "calls", "reads", "writes", "tested_by", "relates_to"]
authorities = []
statuses = ["current", "active", "verified"]
tags = []
include_logic = false
[graph]
allowed_relations = [
"calls",
"depends_on",
"owns",
"reads",
"relates_to",
"tested_by",
"writes",
]
[limits]
max_source_bytes = 500000
max_nodes = 10000
max_query_chars = 500
max_results = 100
max_traversal_depth = 6
max_context_tokens = 12000
max_tool_output_chars = 200000
max_changesets = 100
max_changeset_operations = 100
max_changeset_bytes = 1000000
max_render_views = 20
max_template_bytes = 1000000
max_render_bytes = 1000000
[[profiles]]
id = "development"
families = ["architecture", "operations", "system"]
statuses = ["current", "active", "verified"]
required_nodes = ["architecture.overview"]
token_budget = 8000
dependency_depth = 3
```
Rendering sections are optional. Profiles and limits may also be omitted; runtime defaults then
apply. The required top-level fields are `schema_version`, `project_id`, `title`, `adapter`,
`sources`, `derived`, `changesets`, and `graph`.
## Identity fields
`schema_version` must be `1`.
`project_id` is the stable machine identity. It is lowercase and may contain digits, dots,
underscores, and hyphens after its first character. Do not derive it from a mutable display title.
`title` is the human-readable project name.
`adapter` is `generic` for this file format. Project-owned adapters publish a validated
`adapter_id@adapter_version` identity through their runtime descriptor; changing adapter identity
invalidates incompatible derived state.
The descriptor's byte content contributes to a descriptor hash. Client fragments, launchers, index
evidence, and policy results use that hash to detect drift.
## Canonical sources
`sources.content_roots` lists the project-relative directories containing generic Markdown and TOML
nodes. Each path must resolve beneath the project root. Canonical content roots may not overlap the
derived cache.
`sources.authority_files` lists additional project-relative regular files whose content belongs to
the canonical project identity. They are not automatically parsed as nodes.
A Markdown node contains one TOML metadata block followed by Markdown content:
```markdown
+++
schema_version = 1
id = "architecture.overview"
title = "Architecture overview"
family = "architecture"
authority = "authoritative"
status = "current"
tags = ["architecture"]
summary = "Defines the top-level architecture and ownership."
+++
# Architecture overview
Describe systems, ownership, runtime flow, failure behavior, and proof.
```
Every node ID is project-wide and stable. Every relationship target must resolve. A TOML source may
contain multiple `[[nodes]]` records; proposal-enabled multi-node files need stable
`source_anchor` values where creation or movement requires an exact record boundary.
## Derived state
`derived.cache_root` owns disposable indexes, attestations, extraction caches, render receipts,
projection artifacts, and viewer registry state.
`derived.index` must be inside `derived.cache_root`. Canonical content and cache paths must not
overlap.
Derived state is not a backup. If it is deleted or rejected as corrupt, DocForge rebuilds it from
validated canonical sources.
## Changesets and writers
`changesets.root` is the confined proposal store. It must not overlap canonical content or the
derived cache.
Each `changesets.writers` entry declares:
- a stable writer `id`;
- the node `families` that writer may change;
- allowed `operations`: `create`, `update`, `move`, and/or `delete`.
The descriptor grant is necessary but not sufficient. A process must also select that writer at
startup, and canonical application requires a separately bound matching applier. Capability mode
does not broaden the descriptor grant. See [Policy precedence](POLICY_PRECEDENCE.md).
## Allowed relationships
`graph.allowed_relations` is the exact project vocabulary accepted on edges. It must be nonempty.
Relationship names are stable IDs. DocForge rejects relationships that are not declared and gives
`depends_on` additional cycle validation.
The core does not reinterpret a custom relationship just because its spelling resembles a known
term. Task-context retrieval classifies only the documented versioned aliases and preserves
unknown allowed relationships as `unclassified`.
## Context profiles
Each `profiles` entry defines one bounded context compilation:
- `id` selects the profile;
- `families` and `statuses` filter eligible nodes;
- `required_nodes` names stable nodes that must be present;
- `token_budget` limits compiled content;
- `dependency_depth` bounds relationship expansion.
Profiles choose derived retrieval scope. They do not change node authority or writer permissions.
## Limits
Positive limits bound input, graph, retrieval, proposal, and rendering work. Current defaults are:
- `max_source_bytes = 1000000`
- `max_nodes = 10000`
- `max_query_chars = 500`
- `max_results = 100`
- `max_traversal_depth = 8`
- `max_context_tokens = 32000`
- `max_tool_output_chars = 200000`
- `max_changesets = 1000`
- `max_changeset_operations = 100`
- `max_changeset_bytes = 1000000`
- `max_render_views = 100`
- `max_template_bytes = 1000000`
- `max_render_bytes = 1000000`
Smaller project limits are useful policy. They cannot widen fixed internal worker, package,
response, or cache ceilings. In particular, detached renderer transfer has its own fixed boundary
even if a compatibility descriptor retains a larger `max_render_bytes`.
## Manual rendering
The optional `render` section declares:
- one confined `template_root`;
- one isolated `preview_root`;
- one or more stable views.
Each view uses the built-in `generic_html` renderer, a template beneath `template_root`, one
declared output path, a title, and a family filter. Paths may not overlap canonical content,
authority files, changesets, cache, templates, or previews in unsafe ways.
Templates are inert UTF-8 files with a fixed token vocabulary. They cannot select executable
renderers or commands. See [Rendering and visualization](RENDERING_AND_VISUALIZATION.md).
## Portable graph rendering
The optional `graph_render` section declares an output root and one or more
`portable_graph_html` views. Each view selects exactly one:
- `root`, an exact stable node ID; or
- `query`, a bounded metadata-only lexical seed.
It may then restrict families, relations, authorities, statuses, and tags, plus depth, node, edge,
and work limits. `initial_mode` is `nodes`, `flow`, or `web`. Portable graph contract version 1
requires `include_logic = false`.
The declared `output` is relative to `graph_render.output_root`.
## Paths and confinement
Descriptor paths are project-relative. Absolute paths and parent traversal are rejected. Runtime
validation also rejects symlink escapes, unexpected file types, unsafe overlap, changing path
identity during sensitive reads or publication, and derived outputs outside their declared roots.
The explicit CLI `--project-root` is the only project selector. The MCP process binds it at startup
and exposes no project-switching tool.
## Validate changes safely
After editing the descriptor:
```bash
docforge --project-root /absolute/path/MyProject validate
docforge --project-root /absolute/path/MyProject reindex
docforge --project-root /absolute/path/MyProject check
```
Descriptor or adapter implementation drift makes a project-owned running process fail closed; start
a fresh process after changing those boundaries.
For first-time creation, prefer the create-only [New-project
quickstart](NEW_PROJECT_QUICKSTART.md). For full invariants, read the [Core contract](CONTRACT.md).

View file

@ -5,6 +5,11 @@ DocForge onboarding has two separate outcomes:
1. A generic manual can be configured, indexed, rendered, visualized, and exposed through the MCP.
2. A source graph additionally requires one validated language frontend per source language.
DocForge ships narrow fixed references for Python, JavaScript, TypeScript, and C++. They are useful
for syntax-scoped projects and adoption proof, but production semantic requirements may still
require a project-owned compiler or language-service adapter. Review
[Reference Adapters](REFERENCE_ADAPTERS.md) before selecting a frontend.
The onboarding command never claims that source semantics exist merely because it found source
files. It reports each detected language as `adapter_required` until a project integration supplies
and proves that frontend.
@ -64,6 +69,32 @@ immediately usable through the generic CLI, viewer, and MCP.
The starter overview records detected languages and states that the source graph is unavailable
until a language frontend passes the adapter proof. That limitation is deliberate.
## Configure a fixed reference adapter
The generic onboarding scaffold and fixed reference configuration are separate project routes.
For a syntax-scoped reference project, create `.docforge/reference-adapter.toml`:
```toml
schema_version = 1
project_id = "my-python-project"
title = "My Python project"
language = "python"
source_roots = ["src"]
```
Then start the fixed read-only binding:
```bash
python -I -m docforge.reference_mcp \
--project-root /absolute/path/MyProject \
--capability-mode read
```
The configuration selects only a fixed in-repository provider and cannot name a command or custom
module. JavaScript, TypeScript, and C++ require their respective optional extras; C++ also requires
`compilation_database = "compile_commands.json"`. The complete configuration and supported-fact
contract are in [Reference Adapters](REFERENCE_ADAPTERS.md).
## Complete onboarding checklist
### 1. Repository assessment
@ -118,6 +149,13 @@ For every source language:
- [ ] Define dependency discovery.
- [ ] State unsupported semantic facts explicitly.
Decide whether the project needs production semantic evidence or the narrower syntax-only
reference scope. The Python reference publishes only project-local imports. JavaScript and
TypeScript publish only project-local static relative imports and re-exports. The C++ reference
publishes only directly resolvable project-local quoted includes and does not run a compiler. None
of those references resolves calls, inheritance, types, symbols, runtime behavior, or semantic
ownership.
All frontends emit the same DocForge contracts:
- `AdapterManifest` inventories fingerprinted extraction units and dependencies.
@ -151,6 +189,11 @@ this checklist summarizes.
- [ ] Assign shared symbols to one deterministic source contribution.
- [ ] Record compiler-derived project include dependencies.
These are production semantic-adapter expectations. The built-in C++ reference uses
`compile_commands.json` only as bounded translation-unit inventory and fingerprint evidence. It
parses commands and arguments as inert data, executes no command or compiler, and does not claim
compiler include semantics, symbol ownership, or a Clang-derived graph.
#### Rust
- [ ] Read the Cargo workspace and package graph.
@ -240,6 +283,10 @@ from filenames.
Done when a new session can identify and retrieve the correct project without being told its file
layout.
The fixed reference server is read-only. Generic and custom-adapter client generation, including
the immutable custom launcher boundary, is documented in
[Agent Integration](AGENT_INTEGRATION.md).
### 10. Operating guide and maintenance
- [ ] Record the authority and progressive-reading order.
@ -286,14 +333,18 @@ to one configured project root.
DocForge does not let an MCP call install dependencies, run project builds, modify Git, deploy, or
publish. A project integration may use its own normal development workflow for those actions.
## Frontend packaging direction
## Reference and production frontend boundaries
Reusable language frontends should be separate packages or project-owned adapters over the public
DocForge contracts. They must not put language-specific rules into the graph, index, viewer, or MCP
core.
The public authoring namespace is `docforge.adapter_sdk`. Project-owned and separately distributed
production frontends should build on that contract without putting language-specific rules into
the graph, index, viewer, or MCP core.
Worldforge is the first complete C++ reference integration. A reusable C++ package should be
extracted only after that integration proves stable ownership, compiler dependency invalidation,
and complete/incremental equivalence. Rust and Java frontends should then implement the same
contract using their authoritative build and language tooling rather than copying C++ extraction
rules.
The in-repository Python, JavaScript, TypeScript, and C++ adapters are reference implementations.
Their syntax-scoped behavior is useful without becoming a claim that every project in those
languages has complete semantic coverage. A compiler-backed production C++ adapter may resolve
build flags, calls, types, inheritance, include semantics, and ownership when it can prove those
facts. The reference C++ adapter does none of that and is not a Clang semantic adapter.
New Rust, Java, or other frontends should use their authoritative build and language tooling and
must pass the graph-plus-Logic complete/incremental contract in the
[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md).

View file

@ -0,0 +1,86 @@
# Recovery and performance
DocForge keeps canonical project evidence separate from disposable indexes, caches, receipts, and
rendered projections. Recovery rebuilds derived state from current authority; it does not rewrite
canonical content to make a cache look valid.
For the underlying boundaries, see [core authority](CORE_CONCEPTS_AND_AUTHORITY.md) and the
[security model](SECURITY.md).
## Recovery order
Use the narrowest verified recovery:
1. Run `docforge check` or the corresponding MCP status operation.
2. Run `docforge sync` to repair a missing or stale disposable index when safe.
3. Run `docforge reindex` for an explicit complete rebuild.
4. Restart a long-running MCP or viewer binding after descriptor or adapter implementation drift.
5. Regenerate a derived client fragment, command reference, preview, manual, or portable graph
from current project evidence.
Never delete or rewrite canonical sources, active changesets, or reviewed hashes as cache cleanup.
## Extraction-cache recovery
Incremental adapter contributions are stored in a version-1 disposable extraction cache. Reads are
no-follow, regular-file-only, identity-checked, bounded to 64,000,000 bytes and 10,000 source
records, and fail to a cache miss on malformed or incompatible data. Publication is atomic.
After a miss, current manifest sources are extracted again and the complete assembly is validated.
The active SQLite generation remains authoritative for reads until a new verified index is
published. An extraction cache may therefore be safely ahead of the last index; the two files do
not pretend to be one transaction.
## Index and receipt recovery
A validated SQLite index is a generation-pinned derived snapshot. Missing, corrupt, unattested, or
stale indexes rebuild from the complete project or adapter oracle. Generation receipts and render
receipts are post-publication evidence. Failure to write a receipt after a committed artifact is
reported as degraded committed success, not as permission to repeat a mutation.
The live viewer pins one validated index identity. Index replacement makes the running snapshot
stale and causes a later visualize request to start a fresh worker.
## Proposal and application recovery
Hash or base conflicts are not cache failures. Retrieve the current changeset and diff, then
review the new exact hash. Rebase is allowed only when every touched node, relationship, source,
permission, and graph invariant still matches. A content conflict requires a new proposal.
If a canonical serializer fails its round-trip check, use its reported rollback state. Do not
reapply a changeset whose application may already have committed. See
[migrating from v1](MIGRATING_FROM_V1.md) for rollback planning.
## Milestone 4 scale evidence
The maintained Python reference benchmark creates 334 Python source files and produces:
- 1,002 primary nodes and 1,001 primary edges.
- 334 Logic projections with 2,338 Logic nodes and 2,338 Logic edges.
- Exact complete/incremental primary graph and Logic equality.
- Exact output after corrupt extraction-cache recovery and corrupt-index recovery.
- Zero `ast.parse` calls and zero `extract_source` calls during a warm build.
On the frozen Milestone 4 candidate, operation p95 values were 1.266 to 1.898 seconds. Per-operation
traced peaks were about 61 to 67 MiB, and process high-water was 78,798,848 bytes. These are
regression measurements from one machine, not universal latency promises. The machine-readable
record is `../benchmarks/milestone4-2026-07-29.json`; detailed method and hashes are in
[the Milestone 4 baseline](MILESTONE_4_BASELINE.md).
The zero-parser claim is deliberately Python-only. Focused JavaScript and TypeScript tests prove
their manifests avoid Tree-sitter. The C++ reference manifest currently uses Tree-sitter while
discovering bounded quoted includes, so a warm C++ cache hit is not evidence of zero parser work.
## Maintained gates
Run:
```bash
make gate
make adoption-m4
make benchmark-m4-full
make docs-check
```
The main gate includes smoke benchmarks. Full milestone evidence is recorded separately from a
clean candidate so smoke or dirty-tree results cannot become release claims.

149
docs/REFERENCE_ADAPTERS.md Normal file
View file

@ -0,0 +1,149 @@
# Reference adapters
DocForge includes fixed, syntax-scoped reference adapters for Python, JavaScript, TypeScript, and
C++. They demonstrate the public adapter contract, deterministic complete and incremental
publication, function Logic, cache recovery, and a runnable read-only MCP binding. They are not
compiler or language-service replacements.
Use a reference adapter when its deliberately narrow graph is sufficient or when proving a fresh
DocForge integration. Use the [Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md) for a
production adapter that needs resolved symbols, calls, inheritance, types, build semantics, or
semantic ownership.
## Installation
The Python reference adapter uses the standard library and is available in the base wheel. The
other frontends are separate optional extras:
```bash
python -m pip install docforge
python -m pip install "docforge[javascript]"
python -m pip install "docforge[typescript]"
python -m pip install "docforge[cpp]"
```
Install `docforge[languages]` only when one environment intentionally needs all three Tree-sitter
frontends. JavaScript and TypeScript use distinct grammar packages and distinct extras.
## Fixed project configuration
The runnable reference binding reads exactly
`.docforge/reference-adapter.toml` below the selected project root. The configuration is data only:
it cannot name a provider, Python module, command, arguments, working directory, environment, or
discovery rule.
Reference adapter configuration `.docforge/reference-adapter.toml`:
```toml
schema_version = 1
project_id = "example-python"
title = "Example Python project"
language = "python"
source_roots = ["src"]
```
`language` is one of `python`, `javascript`, `typescript`, or `cpp`. Source roots must be sorted,
unique, non-overlapping project-relative directories outside `.docforge`. The configuration,
roots, and inventoried source files must be regular confined paths without symlink traversal. The
configuration is limited to 65,536 bytes, at most 64 source roots, and 65,536 examined inventory
entries.
C++ additionally requires one explicit project-confined compilation database:
```toml
schema_version = 1
project_id = "example-cpp"
title = "Example C++ project"
language = "cpp"
source_roots = ["include", "src"]
compilation_database = "compile_commands.json"
```
`compilation_database` is forbidden for the other three languages.
## Published scope
All four adapters publish deterministic file, module or translation-unit, class or struct, and
function nodes where their syntax supports those categories. They publish lexical `contains`
relationships, a narrow set of project-local `depends_on` relationships, and function-scoped
Logic. Calls that appear inside Logic are syntax steps, not resolved symbol relationships.
### Python
Python uses `ast` from the standard library for source extraction and Logic. Manifest construction
does not build a Python AST; it tokenizes imports only far enough to publish dependencies that
resolve to another module in the declared source inventory.
It does not import or execute project code. It does not resolve dynamic imports, calls,
inheritance, imported symbols, types, overloads, re-exports, decorators, metaclasses, descriptors,
or runtime-generated behavior.
### JavaScript and TypeScript
JavaScript uses the optional `tree-sitter-javascript` grammar. TypeScript uses the distinct
optional `tree-sitter-typescript` grammar. Their focused incremental tests prove that manifest
construction and an unchanged warm build do not invoke the Tree-sitter extraction parser.
Only static relative imports and re-exports that resolve to another inventoried source file become
dependencies. Dynamic `import()`, `require()`, bare package specifiers, aliases, `tsconfig` paths,
loader hooks, types, interfaces, overloads, calls, inheritance, symbols, and runtime behavior are
not resolved. The adapters never import, compile, transpile, or execute project code.
### C++
C++ treats `compile_commands.json` as the authoritative bounded translation-unit inventory and as
fingerprint evidence. Commands, arguments, directories, and output fields are parsed as inert
data. The adapter executes no compiler, build tool, command, project binary, or project code.
The reference C++ manifest uses the optional `tree-sitter-cpp` grammar to parse inventoried sources
and discover quoted includes. It publishes a dependency only when that quoted include resolves
directly to a real project-local header in the declared roots. Angle-bracket includes, compiler
include paths, frameworks, generated headers, compiler-provided headers, conditional compilation,
and macro expansion are omitted.
This is not a Clang semantic adapter. It does not claim resolved calls, types, templates, aliases,
concepts, references, inheritance, overload ownership, out-of-line semantic ownership, macro
semantics, or compiler include semantics. A warm C++ extraction-cache hit is therefore not evidence
that manifest construction performed zero parser work.
## Complete and incremental proof
Each reference adapter implements:
- `load_projection()` for the complete primary graph compatibility oracle;
- `load_complete_assembly()` for the complete graph-plus-Logic oracle;
- `load_manifest()` and `extract_source()` for incremental extraction; and
- deterministic assembly of the complete current contribution set.
The maintained fixtures prove complete determinism, exact complete/incremental graph and Logic
parity, reverse-dependency invalidation, additions and deletions, corrupt extraction-cache
recovery, confinement, and exact unsupported-fact inventories. See
[Incremental Adapter Indexing](INCREMENTAL_INDEXING.md) for the cache contract and
[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for compatibility and policy limits.
Current fixture evidence is:
| Language | Primary nodes | Relationships | Logic projections |
|---|---:|---:|---:|
| Python | 14 | 13 | 5 |
| JavaScript | 14 | 14 | 5 |
| TypeScript | 13 | 14 | 4 |
| C++ | 17 | 16 | 5 |
These are regression-fixture shapes, not promises for arbitrary repositories.
## Read-only reference server
Start the fixed server with:
```bash
python -I -m docforge.reference_mcp \
--project-root /absolute/path/to/project \
--capability-mode read
```
The server selects one of the four in-repository providers solely from the validated fixed
configuration. Its cache stays below `.docforge/cache/reference-adapter/<language>`. It registers
the 21-tool read surface and no proposal or application tools. See
[Agent Integration](AGENT_INTEGRATION.md) for generated client fragments and
[MCP Boundary](MCP_CONTRACT.md) for the exact tool contract.

View file

@ -0,0 +1,233 @@
# Rendering and visualization
DocForge has three independent ways to present one validated graph generation:
```text
validated generation
├── manual plan → immutable package → detached HTML renderer → declared manual output
├── graph plan → immutable package → detached graph renderer → portable static artifact
└── pinned index → managed loopback viewer → interactive Nodes, Flow, Web, and lazy Logic
```
They share validated facts but not authority, publication, or lifecycle. A manual render does not
publish a portable graph. A portable graph does not start the live viewer. None is canonical
project content.
## Manual rendering
A generic descriptor may declare a manual template root, isolated preview root, and stable views:
```toml
[render]
template_root = ".docforge/templates"
preview_root = ".docforge/previews"
[[render.views]]
id = "manual"
renderer = "generic_html"
template = "manual.html"
output = ".docforge/rendered/manual.html"
title = "My Project Manual"
families = ["architecture", "operations", "system"]
```
The renderer name is fixed to `generic_html`. Templates are confined UTF-8 assets with a fixed
token vocabulary. They cannot name commands, Python modules, executable renderers, or arbitrary
publication paths. Raw HTML in canonical Markdown is disabled by the pinned CommonMark path.
Plan and render identity covers the canonical source hash, optional changeset hash, selected nodes
and relationships, view configuration, template hash, renderer contract, and parser version.
Use:
```bash
docforge --project-root "$PROJECT" render-status
docforge --project-root "$PROJECT" render-status manual --deep
docforge --project-root "$PROJECT" render manual
docforge --project-root "$PROJECT" preview CHANGESET_ID manual
```
Normal status verifies bounded source, configuration, template, output, renderer, and receipt
identities without reconstructing output. `--deep` explicitly runs the side-effect-free full-render
oracle. A changeset preview writes only to the isolated preview root.
Manual `auto` means that a successful canonical application owns regeneration of every declared
manual view. It does not mean background rendering, and ordinary standalone CLI render cannot
select `auto`; use `explicit`.
## Portable graph rendering
Portable graph configuration is separate:
```toml
[graph_render]
output_root = ".docforge/portable-graph"
[[graph_render.views]]
id = "architecture"
renderer = "portable_graph_html"
output = "architecture.html"
title = "Architecture"
root = "architecture.overview"
initial_mode = "web"
depth = 3
max_nodes = 250
max_edges = 1000
max_work = 100000
families = ["architecture", "system"]
relations = ["depends_on", "owns", "calls", "reads", "writes", "tested_by", "relates_to"]
authorities = []
statuses = ["current", "active", "verified"]
tags = []
include_logic = false
```
A view selects exactly one exact `root` node or bounded metadata-only lexical `query`. Exact
filters and node, edge, depth, and work limits close the selection. The initial mode is `nodes`,
`flow`, or `web`. Portable graph contract version 1 excludes function-scoped Logic.
Use:
```bash
docforge --project-root "$PROJECT" graph-plan architecture
docforge --project-root "$PROJECT" graph-render architecture
docforge --project-root "$PROJECT" graph-render-status architecture
```
`graph-plan` validates and returns the generation-pinned plan without publication. `graph-render`
is an explicit local CLI publication action. It commits a content-addressed artifact, renderer
receipt, and bounded generation/view manifest in that order; the manifest is the publication
commit. `graph-render-status` checks only bounded committed evidence and never plans or renders.
MCP may plan and inspect portable graph evidence, but it does not publish the portable artifact.
Use the explicit project-local CLI command for publication.
Portable output is a complete static artifact. JavaScript is progressive enhancement, not a
requirement for the graph facts to be present.
## Live viewer
The live viewer is a managed, read-only browser over one generation-pinned validated SQLite file.
Install the per-user manager once:
```bash
docforge-viewer-manager install-user-service
```
Then operate a project viewer:
```bash
docforge --project-root "$PROJECT" visualize
docforge --project-root "$PROJECT" visualize --node architecture.overview
docforge --project-root "$PROJECT" visualize --query persistence
docforge --project-root "$PROJECT" visualization-status
docforge --project-root "$PROJECT" visualization-stop
```
`visualize` accepts only an optional stable node ID or lexical query, bounded traversal depth, and
the local `--no-open` presentation choice. It does not accept a project root override, database
path, SQL, template, command, renderer, bind address, or module.
The HTTP listener binds to `127.0.0.1` on an operating-system-selected port. A random token is part
of every accepted path. The server supports only `GET` and `HEAD`, sets no-store and restrictive
browser security headers, and has no write, project-selection, arbitrary-query, or static
filesystem endpoint.
The viewer is a snapshot. Index replacement or alteration makes that snapshot fail closed; start a
new visualization to use a new validated generation. Source inspection reads only project-confined
source evidence bound to the pinned generation.
## Nodes, Flow, Web, and Logic
The interactive viewer offers four complementary projections:
- **Nodes** shows a bounded relation-neutral incoming and outgoing neighborhood.
- **Flow** presents semantic contributors toward the focus. Prerequisite-style stored
relationships may be reversed for presentation without changing stored direction.
- **Web** expands the convergence picture with contributors, callers, containers, members, and
contextual relationships.
- **Logic** loads a function-scoped control-flow projection only when requested.
Logic supports the explicit control paths published by Python, JavaScript, TypeScript, and C++
integrations. It stays outside the primary graph, portable graph version 1, search, and
generation-diff receipts.
Hiding a node is presentation-only. In Flow and Web, ancestors without another path to the focus
are removed. In Logic, an omitted-path bridge preserves downstream readability. Restore reverses
the presentation change; neither action mutates the index.
## Immutable plan and worker boundary
Manual and graph plans are versioned, canonical JSON with deterministic ordering and fixed
structural and byte limits. They contain selected graph facts and bounded content, but no live
project object, SQLite handle, absolute project or index path, arbitrary query, command,
executable path, or caller-selected renderer module.
A projection package binds one plan to inert assets, fixed component versions, a closed built-in
renderer identity, and an exact artifact inventory. The detached worker:
- runs one fixed private module through isolated Python;
- uses a trusted working directory and sanitized environment;
- accepts one canonical newline-terminated JSON request;
- returns one bounded canonical JSON response;
- has a fixed renderer allowlist and timeout;
- cannot select graph facts, read project state, choose output paths, or mutate canonical files.
The package contract is bounded, and actual artifact transfer has a fixed 20,000,000-byte ceiling.
A larger descriptor `max_render_bytes` compatibility value does not widen that worker boundary.
## Manual fragment reuse
Manual fragments are disposable semantic cache records, not publication authority. A cold record
is accepted only after byte-exact comparison against a full detached render. On a warm hit, the
worker independently recomputes the expected fragment before reuse.
Corrupt, forged, stale, incompatible, individually oversized, or aggregate-oversized records fall
back to the complete full-render oracle. The current cache is bounded to 10,000 records and
64,000,000 bytes.
Fragment reuse is a correctness and recovery boundary. Do not promise a speedup without current
measurements.
## Projection policy
The version-2 projection policy is independent of capability mode:
```text
manual: auto | explicit | disabled
portable_graph: explicit | disabled
live_viewer: on-demand | disabled
```
Set global CLI flags before the subcommand:
```bash
docforge --project-root "$PROJECT" --manual-render-policy disabled render manual
docforge --project-root "$PROJECT" --portable-graph-policy disabled graph-plan architecture
docforge --project-root "$PROJECT" --live-viewer-policy disabled visualize
```
An active operation blocked by policy fails before hidden work. Receipt-only manual and portable
status remain available. Viewer status and explicit stop remain available when viewer startup is
disabled.
A non-disabled selection also requires its resource: declared manual configuration, declared
portable graph configuration, or viewer runtime. Manual `auto` additionally requires canonical
application. Read [Policy precedence](POLICY_PRECEDENCE.md) for exact composition.
## Failure and recovery
Render input drift detected before replacement fails without publishing a current receipt for stale
output. Portable graph publication uses content-addressed evidence and a final manifest commit.
Status never repairs implicitly.
Recovery is explicit:
- rerun a manual render from validated canonical state;
- use deep manual status for the full, side-effect-free equivalence oracle;
- rerun portable graph publication, or repair only from validated content-addressed evidence;
- stop and restart a stale live viewer against a current validated index.
Do not recover a derived output by editing its receipt or treating it as canonical. See [Recovery
and performance](RECOVERY_AND_PERFORMANCE.md), [Security](SECURITY.md), and the [Viewer
manager](VIEWER_MANAGER.md).

101
docs/SECURITY.md Normal file
View file

@ -0,0 +1,101 @@
# Security model
DocForge is a project-bound knowledge compiler. Its security boundary is an explicit project root,
closed configuration, bounded data, and exact identities. It is not a general process sandbox.
Start with [core authority](CORE_CONCEPTS_AND_AUTHORITY.md), then use
[policy precedence](POLICY_PRECEDENCE.md) to decide which capabilities a server should expose.
## Project and path confinement
Descriptors, reference-adapter configurations, canonical sources, authority files, templates,
changesets, caches, indexes, previews, and declared outputs are resolved against one project root.
DocForge rejects absolute paths where only project-relative paths are allowed, parent traversal,
symbolic-link escapes, unsafe file types, and protected-root overlap. Important reads use
no-follow file descriptors and compare file identity before and after reading.
Confinement protects DocForge operations. It does not stop another process with repository access
from changing files. Long-running bindings revalidate descriptor and adapter implementation
identity and require a restart after drift.
## Untrusted project content
Documentation, source text, templates, adapter metadata, compiler-database entries, and changeset
content are data. They cannot redefine policy or instruct DocForge to execute a command. Generic
manual rendering disables raw HTML, accepts a fixed template vocabulary, and rejects script-like
content. Portable graph and manual workers accept validated inert packages and fixed built-in
renderer identities.
The C++ reference adapter reads `compile_commands.json` only as bounded translation-unit inventory
and fingerprint evidence. It never executes the recorded command, compiler, response file, or
project program.
## Adapter launcher boundary
`AdapterLauncherV1` contains one Python module name and project identity. It contains no command,
shell string, arbitrary argument list, working directory, environment, discovery rule, or callable
selector.
Custom modules must be installed top-level modules. An isolated `python -I` probe resolves the
module without importing it and requires its regular-file origin to remain inside the bound
project. The sole trusted dotted exception is the packaged `docforge.reference_mcp` module. Client
fragments use the exact validated interpreter and canonical fixed arguments with an empty
environment.
This proves that the declared module is resolvable and project-bound. It does not make arbitrary
module code safe. Project owners remain responsible for the implementation they install.
## Mutation boundary
Normal MCP and the fixed reference server are read-only. Proposal tools exist only when a
startup-bound writer is authorized by the descriptor. Canonical application exists only when a
matching applier is explicitly configured.
Every proposal append, rebase, abandonment, and application is hash-bound. Application requires
the exact changeset hash that was reviewed. Source identity, content hashes, permissions,
conflicts, graph validity, and serializer round trips are checked before success. DocForge never
turns prose approval into a fuzzy merge.
## Derived state and publication
SQLite indexes, source-generation receipts, extraction caches, render fragments, previews, and
portable artifacts are disposable. Corrupt, stale, foreign, oversized, or mismatched derived
state is rejected or rebuilt from current project evidence.
Generated command-reference publication serializes cooperating writers and uses no-clobber or
compare-and-swap publication. A raced target is restored or retained for recovery instead of being
silently discarded. Projection publication records when an artifact was committed but later
receipt verification degraded, so a completed mutation is never reported as an ordinary failure.
## Limits and denial-of-service resistance
Inputs, results, traversal, context, changesets, renders, worker protocols, manifests, adapter
assemblies, and extraction caches have explicit count and byte limits. Incremental extraction
caches are capped at 10,000 sources and 64,000,000 bytes. Adapter primary nodes use the project
`max_nodes` limit; edges and Logic have deterministic multipliers over that limit.
Limits reduce accidental and adversarial amplification. An in-process adapter can still allocate
memory before returning data, so only trusted project-owned adapter code should run in the server
process.
## Secrets and network behavior
DocForge does not copy the parent environment into generated client fragments or detached
projection workers. Doctor checks never return environment values. The live viewer binds to
loopback, uses an unguessable URL token, supports read-only methods, and serves no arbitrary
filesystem tree.
Project secrets must not be placed in canonical documentation, adapter configuration, compiler
databases, templates, or changesets. Repository release gates include secret scanning, but that
scan is not a substitute for credential hygiene.
## No-AST boundary
`--no-ast` is a binding policy that preserves the selected adapter and prohibits Logic publication
and retrieval. It is not a parser detector, filesystem sandbox, or promise that unrelated
processes cannot parse source. See [legacy and no-AST operation](LEGACY_AND_NO_AST.md).
## Reporting and recovery
Do not bypass a confinement, identity, policy, hash, or limit error. Preserve the failing evidence,
stop the affected binding, and follow [recovery and performance](RECOVERY_AND_PERFORMANCE.md).

View file

@ -35,6 +35,8 @@ incremental methods while retaining the full loader as a fallback.
relationship keys, source inspection, branch-aware node hiding, panel resizing, zooming, and
managed idle shutdown.
- A generic Markdown/TOML adapter plus contracts for deterministic project-owned adapters.
- A public adapter SDK, optional Python/JavaScript/TypeScript/C++ reference integrations, and one
fixed read-only reference MCP binding.
DocForge does not run shell commands from documentation, mutate Git, build an application, deploy,
publish, choose a project globally, or cross project boundaries.
@ -100,6 +102,49 @@ uv run pytest -q
Use the executables under `/absolute/path/DocForge/.venv/bin/` when DocForge is not installed into
the active shell environment.
For an installed distribution, choose only the language extras the project needs:
```bash
python -m pip install docforge
python -m pip install 'docforge[javascript]'
python -m pip install 'docforge[typescript]'
python -m pip install 'docforge[cpp]'
```
The base wheel contains the Python reference adapter and no Tree-sitter distribution. JavaScript,
TypeScript, and C++ require their matching optional extras. `docforge[languages]` installs all
three optional frontend groups.
### Configure a reference source project
Reference adapters are a narrow alternative to the generic documentation descriptor. Create
`.docforge/reference-adapter.toml`:
```toml reference-adapter
schema_version = 1
project_id = "my-python-project"
title = "My Python Project"
language = "python"
source_roots = ["src"]
```
Start the fixed read-only server:
```bash
python -I -m docforge.reference_mcp \
--project-root /absolute/path/MyProject \
--capability-mode read
```
For C++, set `language = "cpp"` and add a project-relative
`compilation_database = "compile_commands.json"`. The database is inert bounded inventory; the
reference adapter does not execute its commands or compiler.
The reference integrations publish syntax and local static relationships only. They do not claim
resolved calls, types, inheritance, macro behavior, compiler include semantics, runtime behavior,
or semantic ownership. See [reference adapters](REFERENCE_ADAPTERS.md) for exact evidence and
limitations.
### Assess and onboard an unconfigured project
Run a read-only assessment before writing configuration:
@ -478,6 +523,10 @@ Every command emits deterministic JSON:
docforge --project-root /absolute/path/MyProject <command>
```
The sections below group common workflows. The implementation-derived list of all 28 current
commands, exact invocations, 36 generic MCP tools, arguments, and input-schema hashes is the
[generated command reference](COMMAND_REFERENCE.md).
### Project and index commands
```text
@ -590,6 +639,31 @@ declares the named writer, and application requires the same writer/applier iden
`--no-ast` to preserve the no-AST binding. Generic CLI generation refuses project-owned adapters
because it cannot safely reconstruct their composition.
A custom adapter owner supplies the already constructed project and immutable launcher through the
Python API:
```python
from docforge.adapter_launcher import AdapterLauncherV1
from docforge.client_config import generate_adapter_client_configuration
launcher = AdapterLauncherV1.for_project(
project,
module="my_project_docforge",
)
fragment = generate_adapter_client_configuration(
project,
launcher,
"codex",
capability_mode="read",
)
```
The top-level module must be installed for the exact isolated Python environment and resolve to a
regular file inside the project root. The fixed `docforge.reference_mcp` module is the only trusted
dotted exception. Generation probes resolution without importing the custom module, binds current
source availability and policy, and emits no arbitrary command, arguments, working directory, or
environment.
Select projection behavior independently:
```bash
@ -684,6 +758,10 @@ docforge-mcp \
Omit `--proposal-writer` when the MCP client should not create or append proposals.
For `.docforge/reference-adapter.toml`, use the fixed `docforge.reference_mcp` command shown in
[setup](#configure-a-reference-source-project). It exposes exactly the 21 read tools and never
registers proposal or application tools.
Select the session's declared surface explicitly when useful:
```bash
@ -986,6 +1064,28 @@ changes outside this process boundary.
## Troubleshooting
### `optional_dependency_missing`
Install the exact extra named in the error into the same Python environment that starts DocForge:
```bash
python -m pip install 'docforge[javascript]'
python -m pip install 'docforge[typescript]'
python -m pip install 'docforge[cpp]'
```
Do not install every frontend merely to suppress the check. A missing optional parser is a closed,
actionable capability error and does not affect base generic or Python reference operation.
### `adapter_launcher_unavailable` or `invalid_adapter_launcher`
Use one installed top-level Python module whose resolved regular-file origin is inside the project
root, or use the fixed `docforge.reference_mcp` binding. Arbitrary dotted modules, packages,
stdlib modules, missing modules, commands, argument strings, working directories, and environment
injection are rejected. Test the exact generated fragment rather than editing its command by hand.
See [agent integration](AGENT_INTEGRATION.md) and the [security model](SECURITY.md).
### `adapter_restart_required`
The project-local adapter code, its declared descriptor, or another implementation file changed
@ -1128,8 +1228,11 @@ make gate
Use `make benchmark` for the historical Milestone 0 baseline, `make benchmark-m1` for the
counter-gated warm-operation benchmark, `make benchmark-m2` for agent workflow gates, and
`make benchmark-m3-full` for the ten-sample 1,000-node projection, worker, fragment, status,
equivalence, response-size, and memory gates. `make accessibility` runs the generated manual,
portable graph, and live viewer axe and keyboard flows.
equivalence, response-size, and memory gates. `make benchmark-m4-full` runs the 1,002-node adapter
and recovery benchmark. `make adoption-m4` performs the offline fresh-wheel proof.
`make command-reference-check` rejects command-reference drift, and `make docs-check` validates
the maintained documentation graph. `make accessibility` runs the generated manual, portable
graph, and live viewer axe and keyboard flows.
Project-specific vocabulary, extraction rules, and serialization belong in the project adapter.
Generic core behavior must remain deterministic, project-bound, and recoverable.

View file

@ -13,9 +13,26 @@ authors = [{ name = "Worldforge contributors" }]
dependencies = [
"markdown-it-py>=4.2,<5",
"mcp>=1.28,<2",
]
[project.optional-dependencies]
javascript = [
"tree-sitter>=0.25,<0.26",
"tree-sitter-javascript>=0.25,<0.26",
]
typescript = [
"tree-sitter>=0.25,<0.26",
"tree-sitter-typescript>=0.23,<0.24",
]
cpp = [
"tree-sitter>=0.25,<0.26",
"tree-sitter-cpp>=0.23,<0.24",
]
languages = [
"tree-sitter>=0.25,<0.26",
"tree-sitter-cpp>=0.23,<0.24",
"tree-sitter-javascript>=0.25,<0.26",
"tree-sitter-typescript>=0.23,<0.24",
]
[dependency-groups]
@ -23,6 +40,10 @@ dev = [
"jsonschema>=4.25,<5",
"pytest>=9.1,<10",
"ruff>=0.15,<1",
"tree-sitter>=0.25,<0.26",
"tree-sitter-cpp>=0.23,<0.24",
"tree-sitter-javascript>=0.25,<0.26",
"tree-sitter-typescript>=0.23,<0.24",
]
[project.scripts]

View file

@ -0,0 +1,358 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://docforge.local/schema/adapter-client-configuration-v1.json",
"title": "DocForge project-owned adapter client configuration",
"$defs": {
"sha256": {
"type": "string",
"pattern": "^[0-9a-f]{64}$"
},
"project": {
"type": "object",
"required": [
"project_id",
"project_root",
"project_root_fingerprint",
"adapter",
"descriptor_hash"
],
"properties": {
"project_id": {
"type": "string",
"pattern": "^[a-z0-9][a-z0-9._-]{1,127}$"
},
"project_root": {
"type": "string",
"minLength": 1,
"maxLength": 4096
},
"project_root_fingerprint": {
"type": "string",
"pattern": "^[0-9a-f]{16}$"
},
"adapter": {
"type": "string",
"pattern": "^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}@[A-Za-z0-9][A-Za-z0-9_.+-]{0,63}$"
},
"descriptor_hash": { "$ref": "#/$defs/sha256" }
},
"additionalProperties": false
},
"source_availability": {
"type": "object",
"required": [
"schema_version",
"status",
"method",
"revision",
"source_hash"
],
"properties": {
"schema_version": { "const": 1 },
"status": { "const": "available" },
"method": {
"enum": ["incremental-state", "complete-projection"]
},
"revision": {
"type": "string",
"minLength": 1,
"maxLength": 4096
},
"source_hash": { "$ref": "#/$defs/sha256" }
},
"additionalProperties": false
},
"binding": {
"type": "object",
"required": [
"transport",
"capability_mode",
"adapter_policy",
"render_policy",
"command",
"args",
"environment",
"timeouts",
"launcher_hash"
],
"properties": {
"transport": { "const": "stdio" },
"capability_mode": {
"enum": ["read", "proposal", "application"]
},
"adapter_policy": {
"$ref": "https://docforge.local/schema/client-configuration-v1.json#/$defs/adapter_policy"
},
"render_policy": {
"type": "object",
"required": ["manual", "graph", "live_viewer"],
"properties": {
"manual": { "enum": ["auto", "explicit", "disabled"] },
"graph": { "const": "disabled" },
"live_viewer": { "const": "on-demand" }
},
"additionalProperties": false
},
"command": {
"type": "string",
"minLength": 1,
"maxLength": 4096
},
"args": {
"type": "array",
"minItems": 7,
"maxItems": 64,
"prefixItems": [
{ "const": "-I" },
{ "const": "-m" },
{
"type": "string",
"minLength": 1,
"maxLength": 255,
"pattern": "^[A-Za-z_][A-Za-z0-9_]*(\\.[A-Za-z_][A-Za-z0-9_]*){0,31}$",
"not": { "const": "docforge.mcp_server" }
},
{ "const": "--project-root" },
{
"type": "string",
"minLength": 1,
"maxLength": 4096
},
{ "const": "--capability-mode" },
{ "enum": ["read", "proposal", "application"] }
],
"items": {
"type": "string",
"minLength": 1,
"maxLength": 4096
}
},
"environment": {
"type": "object",
"maxProperties": 0
},
"timeouts": {
"type": "object",
"required": ["startup_seconds", "tool_seconds"],
"properties": {
"startup_seconds": {
"type": "integer",
"minimum": 1,
"maximum": 3600
},
"tool_seconds": {
"type": "integer",
"minimum": 1,
"maximum": 86400
}
},
"additionalProperties": false
},
"launcher_hash": { "$ref": "#/$defs/sha256" }
},
"additionalProperties": false
},
"projection_availability": {
"type": "object",
"required": [
"manual_configured",
"portable_graph_configured",
"application_enabled",
"live_viewer_available"
],
"properties": {
"manual_configured": { "type": "boolean" },
"portable_graph_configured": { "type": "boolean" },
"application_enabled": { "type": "boolean" },
"live_viewer_available": { "const": true }
},
"additionalProperties": false
}
},
"type": "object",
"required": [
"status",
"schema_version",
"operation",
"action",
"client",
"server_name",
"project",
"launcher",
"launcher_hash",
"source_availability",
"source_availability_hash",
"binding",
"effective_policy",
"projection_policy",
"projection_policy_hash",
"projection_availability",
"artifact",
"configuration_hash",
"warnings"
],
"properties": {
"status": { "const": "ok" },
"schema_version": { "const": 1 },
"operation": { "const": "adapter_client.configure" },
"action": { "enum": ["preview", "write"] },
"client": { "enum": ["codex", "claude", "openclaw"] },
"server_name": {
"type": "string",
"pattern": "^[a-z0-9][a-z0-9_-]{0,63}$"
},
"project": { "$ref": "#/$defs/project" },
"launcher": {
"$ref": "https://docforge.local/schema/adapter-launcher-v1.json"
},
"launcher_hash": { "$ref": "#/$defs/sha256" },
"source_availability": {
"$ref": "#/$defs/source_availability"
},
"source_availability_hash": { "$ref": "#/$defs/sha256" },
"binding": { "$ref": "#/$defs/binding" },
"effective_policy": {
"$ref": "https://docforge.local/schema/client-configuration-v1.json#/$defs/effective_policy"
},
"projection_policy": {
"$ref": "https://docforge.local/schema/client-configuration-v1.json#/$defs/projection_policy"
},
"projection_policy_hash": { "$ref": "#/$defs/sha256" },
"projection_availability": {
"$ref": "#/$defs/projection_availability"
},
"artifact": {
"$ref": "https://docforge.local/schema/client-configuration-v1.json#/properties/artifact"
},
"configuration_hash": { "$ref": "#/$defs/sha256" },
"warnings": {
"$ref": "https://docforge.local/schema/client-configuration-v1.json#/properties/warnings"
}
},
"allOf": [
{
"if": {
"properties": { "client": { "const": "codex" } },
"required": ["client"]
},
"then": {
"properties": {
"artifact": {
"properties": {
"format": { "const": "codex-toml-fragment-v1" }
}
}
}
}
},
{
"if": {
"properties": { "client": { "const": "claude" } },
"required": ["client"]
},
"then": {
"properties": {
"artifact": {
"properties": {
"format": { "const": "claude-json-fragment-v1" }
}
},
"warnings": {
"contains": {
"properties": {
"code": { "const": "timeout_format_unverified" }
},
"required": ["code"]
}
}
}
}
},
{
"if": {
"properties": { "client": { "const": "openclaw" } },
"required": ["client"]
},
"then": {
"properties": {
"artifact": {
"properties": {
"format": { "const": "openclaw-json-fragment-v1" }
}
}
}
}
},
{
"if": {
"properties": { "action": { "const": "preview" } },
"required": ["action"]
},
"then": {
"properties": {
"artifact": {
"properties": {
"output_path": { "type": "null" },
"write_state": { "const": "not_requested" },
"durability": { "const": "not_applicable" }
}
}
}
},
"else": {
"properties": {
"artifact": {
"properties": {
"write_state": { "enum": ["created", "unchanged"] }
}
}
}
}
},
{
"if": {
"properties": {
"binding": {
"properties": {
"adapter_policy": {
"properties": {
"mode": { "const": "preserve-no-ast" }
},
"required": ["mode"]
}
},
"required": ["adapter_policy"]
}
},
"required": ["binding"]
},
"then": {
"properties": {
"binding": {
"properties": {
"args": {
"contains": { "const": "--no-ast" },
"minContains": 1,
"maxContains": 1
}
}
}
}
},
"else": {
"properties": {
"binding": {
"properties": {
"args": {
"not": {
"contains": { "const": "--no-ast" }
}
}
}
}
}
}
}
],
"additionalProperties": false
}

View file

@ -0,0 +1,46 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://docforge.local/schema/adapter-launcher-v1.json",
"title": "DocForge project-owned adapter launcher",
"type": "object",
"required": [
"schema_version",
"entry_point",
"project_id",
"project_root",
"adapter",
"descriptor_hash",
"module"
],
"properties": {
"schema_version": { "const": 1 },
"entry_point": { "const": "python-module" },
"project_id": {
"type": "string",
"pattern": "^[a-z0-9][a-z0-9._-]{1,127}$"
},
"project_root": {
"type": "string",
"minLength": 1,
"maxLength": 4096
},
"adapter": {
"type": "string",
"pattern": "^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}@[A-Za-z0-9][A-Za-z0-9_.+-]{0,63}$"
},
"descriptor_hash": {
"type": "string",
"pattern": "^[0-9a-f]{64}$"
},
"module": {
"type": "string",
"minLength": 1,
"maxLength": 255,
"anyOf": [
{ "pattern": "^[A-Za-z_][A-Za-z0-9_]*$" },
{ "const": "docforge.reference_mcp" }
]
}
},
"additionalProperties": false
}

View file

@ -0,0 +1,78 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://andraxion.net/docforge/schemas/reference-adapter.schema.json",
"title": "DocForge Reference Adapter Configuration",
"type": "object",
"additionalProperties": false,
"required": [
"schema_version",
"project_id",
"title",
"language",
"source_roots"
],
"properties": {
"schema_version": {
"const": 1
},
"project_id": {
"type": "string",
"pattern": "^[a-z0-9][a-z0-9._-]{1,127}$"
},
"title": {
"type": "string",
"minLength": 1,
"maxLength": 256
},
"language": {
"enum": [
"python",
"javascript",
"typescript",
"cpp"
]
},
"source_roots": {
"type": "array",
"minItems": 1,
"maxItems": 64,
"uniqueItems": true,
"items": {
"type": "string",
"minLength": 1,
"pattern": "^(?!/)(?!.*(?:^|/)\\.\\.(?:/|$)).+$"
}
},
"compilation_database": {
"type": "string",
"minLength": 1,
"pattern": "^(?!/)(?!.*(?:^|/)\\.\\.(?:/|$)).+$"
}
},
"allOf": [
{
"if": {
"properties": {
"language": {
"const": "cpp"
}
},
"required": [
"language"
]
},
"then": {
"required": [
"compilation_database"
]
},
"else": {
"not": {
"required": [
"compilation_database"
]
}
}
}
]
}

View file

@ -21,6 +21,8 @@ from .adapter_validation import (
from .config_validation import ID_PATTERN
from .errors import DocForgeError
from .incremental import (
MAX_EXTRACTION_CACHE_BYTES,
MAX_EXTRACTION_CACHE_SOURCES,
CachedSource,
ExtractionCache,
affected_sources,
@ -40,6 +42,10 @@ from .models import (
)
from .telemetry import increment, stage
MAX_ADAPTER_EDGE_MULTIPLIER = 32
MAX_ADAPTER_LOGIC_NODE_MULTIPLIER = 32
MAX_ADAPTER_LOGIC_EDGE_MULTIPLIER = 64
@dataclass(frozen=True)
class AdapterNode:
@ -147,6 +153,13 @@ class AdapterAssembly:
logic: tuple[LogicProjection, ...] = ()
@runtime_checkable
class CompleteAdapterAssemblyLoader(Protocol):
"""Load one cache-independent complete graph and Logic equivalence oracle."""
def load_complete_assembly(self) -> AdapterAssembly: ...
class AdapterLoader(Protocol):
"""Load one current, deterministic, project-confined adapter projection."""
@ -201,6 +214,14 @@ def _extract_adapter_source(
return loader.extract_source(source)
def _load_complete_adapter_assembly(
loader: CompleteAdapterAssemblyLoader,
) -> AdapterAssembly:
increment("adapter_projection_loads")
with stage("adapter.projection"):
return loader.load_complete_assembly()
@dataclass(frozen=True)
class AdapterImplementation:
"""One confined implementation boundary that must remain stable for a process."""
@ -232,6 +253,38 @@ class _ImplementationSnapshot:
files: tuple[tuple[str, str], ...]
@dataclass(frozen=True)
class AdapterConformanceReport:
"""Stable evidence that one adapter passed its complete and incremental contracts."""
schema_version: int
project_id: str
adapter_id: str
adapter_version: str
revision: str
source_hash: str
assembly_hash: str
node_count: int
edge_count: int
logic_projection_count: int
incremental: bool
def as_dict(self) -> dict[str, object]:
return {
"schema_version": self.schema_version,
"project_id": self.project_id,
"adapter_id": self.adapter_id,
"adapter_version": self.adapter_version,
"revision": self.revision,
"source_hash": self.source_hash,
"assembly_hash": self.assembly_hash,
"node_count": self.node_count,
"edge_count": self.edge_count,
"logic_projection_count": self.logic_projection_count,
"incremental": self.incremental,
}
class AdapterProject:
"""Expose a validated adapter projection through the standard index boundary."""
@ -417,6 +470,9 @@ class AdapterProject:
canonical_sources,
captured,
)
self._enforce_assembly_limits(
AdapterAssembly(projection=projection, logic=self._last_logic)
)
return ProjectSnapshot(
descriptor=self.descriptor,
nodes=projection.core_nodes(),
@ -523,43 +579,57 @@ class AdapterProject:
return self._last_logic
def verify_incremental_equivalence(self) -> dict[str, object]:
"""Prove the incremental and full loader contracts produce the same graph."""
"""Prove incremental graph and Logic output matches an independent complete oracle."""
self.validate_runtime()
if self._incremental_loader is None:
loader = self._incremental_loader
if loader is None:
raise DocForgeError(
"incremental_disabled", "Adapter does not implement incremental extraction"
)
incremental = self._load_incremental()
incremental = self._load_incremental_assembly()
full = _load_adapter_projection(self.loader)
validate_projection(full)
fields = {
"project_id": incremental.project_id == full.project_id,
"adapter_id": incremental.adapter_id == full.adapter_id,
"adapter_version": incremental.adapter_version == full.adapter_version,
"revision": incremental.revision == full.revision,
"source_hash": incremental.source_hash == full.source_hash,
"nodes": incremental.nodes == full.nodes,
"edges": incremental.edges == full.edges,
}
complete = AdapterAssembly(full)
if isinstance(loader, CompleteAdapterAssemblyLoader):
complete = _load_complete_adapter_assembly(loader)
_validate_adapter_assembly(complete)
oracle_fields = _projection_equivalence_fields(complete.projection, full)
oracle_mismatches = [field for field, matches in oracle_fields.items() if not matches]
if oracle_mismatches:
raise DocForgeError(
"complete_oracle_mismatch",
"Complete adapter assembly does not match load_projection()",
fields=oracle_mismatches,
)
elif incremental.logic:
raise DocForgeError(
"complete_logic_oracle_required",
"Incremental Logic publication requires load_complete_assembly()",
)
fields = _assembly_equivalence_fields(incremental, complete)
mismatches = [field for field, matches in fields.items() if not matches]
if mismatches:
raise DocForgeError(
"incremental_mismatch",
"Incremental extraction does not match a full adapter projection",
"Incremental extraction does not match a complete adapter assembly",
fields=mismatches,
)
self.validate_runtime()
return {
"status": "ok",
"project_id": incremental.project_id,
"revision": incremental.revision,
"source_hash": incremental.source_hash,
"node_count": len(incremental.nodes),
"edge_count": len(incremental.edges),
"project_id": incremental.projection.project_id,
"revision": incremental.projection.revision,
"source_hash": incremental.projection.source_hash,
"node_count": len(incremental.projection.nodes),
"edge_count": len(incremental.projection.edges),
"logic_projection_count": len(incremental.logic),
}
def _load_incremental(self) -> AdapterProjection:
return self._load_incremental_assembly().projection
def _load_incremental_assembly(self) -> AdapterAssembly:
loader = self._incremental_loader
if loader is None:
raise DocForgeError("incremental_disabled", "Incremental adapter is not configured")
@ -567,11 +637,14 @@ class AdapterProject:
validate_manifest(manifest)
if manifest.identity() != self._identity:
raise DocForgeError("adapter_changed", "Adapter identity changed during the operation")
self._enforce_manifest_limits(manifest)
cache = load_extraction_cache(
self._cache_path,
project_id=manifest.project_id,
adapter_id=manifest.adapter_id,
adapter_version=manifest.adapter_version,
max_bytes=MAX_EXTRACTION_CACHE_BYTES,
max_sources=MAX_EXTRACTION_CACHE_SOURCES,
)
cached = {source.source_id: source for source in cache.sources} if cache else {}
current = {source.source_id: source for source in manifest.sources}
@ -658,6 +731,7 @@ class AdapterProject:
key=lambda projection: projection.owner_node_id,
)
)
assembly = AdapterAssembly(projection=projection, logic=logic_projections)
identity = (
projection.project_id,
projection.adapter_id,
@ -674,21 +748,8 @@ class AdapterProject:
"invalid_adapter",
"Incremental assembly changed the manifest-bound project identity",
)
validate_projection(projection)
owners = [projection.owner_node_id for projection in logic_projections]
if len(owners) != len(set(owners)):
raise DocForgeError(
"invalid_adapter", "A primary graph node may own only one logic projection"
)
node_ids = {item.node.node_id for item in projection.nodes}
for logic_projection in logic_projections:
validate_logic_projection(logic_projection)
if logic_projection.owner_node_id not in node_ids:
raise DocForgeError(
"invalid_adapter",
"A Logic projection owner must exist in the assembled primary graph",
owner_node_id=logic_projection.owner_node_id,
)
_validate_adapter_assembly(assembly)
self._enforce_assembly_limits(assembly)
stable = loader.load_manifest()
validate_manifest(stable)
if stable != manifest:
@ -704,6 +765,8 @@ class AdapterProject:
adapter_version=manifest.adapter_version,
sources=tuple(cache_records),
),
max_bytes=MAX_EXTRACTION_CACHE_BYTES,
max_sources=MAX_EXTRACTION_CACHE_SOURCES,
)
self._last_logic = logic_projections
self._last_build_report = {
@ -716,7 +779,41 @@ class AdapterProject:
"cache_hit_ids": hits,
"reparsed_source_ids": reparsed,
}
return projection
return assembly
def _enforce_manifest_limits(self, manifest: AdapterManifest) -> None:
if len(manifest.sources) > MAX_EXTRACTION_CACHE_SOURCES:
raise DocForgeError(
"adapter_limit",
"Adapter manifest exceeds the bounded source limit",
maximum=MAX_EXTRACTION_CACHE_SOURCES,
actual=len(manifest.sources),
)
def _enforce_assembly_limits(self, assembly: AdapterAssembly) -> None:
maximum_nodes = self.descriptor.limits.max_nodes
maximum_edges = maximum_nodes * MAX_ADAPTER_EDGE_MULTIPLIER
maximum_logic_nodes = maximum_nodes * MAX_ADAPTER_LOGIC_NODE_MULTIPLIER
maximum_logic_edges = maximum_nodes * MAX_ADAPTER_LOGIC_EDGE_MULTIPLIER
actual_nodes = len(assembly.projection.nodes)
actual_edges = len(assembly.projection.edges)
actual_logic_nodes = sum(len(projection.nodes) for projection in assembly.logic)
actual_logic_edges = sum(len(projection.edges) for projection in assembly.logic)
limits = (
("nodes", actual_nodes, maximum_nodes),
("edges", actual_edges, maximum_edges),
("logic_nodes", actual_logic_nodes, maximum_logic_nodes),
("logic_edges", actual_logic_edges, maximum_logic_edges),
)
for label, actual, maximum in limits:
if actual > maximum:
raise DocForgeError(
"adapter_limit",
f"Adapter assembly exceeds the configured {label.replace('_', ' ')} limit",
kind=label,
maximum=maximum,
actual=actual,
)
def canonical_source_paths(self) -> tuple[Path, ...]:
"""Adapters validate their own source sets before producing a projection."""
@ -1032,6 +1129,141 @@ class AdapterProject:
outputs.add(output)
def _validate_adapter_assembly(assembly: AdapterAssembly) -> None:
validate_projection(assembly.projection)
ordered_logic = tuple(sorted(assembly.logic, key=lambda projection: projection.owner_node_id))
if assembly.logic != ordered_logic:
raise DocForgeError(
"invalid_adapter",
"Complete Logic projections must be deterministically ordered",
)
owners = [projection.owner_node_id for projection in assembly.logic]
if len(owners) != len(set(owners)):
raise DocForgeError(
"invalid_adapter", "A primary graph node may own only one logic projection"
)
node_ids = {item.node.node_id for item in assembly.projection.nodes}
for logic_projection in assembly.logic:
validate_logic_projection(logic_projection)
if logic_projection.owner_node_id not in node_ids:
raise DocForgeError(
"invalid_adapter",
"A Logic projection owner must exist in the assembled primary graph",
owner_node_id=logic_projection.owner_node_id,
)
def _projection_equivalence_fields(
candidate: AdapterProjection,
reference: AdapterProjection,
) -> dict[str, bool]:
return {
"project_id": candidate.project_id == reference.project_id,
"title": candidate.title == reference.title,
"adapter_id": candidate.adapter_id == reference.adapter_id,
"adapter_version": candidate.adapter_version == reference.adapter_version,
"root": candidate.root == reference.root,
"revision": candidate.revision == reference.revision,
"source_hash": candidate.source_hash == reference.source_hash,
"nodes": candidate.nodes == reference.nodes,
"edges": candidate.edges == reference.edges,
}
def _assembly_equivalence_fields(
candidate: AdapterAssembly,
reference: AdapterAssembly,
) -> dict[str, bool]:
return {
**_projection_equivalence_fields(candidate.projection, reference.projection),
"logic": candidate.logic == reference.logic,
}
def _assembly_hash(assembly: AdapterAssembly) -> str:
projection = assembly.projection
payload = {
"projection": {
"project_id": projection.project_id,
"title": projection.title,
"adapter_id": projection.adapter_id,
"adapter_version": projection.adapter_version,
"root": str(projection.root),
"revision": projection.revision,
"source_hash": projection.source_hash,
"nodes": [item.as_dict() for item in projection.nodes],
"edges": [item.as_dict() for item in projection.edges],
},
"logic": [logic_projection.as_dict() for logic_projection in assembly.logic],
}
return hashlib.sha256(
json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()
def _complete_reference_assembly(loader: AdapterLoader) -> AdapterAssembly:
if isinstance(loader, CompleteAdapterAssemblyLoader):
assembly = _load_complete_adapter_assembly(loader)
else:
assembly = AdapterAssembly(_load_adapter_projection(loader))
_validate_adapter_assembly(assembly)
return assembly
def verify_adapter_conformance(
loader: AdapterLoader,
*,
cache_root: Path,
settings: AdapterProjectSettings | None = None,
) -> AdapterConformanceReport:
"""Prove deterministic complete output and, when supported, incremental parity."""
project = AdapterProject(loader, cache_root=cache_root, settings=settings)
project.validate_runtime()
first = _complete_reference_assembly(loader)
second = _complete_reference_assembly(loader)
deterministic_fields = _assembly_equivalence_fields(first, second)
nondeterministic = [field for field, matches in deterministic_fields.items() if not matches]
if nondeterministic:
raise DocForgeError(
"nondeterministic_adapter",
"Repeated complete adapter assemblies do not match",
fields=nondeterministic,
)
if isinstance(loader, CompleteAdapterAssemblyLoader):
graph_oracle = _load_adapter_projection(loader)
validate_projection(graph_oracle)
oracle_fields = _projection_equivalence_fields(first.projection, graph_oracle)
oracle_mismatches = [field for field, matches in oracle_fields.items() if not matches]
if oracle_mismatches:
raise DocForgeError(
"complete_oracle_mismatch",
"Complete adapter assembly does not match load_projection()",
fields=oracle_mismatches,
)
incremental = isinstance(loader, IncrementalAdapterLoader)
if incremental:
project.verify_incremental_equivalence()
project.validate_runtime()
projection = first.projection
return AdapterConformanceReport(
schema_version=1,
project_id=projection.project_id,
adapter_id=projection.adapter_id,
adapter_version=projection.adapter_version,
revision=projection.revision,
source_hash=projection.source_hash,
assembly_hash=_assembly_hash(first),
node_count=len(projection.nodes),
edge_count=len(projection.edges),
logic_projection_count=len(first.logic),
incremental=incremental,
)
@dataclass(frozen=True)
class ShadowArtifact:
"""One named deterministic byte artifact used by a shadow comparison."""

View file

@ -0,0 +1,392 @@
"""Strict, data-only launch contracts for project-owned adapter MCP modules."""
from __future__ import annotations
import json
import re
import stat
import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Literal, cast
from .changeset_contract import document_hash
from .config_validation import ID_PATTERN
from .errors import DocForgeError
from .models import IncrementalStateProject, ProjectService, RuntimeValidatedProject
ADAPTER_LAUNCHER_SCHEMA_VERSION = 1
_PYTHON_MODULE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*){0,31}")
_TOP_LEVEL_PYTHON_MODULE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*")
_ADAPTER_IDENTITY = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}@[A-Za-z0-9][A-Za-z0-9_.+-]{0,63}")
_SHA256 = re.compile(r"[0-9a-f]{64}")
_TRUSTED_DOTTED_MODULE = "docforge.reference_mcp"
_MODULE_PROBE_TIMEOUT_SECONDS = 5
_MAX_MODULE_PROBE_BYTES = 8_192
_MODULE_PROBE = (
"import importlib.util,json,sys;"
"spec=importlib.util.find_spec(sys.argv[1]);"
"result=None if spec is None else {"
"'has_loader':spec.loader is not None,"
"'is_package':spec.submodule_search_locations is not None,"
"'origin':spec.origin};"
"print(json.dumps(result,sort_keys=True,separators=(',',':')))"
)
_run_module_probe = subprocess.run
__all__ = [
"ADAPTER_LAUNCHER_SCHEMA_VERSION",
"AdapterLauncherV1",
"AdapterSourceAvailabilityV1",
"adapter_source_availability",
"validate_adapter_launcher",
]
@dataclass(frozen=True)
class AdapterLauncherV1:
"""One immutable, project-bound Python module launch declaration.
The contract intentionally has no command, shell string, working directory,
environment, arbitrary arguments, discovery rule, or callable selector.
"""
schema_version: int
entry_point: Literal["python-module"]
project_id: str
project_root: Path
adapter: str
descriptor_hash: str
module: str
def __post_init__(self) -> None:
_validate_launcher_fields(self)
@classmethod
def for_project(
cls,
project: ProjectService,
*,
module: str,
) -> AdapterLauncherV1:
"""Bind a structural module entry point to one already constructed adapter project."""
descriptor = project.descriptor
if descriptor.adapter == "generic":
raise DocForgeError(
"adapter_launcher_unavailable",
"Project-owned launcher configuration requires a custom adapter",
)
launcher = cls(
schema_version=ADAPTER_LAUNCHER_SCHEMA_VERSION,
entry_point="python-module",
project_id=descriptor.project_id,
project_root=descriptor.root,
adapter=descriptor.adapter,
descriptor_hash=descriptor.descriptor_hash,
module=module,
)
validate_adapter_launcher(project, launcher)
return launcher
def as_dict(self) -> dict[str, object]:
return {
"schema_version": self.schema_version,
"entry_point": self.entry_point,
"project_id": self.project_id,
"project_root": str(self.project_root),
"adapter": self.adapter,
"descriptor_hash": self.descriptor_hash,
"module": self.module,
}
@property
def launcher_hash(self) -> str:
return document_hash(self.as_dict())
@dataclass(frozen=True)
class AdapterSourceAvailabilityV1:
"""One bounded source identity observed without executing the launcher module."""
schema_version: int
status: Literal["available"]
method: Literal["incremental-state", "complete-projection"]
revision: str
source_hash: str
def __post_init__(self) -> None:
revision = _runtime_value(self.revision)
if (
self.schema_version != 1
or self.status != "available"
or self.method not in {"incremental-state", "complete-projection"}
or not isinstance(revision, str)
or not revision
or _SHA256.fullmatch(self.source_hash) is None
):
raise DocForgeError(
"invalid_adapter_launcher",
"Adapter source availability evidence is invalid",
)
def as_dict(self) -> dict[str, object]:
return {
"schema_version": self.schema_version,
"status": self.status,
"method": self.method,
"revision": self.revision,
"source_hash": self.source_hash,
}
@property
def availability_hash(self) -> str:
return document_hash(self.as_dict())
def validate_adapter_launcher(
project: ProjectService,
launcher: AdapterLauncherV1,
) -> None:
"""Require one launcher to match the exact current adapter project binding."""
_validate_launcher_fields(launcher)
descriptor = project.descriptor
_validate_project_root(descriptor.root)
if descriptor.adapter == "generic":
raise DocForgeError(
"adapter_launcher_unavailable",
"Project-owned launcher configuration requires a custom adapter",
)
if (
launcher.project_id != descriptor.project_id
or launcher.project_root != descriptor.root
or launcher.adapter != descriptor.adapter
or launcher.descriptor_hash != descriptor.descriptor_hash
):
raise DocForgeError(
"adapter_launcher_mismatch",
"Adapter launcher does not match the selected project binding",
project_id=descriptor.project_id,
adapter=descriptor.adapter,
)
if isinstance(project, RuntimeValidatedProject):
project.validate_runtime()
def adapter_source_availability(
project: ProjectService,
launcher: AdapterLauncherV1,
) -> AdapterSourceAvailabilityV1:
"""Capture current source identity without importing or executing the launcher module."""
validate_adapter_launcher(project, launcher)
state = project.incremental_state() if isinstance(project, IncrementalStateProject) else None
if state is not None:
method: Literal["incremental-state", "complete-projection"] = "incremental-state"
revision = state.revision
source_hash = state.source_hash
else:
snapshot = project.load()
descriptor = snapshot.descriptor
if (
descriptor.project_id != launcher.project_id
or descriptor.root != launcher.project_root
or descriptor.adapter != launcher.adapter
or descriptor.descriptor_hash != launcher.descriptor_hash
):
raise DocForgeError(
"adapter_launcher_mismatch",
"Loaded adapter projection drifted from its launcher binding",
)
method = "complete-projection"
revision = snapshot.revision
source_hash = snapshot.source_hash
validate_adapter_launcher(project, launcher)
return AdapterSourceAvailabilityV1(
schema_version=1,
status="available",
method=method,
revision=revision,
source_hash=source_hash,
)
def _validate_launcher_fields(launcher: AdapterLauncherV1) -> None:
if type(launcher.schema_version) is not int or launcher.schema_version != 1:
raise DocForgeError(
"invalid_adapter_launcher",
"Adapter launcher schema version is unsupported",
supported=1,
)
if launcher.entry_point != "python-module":
raise DocForgeError(
"invalid_adapter_launcher",
"Adapter launcher entry point must be one isolated Python module",
)
project_id = _runtime_value(launcher.project_id)
project_root = _runtime_value(launcher.project_root)
adapter = _runtime_value(launcher.adapter)
descriptor_hash = _runtime_value(launcher.descriptor_hash)
module = _runtime_value(launcher.module)
if not isinstance(project_id, str) or ID_PATTERN.fullmatch(project_id) is None:
raise DocForgeError("invalid_adapter_launcher", "Adapter launcher project ID is invalid")
if not isinstance(project_root, Path):
raise DocForgeError("invalid_adapter_launcher", "Adapter launcher project root is invalid")
_validate_project_root(project_root)
if not isinstance(adapter, str) or _ADAPTER_IDENTITY.fullmatch(adapter) is None:
raise DocForgeError("invalid_adapter_launcher", "Adapter launcher identity is invalid")
if not isinstance(descriptor_hash, str) or _SHA256.fullmatch(descriptor_hash) is None:
raise DocForgeError("invalid_adapter_launcher", "Adapter descriptor hash is invalid")
if (
not isinstance(module, str)
or len(module) > 255
or _PYTHON_MODULE.fullmatch(module) is None
or module == "docforge.mcp_server"
or (module != _TRUSTED_DOTTED_MODULE and _TOP_LEVEL_PYTHON_MODULE.fullmatch(module) is None)
):
raise DocForgeError(
"invalid_adapter_launcher",
(
"Adapter launcher module must be one installed project-owned top-level "
"Python module or the fixed DocForge reference binding"
),
)
_validate_isolated_module(project_root, module)
def _validate_isolated_module(project_root: Path, module: str) -> None:
"""Prove ``python -I -m`` can resolve one module without importing project code."""
executable = Path(sys.executable)
try:
executable_status = executable.stat()
except OSError as error:
raise DocForgeError(
"adapter_launcher_unavailable",
"The isolated Python executable is unavailable",
) from error
if not stat.S_ISREG(executable_status.st_mode):
raise DocForgeError(
"adapter_launcher_unavailable",
"The isolated Python executable is not a regular file",
)
try:
completed = _run_module_probe(
[str(executable), "-I", "-c", _MODULE_PROBE, module],
capture_output=True,
text=True,
timeout=_MODULE_PROBE_TIMEOUT_SECONDS,
check=False,
)
except (OSError, subprocess.TimeoutExpired) as error:
raise DocForgeError(
"adapter_launcher_unavailable",
"The isolated adapter launcher probe could not complete",
module=module,
) from error
if (
completed.returncode != 0
or len(completed.stdout.encode("utf-8")) > _MAX_MODULE_PROBE_BYTES
or len(completed.stderr.encode("utf-8")) > _MAX_MODULE_PROBE_BYTES
):
raise DocForgeError(
"adapter_launcher_unavailable",
"The isolated adapter launcher module could not be resolved",
module=module,
)
try:
result: object = json.loads(completed.stdout)
except (UnicodeError, json.JSONDecodeError) as error:
raise DocForgeError(
"adapter_launcher_unavailable",
"The isolated adapter launcher probe returned invalid evidence",
module=module,
) from error
if result is None:
raise DocForgeError(
"adapter_launcher_unavailable",
"The isolated adapter launcher module is not installed",
module=module,
)
if not isinstance(result, dict):
raise DocForgeError(
"adapter_launcher_unavailable",
"The isolated adapter launcher probe returned invalid evidence",
module=module,
)
evidence = cast(dict[object, object], result)
if (
set(evidence) != {"has_loader", "is_package", "origin"}
or evidence["has_loader"] is not True
or evidence["is_package"] is not False
or not isinstance(evidence["origin"], str)
):
raise DocForgeError(
"invalid_adapter_launcher",
"The isolated adapter launcher must resolve to one executable module file",
module=module,
)
origin = Path(evidence["origin"])
try:
origin_status = origin.lstat()
resolved_origin = origin.resolve(strict=True)
except OSError as error:
raise DocForgeError(
"adapter_launcher_unavailable",
"The isolated adapter launcher module origin is unavailable",
module=module,
) from error
if (
not origin.is_absolute()
or stat.S_ISLNK(origin_status.st_mode)
or not stat.S_ISREG(origin_status.st_mode)
or resolved_origin != origin
or origin.suffix != ".py"
):
raise DocForgeError(
"invalid_adapter_launcher",
"The isolated adapter launcher must be one canonical Python module file",
module=module,
)
if module == _TRUSTED_DOTTED_MODULE:
expected = Path(__file__).with_name("reference_mcp.py").resolve(strict=True)
if origin != expected:
raise DocForgeError(
"adapter_launcher_mismatch",
"The fixed reference launcher resolved outside this DocForge installation",
module=module,
)
elif not origin.is_relative_to(project_root):
raise DocForgeError(
"invalid_adapter_launcher",
"The installed adapter launcher module is not owned by the selected project",
module=module,
)
def _validate_project_root(root: Path) -> None:
try:
status = root.lstat()
resolved = root.resolve(strict=True)
except OSError as error:
raise DocForgeError(
"invalid_adapter_launcher",
"Adapter launcher project root is unavailable",
) from error
if (
not root.is_absolute()
or stat.S_ISLNK(status.st_mode)
or not stat.S_ISDIR(status.st_mode)
or resolved != root
):
raise DocForgeError(
"invalid_adapter_launcher",
"Adapter launcher project root must be one canonical real directory",
)
def _runtime_value(value: object) -> object:
"""Keep runtime validation explicit even when static callers are typed."""
return value

View file

@ -0,0 +1,69 @@
"""Public typed surface for project adapter implementations and conformance checks."""
from .adapter_contract import (
AdapterAssembly,
AdapterConformanceReport,
AdapterEdge,
AdapterImplementation,
AdapterLoader,
AdapterManifest,
AdapterNode,
AdapterProject,
AdapterProjection,
AdapterProjectSettings,
AdapterSource,
AdapterSourceProjection,
CompleteAdapterAssemblyLoader,
IncrementalAdapterAssembler,
IncrementalAdapterLoader,
verify_adapter_conformance,
)
from .adapter_validation import (
validate_logic_projection,
validate_manifest,
validate_projection,
validate_source_projection,
)
from .models import (
Edge,
Limits,
LogicEdge,
LogicNode,
LogicProjection,
Node,
ProposalWriter,
RenderConfig,
RenderView,
)
__all__ = [
"AdapterAssembly",
"AdapterConformanceReport",
"AdapterEdge",
"AdapterImplementation",
"AdapterLoader",
"AdapterManifest",
"AdapterNode",
"AdapterProject",
"AdapterProjection",
"AdapterProjectSettings",
"AdapterSource",
"AdapterSourceProjection",
"CompleteAdapterAssemblyLoader",
"Edge",
"IncrementalAdapterAssembler",
"IncrementalAdapterLoader",
"Limits",
"LogicEdge",
"LogicNode",
"LogicProjection",
"Node",
"ProposalWriter",
"RenderConfig",
"RenderView",
"validate_logic_projection",
"validate_manifest",
"validate_projection",
"validate_source_projection",
"verify_adapter_conformance",
]

View file

@ -0,0 +1,25 @@
"""Repository-owned reference adapters built on the public adapter SDK."""
from .javascript import (
JAVASCRIPT_ADAPTER_VERSION,
JAVASCRIPT_EXTRACTOR_VERSION,
JavaScriptReferenceAdapter,
JavaScriptUnsupportedFact,
)
from .python import (
PYTHON_ADAPTER_VERSION,
PYTHON_EXTRACTOR_VERSION,
PythonReferenceAdapter,
PythonUnsupportedFact,
)
__all__ = [
"JAVASCRIPT_ADAPTER_VERSION",
"JAVASCRIPT_EXTRACTOR_VERSION",
"PYTHON_ADAPTER_VERSION",
"PYTHON_EXTRACTOR_VERSION",
"JavaScriptReferenceAdapter",
"JavaScriptUnsupportedFact",
"PythonReferenceAdapter",
"PythonUnsupportedFact",
]

1349
src/docforge/adapters/cpp.py Normal file

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,948 @@
"""Deterministic stdlib-AST reference adapter for explicitly confined Python roots.
The adapter parses Python source as untrusted data. It never imports or executes
project code. Its intentionally narrow dependency model publishes only imports
that resolve to another module in the same declared source inventory.
"""
from __future__ import annotations
import ast
import hashlib
import io
import os
import stat
import tokenize
from collections.abc import Iterable, Sequence
from dataclasses import dataclass
from pathlib import Path, PurePosixPath
from ..adapter_sdk import (
AdapterAssembly,
AdapterEdge,
AdapterManifest,
AdapterNode,
AdapterProjection,
AdapterSource,
AdapterSourceProjection,
Edge,
Node,
)
from ..errors import DocForgeError
from ..python_logic import PythonLogicOwner, analyze_python_source
PYTHON_ADAPTER_ID = "docforge.reference.python"
PYTHON_ADAPTER_VERSION = "1"
PYTHON_EXTRACTOR_VERSION = "stdlib-ast@1"
PYTHON_IDENTITY_VERSION = "python-reference-id@1"
PYTHON_SUPPORT_SCHEMA_VERSION = 1
_SUPPORTED_FACTS = (
"python_file",
"python_module",
"python_class",
"python_function",
"lexical_containment",
"local_import_dependency",
"function_logic",
)
@dataclass(frozen=True)
class PythonUnsupportedFact:
"""One semantic fact this syntax-only reference adapter does not claim."""
code: str
description: str
def as_dict(self) -> dict[str, str]:
return {"code": self.code, "description": self.description}
_UNSUPPORTED_FACTS = (
PythonUnsupportedFact(
"call_resolution",
"Calls are represented only inside function Logic and are not resolved to symbols.",
),
PythonUnsupportedFact(
"dynamic_import_resolution",
"Imports performed through runtime calls are not dependency evidence.",
),
PythonUnsupportedFact(
"inheritance_resolution",
"Class bases are syntax only and are not resolved to local or external types.",
),
PythonUnsupportedFact(
"runtime_generated_facts",
"Decorators, metaclasses, descriptors, and executed module code are never evaluated.",
),
PythonUnsupportedFact(
"symbol_reference_resolution",
"Imported names, variable references, types, overloads, and re-exports are not resolved.",
),
)
@dataclass(frozen=True)
class _SourceRecord:
source_id: str
source_path: str
module_name: str
fingerprint: str
raw: bytes
text: str
tree: ast.Module | None
dependencies: tuple[str, ...] = ()
@dataclass(frozen=True)
class _Definition:
kind: str
qualified_name: str
parent_node_id: str
node: ast.ClassDef | ast.FunctionDef | ast.AsyncFunctionDef
node_id: str
class PythonReferenceAdapter:
"""Reference Python adapter over explicit, non-overlapping source roots."""
def __init__(
self,
root: Path,
*,
source_roots: Iterable[str | Path],
project_id: str = "python-reference",
title: str = "Python reference project",
max_sources: int = 4_096,
max_source_bytes: int = 1_000_000,
max_logic_nodes_per_function: int = 2_000,
) -> None:
resolved_root = root.resolve(strict=True)
if not resolved_root.is_dir():
raise DocForgeError("invalid_adapter", "Python adapter root must be a directory")
if max_sources < 1 or max_source_bytes < 1 or max_logic_nodes_per_function < 2:
raise ValueError("Python adapter limits must be positive")
self.root = resolved_root
self.project_id = project_id
self.title = title
self.max_sources = max_sources
self.max_source_bytes = max_source_bytes
self.max_logic_nodes_per_function = max_logic_nodes_per_function
self.source_roots = self._normalize_source_roots(source_roots)
def support_report(self) -> dict[str, object]:
"""Return deterministic, machine-readable scope and limitation evidence."""
return {
"schema_version": PYTHON_SUPPORT_SCHEMA_VERSION,
"adapter_id": PYTHON_ADAPTER_ID,
"adapter_version": PYTHON_ADAPTER_VERSION,
"extractor_version": PYTHON_EXTRACTOR_VERSION,
"identity_version": PYTHON_IDENTITY_VERSION,
"frontend": "python-stdlib-ast",
"imports_project_code": False,
"executes_project_code": False,
"source_roots": list(self.source_roots),
"supported_facts": list(_SUPPORTED_FACTS),
"unsupported_facts": [fact.as_dict() for fact in _UNSUPPORTED_FACTS],
}
def unsupported_facts(self) -> tuple[PythonUnsupportedFact, ...]:
"""Return the adapter's fixed unsupported semantic fact inventory."""
return _UNSUPPORTED_FACTS
def load_manifest(self) -> AdapterManifest:
records = self._inventory(parse=False)
module_sources = {record.module_name: record.source_id for record in records}
if len(module_sources) != len(records):
raise DocForgeError(
"ambiguous_python_module",
"Declared Python source roots produce duplicate module names",
)
sources = tuple(
sorted(
(
AdapterSource(
source_id=record.source_id,
source_path=record.source_path,
fingerprint=record.fingerprint,
extractor_version=PYTHON_EXTRACTOR_VERSION,
dependencies=self._local_dependencies(record, module_sources),
)
for record in records
),
key=lambda source: source.source_id,
)
)
source_hash = self._source_hash(sources)
return AdapterManifest(
project_id=self.project_id,
title=self.title,
adapter_id=PYTHON_ADAPTER_ID,
adapter_version=PYTHON_ADAPTER_VERSION,
root=self.root,
revision=source_hash[:12],
source_hash=source_hash,
families=("code",),
allowed_relations=("contains", "depends_on"),
sources=sources,
estimated_nodes=max(1, len(sources) * 8),
)
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection:
record = self._read_source(source.source_path)
if record.source_id != source.source_id or record.fingerprint != source.fingerprint:
raise DocForgeError(
"stale_adapter_source",
"Python source changed after its manifest was captured",
source=source.source_path,
)
return self._extract_record(record, source)
def assemble_projection(
self,
manifest: AdapterManifest,
contributions: tuple[AdapterSourceProjection, ...],
) -> AdapterAssembly:
expected = tuple(source.source_id for source in manifest.sources)
actual = tuple(sorted(contribution.source_id for contribution in contributions))
if actual != expected:
raise DocForgeError(
"invalid_adapter",
"Python assembly contributions do not match the current manifest",
)
nodes = tuple(
sorted(
(node for contribution in contributions for node in contribution.nodes),
key=lambda item: item.node.node_id,
)
)
edges = tuple(
sorted(
(edge for contribution in contributions for edge in contribution.edges),
key=lambda item: (
item.edge.source_id,
item.edge.relation,
item.edge.target_id,
),
)
)
logic = tuple(
sorted(
(projection for contribution in contributions for projection in contribution.logic),
key=lambda projection: projection.owner_node_id,
)
)
return AdapterAssembly(
projection=AdapterProjection(
project_id=manifest.project_id,
title=manifest.title,
adapter_id=manifest.adapter_id,
adapter_version=manifest.adapter_version,
root=manifest.root,
revision=manifest.revision,
source_hash=manifest.source_hash,
nodes=nodes,
edges=edges,
),
logic=logic,
)
def load_assembly(self) -> AdapterAssembly:
"""Load one cache-independent complete graph and Logic assembly."""
manifest = self.load_manifest()
contributions = tuple(self.extract_source(source) for source in manifest.sources)
stable = self.load_manifest()
if stable != manifest:
raise DocForgeError(
"source_changed",
"Python sources changed during complete adapter extraction",
)
return self.assemble_projection(manifest, contributions)
def load_complete_assembly(self) -> AdapterAssembly:
"""Implement the SDK's independent complete-assembly oracle."""
return self.load_assembly()
def load_projection(self) -> AdapterProjection:
"""Preserve the first-class legacy complete-projection contract."""
return self.load_assembly().projection
def _normalize_source_roots(self, source_roots: Iterable[str | Path]) -> tuple[str, ...]:
normalized: list[str] = []
for value in source_roots:
candidate = PurePosixPath(Path(value).as_posix())
if (
candidate.is_absolute()
or not candidate.parts
or ".." in candidate.parts
or str(candidate) in {"", "."}
):
raise DocForgeError(
"path_escape",
"Python source roots must be explicit project-relative directories",
)
relative = candidate.as_posix()
absolute = self.root.joinpath(*candidate.parts)
if absolute.is_symlink():
raise DocForgeError("path_escape", "Python source root cannot be a symbolic link")
try:
resolved = absolute.resolve(strict=True)
except OSError as error:
raise DocForgeError(
"invalid_adapter",
"Python source root does not exist",
source_root=relative,
) from error
if (
resolved != absolute
or not resolved.is_dir()
or not resolved.is_relative_to(self.root)
):
raise DocForgeError(
"path_escape",
"Python source root must be a confined real directory",
source_root=relative,
)
normalized.append(relative)
result = tuple(sorted(set(normalized)))
if not result:
raise DocForgeError("invalid_adapter", "At least one Python source root is required")
if len(result) != len(normalized):
raise DocForgeError("invalid_adapter", "Python source roots must be unique")
paths = [PurePosixPath(value) for value in result]
for index, left in enumerate(paths):
for right in paths[index + 1 :]:
if left in right.parents or right in left.parents:
raise DocForgeError(
"invalid_adapter",
"Python source roots must not overlap",
)
return result
def _inventory(self, *, parse: bool = True) -> tuple[_SourceRecord, ...]:
paths: list[str] = []
for source_root in self.source_roots:
absolute_root = self.root.joinpath(*PurePosixPath(source_root).parts)
if (
absolute_root.is_symlink()
or not absolute_root.is_dir()
or absolute_root.resolve(strict=True) != absolute_root
):
raise DocForgeError(
"path_escape",
"Python source root changed after adapter binding",
source_root=source_root,
)
for directory, directory_names, file_names in os.walk(
absolute_root,
topdown=True,
followlinks=False,
):
current = Path(directory)
for name in tuple(directory_names):
child = current / name
if child.is_symlink():
raise DocForgeError(
"path_escape",
"Python source inventory contains a symbolic-link directory",
source=child.relative_to(self.root).as_posix(),
)
directory_names[:] = sorted(
name for name in directory_names if name != "__pycache__"
)
for name in sorted(file_names):
if not name.endswith(".py"):
continue
child = current / name
relative = child.relative_to(self.root).as_posix()
if child.is_symlink():
raise DocForgeError(
"path_escape",
"Python source inventory contains a symbolic-link file",
source=relative,
)
paths.append(relative)
if len(paths) > self.max_sources:
raise DocForgeError(
"adapter_too_large",
"Python source inventory exceeds the configured limit",
maximum=self.max_sources,
)
records = tuple(self._read_source(path, parse=parse) for path in sorted(paths))
modules = [record.module_name for record in records]
if len(modules) != len(set(modules)):
duplicates = sorted(module for module in set(modules) if modules.count(module) > 1)
raise DocForgeError(
"ambiguous_python_module",
"Declared Python source roots produce duplicate module names",
modules=duplicates,
)
return records
def _read_source(self, relative: str, *, parse: bool = True) -> _SourceRecord:
path = PurePosixPath(relative)
if path.is_absolute() or ".." in path.parts or path.suffix != ".py":
raise DocForgeError("path_escape", "Python source path is unsafe")
source_root = self._source_root_for(path)
candidate = self.root.joinpath(*path.parts)
try:
before = candidate.lstat()
except OSError as error:
raise DocForgeError(
"stale_adapter_source",
"Python source is unavailable",
source=relative,
) from error
if (
stat.S_ISLNK(before.st_mode)
or not stat.S_ISREG(before.st_mode)
or candidate.resolve(strict=True) != candidate
or not candidate.is_relative_to(self.root)
):
raise DocForgeError(
"path_escape",
"Python source must be a confined regular file",
source=relative,
)
if before.st_size > self.max_source_bytes:
raise DocForgeError(
"source_too_large",
"Python source exceeds the configured adapter limit",
source=relative,
maximum=self.max_source_bytes,
)
descriptor = -1
try:
flags = os.O_RDONLY | getattr(os, "O_BINARY", 0)
flags |= getattr(os, "O_NOFOLLOW", 0)
descriptor = os.open(candidate, flags)
opened = os.fstat(descriptor)
if not stat.S_ISREG(opened.st_mode) or (opened.st_dev, opened.st_ino) != (
before.st_dev,
before.st_ino,
):
raise DocForgeError(
"path_escape",
"Python source identity changed before it was opened",
source=relative,
)
with os.fdopen(descriptor, "rb") as handle:
descriptor = -1
raw = handle.read(self.max_source_bytes + 1)
after_open = os.fstat(handle.fileno())
after = candidate.lstat()
except DocForgeError:
raise
except OSError as error:
raise DocForgeError(
"stale_adapter_source",
"Python source changed while being read",
source=relative,
) from error
finally:
if descriptor >= 0:
os.close(descriptor)
identity_before = (
before.st_dev,
before.st_ino,
before.st_size,
before.st_mtime_ns,
)
identity_after = (
after.st_dev,
after.st_ino,
after.st_size,
after.st_mtime_ns,
)
identity_opened = (
after_open.st_dev,
after_open.st_ino,
after_open.st_size,
after_open.st_mtime_ns,
)
if (
len(raw) > self.max_source_bytes
or identity_before != identity_opened
or identity_before != identity_after
or len(raw) != after.st_size
):
raise DocForgeError(
"stale_adapter_source",
"Python source changed while being read",
source=relative,
)
text = self._decode_source(raw, relative)
tree: ast.Module | None = None
if parse:
try:
tree = ast.parse(text, filename=relative, type_comments=True)
except SyntaxError as error:
raise DocForgeError(
"invalid_python_source",
"Python source cannot be parsed by the reference adapter",
source=relative,
line=error.lineno,
) from error
module_name = self._module_name(path, PurePosixPath(source_root))
return _SourceRecord(
source_id=self._source_id(relative),
source_path=relative,
module_name=module_name,
fingerprint=hashlib.sha256(raw).hexdigest(),
raw=raw,
text=text,
tree=tree,
)
def _source_root_for(self, path: PurePosixPath) -> str:
matches = [
source_root
for source_root in self.source_roots
if path == PurePosixPath(source_root) or PurePosixPath(source_root) in path.parents
]
if len(matches) != 1:
raise DocForgeError(
"path_escape",
"Python source is outside the declared source roots",
source=path.as_posix(),
)
return matches[0]
@staticmethod
def _decode_source(raw: bytes, source_path: str) -> str:
try:
encoding, _ = tokenize.detect_encoding(io.BytesIO(raw).readline)
return raw.decode(encoding)
except (LookupError, SyntaxError, UnicodeDecodeError) as error:
raise DocForgeError(
"invalid_python_source",
"Python source encoding is invalid",
source=source_path,
) from error
@staticmethod
def _module_name(path: PurePosixPath, source_root: PurePosixPath) -> str:
local = path.relative_to(source_root)
parts = list(local.with_suffix("").parts)
if parts and parts[-1] == "__init__":
parts.pop()
if not parts:
parts = list(source_root.parts)
return ".".join(parts)
@staticmethod
def _source_id(relative: str) -> str:
digest = hashlib.sha256(
f"{PYTHON_IDENTITY_VERSION}\0source\0{relative}".encode()
).hexdigest()[:24]
return f"python.source.{digest}"
@staticmethod
def _file_node_id(source_id: str) -> str:
return source_id.replace("python.source.", "python.file.", 1)
@staticmethod
def _module_node_id(source_id: str) -> str:
return source_id.replace("python.source.", "python.module.", 1)
@staticmethod
def _symbol_node_id(kind: str, source_id: str, qualified_name: str) -> str:
digest = hashlib.sha256(
(f"{PYTHON_IDENTITY_VERSION}\0{kind}\0{source_id}\0{qualified_name}").encode()
).hexdigest()[:24]
return f"python.{kind}.{digest}"
@staticmethod
def _source_hash(sources: Sequence[AdapterSource]) -> str:
digest = hashlib.sha256()
digest.update(PYTHON_ADAPTER_VERSION.encode())
digest.update(PYTHON_EXTRACTOR_VERSION.encode())
digest.update(PYTHON_IDENTITY_VERSION.encode())
for source in sources:
digest.update(b"\0source\0")
digest.update(source.source_id.encode())
digest.update(b"\0path\0")
digest.update(source.source_path.encode())
digest.update(b"\0fingerprint\0")
digest.update(source.fingerprint.encode())
for dependency in source.dependencies:
digest.update(b"\0dependency\0")
digest.update(dependency.encode())
return digest.hexdigest()
def _local_dependencies(
self,
record: _SourceRecord,
module_sources: dict[str, str],
) -> tuple[str, ...]:
dependencies: set[str] = set()
package = (
record.module_name
if record.source_path.endswith("/__init__.py")
else record.module_name.rpartition(".")[0]
)
for level, module, imported_names in self._lexical_imports(record):
candidates: list[str] = []
if not imported_names and level == 0 and module is not None:
candidates.append(module)
else:
base = self._import_from_base(package, level, module)
if base:
candidates.append(base)
candidates.extend(f"{base}.{name}" for name in imported_names if name != "*")
for module_name in candidates:
target = module_sources.get(module_name)
if target is not None and target != record.source_id:
dependencies.add(target)
return tuple(sorted(dependencies))
@staticmethod
def _lexical_imports(
record: _SourceRecord,
) -> tuple[tuple[int, str | None, tuple[str, ...]], ...]:
"""Inventory import dependencies without constructing a Python AST."""
try:
tokens = tuple(tokenize.generate_tokens(io.StringIO(record.text).readline))
except (IndentationError, tokenize.TokenError) as error:
line = error.args[1][0] if len(error.args) > 1 else None
raise DocForgeError(
"invalid_python_source",
"Python source cannot be tokenized by the reference adapter",
source=record.source_path,
line=line,
) from error
statements: list[list[tokenize.TokenInfo]] = []
current: list[tokenize.TokenInfo] = []
nesting = 0
ignored = {
tokenize.ENCODING,
tokenize.INDENT,
tokenize.DEDENT,
tokenize.NL,
tokenize.COMMENT,
}
for token in tokens:
if token.type in ignored:
continue
if token.type == tokenize.OP:
if token.string in "([{":
nesting += 1
elif token.string in ")]}":
nesting = max(0, nesting - 1)
elif token.string == ";" and nesting == 0:
if current:
statements.append(current)
current = []
continue
if token.type in {tokenize.NEWLINE, tokenize.ENDMARKER} and nesting == 0:
if current:
statements.append(current)
current = []
continue
current.append(token)
imports: list[tuple[int, str | None, tuple[str, ...]]] = []
for statement in statements:
for position, token in enumerate(statement):
if token.type != tokenize.NAME or token.string not in {"import", "from"}:
continue
if token.string == "import":
names = PythonReferenceAdapter._imported_names(statement[position + 1 :])
imports.extend((0, name, ()) for name in names)
break
import_position = next(
(
index
for index in range(position + 1, len(statement))
if statement[index].type == tokenize.NAME
and statement[index].string == "import"
),
None,
)
if import_position is None:
break
prefix = statement[position + 1 : import_position]
level = 0
for item in prefix:
if item.type == tokenize.NAME:
break
if item.type == tokenize.OP and set(item.string) == {"."}:
level += len(item.string)
module_parts = [item.string for item in prefix if item.type == tokenize.NAME]
module = ".".join(module_parts) or None
names = PythonReferenceAdapter._imported_names(statement[import_position + 1 :])
imports.append((level, module, names))
break
return tuple(imports)
@staticmethod
def _imported_names(tokens: Sequence[tokenize.TokenInfo]) -> tuple[str, ...]:
names: list[str] = []
current: list[str] = []
skip_alias = False
for token in tokens:
if token.type == tokenize.NAME and token.string == "as":
skip_alias = True
continue
if token.type == tokenize.OP and token.string == ",":
if current:
names.append(".".join(current))
current = []
skip_alias = False
continue
if token.type == tokenize.OP and token.string == "*":
if not skip_alias:
current.append("*")
continue
if token.type == tokenize.NAME and not skip_alias:
current.append(token.string)
if current:
names.append(".".join(current))
return tuple(name for name in names if name)
@staticmethod
def _import_from_base(package: str, level: int, module: str | None) -> str:
if level == 0:
return module or ""
package_parts = package.split(".") if package else []
keep = len(package_parts) - (level - 1)
if keep < 0:
return ""
prefix = package_parts[:keep]
if module:
prefix.extend(module.split("."))
return ".".join(prefix)
def _extract_record(
self,
record: _SourceRecord,
source: AdapterSource,
) -> AdapterSourceProjection:
tree = record.tree
if tree is None:
raise DocForgeError(
"invalid_adapter",
"Python extraction requires a parsed source record",
source=record.source_path,
)
file_id = self._file_node_id(source.source_id)
module_id = self._module_node_id(source.source_id)
nodes: list[AdapterNode] = [
self._node(
node_id=file_id,
title=record.source_path,
kind="file",
qualified_name=record.source_path,
content=record.text,
source_path=record.source_path,
anchor="L1",
),
self._node(
node_id=module_id,
title=record.module_name,
kind="module",
qualified_name=record.module_name,
content=(
ast.get_docstring(tree, clean=False) or f"Python module {record.module_name}."
),
source_path=record.source_path,
anchor="L1",
),
]
edges: list[AdapterEdge] = [self._edge(file_id, "contains", module_id, "syntax")]
definitions = self._definitions(record, module_id)
owners: list[PythonLogicOwner] = []
for definition in definitions:
content = ast.get_source_segment(record.text, definition.node)
if content is None or not content.strip():
content = f"Python {definition.kind} {definition.qualified_name}."
anchor = f"L{definition.node.lineno}"
nodes.append(
self._node(
node_id=definition.node_id,
title=definition.qualified_name,
kind=definition.kind,
qualified_name=f"{record.module_name}.{definition.qualified_name}",
content=content,
source_path=record.source_path,
anchor=anchor,
asynchronous=isinstance(definition.node, ast.AsyncFunctionDef),
)
)
edges.append(
self._edge(
definition.parent_node_id,
"contains",
definition.node_id,
"syntax",
anchor=anchor,
)
)
if definition.kind == "function":
owners.append(
PythonLogicOwner(
owner_node_id=definition.node_id,
qualified_name=definition.qualified_name,
line=definition.node.lineno,
)
)
for dependency in source.dependencies:
edges.append(
self._edge(
module_id,
"depends_on",
self._module_node_id(dependency),
"local_import",
)
)
logic = analyze_python_source(
record.text,
source_id=source.source_id,
owners=owners,
filename=record.source_path,
max_nodes_per_function=self.max_logic_nodes_per_function,
)
return AdapterSourceProjection(
source_id=source.source_id,
fingerprint=source.fingerprint,
nodes=tuple(sorted(nodes, key=lambda item: item.node.node_id)),
edges=tuple(
sorted(
edges,
key=lambda item: (
item.edge.source_id,
item.edge.relation,
item.edge.target_id,
),
)
),
logic=logic,
)
def _definitions(
self,
record: _SourceRecord,
module_node_id: str,
) -> tuple[_Definition, ...]:
tree = record.tree
if tree is None:
raise DocForgeError(
"invalid_adapter",
"Python definition extraction requires a parsed source record",
source=record.source_path,
)
definitions: list[_Definition] = []
qualified_names: set[str] = set()
adapter = self
class Collector(ast.NodeVisitor):
def __init__(self) -> None:
self.scope_names: list[str] = []
self.scope_node_ids: list[str] = [module_node_id]
def _definition(
self,
kind: str,
node: ast.ClassDef | ast.FunctionDef | ast.AsyncFunctionDef,
) -> None:
qualified_name = ".".join((*self.scope_names, node.name))
if qualified_name in qualified_names:
raise DocForgeError(
"ambiguous_python_symbol",
"Python source repeats a class or function identity",
source=record.source_path,
qualified_name=qualified_name,
)
qualified_names.add(qualified_name)
node_id = adapter._symbol_node_id(
kind,
record.source_id,
qualified_name,
)
definitions.append(
_Definition(
kind=kind,
qualified_name=qualified_name,
parent_node_id=self.scope_node_ids[-1],
node=node,
node_id=node_id,
)
)
self.scope_names.append(node.name)
self.scope_node_ids.append(node_id)
self.generic_visit(node)
self.scope_node_ids.pop()
self.scope_names.pop()
def visit_ClassDef(self, node: ast.ClassDef) -> None:
self._definition("class", node)
def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
self._definition("function", node)
def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None:
self._definition("function", node)
Collector().visit(tree)
return tuple(sorted(definitions, key=lambda item: item.node_id))
@staticmethod
def _node(
*,
node_id: str,
title: str,
kind: str,
qualified_name: str,
content: str,
source_path: str,
anchor: str,
asynchronous: bool = False,
) -> AdapterNode:
normalized = content.strip() or f"Python {kind} {qualified_name}."
tags = tuple(sorted({"python", kind, *(("async",) if asynchronous else ())}))
return AdapterNode(
node=Node(
node_id=node_id,
title=title,
family="code",
authority="derived",
status="active",
tags=tags,
summary=f"Python {kind} fact for {qualified_name}.",
content=normalized,
source_path=source_path,
source_anchor=anchor,
content_hash=hashlib.sha256(normalized.encode()).hexdigest(),
),
metadata=(
("extractor", PYTHON_EXTRACTOR_VERSION),
("identity", PYTHON_IDENTITY_VERSION),
("kind", kind),
("qualified_name", qualified_name),
),
)
@staticmethod
def _edge(
source_id: str,
relation: str,
target_id: str,
evidence: str,
*,
anchor: str | None = None,
) -> AdapterEdge:
metadata = [("evidence", evidence)]
if anchor is not None:
metadata.append(("source_anchor", anchor))
return AdapterEdge(
edge=Edge(source_id, relation, target_id),
metadata=tuple(metadata),
)

View file

@ -16,6 +16,12 @@ from dataclasses import dataclass
from pathlib import Path
from typing import Literal, cast
from .adapter_launcher import (
AdapterLauncherV1,
AdapterSourceAvailabilityV1,
adapter_source_availability,
validate_adapter_launcher,
)
from .changeset_contract import document_hash
from .errors import DocForgeError
from .models import ProjectDescriptor, ProjectService
@ -1079,3 +1085,450 @@ def generate_client_configuration(
}
_validate_configuration_result(result, trusted_descriptor=descriptor)
return result
def generate_adapter_client_configuration(
project: ProjectService,
launcher: AdapterLauncherV1,
client: str,
*,
server_name: str | None = None,
capability_mode: str = "read",
proposal_writer: str | None = None,
canonical_applier: str | None = None,
no_ast: bool = False,
manual_render_policy: str | None = None,
portable_graph_policy: str | None = None,
live_viewer_policy: str | None = None,
startup_timeout: int = 30,
tool_timeout: int = 300,
output: Path | None = None,
) -> dict[str, object]:
"""Generate one client fragment for an explicitly constructed project-owned adapter."""
validate_adapter_launcher(project, launcher)
selected_client = _client_name(client)
selected_mode = _capability_mode(capability_mode)
startup_seconds = _bounded_seconds(
startup_timeout,
field="startup_timeout",
maximum=3_600,
)
tool_seconds = _bounded_seconds(
tool_timeout,
field="tool_timeout",
maximum=86_400,
)
descriptor = project.descriptor
_validate_adapter_authority(
descriptor,
selected_mode=selected_mode,
proposal_writer=proposal_writer,
canonical_applier=canonical_applier,
)
source_availability = adapter_source_availability(project, launcher)
fingerprint = project_root_fingerprint(descriptor.root)
selected_name = _validated_server_name(
server_name,
project_id=descriptor.project_id,
fingerprint=fingerprint,
)
executable = Path(os.path.abspath(sys.executable))
try:
executable_status = executable.stat()
except OSError as error:
raise DocForgeError(
"client_configuration_unavailable",
"Current Python executable cannot be inspected",
) from error
if not stat.S_ISREG(executable_status.st_mode) or not os.access(executable, os.X_OK):
raise DocForgeError(
"client_configuration_unavailable",
"Current Python executable is not a runnable regular file",
)
policy = compose_effective_policy(
selected_mode=selected_mode,
capability_source="explicit",
no_ast=no_ast,
diagnostics=False,
render_configured=descriptor.render is not None,
application_enabled=canonical_applier is not None,
)
projection_policy = compose_projection_policy(
manual=manual_render_policy,
portable_graph=portable_graph_policy,
live_viewer=live_viewer_policy,
manual_configured=descriptor.render is not None,
portable_graph_configured=descriptor.graph_render is not None,
application_enabled=canonical_applier is not None,
)
default_projection_policy = compose_projection_policy(
manual_configured=descriptor.render is not None,
portable_graph_configured=descriptor.graph_render is not None,
application_enabled=canonical_applier is not None,
)
arguments = [
"-I",
"-m",
launcher.module,
"--project-root",
str(descriptor.root),
"--capability-mode",
selected_mode,
]
if proposal_writer is not None:
arguments.extend(("--proposal-writer", proposal_writer))
if canonical_applier is not None:
arguments.extend(("--canonical-applier", canonical_applier))
for option, selected, default in (
(
"--manual-render-policy",
projection_policy.manual,
default_projection_policy.manual,
),
(
"--portable-graph-policy",
projection_policy.portable_graph,
default_projection_policy.portable_graph,
),
(
"--live-viewer-policy",
projection_policy.live_viewer,
default_projection_policy.live_viewer,
),
):
if selected != default:
arguments.extend((option, selected))
if no_ast:
arguments.append("--no-ast")
artifact_format, content, warning = _artifact(
selected_client,
server_name=selected_name,
command=str(executable),
arguments=arguments,
startup_timeout=startup_seconds,
tool_timeout=tool_seconds,
)
def validate_current_binding() -> None:
validate_adapter_launcher(project, launcher)
current = adapter_source_availability(project, launcher)
if current != source_availability:
raise DocForgeError(
"source_changed",
"Adapter sources changed during client configuration generation",
)
if output is None:
validate_current_binding()
write_state = "not_requested"
durability = "not_applicable"
publication_warning = None
output_path = None
else:
validate_current_binding()
write_state, durability, publication_warning, published_path = _atomic_write(
output,
content,
validate_binding=validate_current_binding,
)
output_path = str(published_path) if published_path is not None else None
artifact = {
"format": artifact_format,
"content": content,
"content_sha256": hashlib.sha256(content.encode("utf-8")).hexdigest(),
"output_path": output_path,
"write_state": write_state,
"durability": durability,
}
binding = {
"transport": "stdio",
"capability_mode": selected_mode,
"adapter_policy": policy.adapter_policy(),
"render_policy": {
"manual": policy.manual_render,
"graph": policy.graph_render,
"live_viewer": policy.live_viewer,
},
"command": str(executable),
"args": arguments,
"environment": {},
"timeouts": {
"startup_seconds": startup_seconds,
"tool_seconds": tool_seconds,
},
"launcher_hash": launcher.launcher_hash,
}
project_binding = {
"project_id": descriptor.project_id,
"project_root": str(descriptor.root),
"project_root_fingerprint": fingerprint,
"adapter": descriptor.adapter,
"descriptor_hash": descriptor.descriptor_hash,
}
projection_availability = {
"manual_configured": descriptor.render is not None,
"portable_graph_configured": descriptor.graph_render is not None,
"application_enabled": canonical_applier is not None,
"live_viewer_available": True,
}
policy_payload = policy.as_dict()
result: dict[str, object] = {
"status": "ok",
"schema_version": 1,
"operation": "adapter_client.configure",
"action": "write" if output is not None else "preview",
"client": selected_client,
"server_name": selected_name,
"project": project_binding,
"launcher": launcher.as_dict(),
"launcher_hash": launcher.launcher_hash,
"source_availability": source_availability.as_dict(),
"source_availability_hash": source_availability.availability_hash,
"binding": binding,
"effective_policy": policy_payload,
"projection_policy": projection_policy.as_dict(),
"projection_policy_hash": projection_policy.policy_hash,
"projection_availability": projection_availability,
"artifact": artifact,
"warnings": [
*([] if warning is None else [{"code": "timeout_format_unverified"}]),
*([] if publication_warning is None else [{"code": publication_warning}]),
],
}
result["configuration_hash"] = document_hash(_adapter_configuration_hash_payload(result))
_validate_adapter_configuration_result(
result,
project=project,
launcher=launcher,
source_availability=source_availability,
)
return result
def _validate_adapter_authority(
descriptor: ProjectDescriptor,
*,
selected_mode: CapabilityMode,
proposal_writer: str | None,
canonical_applier: str | None,
) -> None:
writer_ids = {writer.writer_id for writer in descriptor.proposal_writers}
if selected_mode == "read":
if proposal_writer is not None or canonical_applier is not None:
raise DocForgeError(
"invalid_capability_binding",
"Read configuration cannot bind proposal or application authority",
)
elif selected_mode == "proposal":
if proposal_writer is None or proposal_writer not in writer_ids:
raise DocForgeError(
"capability_unavailable",
"Proposal configuration requires a descriptor-declared writer",
required="proposal_writer",
)
if canonical_applier is not None:
raise DocForgeError(
"invalid_capability_binding",
"Proposal configuration cannot bind a canonical applier",
)
elif (
proposal_writer is None
or canonical_applier is None
or proposal_writer != canonical_applier
or proposal_writer not in writer_ids
):
raise DocForgeError(
"capability_unavailable",
"Application configuration requires one declared writer/applier identity",
required="matching_declared_writer_and_applier",
)
def _adapter_configuration_hash_payload(result: dict[str, object]) -> dict[str, object]:
artifact = cast(dict[str, object], result["artifact"])
return {
"schema_version": 1,
"client": result["client"],
"server_name": result["server_name"],
"project": result["project"],
"launcher": result["launcher"],
"launcher_hash": result["launcher_hash"],
"source_availability": result["source_availability"],
"source_availability_hash": result["source_availability_hash"],
"binding": result["binding"],
"effective_policy": result["effective_policy"],
"projection_policy": result["projection_policy"],
"projection_policy_hash": result["projection_policy_hash"],
"projection_availability": result["projection_availability"],
"artifact_format": artifact["format"],
"artifact_content_sha256": artifact["content_sha256"],
}
def _validate_adapter_configuration_result(
result: dict[str, object],
*,
project: ProjectService,
launcher: AdapterLauncherV1,
source_availability: AdapterSourceAvailabilityV1,
) -> None:
validate_adapter_launcher(project, launcher)
descriptor = project.descriptor
project_binding = cast(dict[str, object], result["project"])
if (
result["launcher"] != launcher.as_dict()
or result["launcher_hash"] != launcher.launcher_hash
or result["source_availability"] != source_availability.as_dict()
or result["source_availability_hash"] != source_availability.availability_hash
or project_binding
!= {
"project_id": descriptor.project_id,
"project_root": str(descriptor.root),
"project_root_fingerprint": project_root_fingerprint(descriptor.root),
"adapter": descriptor.adapter,
"descriptor_hash": descriptor.descriptor_hash,
}
):
raise AssertionError("Generated adapter client identity drifted")
artifact = cast(dict[str, object], result["artifact"])
binding = cast(dict[str, object], result["binding"])
content = cast(str, artifact["content"])
if artifact["content_sha256"] != hashlib.sha256(content.encode("utf-8")).hexdigest():
raise AssertionError("Generated adapter client content hash drifted")
timeouts = cast(dict[str, object], binding["timeouts"])
expected_format, expected_content, _ = _artifact(
cast(ClientName, result["client"]),
server_name=cast(str, result["server_name"]),
command=cast(str, binding["command"]),
arguments=cast(list[str], binding["args"]),
startup_timeout=cast(int, timeouts["startup_seconds"]),
tool_timeout=cast(int, timeouts["tool_seconds"]),
)
if artifact["format"] != expected_format or content != expected_content:
raise AssertionError("Generated adapter client artifact drifted from its binding")
if binding["environment"] != {} or binding["launcher_hash"] != launcher.launcher_hash:
raise AssertionError("Generated adapter launch boundary drifted")
adapter_policy = cast(dict[str, object], binding["adapter_policy"])
render_policy = cast(dict[str, object], binding["render_policy"])
projection_policy = cast(dict[str, object], result["projection_policy"])
projection_availability = cast(dict[str, object], result["projection_availability"])
arguments = cast(list[str], binding["args"])
prefix = [
"-I",
"-m",
launcher.module,
"--project-root",
str(descriptor.root),
"--capability-mode",
cast(str, binding["capability_mode"]),
]
if arguments[:7] != prefix:
raise AssertionError("Generated adapter client arguments drifted from their binding")
remaining = arguments[7:]
no_ast_argument = "--no-ast" in arguments
if no_ast_argument:
if remaining[-1:] != ["--no-ast"] or arguments.count("--no-ast") != 1:
raise AssertionError("Generated adapter no-AST argument layout drifted")
remaining = remaining[:-1]
projection_arguments: dict[str, str] = {}
authority_arguments: list[str] = []
projection_options = {
"--manual-render-policy": "manual",
"--portable-graph-policy": "portable_graph",
"--live-viewer-policy": "live_viewer",
}
position = 0
while position < len(remaining):
option = remaining[position]
field = projection_options.get(option)
if field is None:
authority_arguments.append(option)
position += 1
continue
if position + 1 >= len(remaining) or option in projection_arguments:
raise AssertionError("Generated adapter projection argument layout drifted")
value = remaining[position + 1]
projection_arguments[option] = value
if projection_policy[field] != value:
raise AssertionError("Generated adapter projection policy argument drifted")
position += 2
mode = binding["capability_mode"]
if (
(mode == "read" and authority_arguments)
or (
mode == "proposal"
and (
len(authority_arguments) != 2
or authority_arguments[0] != "--proposal-writer"
or not authority_arguments[1]
)
)
or (
mode == "application"
and (
len(authority_arguments) != 4
or authority_arguments[0] != "--proposal-writer"
or authority_arguments[2] != "--canonical-applier"
or not authority_arguments[1]
or authority_arguments[1] != authority_arguments[3]
)
)
):
raise AssertionError("Generated adapter authority argument layout drifted")
expected_projection_policy = compose_projection_policy(
manual=projection_arguments.get("--manual-render-policy"),
portable_graph=projection_arguments.get("--portable-graph-policy"),
live_viewer=projection_arguments.get("--live-viewer-policy"),
manual_configured=cast(bool, projection_availability["manual_configured"]),
portable_graph_configured=cast(
bool,
projection_availability["portable_graph_configured"],
),
application_enabled=cast(
bool,
projection_availability["application_enabled"],
),
live_viewer_available=cast(
bool,
projection_availability["live_viewer_available"],
),
)
policy = cast(dict[str, object], result["effective_policy"])
composed_policy = compose_effective_policy(
selected_mode=cast(CapabilityMode, mode),
capability_source="explicit",
no_ast=adapter_policy["mode"] == "preserve-no-ast",
diagnostics=False,
render_configured=render_policy["manual"] != "disabled",
application_enabled=mode == "application",
)
if (
projection_policy != expected_projection_policy.as_dict()
or projection_availability["manual_configured"] != (descriptor.render is not None)
or projection_availability["portable_graph_configured"]
!= (descriptor.graph_render is not None)
or projection_availability["application_enabled"] != (mode == "application")
or projection_availability["live_viewer_available"] is not True
or policy != composed_policy.as_dict()
or adapter_policy != composed_policy.adapter_policy()
or no_ast_argument != (adapter_policy["mode"] == "preserve-no-ast")
or binding["launcher_hash"] != result["launcher_hash"]
):
raise AssertionError("Generated adapter client policy drifted")
if result["projection_policy_hash"] != hashlib.sha256(
json.dumps(
projection_policy,
sort_keys=True,
separators=(",", ":"),
ensure_ascii=False,
).encode("utf-8")
).hexdigest() or result["configuration_hash"] != document_hash(
_adapter_configuration_hash_payload(result)
):
raise AssertionError("Generated adapter client hash drifted")

View file

@ -0,0 +1,242 @@
"""Deterministic command references derived from live CLI and MCP registrations."""
from __future__ import annotations
import argparse
import hashlib
import json
from collections.abc import Collection, Iterable, Mapping
from dataclasses import dataclass
from typing import Any, cast
from mcp.types import Tool
@dataclass(frozen=True)
class CliCommandReference:
"""One CLI command and the exact normalized usage emitted by argparse."""
name: str
invocation: str
@dataclass(frozen=True)
class McpToolReference:
"""One registered MCP tool with arguments derived from its input schema."""
surface: str
name: str
required_arguments: tuple[str, ...]
optional_arguments: tuple[str, ...]
description: str
input_schema_hash: str
def cli_command_references(
parser: argparse.ArgumentParser | None = None,
) -> tuple[CliCommandReference, ...]:
"""Read CLI reference rows from the parser used by ``docforge``."""
effective_parser = parser or _docforge_parser()
commands = _command_parsers(effective_parser)
return tuple(
CliCommandReference(
name=name,
invocation=_normalize_usage(command_parser.format_usage()),
)
for name, command_parser in sorted(commands.items())
)
def mcp_tool_references(
tools: Iterable[Tool],
*,
expected_names: Collection[str] | None = None,
) -> tuple[McpToolReference, ...]:
"""Read MCP reference rows from registered tools returned by ``list_tools``.
``expected_names`` makes documentation generation fail closed when the selected
server surface is incomplete or has drifted.
"""
surface_by_name = _mcp_surface_by_name()
references: list[McpToolReference] = []
observed: set[str] = set()
for tool in tools:
if tool.name in observed:
raise ValueError(f"MCP tool registration repeats {tool.name!r}")
observed.add(tool.name)
try:
surface = surface_by_name[tool.name]
except KeyError as error:
raise ValueError(
f"MCP tool {tool.name!r} has no declared capability surface"
) from error
schema = tool.inputSchema
properties = _schema_properties(schema)
required = _schema_required(schema, properties)
references.append(
McpToolReference(
surface=surface,
name=tool.name,
required_arguments=tuple(sorted(required)),
optional_arguments=tuple(sorted(set(properties) - required)),
description=_normalize_text(tool.description or ""),
input_schema_hash=_canonical_hash(schema),
)
)
if expected_names is not None:
expected = set(expected_names)
if observed != expected:
missing = sorted(expected - observed)
unexpected = sorted(observed - expected)
raise ValueError(
"Registered MCP tools do not match the requested reference surface: "
f"missing={missing!r}, unexpected={unexpected!r}"
)
surface_rank = {"read": 0, "proposal": 1, "application": 2}
return tuple(sorted(references, key=lambda item: (surface_rank[item.surface], item.name)))
def render_cli_reference_markdown(references: Iterable[CliCommandReference]) -> str:
"""Render a deterministic Markdown table for CLI commands."""
rows = sorted(references, key=lambda item: item.name)
lines = [
"| Command | Invocation |",
"|---|---|",
]
lines.extend(
f"| `{_escape_markdown(item.name)}` | `{_escape_markdown(item.invocation)}` |"
for item in rows
)
return "\n".join(lines) + "\n"
def render_mcp_reference_markdown(references: Iterable[McpToolReference]) -> str:
"""Render a deterministic Markdown table for registered MCP tools."""
surface_rank = {"read": 0, "proposal": 1, "application": 2}
rows = sorted(references, key=lambda item: (surface_rank[item.surface], item.name))
lines = [
"| Surface | Tool | Required arguments | Optional arguments | "
"Input schema SHA-256 | Description |",
"|---|---|---|---|---|---|",
]
lines.extend(
"| "
f"{_escape_markdown(item.surface)} | "
f"`{_escape_markdown(item.name)}` | "
f"{_argument_list(item.required_arguments)} | "
f"{_argument_list(item.optional_arguments)} | "
f"`{item.input_schema_hash}` | "
f"{_escape_markdown(item.description) or ''} |"
for item in rows
)
return "\n".join(lines) + "\n"
def render_command_reference_markdown(
cli_references: Iterable[CliCommandReference],
mcp_references: Iterable[McpToolReference],
) -> str:
"""Render the complete deterministic CLI and MCP command reference."""
return (
"# DocForge command reference\n\n"
"> Generated from the live CLI parser and MCP registrations. Do not edit this file "
"by hand.\n\n"
"Global CLI options are documented in `docforge --help` and are not repeated in each "
"command row. The MCP table is the complete generic application-enabled surface; "
"the fixed reference-adapter server exposes only its read rows.\n\n"
"## CLI commands\n\n"
f"{render_cli_reference_markdown(cli_references)}"
"\n## MCP tools\n\n"
f"{render_mcp_reference_markdown(mcp_references)}"
)
def _docforge_parser() -> argparse.ArgumentParser:
from .cli import _parser # pyright: ignore[reportPrivateUsage]
return _parser()
def _command_parsers(
parser: argparse.ArgumentParser,
) -> Mapping[str, argparse.ArgumentParser]:
actions = parser._actions # pyright: ignore[reportPrivateUsage]
subparsers = [
action
for action in actions
if action.dest == "command" and isinstance(action.choices, dict)
]
if len(subparsers) != 1:
raise ValueError("CLI parser must define exactly one command subparser")
return cast(Mapping[str, argparse.ArgumentParser], subparsers[0].choices)
def _mcp_surface_by_name() -> dict[str, str]:
from .mcp_server import APPLICATION_TOOLS, PROPOSAL_TOOLS, READ_TOOLS
groups = {
"read": READ_TOOLS,
"proposal": PROPOSAL_TOOLS,
"application": APPLICATION_TOOLS,
}
surface_by_name: dict[str, str] = {}
for surface, names in groups.items():
for name in names:
if name in surface_by_name:
raise ValueError(f"MCP tool surface declaration repeats {name!r}")
surface_by_name[name] = surface
return surface_by_name
def _normalize_usage(usage: str) -> str:
return _normalize_text(usage.removeprefix("usage: "))
def _normalize_text(value: str) -> str:
return " ".join(value.split())
def _schema_properties(schema: Mapping[str, Any]) -> dict[str, Any]:
raw_properties: object = schema.get("properties", {})
if not isinstance(raw_properties, dict):
raise ValueError("MCP tool input schema properties must be a string-keyed object")
properties: dict[str, Any] = {}
for name, value in cast(dict[object, object], raw_properties).items():
if not isinstance(name, str):
raise ValueError("MCP tool input schema properties must be a string-keyed object")
properties[name] = value
return properties
def _schema_required(schema: Mapping[str, Any], properties: Mapping[str, Any]) -> set[str]:
raw_required: object = schema.get("required", [])
if not isinstance(raw_required, list):
raise ValueError("MCP tool input schema required arguments must be strings")
required_names: set[str] = set()
for name in cast(list[object], raw_required):
if not isinstance(name, str):
raise ValueError("MCP tool input schema required arguments must be strings")
required_names.add(name)
if not required_names <= set(properties):
raise ValueError("MCP tool input schema requires an undeclared argument")
return required_names
def _canonical_hash(payload: Mapping[str, Any]) -> str:
encoded = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
return hashlib.sha256(encoded).hexdigest()
def _argument_list(arguments: tuple[str, ...]) -> str:
if not arguments:
return ""
return ", ".join(f"`{_escape_markdown(argument)}`" for argument in arguments)
def _escape_markdown(value: str) -> str:
return value.replace("\\", "\\\\").replace("|", "\\|").replace("`", "\\`")

View file

@ -4,6 +4,7 @@ from __future__ import annotations
import json
import os
import stat
import tempfile
from dataclasses import dataclass
from pathlib import Path
@ -12,6 +13,8 @@ from typing import Any, cast
from .errors import DocForgeError
EXTRACTION_CACHE_SCHEMA_VERSION = 1
MAX_EXTRACTION_CACHE_BYTES = 64_000_000
MAX_EXTRACTION_CACHE_SOURCES = 10_000
@dataclass(frozen=True)
@ -42,13 +45,39 @@ def load_extraction_cache(
project_id: str,
adapter_id: str,
adapter_version: str,
max_bytes: int = MAX_EXTRACTION_CACHE_BYTES,
max_sources: int = MAX_EXTRACTION_CACHE_SOURCES,
) -> ExtractionCache | None:
"""Read a cache generation, treating malformed or incompatible data as a miss."""
if not path.is_file() or path.is_symlink():
if max_bytes < 1 or max_sources < 1:
raise ValueError("Extraction cache limits must be positive")
try:
descriptor = os.open(
path,
os.O_RDONLY
| getattr(os, "O_CLOEXEC", 0)
| getattr(os, "O_NOFOLLOW", 0)
| getattr(os, "O_NONBLOCK", 0),
)
except OSError:
return None
try:
raw = json.loads(path.read_text(encoding="utf-8"))
with os.fdopen(descriptor, "rb") as handle:
before = os.fstat(handle.fileno())
if not stat.S_ISREG(before.st_mode) or before.st_size > max_bytes:
return None
encoded = handle.read(max_bytes + 1)
after = os.fstat(handle.fileno())
if (
len(encoded) > max_bytes
or before.st_dev != after.st_dev
or before.st_ino != after.st_ino
or before.st_size != after.st_size
or before.st_mtime_ns != after.st_mtime_ns
):
return None
raw = json.loads(encoded.decode("utf-8"))
if not isinstance(raw, dict):
return None
document = cast(dict[str, Any], raw)
@ -62,8 +91,11 @@ def load_extraction_cache(
raw_sources = document.get("sources")
if not isinstance(raw_sources, list):
return None
source_items = cast(list[object], raw_sources)
if len(source_items) > max_sources:
return None
sources: list[CachedSource] = []
for raw_source in cast(list[object], raw_sources):
for raw_source in source_items:
if not isinstance(raw_source, dict):
return None
item = cast(dict[str, object], raw_source)
@ -110,9 +142,24 @@ def load_extraction_cache(
return None
def write_extraction_cache(path: Path, cache: ExtractionCache) -> None:
def write_extraction_cache(
path: Path,
cache: ExtractionCache,
*,
max_bytes: int = MAX_EXTRACTION_CACHE_BYTES,
max_sources: int = MAX_EXTRACTION_CACHE_SOURCES,
) -> None:
"""Atomically publish one validated extraction-cache generation."""
if max_bytes < 1 or max_sources < 1:
raise ValueError("Extraction cache limits must be positive")
if len(cache.sources) > max_sources:
raise DocForgeError(
"cache_limit",
"Incremental extraction cache exceeds its source limit",
maximum=max_sources,
actual=len(cache.sources),
)
path.parent.mkdir(parents=True, exist_ok=True)
document = {
"schema_version": EXTRACTION_CACHE_SCHEMA_VERSION,
@ -131,17 +178,23 @@ def write_extraction_cache(path: Path, cache: ExtractionCache) -> None:
for source in cache.sources
],
}
encoded = json.dumps(document, sort_keys=True, separators=(",", ":")).encode("utf-8") + b"\n"
if len(encoded) > max_bytes:
raise DocForgeError(
"cache_limit",
"Incremental extraction cache exceeds its byte limit",
maximum=max_bytes,
actual=len(encoded),
)
with tempfile.NamedTemporaryFile(
mode="w",
encoding="utf-8",
mode="wb",
prefix="extractions-",
suffix=".json",
dir=path.parent,
delete=False,
) as descriptor:
temporary = Path(descriptor.name)
json.dump(document, descriptor, sort_keys=True, separators=(",", ":"))
descriptor.write("\n")
descriptor.write(encoded)
descriptor.flush()
os.fsync(descriptor.fileno())
try:

View file

@ -0,0 +1,428 @@
"""Strict, data-only configuration for the runnable reference adapters."""
from __future__ import annotations
import hashlib
import os
import stat
import tomllib
from dataclasses import dataclass
from pathlib import Path, PurePosixPath
from typing import Literal, cast
from .config_validation import ID_PATTERN
from .errors import DocForgeError
REFERENCE_ADAPTER_SCHEMA_VERSION = 1
REFERENCE_ADAPTER_CONFIG = PurePosixPath(".docforge/reference-adapter.toml")
MAX_REFERENCE_CONFIG_BYTES = 65_536
MAX_REFERENCE_INVENTORY_ENTRIES = 65_536
ReferenceLanguage = Literal["python", "javascript", "typescript", "cpp"]
REFERENCE_LANGUAGES: tuple[ReferenceLanguage, ...] = (
"python",
"javascript",
"typescript",
"cpp",
)
_CONFIG_FIELDS = frozenset(
{
"schema_version",
"project_id",
"title",
"language",
"source_roots",
"compilation_database",
}
)
_LANGUAGE_SUFFIXES: dict[ReferenceLanguage, frozenset[str]] = {
"python": frozenset({".py"}),
"javascript": frozenset({".cjs", ".js", ".mjs"}),
"typescript": frozenset({".cts", ".mts", ".ts"}),
"cpp": frozenset({".c", ".cc", ".cpp", ".cxx", ".h", ".hh", ".hpp", ".hxx"}),
}
_KNOWN_SOURCE_SUFFIXES = frozenset(
suffix for suffixes in _LANGUAGE_SUFFIXES.values() for suffix in suffixes
)
__all__ = [
"MAX_REFERENCE_CONFIG_BYTES",
"MAX_REFERENCE_INVENTORY_ENTRIES",
"REFERENCE_ADAPTER_CONFIG",
"REFERENCE_ADAPTER_SCHEMA_VERSION",
"REFERENCE_LANGUAGES",
"ReferenceAdapterConfigV1",
"ReferenceLanguage",
"load_reference_adapter_config",
]
@dataclass(frozen=True)
class ReferenceAdapterConfigV1:
"""One immutable, project-confined reference adapter selection."""
schema_version: Literal[1]
project_id: str
title: str
language: ReferenceLanguage
source_roots: tuple[Path, ...]
compilation_database: Path | None
project_root: Path
config_path: Path
config_hash: str
def as_dict(self) -> dict[str, object]:
document: dict[str, object] = {
"schema_version": self.schema_version,
"project_id": self.project_id,
"title": self.title,
"language": self.language,
"source_roots": [
path.relative_to(self.project_root).as_posix() for path in self.source_roots
],
}
if self.compilation_database is not None:
document["compilation_database"] = self.compilation_database.relative_to(
self.project_root
).as_posix()
return document
def load_reference_adapter_config(project_root: str | Path) -> ReferenceAdapterConfigV1:
"""Load the fixed reference-adapter descriptor without executing project content."""
root = _project_root(project_root)
config_path = root.joinpath(*REFERENCE_ADAPTER_CONFIG.parts)
try:
resolved_config = config_path.resolve(strict=True)
except (OSError, ValueError) as error:
raise DocForgeError(
"missing_path",
"Reference adapter configuration does not exist",
path=str(config_path),
) from error
if resolved_config != config_path or not resolved_config.is_relative_to(root):
raise DocForgeError(
"path_escape",
"Reference adapter configuration must not traverse symbolic links",
)
raw = _read_regular_file(
config_path,
field="reference adapter configuration",
maximum_bytes=MAX_REFERENCE_CONFIG_BYTES,
)
try:
document = cast(dict[str, object], tomllib.loads(raw.decode("utf-8")))
except UnicodeDecodeError as error:
raise DocForgeError(
"invalid_reference_config",
"Reference adapter configuration must be UTF-8",
) from error
except tomllib.TOMLDecodeError as error:
raise DocForgeError(
"invalid_reference_config",
f"Invalid reference adapter configuration: {error}",
) from error
unknown = sorted(set(document) - _CONFIG_FIELDS)
if unknown:
raise DocForgeError(
"invalid_reference_config",
"Reference adapter configuration has unknown fields",
fields=unknown,
)
if document.get("schema_version") != REFERENCE_ADAPTER_SCHEMA_VERSION:
raise DocForgeError(
"invalid_reference_config",
"Reference adapter schema_version must be 1",
)
project_id = _string(document, "project_id")
if ID_PATTERN.fullmatch(project_id) is None:
raise DocForgeError(
"invalid_reference_config",
"Reference adapter project_id is not a stable ID",
project_id=project_id,
)
title = _string(document, "title")
if len(title) > 256:
raise DocForgeError(
"invalid_reference_config",
"Reference adapter title exceeds its size limit",
maximum_characters=256,
)
raw_language = _string(document, "language")
if raw_language not in REFERENCE_LANGUAGES:
raise DocForgeError(
"unsupported_reference_language",
"Reference adapter language is not supported",
language=raw_language,
supported=list(REFERENCE_LANGUAGES),
)
language = raw_language
source_roots = _source_roots(root, document.get("source_roots"))
raw_compilation_database = document.get("compilation_database")
compilation_database: Path | None = None
if language == "cpp":
if raw_compilation_database is None:
raise DocForgeError(
"invalid_reference_config",
"C++ reference adapters require compilation_database",
)
compilation_database = _confined_path(
root,
raw_compilation_database,
field="compilation_database",
expected="file",
)
elif raw_compilation_database is not None:
raise DocForgeError(
"invalid_reference_config",
"compilation_database is allowed only for the C++ reference adapter",
language=language,
)
_validate_language_inventory(source_roots, language)
return ReferenceAdapterConfigV1(
schema_version=1,
project_id=project_id,
title=title,
language=language,
source_roots=source_roots,
compilation_database=compilation_database,
project_root=root,
config_path=config_path,
config_hash=hashlib.sha256(raw).hexdigest(),
)
def _project_root(raw: str | Path) -> Path:
path = Path(raw)
if path.is_symlink():
raise DocForgeError("path_escape", "Reference project root must not be a symlink")
try:
root = path.resolve(strict=True)
except (OSError, ValueError) as error:
raise DocForgeError(
"missing_path",
"Reference project root does not exist",
path=str(path),
) from error
if not root.is_dir():
raise DocForgeError("invalid_path", "Reference project root must be a directory")
return root
def _read_regular_file(path: Path, *, field: str, maximum_bytes: int) -> bytes:
try:
before = path.lstat()
except OSError as error:
raise DocForgeError("missing_path", f"{field} does not exist", path=str(path)) from error
if stat.S_ISLNK(before.st_mode) or not stat.S_ISREG(before.st_mode):
raise DocForgeError("invalid_path", f"{field} must be a regular non-symlink file")
if before.st_size > maximum_bytes:
raise DocForgeError(
"invalid_reference_config",
f"{field} exceeds its size limit",
maximum_bytes=maximum_bytes,
)
def identity(value: os.stat_result) -> tuple[int, int, int, int, int, int]:
return (
value.st_dev,
value.st_ino,
value.st_mode,
value.st_size,
value.st_mtime_ns,
value.st_ctime_ns,
)
flags = os.O_RDONLY | getattr(os, "O_CLOEXEC", 0) | getattr(os, "O_NOFOLLOW", 0)
try:
descriptor = os.open(path, flags)
except OSError as error:
raise DocForgeError(
"reference_config_changed",
f"{field} changed while it was opened",
) from error
try:
opened = os.fstat(descriptor)
chunks: list[bytes] = []
remaining = maximum_bytes + 1
while remaining:
chunk = os.read(descriptor, min(remaining, 65_536))
if not chunk:
break
chunks.append(chunk)
remaining -= len(chunk)
raw = b"".join(chunks)
opened_after = os.fstat(descriptor)
except OSError as error:
raise DocForgeError(
"reference_config_changed",
f"{field} changed while it was read",
) from error
finally:
os.close(descriptor)
try:
after = path.lstat()
except OSError as error:
raise DocForgeError(
"reference_config_changed",
f"{field} changed while it was read",
) from error
if (
identity(before) != identity(opened)
or identity(opened) != identity(opened_after)
or identity(opened_after) != identity(after)
or len(raw) != opened_after.st_size
or len(raw) > maximum_bytes
):
raise DocForgeError(
"reference_config_changed",
f"{field} changed while it was read",
)
return raw
def _string(document: dict[str, object], field: str) -> str:
value = document.get(field)
if not isinstance(value, str) or not value or value != value.strip():
raise DocForgeError(
"invalid_reference_config",
f"Reference adapter {field} must be a non-empty normalized string",
)
return value
def _source_roots(root: Path, raw: object) -> tuple[Path, ...]:
if not isinstance(raw, list) or not raw:
raise DocForgeError(
"invalid_reference_config",
"Reference adapter source_roots must be a non-empty string list",
)
values = cast(list[object], raw)
if len(values) > 64:
raise DocForgeError(
"invalid_reference_config",
"Reference adapter source_roots exceeds its item limit",
maximum_items=64,
)
paths = tuple(
_confined_path(root, value, field="source_roots", expected="directory") for value in values
)
if len(paths) != len(set(paths)):
raise DocForgeError(
"invalid_reference_config",
"Reference adapter source_roots contains duplicates",
)
ordered = tuple(sorted(paths))
if paths != ordered:
raise DocForgeError(
"invalid_reference_config",
"Reference adapter source_roots must be sorted",
)
private_root = root / ".docforge"
if any(
path == private_root
or path.is_relative_to(private_root)
or any(
path.is_relative_to(other) or other.is_relative_to(path)
for other in ordered
if other != path
)
for path in ordered
):
raise DocForgeError(
"invalid_reference_config",
"Reference adapter source_roots must be non-overlapping and outside .docforge",
)
return ordered
def _confined_path(root: Path, raw: object, *, field: str, expected: str) -> Path:
if not isinstance(raw, str) or not raw or raw != raw.strip():
raise DocForgeError(
"invalid_reference_config",
f"{field} must be a non-empty normalized relative path",
)
relative = Path(raw)
if relative.is_absolute() or ".." in relative.parts or relative == Path("."):
raise DocForgeError("path_escape", f"{field} must stay inside the project root", path=raw)
candidate = root / relative
if candidate.is_symlink():
raise DocForgeError("path_escape", f"{field} must not be a symlink", path=raw)
try:
resolved = candidate.resolve(strict=True)
except (OSError, ValueError) as error:
raise DocForgeError("missing_path", f"{field} does not exist", path=raw) from error
if resolved != candidate or not resolved.is_relative_to(root):
raise DocForgeError(
"path_escape",
f"{field} must not traverse symlinks or leave the project root",
path=raw,
)
if expected == "directory" and not resolved.is_dir():
raise DocForgeError("invalid_path", f"{field} must identify a directory", path=raw)
if expected == "file" and (not resolved.is_file() or resolved.is_symlink()):
raise DocForgeError(
"invalid_path",
f"{field} must identify a regular non-symlink file",
path=raw,
)
return resolved
def _validate_language_inventory(
source_roots: tuple[Path, ...],
language: ReferenceLanguage,
) -> None:
expected = _LANGUAGE_SUFFIXES[language]
selected = 0
examined = 0
other_languages: set[str] = set()
for source_root in source_roots:
for directory, names, files in os.walk(
source_root,
followlinks=False,
onerror=_inventory_error,
):
base = Path(directory)
for name in (*names, *files):
examined += 1
if examined > MAX_REFERENCE_INVENTORY_ENTRIES:
raise DocForgeError(
"reference_inventory_too_large",
"Reference source inventory exceeds its entry limit",
maximum_entries=MAX_REFERENCE_INVENTORY_ENTRIES,
)
candidate = base / name
if candidate.is_symlink():
raise DocForgeError(
"path_escape",
"Reference source roots must not contain symlinks",
path=candidate.as_posix(),
)
for name in files:
suffix = Path(name).suffix
if suffix in expected:
selected += 1
elif suffix in _KNOWN_SOURCE_SUFFIXES:
other_languages.add(suffix)
if selected == 0:
raise DocForgeError(
"reference_language_mismatch",
"Declared source roots contain no sources for the selected language",
language=language,
observed_suffixes=sorted(other_languages),
)
def _inventory_error(error: OSError) -> None:
raise DocForgeError(
"invalid_path",
"Reference source inventory cannot be read safely",
path=error.filename,
) from error

View file

@ -0,0 +1,196 @@
"""Runnable, read-only MCP binding for fixed in-repository reference adapters."""
from __future__ import annotations
import argparse
import importlib
from pathlib import Path
from typing import cast
from mcp.server.fastmcp import FastMCP
from .adapter_sdk import (
AdapterImplementation,
AdapterLoader,
AdapterProject,
AdapterProjectSettings,
)
from .errors import DocForgeError
from .mcp_server import create_read_only_server
from .reference_config import ReferenceAdapterConfigV1, load_reference_adapter_config
REFERENCE_MCP_MODULE = "docforge.reference_mcp"
_FIXED_PROVIDERS = {
"python": ("docforge.adapters.python", "PythonReferenceAdapter"),
"javascript": ("docforge.adapters.javascript", "JavaScriptReferenceAdapter"),
"typescript": ("docforge.adapters.javascript", "JavaScriptReferenceAdapter"),
"cpp": ("docforge.adapters.cpp", "CppReferenceAdapter"),
}
__all__ = [
"REFERENCE_MCP_MODULE",
"create_reference_project",
"create_reference_server",
"main",
]
def create_reference_project(project_root: str | Path) -> AdapterProject:
"""Construct one configured adapter project from fixed internal providers."""
config = load_reference_adapter_config(project_root)
return _project_from_config(config)
def create_reference_server(
project_root: str | Path,
*,
no_ast: bool = False,
diagnostics: bool = False,
capability_mode: str | None = None,
manual_projection_policy: str | None = None,
portable_graph_policy: str | None = None,
live_viewer_policy: str | None = None,
) -> FastMCP:
"""Create the fixed read-only MCP surface for one reference project."""
config = load_reference_adapter_config(project_root)
project = _project_from_config(config)
return create_read_only_server(
project,
binding_metadata={
"server_module": REFERENCE_MCP_MODULE,
"adapter_mode": "reference",
"reference_language": config.language,
"reference_config_hash": config.config_hash,
},
no_ast=no_ast,
diagnostics=diagnostics,
capability_mode=capability_mode,
manual_projection_policy=manual_projection_policy,
portable_graph_policy=portable_graph_policy,
live_viewer_policy=live_viewer_policy,
)
def _project_from_config(config: ReferenceAdapterConfigV1) -> AdapterProject:
loader = _adapter_loader(config)
cache_root = config.project_root / ".docforge" / "cache" / "reference-adapter" / config.language
return AdapterProject(
loader,
cache_root=cache_root,
settings=AdapterProjectSettings(
descriptor_path=config.config_path,
implementation=AdapterImplementation(files=(config.config_path,)),
),
)
def _adapter_loader(config: ReferenceAdapterConfigV1) -> AdapterLoader:
module_name, class_name = _FIXED_PROVIDERS[config.language]
try:
module = importlib.import_module(module_name)
except ModuleNotFoundError as error:
if error.name != module_name:
raise DocForgeError(
"optional_dependency_missing",
"The selected reference adapter dependency is not installed",
language=config.language,
missing_module=error.name,
install=_install_target(config.language),
) from error
raise DocForgeError(
"reference_adapter_unavailable",
"The selected fixed reference adapter is not available",
language=config.language,
) from error
constructor = getattr(module, class_name, None)
if not callable(constructor):
raise DocForgeError(
"reference_adapter_unavailable",
"The selected fixed reference adapter is not available",
language=config.language,
)
arguments: dict[str, object] = {
"source_roots": tuple(
path.relative_to(config.project_root).as_posix() for path in config.source_roots
),
"project_id": config.project_id,
"title": config.title,
}
if config.language == "cpp":
if config.compilation_database is None:
raise DocForgeError(
"invalid_reference_config",
"C++ reference adapters require compilation_database",
)
arguments["compilation_database"] = config.compilation_database.relative_to(
config.project_root
).as_posix()
loader = constructor(config.project_root, **arguments)
if not callable(getattr(loader, "load_projection", None)):
raise DocForgeError(
"invalid_adapter",
"The selected fixed provider does not implement the adapter contract",
language=config.language,
)
return cast(AdapterLoader, loader)
def _install_target(language: str) -> str:
if language in {"javascript", "typescript"}:
return f"docforge[{language}]"
if language == "cpp":
return "docforge[cpp]"
return "docforge"
def main() -> None:
"""Run the fixed reference binding over stdio."""
parser = argparse.ArgumentParser(prog="python -m docforge.reference_mcp")
parser.add_argument("--project-root", type=Path, required=True)
parser.add_argument(
"--no-ast",
action="store_true",
help=(
"Preserve the existing adapter and forbid AST, Tree-sitter, compiler-AST, "
"and function-Logic extraction changes"
),
)
parser.add_argument(
"--diagnostics",
action="store_true",
help="Attach bounded request-local stage timings and counters",
)
parser.add_argument(
"--capability-mode",
choices=("read",),
help="Expose the fixed read-only project-bound capability surface",
)
parser.add_argument(
"--manual-render-policy",
choices=("auto", "explicit", "disabled"),
)
parser.add_argument(
"--portable-graph-policy",
choices=("explicit", "disabled"),
)
parser.add_argument(
"--live-viewer-policy",
choices=("on-demand", "disabled"),
)
arguments = parser.parse_args()
create_reference_server(
arguments.project_root,
no_ast=arguments.no_ast,
diagnostics=arguments.diagnostics,
capability_mode=arguments.capability_mode,
manual_projection_policy=arguments.manual_render_policy,
portable_graph_policy=arguments.portable_graph_policy,
live_viewer_policy=arguments.live_viewer_policy,
).run(transport="stdio")
if __name__ == "__main__":
main()

View file

@ -9,18 +9,20 @@ data; it is never imported, compiled, or executed.
from __future__ import annotations
import hashlib
import importlib
from collections.abc import Iterable
from dataclasses import dataclass
from functools import lru_cache
import tree_sitter_cpp
import tree_sitter_javascript
from tree_sitter import Language, Node, Parser
from typing import TYPE_CHECKING
from .errors import DocForgeError
from .models import LogicEdge, LogicNode, LogicProjection
if TYPE_CHECKING:
from tree_sitter import Language, Node
_COMMENT_NODE_TYPES = frozenset({"comment"})
_SCRIPT_LANGUAGES = frozenset({"javascript", "typescript"})
@dataclass(frozen=True)
@ -79,7 +81,34 @@ class _LanguageProfile:
def _javascript_profile() -> _LanguageProfile:
return _LanguageProfile(
name="javascript",
language=Language(tree_sitter_javascript.language()),
language=_optional_language(
grammar_module="tree_sitter_javascript",
grammar_function="language",
extra="javascript",
),
root_type="program",
block_types=frozenset({"program", "statement_block"}),
function_types=frozenset(
{"function_declaration", "generator_function_declaration", "method_definition"}
),
loop_types=frozenset(
{"while_statement", "do_statement", "for_statement", "for_in_statement"}
),
return_types=frozenset({"return_statement"}),
raise_types=frozenset({"throw_statement"}),
switch_case_types=frozenset({"switch_case", "switch_default"}),
)
@lru_cache(maxsize=1)
def _typescript_profile() -> _LanguageProfile:
return _LanguageProfile(
name="typescript",
language=_optional_language(
grammar_module="tree_sitter_typescript",
grammar_function="language_typescript",
extra="typescript",
),
root_type="program",
block_types=frozenset({"program", "statement_block"}),
function_types=frozenset(
@ -98,7 +127,11 @@ def _javascript_profile() -> _LanguageProfile:
def _cpp_profile() -> _LanguageProfile:
return _LanguageProfile(
name="cpp",
language=Language(tree_sitter_cpp.language()),
language=_optional_language(
grammar_module="tree_sitter_cpp",
grammar_function="language",
extra="cpp",
),
root_type="translation_unit",
block_types=frozenset({"translation_unit", "compound_statement"}),
function_types=frozenset({"function_definition"}),
@ -116,6 +149,28 @@ def _cpp_profile() -> _LanguageProfile:
)
def _optional_language(
*,
grammar_module: str,
grammar_function: str,
extra: str,
) -> Language:
try:
tree_sitter = importlib.import_module("tree_sitter")
grammar = importlib.import_module(grammar_module)
except ModuleNotFoundError as error:
raise DocForgeError(
"optional_dependency_missing",
"The requested language frontend is not installed",
extra=extra,
install=f"docforge[{extra}]",
missing_module=error.name,
) from error
language_type = tree_sitter.Language
language_factory = getattr(grammar, grammar_function)
return language_type(language_factory())
def analyze_javascript_source(
source: str,
*,
@ -136,12 +191,38 @@ def analyze_javascript_source(
)
def analyze_typescript_source(
source: str,
*,
source_id: str,
owners: Iterable[TreeSitterLogicOwner],
filename: str = "<typescript-source>",
max_nodes_per_function: int = 2_000,
) -> tuple[LogicProjection, ...]:
"""Build control-flow projections for named TypeScript functions and methods."""
return _analyze_tree_sitter_source(
source,
source_id=source_id,
owners=owners,
filename=filename,
profile=_typescript_profile(),
max_nodes_per_function=max_nodes_per_function,
)
def discover_javascript_functions(source: str) -> tuple[DiscoveredFunction, ...]:
"""Return named JavaScript functions, methods, and assigned arrow functions."""
return _discover_functions(source, _javascript_profile())
def discover_typescript_functions(source: str) -> tuple[DiscoveredFunction, ...]:
"""Return named TypeScript functions, methods, and assigned arrow functions."""
return _discover_functions(source, _typescript_profile())
def discover_cpp_functions(source: str) -> tuple[DiscoveredFunction, ...]:
"""Return named C++ functions and methods."""
@ -172,6 +253,8 @@ def _discover_functions(
source: str,
profile: _LanguageProfile,
) -> tuple[DiscoveredFunction, ...]:
from tree_sitter import Parser
raw = source.encode("utf-8")
parser = Parser(profile.language)
tree = parser.parse(raw)
@ -212,6 +295,8 @@ def _analyze_tree_sitter_source(
profile: _LanguageProfile,
max_nodes_per_function: int,
) -> tuple[LogicProjection, ...]:
from tree_sitter import Parser
if max_nodes_per_function < 2:
raise ValueError("max_nodes_per_function must allow entry and exit nodes")
raw = source.encode("utf-8")
@ -293,7 +378,7 @@ def _function_definitions(
def _scope_name(node: Node, raw: bytes, profile: _LanguageProfile) -> str | None:
if profile.name == "javascript" and node.type in {"class_declaration", "class"}:
if profile.name in _SCRIPT_LANGUAGES and node.type in {"class_declaration", "class"}:
return _field_text(node, "name", raw)
if profile.name == "cpp" and node.type in {
"namespace_definition",
@ -307,11 +392,11 @@ def _scope_name(node: Node, raw: bytes, profile: _LanguageProfile) -> str | None
def _function_name(node: Node, raw: bytes, profile: _LanguageProfile) -> str | None:
if node.type in profile.function_types:
if profile.name == "javascript":
if profile.name in _SCRIPT_LANGUAGES:
return _field_text(node, "name", raw)
declarator = node.child_by_field_name("declarator")
return _declarator_name(declarator, raw) if declarator is not None else None
if profile.name != "javascript" or node.type != "variable_declarator":
if profile.name not in _SCRIPT_LANGUAGES or node.type != "variable_declarator":
return None
value = node.child_by_field_name("value")
if value is None or value.type not in {"arrow_function", "function_expression"}:

View file

@ -0,0 +1,19 @@
[
{
"arguments": [
"c++",
"-std=c++20",
"-Iinclude",
"-c",
"src/main.cpp"
],
"directory": ".",
"file": "src/main.cpp"
},
{
"command": "c++ -std=c++20 -Iinclude -c src/worker.cpp",
"directory": ".",
"file": "src/worker.cpp",
"output": "worker.o"
}
]

View file

@ -0,0 +1,15 @@
#pragma once
#include "detail.hpp"
namespace demo {
struct Config {
int limit;
};
inline int effective_limit(const Config& config) {
return normalize(config.limit);
}
} // namespace demo

View file

@ -0,0 +1,12 @@
#pragma once
namespace demo {
inline int normalize(int value) {
if (value < 0) {
return 0;
}
return value;
}
} // namespace demo

View file

@ -0,0 +1,17 @@
#include "../include/config.hpp"
namespace demo {
int execute(int value) {
Config config{value};
if (value == 0) {
return effective_limit(config);
}
return normalize(value);
}
} // namespace demo
int main() {
return demo::execute(3);
}

View file

@ -0,0 +1,17 @@
#include "../include/config.hpp"
namespace demo {
class Worker {
public:
int run(int value) {
for (int attempt = 0; attempt < 2; ++attempt) {
if (value > attempt) {
return effective_limit(Config{value});
}
}
return 0;
}
};
} // namespace demo

View file

@ -0,0 +1,8 @@
import { Service } from "./service.mjs";
export { clamp } from "./shared.js";
throw new Error("The reference adapter must never execute project code");
export function applicationName() {
return Service.name;
}

View file

@ -0,0 +1,14 @@
import { clamp } from "./shared.js";
export class Service {
run(value) {
return clamp(value);
}
}
export const buildService = (enabled) => {
if (enabled) {
return new Service();
}
return null;
};

View file

@ -0,0 +1,11 @@
export const DEFAULT_LIMIT = 3;
export function clamp(value, limit = DEFAULT_LIMIT) {
if (value < 0) {
return 0;
}
if (value > limit) {
return limit;
}
return value;
}

View file

@ -0,0 +1,5 @@
export { Service } from "./service.mjs";
export function execute(worker, value) {
return worker.run(value);
}

View file

@ -0,0 +1,7 @@
"""Small multi-file package for the Python reference adapter."""
from .service import Service
def package_name() -> str:
return Service.__name__

View file

@ -0,0 +1,18 @@
"""Service behavior with class, method, function, and local import evidence."""
import json
from .shared import clamp
raise RuntimeError("The reference adapter must never execute project code")
class Service:
def run(self, value: int) -> int:
return clamp(value) if json.__name__ else value
async def build_service(enabled: bool) -> Service | None:
if enabled:
return Service()
return None

View file

@ -0,0 +1,11 @@
"""Shared values used by dependent modules."""
DEFAULT_LIMIT = 3
def clamp(value: int, limit: int = DEFAULT_LIMIT) -> int:
if value < 0:
return 0
if value > limit:
return limit
return value

View file

@ -0,0 +1,8 @@
"""A transitive dependent used to prove reverse invalidation."""
from sample.service import Service
def execute(value: int) -> int:
worker = Service()
return worker.run(value)

View file

@ -0,0 +1,8 @@
import { Service } from "./service";
export type { Choice } from "./types";
throw new Error("The reference adapter must never execute project code");
export function applicationName(): string {
return Service.name;
}

View file

@ -0,0 +1,17 @@
import type { Choice } from "./types";
export class Service {
run(choice: Choice): number {
if (choice.enabled) {
return choice.value;
}
return 0;
}
}
export const buildService = (choice: Choice): Service | null => {
if (choice.enabled) {
return new Service();
}
return null;
};

View file

@ -0,0 +1,6 @@
export interface Choice {
enabled: boolean;
value: number;
}
export const DEFAULT_LIMIT: number = 3;

View file

@ -0,0 +1,6 @@
import { Service } from "./service";
import type { Choice } from "./types";
export function execute(worker: Service, choice: Choice): number {
return worker.run(choice);
}

View file

@ -39,6 +39,7 @@ from docforge.mcp_server import (
)
from docforge.models import (
Edge,
Limits,
LogicEdge,
LogicNode,
LogicProjection,
@ -248,6 +249,36 @@ class NonLogicIncrementalLoader(IncrementalLoader):
return replace(super().extract_source(source), logic=())
class CompleteLogicIncrementalLoader(IncrementalLoader):
"""Supply an independent complete Logic oracle alongside the legacy graph oracle."""
def __init__(self, root: Path, *, mismatched_logic: bool = False) -> None:
super().__init__(root)
self.mismatched_logic = mismatched_logic
def load_complete_assembly(self) -> AdapterAssembly:
projection = super().load_projection()
if "guide.workflow" not in self.sources:
return AdapterAssembly(projection)
manifest = self.load_manifest()
workflow = next(
source for source in manifest.sources if source.source_id == "guide.workflow"
)
logic = self.extract_source(workflow).logic
if self.mismatched_logic:
current = logic[0]
logic = (
replace(
current,
nodes=(
current.nodes[0],
replace(current.nodes[1], label="Different complete return"),
),
),
)
return AdapterAssembly(projection, logic)
class AdapterContractTests(unittest.TestCase):
def projection(self, root: Path) -> AdapterProjection:
foundation = Node(
@ -311,6 +342,23 @@ class AdapterContractTests(unittest.TestCase):
self.assertEqual("Workflow", index.get_node("guide.workflow")["node"]["title"])
self.assertEqual(projection.identity(), projection.identity())
def test_adapter_project_enforces_graph_bounds_before_publication(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
project = AdapterProject(
Loader(self.projection(root)),
cache_root=root / ".cache" / "bounded",
settings=AdapterProjectSettings(limits=Limits(max_nodes=1)),
)
with self.assertRaises(DocForgeError) as captured:
project.load()
self.assertEqual("adapter_limit", captured.exception.code)
self.assertEqual("nodes", captured.exception.details["kind"])
self.assertEqual(1, captured.exception.details["maximum"])
self.assertEqual(2, captured.exception.details["actual"])
def test_projection_rejects_unsorted_metadata_graph_and_identity_changes(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
@ -782,6 +830,46 @@ class AdapterContractTests(unittest.TestCase):
self.assertEqual("ok", equivalent["status"])
self.assertEqual(1, equivalent["node_count"])
def test_logic_equivalence_requires_an_independent_complete_oracle(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
project = AdapterProject(
IncrementalLoader(root),
cache_root=root / ".cache" / "logic-without-oracle",
)
with self.assertRaises(DocForgeError) as captured:
project.verify_incremental_equivalence()
self.assertEqual("complete_logic_oracle_required", captured.exception.code)
def test_logic_equivalence_uses_the_complete_assembly_oracle(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
project = AdapterProject(
CompleteLogicIncrementalLoader(root),
cache_root=root / ".cache" / "logic-oracle",
)
result = project.verify_incremental_equivalence()
self.assertEqual("ok", result["status"])
self.assertEqual(1, result["logic_projection_count"])
def test_logic_equivalence_reports_exact_mismatch_field(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
project = AdapterProject(
CompleteLogicIncrementalLoader(root, mismatched_logic=True),
cache_root=root / ".cache" / "logic-mismatch",
)
with self.assertRaises(DocForgeError) as captured:
project.verify_incremental_equivalence()
self.assertEqual("incremental_mismatch", captured.exception.code)
self.assertEqual(["logic"], captured.exception.details["fields"])
def test_incremental_adapter_can_assemble_overlapping_source_evidence(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()

View file

@ -0,0 +1,475 @@
from __future__ import annotations
import asyncio
import json
import os
import shutil
import subprocess
import sys
import tempfile
import tomllib
import unittest
import venv
from dataclasses import replace
from pathlib import Path
from unittest import mock
from jsonschema import Draft202012Validator
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from referencing import Registry, Resource
from docforge.adapter_contract import AdapterNode, AdapterProject, AdapterProjection
from docforge.adapter_launcher import AdapterLauncherV1
from docforge.client_config import (
_validate_adapter_configuration_result,
generate_adapter_client_configuration,
generate_client_configuration,
)
from docforge.errors import DocForgeError
from docforge.models import Node
from docforge.project import Project
from docforge.reference_mcp import (
REFERENCE_MCP_MODULE,
create_reference_project,
)
ROOT = Path(__file__).resolve().parents[1]
FIXTURES = ROOT / "tests" / "fixtures"
SCHEMAS = ROOT / "schemas"
LAUNCHER_SCHEMA = json.loads((SCHEMAS / "adapter-launcher.schema.json").read_text(encoding="utf-8"))
ADAPTER_CONFIGURATION_SCHEMA = json.loads(
(SCHEMAS / "adapter-client-configuration.schema.json").read_text(encoding="utf-8")
)
GENERIC_CONFIGURATION_SCHEMA = json.loads(
(SCHEMAS / "client-configuration.schema.json").read_text(encoding="utf-8")
)
SCHEMA_REGISTRY = Registry().with_resources(
(
(LAUNCHER_SCHEMA["$id"], Resource.from_contents(LAUNCHER_SCHEMA)),
(
GENERIC_CONFIGURATION_SCHEMA["$id"],
Resource.from_contents(GENERIC_CONFIGURATION_SCHEMA),
),
(
ADAPTER_CONFIGURATION_SCHEMA["$id"],
Resource.from_contents(ADAPTER_CONFIGURATION_SCHEMA),
),
)
)
ADAPTER_CONFIGURATION_VALIDATOR = Draft202012Validator(
ADAPTER_CONFIGURATION_SCHEMA,
registry=SCHEMA_REGISTRY,
)
class Loader:
def __init__(self, projection: AdapterProjection) -> None:
self.projection = projection
self.load_calls = 0
def load_projection(self) -> AdapterProjection:
self.load_calls += 1
return self.projection
class DriftingLoader(Loader):
def __init__(
self,
projection: AdapterProjection,
changed: AdapterProjection,
) -> None:
super().__init__(projection)
self.changed = changed
def load_projection(self) -> AdapterProjection:
self.load_calls += 1
return self.projection if self.load_calls < 3 else self.changed
class AdapterLauncherTests(unittest.TestCase):
def copy_fixture(self, destination: Path) -> Path:
root = destination / "alpha"
shutil.copytree(FIXTURES / "alpha", root)
return root
def copy_reference_fixture(self, destination: Path) -> Path:
root = destination / "reference-python"
shutil.copytree(FIXTURES / "reference-python", root)
config = root / ".docforge" / "reference-adapter.toml"
config.parent.mkdir(parents=True)
config.write_text(
"\n".join(
(
"schema_version = 1",
'project_id = "reference-python"',
'title = "Runnable Python reference"',
'language = "python"',
'source_roots = ["src"]',
"",
)
),
encoding="utf-8",
)
return root.resolve()
def projection(
self,
root: Path,
*,
source_hash: str = "3" * 64,
) -> AdapterProjection:
node = Node(
node_id="source.entry",
title="Entry",
family="source",
authority="derived",
status="active",
tags=("python",),
summary="Reference adapter entry.",
content="Reference adapter content.",
source_path="src/entry.py",
source_anchor="L1",
content_hash="2" * 64,
)
return AdapterProjection(
project_id="adapter-client-fixture",
title="Adapter client fixture",
adapter_id="fixture-client",
adapter_version="1",
root=root,
revision="fixture-revision",
source_hash=source_hash,
nodes=(AdapterNode(node),),
edges=(),
)
def project(
self,
root: Path,
*,
loader: Loader | None = None,
) -> tuple[AdapterProject, Loader]:
effective_loader = loader or Loader(self.projection(root))
return (
AdapterProject(
effective_loader,
cache_root=root / ".docforge-cache" / "adapter-client",
),
effective_loader,
)
def test_launcher_and_all_client_fragments_are_deterministic_and_schema_valid(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
project, _ = self.project(root)
launcher = AdapterLauncherV1.for_project(
project,
module=REFERENCE_MCP_MODULE,
)
Draft202012Validator(LAUNCHER_SCHEMA).validate(launcher.as_dict())
previous = os.environ.get("DOCFORGE_ADAPTER_LAUNCHER_SECRET")
os.environ["DOCFORGE_ADAPTER_LAUNCHER_SECRET"] = "must-not-appear"
try:
with mock.patch("docforge.client_config.subprocess.run") as executed:
results: dict[str, dict[str, object]] = {}
for client in ("codex", "claude", "openclaw"):
first = generate_adapter_client_configuration(
project,
launcher,
client,
no_ast=True,
)
second = generate_adapter_client_configuration(
project,
launcher,
client,
no_ast=True,
)
self.assertEqual(first, second)
ADAPTER_CONFIGURATION_VALIDATOR.validate(first)
self.assertEqual(launcher.launcher_hash, first["launcher_hash"])
self.assertEqual(
launcher.launcher_hash,
first["binding"]["launcher_hash"],
)
self.assertEqual(
[
"-I",
"-m",
REFERENCE_MCP_MODULE,
"--project-root",
str(root),
"--capability-mode",
"read",
"--no-ast",
],
first["binding"]["args"],
)
self.assertEqual({}, first["binding"]["environment"])
self.assertNotIn("cwd", first["binding"])
self.assertNotIn(
"must-not-appear",
json.dumps(first, sort_keys=True),
)
results[client] = first
executed.assert_not_called()
finally:
if previous is None:
os.environ.pop("DOCFORGE_ADAPTER_LAUNCHER_SECRET", None)
else:
os.environ["DOCFORGE_ADAPTER_LAUNCHER_SECRET"] = previous
self.assertIn(
"mcp_servers",
tomllib.loads(results["codex"]["artifact"]["content"]),
)
self.assertIn(
"mcpServers",
json.loads(results["claude"]["artifact"]["content"]),
)
self.assertIn(
"mcp",
json.loads(results["openclaw"]["artifact"]["content"]),
)
def test_launcher_rejects_module_path_entry_point_and_argument_injection(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
project, _ = self.project(root)
launcher = AdapterLauncherV1.for_project(
project,
module=REFERENCE_MCP_MODULE,
)
for module in (
"-m",
"fixture_adapter.mcp_server",
"fixture_adapter.mcp_server --debug",
"fixture_adapter:mcp_server",
"fixture_adapter/mcp_server",
".fixture_adapter",
"docforge.mcp_server",
"fixture_adapter.$server",
):
with self.subTest(module=module), self.assertRaises(DocForgeError) as captured:
replace(launcher, module=module)
self.assertEqual("invalid_adapter_launcher", captured.exception.code)
with self.assertRaises(DocForgeError):
replace(launcher, project_root=Path("relative/project"))
with self.assertRaises(DocForgeError):
replace(launcher, entry_point="console-script") # type: ignore[arg-type]
payload = launcher.as_dict()
payload["arguments"] = ["--shell", "command"]
with self.assertRaises(TypeError):
AdapterLauncherV1(**payload) # type: ignore[arg-type]
result = generate_adapter_client_configuration(project, launcher, "codex")
result["binding"]["args"].append("--arbitrary")
with self.assertRaisesRegex(AssertionError, "Generated adapter client"):
_validate_adapter_configuration_result(
result,
project=project,
launcher=launcher,
source_availability=project_source_availability(result),
)
def test_wrong_project_descriptor_and_source_drift_fail_closed(self) -> None:
with tempfile.TemporaryDirectory() as directory:
parent = Path(directory)
first_root = (parent / "first").resolve()
second_root = (parent / "second").resolve()
first_root.mkdir()
second_root.mkdir()
first, _ = self.project(first_root)
second, _ = self.project(second_root)
launcher = AdapterLauncherV1.for_project(
first,
module=REFERENCE_MCP_MODULE,
)
with self.assertRaises(DocForgeError) as wrong_project:
generate_adapter_client_configuration(second, launcher, "codex")
self.assertEqual("adapter_launcher_mismatch", wrong_project.exception.code)
with self.assertRaises(DocForgeError) as descriptor_drift:
generate_adapter_client_configuration(
first,
replace(launcher, descriptor_hash="0" * 64),
"codex",
)
self.assertEqual("adapter_launcher_mismatch", descriptor_drift.exception.code)
base = self.projection(first_root)
changed = replace(base, source_hash="4" * 64)
drifting, _ = self.project(
first_root,
loader=DriftingLoader(base, changed),
)
drifting_launcher = AdapterLauncherV1.for_project(
drifting,
module=REFERENCE_MCP_MODULE,
)
with self.assertRaises(DocForgeError) as source_drift:
generate_adapter_client_configuration(
drifting,
drifting_launcher,
"openclaw",
)
self.assertEqual("source_changed", source_drift.exception.code)
def test_generic_configuration_remains_separate_and_compatible(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
generic = Project.open(root)
result = generate_client_configuration(generic, "codex", no_ast=True)
Draft202012Validator(GENERIC_CONFIGURATION_SCHEMA).validate(result)
self.assertEqual("client.configure", result["operation"])
self.assertNotIn("launcher", result)
with self.assertRaises(DocForgeError) as generic_launcher:
AdapterLauncherV1.for_project(
generic,
module=REFERENCE_MCP_MODULE,
)
self.assertEqual("adapter_launcher_unavailable", generic_launcher.exception.code)
custom_root = (Path(directory) / "custom").resolve()
custom_root.mkdir()
custom, _ = self.project(custom_root)
with self.assertRaises(DocForgeError) as generic_api:
generate_client_configuration(custom, "codex")
self.assertEqual("missing_config", generic_api.exception.code)
def test_installed_project_owned_top_level_module_launches_without_probe_execution(
self,
) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
environment = root / ".launcher-venv"
venv.EnvBuilder(with_pip=False).create(environment)
executable = environment / "bin" / "python"
site_packages = Path(
subprocess.run(
[
str(executable),
"-I",
"-c",
"import site; print(site.getsitepackages()[0])",
],
check=True,
capture_output=True,
text=True,
).stdout.strip()
)
(site_packages / "fixture-adapter.pth").write_text(
f"{root}\n",
encoding="utf-8",
)
sentinel = root / "fixture-adapter-executed"
(root / "fixture_adapter.py").write_text(
"\n".join(
(
"import argparse",
"import json",
"from pathlib import Path",
"parser = argparse.ArgumentParser()",
'parser.add_argument("--project-root", required=True)',
(
'parser.add_argument("--capability-mode", '
'choices=("read",), required=True)'
),
"arguments = parser.parse_args()",
f"Path({str(sentinel)!r}).write_text('executed', encoding='utf-8')",
(
"print(json.dumps({'project_root': arguments.project_root}, "
"sort_keys=True))"
),
"",
)
),
encoding="utf-8",
)
project, _ = self.project(root)
with mock.patch.object(sys, "executable", str(executable)):
launcher = AdapterLauncherV1.for_project(
project,
module="fixture_adapter",
)
result = generate_adapter_client_configuration(
project,
launcher,
"codex",
)
self.assertFalse(sentinel.exists())
Draft202012Validator(LAUNCHER_SCHEMA).validate(launcher.as_dict())
binding = result["binding"]
completed = subprocess.run(
[binding["command"], *binding["args"]],
check=True,
capture_output=True,
text=True,
timeout=10,
)
self.assertEqual({"project_root": str(root)}, json.loads(completed.stdout))
self.assertEqual("executed", sentinel.read_text(encoding="utf-8"))
def test_uninstalled_packages_and_non_project_modules_fail_closed(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
project, _ = self.project(root)
for module, code in (
("fixture_adapter", "adapter_launcher_unavailable"),
("os", "adapter_launcher_unavailable"),
("json", "invalid_adapter_launcher"),
):
with self.subTest(module=module), self.assertRaises(DocForgeError) as captured:
AdapterLauncherV1.for_project(project, module=module)
self.assertEqual(code, captured.exception.code)
def test_generated_reference_binding_starts_a_real_isolated_mcp_process(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.copy_reference_fixture(Path(directory))
project = create_reference_project(root)
launcher = AdapterLauncherV1.for_project(
project,
module=REFERENCE_MCP_MODULE,
)
result = generate_adapter_client_configuration(
project,
launcher,
"codex",
)
binding = result["binding"]
async def inspect() -> tuple[str, ...]:
parameters = StdioServerParameters(
command=binding["command"],
args=binding["args"],
env=binding["environment"],
)
async with (
stdio_client(parameters) as streams,
ClientSession(*streams) as session,
):
await session.initialize()
return tuple(tool.name for tool in (await session.list_tools()).tools)
tools = asyncio.run(inspect())
self.assertIn("docforge_bootstrap", tools)
def project_source_availability(result: dict[str, object]):
from docforge.adapter_launcher import AdapterSourceAvailabilityV1
payload = result["source_availability"]
return AdapterSourceAvailabilityV1(
schema_version=payload["schema_version"],
status=payload["status"],
method=payload["method"],
revision=payload["revision"],
source_hash=payload["source_hash"],
)
if __name__ == "__main__":
unittest.main()

261
tests/test_adapter_sdk.py Normal file
View file

@ -0,0 +1,261 @@
from __future__ import annotations
import hashlib
import tempfile
import unittest
from dataclasses import replace
from pathlib import Path
import docforge.adapter_sdk as sdk
from docforge.errors import DocForgeError
class ReferenceIncrementalAdapter:
"""Small public-SDK-only adapter used to prove the conformance entry point."""
def __init__(self, root: Path) -> None:
self.root = root
@staticmethod
def _hash(value: str) -> str:
return hashlib.sha256(value.encode()).hexdigest()
def _node(self) -> sdk.AdapterNode:
content = "A deterministic reference function."
return sdk.AdapterNode(
sdk.Node(
node_id="reference.run",
title="Reference run",
family="source",
authority="derived",
status="current",
tags=("python",),
summary=content,
content=content,
source_path="src/reference.py",
source_anchor="L1",
content_hash=self._hash(content),
)
)
@staticmethod
def _logic() -> sdk.LogicProjection:
return sdk.LogicProjection(
owner_node_id="reference.run",
source_id="source.reference",
nodes=(
sdk.LogicNode("entry", "entry", "Enter run", "L1"),
sdk.LogicNode("return", "return", "Return", "L2"),
),
edges=(sdk.LogicEdge("entry", "return", "return", "RETURN", 0),),
)
def load_manifest(self) -> sdk.AdapterManifest:
fingerprint = self._hash("def run(): return True\n")
return sdk.AdapterManifest(
project_id="sdk-reference",
title="SDK reference",
adapter_id="python-reference",
adapter_version="1",
root=self.root,
revision=fingerprint[:12],
source_hash=fingerprint,
families=("source",),
allowed_relations=(),
sources=(
sdk.AdapterSource(
source_id="source.reference",
source_path="src/reference.py",
fingerprint=fingerprint,
extractor_version="python-reference@1",
),
),
estimated_nodes=1,
)
def extract_source(self, source: sdk.AdapterSource) -> sdk.AdapterSourceProjection:
return sdk.AdapterSourceProjection(
source_id=source.source_id,
fingerprint=source.fingerprint,
nodes=(self._node(),),
edges=(),
logic=(self._logic(),),
)
def load_projection(self) -> sdk.AdapterProjection:
manifest = self.load_manifest()
return sdk.AdapterProjection(
project_id=manifest.project_id,
title=manifest.title,
adapter_id=manifest.adapter_id,
adapter_version=manifest.adapter_version,
root=manifest.root,
revision=manifest.revision,
source_hash=manifest.source_hash,
nodes=(self._node(),),
edges=(),
)
def load_complete_assembly(self) -> sdk.AdapterAssembly:
return sdk.AdapterAssembly(
projection=self.load_projection(),
logic=(self._logic(),),
)
class LegacyAdapter:
def __init__(self, projection: sdk.AdapterProjection) -> None:
self.projection = projection
def load_projection(self) -> sdk.AdapterProjection:
return self.projection
class LogicWithoutCompleteAdapter:
def __init__(self, root: Path) -> None:
self.reference = ReferenceIncrementalAdapter(root)
def load_manifest(self) -> sdk.AdapterManifest:
return self.reference.load_manifest()
def extract_source(self, source: sdk.AdapterSource) -> sdk.AdapterSourceProjection:
return self.reference.extract_source(source)
def load_projection(self) -> sdk.AdapterProjection:
return self.reference.load_projection()
class NondeterministicCompleteAdapter(ReferenceIncrementalAdapter):
def __init__(self, root: Path) -> None:
super().__init__(root)
self.complete_calls = 0
def load_complete_assembly(self) -> sdk.AdapterAssembly:
self.complete_calls += 1
assembly = super().load_complete_assembly()
if self.complete_calls % 2 == 0:
logic = assembly.logic[0]
return replace(
assembly,
logic=(
replace(
logic,
nodes=(
logic.nodes[0],
replace(logic.nodes[1], label="Unstable return"),
),
),
),
)
return assembly
class AdapterSdkTests(unittest.TestCase):
def test_public_facade_contains_the_complete_authoring_surface(self) -> None:
expected = {
"AdapterAssembly",
"AdapterConformanceReport",
"AdapterManifest",
"AdapterNode",
"AdapterProjection",
"AdapterSource",
"AdapterSourceProjection",
"CompleteAdapterAssemblyLoader",
"Edge",
"LogicEdge",
"LogicNode",
"LogicProjection",
"Node",
"verify_adapter_conformance",
}
self.assertLessEqual(expected, set(sdk.__all__))
for name in expected:
with self.subTest(name=name):
self.assertTrue(hasattr(sdk, name))
def test_conformance_proves_deterministic_complete_graph_and_logic_parity(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
report = sdk.verify_adapter_conformance(
ReferenceIncrementalAdapter(root),
cache_root=root / ".cache" / "conformance",
)
self.assertEqual(1, report.schema_version)
self.assertTrue(report.incremental)
self.assertEqual(1, report.node_count)
self.assertEqual(0, report.edge_count)
self.assertEqual(1, report.logic_projection_count)
self.assertEqual(64, len(report.assembly_hash))
self.assertEqual(report.assembly_hash, report.as_dict()["assembly_hash"])
def test_conformance_keeps_load_projection_only_adapters_first_class(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
content = "Legacy adapter content."
projection = sdk.AdapterProjection(
project_id="legacy-sdk",
title="Legacy SDK",
adapter_id="legacy",
adapter_version="1",
root=root,
revision="legacy",
source_hash=hashlib.sha256(content.encode()).hexdigest(),
nodes=(
sdk.AdapterNode(
sdk.Node(
node_id="legacy.node",
title="Legacy node",
family="guide",
authority="authoritative",
status="current",
tags=("legacy",),
summary=content,
content=content,
source_path="docs/legacy.md",
source_anchor=None,
content_hash=hashlib.sha256(content.encode()).hexdigest(),
)
),
),
edges=(),
)
report = sdk.verify_adapter_conformance(
LegacyAdapter(projection),
cache_root=root / ".cache" / "legacy-conformance",
)
self.assertFalse(report.incremental)
self.assertEqual(0, report.logic_projection_count)
self.assertEqual(projection.identity(), projection.identity())
def test_conformance_requires_a_complete_oracle_for_incremental_logic(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
with self.assertRaises(DocForgeError) as captured:
sdk.verify_adapter_conformance(
LogicWithoutCompleteAdapter(root),
cache_root=root / ".cache" / "missing-logic-oracle",
)
self.assertEqual("complete_logic_oracle_required", captured.exception.code)
def test_conformance_rejects_nondeterministic_complete_logic(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
with self.assertRaises(DocForgeError) as captured:
sdk.verify_adapter_conformance(
NondeterministicCompleteAdapter(root),
cache_root=root / ".cache" / "nondeterministic",
)
self.assertEqual("nondeterministic_adapter", captured.exception.code)
self.assertEqual(["logic"], captured.exception.details["fields"])
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,141 @@
from __future__ import annotations
import argparse
import hashlib
import json
import shutil
import tempfile
import unittest
from pathlib import Path
from mcp.shared.memory import create_connected_server_and_client_session
from mcp.types import Tool
from docforge.cli import _parser
from docforge.command_reference import (
cli_command_references,
mcp_tool_references,
render_cli_reference_markdown,
render_command_reference_markdown,
render_mcp_reference_markdown,
)
from docforge.index import ProjectIndex
from docforge.mcp_server import ALL_TOOLS, APPLICATION_TOOLS, create_server
from docforge.project import Project
ROOT = Path(__file__).resolve().parents[1]
FIXTURES = ROOT / "tests" / "fixtures"
class CommandReferenceTests(unittest.IsolatedAsyncioTestCase):
def copy_fixture(self, destination: Path) -> Path:
root = destination / "alpha"
shutil.copytree(FIXTURES / "alpha", root)
return root
def test_cli_reference_is_derived_from_the_real_parser(self) -> None:
parser = _parser()
references = cli_command_references(parser)
subparsers = next(
action
for action in parser._actions # pyright: ignore[reportPrivateUsage]
if isinstance(
action,
argparse._SubParsersAction, # pyright: ignore[reportPrivateUsage]
)
)
expected = {
name: " ".join(command.format_usage().removeprefix("usage: ").split())
for name, command in subparsers.choices.items()
}
self.assertEqual(tuple(sorted(expected)), tuple(item.name for item in references))
self.assertEqual(expected, {item.name: item.invocation for item in references})
rendered = render_cli_reference_markdown(references)
self.assertEqual(rendered, render_cli_reference_markdown(reversed(references)))
self.assertIn("| `search` | `docforge search [-h] [--limit LIMIT] query` |", rendered)
self.assertIn(
"| `graph-render-status` | `docforge graph-render-status [-h] [view_id]` |",
rendered,
)
async def test_mcp_reference_is_derived_from_registered_tool_metadata(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
ProjectIndex(Project.open(root)).build()
async with create_connected_server_and_client_session(
create_server(
root,
"alpha-editor",
canonical_applier_id="alpha-editor",
),
raise_exceptions=True,
) as session:
tools = (await session.list_tools()).tools
expected_names = (*ALL_TOOLS, *APPLICATION_TOOLS)
references = mcp_tool_references(tools, expected_names=expected_names)
by_name = {item.name: item for item in references}
tools_by_name = {tool.name: tool for tool in tools}
self.assertEqual(set(expected_names), set(by_name))
for name, tool in tools_by_name.items():
with self.subTest(tool=name):
schema_hash = hashlib.sha256(
json.dumps(
tool.inputSchema,
sort_keys=True,
separators=(",", ":"),
).encode()
).hexdigest()
properties = set(tool.inputSchema.get("properties", {}))
required = set(tool.inputSchema.get("required", []))
reference = by_name[name]
self.assertEqual(" ".join((tool.description or "").split()), reference.description)
self.assertEqual(tuple(sorted(required)), reference.required_arguments)
self.assertEqual(
tuple(sorted(properties - required)),
reference.optional_arguments,
)
self.assertEqual(schema_hash, reference.input_schema_hash)
rendered = render_mcp_reference_markdown(references)
self.assertEqual(rendered, render_mcp_reference_markdown(reversed(references)))
self.assertIn("| read | `docforge_bootstrap` | — | — |", rendered)
self.assertIn(
"| application | `docforge_apply_changeset` | "
"`changeset_id`, `expected_changeset_hash` | — |",
rendered,
)
complete = render_command_reference_markdown(
cli_command_references(),
references,
)
self.assertEqual(1, complete.count("## CLI commands"))
self.assertEqual(1, complete.count("## MCP tools"))
self.assertTrue(complete.startswith("# DocForge command reference\n"))
self.assertIn("Do not edit this file by hand.", complete)
self.assertIn("fixed reference-adapter server exposes only its read rows", complete)
def test_mcp_reference_fails_closed_on_registration_drift(self) -> None:
known = Tool(
name="docforge_bootstrap",
description="Bootstrap.",
inputSchema={"type": "object", "properties": {}},
)
unknown = Tool(
name="docforge_unregistered",
description="Unknown.",
inputSchema={"type": "object", "properties": {}},
)
with self.assertRaisesRegex(ValueError, "no declared capability surface"):
mcp_tool_references((unknown,))
with self.assertRaisesRegex(ValueError, "missing="):
mcp_tool_references((known,), expected_names=ALL_TOOLS)
with self.assertRaisesRegex(ValueError, "repeats"):
mcp_tool_references((known, known))
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,404 @@
from __future__ import annotations
import asyncio
import contextlib
import hashlib
import io
import os
import tempfile
import threading
import unittest
from pathlib import Path
from unittest import mock
from docforge.mcp_server import ALL_TOOLS, APPLICATION_TOOLS
from tools import generate_command_reference as command_reference_tool
from tools.generate_command_reference import (
EXPECTED_CLI_ROWS,
EXPECTED_MCP_ROWS,
CommandReferenceDrift,
CommandReferenceToolError,
collect_command_reference_rows,
generate_command_reference_bytes,
main,
publish_or_check_command_reference,
)
ROOT = Path(__file__).resolve().parents[1]
ALPHA = ROOT / "tests" / "fixtures" / "alpha"
def _tree_snapshot(root: Path) -> tuple[tuple[str, str, str], ...]:
records: list[tuple[str, str, str]] = []
for path in sorted(root.rglob("*")):
relative = path.relative_to(root).as_posix()
if path.is_symlink():
records.append((relative, "symlink", os.readlink(path)))
elif path.is_dir():
records.append((relative, "directory", ""))
else:
records.append(
(
relative,
"file",
hashlib.sha256(path.read_bytes()).hexdigest(),
)
)
return tuple(records)
class CommandReferenceToolTests(unittest.TestCase):
def test_generated_bytes_are_stable_and_never_mutate_the_fixture(self) -> None:
before = _tree_snapshot(ALPHA)
first = generate_command_reference_bytes(ALPHA)
middle = _tree_snapshot(ALPHA)
second = generate_command_reference_bytes(ALPHA)
self.assertEqual(first, second)
self.assertEqual(before, middle)
self.assertEqual(before, _tree_snapshot(ALPHA))
self.assertTrue(first.endswith(b"\n"))
self.assertTrue(first.startswith(b"# DocForge command reference\n"))
self.assertIn(b"Do not edit this file by hand.", first)
self.assertEqual(1, first.count(b"## CLI commands"))
self.assertEqual(1, first.count(b"## MCP tools"))
def test_registration_collection_is_complete(self) -> None:
cli_rows, mcp_rows = asyncio.run(
collect_command_reference_rows(
ALPHA,
proposal_writer="alpha-editor",
canonical_applier="alpha-editor",
)
)
self.assertEqual(EXPECTED_CLI_ROWS, len(cli_rows))
self.assertEqual(EXPECTED_MCP_ROWS, len(mcp_rows))
self.assertEqual(len(cli_rows), len({item.name for item in cli_rows}))
self.assertEqual(
set((*ALL_TOOLS, *APPLICATION_TOOLS)),
{item.name for item in mcp_rows},
)
self.assertEqual(
{"read", "proposal", "application"},
{item.surface for item in mcp_rows},
)
def test_atomic_write_check_and_drift_detection(self) -> None:
content = generate_command_reference_bytes(ALPHA)
with tempfile.TemporaryDirectory() as directory:
repository = Path(directory).resolve()
(repository / "generated").mkdir()
output = Path("generated/command-reference.md")
target = repository / output
self.assertEqual(
"written",
publish_or_check_command_reference(
content,
repository_root=repository,
project_root=ALPHA,
output=output,
check=False,
),
)
self.assertEqual(content, target.read_bytes())
self.assertEqual(
"current",
publish_or_check_command_reference(
content,
repository_root=repository,
project_root=ALPHA,
output=output,
check=True,
),
)
target.write_text("stale\n", encoding="utf-8")
with self.assertRaises(CommandReferenceDrift):
publish_or_check_command_reference(
content,
repository_root=repository,
project_root=ALPHA,
output=output,
check=True,
)
self.assertEqual(b"stale\n", target.read_bytes())
self.assertEqual(
"written",
publish_or_check_command_reference(
content,
repository_root=repository,
project_root=ALPHA,
output=output,
check=False,
),
)
self.assertEqual(content, target.read_bytes())
self.assertEqual(
"unchanged",
publish_or_check_command_reference(
content,
repository_root=repository,
project_root=ALPHA,
output=output,
check=False,
),
)
def test_output_confinement_rejects_escapes_and_symlinks(self) -> None:
content = b"reference\n"
with tempfile.TemporaryDirectory() as directory:
repository = Path(directory).resolve()
generated = repository / "generated"
generated.mkdir()
outside = repository.parent / f"{repository.name}-outside.md"
symlink = generated / "reference.md"
symlink.symlink_to(outside)
for output in (Path("../escape.md"), Path("generated/reference.md")):
with (
self.subTest(output=output),
self.assertRaises(CommandReferenceToolError),
):
publish_or_check_command_reference(
content,
repository_root=repository,
project_root=ALPHA,
output=output,
check=False,
)
symlink.unlink()
(repository / "linked").symlink_to(generated, target_is_directory=True)
with self.assertRaises(CommandReferenceToolError):
publish_or_check_command_reference(
content,
repository_root=repository,
project_root=ALPHA,
output=Path("linked/reference.md"),
check=False,
)
with self.assertRaises(CommandReferenceToolError):
publish_or_check_command_reference(
content,
repository_root=ROOT,
project_root=ALPHA,
output=Path("tests/fixtures/alpha/generated-reference.md"),
check=False,
)
def test_existing_target_race_is_restored_without_overwrite(self) -> None:
with tempfile.TemporaryDirectory() as directory:
repository = Path(directory).resolve()
generated = repository / "generated"
generated.mkdir()
output = Path("generated/reference.md")
target = repository / output
target.write_bytes(b"initial\n")
original_exchange = command_reference_tool._rename_exchange
injected = False
def exchange_after_race(directory_fd: int, first: str, second: str) -> None:
nonlocal injected
if not injected:
injected = True
target.write_bytes(b"concurrent\n")
original_exchange(directory_fd, first, second)
with (
mock.patch.object(
command_reference_tool,
"_rename_exchange",
side_effect=exchange_after_race,
),
self.assertRaisesRegex(
CommandReferenceToolError,
"changed during atomic publication",
),
):
publish_or_check_command_reference(
b"generated\n",
repository_root=repository,
project_root=ALPHA,
output=output,
check=False,
)
self.assertEqual(b"concurrent\n", target.read_bytes())
self.assertEqual(
[],
list(generated.glob(".reference.md.docforge-command-reference-*")),
)
def test_missing_target_race_is_never_clobbered(self) -> None:
with tempfile.TemporaryDirectory() as directory:
repository = Path(directory).resolve()
generated = repository / "generated"
generated.mkdir()
output = Path("generated/reference.md")
target = repository / output
original_link = command_reference_tool._link_no_replace
injected = False
def link_after_race(directory_fd: int, source: str, destination: str) -> None:
nonlocal injected
if not injected:
injected = True
target.write_bytes(b"concurrent\n")
original_link(directory_fd, source, destination)
with (
mock.patch.object(
command_reference_tool,
"_link_no_replace",
side_effect=link_after_race,
),
self.assertRaisesRegex(CommandReferenceToolError, "appeared"),
):
publish_or_check_command_reference(
b"generated\n",
repository_root=repository,
project_root=ALPHA,
output=output,
check=False,
)
self.assertEqual(b"concurrent\n", target.read_bytes())
self.assertEqual(
[],
list(generated.glob(".reference.md.docforge-command-reference-*")),
)
def test_concurrent_generators_are_serialized_around_inspection_and_write(self) -> None:
with tempfile.TemporaryDirectory() as directory:
repository = Path(directory).resolve()
(repository / "generated").mkdir()
output = Path("generated/reference.md")
target = repository / output
target.write_bytes(b"initial\n")
first_entered = threading.Event()
release_first = threading.Event()
second_started = threading.Event()
second_entered = threading.Event()
results: list[str] = []
errors: list[BaseException] = []
original_replace = command_reference_tool._atomic_replace
invocation_count = 0
invocation_lock = threading.Lock()
def blocking_replace(
directory_fd: int,
name: str,
content: bytes,
*,
expected: object,
expected_content: bytes | None,
) -> None:
nonlocal invocation_count
with invocation_lock:
invocation_count += 1
invocation = invocation_count
if invocation == 1:
first_entered.set()
if not release_first.wait(timeout=5):
raise AssertionError("Timed out waiting to release first generator")
else:
second_entered.set()
original_replace(
directory_fd,
name,
content,
expected=expected, # type: ignore[arg-type]
expected_content=expected_content,
)
def publish(content: bytes, *, started: threading.Event | None = None) -> None:
if started is not None:
started.set()
try:
results.append(
publish_or_check_command_reference(
content,
repository_root=repository,
project_root=ALPHA,
output=output,
check=False,
)
)
except BaseException as error:
errors.append(error)
with mock.patch.object(
command_reference_tool,
"_atomic_replace",
side_effect=blocking_replace,
):
first = threading.Thread(target=publish, args=(b"first\n",))
second = threading.Thread(
target=publish,
args=(b"second\n",),
kwargs={"started": second_started},
)
first.start()
self.assertTrue(first_entered.wait(timeout=5))
second.start()
self.assertTrue(second_started.wait(timeout=5))
self.assertFalse(second_entered.wait(timeout=0.1))
release_first.set()
first.join(timeout=5)
second.join(timeout=5)
self.assertFalse(first.is_alive())
self.assertFalse(second.is_alive())
self.assertEqual([], errors)
self.assertEqual(["written", "written"], results)
self.assertEqual(b"second\n", target.read_bytes())
def test_command_runs_with_explicit_repository_and_project_roots(self) -> None:
with tempfile.TemporaryDirectory() as directory:
repository = Path(directory).resolve()
(repository / "generated").mkdir()
output = Path("generated/reference.md")
stdout = io.StringIO()
with contextlib.redirect_stdout(stdout):
status = main(
(
"--repository-root",
str(repository),
"--project-root",
str(ALPHA),
"--output",
str(output),
)
)
self.assertEqual(0, status)
self.assertTrue((repository / output).is_file())
self.assertIn('"cli_rows":28', stdout.getvalue())
self.assertIn('"mcp_rows":36', stdout.getvalue())
target = repository / output
target.write_text("drift\n", encoding="utf-8")
stderr = io.StringIO()
with contextlib.redirect_stderr(stderr):
check_status = main(
(
"--repository-root",
str(repository),
"--project-root",
str(ALPHA),
"--output",
str(output),
"--check",
)
)
self.assertEqual(1, check_status)
self.assertEqual("drift\n", target.read_text(encoding="utf-8"))
self.assertIn("missing or stale", stderr.getvalue())
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,486 @@
from __future__ import annotations
import json
import shutil
import tempfile
import unittest
from pathlib import Path
from typing import cast
from unittest.mock import patch
from docforge.adapter_sdk import (
AdapterProject,
AdapterSource,
AdapterSourceProjection,
verify_adapter_conformance,
)
from docforge.adapters.cpp import (
CPP_ADAPTER_ID,
CPP_ADAPTER_VERSION,
CPP_EXTRACTOR_VERSION,
CppReferenceAdapter,
)
from docforge.errors import DocForgeError
from docforge.index import ProjectIndex
ROOT = Path(__file__).resolve().parents[1]
FIXTURE = ROOT / "tests" / "fixtures" / "reference-cpp"
class RecordingCppAdapter(CppReferenceAdapter):
def __init__(self, root: Path) -> None:
super().__init__(
root,
source_roots=("include", "src"),
compilation_database="compile_commands.json",
project_id="reference-cpp",
title="Reference C++ fixture",
)
self.extracted_paths: list[str] = []
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection:
self.extracted_paths.append(source.source_path)
return super().extract_source(source)
class CppReferenceAdapterTests(unittest.TestCase):
def copy_fixture(self, parent: Path) -> Path:
root = parent / "reference-cpp"
shutil.copytree(FIXTURE, root)
return root.resolve()
def adapter(self, root: Path) -> RecordingCppAdapter:
return RecordingCppAdapter(root)
@staticmethod
def build_metrics(result: dict[str, object]) -> dict[str, object]:
return cast(dict[str, object], result["build"])
@staticmethod
def read_database(root: Path) -> list[dict[str, object]]:
return cast(
list[dict[str, object]],
json.loads((root / "compile_commands.json").read_text(encoding="utf-8")),
)
@staticmethod
def write_database(root: Path, entries: list[dict[str, object]]) -> None:
(root / "compile_commands.json").write_text(
json.dumps(entries, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
def test_complete_assembly_is_deterministic_conformant_and_syntax_scoped(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
adapter = self.adapter(root)
first = adapter.load_assembly()
second = adapter.load_complete_assembly()
projection = adapter.load_projection()
self.assertEqual(first, second)
self.assertEqual(first.projection, projection)
self.assertEqual(CPP_ADAPTER_ID, projection.adapter_id)
self.assertEqual(CPP_ADAPTER_VERSION, projection.adapter_version)
self.assertEqual(17, len(projection.nodes))
self.assertEqual(16, len(projection.edges))
self.assertEqual(5, len(first.logic))
self.assertEqual(
list(projection.nodes),
sorted(projection.nodes, key=lambda item: item.node.node_id),
)
self.assertEqual(
{"contains", "depends_on"},
{item.edge.relation for item in projection.edges},
)
self.assertTrue(
any(
item.node.title == "demo.Worker" and ("kind", "class") in item.metadata
for item in projection.nodes
)
)
self.assertTrue(
any(
item.node.title == "demo.Config" and ("kind", "struct") in item.metadata
for item in projection.nodes
)
)
self.assertTrue(
any(
item.node.title == "demo.Worker.run" and ("kind", "function") in item.metadata
for item in projection.nodes
)
)
self.assertEqual(
{"include/config.hpp", "include/detail.hpp", "src/main.cpp", "src/worker.cpp"},
{item.node.source_path for item in projection.nodes},
)
self.assertFalse(
{"calls", "inherits", "type_of"} & {item.edge.relation for item in projection.edges}
)
report = adapter.support_report()
self.assertEqual(1, report["schema_version"])
self.assertEqual(CPP_EXTRACTOR_VERSION, report["extractor_version"])
self.assertEqual(["include", "src"], report["source_roots"])
self.assertTrue(report["compilation_database_is_authoritative"])
self.assertFalse(report["executes_compiler"])
self.assertFalse(report["executes_project_code"])
self.assertEqual(
[
"call_resolution",
"type_resolution",
"inheritance_resolution",
"overload_ownership",
"macro_semantics",
"compiler_command_execution",
"nonlocal_include_resolution",
],
[item.code for item in adapter.unsupported_facts()],
)
self.assertFalse((root / ".cache").exists())
conformance = verify_adapter_conformance(
adapter,
cache_root=root / ".cache" / "conformance",
)
self.assertEqual("reference-cpp", conformance.project_id)
self.assertEqual(17, conformance.node_count)
self.assertEqual(16, conformance.edge_count)
self.assertEqual(5, conformance.logic_projection_count)
self.assertTrue(conformance.incremental)
def test_incremental_cold_warm_source_header_and_flag_invalidation(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
adapter = self.adapter(root)
project = AdapterProject(adapter, cache_root=root / ".cache" / "incremental")
index = ProjectIndex(project)
cold = index.build()
self.assertEqual(4, self.build_metrics(cold)["reparsed_sources"])
self.assertEqual(
[
"include/config.hpp",
"include/detail.hpp",
"src/main.cpp",
"src/worker.cpp",
],
sorted(adapter.extracted_paths),
)
adapter.extracted_paths.clear()
warm = index.build()
self.assertEqual(4, self.build_metrics(warm)["cache_hits"])
self.assertEqual(0, self.build_metrics(warm)["reparsed_sources"])
self.assertEqual([], adapter.extracted_paths)
main = root / "src" / "main.cpp"
main.write_text(
main.read_text(encoding="utf-8").replace(
"return demo::execute(3);",
"return demo::execute(4);",
),
encoding="utf-8",
)
adapter.extracted_paths.clear()
source_changed = index.build()
self.assertEqual(1, self.build_metrics(source_changed)["invalidated_sources"])
self.assertEqual(["src/main.cpp"], adapter.extracted_paths)
detail = root / "include" / "detail.hpp"
detail.write_text(
detail.read_text(encoding="utf-8").replace(
"return value;",
"return value + 0;",
),
encoding="utf-8",
)
adapter.extracted_paths.clear()
header_changed = index.build()
self.assertEqual(4, self.build_metrics(header_changed)["invalidated_sources"])
self.assertEqual(
[
"include/config.hpp",
"include/detail.hpp",
"src/main.cpp",
"src/worker.cpp",
],
sorted(adapter.extracted_paths),
)
entries = self.read_database(root)
arguments = cast(list[str], entries[0]["arguments"])
arguments.append("-DREFERENCE_FLAG=1")
self.write_database(root, entries)
adapter.extracted_paths.clear()
flag_changed = index.build()
self.assertEqual(1, self.build_metrics(flag_changed)["invalidated_sources"])
self.assertEqual(["src/main.cpp"], adapter.extracted_paths)
equivalent = project.verify_incremental_equivalence()
self.assertEqual("ok", equivalent["status"])
self.assertEqual(17, equivalent["node_count"])
self.assertEqual(16, equivalent["edge_count"])
self.assertEqual(5, equivalent["logic_projection_count"])
def test_add_delete_and_corrupt_cache_recover_without_stale_facts(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
adapter = self.adapter(root)
cache_root = root / ".cache" / "incremental"
project = AdapterProject(adapter, cache_root=cache_root)
index = ProjectIndex(project)
index.build()
extra = root / "src" / "extra.cpp"
extra.write_text(
'#include "../include/config.hpp"\n\n'
"int extra(int value) {\n"
" return demo::normalize(value);\n"
"}\n",
encoding="utf-8",
)
entries = self.read_database(root)
entries.append(
{
"arguments": ["c++", "-std=c++20", "-c", "src/extra.cpp"],
"directory": ".",
"file": "src/extra.cpp",
}
)
self.write_database(root, entries)
adapter.extracted_paths.clear()
added = index.build()
self.assertEqual(1, self.build_metrics(added)["reparsed_sources"])
self.assertEqual(["src/extra.cpp"], adapter.extracted_paths)
self.assertIn(
"src/extra.cpp",
{node.source_path for node in project.load().nodes},
)
worker = root / "src" / "worker.cpp"
entries = [
entry for entry in self.read_database(root) if entry["file"] != "src/worker.cpp"
]
self.write_database(root, entries)
worker.unlink()
adapter.extracted_paths.clear()
deleted = index.build()
self.assertEqual(1, self.build_metrics(deleted)["deleted_sources"])
self.assertNotIn(
"src/worker.cpp",
{node.source_path for node in project.load().nodes},
)
(cache_root / "extractions.json").write_text("{broken", encoding="utf-8")
adapter.extracted_paths.clear()
recovered = index.build()
self.assertEqual(4, self.build_metrics(recovered)["reparsed_sources"])
self.assertEqual(
[
"include/config.hpp",
"include/detail.hpp",
"src/extra.cpp",
"src/main.cpp",
],
sorted(adapter.extracted_paths),
)
self.assertEqual("ok", project.verify_incremental_equivalence()["status"])
def test_database_shape_bounds_duplicates_and_paths_fail_closed(self) -> None:
invalid_documents: list[tuple[str, list[dict[str, object]], str]] = []
baseline = cast(
list[dict[str, object]],
json.loads((FIXTURE / "compile_commands.json").read_text(encoding="utf-8")),
)
both = json.loads(json.dumps(baseline))
both[0]["command"] = "c++ -c src/main.cpp"
invalid_documents.append(("both command forms", both, "invalid_compilation_database"))
neither = json.loads(json.dumps(baseline))
del neither[1]["command"]
invalid_documents.append(("neither command form", neither, "invalid_compilation_database"))
bad_argument = json.loads(json.dumps(baseline))
bad_argument[0]["arguments"] = ["c++", 7]
invalid_documents.append(
("non-string argument", bad_argument, "invalid_compilation_database")
)
control_command = json.loads(json.dumps(baseline))
control_command[1]["command"] = "c++ -c src/worker.cpp\nmalicious"
invalid_documents.append(
("control command", control_command, "invalid_compilation_database")
)
unknown_key = json.loads(json.dumps(baseline))
unknown_key[0]["environment"] = {"PATH": "/tmp"}
invalid_documents.append(("unknown key", unknown_key, "invalid_compilation_database"))
duplicate = json.loads(json.dumps(baseline))
duplicate.append(json.loads(json.dumps(duplicate[0])))
invalid_documents.append(("duplicate TU", duplicate, "duplicate_translation_unit"))
escaped_directory = json.loads(json.dumps(baseline))
escaped_directory[0]["directory"] = ".."
invalid_documents.append(("escaped directory", escaped_directory, "path_escape"))
escaped_file = json.loads(json.dumps(baseline))
escaped_file[0]["file"] = "../outside.cpp"
invalid_documents.append(("escaped file", escaped_file, "path_escape"))
for name, document, code in invalid_documents:
with self.subTest(name=name), tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
self.write_database(root, document)
adapter = self.adapter(root)
with self.assertRaises(DocForgeError) as captured:
adapter.load_manifest()
self.assertEqual(code, captured.exception.code)
with tempfile.TemporaryDirectory() as directory:
parent = Path(directory)
root = self.copy_fixture(parent)
outside_database = parent / "compile_commands.json"
outside_database.write_text("[]\n", encoding="utf-8")
database = root / "compile_commands.json"
database.unlink()
database.symlink_to(outside_database)
with self.assertRaises(DocForgeError) as database_link:
self.adapter(root)
self.assertEqual("path_escape", database_link.exception.code)
with tempfile.TemporaryDirectory() as directory:
parent = Path(directory)
root = self.copy_fixture(parent)
outside_source = parent / "worker.cpp"
outside_source.write_text("int outside() { return 1; }\n", encoding="utf-8")
worker = root / "src" / "worker.cpp"
worker.unlink()
worker.symlink_to(outside_source)
adapter = self.adapter(root)
with self.assertRaises(DocForgeError) as source_link:
adapter.load_manifest()
self.assertEqual("path_escape", source_link.exception.code)
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
bounded_adapters = (
(
CppReferenceAdapter(
root,
source_roots=("include", "src"),
compilation_database="compile_commands.json",
max_database_bytes=1,
),
"source_too_large",
),
(
CppReferenceAdapter(
root,
source_roots=("include", "src"),
compilation_database="compile_commands.json",
max_source_bytes=1,
),
"source_too_large",
),
(
CppReferenceAdapter(
root,
source_roots=("include", "src"),
compilation_database="compile_commands.json",
max_command_chars=32,
),
"invalid_compilation_database",
),
(
CppReferenceAdapter(
root,
source_roots=("include", "src"),
compilation_database="compile_commands.json",
max_arguments=2,
),
"invalid_compilation_database",
),
(
CppReferenceAdapter(
root,
source_roots=("include", "src"),
compilation_database="compile_commands.json",
max_translation_units=1,
),
"adapter_too_large",
),
(
CppReferenceAdapter(
root,
source_roots=("include", "src"),
compilation_database="compile_commands.json",
max_headers=1,
),
"adapter_too_large",
),
)
for adapter, code in bounded_adapters:
with self.subTest(limit=code), self.assertRaises(DocForgeError) as bounded:
adapter.load_manifest()
self.assertEqual(code, bounded.exception.code)
def test_source_roots_no_ast_and_optional_extra_fail_closed(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
with self.assertRaises(DocForgeError) as translation_unit_outside:
CppReferenceAdapter(
root,
source_roots=("include",),
compilation_database="compile_commands.json",
).load_manifest()
self.assertEqual("path_escape", translation_unit_outside.exception.code)
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
other = root / "other"
other.mkdir()
(other / "outside.hpp").write_text(
"#pragma once\nstruct Outside {};\n",
encoding="utf-8",
)
main = root / "src" / "main.cpp"
main.write_text(
'#include "../other/outside.hpp"\n' + main.read_text(encoding="utf-8"),
encoding="utf-8",
)
with self.assertRaises(DocForgeError) as header_outside:
self.adapter(root).load_manifest()
self.assertEqual("path_escape", header_outside.exception.code)
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
adapter = self.adapter(root)
project = AdapterProject(adapter, cache_root=root / ".cache" / "no-ast")
with self.assertRaises(DocForgeError) as no_ast:
ProjectIndex(project, allow_logic=False).build()
self.assertEqual("adapter_policy_forbids_logic", no_ast.exception.code)
self.assertFalse(project.descriptor.index_path.exists())
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
adapter = self.adapter(root)
missing = ModuleNotFoundError(
"No module named 'tree_sitter'",
name="tree_sitter",
)
with (
patch(
"docforge.adapters.cpp.importlib.import_module",
side_effect=missing,
),
self.assertRaises(DocForgeError) as optional,
):
adapter.load_manifest()
self.assertEqual("optional_dependency_missing", optional.exception.code)
self.assertEqual("cpp", optional.exception.details["extra"])
self.assertEqual("docforge[cpp]", optional.exception.details["install"])

198
tests/test_docs_check.py Normal file
View file

@ -0,0 +1,198 @@
from __future__ import annotations
import shutil
import tempfile
import unittest
from pathlib import Path
from tools.check_documentation import (
COMMAND_REFERENCE_NOTICE,
MAX_DIAGNOSTICS,
DocumentationPolicy,
check_documentation,
)
ROOT = Path(__file__).resolve().parents[1]
SCHEMA = ROOT / "schemas/reference-adapter.schema.json"
class DocumentationCheckTests(unittest.TestCase):
def repository(self) -> Path:
root = Path(self.enterContext(tempfile.TemporaryDirectory()))
(root / "docs").mkdir()
(root / "schemas").mkdir()
shutil.copyfile(SCHEMA, root / "schemas/reference-adapter.schema.json")
(root / "pending.txt").write_text("", encoding="utf-8")
return root
@staticmethod
def policy(*required: str) -> DocumentationPolicy:
return DocumentationPolicy(
required_pages=tuple(Path(item) for item in required),
pending_inventory=Path("pending.txt"),
)
@staticmethod
def command_reference() -> str:
return f"# DocForge command reference\n\n{COMMAND_REFERENCE_NOTICE}\n\n## CLI commands\n"
def test_valid_graph_h1_anchors_and_reference_adapter_toml_pass(self) -> None:
root = self.repository()
(root / "README.md").write_text(
"# Documentation\n\n"
"[Guide](docs/GUIDE.md#adapter-setup) and "
"[commands](docs/COMMAND_REFERENCE.md). [Top](#documentation)\n",
encoding="utf-8",
)
(root / "docs/GUIDE.md").write_text(
"# Guide\n\n"
"## Adapter *setup*\n\n"
"Reference adapter configuration `.docforge/reference-adapter.toml`:\n\n"
"```toml\n"
"schema_version = 1\n"
'project_id = "example-project"\n'
'title = "Example"\n'
'language = "python"\n'
'source_roots = ["src"]\n'
"```\n\n"
"[Back](../README.md#documentation)\n",
encoding="utf-8",
)
(root / "docs/COMMAND_REFERENCE.md").write_text(
self.command_reference(),
encoding="utf-8",
)
report = check_documentation(
root,
policy=self.policy(
"README.md",
"docs/GUIDE.md",
"docs/COMMAND_REFERENCE.md",
),
)
self.assertTrue(report.ok, [item.render() for item in report.diagnostics])
self.assertEqual(3, report.markdown_pages)
self.assertEqual(1, report.reference_adapter_examples)
def test_links_anchors_h1_and_generated_notice_fail_closed(self) -> None:
root = self.repository()
(root / "README.md").write_text(
"# Documentation\n\n"
"[bad anchor](docs/GUIDE.md#missing) "
"[missing file](docs/ABSENT.md) "
"[commands](docs/COMMAND_REFERENCE.md).\n",
encoding="utf-8",
)
(root / "docs/GUIDE.md").write_text(
"# Guide\n\n# Duplicate\n",
encoding="utf-8",
)
(root / "docs/COMMAND_REFERENCE.md").write_text(
"# DocForge command reference\n\n## CLI commands\n",
encoding="utf-8",
)
report = check_documentation(
root,
policy=self.policy(
"README.md",
"docs/GUIDE.md",
"docs/COMMAND_REFERENCE.md",
),
)
self.assertFalse(report.ok)
self.assertEqual(
{"DOC016", "DOC017", "DOC020", "DOC028"},
{item.code for item in report.diagnostics},
)
def test_only_prior_milestone_records_are_reachability_exempt(self) -> None:
root = self.repository()
(root / "README.md").write_text("# Documentation\n", encoding="utf-8")
(root / "docs/MILESTONE_2_BASELINE.md").write_text(
"# Historical evidence\n",
encoding="utf-8",
)
(root / "docs/ORPHAN.md").write_text("# Orphan\n", encoding="utf-8")
report = check_documentation(
root,
policy=self.policy("README.md"),
)
orphans = [item.path for item in report.diagnostics if item.code == "DOC023"]
self.assertEqual(["docs/ORPHAN.md"], orphans)
def test_pending_page_becomes_a_hard_failure_as_soon_as_it_exists(self) -> None:
root = self.repository()
(root / "README.md").write_text("# Documentation\n", encoding="utf-8")
(root / "pending.txt").write_text("docs/FUTURE.md\n", encoding="utf-8")
policy = self.policy("README.md", "docs/FUTURE.md")
pending = check_documentation(root, policy=policy)
self.assertTrue(pending.ok)
self.assertEqual(1, pending.pending_pages)
(root / "docs/FUTURE.md").write_text("# Future\n", encoding="utf-8")
stale = check_documentation(root, policy=policy)
self.assertIn("DOC014", {item.code for item in stale.diagnostics})
self.assertIn("DOC023", {item.code for item in stale.diagnostics})
def test_unknown_pending_page_and_invalid_toml_are_reported(self) -> None:
root = self.repository()
(root / "README.md").write_text(
"# Documentation\n\n[Adapter](docs/ADAPTER.md)\n",
encoding="utf-8",
)
(root / "docs/ADAPTER.md").write_text(
"# Adapter\n\n"
"```toml reference-adapter\n"
"schema_version = 1\n"
'project_id = "cpp-example"\n'
'title = "C++"\n'
'language = "cpp"\n'
'source_roots = ["src"]\n'
"```\n",
encoding="utf-8",
)
(root / "pending.txt").write_text("docs/UNKNOWN.md\n", encoding="utf-8")
report = check_documentation(
root,
policy=self.policy("README.md", "docs/ADAPTER.md"),
)
self.assertIn("DOC012", {item.code for item in report.diagnostics})
schema_failures = [item for item in report.diagnostics if item.code == "DOC026"]
self.assertEqual(1, len(schema_failures))
self.assertIn("compilation_database", schema_failures[0].message)
def test_diagnostics_are_deterministically_bounded(self) -> None:
root = self.repository()
(root / "README.md").write_text("# Documentation\n", encoding="utf-8")
for index in range(MAX_DIAGNOSTICS + 7):
(root / "docs" / f"ORPHAN_{index:03d}.md").write_text(
f"# Orphan {index}\n",
encoding="utf-8",
)
report = check_documentation(
root,
policy=self.policy("README.md"),
)
self.assertEqual(MAX_DIAGNOSTICS, len(report.diagnostics))
self.assertEqual(7, report.omitted_diagnostics)
self.assertEqual(
sorted(report.diagnostics),
list(report.diagnostics),
)
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,176 @@
from __future__ import annotations
import json
import tempfile
import unittest
from pathlib import Path
from tools.check_documentation import (
DocumentationPolicy,
check_documentation,
)
class DocumentationCheckTests(unittest.TestCase):
def repository(self, parent: Path) -> Path:
root = parent / "repository"
(root / "docs").mkdir(parents=True)
(root / "schemas").mkdir()
(root / "schemas" / "reference-adapter.schema.json").write_text(
json.dumps(
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"additionalProperties": False,
"required": [
"schema_version",
"project_id",
"title",
"language",
"source_roots",
],
"properties": {
"schema_version": {"const": 1},
"project_id": {"type": "string"},
"title": {"type": "string"},
"language": {"enum": ["python", "javascript", "typescript", "cpp"]},
"source_roots": {
"type": "array",
"minItems": 1,
"items": {"type": "string"},
},
"compilation_database": {"type": "string"},
},
}
),
encoding="utf-8",
)
return root
def test_linked_pages_h1_anchors_and_reference_example_pass(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.repository(Path(directory))
(root / "README.md").write_text(
"# Project\n\n[Guide](docs/GUIDE.md#reference-adapter-configuration)\n",
encoding="utf-8",
)
(root / "docs" / "GUIDE.md").write_text(
"\n".join(
(
"# Guide",
"",
"## Reference adapter configuration",
"",
"`.docforge/reference-adapter.toml`:",
"",
"```toml",
"schema_version = 1",
'project_id = "fixture"',
'title = "Fixture"',
'language = "python"',
'source_roots = ["src"]',
"```",
"",
)
),
encoding="utf-8",
)
report = check_documentation(
root,
policy=DocumentationPolicy(
required_pages=(Path("README.md"), Path("docs/GUIDE.md")),
pending_inventory=None,
),
)
self.assertTrue(report.ok, [item.render() for item in report.diagnostics])
self.assertEqual(2, report.markdown_pages)
self.assertEqual(1, report.reference_adapter_examples)
def test_missing_anchor_orphan_and_duplicate_h1_fail_deterministically(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.repository(Path(directory))
(root / "README.md").write_text(
"# Project\n\n[Broken](docs/GUIDE.md#missing)\n",
encoding="utf-8",
)
(root / "docs" / "GUIDE.md").write_text(
"# Guide\n\n# Duplicate\n",
encoding="utf-8",
)
(root / "docs" / "ORPHAN.md").write_text("# Orphan\n", encoding="utf-8")
report = check_documentation(
root,
policy=DocumentationPolicy(
required_pages=(Path("README.md"), Path("docs/GUIDE.md")),
pending_inventory=None,
),
)
self.assertFalse(report.ok)
codes = [item.code for item in report.diagnostics]
self.assertEqual(["DOC016", "DOC017", "DOC023"], sorted(codes))
def test_pending_inventory_allows_absence_but_rejects_existing_page(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.repository(Path(directory))
(root / "README.md").write_text("# Project\n", encoding="utf-8")
pending_path = root / "pending.txt"
pending_path.write_text("docs/FUTURE.md\n", encoding="utf-8")
policy = DocumentationPolicy(
required_pages=(Path("README.md"), Path("docs/FUTURE.md")),
pending_inventory=Path("pending.txt"),
)
missing = check_documentation(root, policy=policy)
self.assertTrue(missing.ok)
self.assertEqual(1, missing.pending_pages)
(root / "docs" / "FUTURE.md").write_text("# Future\n", encoding="utf-8")
stale = check_documentation(root, policy=policy)
self.assertEqual(["DOC014", "DOC023"], [item.code for item in stale.diagnostics])
def test_invalid_reference_example_reports_schema_path(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.repository(Path(directory))
(root / "README.md").write_text(
"# Project\n\n[Guide](docs/GUIDE.md)\n",
encoding="utf-8",
)
(root / "docs" / "GUIDE.md").write_text(
"\n".join(
(
"# Guide",
"",
"Reference adapter configuration:",
"",
"```toml",
'language = "python"',
'source_roots = ["src"]',
"```",
"",
)
),
encoding="utf-8",
)
report = check_documentation(
root,
policy=DocumentationPolicy(
required_pages=(Path("README.md"), Path("docs/GUIDE.md")),
pending_inventory=None,
),
)
schema_errors = [item for item in report.diagnostics if item.code == "DOC026"]
self.assertEqual(3, len(schema_errors))
messages = " ".join(item.message for item in schema_errors)
self.assertIn("schema_version", messages)
self.assertIn("project_id", messages)
self.assertIn("title", messages)
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,102 @@
from __future__ import annotations
import hashlib
import os
import tempfile
import unittest
from pathlib import Path
from docforge.errors import DocForgeError
from docforge.incremental import (
CachedSource,
ExtractionCache,
load_extraction_cache,
write_extraction_cache,
)
class ExtractionCacheBoundsTests(unittest.TestCase):
@staticmethod
def cache(*, payload: str = "ok", count: int = 1) -> ExtractionCache:
sources = tuple(
CachedSource(
source_id=f"source-{index}",
source_path=f"src/{index}.txt",
fingerprint=hashlib.sha256(str(index).encode()).hexdigest(),
extractor_version="fixture@1",
dependencies=(),
payload={"content": payload},
)
for index in range(count)
)
return ExtractionCache("fixture", "fixture-adapter", "1", sources)
def test_bounded_round_trip_and_oversized_read_is_a_cache_miss(self) -> None:
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "extractions.json"
write_extraction_cache(path, self.cache(), max_bytes=1_000, max_sources=2)
loaded = load_extraction_cache(
path,
project_id="fixture",
adapter_id="fixture-adapter",
adapter_version="1",
max_bytes=1_000,
max_sources=2,
)
self.assertEqual(self.cache(), loaded)
with path.open("ab") as handle:
handle.write(b" " * 1_000)
self.assertIsNone(
load_extraction_cache(
path,
project_id="fixture",
adapter_id="fixture-adapter",
adapter_version="1",
max_bytes=1_000,
max_sources=2,
)
)
def test_write_rejects_byte_and_source_limits_without_replacing_target(self) -> None:
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "extractions.json"
path.write_bytes(b"preserved\n")
for cache, maximum_bytes, maximum_sources in (
(self.cache(payload="x" * 500), 100, 2),
(self.cache(count=2), 1_000, 1),
):
with self.assertRaises(DocForgeError) as captured:
write_extraction_cache(
path,
cache,
max_bytes=maximum_bytes,
max_sources=maximum_sources,
)
self.assertEqual("cache_limit", captured.exception.code)
self.assertEqual(b"preserved\n", path.read_bytes())
def test_symlink_and_non_regular_targets_are_cache_misses(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
real = root / "real.json"
write_extraction_cache(real, self.cache(), max_bytes=1_000, max_sources=2)
linked = root / "linked.json"
linked.symlink_to(real)
fifo = root / "fifo"
os.mkfifo(fifo)
for path in (linked, fifo):
self.assertIsNone(
load_extraction_cache(
path,
project_id="fixture",
adapter_id="fixture-adapter",
adapter_version="1",
max_bytes=1_000,
max_sources=2,
)
)
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,267 @@
from __future__ import annotations
import shutil
import tempfile
import unittest
from pathlib import Path
from typing import cast
from unittest import mock
import docforge.adapters.javascript as javascript_adapter
from docforge.adapter_sdk import (
AdapterProject,
AdapterSource,
AdapterSourceProjection,
verify_adapter_conformance,
)
from docforge.adapters.javascript import (
JAVASCRIPT_ADAPTER_ID,
JAVASCRIPT_ADAPTER_VERSION,
JAVASCRIPT_EXTRACTOR_VERSION,
JavaScriptReferenceAdapter,
)
from docforge.errors import DocForgeError
from docforge.index import ProjectIndex
ROOT = Path(__file__).resolve().parents[1]
FIXTURES = ROOT / "tests" / "fixtures"
class RecordingScriptAdapter(JavaScriptReferenceAdapter):
def __init__(self, root: Path, project_id: str) -> None:
super().__init__(
root,
source_roots=("src",),
project_id=project_id,
title=f"{project_id} fixture",
)
self.extracted_paths: list[str] = []
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection:
self.extracted_paths.append(source.source_path)
return super().extract_source(source)
class JavaScriptReferenceAdapterTests(unittest.TestCase):
CASES = (
("reference-javascript", "javascript", 14, 14, 5, "src/app/shared.js"),
("reference-typescript", "typescript", 13, 14, 4, "src/app/types.mts"),
)
def copy_fixture(self, parent: Path, fixture: str) -> Path:
root = parent / fixture
shutil.copytree(FIXTURES / fixture, root)
return root.resolve()
@staticmethod
def build_metrics(result: dict[str, object]) -> dict[str, object]:
return cast(dict[str, object], result["build"])
def test_both_grammars_produce_deterministic_complete_logic_assemblies(self) -> None:
for fixture, language, nodes, edges, logic, _shared in self.CASES:
with self.subTest(language=language), tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory), fixture)
adapter = RecordingScriptAdapter(root, fixture)
first = adapter.load_assembly()
second = adapter.load_complete_assembly()
projection = adapter.load_projection()
self.assertEqual(first, second)
self.assertEqual(first.projection, projection)
self.assertEqual(JAVASCRIPT_ADAPTER_ID, projection.adapter_id)
self.assertEqual(JAVASCRIPT_ADAPTER_VERSION, projection.adapter_version)
self.assertEqual(nodes, len(projection.nodes))
self.assertEqual(edges, len(projection.edges))
self.assertEqual(logic, len(first.logic))
self.assertEqual(
{language},
{dict(item.metadata)["language"] for item in projection.nodes},
)
self.assertEqual(
{"contains", "depends_on"},
{item.edge.relation for item in projection.edges},
)
self.assertTrue(
any(
item.node.title == "Service.run"
and dict(item.metadata)["kind"] == "function"
for item in projection.nodes
)
)
self.assertFalse((root / ".cache").exists())
report = adapter.support_report()
self.assertEqual(1, report["schema_version"])
self.assertEqual(JAVASCRIPT_EXTRACTOR_VERSION, report["extractor_version"])
self.assertFalse(report["imports_project_code"])
self.assertFalse(report["executes_project_code"])
self.assertEqual(
[
"call_resolution",
"dynamic_module_resolution",
"inheritance_resolution",
"module_configuration_resolution",
"runtime_generated_facts",
"type_and_symbol_resolution",
],
[item.code for item in adapter.unsupported_facts()],
)
conformance = verify_adapter_conformance(
adapter,
cache_root=root / ".cache" / "conformance",
)
self.assertEqual(nodes, conformance.node_count)
self.assertEqual(edges, conformance.edge_count)
self.assertEqual(logic, conformance.logic_projection_count)
self.assertTrue(conformance.incremental)
def test_cold_warm_reverse_dependency_and_exact_equivalence_for_both_grammars(
self,
) -> None:
for fixture, language, nodes, edges, logic, shared_path in self.CASES:
with self.subTest(language=language), tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory), fixture)
adapter = RecordingScriptAdapter(root, fixture)
project = AdapterProject(adapter, cache_root=root / ".cache" / "incremental")
index = ProjectIndex(project)
cold = index.build()
self.assertEqual(4, self.build_metrics(cold)["reparsed_sources"])
self.assertEqual(4, len(adapter.extracted_paths))
adapter.extracted_paths.clear()
with mock.patch.object(
adapter,
"_parse",
wraps=adapter._parse, # pyright: ignore[reportPrivateUsage]
) as parsed:
warm = index.build()
self.assertEqual(4, self.build_metrics(warm)["cache_hits"])
self.assertEqual(0, self.build_metrics(warm)["reparsed_sources"])
self.assertEqual([], adapter.extracted_paths)
parsed.assert_not_called()
shared = root / shared_path
shared.write_text(
shared.read_text(encoding="utf-8") + "\n// changed dependency evidence\n",
encoding="utf-8",
)
adapter.extracted_paths.clear()
with mock.patch.object(
adapter,
"_parse",
wraps=adapter._parse, # pyright: ignore[reportPrivateUsage]
) as parsed:
changed = index.build()
self.assertEqual(4, self.build_metrics(changed)["invalidated_sources"])
self.assertEqual(4, len(adapter.extracted_paths))
self.assertEqual(4, parsed.call_count)
equivalent = project.verify_incremental_equivalence()
self.assertEqual("ok", equivalent["status"])
self.assertEqual(nodes, equivalent["node_count"])
self.assertEqual(edges, equivalent["edge_count"])
self.assertEqual(logic, equivalent["logic_projection_count"])
def test_add_delete_and_corrupt_cache_recovery_for_both_grammars(self) -> None:
additions = {
"javascript": (
"src/app/extra.js",
"export function extra() { return 'extra'; }\n",
),
"typescript": (
"src/app/extra.ts",
"export function extra(): string { return 'extra'; }\n",
),
}
for fixture, language, _nodes, _edges, _logic, _shared_path in self.CASES:
with self.subTest(language=language), tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory), fixture)
adapter = RecordingScriptAdapter(root, fixture)
cache_root = root / ".cache" / "incremental"
project = AdapterProject(adapter, cache_root=cache_root)
index = ProjectIndex(project)
index.build()
relative, content = additions[language]
added_path = root / relative
added_path.write_text(content, encoding="utf-8")
adapter.extracted_paths.clear()
added = index.build()
self.assertEqual(1, self.build_metrics(added)["reparsed_sources"])
self.assertEqual([relative], adapter.extracted_paths)
self.assertIn(relative, {node.source_path for node in project.load().nodes})
added_path.unlink()
adapter.extracted_paths.clear()
deleted = index.build()
self.assertEqual(1, self.build_metrics(deleted)["deleted_sources"])
self.assertNotIn(relative, {node.source_path for node in project.load().nodes})
(cache_root / "extractions.json").write_text("{broken", encoding="utf-8")
adapter.extracted_paths.clear()
recovered = index.build()
self.assertEqual(4, self.build_metrics(recovered)["reparsed_sources"])
self.assertEqual(4, len(adapter.extracted_paths))
self.assertEqual("ok", project.verify_incremental_equivalence()["status"])
def test_confinement_and_no_ast_reject_logic_for_both_grammars(self) -> None:
for fixture, language, _nodes, _edges, _logic, _shared_path in self.CASES:
with self.subTest(language=language), tempfile.TemporaryDirectory() as directory:
parent = Path(directory)
root = self.copy_fixture(parent, fixture)
for source_root in (root / "src", Path("../outside")):
with self.assertRaises(DocForgeError) as captured:
JavaScriptReferenceAdapter(root, source_roots=(source_root,))
self.assertEqual("path_escape", captured.exception.code)
outside = parent / "outside"
outside.mkdir()
linked_root = root / "linked"
linked_root.symlink_to(outside, target_is_directory=True)
with self.assertRaises(DocForgeError) as linked:
JavaScriptReferenceAdapter(root, source_roots=("linked",))
self.assertEqual("path_escape", linked.exception.code)
adapter = RecordingScriptAdapter(root, fixture)
project = AdapterProject(adapter, cache_root=root / ".cache" / "no-ast")
with self.assertRaises(DocForgeError) as no_ast:
ProjectIndex(project, allow_logic=False).build()
self.assertEqual("adapter_policy_forbids_logic", no_ast.exception.code)
self.assertFalse(project.descriptor.index_path.exists())
def test_missing_optional_frontends_report_exact_install_remediation(self) -> None:
for fixture, language, _nodes, _edges, _logic, _shared_path in self.CASES:
grammar_module = (
"tree_sitter_javascript" if language == "javascript" else "tree_sitter_typescript"
)
with self.subTest(language=language), tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory), fixture)
adapter = RecordingScriptAdapter(root, fixture)
missing = ModuleNotFoundError(
f"No module named {grammar_module!r}",
name=grammar_module,
)
with (
mock.patch.object(
javascript_adapter.importlib,
"import_module",
side_effect=missing,
),
self.assertRaises(DocForgeError) as captured,
):
source = adapter.load_manifest().sources[0]
adapter.extract_source(source)
self.assertEqual("optional_dependency_missing", captured.exception.code)
self.assertEqual(language, captured.exception.details["extra"])
self.assertEqual(
f"docforge[{language}]",
captured.exception.details["install"],
)
self.assertEqual(
grammar_module,
captured.exception.details["missing_module"],
)

View file

@ -0,0 +1,137 @@
from __future__ import annotations
import json
import subprocess
import tempfile
import unittest
from pathlib import Path
from unittest import mock
from tools.milestone4_adoption import (
AdoptionProofError,
find_wheel,
parse_evidence,
run_adoption_proof,
)
def _evidence() -> dict[str, object]:
return {
"schema_version": 1,
"base_frontend_modules": [],
"base_frontend_distributions": [],
"python": {"build_status": "ok", "check_status": "ok", "node_count": 3},
"mcp": {
"bootstrap_status": "ok",
"node_id": "python.class.service",
"read_tool_count": 20,
},
"cpp_without_extra": {
"code": "optional_dependency_missing",
"install": "docforge[cpp]",
"missing_module": "tree_sitter",
},
}
class Milestone4AdoptionTests(unittest.TestCase):
def test_evidence_requires_base_isolation_python_mcp_and_actionable_cpp_failure(
self,
) -> None:
self.assertEqual(_evidence(), parse_evidence(json.dumps(_evidence())))
for mutation in (
{"base_frontend_modules": ["tree_sitter"]},
{"python": {"build_status": "error", "check_status": "ok"}},
{"mcp": {"bootstrap_status": "error"}},
{
"cpp_without_extra": {
"code": "optional_dependency_missing",
"install": "docforge[languages]",
}
},
):
evidence = {**_evidence(), **mutation}
with self.subTest(mutation=mutation), self.assertRaises(AdoptionProofError):
parse_evidence(json.dumps(evidence))
def test_wheel_discovery_rejects_missing_or_ambiguous_artifacts(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
with self.assertRaises(AdoptionProofError):
find_wheel(root)
first = root / "docforge-1-py3-none-any.whl"
first.touch()
self.assertEqual(first, find_wheel(root))
(root / "docforge-2-py3-none-any.whl").touch()
with self.assertRaises(AdoptionProofError):
find_wheel(root)
def test_orchestration_is_offline_locked_and_installs_the_wheel_without_extras(
self,
) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
(root / "pyproject.toml").write_text("[project]\n", encoding="utf-8")
(root / "uv.lock").write_text("version = 1\n", encoding="utf-8")
commands: list[list[str]] = []
def execute(
arguments: list[str],
*,
cwd: Path,
timeout: int,
environment: dict[str, str] | None = None,
) -> subprocess.CompletedProcess[str]:
del cwd, timeout
commands.append(arguments)
self.assertIsNotNone(environment)
assert environment is not None
self.assertEqual("1", environment["UV_OFFLINE"])
self.assertNotIn("PYTHONPATH", environment)
if arguments[1] == "build":
output = Path(arguments[arguments.index("--out-dir") + 1])
output.mkdir()
(output / "docforge-1-py3-none-any.whl").write_bytes(b"wheel")
elif arguments[1] == "export":
output = Path(arguments[arguments.index("--output-file") + 1])
output.write_text("mcp==1\n", encoding="utf-8")
elif arguments[1] == "venv":
virtual_environment = Path(arguments[-1])
(virtual_environment / "bin").mkdir(parents=True)
(virtual_environment / "bin" / "python").touch()
elif arguments[0].endswith("/bin/python"):
return subprocess.CompletedProcess(
arguments,
0,
stdout=json.dumps(_evidence()),
stderr="",
)
return subprocess.CompletedProcess(arguments, 0, stdout="", stderr="")
with mock.patch(
"tools.milestone4_adoption._run_checked",
side_effect=execute,
):
result = run_adoption_proof(root, uv="uv")
self.assertEqual("offline", result["network"])
wheel = result["wheel"]
self.assertIsInstance(wheel, dict)
assert isinstance(wheel, dict)
self.assertEqual("docforge-1-py3-none-any.whl", wheel["filename"])
network_commands = (
command
for command in commands
if command[0] == "uv" and command[1] in {"build", "export", "pip"}
)
self.assertTrue(all("--offline" in command for command in network_commands))
export = next(command for command in commands if command[1] == "export")
self.assertIn("--frozen", export)
self.assertIn("--no-dev", export)
wheel_install = next(command for command in commands if command[1:3] == ["pip", "install"])
self.assertIn("--no-deps", wheel_install)
if __name__ == "__main__":
unittest.main()

View file

@ -43,6 +43,7 @@ PUBLIC_IMPORTS = {
),
"docforge.adapter_contract": (
"AdapterAssembly",
"AdapterConformanceReport",
"AdapterEdge",
"AdapterImplementation",
"AdapterLoader",
@ -53,15 +54,41 @@ PUBLIC_IMPORTS = {
"AdapterProjectSettings",
"AdapterSource",
"AdapterSourceProjection",
"CompleteAdapterAssemblyLoader",
"IncrementalAdapterAssembler",
"IncrementalAdapterLoader",
"verify_adapter_conformance",
),
"docforge.adapter_sdk": (
"AdapterAssembly",
"AdapterConformanceReport",
"AdapterLoader",
"AdapterManifest",
"AdapterProject",
"AdapterProjection",
"AdapterSourceProjection",
"CompleteAdapterAssemblyLoader",
"IncrementalAdapterLoader",
"LogicProjection",
"Node",
"verify_adapter_conformance",
),
"docforge.application": (
"CanonicalApplier",
"CanonicalApplicationService",
"GenericCanonicalApplier",
),
"docforge.client_config": ("generate_client_configuration",),
"docforge.adapter_launcher": (
"ADAPTER_LAUNCHER_SCHEMA_VERSION",
"AdapterLauncherV1",
"AdapterSourceAvailabilityV1",
"adapter_source_availability",
"validate_adapter_launcher",
),
"docforge.client_config": (
"generate_adapter_client_configuration",
"generate_client_configuration",
),
"docforge.doctor": ("run_doctor",),
"docforge.index": ("ProjectIndex",),
"docforge.graph_projection": (
@ -118,6 +145,18 @@ PUBLIC_IMPORTS = {
"projection_hash",
),
"docforge.projection_worker": ("render_projection_in_worker",),
"docforge.reference_config": (
"REFERENCE_ADAPTER_CONFIG",
"REFERENCE_ADAPTER_SCHEMA_VERSION",
"REFERENCE_LANGUAGES",
"ReferenceAdapterConfigV1",
"load_reference_adapter_config",
),
"docforge.reference_mcp": (
"REFERENCE_MCP_MODULE",
"create_reference_project",
"create_reference_server",
),
"docforge.retrieval": (
"ContextCapsuleV1",
"RetrievalPlanV1",

View file

@ -0,0 +1,282 @@
from __future__ import annotations
import shutil
import tempfile
import unittest
from pathlib import Path
from typing import cast
from unittest import mock
import docforge.adapters.python as python_adapter
from docforge.adapter_sdk import (
AdapterProject,
AdapterSource,
AdapterSourceProjection,
verify_adapter_conformance,
)
from docforge.adapters.python import (
PYTHON_ADAPTER_ID,
PYTHON_ADAPTER_VERSION,
PYTHON_EXTRACTOR_VERSION,
PythonReferenceAdapter,
)
from docforge.errors import DocForgeError
from docforge.index import ProjectIndex
ROOT = Path(__file__).resolve().parents[1]
FIXTURE = ROOT / "tests" / "fixtures" / "reference-python"
class RecordingPythonAdapter(PythonReferenceAdapter):
def __init__(self, root: Path) -> None:
super().__init__(
root,
source_roots=("src",),
project_id="reference-python",
title="Reference Python fixture",
)
self.extracted_paths: list[str] = []
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection:
self.extracted_paths.append(source.source_path)
return super().extract_source(source)
class PythonReferenceAdapterTests(unittest.TestCase):
def copy_fixture(self, parent: Path) -> Path:
root = parent / "reference-python"
shutil.copytree(FIXTURE, root)
return root.resolve()
def adapter(self, root: Path) -> RecordingPythonAdapter:
return RecordingPythonAdapter(root)
@staticmethod
def build_metrics(result: dict[str, object]) -> dict[str, object]:
return cast(dict[str, object], result["build"])
def test_complete_assembly_is_exact_ordered_cache_independent_and_scoped(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
adapter = self.adapter(root)
first = adapter.load_assembly()
second = adapter.load_complete_assembly()
projection = adapter.load_projection()
self.assertEqual(first, second)
self.assertEqual(first.projection, projection)
self.assertEqual(PYTHON_ADAPTER_ID, projection.adapter_id)
self.assertEqual(PYTHON_ADAPTER_VERSION, projection.adapter_version)
self.assertEqual(14, len(projection.nodes))
self.assertEqual(13, len(projection.edges))
self.assertEqual(5, len(first.logic))
self.assertEqual(
list(projection.nodes),
sorted(projection.nodes, key=lambda item: item.node.node_id),
)
self.assertEqual(
{"contains", "depends_on"}, {item.edge.relation for item in projection.edges}
)
self.assertFalse(any(item.node.title == "json" for item in projection.nodes))
self.assertTrue(
any(
item.node.title == "Service" and ("kind", "class") in item.metadata
for item in projection.nodes
)
)
self.assertTrue(
any(
item.node.title == "Service.run" and ("kind", "function") in item.metadata
for item in projection.nodes
)
)
report = adapter.support_report()
self.assertEqual(1, report["schema_version"])
self.assertEqual(PYTHON_EXTRACTOR_VERSION, report["extractor_version"])
self.assertFalse(report["imports_project_code"])
self.assertFalse(report["executes_project_code"])
self.assertEqual(
[
"call_resolution",
"dynamic_import_resolution",
"inheritance_resolution",
"runtime_generated_facts",
"symbol_reference_resolution",
],
[item.code for item in adapter.unsupported_facts()],
)
self.assertFalse((root / ".cache").exists())
conformance = verify_adapter_conformance(
adapter,
cache_root=root / ".cache" / "conformance",
)
self.assertEqual("reference-python", conformance.project_id)
self.assertEqual(14, conformance.node_count)
self.assertEqual(13, conformance.edge_count)
self.assertEqual(5, conformance.logic_projection_count)
self.assertTrue(conformance.incremental)
def test_incremental_cold_warm_reverse_dependencies_and_logic_equivalence(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
adapter = self.adapter(root)
project = AdapterProject(adapter, cache_root=root / ".cache" / "incremental")
index = ProjectIndex(project)
cold = index.build()
self.assertEqual(4, self.build_metrics(cold)["reparsed_sources"])
self.assertEqual(
[
"src/sample/__init__.py",
"src/sample/service.py",
"src/sample/shared.py",
"src/sample/worker.py",
],
sorted(adapter.extracted_paths),
)
adapter.extracted_paths.clear()
original_parse = python_adapter.ast.parse
with mock.patch.object(
python_adapter.ast,
"parse",
wraps=original_parse,
) as parsed:
warm = index.build()
self.assertEqual(4, self.build_metrics(warm)["cache_hits"])
self.assertEqual(0, self.build_metrics(warm)["reparsed_sources"])
self.assertEqual([], adapter.extracted_paths)
parsed.assert_not_called()
shared = root / "src" / "sample" / "shared.py"
shared.write_text(
shared.read_text(encoding="utf-8").replace(
"DEFAULT_LIMIT = 3", "DEFAULT_LIMIT = 5"
),
encoding="utf-8",
)
adapter.extracted_paths.clear()
with mock.patch.object(
python_adapter.ast,
"parse",
wraps=original_parse,
) as parsed:
changed = index.build()
self.assertEqual(4, self.build_metrics(changed)["invalidated_sources"])
self.assertGreater(parsed.call_count, 0)
self.assertEqual(
[
"src/sample/__init__.py",
"src/sample/service.py",
"src/sample/shared.py",
"src/sample/worker.py",
],
sorted(adapter.extracted_paths),
)
equivalent = project.verify_incremental_equivalence()
self.assertEqual("ok", equivalent["status"])
self.assertEqual(14, equivalent["node_count"])
self.assertEqual(13, equivalent["edge_count"])
self.assertEqual(5, equivalent["logic_projection_count"])
def test_add_rename_delete_and_corrupt_cache_recover_without_stale_facts(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
adapter = self.adapter(root)
cache_root = root / ".cache" / "incremental"
project = AdapterProject(adapter, cache_root=cache_root)
index = ProjectIndex(project)
index.build()
extra = root / "src" / "sample" / "extra.py"
extra.write_text(
'"""Added source."""\n\n\ndef extra() -> str:\n return "extra"\n',
encoding="utf-8",
)
adapter.extracted_paths.clear()
added = index.build()
self.assertEqual(1, self.build_metrics(added)["reparsed_sources"])
self.assertEqual(["src/sample/extra.py"], adapter.extracted_paths)
self.assertIn(
"src/sample/extra.py",
{node.source_path for node in project.load().nodes},
)
worker = root / "src" / "sample" / "worker.py"
runner = root / "src" / "sample" / "runner.py"
worker.rename(runner)
adapter.extracted_paths.clear()
renamed = index.build()
self.assertEqual(1, self.build_metrics(renamed)["deleted_sources"])
self.assertEqual(1, self.build_metrics(renamed)["reparsed_sources"])
renamed_paths = {node.source_path for node in project.load().nodes}
self.assertNotIn("src/sample/worker.py", renamed_paths)
self.assertIn("src/sample/runner.py", renamed_paths)
extra.unlink()
adapter.extracted_paths.clear()
deleted = index.build()
self.assertEqual(1, self.build_metrics(deleted)["deleted_sources"])
self.assertNotIn(
"src/sample/extra.py",
{node.source_path for node in project.load().nodes},
)
(cache_root / "extractions.json").write_text("{broken", encoding="utf-8")
adapter.extracted_paths.clear()
recovered = index.build()
self.assertEqual(4, self.build_metrics(recovered)["reparsed_sources"])
self.assertEqual(
[
"src/sample/__init__.py",
"src/sample/runner.py",
"src/sample/service.py",
"src/sample/shared.py",
],
sorted(adapter.extracted_paths),
)
self.assertEqual("ok", project.verify_incremental_equivalence()["status"])
def test_confinement_and_no_ast_policy_reject_logic_without_importing_code(self) -> None:
with tempfile.TemporaryDirectory() as directory:
parent = Path(directory)
root = self.copy_fixture(parent)
for source_root in (root / "src", Path("../outside")):
with self.subTest(source_root=source_root):
with self.assertRaises(DocForgeError) as captured:
PythonReferenceAdapter(root, source_roots=(source_root,))
self.assertEqual("path_escape", captured.exception.code)
outside = parent / "outside"
outside.mkdir()
linked_root = root / "linked"
linked_root.symlink_to(outside, target_is_directory=True)
with self.assertRaises(DocForgeError) as linked:
PythonReferenceAdapter(root, source_roots=("linked",))
self.assertEqual("path_escape", linked.exception.code)
external_source = outside / "external.py"
external_source.write_text("def outside():\n return True\n", encoding="utf-8")
worker = root / "src" / "sample" / "worker.py"
worker.unlink()
worker.symlink_to(external_source)
adapter = self.adapter(root)
with self.assertRaises(DocForgeError) as source_link:
adapter.load_manifest()
self.assertEqual("path_escape", source_link.exception.code)
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
adapter = self.adapter(root)
project = AdapterProject(adapter, cache_root=root / ".cache" / "no-ast")
with self.assertRaises(DocForgeError) as no_ast:
ProjectIndex(project, allow_logic=False).build()
self.assertEqual("adapter_policy_forbids_logic", no_ast.exception.code)
self.assertFalse(project.descriptor.index_path.exists())

296
tests/test_reference_mcp.py Normal file
View file

@ -0,0 +1,296 @@
from __future__ import annotations
import io
import json
import shutil
import sys
import tempfile
import unittest
from contextlib import redirect_stderr
from pathlib import Path
from unittest import mock
from jsonschema import Draft202012Validator
from mcp.shared.memory import create_connected_server_and_client_session
from docforge.adapter_launcher import AdapterLauncherV1
from docforge.client_config import generate_adapter_client_configuration
from docforge.errors import DocForgeError
from docforge.index import ProjectIndex
from docforge.mcp_server import READ_TOOLS
from docforge.reference_config import (
REFERENCE_ADAPTER_CONFIG,
load_reference_adapter_config,
)
from docforge.reference_mcp import (
REFERENCE_MCP_MODULE,
create_reference_project,
create_reference_server,
main,
)
ROOT = Path(__file__).resolve().parents[1]
FIXTURE = ROOT / "tests" / "fixtures" / "reference-python"
CPP_FIXTURE = ROOT / "tests" / "fixtures" / "reference-cpp"
SCHEMA = json.loads(
(ROOT / "schemas" / "reference-adapter.schema.json").read_text(encoding="utf-8")
)
class ReferenceMcpTests(unittest.IsolatedAsyncioTestCase):
def copy_fixture(self, parent: Path) -> Path:
root = parent / "reference-python"
shutil.copytree(FIXTURE, root)
return root.resolve()
def copy_cpp_fixture(self, parent: Path) -> Path:
root = parent / "reference-cpp"
shutil.copytree(CPP_FIXTURE, root)
return root.resolve()
@staticmethod
def write_config(
root: Path,
*,
language: str = "python",
source_roots: tuple[str, ...] = ("src",),
extra: str = "",
) -> Path:
config = root.joinpath(*REFERENCE_ADAPTER_CONFIG.parts)
config.parent.mkdir(parents=True, exist_ok=True)
quoted_roots = ", ".join(json.dumps(item) for item in source_roots)
config.write_text(
"\n".join(
(
"schema_version = 1",
'project_id = "reference-python"',
'title = "Runnable Python reference"',
f"language = {json.dumps(language)}",
f"source_roots = [{quoted_roots}]",
extra,
"",
)
),
encoding="utf-8",
)
return config
async def test_python_config_build_check_and_real_mcp_retrieval(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
self.write_config(root)
config = load_reference_adapter_config(root)
Draft202012Validator(SCHEMA).validate(config.as_dict())
self.assertEqual("python", config.language)
self.assertEqual((root / "src",), config.source_roots)
self.assertIsNone(config.compilation_database)
self.assertEqual(64, len(config.config_hash))
project = create_reference_project(root)
index = ProjectIndex(project)
built = index.build()
checked = index.check()
self.assertEqual("ok", built["status"])
self.assertEqual("ok", checked["status"])
self.assertTrue(project.descriptor.index_path.is_relative_to(root / ".docforge"))
async with create_connected_server_and_client_session(
create_reference_server(root, capability_mode="read"),
raise_exceptions=True,
) as session:
tools = tuple(tool.name for tool in (await session.list_tools()).tools)
bootstrap = await session.call_tool("docforge_bootstrap", {})
search = await session.call_tool(
"docforge_search",
{"query": "Service", "limit": 5},
)
node_id = search.structuredContent["results"][0]["node_id"]
node = await session.call_tool("docforge_get_node", {"node_id": node_id})
self.assertEqual(READ_TOOLS, tools)
self.assertEqual("ok", bootstrap.structuredContent["status"])
self.assertEqual(
REFERENCE_MCP_MODULE,
bootstrap.structuredContent["binding"]["server_module"],
)
self.assertEqual(
"python",
bootstrap.structuredContent["binding"]["reference_language"],
)
self.assertGreater(search.structuredContent["count"], 0)
self.assertEqual(node_id, node.structuredContent["node"]["node_id"])
async def test_cpp_config_build_check_and_real_mcp_retrieval(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.copy_cpp_fixture(Path(directory))
self.write_config(
root,
language="cpp",
source_roots=("include", "src"),
extra='compilation_database = "compile_commands.json"',
)
config = load_reference_adapter_config(root)
Draft202012Validator(SCHEMA).validate(config.as_dict())
project = create_reference_project(root)
index = ProjectIndex(project)
self.assertEqual("ok", index.build()["status"])
self.assertEqual("ok", index.check()["status"])
async with create_connected_server_and_client_session(
create_reference_server(root, capability_mode="read"),
raise_exceptions=True,
) as session:
bootstrap = await session.call_tool("docforge_bootstrap", {})
search = await session.call_tool(
"docforge_search",
{"query": "worker", "limit": 5},
)
node_id = search.structuredContent["results"][0]["node_id"]
node = await session.call_tool("docforge_get_node", {"node_id": node_id})
self.assertEqual("cpp", config.language)
self.assertEqual(root / "compile_commands.json", config.compilation_database)
self.assertEqual(
"cpp",
bootstrap.structuredContent["binding"]["reference_language"],
)
self.assertEqual(node_id, node.structuredContent["node"]["node_id"])
def test_launcher_generates_exact_isolated_module_invocation(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
self.write_config(root)
project = create_reference_project(root)
launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE)
with mock.patch("docforge.client_config.subprocess.run") as executed:
result = generate_adapter_client_configuration(
project,
launcher,
"codex",
capability_mode="read",
)
self.assertEqual(
[
"-I",
"-m",
REFERENCE_MCP_MODULE,
"--project-root",
str(root),
"--capability-mode",
"read",
],
result["binding"]["args"],
)
self.assertEqual({}, result["binding"]["environment"])
self.assertNotIn("cwd", result["binding"])
executed.assert_not_called()
def test_config_drift_requires_runtime_restart(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
config_path = self.write_config(root)
project = create_reference_project(root)
project.validate_runtime()
config_path.write_text(
config_path.read_text(encoding="utf-8").replace(
"Runnable Python reference",
"Changed Python reference",
),
encoding="utf-8",
)
with self.assertRaises(DocForgeError) as captured:
project.validate_runtime()
self.assertEqual("adapter_restart_required", captured.exception.code)
self.assertEqual(
[REFERENCE_ADAPTER_CONFIG.as_posix()],
captured.exception.details["changed"],
)
def test_config_rejects_unknown_fields_languages_and_language_mismatch(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
for language, extra, code in (
("ruby", "", "unsupported_reference_language"),
("javascript", "", "reference_language_mismatch"),
("python", 'command = "python project.py"', "invalid_reference_config"),
(
"python",
'compilation_database = "compile_commands.json"',
"invalid_reference_config",
),
("cpp", "", "invalid_reference_config"),
):
with self.subTest(language=language, extra=extra):
self.write_config(root, language=language, extra=extra)
with self.assertRaises(DocForgeError) as captured:
load_reference_adapter_config(root)
self.assertEqual(code, captured.exception.code)
def test_config_rejects_path_escape_and_source_symlinks(self) -> None:
with tempfile.TemporaryDirectory() as directory:
parent = Path(directory)
root = self.copy_fixture(parent)
outside = parent / "outside"
outside.mkdir()
self.write_config(root, source_roots=("../outside",))
with self.assertRaises(DocForgeError) as escaped:
load_reference_adapter_config(root)
self.assertEqual("path_escape", escaped.exception.code)
linked = root / "linked"
linked.symlink_to(outside, target_is_directory=True)
self.write_config(root, source_roots=("linked",))
with self.assertRaises(DocForgeError) as linked_root:
load_reference_adapter_config(root)
self.assertEqual("path_escape", linked_root.exception.code)
self.write_config(root)
external_source = outside / "external.py"
external_source.write_text("def external(): pass\n", encoding="utf-8")
source_link = root / "src" / "external.py"
source_link.symlink_to(external_source)
with self.assertRaises(DocForgeError) as linked_source:
load_reference_adapter_config(root)
self.assertEqual("path_escape", linked_source.exception.code)
def test_config_file_must_be_a_regular_non_symlink(self) -> None:
with tempfile.TemporaryDirectory() as directory:
parent = Path(directory)
root = self.copy_fixture(parent)
config_path = self.write_config(root)
outside = parent / "reference-adapter.toml"
shutil.copyfile(config_path, outside)
config_path.unlink()
config_path.symlink_to(outside)
with self.assertRaises(DocForgeError) as captured:
load_reference_adapter_config(root)
self.assertEqual("path_escape", captured.exception.code)
def test_reference_module_cli_advertises_read_mode_only(self) -> None:
with (
redirect_stderr(io.StringIO()),
mock.patch.object(
sys,
"argv",
[
REFERENCE_MCP_MODULE,
"--project-root",
"/irrelevant",
"--capability-mode",
"proposal",
],
),
mock.patch("docforge.reference_mcp.create_reference_server") as create_server,
self.assertRaises(SystemExit) as captured,
):
main()
self.assertEqual(2, captured.exception.code)
create_server.assert_not_called()

View file

@ -7,6 +7,8 @@ from docforge.treesitter_logic import (
TreeSitterLogicOwner,
analyze_cpp_source,
analyze_javascript_source,
analyze_typescript_source,
discover_typescript_functions,
)
@ -200,5 +202,44 @@ public:
self.assertEqual(missing.exception.code, "missing_logic_owner")
class TypeScriptLogicTests(unittest.TestCase):
def test_typed_function_and_method_use_the_typescript_grammar(self) -> None:
source = """
interface Choice {
enabled: boolean;
}
function choose(choice: Choice): number {
if (choice.enabled) {
return 1;
}
return 0;
}
class Worker {
run(choice: Choice): number {
return choose(choice);
}
}
""".strip()
discovered = discover_typescript_functions(source)
self.assertEqual(
[("Worker.run", 13), ("choose", 5)],
[(item.qualified_name, item.line) for item in discovered],
)
projection = analyze_typescript_source(
source,
source_id="source.typescript",
owners=(TreeSitterLogicOwner("ts.symbol.choose", "choose", 5),),
)[0]
self.assertIn("choice.enabled", {node.label for node in projection.nodes})
self.assertEqual(
{"return"},
{edge.relation for edge in projection.edges if edge.relation == "return"},
)
if __name__ == "__main__":
unittest.main()

View file

@ -0,0 +1,698 @@
"""Validate the maintained DocForge Markdown documentation graph."""
from __future__ import annotations
import argparse
import html
import json
import re
import stat
import sys
import tomllib
import unicodedata
from collections import deque
from collections.abc import Iterable, Sequence
from dataclasses import dataclass
from pathlib import Path
from typing import Protocol, cast
from urllib.parse import unquote, urlsplit
from jsonschema import Draft202012Validator
from jsonschema.exceptions import ValidationError
from markdown_it import MarkdownIt
from markdown_it.token import Token
REPOSITORY_ROOT = Path(__file__).resolve().parents[1]
DEFAULT_PENDING_INVENTORY = Path("tools/docs_pending_m4_pages.txt")
COMMAND_REFERENCE = Path("docs/COMMAND_REFERENCE.md")
COMMAND_REFERENCE_H1 = "DocForge command reference"
COMMAND_REFERENCE_NOTICE = (
"> Generated from the live CLI parser and MCP registrations. Do not edit this file by hand."
)
REFERENCE_ADAPTER_SCHEMA = Path("schemas/reference-adapter.schema.json")
MAX_MARKDOWN_PAGES = 256
MAX_PAGE_BYTES = 2_000_000
MAX_TOTAL_BYTES = 20_000_000
MAX_DIAGNOSTICS = 100
REQUIRED_MILESTONE_4_PAGES = (
Path("README.md"),
Path("docs/ADAPTER_AUTHORING_GUIDE.md"),
Path("docs/AGENT_INTEGRATION.md"),
Path("docs/APPLICATION_DECISION.md"),
Path("docs/COMMAND_REFERENCE.md"),
Path("docs/COMPATIBILITY.md"),
Path("docs/CONTRACT.md"),
Path("docs/CORE_CONCEPTS_AND_AUTHORITY.md"),
Path("docs/INCREMENTAL_INDEXING.md"),
Path("docs/LEGACY_AND_NO_AST.md"),
Path("docs/MCP_CONTRACT.md"),
Path("docs/MIGRATING_FROM_V1.md"),
Path("docs/MILESTONE_4_BASELINE.md"),
Path("docs/MILESTONE_4_CLOSEOUT.md"),
Path("docs/NEW_PROJECT_QUICKSTART.md"),
Path("docs/POLICY_PRECEDENCE.md"),
Path("docs/PROJECT_DESCRIPTOR.md"),
Path("docs/PROJECT_ONBOARDING.md"),
Path("docs/RECOVERY_AND_PERFORMANCE.md"),
Path("docs/REFERENCE_ADAPTERS.md"),
Path("docs/RENDERING_AND_VISUALIZATION.md"),
Path("docs/SECURITY.md"),
Path("docs/USER_MANUAL.md"),
Path("docs/VIEWER_MANAGER.md"),
)
_HISTORICAL_RECORD = re.compile(r"^docs/MILESTONE_[0-3]_(?:BASELINE|CLOSEOUT)\.md$")
_HTML_ID = re.compile(
r"""\bid\s*=\s*(?:"([^"]+)"|'([^']+)')""",
re.IGNORECASE,
)
_REFERENCE_ADAPTER_KEY = re.compile(r"(?m)^(?:language|source_roots|compilation_database)\s*=")
_URI_SCHEME = re.compile(r"^[A-Za-z][A-Za-z0-9+.-]*:")
@dataclass(frozen=True, order=True)
class Diagnostic:
"""One deterministic documentation validation failure."""
path: str
line: int
code: str
message: str
def render(self) -> str:
"""Render one bounded compiler-style diagnostic."""
return f"{self.path}:{self.line}: {self.code}: {self.message}"
@dataclass(frozen=True)
class DocumentationPolicy:
"""Repository-relative pages and temporary inventory accepted by one run."""
required_pages: tuple[Path, ...] = REQUIRED_MILESTONE_4_PAGES
pending_inventory: Path | None = DEFAULT_PENDING_INVENTORY
@dataclass(frozen=True)
class DocumentationReport:
"""Bounded deterministic result from one complete documentation check."""
diagnostics: tuple[Diagnostic, ...]
omitted_diagnostics: int
markdown_pages: int
pending_pages: int
reference_adapter_examples: int
@property
def ok(self) -> bool:
"""Return whether every documentation invariant passed."""
return not self.diagnostics and self.omitted_diagnostics == 0
@dataclass(frozen=True)
class _Page:
relative: Path
source: str
tokens: tuple[Token, ...]
anchors: frozenset[str]
class _Diagnostics:
def __init__(self) -> None:
self._items: list[Diagnostic] = []
self._omitted = 0
def add(self, path: Path | str, line: int, code: str, message: str) -> None:
relative = path.as_posix() if isinstance(path, Path) else path
item = Diagnostic(relative, max(1, line), code, message)
if len(self._items) < MAX_DIAGNOSTICS:
self._items.append(item)
else:
self._omitted += 1
def result(self) -> tuple[tuple[Diagnostic, ...], int]:
return tuple(sorted(self._items)), self._omitted
class _Validator(Protocol):
def iter_errors(self, instance: object) -> Iterable[ValidationError]: ...
def check_documentation(
repository_root: Path,
*,
policy: DocumentationPolicy | None = None,
) -> DocumentationReport:
"""Check one repository's maintained Markdown pages without modifying it."""
root = repository_root.resolve(strict=True)
if not root.is_dir():
raise ValueError(f"repository root is not a directory: {root}")
effective_policy = policy or DocumentationPolicy()
diagnostics = _Diagnostics()
parser = MarkdownIt("commonmark", {"html": True, "typographer": False})
page_paths = _discover_pages(root, diagnostics)
pages = _load_pages(root, page_paths, parser, diagnostics)
pending = _load_pending_inventory(root, effective_policy, diagnostics)
_check_required_inventory(
root,
effective_policy.required_pages,
pending,
diagnostics,
)
_check_h1s(pages, diagnostics)
link_graph = _check_links(root, pages, parser, diagnostics)
_check_reachability(pages, link_graph, diagnostics)
reference_examples = _check_reference_adapter_examples(root, pages, diagnostics)
_check_command_reference(pages, diagnostics)
items, omitted = diagnostics.result()
return DocumentationReport(
diagnostics=items,
omitted_diagnostics=omitted,
markdown_pages=len(pages),
pending_pages=len(pending),
reference_adapter_examples=reference_examples,
)
def _discover_pages(root: Path, diagnostics: _Diagnostics) -> tuple[Path, ...]:
candidates = [Path("README.md")]
docs = root / "docs"
if not docs.is_dir():
diagnostics.add("docs", 1, "DOC001", "maintained documentation directory is missing")
return tuple(candidates)
candidates.extend(
path.relative_to(root)
for path in docs.rglob("*.md")
if not path.is_symlink() and path.is_file()
)
unique = tuple(sorted(set(candidates), key=Path.as_posix))
if len(unique) > MAX_MARKDOWN_PAGES:
diagnostics.add(
"docs",
1,
"DOC002",
f"found {len(unique)} Markdown pages; limit is {MAX_MARKDOWN_PAGES}",
)
return unique[:MAX_MARKDOWN_PAGES]
return unique
def _load_pages(
root: Path,
paths: tuple[Path, ...],
parser: MarkdownIt,
diagnostics: _Diagnostics,
) -> dict[Path, _Page]:
pages: dict[Path, _Page] = {}
total_bytes = 0
for relative in paths:
absolute = root / relative
try:
metadata = absolute.lstat()
except FileNotFoundError:
diagnostics.add(relative, 1, "DOC003", "maintained Markdown page is missing")
continue
if not stat.S_ISREG(metadata.st_mode):
diagnostics.add(relative, 1, "DOC004", "maintained Markdown page is not a regular file")
continue
if metadata.st_size > MAX_PAGE_BYTES:
diagnostics.add(
relative,
1,
"DOC005",
f"page is {metadata.st_size} bytes; limit is {MAX_PAGE_BYTES}",
)
continue
total_bytes += metadata.st_size
if total_bytes > MAX_TOTAL_BYTES:
diagnostics.add(
relative,
1,
"DOC006",
f"total Markdown input exceeds {MAX_TOTAL_BYTES} bytes",
)
break
try:
source = absolute.read_text(encoding="utf-8")
except UnicodeDecodeError:
diagnostics.add(relative, 1, "DOC007", "page is not valid UTF-8")
continue
tokens = tuple(parser.parse(source))
pages[relative] = _Page(
relative=relative,
source=source,
tokens=tokens,
anchors=_anchors(tokens),
)
return pages
def _load_pending_inventory(
root: Path,
policy: DocumentationPolicy,
diagnostics: _Diagnostics,
) -> frozenset[Path]:
inventory = policy.pending_inventory
if inventory is None:
return frozenset()
if inventory.is_absolute():
diagnostics.add(
inventory.as_posix(),
1,
"DOC008",
"pending inventory path must be repository-relative",
)
return frozenset()
absolute = root / inventory
try:
lines = absolute.read_text(encoding="utf-8").splitlines()
except FileNotFoundError:
diagnostics.add(inventory, 1, "DOC009", "pending inventory file is missing")
return frozenset()
except UnicodeDecodeError:
diagnostics.add(inventory, 1, "DOC010", "pending inventory is not valid UTF-8")
return frozenset()
required = set(policy.required_pages)
pending: set[Path] = set()
for line_number, raw in enumerate(lines, start=1):
value = raw.strip()
if not value or value.startswith("#"):
continue
path = Path(value)
if path.is_absolute() or ".." in path.parts or path.as_posix() != value:
diagnostics.add(
inventory,
line_number,
"DOC011",
f"unsafe pending inventory path: {value!r}",
)
continue
if path not in required:
diagnostics.add(
inventory,
line_number,
"DOC012",
f"pending page is not in the required inventory: {value}",
)
continue
if path in pending:
diagnostics.add(
inventory,
line_number,
"DOC013",
f"duplicate pending page: {value}",
)
continue
pending.add(path)
return frozenset(pending)
def _check_required_inventory(
root: Path,
required_pages: tuple[Path, ...],
pending: frozenset[Path],
diagnostics: _Diagnostics,
) -> None:
for relative in sorted(set(required_pages), key=Path.as_posix):
exists = (root / relative).is_file()
if relative in pending:
if exists:
diagnostics.add(
relative,
1,
"DOC014",
"page exists but remains in the pending inventory",
)
continue
if not exists:
diagnostics.add(relative, 1, "DOC015", "required Milestone 4 page is missing")
def _check_h1s(pages: dict[Path, _Page], diagnostics: _Diagnostics) -> None:
for relative, page in pages.items():
h1_lines = [
_token_line(token)
for token in page.tokens
if token.type == "heading_open" and token.tag == "h1"
]
if len(h1_lines) != 1:
rendered = ", ".join(str(line) for line in h1_lines) or "none"
diagnostics.add(
relative,
h1_lines[1] if len(h1_lines) > 1 else 1,
"DOC016",
f"expected exactly one H1; found {len(h1_lines)} (lines: {rendered})",
)
def _check_links(
root: Path,
pages: dict[Path, _Page],
parser: MarkdownIt,
diagnostics: _Diagnostics,
) -> dict[Path, frozenset[Path]]:
graph: dict[Path, frozenset[Path]] = {}
anchor_cache = {relative: page.anchors for relative, page in pages.items()}
for relative, page in pages.items():
destinations: set[Path] = set()
for href, line in _page_links(page.tokens):
target = _resolve_local_link(root, relative, href, line, diagnostics)
if target is None:
continue
target_relative, fragment = target
if target_relative in pages:
destinations.add(target_relative)
if not fragment:
continue
anchors = anchor_cache.get(target_relative)
if anchors is None:
anchors = _load_link_target_anchors(
root,
target_relative,
parser,
diagnostics,
)
if anchors is None:
continue
anchor_cache[target_relative] = anchors
if fragment not in anchors:
diagnostics.add(
relative,
line,
"DOC017",
f"missing anchor #{fragment} in {target_relative.as_posix()}",
)
graph[relative] = frozenset(destinations)
return graph
def _page_links(tokens: tuple[Token, ...]) -> Iterable[tuple[str, int]]:
for token in tokens:
children = token.children or []
for child in children:
attribute = "href" if child.type == "link_open" else "src"
if child.type not in {"link_open", "image"}:
continue
raw = child.attrs.get(attribute)
if isinstance(raw, str):
yield raw, _token_line(token)
def _resolve_local_link(
root: Path,
source: Path,
href: str,
line: int,
diagnostics: _Diagnostics,
) -> tuple[Path, str] | None:
if not href or href.startswith("//") or _URI_SCHEME.match(href):
return None
try:
parsed = urlsplit(href)
except ValueError as error:
diagnostics.add(source, line, "DOC018", f"invalid link {href!r}: {error}")
return None
if parsed.netloc:
return None
decoded_path = unquote(parsed.path)
fragment = unquote(parsed.fragment)
if "\x00" in decoded_path or decoded_path.startswith("/"):
diagnostics.add(source, line, "DOC019", f"local link escapes the repository: {href!r}")
return None
absolute = (root / source if not decoded_path else root / source.parent / decoded_path).resolve(
strict=False
)
try:
target = absolute.relative_to(root)
except ValueError:
diagnostics.add(source, line, "DOC019", f"local link escapes the repository: {href!r}")
return None
if not absolute.is_file():
diagnostics.add(source, line, "DOC020", f"missing local link target: {target.as_posix()}")
return None
return target, fragment
def _load_link_target_anchors(
root: Path,
relative: Path,
parser: MarkdownIt,
diagnostics: _Diagnostics,
) -> frozenset[str] | None:
if relative.suffix.lower() not in {".md", ".markdown"}:
diagnostics.add(
relative,
1,
"DOC021",
"link uses an anchor on a non-Markdown target",
)
return None
absolute = root / relative
try:
metadata = absolute.lstat()
if not stat.S_ISREG(metadata.st_mode) or metadata.st_size > MAX_PAGE_BYTES:
raise ValueError("target is not a bounded regular Markdown file")
source = absolute.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError, ValueError) as error:
diagnostics.add(relative, 1, "DOC022", f"cannot inspect linked Markdown anchors: {error}")
return None
return _anchors(tuple(parser.parse(source)))
def _anchors(tokens: tuple[Token, ...]) -> frozenset[str]:
anchors: set[str] = set()
slug_counts: dict[str, int] = {}
for index, token in enumerate(tokens):
if token.type in {"html_block", "html_inline"}:
anchors.update(_html_ids(token.content))
if token.children:
for child in token.children:
if child.type == "html_inline":
anchors.update(_html_ids(child.content))
if token.type != "heading_open" or index + 1 >= len(tokens):
continue
inline = tokens[index + 1]
base = _heading_slug(_inline_text(inline))
duplicate = slug_counts.get(base, 0)
slug_counts[base] = duplicate + 1
anchors.add(base if duplicate == 0 else f"{base}-{duplicate}")
return frozenset(anchors)
def _inline_text(token: Token) -> str:
if not token.children:
return token.content
pieces: list[str] = []
for child in token.children:
if child.type in {"text", "code_inline"}:
pieces.append(child.content)
elif child.type in {"softbreak", "hardbreak"}:
pieces.append(" ")
elif child.type == "image":
pieces.append(child.content)
elif child.type == "html_inline":
pieces.append(re.sub(r"<[^>]*>", "", child.content))
return "".join(pieces)
def _heading_slug(value: str) -> str:
characters: list[str] = []
for character in html.unescape(value).casefold().strip():
category = unicodedata.category(character)
if category[0] in {"L", "M", "N"} or character in {"-", "_"}:
characters.append(character)
elif character.isspace():
characters.append("-")
return "".join(characters)
def _html_ids(value: str) -> set[str]:
return {first or second for first, second in _HTML_ID.findall(value)}
def _check_reachability(
pages: dict[Path, _Page],
graph: dict[Path, frozenset[Path]],
diagnostics: _Diagnostics,
) -> None:
root_page = Path("README.md")
if root_page not in pages:
return
reached = {root_page}
queue = deque([root_page])
while queue:
current = queue.popleft()
for target in sorted(graph.get(current, frozenset()), key=Path.as_posix):
if target not in reached:
reached.add(target)
queue.append(target)
for relative in sorted(set(pages) - reached, key=Path.as_posix):
if _HISTORICAL_RECORD.fullmatch(relative.as_posix()):
continue
diagnostics.add(
relative,
1,
"DOC023",
"maintained page is not reachable from README.md",
)
def _check_reference_adapter_examples(
root: Path,
pages: dict[Path, _Page],
diagnostics: _Diagnostics,
) -> int:
schema_path = root / REFERENCE_ADAPTER_SCHEMA
try:
schema_document = cast(
dict[str, object],
json.loads(schema_path.read_text(encoding="utf-8")),
)
Draft202012Validator.check_schema(schema_document)
except (OSError, UnicodeDecodeError, json.JSONDecodeError, TypeError) as error:
diagnostics.add(
REFERENCE_ADAPTER_SCHEMA,
1,
"DOC024",
f"cannot load reference-adapter schema: {error}",
)
return 0
validator = cast(_Validator, Draft202012Validator(schema_document))
count = 0
for relative, page in pages.items():
previous_inline: Token | None = None
for token in page.tokens:
if token.type == "inline":
previous_inline = token
continue
if token.type != "fence" or token.info.split(maxsplit=1)[:1] != ["toml"]:
continue
if not _is_reference_adapter_example(token, previous_inline):
continue
count += 1
line = _token_line(token)
try:
document = tomllib.loads(token.content)
except tomllib.TOMLDecodeError as error:
diagnostics.add(
relative,
line,
"DOC025",
f"invalid reference-adapter TOML example: {error}",
)
continue
errors = sorted(
validator.iter_errors(document),
key=lambda error: tuple(str(part) for part in error.absolute_path),
)
for error in errors:
location = ".".join(str(part) for part in error.absolute_path) or "<root>"
diagnostics.add(
relative,
line,
"DOC026",
f"reference-adapter example {location}: {error.message}",
)
return count
def _is_reference_adapter_example(token: Token, previous_inline: Token | None) -> bool:
info = token.info.casefold()
if "reference-adapter" in info or "reference_adapter" in info:
return True
if previous_inline is not None and previous_inline.map is not None and token.map is not None:
adjacent = token.map[0] - previous_inline.map[1] <= 1
context = previous_inline.content.casefold()
if adjacent and (
"reference-adapter.toml" in context or "reference adapter configuration" in context
):
return True
keys = set(_REFERENCE_ADAPTER_KEY.findall(token.content))
return {"language", "source_roots"}.issubset(keys)
def _check_command_reference(
pages: dict[Path, _Page],
diagnostics: _Diagnostics,
) -> None:
page = pages.get(COMMAND_REFERENCE)
if page is None:
return
lines = page.source.splitlines()
if not lines or lines[0] != f"# {COMMAND_REFERENCE_H1}":
diagnostics.add(
COMMAND_REFERENCE,
1,
"DOC027",
f"generated reference must begin with '# {COMMAND_REFERENCE_H1}'",
)
if COMMAND_REFERENCE_NOTICE not in lines[:5]:
diagnostics.add(
COMMAND_REFERENCE,
2,
"DOC028",
"generated reference notice is missing or changed",
)
def _token_line(token: Token) -> int:
return token.map[0] + 1 if token.map else 1
def _parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
description="Check deterministic DocForge documentation invariants."
)
parser.add_argument("--repository-root", type=Path, default=REPOSITORY_ROOT)
parser.add_argument(
"--pending-inventory",
type=Path,
default=DEFAULT_PENDING_INVENTORY,
help="repository-relative list of required pages not authored yet",
)
return parser
def main(arguments: Sequence[str] | None = None) -> int:
parsed = _parser().parse_args(arguments)
try:
report = check_documentation(
parsed.repository_root,
policy=DocumentationPolicy(pending_inventory=parsed.pending_inventory),
)
except Exception as error:
print(f"documentation check failed: {error}", file=sys.stderr)
return 2
if not report.ok:
for diagnostic in report.diagnostics:
print(diagnostic.render(), file=sys.stderr)
if report.omitted_diagnostics:
print(
f"... {report.omitted_diagnostics} additional diagnostics omitted "
f"(limit {MAX_DIAGNOSTICS})",
file=sys.stderr,
)
return 1
print(
json.dumps(
{
"markdown_pages": report.markdown_pages,
"pending_pages": report.pending_pages,
"reference_adapter_examples": report.reference_adapter_examples,
"status": "ok",
},
sort_keys=True,
separators=(",", ":"),
)
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -0,0 +1 @@
# Milestone 4 required-page inventory is complete.

View file

@ -0,0 +1,533 @@
"""Generate or verify the deterministic DocForge CLI and MCP command reference."""
from __future__ import annotations
import argparse
import asyncio
import contextlib
import ctypes
import errno
import fcntl
import json
import os
import secrets
import shutil
import stat
import sys
import tempfile
from collections.abc import Sequence
from dataclasses import dataclass
from pathlib import Path
from typing import Literal, Protocol, cast
from mcp.shared.memory import create_connected_server_and_client_session
from docforge.command_reference import (
CliCommandReference,
McpToolReference,
cli_command_references,
mcp_tool_references,
render_command_reference_markdown,
)
from docforge.index import ProjectIndex
from docforge.mcp_server import ALL_TOOLS, APPLICATION_TOOLS, create_server
from docforge.project import Project
REPOSITORY_ROOT = Path(__file__).resolve().parents[1]
DEFAULT_PROJECT_ROOT = REPOSITORY_ROOT / "tests" / "fixtures" / "alpha"
EXPECTED_CLI_ROWS = 28
EXPECTED_MCP_ROWS = 36
MAX_REFERENCE_BYTES = 5_000_000
RENAME_EXCHANGE = 2
class CommandReferenceToolError(RuntimeError):
"""One safe repository-tool validation or publication failure."""
class CommandReferenceDrift(CommandReferenceToolError):
"""The checked output does not match current registered command metadata."""
@dataclass(frozen=True)
class _FileIdentity:
device: int
inode: int
mode: int
owner: int
group: int
size: int
modified_ns: int
changed_ns: int
class _RenameAt2(Protocol):
argtypes: list[object]
restype: object
def __call__(
self,
old_directory_fd: int,
old_name: bytes,
new_directory_fd: int,
new_name: bytes,
flags: int,
/,
) -> int: ...
async def collect_command_reference_rows(
project_root: Path,
*,
proposal_writer: str,
canonical_applier: str,
) -> tuple[tuple[CliCommandReference, ...], tuple[McpToolReference, ...]]:
"""Collect the real CLI parser and full project-bound MCP registration surface."""
source = _safe_project_root(project_root)
with tempfile.TemporaryDirectory(prefix="docforge-command-reference-") as directory:
copied = Path(directory) / "project"
shutil.copytree(source, copied, symlinks=True)
ProjectIndex(Project.open(copied)).build()
server = create_server(
copied,
proposal_writer,
canonical_applier_id=canonical_applier,
)
async with create_connected_server_and_client_session(
server,
raise_exceptions=True,
) as session:
tools = tuple((await session.list_tools()).tools)
cli_rows = cli_command_references()
expected_mcp_names = (*ALL_TOOLS, *APPLICATION_TOOLS)
mcp_rows = mcp_tool_references(tools, expected_names=expected_mcp_names)
if len(cli_rows) != EXPECTED_CLI_ROWS:
raise CommandReferenceToolError(
f"Expected {EXPECTED_CLI_ROWS} CLI rows, found {len(cli_rows)}"
)
if len(mcp_rows) != EXPECTED_MCP_ROWS:
raise CommandReferenceToolError(
f"Expected {EXPECTED_MCP_ROWS} MCP rows, found {len(mcp_rows)}"
)
return cli_rows, mcp_rows
def generate_command_reference_bytes(
project_root: Path,
*,
proposal_writer: str = "alpha-editor",
canonical_applier: str = "alpha-editor",
) -> bytes:
"""Render one stable Markdown document from an isolated project copy."""
cli_rows, mcp_rows = asyncio.run(
collect_command_reference_rows(
project_root,
proposal_writer=proposal_writer,
canonical_applier=canonical_applier,
)
)
rendered = render_command_reference_markdown(cli_rows, mcp_rows).encode("utf-8")
if len(rendered) > MAX_REFERENCE_BYTES:
raise CommandReferenceToolError("Generated command reference exceeds its byte limit")
return rendered
def publish_or_check_command_reference(
content: bytes,
*,
repository_root: Path,
project_root: Path,
output: Path,
check: bool,
) -> Literal["current", "unchanged", "written"]:
"""Safely check or atomically publish one repository-confined Markdown file."""
if len(content) > MAX_REFERENCE_BYTES:
raise CommandReferenceToolError("Command reference exceeds its byte limit")
root = _safe_repository_root(repository_root)
source = _safe_project_root(project_root)
relative = _safe_output_relative(root, source, output)
parent_parts = relative.parts[:-1]
target_name = relative.name
parent_fd = _open_relative_directory(root, parent_parts)
try:
fcntl.flock(parent_fd, fcntl.LOCK_EX)
try:
initial = _file_identity(parent_fd, target_name)
existing = (
_read_regular_file(parent_fd, target_name, limit=MAX_REFERENCE_BYTES)
if initial is not None
else None
)
if _file_identity(parent_fd, target_name) != initial:
raise CommandReferenceToolError("Output target changed during inspection")
if check:
if existing != content:
raise CommandReferenceDrift(
f"Command reference is missing or stale: {relative.as_posix()}"
)
return "current"
if existing == content:
return "unchanged"
_atomic_replace(
parent_fd,
target_name,
content,
expected=initial,
expected_content=existing,
)
return "written"
finally:
fcntl.flock(parent_fd, fcntl.LOCK_UN)
finally:
os.close(parent_fd)
def _safe_repository_root(path: Path) -> Path:
if path.is_symlink():
raise CommandReferenceToolError("Repository root must not be a symbolic link")
try:
resolved = path.resolve(strict=True)
except OSError as error:
raise CommandReferenceToolError("Repository root does not exist") from error
if not resolved.is_dir():
raise CommandReferenceToolError("Repository root must be a directory")
return resolved
def _safe_project_root(path: Path) -> Path:
if path.is_symlink():
raise CommandReferenceToolError("Project fixture root must not be a symbolic link")
try:
resolved = path.resolve(strict=True)
except OSError as error:
raise CommandReferenceToolError("Project fixture root does not exist") from error
if not resolved.is_dir():
raise CommandReferenceToolError("Project fixture root must be a directory")
return resolved
def _safe_output_relative(root: Path, project_root: Path, output: Path) -> Path:
candidate = output if output.is_absolute() else root / output
absolute = Path(os.path.abspath(candidate))
if absolute == root or not absolute.is_relative_to(root):
raise CommandReferenceToolError("Output must remain inside the repository root")
if absolute == project_root or absolute.is_relative_to(project_root):
raise CommandReferenceToolError("Output must not modify the source project fixture")
relative = absolute.relative_to(root)
if relative.suffix != ".md":
raise CommandReferenceToolError("Output must be one Markdown file")
if not relative.name or len(relative.parts) < 2:
raise CommandReferenceToolError("Output must be below an existing repository directory")
return relative
def _open_relative_directory(root: Path, parts: tuple[str, ...]) -> int:
flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW
descriptor = os.open(root, flags)
try:
for part in parts:
if part in {"", ".", ".."}:
raise CommandReferenceToolError("Output directory is unsafe")
try:
next_descriptor = os.open(part, flags, dir_fd=descriptor)
except OSError as error:
raise CommandReferenceToolError(
"Output parent must be an existing non-symlink directory"
) from error
os.close(descriptor)
descriptor = next_descriptor
return descriptor
except Exception:
os.close(descriptor)
raise
def _file_identity(directory_fd: int, name: str) -> _FileIdentity | None:
try:
status = os.stat(name, dir_fd=directory_fd, follow_symlinks=False)
except FileNotFoundError:
return None
if not stat.S_ISREG(status.st_mode):
raise CommandReferenceToolError("Output target must be a regular file")
return _FileIdentity(
device=status.st_dev,
inode=status.st_ino,
mode=status.st_mode,
owner=status.st_uid,
group=status.st_gid,
size=status.st_size,
modified_ns=status.st_mtime_ns,
changed_ns=status.st_ctime_ns,
)
def _read_regular_file(directory_fd: int, name: str, *, limit: int) -> bytes:
try:
descriptor = os.open(name, os.O_RDONLY | os.O_NOFOLLOW, dir_fd=directory_fd)
except OSError as error:
raise CommandReferenceToolError("Output target could not be opened safely") from error
try:
before = os.fstat(descriptor)
if not stat.S_ISREG(before.st_mode) or before.st_size > limit:
raise CommandReferenceToolError("Output target is not a bounded regular file")
chunks: list[bytes] = []
remaining = limit + 1
while remaining:
chunk = os.read(descriptor, min(remaining, 64 * 1024))
if not chunk:
break
chunks.append(chunk)
remaining -= len(chunk)
content = b"".join(chunks)
after = os.fstat(descriptor)
if len(content) > limit or _identity_from_stat(before) != _identity_from_stat(after):
raise CommandReferenceToolError("Output target changed during inspection")
return content
finally:
os.close(descriptor)
def _identity_from_stat(status: os.stat_result) -> _FileIdentity:
return _FileIdentity(
device=status.st_dev,
inode=status.st_ino,
mode=status.st_mode,
owner=status.st_uid,
group=status.st_gid,
size=status.st_size,
modified_ns=status.st_mtime_ns,
changed_ns=status.st_ctime_ns,
)
def _atomic_replace(
directory_fd: int,
name: str,
content: bytes,
*,
expected: _FileIdentity | None,
expected_content: bytes | None,
) -> None:
temporary_name = f".{name}.docforge-command-reference-{secrets.token_hex(12)}"
flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW
try:
descriptor = os.open(temporary_name, flags, 0o644, dir_fd=directory_fd)
except OSError as error:
raise CommandReferenceToolError("Could not create atomic output temporary") from error
try:
view = memoryview(content)
while view:
written = os.write(descriptor, view)
if written <= 0:
raise CommandReferenceToolError("Could not write command reference output")
view = view[written:]
os.fsync(descriptor)
except Exception:
os.close(descriptor)
_unlink_at(directory_fd, temporary_name)
raise
else:
os.close(descriptor)
temporary_contains_only_staged_content = True
try:
staged = _file_identity(directory_fd, temporary_name)
if staged is None:
raise CommandReferenceToolError("Atomic output temporary disappeared")
if _file_identity(directory_fd, name) != expected:
raise CommandReferenceToolError("Output target changed before atomic publication")
if expected is None:
_link_no_replace(directory_fd, temporary_name, name)
_unlink_at(directory_fd, temporary_name)
os.fsync(directory_fd)
return
if expected_content is None:
raise CommandReferenceToolError("Expected output content was not captured")
_rename_exchange(directory_fd, temporary_name, name)
temporary_contains_only_staged_content = False
displaced = _file_identity(directory_fd, temporary_name)
published = _file_identity(directory_fd, name)
displaced_content = _read_regular_file(
directory_fd,
temporary_name,
limit=MAX_REFERENCE_BYTES,
)
if (
displaced is None
or published is None
or not _same_identity_after_rename(displaced, expected)
or displaced_content != expected_content
or not _same_identity_after_rename(published, staged)
):
try:
_rename_exchange(directory_fd, temporary_name, name)
except Exception as error:
raise CommandReferenceToolError(
"Output target raced publication; displaced data was retained "
f"in {temporary_name}"
) from error
temporary_contains_only_staged_content = True
if not _same_identity_after_rename(
_file_identity(directory_fd, name),
displaced,
) or not _same_identity_after_rename(
_file_identity(directory_fd, temporary_name),
staged,
):
raise CommandReferenceToolError(
"Output target raced publication and could not be safely restored"
)
raise CommandReferenceToolError("Output target changed during atomic publication")
_unlink_at(directory_fd, temporary_name)
os.fsync(directory_fd)
except Exception:
if temporary_contains_only_staged_content:
_unlink_at(directory_fd, temporary_name)
raise
def _same_identity_after_rename(
actual: _FileIdentity | None,
expected: _FileIdentity | None,
) -> bool:
if actual is None or expected is None:
return False
return (
actual.device,
actual.inode,
actual.mode,
actual.owner,
actual.group,
actual.size,
actual.modified_ns,
) == (
expected.device,
expected.inode,
expected.mode,
expected.owner,
expected.group,
expected.size,
expected.modified_ns,
)
def _link_no_replace(directory_fd: int, source: str, target: str) -> None:
try:
os.link(
source,
target,
src_dir_fd=directory_fd,
dst_dir_fd=directory_fd,
follow_symlinks=False,
)
except FileExistsError as error:
raise CommandReferenceToolError(
"Output target appeared during atomic publication"
) from error
except OSError as error:
raise CommandReferenceToolError("Could not publish atomic output") from error
def _rename_exchange(directory_fd: int, first: str, second: str) -> None:
rename_at2 = _load_rename_at2()
ctypes.set_errno(0)
result = rename_at2(
directory_fd,
os.fsencode(first),
directory_fd,
os.fsencode(second),
RENAME_EXCHANGE,
)
if result == 0:
return
error_number = ctypes.get_errno()
if error_number in {errno.ENOSYS, errno.EINVAL, errno.EOPNOTSUPP}:
raise CommandReferenceToolError(
"Atomic exchange publication is unavailable on this filesystem"
)
raise CommandReferenceToolError("Could not exchange atomic output") from OSError(
error_number,
os.strerror(error_number),
)
def _load_rename_at2() -> _RenameAt2:
library = ctypes.CDLL(None, use_errno=True)
try:
rename_at2 = cast(_RenameAt2, library.renameat2)
except AttributeError as error:
raise CommandReferenceToolError(
"Atomic exchange publication is unavailable on this platform"
) from error
rename_at2.argtypes = [
ctypes.c_int,
ctypes.c_char_p,
ctypes.c_int,
ctypes.c_char_p,
ctypes.c_uint,
]
rename_at2.restype = ctypes.c_int
return rename_at2
def _unlink_at(directory_fd: int, name: str) -> None:
with contextlib.suppress(FileNotFoundError):
os.unlink(name, dir_fd=directory_fd)
def _parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
description="Generate or check deterministic DocForge command-reference Markdown."
)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--check", action="store_true")
parser.add_argument("--repository-root", type=Path, default=REPOSITORY_ROOT)
parser.add_argument("--project-root", type=Path, default=DEFAULT_PROJECT_ROOT)
parser.add_argument("--proposal-writer", default="alpha-editor")
parser.add_argument("--canonical-applier", default="alpha-editor")
return parser
def main(arguments: Sequence[str] | None = None) -> int:
parsed = _parser().parse_args(arguments)
try:
content = generate_command_reference_bytes(
parsed.project_root,
proposal_writer=parsed.proposal_writer,
canonical_applier=parsed.canonical_applier,
)
state = publish_or_check_command_reference(
content,
repository_root=parsed.repository_root,
project_root=parsed.project_root,
output=parsed.output,
check=parsed.check,
)
except CommandReferenceDrift as error:
print(str(error), file=sys.stderr)
return 1
except Exception as error:
print(f"command-reference generation failed: {error}", file=sys.stderr)
return 2
print(
json.dumps(
{
"status": state,
"output": str(parsed.output),
"cli_rows": EXPECTED_CLI_ROWS,
"mcp_rows": EXPECTED_MCP_ROWS,
},
sort_keys=True,
separators=(",", ":"),
)
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -0,0 +1,430 @@
#!/usr/bin/env python3
"""Run the offline Milestone 4 fresh-wheel adoption proof."""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
from typing import cast
PROOF_SCHEMA_VERSION = 1
COMMAND_TIMEOUT_SECONDS = 120
VALIDATION_TIMEOUT_SECONDS = 45
OPTIONAL_FRONTEND_MODULES = (
"tree_sitter",
"tree_sitter_cpp",
"tree_sitter_javascript",
"tree_sitter_typescript",
)
_VALIDATION_PROGRAM = r"""
from __future__ import annotations
import asyncio
import importlib.metadata
import importlib.util
import json
import sys
from pathlib import Path
from docforge.errors import DocForgeError
from docforge.index import ProjectIndex
from docforge.reference_mcp import create_reference_project
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
def write_config(
root: Path,
*,
language: str,
source_roots: tuple[str, ...],
compilation_database: str | None = None,
) -> None:
config = root / ".docforge" / "reference-adapter.toml"
config.parent.mkdir(parents=True)
roots = ", ".join(json.dumps(item) for item in source_roots)
lines = [
"schema_version = 1",
'project_id = "fresh-wheel-adoption"',
'title = "Fresh wheel adoption proof"',
f"language = {json.dumps(language)}",
f"source_roots = [{roots}]",
]
if compilation_database is not None:
lines.append(f"compilation_database = {json.dumps(compilation_database)}")
config.write_text("\n".join((*lines, "")), encoding="utf-8")
async def retrieve(python: Path, root: Path) -> dict[str, object]:
parameters = StdioServerParameters(
command=str(python),
args=[
"-I",
"-m",
"docforge.reference_mcp",
"--project-root",
str(root),
"--capability-mode",
"read",
],
env={},
)
async with (
stdio_client(parameters) as streams,
ClientSession(*streams) as session,
):
await session.initialize()
tools = tuple(tool.name for tool in (await session.list_tools()).tools)
bootstrap = await session.call_tool("docforge_bootstrap", {})
search = await session.call_tool(
"docforge_search",
{"query": "Service", "limit": 5},
)
if search.isError or not search.structuredContent:
raise RuntimeError("fresh-wheel reference MCP search failed")
results = search.structuredContent.get("results")
if not isinstance(results, list) or not results:
raise RuntimeError("fresh-wheel reference MCP search returned no evidence")
first = results[0]
if not isinstance(first, dict) or not isinstance(first.get("node_id"), str):
raise RuntimeError("fresh-wheel reference MCP search returned invalid evidence")
node_id = first["node_id"]
node = await session.call_tool("docforge_get_node", {"node_id": node_id})
if bootstrap.isError or not bootstrap.structuredContent:
raise RuntimeError("fresh-wheel reference MCP bootstrap failed")
if node.isError or not node.structuredContent:
raise RuntimeError("fresh-wheel reference MCP node retrieval failed")
retrieved = node.structuredContent.get("node")
if not isinstance(retrieved, dict) or retrieved.get("node_id") != node_id:
raise RuntimeError("fresh-wheel reference MCP returned the wrong node")
return {
"bootstrap_status": bootstrap.structuredContent.get("status"),
"node_id": node_id,
"read_tool_count": len(tools),
}
workspace = Path(sys.argv[1]).resolve(strict=True)
python = Path(sys.executable).absolute()
present_modules = [
name
for name in (
"tree_sitter",
"tree_sitter_cpp",
"tree_sitter_javascript",
"tree_sitter_typescript",
)
if importlib.util.find_spec(name) is not None
]
installed_frontends = sorted(
distribution.metadata["Name"]
for distribution in importlib.metadata.distributions()
if distribution.metadata["Name"].casefold().startswith("tree-sitter")
)
if present_modules or installed_frontends:
raise RuntimeError(
f"base wheel unexpectedly installed language frontends: "
f"{present_modules or installed_frontends}"
)
python_root = workspace / "python-project"
(python_root / "src" / "sample").mkdir(parents=True)
(python_root / "src" / "sample" / "__init__.py").write_text("", encoding="utf-8")
(python_root / "src" / "sample" / "service.py").write_text(
"class Service:\n"
" def execute(self, ready: bool) -> str:\n"
" if ready:\n"
' return "ready"\n'
' return "waiting"\n',
encoding="utf-8",
)
write_config(python_root, language="python", source_roots=("src",))
project = create_reference_project(python_root)
index = ProjectIndex(project)
built = index.build()
checked = index.check()
if built.get("status") != "ok" or checked.get("status") != "ok":
raise RuntimeError("fresh-wheel Python reference build/check failed")
mcp = asyncio.run(retrieve(python, python_root))
if mcp["bootstrap_status"] != "ok":
raise RuntimeError("fresh-wheel reference MCP bootstrap did not report ok")
cpp_root = workspace / "cpp-project"
(cpp_root / "src").mkdir(parents=True)
(cpp_root / "src" / "main.cpp").write_text("int main() { return 0; }\n", encoding="utf-8")
(cpp_root / "compile_commands.json").write_text(
json.dumps(
[
{
"directory": str(cpp_root),
"file": "src/main.cpp",
"arguments": ["c++", "-c", "src/main.cpp"],
}
],
sort_keys=True,
)
+ "\n",
encoding="utf-8",
)
write_config(
cpp_root,
language="cpp",
source_roots=("src",),
compilation_database="compile_commands.json",
)
try:
ProjectIndex(create_reference_project(cpp_root)).build()
except DocForgeError as error:
if error.code != "optional_dependency_missing":
raise
if error.details.get("install") != "docforge[cpp]":
raise RuntimeError("C++ missing-extra failure was not actionable") from error
cpp_failure = {
"code": error.code,
"install": error.details["install"],
"missing_module": error.details.get("missing_module"),
}
else:
raise RuntimeError("C++ reference build unexpectedly worked without docforge[cpp]")
print(
json.dumps(
{
"schema_version": 1,
"base_frontend_modules": present_modules,
"base_frontend_distributions": installed_frontends,
"python": {
"build_status": built["status"],
"check_status": checked["status"],
"node_count": checked["node_count"],
},
"mcp": mcp,
"cpp_without_extra": cpp_failure,
},
sort_keys=True,
)
)
"""
class AdoptionProofError(RuntimeError):
"""The disposable adoption proof could not be completed."""
def _run_checked(
arguments: list[str],
*,
cwd: Path,
timeout: int,
environment: dict[str, str] | None = None,
) -> subprocess.CompletedProcess[str]:
result = subprocess.run(
arguments,
cwd=cwd,
env=environment,
text=True,
capture_output=True,
check=False,
timeout=timeout,
)
if result.returncode != 0:
detail = result.stderr.strip() or result.stdout.strip() or "no command output"
raise AdoptionProofError(
f"command failed with exit {result.returncode}: {arguments[0]}\n{detail}"
)
return result
def find_wheel(directory: Path) -> Path:
wheels = tuple(sorted(directory.glob("docforge-*.whl")))
if len(wheels) != 1:
raise AdoptionProofError(f"expected exactly one DocForge wheel, found {len(wheels)}")
return wheels[0]
def parse_evidence(raw: str) -> dict[str, object]:
try:
loaded: object = json.loads(raw)
except json.JSONDecodeError as error:
raise AdoptionProofError("fresh-wheel validation returned invalid JSON") from error
if not isinstance(loaded, dict):
raise AdoptionProofError("fresh-wheel validation returned the wrong schema")
document = cast(dict[str, object], loaded)
if document.get("schema_version") != PROOF_SCHEMA_VERSION:
raise AdoptionProofError("fresh-wheel validation returned the wrong schema")
if document.get("base_frontend_modules") != []:
raise AdoptionProofError("fresh-wheel validation found optional frontend modules")
if document.get("base_frontend_distributions") != []:
raise AdoptionProofError("fresh-wheel validation found optional frontend distributions")
raw_python = document.get("python")
raw_mcp = document.get("mcp")
raw_cpp = document.get("cpp_without_extra")
python = cast(dict[str, object], raw_python) if isinstance(raw_python, dict) else None
mcp = cast(dict[str, object], raw_mcp) if isinstance(raw_mcp, dict) else None
cpp = cast(dict[str, object], raw_cpp) if isinstance(raw_cpp, dict) else None
if (
python is None
or python.get("build_status") != "ok"
or python.get("check_status") != "ok"
or mcp is None
or mcp.get("bootstrap_status") != "ok"
or cpp is None
or cpp.get("code") != "optional_dependency_missing"
or cpp.get("install") != "docforge[cpp]"
):
raise AdoptionProofError("fresh-wheel validation evidence is incomplete")
return document
def run_adoption_proof(repository_root: Path, *, uv: str = "uv") -> dict[str, object]:
"""Build and exercise a base wheel without using the network."""
root = repository_root.resolve(strict=True)
if not root.joinpath("pyproject.toml").is_file() or not root.joinpath("uv.lock").is_file():
raise AdoptionProofError("repository root lacks pyproject.toml or uv.lock")
offline_environment = {
**os.environ,
"UV_OFFLINE": "1",
"UV_PYTHON_DOWNLOADS": "never",
}
offline_environment.pop("PYTHONPATH", None)
with tempfile.TemporaryDirectory(prefix="docforge-m4-adoption-") as directory:
workspace = Path(directory).resolve()
distributions = workspace / "dist"
requirements = workspace / "base-requirements.txt"
virtual_environment = workspace / "venv"
python = virtual_environment / "bin" / "python"
_run_checked(
[
uv,
"build",
"--wheel",
"--offline",
"--no-python-downloads",
"--out-dir",
str(distributions),
str(root),
],
cwd=root,
timeout=COMMAND_TIMEOUT_SECONDS,
environment=offline_environment,
)
wheel = find_wheel(distributions)
_run_checked(
[
uv,
"export",
"--frozen",
"--offline",
"--no-dev",
"--no-emit-project",
"--output-file",
str(requirements),
],
cwd=root,
timeout=COMMAND_TIMEOUT_SECONDS,
environment=offline_environment,
)
exported = requirements.read_text(encoding="utf-8").casefold()
if any(module.replace("_", "-") in exported for module in OPTIONAL_FRONTEND_MODULES):
raise AdoptionProofError("locked base dependency export contains a language frontend")
_run_checked(
[
uv,
"venv",
"--python",
sys.executable,
"--no-project",
"--no-python-downloads",
str(virtual_environment),
],
cwd=workspace,
timeout=COMMAND_TIMEOUT_SECONDS,
environment=offline_environment,
)
_run_checked(
[
uv,
"pip",
"sync",
"--python",
str(python),
"--offline",
"--strict",
str(requirements),
],
cwd=workspace,
timeout=COMMAND_TIMEOUT_SECONDS,
environment=offline_environment,
)
_run_checked(
[
uv,
"pip",
"install",
"--python",
str(python),
"--offline",
"--strict",
"--no-deps",
str(wheel),
],
cwd=workspace,
timeout=COMMAND_TIMEOUT_SECONDS,
environment=offline_environment,
)
validation = _run_checked(
[str(python), "-I", "-c", _VALIDATION_PROGRAM, str(workspace)],
cwd=workspace,
timeout=VALIDATION_TIMEOUT_SECONDS,
environment=offline_environment,
)
evidence = parse_evidence(validation.stdout.strip())
evidence["wheel"] = {
"filename": wheel.name,
"sha256": hashlib.sha256(wheel.read_bytes()).hexdigest(),
}
evidence["network"] = "offline"
return evidence
def main() -> None:
parser = argparse.ArgumentParser(
description="Run the offline Milestone 4 fresh-wheel adoption proof"
)
parser.add_argument(
"--repository-root",
type=Path,
default=Path(__file__).resolve().parents[1],
)
parser.add_argument("--uv", default=shutil.which("uv") or "uv")
parser.add_argument("--output", type=Path)
arguments = parser.parse_args()
try:
evidence = run_adoption_proof(arguments.repository_root, uv=arguments.uv)
except (AdoptionProofError, OSError, subprocess.SubprocessError) as error:
print(json.dumps({"status": "error", "error": str(error)}, sort_keys=True))
raise SystemExit(1) from error
payload = json.dumps({"status": "ok", **evidence}, indent=2, sort_keys=True) + "\n"
if arguments.output is not None:
arguments.output.write_text(payload, encoding="utf-8")
print(payload, end="")
if __name__ == "__main__":
main()

View file

@ -0,0 +1,498 @@
"""Milestone 4 Python adapter SDK, incremental, and recovery benchmark gates."""
from __future__ import annotations
import argparse
import ast
import gc
import hashlib
import json
import math
import platform
import resource
import statistics
import subprocess
import sys
import tempfile
import time
import tracemalloc
from collections.abc import Callable, Generator, Mapping
from contextlib import contextmanager
from dataclasses import dataclass
from pathlib import Path
from typing import cast
from unittest import mock
from docforge.adapter_sdk import (
AdapterAssembly,
AdapterProject,
AdapterSource,
AdapterSourceProjection,
)
from docforge.adapters.python import PythonReferenceAdapter
from docforge.index import ProjectIndex
ROOT = Path(__file__).resolve().parents[1]
FULL_SOURCE_COUNT = 334
SMOKE_SOURCE_COUNT = 12
NODES_PER_SOURCE = 3
DEFAULT_FULL_SAMPLES = 3
DEFAULT_SMOKE_SAMPLES = 1
# These are regression tripwires, not performance targets. They deliberately
# leave several times the Milestone 3 1,000-node in-process allowances for the
# additional AST, Logic, extraction-cache, and SQLite work in this gate.
MAX_TRACED_PEAK_BYTES = 256 * 1024 * 1024
MAX_PROCESS_PEAK_BYTES = 512 * 1024 * 1024
MAX_RESPONSE_BYTES = 256 * 1024
COLD_LIMIT_MS = 30_000.0
WARM_LIMIT_MS = 20_000.0
EQUIVALENCE_LIMIT_MS = 30_000.0
RECOVERY_LIMIT_MS = 30_000.0
@dataclass(frozen=True)
class _InstrumentedResult:
value: Mapping[str, object]
ast_parse_calls: int
extraction_calls: int
class _RecordingPythonAdapter(PythonReferenceAdapter):
"""Count extraction entry points without weakening the production adapter."""
def __init__(self, root: Path) -> None:
super().__init__(
root,
source_roots=("src",),
project_id="milestone4-python-benchmark",
title="Milestone 4 Python adapter benchmark",
)
self.extraction_calls = 0
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection:
self.extraction_calls += 1
return super().extract_source(source)
def _parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
description=("Gate DocForge2 Milestone 4 adapter behavior on a disposable Python project.")
)
parser.add_argument("--mode", choices=("smoke", "full"), default="full")
parser.add_argument("--sources", type=int)
parser.add_argument("--samples", type=int)
parser.add_argument("--output", type=Path)
return parser
def encode_report(value: object) -> str:
"""Serialize benchmark evidence in one deterministic JSON representation."""
return json.dumps(value, sort_keys=True, indent=2, ensure_ascii=False) + "\n"
def _compact_json(value: object) -> bytes:
return json.dumps(
value,
sort_keys=True,
separators=(",", ":"),
ensure_ascii=False,
).encode("utf-8")
def _sha256(value: object) -> str:
return hashlib.sha256(_compact_json(value)).hexdigest()
def _git(arguments: list[str]) -> str:
return subprocess.run(
["git", *arguments],
cwd=ROOT,
check=True,
capture_output=True,
text=True,
).stdout.strip()
def write_python_fixture(root: Path, source_count: int) -> None:
"""Write a deterministic project with three primary nodes per source."""
if source_count < 2:
raise ValueError("The Milestone 4 fixture requires at least two sources")
package = root / "src" / "benchmark"
package.mkdir(parents=True)
for index in range(source_count):
dependency = ""
call = f"value = {index}"
if index:
dependency = f"from benchmark.module_{index - 1:04d} import compute_{index - 1:04d}\n\n"
call = f"value = compute_{index - 1:04d}(value)"
source = (
f'"""Synthetic adapter benchmark module {index:04d}."""\n\n'
f"{dependency}"
f"def compute_{index:04d}(value: int) -> int:\n"
f' """Return one deterministic branch result."""\n'
f" {call}\n"
" if value % 2:\n"
f" return value + {index + 1}\n"
" return value\n"
)
(package / f"module_{index:04d}.py").write_text(source, encoding="utf-8")
@contextmanager
def _count_ast_parses() -> Generator[mock.MagicMock]:
with mock.patch.object(ast, "parse", wraps=ast.parse) as parsed:
yield parsed
def _instrument(
adapter: _RecordingPythonAdapter,
operation: Callable[[], Mapping[str, object]],
) -> _InstrumentedResult:
adapter.extraction_calls = 0
with _count_ast_parses() as counter:
value = operation()
return _InstrumentedResult(
value=value,
ast_parse_calls=counter.call_count,
extraction_calls=adapter.extraction_calls,
)
def _build_summary(sample: _InstrumentedResult) -> dict[str, object]:
value = sample.value
build = cast(Mapping[str, object], value["build"])
return {
"status": value["status"],
"revision": value["revision"],
"source_hash": value["source_hash"],
"node_hash": value["node_hash"],
"node_count": value["node_count"],
"edge_hash": value["edge_hash"],
"edge_count": value["edge_count"],
"logic_hash": value["logic_hash"],
"logic_projection_count": value["logic_projection_count"],
"logic_node_count": value["logic_node_count"],
"logic_edge_count": value["logic_edge_count"],
"cache_hits": build["cache_hits"],
"reparsed_sources": build["reparsed_sources"],
"invalidated_sources": build["invalidated_sources"],
"deleted_sources": build["deleted_sources"],
"total_sources": build["total_sources"],
"ast_parse_calls": sample.ast_parse_calls,
"extraction_calls": sample.extraction_calls,
}
def _equivalence_summary(sample: _InstrumentedResult) -> dict[str, object]:
return {
**sample.value,
"ast_parse_calls": sample.ast_parse_calls,
"extraction_calls": sample.extraction_calls,
}
def _synchronization_summary(sample: _InstrumentedResult) -> dict[str, object]:
synchronization = cast(Mapping[str, object], sample.value["synchronization"])
build = cast(Mapping[str, object], synchronization["build"])
return {
"status": sample.value["status"],
"revision": sample.value["revision"],
"source_hash": sample.value["source_hash"],
"node_hash": sample.value["node_hash"],
"node_count": sample.value["node_count"],
"edge_hash": sample.value["edge_hash"],
"edge_count": sample.value["edge_count"],
"logic_hash": sample.value["logic_hash"],
"logic_projection_count": sample.value["logic_projection_count"],
"action": synchronization["action"],
"initial_error_code": cast(Mapping[str, object], synchronization["initial_error"])["code"],
"cache_hits": build["cache_hits"],
"reparsed_sources": build["reparsed_sources"],
"ast_parse_calls": sample.ast_parse_calls,
"extraction_calls": sample.extraction_calls,
}
def _measure(
operation: Callable[[], _InstrumentedResult],
*,
samples: int,
p95_limit_ms: float,
summary: Callable[[_InstrumentedResult], Mapping[str, object]],
) -> tuple[dict[str, object], _InstrumentedResult]:
durations: list[float] = []
traced_peaks: list[int] = []
response_sizes: list[int] = []
stable_summary: dict[str, object] | None = None
last: _InstrumentedResult | None = None
for _ in range(samples):
gc.collect()
tracemalloc.start()
started = time.perf_counter_ns()
try:
current = operation()
elapsed_ms = (time.perf_counter_ns() - started) / 1_000_000
_, traced_peak = tracemalloc.get_traced_memory()
finally:
tracemalloc.stop()
current_summary = dict(summary(current))
if stable_summary is None:
stable_summary = current_summary
elif current_summary != stable_summary:
raise RuntimeError("Milestone 4 operation changed its deterministic result")
response_bytes = len(_compact_json(current.value))
if response_bytes > MAX_RESPONSE_BYTES:
raise RuntimeError(
"Milestone 4 response exceeded its fixed byte boundary: "
f"{response_bytes} > {MAX_RESPONSE_BYTES}"
)
if traced_peak > MAX_TRACED_PEAK_BYTES:
raise RuntimeError(
"Milestone 4 operation exceeded its traced-memory boundary: "
f"{traced_peak} > {MAX_TRACED_PEAK_BYTES}"
)
durations.append(elapsed_ms)
traced_peaks.append(traced_peak)
response_sizes.append(response_bytes)
last = current
ordered = sorted(durations)
p95_index = max(0, math.ceil(len(ordered) * 0.95) - 1)
p95 = ordered[p95_index]
if p95 > p95_limit_ms:
raise RuntimeError(f"Milestone 4 operation p95 {p95:.3f} ms exceeds {p95_limit_ms:.3f} ms")
assert stable_summary is not None
assert last is not None
return (
{
"samples": samples,
"median_ms": round(statistics.median(ordered), 3),
"p95_ms": round(p95, 3),
"min_ms": round(ordered[0], 3),
"max_ms": round(ordered[-1], 3),
"p95_limit_ms": p95_limit_ms,
"maximum_response_bytes": max(response_sizes),
"response_limit_bytes": MAX_RESPONSE_BYTES,
"maximum_traced_peak_bytes": max(traced_peaks),
"traced_peak_limit_bytes": MAX_TRACED_PEAK_BYTES,
"stable_result": stable_summary,
},
last,
)
def _semantic_assembly_hash(assembly: AdapterAssembly) -> str:
projection = assembly.projection
return _sha256(
{
"project_id": projection.project_id,
"title": projection.title,
"adapter_id": projection.adapter_id,
"adapter_version": projection.adapter_version,
"revision": projection.revision,
"source_hash": projection.source_hash,
"nodes": [item.as_dict() for item in projection.nodes],
"edges": [item.as_dict() for item in projection.edges],
"logic": [item.as_dict() for item in assembly.logic],
}
)
def _benchmark(root: Path, source_count: int, samples: int) -> dict[str, object]:
write_python_fixture(root, source_count)
adapter = _RecordingPythonAdapter(root)
cache_root = root / ".docforge" / "benchmark"
project = AdapterProject(adapter, cache_root=cache_root)
index = ProjectIndex(project)
cold, cold_sample = _measure(
lambda: _instrument(adapter, index.build),
samples=1,
p95_limit_ms=COLD_LIMIT_MS,
summary=_build_summary,
)
cold_summary = _build_summary(cold_sample)
expected_nodes = source_count * NODES_PER_SOURCE
if (
cold_summary["node_count"] != expected_nodes
or cold_summary["logic_projection_count"] != source_count
or cold_summary["reparsed_sources"] != source_count
or cast(int, cold_summary["ast_parse_calls"]) < source_count
or cold_summary["extraction_calls"] != source_count
):
raise RuntimeError("Cold adapter build did not cover the complete synthetic fixture")
warm, warm_sample = _measure(
lambda: _instrument(adapter, index.build),
samples=samples,
p95_limit_ms=WARM_LIMIT_MS,
summary=_build_summary,
)
warm_summary = _build_summary(warm_sample)
if (
warm_summary["cache_hits"] != source_count
or warm_summary["reparsed_sources"] != 0
or warm_summary["ast_parse_calls"] != 0
or warm_summary["extraction_calls"] != 0
):
raise RuntimeError("Warm adapter build performed forbidden parsing or extraction")
equivalence, equivalence_sample = _measure(
lambda: _instrument(adapter, project.verify_incremental_equivalence),
samples=1,
p95_limit_ms=EQUIVALENCE_LIMIT_MS,
summary=_equivalence_summary,
)
equivalence_summary = _equivalence_summary(equivalence_sample)
if (
equivalence_summary["status"] != "ok"
or equivalence_summary["node_count"] != expected_nodes
or equivalence_summary["logic_projection_count"] != source_count
):
raise RuntimeError("Full and incremental graph plus Logic output was not equivalent")
complete_assembly = adapter.load_complete_assembly()
assembly_hash = _semantic_assembly_hash(complete_assembly)
extraction_cache = cache_root / "extractions.json"
extraction_cache.write_bytes(b"{broken")
cache_recovery, cache_recovery_sample = _measure(
lambda: _instrument(adapter, index.build),
samples=1,
p95_limit_ms=RECOVERY_LIMIT_MS,
summary=_build_summary,
)
cache_recovery_summary = _build_summary(cache_recovery_sample)
if (
cache_recovery_summary["reparsed_sources"] != source_count
or cast(int, cache_recovery_summary["ast_parse_calls"]) < source_count
or cache_recovery_summary["node_hash"] != cold_summary["node_hash"]
or cache_recovery_summary["logic_hash"] != cold_summary["logic_hash"]
):
raise RuntimeError("Corrupt extraction-cache recovery changed adapter output")
index.path.write_bytes(b"not-a-sqlite-index")
index_recovery, index_recovery_sample = _measure(
lambda: _instrument(adapter, index.synchronize),
samples=1,
p95_limit_ms=RECOVERY_LIMIT_MS,
summary=_synchronization_summary,
)
index_recovery_summary = _synchronization_summary(index_recovery_sample)
if (
index_recovery_summary["action"] != "rebuilt"
or index_recovery_summary["cache_hits"] != source_count
or index_recovery_summary["reparsed_sources"] != 0
or index_recovery_summary["ast_parse_calls"] != 0
or index_recovery_summary["extraction_calls"] != 0
or index_recovery_summary["node_hash"] != cold_summary["node_hash"]
or index_recovery_summary["logic_hash"] != cold_summary["logic_hash"]
):
raise RuntimeError("Corrupt derived-index recovery changed adapter output")
evidence = {
"source_count": source_count,
"node_count": expected_nodes,
"edge_count": cold_summary["edge_count"],
"logic_projection_count": source_count,
"logic_node_count": cold_summary["logic_node_count"],
"logic_edge_count": cold_summary["logic_edge_count"],
"source_hash": cold_summary["source_hash"],
"node_hash": cold_summary["node_hash"],
"edge_hash": cold_summary["edge_hash"],
"logic_hash": cold_summary["logic_hash"],
"complete_assembly_hash": assembly_hash,
"full_incremental_graph_and_logic_exact": True,
"warm_zero_ast_parse": True,
"warm_zero_source_extraction": True,
"cache_recovery_exact": True,
"index_recovery_exact": True,
}
return {
"fixture": {
"kind": "synthetic_python_reference_adapter",
"source_count": source_count,
"nodes_per_source": NODES_PER_SOURCE,
"expected_node_count": expected_nodes,
},
"operations": {
"cold_incremental_build": cold,
"warm_incremental_build": warm,
"full_incremental_equivalence": equivalence,
"corrupt_extraction_cache_recovery": cache_recovery,
"corrupt_index_recovery": index_recovery,
},
"evidence": evidence,
"evidence_sha256": _sha256(evidence),
}
def main() -> int:
arguments = _parser().parse_args()
default_sources = FULL_SOURCE_COUNT if arguments.mode == "full" else SMOKE_SOURCE_COUNT
default_samples = DEFAULT_FULL_SAMPLES if arguments.mode == "full" else DEFAULT_SMOKE_SAMPLES
source_count = default_sources if arguments.sources is None else arguments.sources
samples = default_samples if arguments.samples is None else arguments.samples
if not 2 <= source_count <= FULL_SOURCE_COUNT:
raise SystemExit(f"--sources must be between 2 and {FULL_SOURCE_COUNT}")
if arguments.mode == "full" and source_count != FULL_SOURCE_COUNT:
raise SystemExit(f"--mode full requires exactly {FULL_SOURCE_COUNT} sources")
if samples < 1:
raise SystemExit("--samples must be positive")
with tempfile.TemporaryDirectory(prefix="docforge-milestone4-") as directory:
measurement = _benchmark(Path(directory).resolve(), source_count, samples)
process_peak_bytes = int(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss) * 1024
if process_peak_bytes > MAX_PROCESS_PEAK_BYTES:
raise RuntimeError(
"Milestone 4 process peak exceeded its memory boundary: "
f"{process_peak_bytes} > {MAX_PROCESS_PEAK_BYTES}"
)
status = _git(["status", "--porcelain"])
report: dict[str, object] = {
"schema_version": 1,
"benchmark": "docforge2_milestone4",
"mode": arguments.mode,
"source": {
"revision": _git(["rev-parse", "HEAD"]),
"dirty": bool(status),
},
"environment": {
"platform": platform.platform(),
"machine": platform.machine(),
"python": platform.python_version(),
"implementation": platform.python_implementation(),
},
"method": {
"clock": "time.perf_counter_ns",
"in_process_peak_memory": "tracemalloc per measured invocation",
"process_peak_memory": "resource.getrusage(RUSAGE_SELF).ru_maxrss",
"response_size": "UTF-8 bytes of compact sorted JSON",
"ast_instrumentation": "temporary counter around stdlib ast.parse",
"extraction_instrumentation": "reference adapter extract_source entry counter",
"samples": samples,
"full_mode_source_requirement": FULL_SOURCE_COUNT,
"threshold_basis": (
"Regression tripwires leave several times the Milestone 3 1,000-node "
"allowances for AST, Logic, extraction-cache, and SQLite work."
),
},
**measurement,
"memory": {
"process_peak_bytes": process_peak_bytes,
"process_peak_limit_bytes": MAX_PROCESS_PEAK_BYTES,
},
}
encoded = encode_report(report)
if arguments.output is not None:
output = arguments.output.resolve()
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(encoded, encoding="utf-8")
sys.stdout.write(encoded)
return 0
if __name__ == "__main__":
raise SystemExit(main())

54
uv.lock generated
View file

@ -211,9 +211,26 @@ source = { editable = "." }
dependencies = [
{ name = "markdown-it-py" },
{ name = "mcp" },
]
[package.optional-dependencies]
cpp = [
{ name = "tree-sitter" },
{ name = "tree-sitter-cpp" },
]
javascript = [
{ name = "tree-sitter" },
{ name = "tree-sitter-javascript" },
]
languages = [
{ name = "tree-sitter" },
{ name = "tree-sitter-cpp" },
{ name = "tree-sitter-javascript" },
{ name = "tree-sitter-typescript" },
]
typescript = [
{ name = "tree-sitter" },
{ name = "tree-sitter-typescript" },
]
[package.dev-dependencies]
@ -221,22 +238,38 @@ dev = [
{ name = "jsonschema" },
{ name = "pytest" },
{ name = "ruff" },
{ name = "tree-sitter" },
{ name = "tree-sitter-cpp" },
{ name = "tree-sitter-javascript" },
{ name = "tree-sitter-typescript" },
]
[package.metadata]
requires-dist = [
{ name = "markdown-it-py", specifier = ">=4.2,<5" },
{ name = "mcp", specifier = ">=1.28,<2" },
{ name = "tree-sitter", specifier = ">=0.25,<0.26" },
{ name = "tree-sitter-cpp", specifier = ">=0.23,<0.24" },
{ name = "tree-sitter-javascript", specifier = ">=0.25,<0.26" },
{ name = "tree-sitter", marker = "extra == 'cpp'", specifier = ">=0.25,<0.26" },
{ name = "tree-sitter", marker = "extra == 'javascript'", specifier = ">=0.25,<0.26" },
{ name = "tree-sitter", marker = "extra == 'languages'", specifier = ">=0.25,<0.26" },
{ name = "tree-sitter", marker = "extra == 'typescript'", specifier = ">=0.25,<0.26" },
{ name = "tree-sitter-cpp", marker = "extra == 'cpp'", specifier = ">=0.23,<0.24" },
{ name = "tree-sitter-cpp", marker = "extra == 'languages'", specifier = ">=0.23,<0.24" },
{ name = "tree-sitter-javascript", marker = "extra == 'javascript'", specifier = ">=0.25,<0.26" },
{ name = "tree-sitter-javascript", marker = "extra == 'languages'", specifier = ">=0.25,<0.26" },
{ name = "tree-sitter-typescript", marker = "extra == 'languages'", specifier = ">=0.23,<0.24" },
{ name = "tree-sitter-typescript", marker = "extra == 'typescript'", specifier = ">=0.23,<0.24" },
]
provides-extras = ["javascript", "typescript", "cpp", "languages"]
[package.metadata.requires-dev]
dev = [
{ name = "jsonschema", specifier = ">=4.25,<5" },
{ name = "pytest", specifier = ">=9.1,<10" },
{ name = "ruff", specifier = ">=0.15,<1" },
{ name = "tree-sitter", specifier = ">=0.25,<0.26" },
{ name = "tree-sitter-cpp", specifier = ">=0.23,<0.24" },
{ name = "tree-sitter-javascript", specifier = ">=0.25,<0.26" },
{ name = "tree-sitter-typescript", specifier = ">=0.23,<0.24" },
]
[[package]]
@ -804,6 +837,21 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/2e/1f/f9eba1038b7d4394410f3c0a6ec2122b590cd7acb03f196e52fa57ebbe72/tree_sitter_javascript-0.25.0-cp310-abi3-win_arm64.whl", hash = "sha256:622a69d677aa7f6ee2931d8c77c981a33f0ebb6d275aa9d43d3397c879a9bb0b", size = 61668, upload-time = "2025-09-01T07:13:43.803Z" },
]
[[package]]
name = "tree-sitter-typescript"
version = "0.23.2"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/1e/fc/bb52958f7e399250aee093751e9373a6311cadbe76b6e0d109b853757f35/tree_sitter_typescript-0.23.2.tar.gz", hash = "sha256:7b167b5827c882261cb7a50dfa0fb567975f9b315e87ed87ad0a0a3aedb3834d", size = 773053, upload-time = "2024-11-11T02:36:11.396Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/28/95/4c00680866280e008e81dd621fd4d3f54aa3dad1b76b857a19da1b2cc426/tree_sitter_typescript-0.23.2-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:3cd752d70d8e5371fdac6a9a4df9d8924b63b6998d268586f7d374c9fba2a478", size = 286677, upload-time = "2024-11-11T02:35:58.839Z" },
{ url = "https://files.pythonhosted.org/packages/8f/2f/1f36fda564518d84593f2740d5905ac127d590baf5c5753cef2a88a89c15/tree_sitter_typescript-0.23.2-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:c7cc1b0ff5d91bac863b0e38b1578d5505e718156c9db577c8baea2557f66de8", size = 302008, upload-time = "2024-11-11T02:36:00.733Z" },
{ url = "https://files.pythonhosted.org/packages/96/2d/975c2dad292aa9994f982eb0b69cc6fda0223e4b6c4ea714550477d8ec3a/tree_sitter_typescript-0.23.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4b1eed5b0b3a8134e86126b00b743d667ec27c63fc9de1b7bb23168803879e31", size = 351987, upload-time = "2024-11-11T02:36:02.669Z" },
{ url = "https://files.pythonhosted.org/packages/49/d1/a71c36da6e2b8a4ed5e2970819b86ef13ba77ac40d9e333cb17df6a2c5db/tree_sitter_typescript-0.23.2-cp39-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e96d36b85bcacdeb8ff5c2618d75593ef12ebaf1b4eace3477e2bdb2abb1752c", size = 344960, upload-time = "2024-11-11T02:36:04.443Z" },
{ url = "https://files.pythonhosted.org/packages/7f/cb/f57b149d7beed1a85b8266d0c60ebe4c46e79c9ba56bc17b898e17daf88e/tree_sitter_typescript-0.23.2-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:8d4f0f9bcb61ad7b7509d49a1565ff2cc363863644a234e1e0fe10960e55aea0", size = 340245, upload-time = "2024-11-11T02:36:06.473Z" },
{ url = "https://files.pythonhosted.org/packages/8b/ab/dd84f0e2337296a5f09749f7b5483215d75c8fa9e33738522e5ed81f7254/tree_sitter_typescript-0.23.2-cp39-abi3-win_amd64.whl", hash = "sha256:3f730b66396bc3e11811e4465c41ee45d9e9edd6de355a58bbbc49fa770da8f9", size = 278015, upload-time = "2024-11-11T02:36:07.631Z" },
{ url = "https://files.pythonhosted.org/packages/9f/e4/81f9a935789233cf412a0ed5fe04c883841d2c8fb0b7e075958a35c65032/tree_sitter_typescript-0.23.2-cp39-abi3-win_arm64.whl", hash = "sha256:05db58f70b95ef0ea126db5560f3775692f609589ed6f8dd0af84b7f19f1cbb7", size = 274052, upload-time = "2024-11-11T02:36:09.514Z" },
]
[[package]]
name = "typing-extensions"
version = "4.16.0"