Close Milestone 4 with adapter adoption evidence
This commit is contained in:
parent
95271dcf2e
commit
6d06195950
27 changed files with 2870 additions and 325 deletions
|
|
@ -1,72 +1,15 @@
|
||||||
# Milestone state
|
# Milestone state
|
||||||
|
|
||||||
```text
|
```text
|
||||||
Last completed milestone: 3 — independent projections
|
Last completed milestone: 4 — adapter SDK and product documentation.
|
||||||
Evidence: d6d9f47672c513c4927c5c42ad97a175ed651a74 is merged and pushed on main and dev.
|
Evidence: Frozen executable candidate 95271dcf2e48045b9d3aed9b9ea09c7fc155692c; 347 tests and 402 subtests; 3 accessibility flows; offline fresh-wheel adoption; clean 1,002-node graph-plus-Logic benchmark; no secret-scan findings.
|
||||||
Active milestone: 4 — adapter SDK and product documentation.
|
Active milestone: None.
|
||||||
Outcome: New projects can adopt DocForge incrementally, and users can understand the CLI, MCP server, policies, adapters, renderers, and agent integration without reading implementation code.
|
Outcome: New projects can adopt DocForge incrementally, and users can understand the CLI, MCP server, policies, adapters, renderers, and agent integration without reading implementation code.
|
||||||
Status: Active. The first gate freezes and proves the adapter SDK before language references.
|
Status: Milestone 4 is closed. Milestone 5 has not started.
|
||||||
Next directional milestone: 5 — stabilization and first DocForge2 release.
|
Next directional milestone: 5 — stabilization and first DocForge2 release.
|
||||||
```
|
```
|
||||||
|
|
||||||
## Milestone 4 contract
|
Milestone 5 remains directional context. Do not begin release stabilization, versioning, tagging,
|
||||||
|
or publication without a new active-slice contract based on the Milestone 4 evidence.
|
||||||
|
|
||||||
Milestone 4 closes only when all of the following are true:
|
The completed Milestone 4 contract and its exclusions are preserved in `SLICE_HISTORY.md`.
|
||||||
|
|
||||||
1. A typed public adapter SDK and repository-native conformance kit make complete, incremental,
|
|
||||||
Logic, confinement, restart, cache-recovery, no-AST, and retrieval contracts usable without
|
|
||||||
reading core implementation.
|
|
||||||
2. Repository-owned runnable reference integrations exist for Python, JavaScript/TypeScript, and
|
|
||||||
C++. They declare their evidence authorities and unsupported facts. Syntax-only evidence never
|
|
||||||
claims compiler-resolved calls, inheritance, types, or ownership.
|
|
||||||
3. Heavy language frontends are optional. A base wheel and no-AST operation work without them.
|
|
||||||
Missing optional features fail with closed, actionable errors.
|
|
||||||
4. CLI and MCP reference tables are generated deterministically from implementation-owned
|
|
||||||
metadata and a repository gate rejects drift.
|
|
||||||
5. Custom adapters can declare one versioned, immutable, project-bound launcher. Codex, Claude,
|
|
||||||
and OpenClaw client configuration is generated from that launcher without arbitrary discovery
|
|
||||||
or execution.
|
|
||||||
6. Dedicated onboarding, authority, descriptor, policy, adapter, language, legacy/no-AST,
|
|
||||||
rendering, agent-integration, security, recovery/performance, and DocForge v1 migration
|
|
||||||
guidance passes a fresh-wheel adoption proof.
|
|
||||||
7. Existing package, import, CLI, MCP, legacy-adapter, generic-project, no-AST, projection,
|
|
||||||
application, and recovery behavior remains additively compatible.
|
|
||||||
|
|
||||||
## Fixed decisions
|
|
||||||
|
|
||||||
- Milestone 4 provides stable in-repository SDK APIs and reference integrations. It does not
|
|
||||||
promise separately published language-adapter distributions.
|
|
||||||
- Complete adapter evidence includes primary graph and function Logic. An incremental adapter that
|
|
||||||
publishes Logic must expose an independent complete Logic oracle before equivalence can pass.
|
|
||||||
- Python reference evidence may use the standard-library AST. JavaScript/TypeScript and C++
|
|
||||||
reference evidence uses optional pinned parsing frontends for syntax plus declared build
|
|
||||||
inventory. It omits semantic facts that require a compiler, language server, or SCIP evidence.
|
|
||||||
- C++ inventory is bound to a project-confined compilation database. WorldForge is not used or
|
|
||||||
changed.
|
|
||||||
- JavaScript and TypeScript are both exercised. TypeScript support requires its own optional
|
|
||||||
pinned grammar; it is not silently parsed as JavaScript.
|
|
||||||
- Adapter extraction caches remain disposable and may become safely ahead of the last published
|
|
||||||
SQLite index. The index remains the active validated generation. Recovery must prove this
|
|
||||||
behavior instead of claiming one joint cache-and-index transaction.
|
|
||||||
- Generated references and templates are derived artifacts. Implementation metadata is their
|
|
||||||
authority.
|
|
||||||
|
|
||||||
## Exclusions
|
|
||||||
|
|
||||||
- No WorldForge, ScrapeStation, legacy-repository, or production-binding change.
|
|
||||||
- No remote adapter execution, arbitrary adapter commands, renderer ecosystem, render farm,
|
|
||||||
storage rewrite, or cross-project graph.
|
|
||||||
- No compiler or language-server installation by DocForge.
|
|
||||||
- No self-hosting requirement.
|
|
||||||
- No version, release tag, or Forgejo release before Milestone 5.
|
|
||||||
|
|
||||||
## Slice order
|
|
||||||
|
|
||||||
1. Freeze complete graph-plus-Logic equivalence, the SDK facade, conformance protocol, schemas, and
|
|
||||||
baseline measurements.
|
|
||||||
2. Add deterministic command metadata, generated CLI/MCP references, and a drift gate.
|
|
||||||
3. Add the Python reference integration and scaled complete/incremental benchmark.
|
|
||||||
4. Add optional JavaScript/TypeScript and C++ reference integrations with bounded evidence.
|
|
||||||
5. Add versioned adapter launcher metadata and generated agent configurations.
|
|
||||||
6. Freeze the candidate, perform one documentation closeout, and run focused, complete,
|
|
||||||
fresh-wheel, adoption, benchmark, accessibility, build, and secret-scan gates.
|
|
||||||
|
|
|
||||||
|
|
@ -784,3 +784,108 @@ proof. Gitleaks 8.30.1 found no findings across the six Milestone 3 commits or c
|
||||||
Milestone 3 is complete. No tag, release, production integration repointing, WorldForge change,
|
Milestone 3 is complete. No tag, release, production integration repointing, WorldForge change,
|
||||||
ScrapeStation change, storage rewrite, or self-hosting dependency was introduced. Milestone 4
|
ScrapeStation change, storage rewrite, or self-hosting dependency was introduced. Milestone 4
|
||||||
remains directional and has not started.
|
remains directional and has not started.
|
||||||
|
|
||||||
|
## Milestone 4 — complete: adapter SDK and product documentation
|
||||||
|
|
||||||
|
Milestone 4 activated from the verified Milestone 3 closeout. Its scope was deliberately additive:
|
||||||
|
freeze an adapter-authoring boundary, prove narrow language references, make heavy frontends
|
||||||
|
optional, generate command and client integration evidence, and document adoption. WorldForge,
|
||||||
|
ScrapeStation, the legacy repository, production bindings, self-hosting, storage replacement, and
|
||||||
|
release publication remained excluded.
|
||||||
|
|
||||||
|
### Adapter SDK and conformance
|
||||||
|
|
||||||
|
`docforge.adapter_sdk` is now the public authoring facade for typed projections, manifests, source
|
||||||
|
contributions, complete assemblies, incremental loaders, project bindings, core graph models, and
|
||||||
|
conformance reports. Complete evidence includes primary graph and function Logic. An incremental
|
||||||
|
adapter that publishes Logic must provide an independent `load_complete_assembly()` oracle.
|
||||||
|
|
||||||
|
The conformance helper repeats complete loading for determinism, compares the complete assembly
|
||||||
|
with `load_projection()`, and proves exact complete/incremental graph-plus-Logic parity. It does not
|
||||||
|
pretend to replace separate confinement, restart, no-AST, corrupt-cache, or retrieval tests.
|
||||||
|
|
||||||
|
An adversarial review found that extraction caches and adapter assemblies lacked aggregate
|
||||||
|
boundaries. Version-1 cache reads and writes are now regular-file-only, identity-checked, capped at
|
||||||
|
64,000,000 bytes and 10,000 sources, and fail safely to a miss. Adapter projects enforce primary
|
||||||
|
node limits and deterministic edge and Logic multipliers before publication.
|
||||||
|
|
||||||
|
### Reference language evidence
|
||||||
|
|
||||||
|
The Python reference uses the standard-library AST and publishes files, modules, classes,
|
||||||
|
functions, arguments, local static imports, and function Logic. Its manifest uses tokenization
|
||||||
|
rather than AST parsing, so unchanged warm builds perform no syntax parse.
|
||||||
|
|
||||||
|
JavaScript and TypeScript use separate pinned optional Tree-sitter grammars. They publish files,
|
||||||
|
modules, classes, functions/methods, static project-relative imports/re-exports, and function
|
||||||
|
Logic. Their manifests use a closed comment/string-aware ESM scan and do not load Tree-sitter.
|
||||||
|
|
||||||
|
C++ uses explicit non-overlapping source roots and one confined `compile_commands.json` as
|
||||||
|
translation-unit inventory and fingerprint evidence. It validates entries but never executes a
|
||||||
|
command, compiler, response file, or project program. It publishes syntax and directly resolvable
|
||||||
|
project-local quoted includes. Its manifest currently parses for include discovery, so no
|
||||||
|
zero-warm-parser claim is made.
|
||||||
|
|
||||||
|
None of the references claim compiler-resolved calls, inheritance, types, symbol references,
|
||||||
|
compiler include semantics, macro semantics, runtime behavior, or semantic ownership.
|
||||||
|
|
||||||
|
The first packaging proof exposed that language parsers were inherited as mandatory dependencies.
|
||||||
|
The base wheel now depends only on Markdown and MCP packages. Python works in the base wheel;
|
||||||
|
JavaScript, TypeScript, C++, and all-language extras install their respective Tree-sitter
|
||||||
|
frontends. Missing extras return a closed error with the exact install target.
|
||||||
|
|
||||||
|
### Fixed reference MCP and launchers
|
||||||
|
|
||||||
|
A strict `.docforge/reference-adapter.toml` selects one fixed reference language, project
|
||||||
|
identity, and explicit source roots. C++ additionally requires the compilation database. The fixed
|
||||||
|
`python -m docforge.reference_mcp` binding constructs only repository-owned providers and exposes
|
||||||
|
the 21 read tools.
|
||||||
|
|
||||||
|
The initial launcher design generated `python -I -m` for arbitrary project module names without
|
||||||
|
proving that isolated Python could resolve them. The corrected `AdapterLauncherV1` accepts only an
|
||||||
|
installed project-confined top-level module or the exact trusted `docforge.reference_mcp`
|
||||||
|
exception. A fixed bounded `find_spec` probe runs under isolated Python without importing project
|
||||||
|
code. Tests then launch both a real project module and the real reference MCP process.
|
||||||
|
|
||||||
|
Generated Codex, Claude, and OpenClaw fragments bind the exact launcher, descriptor, source
|
||||||
|
availability, effective and projection policy, interpreter, canonical arguments, empty
|
||||||
|
environment, artifact bytes, and hashes. There is no arbitrary command, argument list, working
|
||||||
|
directory, environment, discovery, or callable selector.
|
||||||
|
|
||||||
|
### Generated reference and documentation gates
|
||||||
|
|
||||||
|
CLI tables are derived from the real argparse subcommands. MCP tables are derived from a real
|
||||||
|
application-enabled server's `list_tools()` registrations, including capability surface,
|
||||||
|
arguments, descriptions, and input-schema hashes. The generated artifact contains 28 CLI rows and
|
||||||
|
36 MCP rows.
|
||||||
|
|
||||||
|
An adversarial review found a target-change race between the generator's identity check and
|
||||||
|
replacement. Publication now locks cooperating generators, uses no-clobber publication for a
|
||||||
|
missing target, and uses Linux atomic exchange plus displaced-byte and file-identity verification
|
||||||
|
for an existing target. Raced data is restored or retained for recovery rather than discarded.
|
||||||
|
|
||||||
|
`docs-check` validates generated drift, local files and anchors, one H1 per page, README-rooted
|
||||||
|
reachability, required Milestone 4 inventory, the generated notice, and every documented
|
||||||
|
reference-adapter TOML example against the packaged schema.
|
||||||
|
|
||||||
|
### Candidate, adoption, and scale evidence
|
||||||
|
|
||||||
|
Frozen executable candidate `95271dcf2e48045b9d3aed9b9ea09c7fc155692c` passed formatting,
|
||||||
|
Python and web lint, strict Pyright, compilation, 142 contract tests and 268 subtests, 347 total
|
||||||
|
tests and 402 subtests, three accessibility flows, lock and dependency checks, package builds,
|
||||||
|
offline wheel adoption, and every maintained smoke benchmark.
|
||||||
|
|
||||||
|
The offline adoption proof installed the base wheel from the lock without network access. It
|
||||||
|
proved no Tree-sitter package or module was installed, built and checked a real Python reference
|
||||||
|
project, performed isolated MCP bootstrap/search/get-node over the exact 21 read tools, and proved
|
||||||
|
that base-only C++ fails with `optional_dependency_missing` and `docforge[cpp]` remediation.
|
||||||
|
|
||||||
|
The clean 334-source benchmark produced 1,002 primary nodes, 1,001 primary edges, 334 Logic
|
||||||
|
projections, 2,338 Logic nodes, and 2,338 Logic edges. Complete and incremental output matched
|
||||||
|
exactly. Warm build p95 was 1,265.387 ms with zero `ast.parse` and zero `extract_source` calls.
|
||||||
|
Complete equivalence, corrupt cache recovery, and corrupt index recovery were all below 1.9
|
||||||
|
seconds. The largest per-operation traced peak was 69,997,166 bytes and process high-water was
|
||||||
|
78,798,848 bytes. Exact hashes and method are recorded in
|
||||||
|
`benchmarks/milestone4-2026-07-29.json` and `docs/MILESTONE_4_BASELINE.md`.
|
||||||
|
|
||||||
|
Milestone 4 is complete. No tag or release was created. Milestone 5 remains unstarted until its own
|
||||||
|
active-slice contract is frozen.
|
||||||
|
|
|
||||||
409
README.md
409
README.md
|
|
@ -1,88 +1,107 @@
|
||||||
# DocForge
|
# DocForge
|
||||||
|
|
||||||
DocForge is a project-scoped documentation graph for people and AI agents. It validates canonical
|
DocForge is a project-scoped documentation and source graph for people and AI agents. Canonical
|
||||||
documentation, builds a disposable search and relationship index, compiles bounded context, renders
|
project files remain authoritative; DocForge validates them, builds disposable search and
|
||||||
declared manuals, visualizes project structure, and manages reviewable documentation changesets.
|
relationship indexes, compiles bounded context, manages reviewable changesets, renders declared
|
||||||
|
manuals and portable graph views, and serves one project-bound MCP surface.
|
||||||
|
|
||||||
## What it does
|
DocForge never treats indexed text as instructions. It does not run project build commands,
|
||||||
|
compilers, Git operations, deployments, or arbitrary renderers, and it does not select projects
|
||||||
|
globally.
|
||||||
|
|
||||||
- Validates stable Markdown/TOML nodes and typed relationships.
|
DocForge 1.0.0 remains the first stable release in this product line. This checkout is the
|
||||||
- Builds a deterministic SQLite search and graph index.
|
post-1.0 development line (`1.3.0.dev0`) and contains the Milestone 4 adapter-platform work.
|
||||||
- Exposes project-bound CLI and MCP query surfaces.
|
Milestone 4 is not a new product release, and Milestone 5 release and publication work remains
|
||||||
- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, and bounded
|
separate.
|
||||||
continuation.
|
|
||||||
- Records one bounded, versioned latest-generation graph transition without creating a history
|
|
||||||
database.
|
|
||||||
- Generates deterministic project-bound Codex, Claude, and OpenClaw client fragments without
|
|
||||||
copying ambient secrets.
|
|
||||||
- Diagnoses one client binding through bounded read-only checks without starting MCP or rebuilding
|
|
||||||
project state.
|
|
||||||
- Automatically synchronizes disposable indexes before MCP work.
|
|
||||||
- Creates, validates, diffs, and previews isolated changesets.
|
|
||||||
- Registers complete proposals atomically without caller-managed hash chaining.
|
|
||||||
- Applies one explicitly approved changeset hash through CLI or gated MCP.
|
|
||||||
- Supports opt-in incremental adapters with reverse-dependency invalidation and full-build
|
|
||||||
equivalence checks.
|
|
||||||
- Detects project-local adapter implementation and configuration changes and requires a fresh
|
|
||||||
project-bound process before any further MCP work.
|
|
||||||
- Keeps function-scoped control-flow projections separate from the primary architecture graph.
|
|
||||||
- Compiles manuals and portable graph artifacts from separate versioned, generation-pinned plans
|
|
||||||
and immutable packages.
|
|
||||||
- Runs built-in manual and graph renderers in fixed detached workers with validated receipts,
|
|
||||||
bounded transfer, and no project-path authority.
|
|
||||||
- Enforces independent manual, portable-graph, and live-viewer policy while keeping status
|
|
||||||
receipt-only.
|
|
||||||
- Runs a managed loopback graph browser with neighborhood, semantic Flow, convergence Web,
|
|
||||||
function-scoped Logic, source inspection, and branch-aware node hiding.
|
|
||||||
- Supports generic documentation projects and project-owned source adapters.
|
|
||||||
|
|
||||||
DocForge never treats indexed text as instructions. It does not run shell commands, mutate Git,
|
## Start here
|
||||||
build applications, deploy, publish, or select projects globally.
|
|
||||||
|
|
||||||
For implementation projects, the recommended cadence is to read canonical documentation during
|
- New installation or first project: [New-project quickstart](docs/NEW_PROJECT_QUICKSTART.md)
|
||||||
intake, keep it read-only through implementation and focused testing, freeze and validate a release
|
- Mental model and authority: [Core concepts and authority](docs/CORE_CONCEPTS_AND_AUTHORITY.md)
|
||||||
candidate, then perform one atomic documentation closeout before the final commit and tag. This
|
- Complete configuration shape: [Project descriptor](docs/PROJECT_DESCRIPTOR.md)
|
||||||
keeps the manual authoritative without using it as an implementation notebook.
|
- Fixed runnable examples: [Reference adapters](docs/REFERENCE_ADAPTERS.md)
|
||||||
|
- Agent and client setup: [Agent integration](docs/AGENT_INTEGRATION.md)
|
||||||
|
- Exact live command and tool inventory: [Generated command reference](docs/COMMAND_REFERENCE.md)
|
||||||
|
- Task-oriented operating guide: [User manual](docs/USER_MANUAL.md)
|
||||||
|
|
||||||
## Release 1
|
## Current capabilities
|
||||||
|
|
||||||
DocForge 1.0.0 is the first stable product release. It combines the project-scoped graph, CLI and
|
- Validates Markdown and TOML nodes, stable IDs, typed relationships, project limits, and confined
|
||||||
MCP query surfaces, reviewable hash-approved changesets, generic and project-owned adapters,
|
paths.
|
||||||
declared rendering, and the complete Nodes/Flow/Web visualization model in one supported release.
|
- Builds a deterministic, disposable SQLite graph and search index with integrity and generation
|
||||||
|
evidence.
|
||||||
|
- Exposes project-bound CLI and MCP read, proposal, application, rendering, and visualization
|
||||||
|
surfaces according to the startup policy.
|
||||||
|
- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, bounded
|
||||||
|
output, and deterministic continuation.
|
||||||
|
- Creates isolated documentation changesets, validates complete projected graphs, and applies only
|
||||||
|
one explicitly approved changeset hash through a separately bound canonical applier.
|
||||||
|
- Supports complete-projection adapters and opt-in incremental adapters with reverse-dependency
|
||||||
|
invalidation, bounded extraction caches, and a clean full-build equivalence oracle.
|
||||||
|
- Publishes function-scoped Logic separately from the primary graph for Python, JavaScript,
|
||||||
|
TypeScript, and C++ integrations that provide it.
|
||||||
|
- Compiles manual and portable graph plans into immutable packages for fixed detached renderers,
|
||||||
|
then records bounded receipts.
|
||||||
|
- Runs a managed loopback graph viewer with Nodes, Flow, Web, lazy Logic, source inspection, and
|
||||||
|
branch-aware hiding.
|
||||||
|
- Generates deterministic Codex, Claude, and OpenClaw client fragments without copying ambient
|
||||||
|
environment values or secrets.
|
||||||
|
|
||||||
The post-1.0 incremental compiler is a backward-compatible, optional enhancement. Existing Release
|
## Adapter platform
|
||||||
1 adapters that implement only `load_projection()` continue to use the original complete-projection
|
|
||||||
path without modification. Adapters gain incremental performance only when they additionally
|
|
||||||
implement the source manifest and extraction methods. Incremental adapters must retain
|
|
||||||
`load_projection()` as their clean-rebuild fallback and equivalence oracle.
|
|
||||||
|
|
||||||
MCP bindings can additionally select `--no-ast` when an owner wants to preserve an existing
|
Adapter authors use the public `docforge.adapter_sdk` surface for graph types, complete and
|
||||||
non-AST adapter. The binding advertises that policy to clients, forbids adapter rewrites that add
|
incremental contracts, validation helpers, and `verify_adapter_conformance()`. Existing adapters
|
||||||
AST, Tree-sitter, compiler-AST, or function-Logic extraction, blocks the Logic tool, and rejects
|
that implement only `load_projection()` remain supported. Incremental adapters add a manifest and
|
||||||
nonempty Logic publication. Complete-projection adapters continue unchanged, and non-AST
|
source extraction while retaining `load_projection()` as the complete graph oracle. An adapter
|
||||||
incremental fingerprinting and caching remain allowed.
|
that publishes Logic incrementally also supplies `load_complete_assembly()` so the complete oracle
|
||||||
|
covers both graph and Logic.
|
||||||
|
|
||||||
DocForge2 bindings may also declare
|
The repository includes bounded reference integrations for:
|
||||||
`--capability-mode read|proposal|application|operator`. Bootstrap returns one versioned effective
|
|
||||||
policy and the actual startup-gated capabilities. Existing tool surfaces and the legacy no-AST
|
|
||||||
payload remain compatible.
|
|
||||||
|
|
||||||
`docforge_get_task_context` is an additive read tool for `change`, `implementation`, `failure`,
|
- Python, using the standard-library AST and publishing only project-local imports;
|
||||||
`ownership`, `test`, `operation`, and `release` work. It derives a closed version-1 retrieval plan,
|
- JavaScript and TypeScript, using their distinct optional Tree-sitter grammars and publishing only
|
||||||
executes it against one immutable index generation, and returns a hash-bound context capsule.
|
project-local static relative imports and re-exports;
|
||||||
Project relation names remain authoritative. DocForge classifies only its versioned alias set and
|
- C++, using `compile_commands.json` as translation-unit inventory and fingerprint evidence, without
|
||||||
preserves every unknown relation as `unclassified` instead of guessing semantics.
|
executing its commands or a compiler, and publishing only directly resolvable project-local
|
||||||
|
quoted includes.
|
||||||
|
|
||||||
`docforge_get_generation_diff` reports the latest verified primary-graph transition through one
|
These integrations demonstrate the adapter contract; they do not claim resolved calls,
|
||||||
bounded disposable receipt. It includes exact node and edge change counts, hash-bound retained
|
inheritance, types, runtime behavior, macro expansion, compiler include semantics, or semantic
|
||||||
details, and explicit truncation. Paged results use one top-level cursor and a versioned
|
ownership. Python works from the base wheel. Install the `javascript`, `typescript`, or `cpp`
|
||||||
`receipt_header`; `stored_receipt_hash` identifies the complete persisted receipt. The read never
|
extra for the corresponding grammar, or `languages` for all three:
|
||||||
exposes Logic details, loads canonical source, repairs derived state, or invents history.
|
|
||||||
|
|
||||||
## Independent projections
|
```bash
|
||||||
|
uv pip install "/absolute/path/DocForge[javascript]"
|
||||||
|
uv pip install "/absolute/path/DocForge[typescript]"
|
||||||
|
uv pip install "/absolute/path/DocForge[cpp]"
|
||||||
|
uv pip install "/absolute/path/DocForge[languages]"
|
||||||
|
```
|
||||||
|
|
||||||
Manual compilation, portable graph rendering, and the live viewer consume the same validated graph
|
Reference projects use the fixed `.docforge/reference-adapter.toml` descriptor and the installed
|
||||||
generation through separate boundaries:
|
`docforge.reference_mcp` module. That server is read-only and exposes no proposal or application
|
||||||
|
surface.
|
||||||
|
|
||||||
|
Project-owned adapters use `AdapterLauncherV1` and
|
||||||
|
`generate_adapter_client_configuration()` to produce a client fragment from an explicitly
|
||||||
|
constructed adapter project. The launcher is immutable and contains no command, caller arguments,
|
||||||
|
working directory, environment, discovery rule, or callable selector. It resolves through isolated
|
||||||
|
Python to one installed, project-owned top-level module; the only trusted dotted exception is the
|
||||||
|
fixed `docforge.reference_mcp` binding. Generic `docforge configure` intentionally refuses custom
|
||||||
|
adapters because it cannot safely reconstruct project-owned composition.
|
||||||
|
|
||||||
|
See the [Adapter authoring guide](docs/ADAPTER_AUTHORING_GUIDE.md), [Reference
|
||||||
|
adapters](docs/REFERENCE_ADAPTERS.md), and [Agent integration](docs/AGENT_INTEGRATION.md) for the
|
||||||
|
supported routes.
|
||||||
|
|
||||||
|
## Authority and projections
|
||||||
|
|
||||||
|
Canonical Markdown, TOML, adapter-declared sources, and descriptor files own project facts. The
|
||||||
|
SQLite index, extraction cache, render packages, previews, portable artifacts, receipts, client
|
||||||
|
fragments, and viewer processes are derived and replaceable.
|
||||||
|
|
||||||
|
The primary graph contains project nodes and relationships. Logic is a separate, lazy,
|
||||||
|
function-scoped control-flow projection. Manual output, portable graph output, and the live viewer
|
||||||
|
are independent consumers of one validated generation:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
validated generation
|
validated generation
|
||||||
|
|
@ -91,190 +110,142 @@ validated generation
|
||||||
└── pinned index → managed read-only live viewer
|
└── pinned index → managed read-only live viewer
|
||||||
```
|
```
|
||||||
|
|
||||||
Plans, packages, and receipts are canonical, versioned, hash-identified, bounded, and contain no
|
The version-2 projection policy selects each consumer independently:
|
||||||
project object, SQLite handle, absolute project path, command, or caller-selected renderer module.
|
|
||||||
Renderers cannot select graph facts, crawl canonical sources, choose publication paths, or mutate
|
|
||||||
project state.
|
|
||||||
|
|
||||||
Declare portable graph output separately from manual views:
|
```text
|
||||||
|
manual: auto | explicit | disabled
|
||||||
```toml
|
portable_graph: explicit | disabled
|
||||||
[graph_render]
|
live_viewer: on-demand | disabled
|
||||||
output_root = ".docforge/portable-graph"
|
|
||||||
|
|
||||||
[[graph_render.views]]
|
|
||||||
id = "architecture"
|
|
||||||
renderer = "portable_graph_html"
|
|
||||||
output = "architecture.html"
|
|
||||||
title = "Architecture"
|
|
||||||
root = "architecture.overview"
|
|
||||||
initial_mode = "web"
|
|
||||||
depth = 3
|
|
||||||
max_nodes = 250
|
|
||||||
max_edges = 1000
|
|
||||||
max_work = 100000
|
|
||||||
include_logic = false
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Plan, publish, and inspect it explicitly:
|
The process capability policy is separate. Capability mode controls the registered read,
|
||||||
|
proposal, application, or reserved operator surface; descriptor writers and the startup-bound
|
||||||
|
canonical applier determine whether mutations are actually authorized. `--no-ast` is a restrictive
|
||||||
|
binding policy over adapter evolution, Logic publication, and Logic retrieval. It is not a parser
|
||||||
|
inspection mechanism or a filesystem sandbox, and the AST/Tree-sitter reference integrations
|
||||||
|
should not be presented as no-AST adapters.
|
||||||
|
|
||||||
|
Read [Policy precedence](docs/POLICY_PRECEDENCE.md), [Rendering and
|
||||||
|
visualization](docs/RENDERING_AND_VISUALIZATION.md), and [Legacy and no-AST
|
||||||
|
operation](docs/LEGACY_AND_NO_AST.md) before changing those boundaries.
|
||||||
|
|
||||||
|
## Five-minute generic project
|
||||||
|
|
||||||
|
Requirements are Python 3.12 or newer and an installed DocForge environment. From a development
|
||||||
|
checkout, `uv sync --group dev` creates `.venv`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
.venv/bin/docforge --project-root "$PROJECT" graph-plan architecture
|
git clone <repository-url> /absolute/path/DocForge
|
||||||
.venv/bin/docforge --project-root "$PROJECT" graph-render architecture
|
cd /absolute/path/DocForge
|
||||||
.venv/bin/docforge --project-root "$PROJECT" graph-render-status architecture
|
|
||||||
```
|
|
||||||
|
|
||||||
The version-2 projection policy independently selects manual
|
|
||||||
`auto|explicit|disabled`, portable graph `explicit|disabled`, and live viewer
|
|
||||||
`on-demand|disabled`. Use `--manual-render-policy`, `--portable-graph-policy`, and
|
|
||||||
`--live-viewer-policy` on CLI/MCP startup or generated client configuration. Status remains
|
|
||||||
available when the corresponding active operation is disabled.
|
|
||||||
|
|
||||||
Manual fragment reuse is disposable. Cold record publication is guarded by byte-exact comparison
|
|
||||||
with a full detached render; warm records are independently recomputed and validated inside the
|
|
||||||
worker. Full rendering remains the recovery and equivalence oracle. Portable artifacts commit
|
|
||||||
content-addressed output and renderer evidence before one bounded generation/view manifest; status
|
|
||||||
does not plan or render.
|
|
||||||
|
|
||||||
## Graph views
|
|
||||||
|
|
||||||
The browser presents the primary architecture graph through three complementary views and loads a
|
|
||||||
fourth function-scoped view only when requested:
|
|
||||||
|
|
||||||
- **Nodes** shows a bounded, relation-neutral neighborhood around the focus. It is the broad
|
|
||||||
inspection view for seeing stored incoming and outgoing relationships without changing their
|
|
||||||
direction. Semantic cards distinguish structure, behavior, dependencies, execution, data,
|
|
||||||
evidence, context, and other relationships.
|
|
||||||
- **Flow** shows semantic origin-to-destination paths that terminate at the focus. DocForge
|
|
||||||
reverses prerequisite-style relationships for presentation, so imports, dependencies, reads,
|
|
||||||
inheritance, definitions, and tests flow toward the thing they help create or exercise.
|
|
||||||
- **Web** shows the larger convergence picture: Flow contributors plus contextual relationships,
|
|
||||||
callers, containers, and direct members or execution dependencies owned by the focus.
|
|
||||||
- **Logic** shows the possible static control paths inside a focused Python, JavaScript, or C++
|
|
||||||
function or method. Entry, decisions, actions, loops, convergence points, returns, and
|
|
||||||
exceptions connect through explicit
|
|
||||||
`TRUE`, `FALSE`, `NEXT`, `CASE`, `LOOP`, `RETURN`, and `RAISE` paths. Logic is stored separately
|
|
||||||
and does not add statement-level noise to Nodes, Flow, Web, or search.
|
|
||||||
|
|
||||||
Graph cards show the node's readable leaf name and kind without clipping either value. The full
|
|
||||||
qualified identity remains available in the tooltip, compact descriptor, and full inspector.
|
|
||||||
The left browser panel can combine text, family, node-kind, language, and capability filters.
|
|
||||||
Quick presets expose Logic-ready nodes, Python callables, tests, routes, and documentation without
|
|
||||||
requiring users to know stable IDs. Selecting a canvas node emphasizes its directly connected
|
|
||||||
neighbors and edges while muting unrelated paths.
|
|
||||||
|
|
||||||
**Hide node** removes noise without changing the index. In Flow and Web, hiding a contributor also
|
|
||||||
removes upstream ancestors that no longer have a path to the focus. Nodes between the hidden
|
|
||||||
contributor and the focus stay visible, and alternate ancestor paths remain intact. In Logic,
|
|
||||||
hiding a step inserts an explicit omitted-path bridge so downstream control flow remains readable.
|
|
||||||
**Restore hidden** restores the presentation.
|
|
||||||
|
|
||||||
## Five-minute start
|
|
||||||
|
|
||||||
Requirements are Python 3.12+, `uv`, and Node.js/npm.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git clone forgejo@repo.andraxion.net:administrator/DocForge2.git /absolute/path/DocForge2
|
|
||||||
cd /absolute/path/DocForge2
|
|
||||||
uv sync --group dev
|
uv sync --group dev
|
||||||
npm ci
|
|
||||||
|
|
||||||
PROJECT=/absolute/path/MyProject
|
PROJECT=/absolute/path/MyProject
|
||||||
|
.venv/bin/docforge --project-root "$PROJECT" onboard
|
||||||
|
.venv/bin/docforge --project-root "$PROJECT" onboard \
|
||||||
|
--scaffold \
|
||||||
|
--project-id my-project \
|
||||||
|
--title "My Project"
|
||||||
|
```
|
||||||
|
|
||||||
|
The first command is read-only. Scaffolding is explicit and create-only: it writes a generic
|
||||||
|
descriptor, one canonical overview node, and a built-in manual template, then indexes and renders
|
||||||
|
them. Detected source languages remain `adapter_required` until a validated frontend is selected.
|
||||||
|
|
||||||
|
Operate the configured project:
|
||||||
|
|
||||||
|
```bash
|
||||||
.venv/bin/docforge --project-root "$PROJECT" validate
|
.venv/bin/docforge --project-root "$PROJECT" validate
|
||||||
.venv/bin/docforge --project-root "$PROJECT" reindex
|
.venv/bin/docforge --project-root "$PROJECT" reindex
|
||||||
|
.venv/bin/docforge --project-root "$PROJECT" search architecture
|
||||||
.venv/bin/docforge --project-root "$PROJECT" visualize
|
.venv/bin/docforge --project-root "$PROJECT" visualize
|
||||||
```
|
```
|
||||||
|
|
||||||
Install the persistent per-user graph viewer once:
|
Install the persistent per-user graph viewer manager once when using the live viewer:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
.venv/bin/docforge-viewer-manager install-user-service
|
.venv/bin/docforge-viewer-manager install-user-service
|
||||||
```
|
```
|
||||||
|
|
||||||
Start an MCP server for one project:
|
Start one generic, read-only MCP server:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
.venv/bin/docforge-mcp \
|
.venv/bin/docforge-mcp \
|
||||||
--project-root "$PROJECT" \
|
--project-root "$PROJECT" \
|
||||||
--proposal-writer project-editor
|
--capability-mode read
|
||||||
```
|
```
|
||||||
|
|
||||||
Add `--canonical-applier project-editor` only when that MCP integration should expose the
|
Add a descriptor-authorized proposal writer only when the client should create proposals. Add a
|
||||||
hash-bound `docforge_apply_changeset` tool.
|
matching `--canonical-applier` only when that integration should expose exact-hash application.
|
||||||
|
See the [quickstart](docs/NEW_PROJECT_QUICKSTART.md) for the reference-adapter and generated-client
|
||||||
Preview a read-only Codex fragment and diagnose an installed binding:
|
paths.
|
||||||
|
|
||||||
```bash
|
|
||||||
.venv/bin/docforge configure codex --project "$PROJECT"
|
|
||||||
.venv/bin/docforge doctor --client codex --project "$PROJECT"
|
|
||||||
```
|
|
||||||
|
|
||||||
Pass `--output /absolute/path/docforge.toml` only when creating a standalone fragment. DocForge
|
|
||||||
never replaces or merges an existing different client file.
|
|
||||||
|
|
||||||
For an unconfigured codebase, begin with a read-only language and documentation assessment:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
.venv/bin/docforge --project-root /absolute/path/MyProject onboard
|
|
||||||
```
|
|
||||||
|
|
||||||
Add `--scaffold`, a stable project ID, and a title to create, index, and render a generic starter
|
|
||||||
manual. Source files are reported separately and require a validated language frontend before
|
|
||||||
DocForge describes them as a source graph.
|
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
- [User manual](docs/USER_MANUAL.md) — features, setup, visualization, CLI, MCP, apply, adapters,
|
### Learn and operate
|
||||||
and troubleshooting.
|
|
||||||
- [Core contract](docs/CONTRACT.md) — invariants and security boundary.
|
- [New-project quickstart](docs/NEW_PROJECT_QUICKSTART.md)
|
||||||
- [Milestone 0 compatibility](docs/COMPATIBILITY.md) — preserved package, CLI, MCP, adapter,
|
- [Core concepts and authority](docs/CORE_CONCEPTS_AND_AUTHORITY.md)
|
||||||
schema, changeset, rendering, and no-AST guarantees.
|
- [Project descriptor](docs/PROJECT_DESCRIPTOR.md)
|
||||||
- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) — validation evidence, cold and warm
|
- [Policy precedence](docs/POLICY_PRECEDENCE.md)
|
||||||
performance, memory, rendering and response sizes, bottlenecks, and missing coverage.
|
- [Reference adapters](docs/REFERENCE_ADAPTERS.md)
|
||||||
- [Milestone 0 closeout](docs/MILESTONE_0_CLOSEOUT.md) — lineage, migration, security scan,
|
- [Agent integration](docs/AGENT_INTEGRATION.md)
|
||||||
repository state, and fresh-clone proof.
|
- [User manual](docs/USER_MANUAL.md)
|
||||||
- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) — warm operation latency, structured work,
|
- [Generated command reference](docs/COMMAND_REFERENCE.md)
|
||||||
status, retrieval, and memory measurements.
|
- [Project onboarding](docs/PROJECT_ONBOARDING.md)
|
||||||
- [Milestone 1 closeout](docs/MILESTONE_1_CLOSEOUT.md) — fast-core contracts, adversarial
|
- [Rendering and visualization](docs/RENDERING_AND_VISUALIZATION.md)
|
||||||
validation, compatibility boundaries, and exact candidate evidence.
|
- [Recovery and performance](docs/RECOVERY_AND_PERFORMANCE.md)
|
||||||
- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) — task context, generation diff, client
|
- [Security](docs/SECURITY.md)
|
||||||
configuration, doctor, response-size, counter, and memory measurements.
|
|
||||||
- [Milestone 2 closeout](docs/MILESTONE_2_CLOSEOUT.md) — implemented contracts, adversarial
|
### Contracts and compatibility
|
||||||
validation, exclusions, and exact candidate evidence.
|
|
||||||
- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) — manual, fragment, worker, portable graph,
|
- [Core contract](docs/CONTRACT.md)
|
||||||
status, equivalence, response-size, and memory measurements.
|
- [MCP contract](docs/MCP_CONTRACT.md)
|
||||||
- [Milestone 3 closeout](docs/MILESTONE_3_CLOSEOUT.md) — independent projection contracts,
|
- [Compatibility contract](docs/COMPATIBILITY.md)
|
||||||
adversarial validation, compatibility boundaries, and exact candidate evidence.
|
- [Adapter authoring guide](docs/ADAPTER_AUTHORING_GUIDE.md)
|
||||||
- [MCP contract](docs/MCP_CONTRACT.md) — exact tool and process boundary.
|
- [Incremental indexing](docs/INCREMENTAL_INDEXING.md)
|
||||||
- [Viewer manager](docs/VIEWER_MANAGER.md) — native service setup and lifecycle.
|
- [Adapter application decision](docs/APPLICATION_DECISION.md)
|
||||||
- [Adapter decision](docs/APPLICATION_DECISION.md) — why custom adapters own canonical
|
- [Legacy and no-AST operation](docs/LEGACY_AND_NO_AST.md)
|
||||||
serialization.
|
- [Migrating from version 1](docs/MIGRATING_FROM_V1.md)
|
||||||
- [Incremental adapter indexing](docs/INCREMENTAL_INDEXING.md) — source-scoped extraction,
|
- [Viewer manager](docs/VIEWER_MANAGER.md)
|
||||||
invalidation, equivalence, relationship changes, and the lazy Logic boundary.
|
|
||||||
- [Project onboarding](docs/PROJECT_ONBOARDING.md) — repository assessment, safe manual
|
### Milestone evidence
|
||||||
scaffolding, language frontends, source/manual integration, proof, and MCP activation.
|
|
||||||
- [Language adapter authoring](docs/ADAPTER_AUTHORING_GUIDE.md) — implementation sequence,
|
- [Milestone 4 baseline](docs/MILESTONE_4_BASELINE.md)
|
||||||
stable identities, overlap ownership, normalization, incremental equivalence, troubleshooting,
|
- [Milestone 4 closeout](docs/MILESTONE_4_CLOSEOUT.md)
|
||||||
and the complete adapter proof matrix.
|
- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) and [closeout](docs/MILESTONE_3_CLOSEOUT.md)
|
||||||
|
- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) and [closeout](docs/MILESTONE_2_CLOSEOUT.md)
|
||||||
|
- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) and [closeout](docs/MILESTONE_1_CLOSEOUT.md)
|
||||||
|
- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) and [closeout](docs/MILESTONE_0_CLOSEOUT.md)
|
||||||
|
|
||||||
|
Historical milestone records preserve the facts and dependency observations of their frozen
|
||||||
|
candidates. Use the current guides and contracts for present behavior.
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
Run the complete repository-native gate:
|
Run the repository-native gate:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
make gate
|
make gate
|
||||||
```
|
```
|
||||||
|
|
||||||
Focused entry points are available as `make contract`, `make test`, `make type`,
|
Milestone 4 maintenance entry points include:
|
||||||
`make benchmark-smoke`, `make benchmark`, `make benchmark-m1-smoke`, and
|
|
||||||
`make benchmark-m1`. Milestone 2 adds `make benchmark-m2-smoke` and `make benchmark-m2`.
|
|
||||||
Milestone 3 adds `make accessibility`, `make benchmark-m3-smoke`, `make benchmark-m3`, and
|
|
||||||
`make benchmark-m3-full`.
|
|
||||||
|
|
||||||
The committed 1,000-node baseline and its measurement method are under `benchmarks/`.
|
```bash
|
||||||
|
make adoption-m4
|
||||||
|
make benchmark-m4-smoke
|
||||||
|
make benchmark-m4
|
||||||
|
make benchmark-m4-full
|
||||||
|
make command-reference-check
|
||||||
|
make docs-check
|
||||||
|
```
|
||||||
|
|
||||||
Pass `--diagnostics` to `docforge` or `docforge-mcp` to attach bounded request-local stage timings
|
`adoption-m4` builds and exercises a fresh base wheel without Tree-sitter packages and proves the
|
||||||
and compiler-work counters. Diagnostics are disabled by default and are dropped before primary MCP
|
Python reference plus a real isolated read-only MCP retrieval. The optional-language integrations
|
||||||
results when the configured output budget is tight.
|
have their own focused tests and extras. `benchmark-m4` runs the maintained full adapter workload;
|
||||||
|
the smoke target is for routine gate coverage, not final performance evidence.
|
||||||
|
|
||||||
See [AGENTS.md](AGENTS.md) before changing core boundaries.
|
Pass `--diagnostics` to `docforge` or `docforge-mcp` for bounded request-local timings and compiler
|
||||||
|
work counters. Diagnostics are disabled by default and do not displace a primary result when the
|
||||||
|
configured output budget is tight.
|
||||||
|
|
||||||
|
Read [AGENTS.md](AGENTS.md) before changing core boundaries.
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,50 @@
|
||||||
# Completed slices
|
# Completed slices
|
||||||
|
|
||||||
|
## DocForge2 Milestone 4 adapter SDK and product documentation
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Added stable `docforge.adapter_sdk` authoring imports and a conformance helper for deterministic
|
||||||
|
complete and exact complete/incremental primary graph plus Logic equivalence.
|
||||||
|
- Added bounded Python, JavaScript, TypeScript, and C++ reference integrations with explicit
|
||||||
|
unsupported-fact reports and optional heavy frontends.
|
||||||
|
- Added a closed `.docforge/reference-adapter.toml` contract and fixed read-only reference MCP
|
||||||
|
server.
|
||||||
|
- Added immutable project-bound launcher declarations and deterministic Codex, Claude, and OpenClaw
|
||||||
|
configuration generation for custom adapters.
|
||||||
|
- Generated CLI and MCP references from live implementation metadata and added race-safe
|
||||||
|
publication plus repository drift checks.
|
||||||
|
- Added strict documentation-graph and fresh-wheel adoption gates and dedicated onboarding,
|
||||||
|
authority, descriptor, policy, adapter, legacy/no-AST, rendering, agent-integration, security,
|
||||||
|
recovery/performance, and v1 migration guides.
|
||||||
|
|
||||||
|
### Verification
|
||||||
|
|
||||||
|
- The executable candidate passed 347 tests and 402 subtests, 142 contract tests and 268 subtests,
|
||||||
|
three accessibility flows, lint, strict types, compilation, dependencies, builds, adoption, and
|
||||||
|
every maintained smoke benchmark.
|
||||||
|
- The offline base wheel contained no Tree-sitter distribution and ran a real Python reference
|
||||||
|
build/check and isolated 21-tool MCP retrieval flow. Missing C++ support failed with the exact
|
||||||
|
`docforge[cpp]` remediation.
|
||||||
|
- The clean 334-source benchmark produced 1,002 nodes and exact full/incremental graph and Logic
|
||||||
|
evidence, warm zero Python parsing/extraction, and exact corrupt-cache and corrupt-index
|
||||||
|
recovery below every latency, memory, and response gate.
|
||||||
|
- Gitleaks found no findings in the Milestone 4 commit range or candidate tree.
|
||||||
|
|
||||||
|
### Limits
|
||||||
|
|
||||||
|
- The reference integrations publish syntax and local static relationships. They do not claim
|
||||||
|
compiler-resolved calls, types, inheritance, macros, runtime behavior, or semantic ownership.
|
||||||
|
- C++ compilation-database commands are inert and never executed.
|
||||||
|
- C++ manifest include discovery uses Tree-sitter and makes no zero-warm-parser claim.
|
||||||
|
- No WorldForge, ScrapeStation, legacy repository, production binding, language distribution,
|
||||||
|
self-hosting, tag, or release was changed.
|
||||||
|
|
||||||
|
### Next gate
|
||||||
|
|
||||||
|
Milestone 5 remains directional. Create a new active-slice contract before stabilization,
|
||||||
|
versioning, tagging, or publication.
|
||||||
|
|
||||||
## DocForge2 Milestone 3 independent projections
|
## DocForge2 Milestone 3 independent projections
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|
|
||||||
|
|
@ -42,6 +42,14 @@ make benchmark-m3
|
||||||
make benchmark-m3-full
|
make benchmark-m3-full
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Run the Milestone 4 adapter SDK smoke, maintained full, and fresh-wheel adoption gates:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make benchmark-m4-smoke
|
||||||
|
make benchmark-m4-full
|
||||||
|
make adoption-m4
|
||||||
|
```
|
||||||
|
|
||||||
The benchmark creates canonical sources, derived state, changesets, rendered output, and caches
|
The benchmark creates canonical sources, derived state, changesets, rendered output, and caches
|
||||||
only in a disposable temporary directory. It does not read another project, self-host DocForge, or
|
only in a disposable temporary directory. It does not read another project, self-host DocForge, or
|
||||||
mutate repository content.
|
mutate repository content.
|
||||||
|
|
@ -71,6 +79,13 @@ sweeps, add/change/delete/reorder equivalence, portable publication, receipt-onl
|
||||||
memory, detached worker peak memory, and response size. Its interpretation is in
|
memory, detached worker peak memory, and response size. Its interpretation is in
|
||||||
[`docs/MILESTONE_3_BASELINE.md`](../docs/MILESTONE_3_BASELINE.md).
|
[`docs/MILESTONE_3_BASELINE.md`](../docs/MILESTONE_3_BASELINE.md).
|
||||||
|
|
||||||
|
`milestone4-2026-07-29.json` is the clean-tree adapter SDK and reference-Python baseline captured
|
||||||
|
from commit `95271dcf2e48045b9d3aed9b9ea09c7fc155692c`. It measures cold and warm
|
||||||
|
incremental builds, complete/incremental graph-plus-Logic equivalence, corrupt extraction-cache
|
||||||
|
recovery, corrupt-index recovery, parser and extraction counters, traced and process memory, and
|
||||||
|
bounded response size. Its interpretation is in
|
||||||
|
[`docs/MILESTONE_4_BASELINE.md`](../docs/MILESTONE_4_BASELINE.md).
|
||||||
|
|
||||||
The generic fixtures expose whole-source and projection scaling. They do not replace incremental
|
The generic fixtures expose whole-source and projection scaling. They do not replace incremental
|
||||||
adapter equivalence tests. Milestone 3's portable fixture contains 1,000 Nodes/Flow/Web nodes and
|
adapter equivalence tests. Milestone 3's portable fixture contains 1,000 Nodes/Flow/Web nodes and
|
||||||
999 edges; portable version 1 deliberately excludes Logic.
|
999 edges; portable version 1 deliberately excludes Logic.
|
||||||
|
|
@ -89,4 +104,5 @@ backlinks, both traversal directions, paged context, render receipt states, and
|
||||||
freshness. The reported p95 uses the nearest-rank method; with ten samples it is the maximum.
|
freshness. The reported p95 uses the nearest-rank method; with ten samples it is the maximum.
|
||||||
`process_peak_rss_kib` is the cumulative main-process `RUSAGE_SELF` high-water mark and excludes the
|
`process_peak_rss_kib` is the cumulative main-process `RUSAGE_SELF` high-water mark and excludes the
|
||||||
detached viewer worker. It is diagnostic and not operation-local. Milestone 3 memory gates use
|
detached viewer worker. It is diagnostic and not operation-local. Milestone 3 memory gates use
|
||||||
per-operation `tracemalloc` peaks and detached worker receipt peaks instead.
|
per-operation `tracemalloc` peaks and detached worker receipt peaks instead. Milestone 4 uses
|
||||||
|
per-operation `tracemalloc` peaks and a cumulative benchmark-process high-water gate.
|
||||||
|
|
|
||||||
205
benchmarks/milestone4-2026-07-29.json
Normal file
205
benchmarks/milestone4-2026-07-29.json
Normal file
|
|
@ -0,0 +1,205 @@
|
||||||
|
{
|
||||||
|
"benchmark": "docforge2_milestone4",
|
||||||
|
"environment": {
|
||||||
|
"implementation": "CPython",
|
||||||
|
"machine": "x86_64",
|
||||||
|
"platform": "Linux-7.1.3-200.nobara.fc44.x86_64-x86_64-with-glibc2.43",
|
||||||
|
"python": "3.14.6"
|
||||||
|
},
|
||||||
|
"evidence": {
|
||||||
|
"cache_recovery_exact": true,
|
||||||
|
"complete_assembly_hash": "2888182ab765fbffe3ba873c1613345640e8e6d89be74ddfca7452c0a5056345",
|
||||||
|
"edge_count": 1001,
|
||||||
|
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
|
||||||
|
"full_incremental_graph_and_logic_exact": true,
|
||||||
|
"index_recovery_exact": true,
|
||||||
|
"logic_edge_count": 2338,
|
||||||
|
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
|
||||||
|
"logic_node_count": 2338,
|
||||||
|
"logic_projection_count": 334,
|
||||||
|
"node_count": 1002,
|
||||||
|
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
|
||||||
|
"source_count": 334,
|
||||||
|
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
|
||||||
|
"warm_zero_ast_parse": true,
|
||||||
|
"warm_zero_source_extraction": true
|
||||||
|
},
|
||||||
|
"evidence_sha256": "4a8db461a311b5df4abd5aa00063e9a347d8b9dba19b30e35684f561d5271549",
|
||||||
|
"fixture": {
|
||||||
|
"expected_node_count": 1002,
|
||||||
|
"kind": "synthetic_python_reference_adapter",
|
||||||
|
"nodes_per_source": 3,
|
||||||
|
"source_count": 334
|
||||||
|
},
|
||||||
|
"memory": {
|
||||||
|
"process_peak_bytes": 78798848,
|
||||||
|
"process_peak_limit_bytes": 536870912
|
||||||
|
},
|
||||||
|
"method": {
|
||||||
|
"ast_instrumentation": "temporary counter around stdlib ast.parse",
|
||||||
|
"clock": "time.perf_counter_ns",
|
||||||
|
"extraction_instrumentation": "reference adapter extract_source entry counter",
|
||||||
|
"full_mode_source_requirement": 334,
|
||||||
|
"in_process_peak_memory": "tracemalloc per measured invocation",
|
||||||
|
"process_peak_memory": "resource.getrusage(RUSAGE_SELF).ru_maxrss",
|
||||||
|
"response_size": "UTF-8 bytes of compact sorted JSON",
|
||||||
|
"samples": 3,
|
||||||
|
"threshold_basis": "Regression tripwires leave several times the Milestone 3 1,000-node allowances for AST, Logic, extraction-cache, and SQLite work."
|
||||||
|
},
|
||||||
|
"mode": "full",
|
||||||
|
"operations": {
|
||||||
|
"cold_incremental_build": {
|
||||||
|
"max_ms": 1599.76,
|
||||||
|
"maximum_response_bytes": 14580,
|
||||||
|
"maximum_traced_peak_bytes": 68573540,
|
||||||
|
"median_ms": 1599.76,
|
||||||
|
"min_ms": 1599.76,
|
||||||
|
"p95_limit_ms": 30000.0,
|
||||||
|
"p95_ms": 1599.76,
|
||||||
|
"response_limit_bytes": 262144,
|
||||||
|
"samples": 1,
|
||||||
|
"stable_result": {
|
||||||
|
"ast_parse_calls": 668,
|
||||||
|
"cache_hits": 0,
|
||||||
|
"deleted_sources": 0,
|
||||||
|
"edge_count": 1001,
|
||||||
|
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
|
||||||
|
"extraction_calls": 334,
|
||||||
|
"invalidated_sources": 334,
|
||||||
|
"logic_edge_count": 2338,
|
||||||
|
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
|
||||||
|
"logic_node_count": 2338,
|
||||||
|
"logic_projection_count": 334,
|
||||||
|
"node_count": 1002,
|
||||||
|
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
|
||||||
|
"reparsed_sources": 334,
|
||||||
|
"revision": "30ef23aa4106",
|
||||||
|
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
|
||||||
|
"status": "ok",
|
||||||
|
"total_sources": 334
|
||||||
|
},
|
||||||
|
"traced_peak_limit_bytes": 268435456
|
||||||
|
},
|
||||||
|
"corrupt_extraction_cache_recovery": {
|
||||||
|
"max_ms": 1695.406,
|
||||||
|
"maximum_response_bytes": 14580,
|
||||||
|
"maximum_traced_peak_bytes": 69776923,
|
||||||
|
"median_ms": 1695.406,
|
||||||
|
"min_ms": 1695.406,
|
||||||
|
"p95_limit_ms": 30000.0,
|
||||||
|
"p95_ms": 1695.406,
|
||||||
|
"response_limit_bytes": 262144,
|
||||||
|
"samples": 1,
|
||||||
|
"stable_result": {
|
||||||
|
"ast_parse_calls": 668,
|
||||||
|
"cache_hits": 0,
|
||||||
|
"deleted_sources": 0,
|
||||||
|
"edge_count": 1001,
|
||||||
|
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
|
||||||
|
"extraction_calls": 334,
|
||||||
|
"invalidated_sources": 334,
|
||||||
|
"logic_edge_count": 2338,
|
||||||
|
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
|
||||||
|
"logic_node_count": 2338,
|
||||||
|
"logic_projection_count": 334,
|
||||||
|
"node_count": 1002,
|
||||||
|
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
|
||||||
|
"reparsed_sources": 334,
|
||||||
|
"revision": "30ef23aa4106",
|
||||||
|
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
|
||||||
|
"status": "ok",
|
||||||
|
"total_sources": 334
|
||||||
|
},
|
||||||
|
"traced_peak_limit_bytes": 268435456
|
||||||
|
},
|
||||||
|
"corrupt_index_recovery": {
|
||||||
|
"max_ms": 1707.879,
|
||||||
|
"maximum_response_bytes": 14779,
|
||||||
|
"maximum_traced_peak_bytes": 68561924,
|
||||||
|
"median_ms": 1707.879,
|
||||||
|
"min_ms": 1707.879,
|
||||||
|
"p95_limit_ms": 30000.0,
|
||||||
|
"p95_ms": 1707.879,
|
||||||
|
"response_limit_bytes": 262144,
|
||||||
|
"samples": 1,
|
||||||
|
"stable_result": {
|
||||||
|
"action": "rebuilt",
|
||||||
|
"ast_parse_calls": 0,
|
||||||
|
"cache_hits": 334,
|
||||||
|
"edge_count": 1001,
|
||||||
|
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
|
||||||
|
"extraction_calls": 0,
|
||||||
|
"initial_error_code": "invalid_index",
|
||||||
|
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
|
||||||
|
"logic_projection_count": 334,
|
||||||
|
"node_count": 1002,
|
||||||
|
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
|
||||||
|
"reparsed_sources": 0,
|
||||||
|
"revision": "30ef23aa4106",
|
||||||
|
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
|
||||||
|
"status": "ok"
|
||||||
|
},
|
||||||
|
"traced_peak_limit_bytes": 268435456
|
||||||
|
},
|
||||||
|
"full_incremental_equivalence": {
|
||||||
|
"max_ms": 1897.919,
|
||||||
|
"maximum_response_bytes": 230,
|
||||||
|
"maximum_traced_peak_bytes": 64305919,
|
||||||
|
"median_ms": 1897.919,
|
||||||
|
"min_ms": 1897.919,
|
||||||
|
"p95_limit_ms": 30000.0,
|
||||||
|
"p95_ms": 1897.919,
|
||||||
|
"response_limit_bytes": 262144,
|
||||||
|
"samples": 1,
|
||||||
|
"stable_result": {
|
||||||
|
"ast_parse_calls": 1336,
|
||||||
|
"edge_count": 1001,
|
||||||
|
"extraction_calls": 668,
|
||||||
|
"logic_projection_count": 334,
|
||||||
|
"node_count": 1002,
|
||||||
|
"project_id": "milestone4-python-benchmark",
|
||||||
|
"revision": "30ef23aa4106",
|
||||||
|
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
|
||||||
|
"status": "ok"
|
||||||
|
},
|
||||||
|
"traced_peak_limit_bytes": 268435456
|
||||||
|
},
|
||||||
|
"warm_incremental_build": {
|
||||||
|
"max_ms": 1265.387,
|
||||||
|
"maximum_response_bytes": 14578,
|
||||||
|
"maximum_traced_peak_bytes": 69997166,
|
||||||
|
"median_ms": 1244.427,
|
||||||
|
"min_ms": 1239.941,
|
||||||
|
"p95_limit_ms": 20000.0,
|
||||||
|
"p95_ms": 1265.387,
|
||||||
|
"response_limit_bytes": 262144,
|
||||||
|
"samples": 3,
|
||||||
|
"stable_result": {
|
||||||
|
"ast_parse_calls": 0,
|
||||||
|
"cache_hits": 334,
|
||||||
|
"deleted_sources": 0,
|
||||||
|
"edge_count": 1001,
|
||||||
|
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
|
||||||
|
"extraction_calls": 0,
|
||||||
|
"invalidated_sources": 0,
|
||||||
|
"logic_edge_count": 2338,
|
||||||
|
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
|
||||||
|
"logic_node_count": 2338,
|
||||||
|
"logic_projection_count": 334,
|
||||||
|
"node_count": 1002,
|
||||||
|
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
|
||||||
|
"reparsed_sources": 0,
|
||||||
|
"revision": "30ef23aa4106",
|
||||||
|
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
|
||||||
|
"status": "ok",
|
||||||
|
"total_sources": 334
|
||||||
|
},
|
||||||
|
"traced_peak_limit_bytes": 268435456
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"schema_version": 1,
|
||||||
|
"source": {
|
||||||
|
"dirty": false,
|
||||||
|
"revision": "95271dcf2e48045b9d3aed9b9ea09c7fc155692c"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -8,6 +8,30 @@ Use this guide after the repository assessment in
|
||||||
[Project Onboarding](PROJECT_ONBOARDING.md). The onboarding checklist decides whether an adapter
|
[Project Onboarding](PROJECT_ONBOARDING.md). The onboarding checklist decides whether an adapter
|
||||||
is needed. This guide defines how to build and prove one.
|
is needed. This guide defines how to build and prove one.
|
||||||
|
|
||||||
|
## Public authoring surface
|
||||||
|
|
||||||
|
Adapter authors should import typed contracts, validation helpers, graph models, and the
|
||||||
|
conformance helper from `docforge.adapter_sdk`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from docforge.adapter_sdk import (
|
||||||
|
AdapterAssembly,
|
||||||
|
AdapterImplementation,
|
||||||
|
AdapterManifest,
|
||||||
|
AdapterProject,
|
||||||
|
AdapterProjectSettings,
|
||||||
|
AdapterProjection,
|
||||||
|
AdapterSource,
|
||||||
|
AdapterSourceProjection,
|
||||||
|
LogicProjection,
|
||||||
|
verify_adapter_conformance,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Language-specific modules under `docforge.adapters` are repository reference implementations, not
|
||||||
|
the general authoring namespace. Their exact, deliberately narrow behavior is documented in
|
||||||
|
[Reference Adapters](REFERENCE_ADAPTERS.md).
|
||||||
|
|
||||||
## Required outcome
|
## Required outcome
|
||||||
|
|
||||||
A production adapter must provide one reproducible public graph from authoritative project
|
A production adapter must provide one reproducible public graph from authoritative project
|
||||||
|
|
@ -51,11 +75,39 @@ class MyAdapter:
|
||||||
manifest: AdapterManifest,
|
manifest: AdapterManifest,
|
||||||
contributions: tuple[AdapterSourceProjection, ...],
|
contributions: tuple[AdapterSourceProjection, ...],
|
||||||
) -> AdapterAssembly: ...
|
) -> AdapterAssembly: ...
|
||||||
|
def load_complete_assembly(self) -> AdapterAssembly: ...
|
||||||
def load_projection(self) -> AdapterProjection: ...
|
def load_projection(self) -> AdapterProjection: ...
|
||||||
```
|
```
|
||||||
|
|
||||||
`assemble_projection()` is optional only when extraction units already have disjoint ownership.
|
`assemble_projection()` is optional only when extraction units already have disjoint ownership.
|
||||||
`load_projection()` is always required. It is the clean rebuild and equivalence oracle.
|
`load_projection()` is always required. It is the clean primary-graph compatibility oracle.
|
||||||
|
`load_complete_assembly()` supplies the cache-independent complete graph-plus-Logic oracle. It is
|
||||||
|
required when incremental contributions publish Logic; without it, DocForge cannot prove Logic
|
||||||
|
parity.
|
||||||
|
|
||||||
|
## What the conformance helper proves
|
||||||
|
|
||||||
|
Run the public helper against a fresh confined cache root:
|
||||||
|
|
||||||
|
```python
|
||||||
|
report = verify_adapter_conformance(
|
||||||
|
MyAdapter(project_root),
|
||||||
|
cache_root=project_root / ".docforge" / "cache" / "adapter-conformance",
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
`verify_adapter_conformance()` proves:
|
||||||
|
|
||||||
|
- two repeated complete assemblies are exactly deterministic, including Logic;
|
||||||
|
- when `load_complete_assembly()` exists, its primary graph exactly matches
|
||||||
|
`load_projection()`; and
|
||||||
|
- when the incremental methods exist, the assembled incremental graph and Logic exactly match the
|
||||||
|
independent complete oracle.
|
||||||
|
|
||||||
|
The report records stable identity, counts, and an assembly hash. This helper does not by itself
|
||||||
|
prove process confinement, implementation restart behavior, corrupt-cache recovery, warm zero
|
||||||
|
parsing, retrieval, or every case in the proof matrix below. Keep those as separate focused and
|
||||||
|
integration tests.
|
||||||
|
|
||||||
## Step 1: define authority before parsing
|
## Step 1: define authority before parsing
|
||||||
|
|
||||||
|
|
@ -324,6 +376,25 @@ The manifest remains a current source snapshot, not a Git-index snapshot. A Git-
|
||||||
omit a deleted source whether its deletion is unstaged or staged. Staging is never a required
|
omit a deleted source whether its deletion is unstaged or staged. Staging is never a required
|
||||||
DocForge synchronization step.
|
DocForge synchronization step.
|
||||||
|
|
||||||
|
### Current aggregate bounds
|
||||||
|
|
||||||
|
DocForge limits one extraction-cache generation to 10,000 source contributions and 64,000,000
|
||||||
|
encoded bytes. Malformed, incompatible, missing, oversized, or unsafe cache data is treated as a
|
||||||
|
cache miss.
|
||||||
|
|
||||||
|
The assembled graph is bounded by the effective project `Limits.max_nodes`. When adapter settings
|
||||||
|
do not override limits, DocForge selects at least 10,000 nodes and raises that bound to the
|
||||||
|
manifest's `estimated_nodes` when larger. Aggregate assembly ceilings are:
|
||||||
|
|
||||||
|
- nodes: `max_nodes`;
|
||||||
|
- relationships: `max_nodes * 32`;
|
||||||
|
- Logic nodes across all functions: `max_nodes * 32`; and
|
||||||
|
- Logic edges across all functions: `max_nodes * 64`.
|
||||||
|
|
||||||
|
Adapters should set realistic limits and fail before retaining unbounded frontend evidence. The
|
||||||
|
cache and assembly bounds do not replace each adapter's own bounded source, command, parser, or
|
||||||
|
per-function limits.
|
||||||
|
|
||||||
## Step 9: keep the complete path independent
|
## Step 9: keep the complete path independent
|
||||||
|
|
||||||
The full rebuild must not read the incremental extraction cache. Otherwise equivalence compares
|
The full rebuild must not read the incremental extraction cache. Otherwise equivalence compares
|
||||||
|
|
@ -470,7 +541,8 @@ For every such change:
|
||||||
- [ ] Two independent complete builds match exactly.
|
- [ ] Two independent complete builds match exactly.
|
||||||
- [ ] Incremental extraction uses authoritative dependencies.
|
- [ ] Incremental extraction uses authoritative dependencies.
|
||||||
- [ ] The complete oracle is independent of the cache.
|
- [ ] The complete oracle is independent of the cache.
|
||||||
- [ ] Complete and incremental public projections match exactly.
|
- [ ] Complete and incremental public graph and Logic projections match exactly.
|
||||||
|
- [ ] `verify_adapter_conformance()` passes, and the separate proof-matrix cases also pass.
|
||||||
- [ ] Corrupt, missing, and interrupted cache cases fail safely.
|
- [ ] Corrupt, missing, and interrupted cache cases fail safely.
|
||||||
- [ ] Session composition and family isolation are proven.
|
- [ ] Session composition and family isolation are proven.
|
||||||
- [ ] Viewer, query, context, and Logic retrieval are proven.
|
- [ ] Viewer, query, context, and Logic retrieval are proven.
|
||||||
|
|
|
||||||
105
docs/AGENT_INTEGRATION.md
Normal file
105
docs/AGENT_INTEGRATION.md
Normal file
|
|
@ -0,0 +1,105 @@
|
||||||
|
# Agent integration
|
||||||
|
|
||||||
|
DocForge generates deterministic, project-bound MCP client fragments for Codex, Claude, and
|
||||||
|
OpenClaw. Generic projects and custom adapters use different configuration APIs, but both produce
|
||||||
|
fixed standard-input/output bindings with empty generated environments and explicit capability
|
||||||
|
policy.
|
||||||
|
|
||||||
|
## Fixed reference binding
|
||||||
|
|
||||||
|
Configure one of the in-repository adapters with
|
||||||
|
`.docforge/reference-adapter.toml` as described in
|
||||||
|
[Reference Adapters](REFERENCE_ADAPTERS.md), then start:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -I -m docforge.reference_mcp \
|
||||||
|
--project-root /absolute/path/to/project \
|
||||||
|
--capability-mode read
|
||||||
|
```
|
||||||
|
|
||||||
|
`docforge.reference_mcp` is a fixed trusted module in the installed DocForge distribution. It
|
||||||
|
selects a provider only from the validated reference configuration and reports binding metadata
|
||||||
|
containing:
|
||||||
|
|
||||||
|
- `server_module = "docforge.reference_mcp"`;
|
||||||
|
- `adapter_mode = "reference"`;
|
||||||
|
- the selected `reference_language`; and
|
||||||
|
- the exact `reference_config_hash`.
|
||||||
|
|
||||||
|
The reference binding registers exactly the 21 read tools listed in
|
||||||
|
[MCP Boundary](MCP_CONTRACT.md#read-tools). It has no proposal or canonical-application surface.
|
||||||
|
|
||||||
|
## Generate a reference client fragment
|
||||||
|
|
||||||
|
The public launcher and generator APIs are:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from docforge.adapter_launcher import AdapterLauncherV1
|
||||||
|
from docforge.client_config import generate_adapter_client_configuration
|
||||||
|
from docforge.reference_mcp import REFERENCE_MCP_MODULE, create_reference_project
|
||||||
|
|
||||||
|
root = Path("/absolute/path/to/project")
|
||||||
|
project = create_reference_project(root)
|
||||||
|
launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE)
|
||||||
|
|
||||||
|
plan = generate_adapter_client_configuration(
|
||||||
|
project,
|
||||||
|
launcher,
|
||||||
|
"codex", # "codex", "claude", or "openclaw"
|
||||||
|
capability_mode="read",
|
||||||
|
)
|
||||||
|
print(plan["artifact"]["content"])
|
||||||
|
```
|
||||||
|
|
||||||
|
Pass `output=Path(...)` only when the caller has selected an exact destination. Publication is an
|
||||||
|
atomic create-or-exact-match operation; it does not merge or replace different existing content.
|
||||||
|
The result includes the launcher, source-availability, project, policy, artifact, and configuration
|
||||||
|
hashes needed to inspect the binding before use.
|
||||||
|
|
||||||
|
The fixed reference module accepts read mode only. Do not request proposal or application mode for
|
||||||
|
it.
|
||||||
|
|
||||||
|
## Custom project-owned adapter launchers
|
||||||
|
|
||||||
|
Generic `docforge configure` intentionally refuses a custom adapter project. Construct the
|
||||||
|
project-owned `ProjectService`, then use `AdapterLauncherV1` and
|
||||||
|
`generate_adapter_client_configuration()` as the custom-adapter route.
|
||||||
|
|
||||||
|
`AdapterLauncherV1` schema version 1 binds:
|
||||||
|
|
||||||
|
- one project ID, canonical absolute project root, adapter identity, and descriptor hash;
|
||||||
|
- `entry_point = "python-module"`; and
|
||||||
|
- one installed project-owned top-level Python module.
|
||||||
|
|
||||||
|
The only trusted dotted-module exception is the fixed `docforge.reference_mcp` binding. A
|
||||||
|
project-owned module is resolved through isolated Python without importing or executing it during
|
||||||
|
the probe, and it must resolve to one canonical regular `.py` file inside the project root.
|
||||||
|
|
||||||
|
The launcher contract has no arbitrary command, command arguments, shell string, working
|
||||||
|
directory, environment, callable selector, discovery rule, or module-reload mechanism. Generated
|
||||||
|
bindings invoke the current Python executable as `python -I -m <module>` with only DocForge's
|
||||||
|
validated project, capability, render-policy, authority, and no-AST options. Source identity and
|
||||||
|
the project binding are revalidated before publication; drift fails closed.
|
||||||
|
|
||||||
|
Proposal and application modes are available only to a custom module that implements those fixed
|
||||||
|
server arguments and only when the project descriptor declares the matching writer and canonical
|
||||||
|
applier authority. Generating a mode does not manufacture that authority.
|
||||||
|
|
||||||
|
## Session workflow
|
||||||
|
|
||||||
|
After registering the generated fragment in the selected client:
|
||||||
|
|
||||||
|
1. Start a new MCP process or client session.
|
||||||
|
2. Call `docforge_bootstrap`.
|
||||||
|
3. Verify project ID, root fingerprint, adapter identity, revision, source hash, binding metadata,
|
||||||
|
effective policy, and registered tools.
|
||||||
|
4. Retrieve exact or bounded project evidence through the fixed read tools.
|
||||||
|
5. Restart the process if `adapter_restart_required` reports an implementation or configuration
|
||||||
|
change.
|
||||||
|
|
||||||
|
Document text returned by DocForge is untrusted project content. It never overrides client, user,
|
||||||
|
or project authority. See [MCP Boundary](MCP_CONTRACT.md) for synchronization, pagination,
|
||||||
|
retrieval, proposal, and application rules, and
|
||||||
|
[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) before selecting `no_ast=True`.
|
||||||
|
|
@ -31,16 +31,31 @@ model, index, rendering, application, and MCP factory names imported from these
|
||||||
remain supported:
|
remain supported:
|
||||||
|
|
||||||
- `docforge.adapter_contract`
|
- `docforge.adapter_contract`
|
||||||
|
- `docforge.adapter_sdk`
|
||||||
|
- `docforge.adapter_launcher`
|
||||||
- `docforge.application`
|
- `docforge.application`
|
||||||
|
- `docforge.client_config`
|
||||||
- `docforge.index`
|
- `docforge.index`
|
||||||
- `docforge.mcp_server`
|
- `docforge.mcp_server`
|
||||||
- `docforge.models`
|
- `docforge.models`
|
||||||
- `docforge.policy`
|
- `docforge.policy`
|
||||||
- `docforge.render_contract`
|
- `docforge.render_contract`
|
||||||
|
- `docforge.reference_config`
|
||||||
|
- `docforge.reference_mcp`
|
||||||
|
|
||||||
Names beginning with an underscore are implementation details. New public names may be added
|
Names beginning with an underscore are implementation details. New public names may be added
|
||||||
without breaking this contract.
|
without breaking this contract.
|
||||||
|
|
||||||
|
The repository Python, JavaScript/TypeScript, and C++ adapters are supported reference
|
||||||
|
implementations. Their documented configuration, evidence limits, and unsupported-fact reports are
|
||||||
|
compatibility surfaces; their internal parser helpers are not adapter-authoring imports.
|
||||||
|
|
||||||
|
Milestone 4 adds three version-1 schemas without changing existing descriptor or result schemas:
|
||||||
|
|
||||||
|
- `schemas/adapter-launcher.schema.json`
|
||||||
|
- `schemas/adapter-client-configuration.schema.json`
|
||||||
|
- `schemas/reference-adapter.schema.json`
|
||||||
|
|
||||||
## CLI and MCP surfaces
|
## CLI and MCP surfaces
|
||||||
|
|
||||||
Existing `docforge` command names and arguments remain supported. Existing `docforge-mcp` tool
|
Existing `docforge` command names and arguments remain supported. Existing `docforge-mcp` tool
|
||||||
|
|
@ -233,6 +248,11 @@ Milestone 3 adds `ManualRenderPlanV1`, `GraphViewPlanV1`, projection package and
|
||||||
and projection policy version 2. These are additive submodule and schema contracts. They do not
|
and projection policy version 2. These are additive submodule and schema contracts. They do not
|
||||||
change the legacy task-context, adapter, changeset, or effective-policy contracts described above.
|
change the legacy task-context, adapter, changeset, or effective-policy contracts described above.
|
||||||
|
|
||||||
|
Milestone 4 adds `docforge.adapter_sdk`, the fixed reference configuration and read-only MCP
|
||||||
|
binding, immutable adapter launchers, and generated adapter client fragments. A legacy adapter with
|
||||||
|
only `load_projection()` remains first-class and need not adopt incremental extraction, Logic, a
|
||||||
|
reference configuration, or launcher metadata.
|
||||||
|
|
||||||
## Safety boundary
|
## Safety boundary
|
||||||
|
|
||||||
DocForge remains bound to one explicit project root. It rejects absolute paths, root escapes, and
|
DocForge remains bound to one explicit project root. It rejects absolute paths, root escapes, and
|
||||||
|
|
@ -245,8 +265,9 @@ publication, or project switching.
|
||||||
Milestone 0 records rather than redesigns these areas:
|
Milestone 0 records rather than redesigns these areas:
|
||||||
|
|
||||||
- Generic warm reads still repeat whole-project discovery, parsing, and validation.
|
- Generic warm reads still repeat whole-project discovery, parsing, and validation.
|
||||||
- Tree-sitter and the JavaScript and C++ grammars remain mandatory installation dependencies even
|
- The base wheel intentionally omits Tree-sitter. JavaScript, TypeScript, and C++ syntax evidence
|
||||||
when their runtime modules are unused.
|
requires the matching `docforge[javascript]`, `docforge[typescript]`, or `docforge[cpp]` extra.
|
||||||
|
Python reference evidence uses the standard library and remains available in the base wheel.
|
||||||
- Several version strings and defaults remain duplicated.
|
- Several version strings and defaults remain duplicated.
|
||||||
- One individually oversized context entry is represented as explicit bounded omission evidence;
|
- One individually oversized context entry is represented as explicit bounded omission evidence;
|
||||||
callers use targeted retrieval for that node.
|
callers use targeted retrieval for that node.
|
||||||
|
|
|
||||||
|
|
@ -29,6 +29,9 @@ commit when Git is available; it cannot change repository state.
|
||||||
- Projection package: `schemas/projection-package.schema.json`, version 1.
|
- Projection package: `schemas/projection-package.schema.json`, version 1.
|
||||||
- Projection receipt: `schemas/projection-receipt.schema.json`, version 1.
|
- Projection receipt: `schemas/projection-receipt.schema.json`, version 1.
|
||||||
- Independent projection policy: `schemas/projection-policy.schema.json`, version 2.
|
- Independent projection policy: `schemas/projection-policy.schema.json`, version 2.
|
||||||
|
- Adapter launcher: `schemas/adapter-launcher.schema.json`, version 1.
|
||||||
|
- Adapter client configuration: `schemas/adapter-client-configuration.schema.json`, version 1.
|
||||||
|
- Reference adapter configuration: `schemas/reference-adapter.schema.json`, version 1.
|
||||||
- Index schema: version 3, disposable and reproducible.
|
- Index schema: version 3, disposable and reproducible.
|
||||||
- Index attestation: schema version 1, disposable and reproducible.
|
- Index attestation: schema version 1, disposable and reproducible.
|
||||||
- Core, CLI, and MCP server: version 1.3.0.dev0.
|
- Core, CLI, and MCP server: version 1.3.0.dev0.
|
||||||
|
|
@ -103,6 +106,42 @@ complete projection, opens SQLite, starts MCP, executes the configured command,
|
||||||
builds, renders, starts a viewer, or writes configuration. Unprovable client behavior is a warning,
|
builds, renders, starts a viewer, or writes configuration. Unprovable client behavior is a warning,
|
||||||
not an invented success.
|
not an invented success.
|
||||||
|
|
||||||
|
## Public adapter SDK and reference binding
|
||||||
|
|
||||||
|
`docforge.adapter_sdk` is the stable adapter-authoring import boundary. It exposes the typed
|
||||||
|
projection, manifest, source contribution, complete assembly, project wrapper, graph model, and
|
||||||
|
conformance contracts needed by an adapter without requiring authors to import core implementation
|
||||||
|
modules.
|
||||||
|
|
||||||
|
Complete evidence includes the primary graph and function Logic. An incremental adapter that
|
||||||
|
publishes Logic must implement `load_complete_assembly()` as an independent complete oracle.
|
||||||
|
`verify_adapter_conformance()` proves repeated complete determinism, equality between the complete
|
||||||
|
assembly and `load_projection()`, and exact complete/incremental graph-plus-Logic parity. Separate
|
||||||
|
tests remain responsible for confinement, restart behavior, no-AST behavior, cache recovery, and
|
||||||
|
retrieval.
|
||||||
|
|
||||||
|
Adapter assemblies are bounded before publication. Primary nodes use the descriptor `max_nodes`
|
||||||
|
limit. Primary edges, Logic nodes, and Logic edges use fixed deterministic multipliers over that
|
||||||
|
limit. Version-1 extraction caches are regular-file-only, bounded to 10,000 sources and
|
||||||
|
64,000,000 bytes, and are treated as misses when corrupt, oversized, foreign, or incompatible.
|
||||||
|
|
||||||
|
`.docforge/reference-adapter.toml` is a closed version-1 selection among `python`, `javascript`,
|
||||||
|
`typescript`, and `cpp`. It declares one project identity and explicit non-overlapping source
|
||||||
|
roots. C++ additionally requires a confined `compile_commands.json`. It cannot declare a command,
|
||||||
|
module, environment, writer, applier, or remote endpoint.
|
||||||
|
|
||||||
|
`python -m docforge.reference_mcp --project-root ROOT` constructs only the selected fixed
|
||||||
|
repository reference adapter and exposes the read surface. It never registers proposal or
|
||||||
|
application tools.
|
||||||
|
|
||||||
|
`AdapterLauncherV1` is an immutable project-bound Python-module declaration. It accepts no
|
||||||
|
arbitrary command, arguments, working directory, environment, discovery, callable selector, or
|
||||||
|
module reload. Custom launchers resolve one installed top-level module through isolated Python and
|
||||||
|
require its origin inside the project root. The fixed `docforge.reference_mcp` module is the only
|
||||||
|
trusted dotted exception. `generate_adapter_client_configuration()` binds generated Codex,
|
||||||
|
Claude, and OpenClaw fragments to that launcher, current source availability, effective policy,
|
||||||
|
descriptor, interpreter, and exact artifact bytes.
|
||||||
|
|
||||||
## Isolated proposal model
|
## Isolated proposal model
|
||||||
|
|
||||||
Create, update, move, and delete are ordered node operations inside an isolated changeset. Every
|
Create, update, move, and delete are ordered node operations inside an isolated changeset. Every
|
||||||
|
|
|
||||||
165
docs/CORE_CONCEPTS_AND_AUTHORITY.md
Normal file
165
docs/CORE_CONCEPTS_AND_AUTHORITY.md
Normal file
|
|
@ -0,0 +1,165 @@
|
||||||
|
# Core concepts and authority
|
||||||
|
|
||||||
|
DocForge is a project-bound knowledge compiler. It turns explicit canonical project facts into
|
||||||
|
validated graphs and bounded derived views without transferring authority to the index, an agent,
|
||||||
|
or a renderer.
|
||||||
|
|
||||||
|
## One project, one explicit root
|
||||||
|
|
||||||
|
Every operation is bound to one canonical real project directory. Paths in descriptors and adapter
|
||||||
|
configuration are project-relative and confined beneath that root. A CLI or MCP process does not
|
||||||
|
discover or switch projects after startup.
|
||||||
|
|
||||||
|
The project root determines:
|
||||||
|
|
||||||
|
- which descriptor and canonical sources may be read;
|
||||||
|
- where derived cache and changeset roots may exist;
|
||||||
|
- which project identity, revision, source hash, and generation appear in results;
|
||||||
|
- which writer, applier, rendering, and viewer policies can be selected.
|
||||||
|
|
||||||
|
Generated client fragments preserve that binding. They are machine-local configuration
|
||||||
|
projections, not portable project authority.
|
||||||
|
|
||||||
|
## Canonical facts and derived evidence
|
||||||
|
|
||||||
|
Canonical inputs own the facts:
|
||||||
|
|
||||||
|
- generic Markdown or TOML node sources;
|
||||||
|
- authority files named by a generic descriptor;
|
||||||
|
- an adapter's declared canonical sources and implementation boundary;
|
||||||
|
- `.docforge/project.toml` for a generic project;
|
||||||
|
- `.docforge/reference-adapter.toml` for a fixed reference integration.
|
||||||
|
|
||||||
|
Everything DocForge builds from those inputs is derived:
|
||||||
|
|
||||||
|
- SQLite indexes, attestations, generation receipts, and generation diffs;
|
||||||
|
- incremental extraction caches;
|
||||||
|
- task-context capsules and query responses;
|
||||||
|
- changeset previews;
|
||||||
|
- render plans, immutable packages, fragments, artifacts, and receipts;
|
||||||
|
- portable graph publications and live-viewer processes;
|
||||||
|
- generated Codex, Claude, and OpenClaw fragments.
|
||||||
|
|
||||||
|
Derived state may be discarded and rebuilt. A derived artifact can prove what it was bound to, but
|
||||||
|
it cannot override current canonical content.
|
||||||
|
|
||||||
|
## Nodes, relationships, and Logic
|
||||||
|
|
||||||
|
The primary graph contains nodes and directed relationships.
|
||||||
|
|
||||||
|
A node has a stable project-wide ID, title, family, authority, status, tags, summary, content,
|
||||||
|
source identity, and content hash. A generic Markdown file contains one node beneath a TOML
|
||||||
|
metadata block; a generic TOML source may contain multiple nodes. Adapter nodes use the same public
|
||||||
|
graph contract.
|
||||||
|
|
||||||
|
A relationship is an exact `(source, relation, target)` triple. The project descriptor defines the
|
||||||
|
allowed relation names. DocForge gives `depends_on` special acyclic validation, but it does not
|
||||||
|
invent domain meaning for a project's other relation names. Retrieval recognizes only a versioned
|
||||||
|
alias set for task planning and reports unknown allowed relations as `unclassified`.
|
||||||
|
|
||||||
|
Logic is deliberately separate. It is a lazy function-scoped control-flow projection owned by one
|
||||||
|
primary node. Decisions, actions, loops, convergence points, returns, and exceptions connect
|
||||||
|
through explicit branch edges. Logic does not add statement-level nodes to ordinary Nodes, Flow,
|
||||||
|
Web, search, or generation-diff results. Python, JavaScript, TypeScript, and C++ integrations may
|
||||||
|
publish Logic when their adapter contract supports it.
|
||||||
|
|
||||||
|
## Authority, status, family, and tags
|
||||||
|
|
||||||
|
These fields answer different questions:
|
||||||
|
|
||||||
|
- `authority` describes the role of the content. The generic vocabulary is `authoritative`,
|
||||||
|
`approved_plan`, `derived`, `proposal`, and `historical`.
|
||||||
|
- `status` is project-defined lifecycle state such as `current`, `active`, or `verified`.
|
||||||
|
- `family` is a project-defined content grouping used for filtering, profiles, rendering, and
|
||||||
|
writer permissions.
|
||||||
|
- `tags` are exact project labels for retrieval and presentation.
|
||||||
|
|
||||||
|
An `authoritative` node can still become stale; authority is not a freshness claim. A `derived`
|
||||||
|
node is still canonical if it is stored in a declared canonical source; the label describes its
|
||||||
|
role, not whether DocForge may silently regenerate it. Status and authority never grant an MCP
|
||||||
|
writer permission.
|
||||||
|
|
||||||
|
## Validation, synchronization, and generations
|
||||||
|
|
||||||
|
Validation loads the complete canonical graph and rejects unsafe paths, invalid source formats,
|
||||||
|
duplicate IDs, unresolved relationships, prohibited cycles, violated project limits, and
|
||||||
|
adapter-specific contract failures.
|
||||||
|
|
||||||
|
The disposable index is published atomically only after the complete graph and SQLite integrity
|
||||||
|
checks pass. Its attestation binds the whole index file. A successful replacement is the derived
|
||||||
|
publication commit point; later receipt-writing trouble is reported as degraded evidence rather
|
||||||
|
than as a false claim that the replacement failed.
|
||||||
|
|
||||||
|
MCP operations automatically synchronize derived state under a project lock before normal work.
|
||||||
|
A graph generation identifies one validated indexed snapshot. Generation-pinned retrieval and
|
||||||
|
rendering do not reopen mutable sources behind an older snapshot.
|
||||||
|
|
||||||
|
The latest generation diff is one bounded primary-graph transition, not a history database. It
|
||||||
|
contains no Logic details or source text.
|
||||||
|
|
||||||
|
## Complete and incremental adapters
|
||||||
|
|
||||||
|
`load_projection()` is the compatibility baseline and clean graph oracle. An incremental adapter
|
||||||
|
adds:
|
||||||
|
|
||||||
|
- `load_manifest()` for cheap project identity, source inventory, fingerprints, and dependencies;
|
||||||
|
- `extract_source()` for one cacheable source contribution;
|
||||||
|
- optionally `assemble_projection()` to normalize overlapping contributions.
|
||||||
|
|
||||||
|
When incremental contributions contain Logic, `load_complete_assembly()` supplies a
|
||||||
|
cache-independent complete graph-plus-Logic oracle. Warm cache behavior is an optimization, never a
|
||||||
|
different authority path. Corrupt or incompatible extraction caches are treated as misses, and a
|
||||||
|
clean complete build remains the equivalence and recovery boundary.
|
||||||
|
|
||||||
|
The public types, validators, and conformance helper are exported from `docforge.adapter_sdk`. See
|
||||||
|
the [Adapter authoring guide](ADAPTER_AUTHORING_GUIDE.md) and [Incremental
|
||||||
|
indexing](INCREMENTAL_INDEXING.md).
|
||||||
|
|
||||||
|
## Proposals are not canonical changes
|
||||||
|
|
||||||
|
A changeset is an isolated, ordered proposal over an exact canonical base. Each operation names
|
||||||
|
preconditions, and the final changeset has a content-derived hash. Validation projects the complete
|
||||||
|
resulting graph before application.
|
||||||
|
|
||||||
|
Canonical application requires:
|
||||||
|
|
||||||
|
1. a writer declared in the descriptor;
|
||||||
|
2. a process started with the matching proposal and application authority;
|
||||||
|
3. one explicit final changeset hash;
|
||||||
|
4. unchanged source, relationship, permission, and graph preconditions.
|
||||||
|
|
||||||
|
Application does not perform Git mutation, build, deployment, or publication. Until exact-hash
|
||||||
|
application succeeds, canonical project files remain unchanged.
|
||||||
|
|
||||||
|
## Three independent output projections
|
||||||
|
|
||||||
|
Manual rendering, portable graph publication, and the live viewer are separate:
|
||||||
|
|
||||||
|
- a manual is a declared derived HTML view over selected nodes;
|
||||||
|
- a portable graph is a content-addressed static Nodes, Flow, or Web artifact;
|
||||||
|
- the live viewer is a managed loopback process pinned to one validated index generation and can
|
||||||
|
request lazy Logic.
|
||||||
|
|
||||||
|
Their policies compose independently. Disabling one does not transfer its authority to another.
|
||||||
|
None is canonical documentation or a retrieval authority. See [Rendering and
|
||||||
|
visualization](RENDERING_AND_VISUALIZATION.md).
|
||||||
|
|
||||||
|
## Binding policy is not project truth
|
||||||
|
|
||||||
|
Capability mode, no-AST preservation, diagnostics, and projection modes describe one running
|
||||||
|
process or generated client binding. They do not rewrite the descriptor or canonical graph.
|
||||||
|
|
||||||
|
`--no-ast` forbids AST-family adapter evolution and Logic publication/retrieval for that binding.
|
||||||
|
It does not inspect parser implementation, sandbox the filesystem, or convert an existing
|
||||||
|
AST/Tree-sitter adapter into a no-AST adapter. Read [Policy precedence](POLICY_PRECEDENCE.md) and
|
||||||
|
[Legacy and no-AST operation](LEGACY_AND_NO_AST.md).
|
||||||
|
|
||||||
|
## Trust the narrowest evidence
|
||||||
|
|
||||||
|
DocForge reports stable IDs, hashes, generations, omissions, truncation, and provenance limits so a
|
||||||
|
consumer can distinguish proof from inference. A source inventory is not a semantic graph; a
|
||||||
|
syntax-level relationship is not compiler resolution; a configuration doctor is not a connection
|
||||||
|
test; a viewer snapshot is not continuous monitoring; and a benchmark is not a release.
|
||||||
|
|
||||||
|
Continue with the [Project descriptor](PROJECT_DESCRIPTOR.md), [New-project
|
||||||
|
quickstart](NEW_PROJECT_QUICKSTART.md), or [Core contract](CONTRACT.md).
|
||||||
|
|
@ -4,6 +4,10 @@ DocForge Release 1 adapters return one complete immutable projection. That contr
|
||||||
supported. The incremental compiler adds an opt-in source-scoped contract that avoids reparsing
|
supported. The incremental compiler adds an opt-in source-scoped contract that avoids reparsing
|
||||||
unchanged files while preserving the same validated, atomically published graph.
|
unchanged files while preserving the same validated, atomically published graph.
|
||||||
|
|
||||||
|
Import these contracts from the public `docforge.adapter_sdk` facade. See
|
||||||
|
[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for the preserved one-method contract and
|
||||||
|
[Reference Adapters](REFERENCE_ADAPTERS.md) for the four maintained implementations.
|
||||||
|
|
||||||
## Release 1 compatibility
|
## Release 1 compatibility
|
||||||
|
|
||||||
The incremental interface is additive:
|
The incremental interface is additive:
|
||||||
|
|
@ -34,18 +38,26 @@ Incremental indexing is an extraction optimization. It does not weaken publicati
|
||||||
An interrupted extraction never replaces the last validated SQLite index. A malformed,
|
An interrupted extraction never replaces the last validated SQLite index. A malformed,
|
||||||
incompatible, or missing cache is a cache miss, not a partial graph.
|
incompatible, or missing cache is a cache miss, not a partial graph.
|
||||||
|
|
||||||
|
One cache generation is bounded to 10,000 source contributions and 64,000,000 encoded bytes.
|
||||||
|
Aggregate graph and Logic assembly limits are described in the
|
||||||
|
[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md#current-aggregate-bounds).
|
||||||
|
|
||||||
## Adapter contract
|
## Adapter contract
|
||||||
|
|
||||||
An incremental loader implements all three methods:
|
An incremental loader implements the first, second, and fourth methods. It implements
|
||||||
|
`load_complete_assembly()` as well when it publishes Logic:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
class MyAdapter:
|
class MyAdapter:
|
||||||
def load_manifest(self) -> AdapterManifest: ...
|
def load_manifest(self) -> AdapterManifest: ...
|
||||||
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ...
|
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ...
|
||||||
|
def load_complete_assembly(self) -> AdapterAssembly: ...
|
||||||
def load_projection(self) -> AdapterProjection: ...
|
def load_projection(self) -> AdapterProjection: ...
|
||||||
```
|
```
|
||||||
|
|
||||||
`load_projection()` remains the deterministic full-rebuild fallback and equivalence oracle.
|
`load_projection()` remains the deterministic full-rebuild primary-graph oracle. An incremental
|
||||||
|
adapter that publishes Logic must additionally implement `load_complete_assembly()` as the
|
||||||
|
cache-independent complete graph-plus-Logic oracle.
|
||||||
|
|
||||||
Each `AdapterSource` declares:
|
Each `AdapterSource` declares:
|
||||||
|
|
||||||
|
|
@ -137,7 +149,8 @@ fresh process must import and validate the current adapter.
|
||||||
```
|
```
|
||||||
|
|
||||||
Adapters can call `AdapterProject.verify_incremental_equivalence()` in release and contract tests.
|
Adapters can call `AdapterProject.verify_incremental_equivalence()` in release and contract tests.
|
||||||
The check compares project identity, revision, source hash, nodes, and relationships against
|
The check compares project identity, revision, source hash, nodes, relationships, and Logic against
|
||||||
|
the independent complete assembly. It also requires the complete assembly's primary graph to match
|
||||||
`load_projection()`.
|
`load_projection()`.
|
||||||
|
|
||||||
## Manual changes and relationships
|
## Manual changes and relationships
|
||||||
|
|
@ -170,12 +183,29 @@ logic empty until they implement a language analyzer.
|
||||||
This boundary prevents thousands of boolean expressions and basic blocks from polluting Nodes,
|
This boundary prevents thousands of boolean expressions and basic blocks from polluting Nodes,
|
||||||
Flow, Web, ordinary search, or architectural traversal. The Logic tab and `docforge_get_logic`
|
Flow, Web, ordinary search, or architectural traversal. The Logic tab and `docforge_get_logic`
|
||||||
request one function-scoped projection on demand. The built-in analyzers cover Python,
|
request one function-scoped projection on demand. The built-in analyzers cover Python,
|
||||||
JavaScript, and C++. Python uses the standard-library AST. JavaScript and C++ share pinned
|
JavaScript, TypeScript, and C++. Python uses the standard-library AST. JavaScript, TypeScript, and
|
||||||
Tree-sitter infrastructure with thin language-aware control-flow profiles. Parsers run only while
|
C++ use distinct optional Tree-sitter grammars with thin language-aware control-flow profiles.
|
||||||
extracting a changed source contribution; ordinary graph reads do not load or execute them. A
|
Ordinary graph reads use stored projections and do not load or execute these parsers. A grammar
|
||||||
grammar alone supplies syntax, not control-flow meaning, so each new language still needs a small
|
alone supplies syntax, not control-flow meaning, so each new language still needs a profile for
|
||||||
semantic profile for its branch, loop, case, exception, and termination constructs. All analyzers
|
its branch, loop, case, exception, and termination constructs. All analyzers report possible
|
||||||
report possible static paths; they do not claim runtime branch outcomes.
|
static paths; they do not claim runtime branch outcomes.
|
||||||
|
|
||||||
|
## Manifest and warm-parser scope
|
||||||
|
|
||||||
|
Parser work is language-specific and must be measured at the correct boundary:
|
||||||
|
|
||||||
|
- Python manifest construction fingerprints source and tokenizes local imports without calling
|
||||||
|
`ast.parse`.
|
||||||
|
- JavaScript and TypeScript manifest construction lexes static relative module specifiers without
|
||||||
|
invoking their distinct Tree-sitter extraction parsers. Focused tests prove this behavior on an
|
||||||
|
unchanged warm build.
|
||||||
|
- The C++ reference manifest parses inventoried sources with `tree-sitter-cpp` to discover quoted
|
||||||
|
include dependencies. A warm C++ extraction-cache hit is not a zero-parser claim.
|
||||||
|
|
||||||
|
The maintained Python benchmark instruments the unchanged warm path and requires zero
|
||||||
|
`ast.parse` calls and zero `extract_source` calls. That exact zero-parser benchmark claim is Python
|
||||||
|
only. JavaScript and TypeScript retain focused parser-free-manifest tests; C++ deliberately does
|
||||||
|
not.
|
||||||
|
|
||||||
## Full rebuilds
|
## Full rebuilds
|
||||||
|
|
||||||
|
|
|
||||||
80
docs/LEGACY_AND_NO_AST.md
Normal file
80
docs/LEGACY_AND_NO_AST.md
Normal file
|
|
@ -0,0 +1,80 @@
|
||||||
|
# Legacy adapters and no-AST policy
|
||||||
|
|
||||||
|
DocForge preserves the original one-method adapter contract while offering incremental extraction,
|
||||||
|
complete graph-plus-Logic assemblies, and an independently selectable no-AST MCP policy. These are
|
||||||
|
separate compatibility boundaries.
|
||||||
|
|
||||||
|
## One-method adapters remain valid
|
||||||
|
|
||||||
|
An existing adapter that implements only:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def load_projection(self) -> AdapterProjection: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
continues to work. It receives the same projection validation, indexing, querying, visualization,
|
||||||
|
and MCP behavior as before. It does not receive extraction-cache speedups and publishes no Logic
|
||||||
|
through the complete assembly contract.
|
||||||
|
|
||||||
|
Incremental adoption is additive. Implement `load_manifest()` and `extract_source()` while keeping
|
||||||
|
`load_projection()` as the independent complete graph oracle. If incremental contributions publish
|
||||||
|
Logic, also implement `load_complete_assembly()` so complete and incremental graph-plus-Logic
|
||||||
|
output can be compared exactly. The public authoring surface is `docforge.adapter_sdk`; see the
|
||||||
|
[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md).
|
||||||
|
|
||||||
|
## What no-AST means
|
||||||
|
|
||||||
|
`--no-ast` is an immutable policy on one MCP process binding. It preserves the configured adapter
|
||||||
|
strategy but forbids AST, Tree-sitter, compiler-AST, and function-Logic evolution for that binding.
|
||||||
|
It also rejects nonempty Logic before index publication and rejects a pre-existing index that
|
||||||
|
contains Logic before reads or live visualization.
|
||||||
|
|
||||||
|
Start the generic server with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docforge-mcp --project-root /absolute/path/to/project --no-ast
|
||||||
|
```
|
||||||
|
|
||||||
|
Project-owned integrations select the same policy with
|
||||||
|
`create_project_server(..., no_ast=True)` or `create_read_only_server(..., no_ast=True)`.
|
||||||
|
`docforge_bootstrap` and `docforge_get_contract` report the effective policy, and
|
||||||
|
`docforge_get_logic` returns `adapter_policy_forbids_logic`.
|
||||||
|
|
||||||
|
Changing this policy requires a new process. It does not rewrite or reload an adapter in place.
|
||||||
|
|
||||||
|
## What no-AST does not mean
|
||||||
|
|
||||||
|
No-AST is not:
|
||||||
|
|
||||||
|
- an inspection mechanism that proves which parser an arbitrary adapter uses;
|
||||||
|
- a filesystem, Python-import, or process sandbox;
|
||||||
|
- a promise that the adapter performs no non-AST fingerprinting, dependency discovery, caching, or
|
||||||
|
complete-projection loading; or
|
||||||
|
- permission to change project files, adapter code, or the configured capability surface.
|
||||||
|
|
||||||
|
Repository permissions and project instructions remain responsible for processes that have direct
|
||||||
|
filesystem access. The MCP boundary still excludes shell execution, arbitrary file operations,
|
||||||
|
Git mutation, builds, deployment, publication, project switching, and cross-project retrieval.
|
||||||
|
|
||||||
|
## Reference adapters are not no-AST adapters
|
||||||
|
|
||||||
|
The built-in Python reference extracts syntax and Logic with the standard-library AST. The
|
||||||
|
JavaScript, TypeScript, and C++ references extract syntax and Logic with their optional
|
||||||
|
Tree-sitter grammars. Their normal projections contain Logic, so they must not be presented or
|
||||||
|
configured as no-AST adapters.
|
||||||
|
|
||||||
|
The narrower performance statements in [Reference Adapters](REFERENCE_ADAPTERS.md) concern
|
||||||
|
specific manifest and warm-cache paths. A parser-free manifest does not turn an adapter that
|
||||||
|
publishes AST-derived Logic into a no-AST adapter.
|
||||||
|
|
||||||
|
## Migration choices
|
||||||
|
|
||||||
|
Keep a one-method adapter when its complete rebuild is acceptable and no function Logic is needed.
|
||||||
|
Adopt incremental extraction when measured source parsing dominates synchronization. Adopt the
|
||||||
|
complete assembly contract when Logic is published or overlapping raw evidence needs deterministic
|
||||||
|
ownership.
|
||||||
|
|
||||||
|
Choose no-AST only when the project binding must forbid AST-derived publication and future adapter
|
||||||
|
upgrades of that kind. It is a binding policy, not a substitute for adapter confinement,
|
||||||
|
conformance tests, restart detection, cache recovery, or the full proof matrix in
|
||||||
|
[Incremental Adapter Indexing](INCREMENTAL_INDEXING.md).
|
||||||
|
|
@ -28,6 +28,30 @@ Bootstrap and contract results include `effective_policy` schema version 1 plus
|
||||||
registered surface and startup-bound proposal/application access. The project descriptor remains
|
registered surface and startup-bound proposal/application access. The project descriptor remains
|
||||||
schema version 1 and does not silently acquire machine-specific process policy.
|
schema version 1 and does not silently acquire machine-specific process policy.
|
||||||
|
|
||||||
|
## Fixed reference binding
|
||||||
|
|
||||||
|
`docforge.reference_mcp` is the fixed runnable binding for the in-repository Python, JavaScript,
|
||||||
|
TypeScript, and C++ reference adapters:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -I -m docforge.reference_mcp \
|
||||||
|
--project-root /absolute/project \
|
||||||
|
--capability-mode read
|
||||||
|
```
|
||||||
|
|
||||||
|
It loads only `.docforge/reference-adapter.toml`, selects only the fixed provider for the declared
|
||||||
|
language, and constructs the same project-bound read surface through `create_read_only_server()`.
|
||||||
|
Bootstrap binding metadata includes `server_module = "docforge.reference_mcp"`,
|
||||||
|
`adapter_mode = "reference"`, the selected `reference_language`, and the exact
|
||||||
|
`reference_config_hash`.
|
||||||
|
|
||||||
|
The reference command accepts read capability mode only. It registers all 21 read tools below and
|
||||||
|
no isolated proposal or canonical-application tools. The configuration cannot select a provider,
|
||||||
|
module, command, arguments, working directory, environment, or arbitrary discovery behavior. See
|
||||||
|
[Reference Adapters](REFERENCE_ADAPTERS.md) for language scope and
|
||||||
|
[Agent Integration](AGENT_INTEGRATION.md) for the immutable launcher and generated client
|
||||||
|
fragments.
|
||||||
|
|
||||||
## Read tools
|
## Read tools
|
||||||
|
|
||||||
- `docforge_bootstrap`
|
- `docforge_bootstrap`
|
||||||
|
|
@ -45,6 +69,8 @@ schema version 1 and does not silently acquire machine-specific process policy.
|
||||||
- `docforge_get_task_context`
|
- `docforge_get_task_context`
|
||||||
- `docforge_validate_project`
|
- `docforge_validate_project`
|
||||||
- `docforge_render_status`
|
- `docforge_render_status`
|
||||||
|
- `docforge_graph_plan`
|
||||||
|
- `docforge_graph_render_status`
|
||||||
- `docforge_visualize`
|
- `docforge_visualize`
|
||||||
- `docforge_stop_visualization`
|
- `docforge_stop_visualization`
|
||||||
- `docforge_visualization_status`
|
- `docforge_visualization_status`
|
||||||
|
|
@ -349,3 +375,6 @@ permissions and project instructions remain responsible for that broader boundar
|
||||||
The legacy `adapter_policy` object remains byte-compatible. It is now a projection of the
|
The legacy `adapter_policy` object remains byte-compatible. It is now a projection of the
|
||||||
versioned `effective_policy`; `--no-ast` restrictively overrides adapter evolution, AST analysis,
|
versioned `effective_policy`; `--no-ast` restrictively overrides adapter evolution, AST analysis,
|
||||||
and Logic indexing without widening any other capability.
|
and Logic indexing without widening any other capability.
|
||||||
|
|
||||||
|
See [Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for the compatibility boundary and
|
||||||
|
why the AST and Tree-sitter reference adapters are not no-AST adapters.
|
||||||
|
|
|
||||||
79
docs/MIGRATING_FROM_V1.md
Normal file
79
docs/MIGRATING_FROM_V1.md
Normal file
|
|
@ -0,0 +1,79 @@
|
||||||
|
# Migrating from DocForge v1
|
||||||
|
|
||||||
|
DocForge2 preserves the `docforge` distribution, Python package, CLI command, MCP tool prefix,
|
||||||
|
generic project descriptor version, and legacy adapter entry point. Migration is an additive
|
||||||
|
validation exercise, not a canonical-content rewrite.
|
||||||
|
|
||||||
|
Read [compatibility](COMPATIBILITY.md), [legacy and no-AST operation](LEGACY_AND_NO_AST.md), and
|
||||||
|
[recovery](RECOVERY_AND_PERFORMANCE.md) before changing a production binding.
|
||||||
|
|
||||||
|
## What remains compatible
|
||||||
|
|
||||||
|
- Generic Markdown and TOML projects keep `.docforge/project.toml` schema version 1.
|
||||||
|
- A custom adapter implementing only `load_projection()` remains valid.
|
||||||
|
- Existing canonical nodes, stable IDs, source paths, relationship vocabulary, changesets, and
|
||||||
|
reviewed hashes are not silently rewritten.
|
||||||
|
- Existing CLI and MCP names remain available. New fields, commands, tools, plans, policies, and
|
||||||
|
reference integrations are additive.
|
||||||
|
- `docforge-mcp --no-ast` remains the shorthand for the preserved no-AST binding.
|
||||||
|
|
||||||
|
Disposable index and cache schemas may change. Rebuild them rather than copying them as authority.
|
||||||
|
|
||||||
|
## Recommended migration
|
||||||
|
|
||||||
|
1. Record the v1 package version, adapter identity, project descriptor, canonical source hash,
|
||||||
|
active changesets, generated client configuration, and current rendered outputs.
|
||||||
|
2. Back up canonical sources and active proposal files. Derived `.docforge/cache`, preview,
|
||||||
|
portable-graph, and viewer state need not be authoritative backups.
|
||||||
|
3. Install the DocForge2 candidate in a separate environment. Do not repoint the production MCP
|
||||||
|
binding yet.
|
||||||
|
4. Run the existing adapter through complete loading and `ProjectIndex.check()`. For incremental
|
||||||
|
adapters, verify complete/incremental equivalence. Logic-producing incremental adapters must
|
||||||
|
implement the independent complete assembly oracle.
|
||||||
|
5. Compare project ID, adapter, revision, source hash, primary node and edge hashes, Logic policy,
|
||||||
|
and retrieval results with the v1 evidence.
|
||||||
|
6. Recompose effective capability and projection policy. Do not assume that a new default grants
|
||||||
|
proposal, application, rendering, or viewer access.
|
||||||
|
7. Rebuild disposable indexes, extraction caches, fragments, receipts, previews, and portable
|
||||||
|
artifacts from the validated candidate.
|
||||||
|
8. Generate a new client fragment. Generic projects use `docforge configure`; custom adapters use
|
||||||
|
`AdapterLauncherV1` with `generate_adapter_client_configuration()`.
|
||||||
|
9. Run doctor, real MCP bootstrap, representative retrieval, rendering status, recovery, and
|
||||||
|
exact-hash proposal/application checks in a disposable or shadow environment.
|
||||||
|
10. Repoint one production binding only after the candidate and rollback procedure pass.
|
||||||
|
|
||||||
|
## Adopting reference adapters
|
||||||
|
|
||||||
|
The repository reference adapters are narrow syntax evidence, not automatic semantic replacements
|
||||||
|
for mature v1 integrations. Python publishes modules, classes, functions, and local imports.
|
||||||
|
JavaScript and TypeScript publish syntax plus static project-relative imports and re-exports.
|
||||||
|
C++ uses a confined `compile_commands.json` as inert translation-unit inventory and publishes
|
||||||
|
directly resolvable project-local quoted includes.
|
||||||
|
|
||||||
|
Do not replace a compiler-, language-server-, or project-owned semantic adapter if the task depends
|
||||||
|
on resolved calls, types, inheritance, macros, compiler include semantics, runtime facts, or
|
||||||
|
ownership. Use [reference adapters](REFERENCE_ADAPTERS.md) as a measured starting point.
|
||||||
|
|
||||||
|
## Changesets
|
||||||
|
|
||||||
|
Existing changesets remain bound to their original project, root, base revision, source hash,
|
||||||
|
writer, operations, and exact hash. Do not edit a changeset to make it look current. Retrieve and
|
||||||
|
validate it through the candidate. Rebase only when every precondition remains true; otherwise
|
||||||
|
create and review a new proposal.
|
||||||
|
|
||||||
|
Before enabling canonical application, prove the exact serializer round trip and recovery behavior
|
||||||
|
for the selected adapter. The fixed reference MCP server is intentionally read-only.
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
Keep the v1 environment, configuration fragment, and service definition until the DocForge2
|
||||||
|
binding has passed live validation. To roll back:
|
||||||
|
|
||||||
|
1. Stop the DocForge2 MCP/viewer processes.
|
||||||
|
2. Restore the previous client or service binding.
|
||||||
|
3. Restore canonical sources only if a verified application receipt says they changed and the
|
||||||
|
adapter-specific rollback requires it.
|
||||||
|
4. Discard DocForge2-derived indexes, receipts, fragments, previews, and portable artifacts.
|
||||||
|
5. Restart v1 and verify its recorded project/source identity.
|
||||||
|
|
||||||
|
Never move a published tag or reuse a version identity for a corrected release.
|
||||||
95
docs/MILESTONE_4_BASELINE.md
Normal file
95
docs/MILESTONE_4_BASELINE.md
Normal file
|
|
@ -0,0 +1,95 @@
|
||||||
|
# Milestone 4 baseline
|
||||||
|
|
||||||
|
## Scope and method
|
||||||
|
|
||||||
|
This baseline records the adapter SDK, Python reference adapter, incremental equivalence, and
|
||||||
|
recovery behavior completed in Milestone 4. It was captured on 2026-07-29 from clean executable
|
||||||
|
candidate `95271dcf2e48045b9d3aed9b9ea09c7fc155692c`.
|
||||||
|
|
||||||
|
The maintained command was:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python tools/milestone4_benchmark.py \
|
||||||
|
--mode full \
|
||||||
|
--output benchmarks/milestone4-2026-07-29.json
|
||||||
|
```
|
||||||
|
|
||||||
|
The synthetic project contains 334 Python files. Each file contributes one module, one function,
|
||||||
|
and one argument node, for 1,002 primary nodes. Imports form a deterministic chain. Durations use
|
||||||
|
`time.perf_counter_ns()` and nearest-rank p95. Each ordinary result is serialized as compact sorted
|
||||||
|
JSON for its response-size gate. Per-operation memory uses `tracemalloc`; cumulative process
|
||||||
|
high-water uses `RUSAGE_SELF`.
|
||||||
|
|
||||||
|
Environment:
|
||||||
|
|
||||||
|
- Linux 7.1.3-200.nobara.fc44.x86_64 with glibc 2.43.
|
||||||
|
- CPython 3.14.6.
|
||||||
|
- x86_64.
|
||||||
|
- Three warm samples; cold, equivalence, and recovery operations run once.
|
||||||
|
- Per-operation traced-memory ceiling: 268,435,456 bytes.
|
||||||
|
- Process high-water ceiling: 536,870,912 bytes.
|
||||||
|
- Response ceiling: 524,288 bytes.
|
||||||
|
|
||||||
|
The complete machine-readable result is
|
||||||
|
[`benchmarks/milestone4-2026-07-29.json`](../benchmarks/milestone4-2026-07-29.json).
|
||||||
|
Its SHA-256 is `b6a871dde730a119fc0a138c47bd25f2c533ed3171a9a3d098075aef08173600`.
|
||||||
|
The stable evidence payload SHA-256 is
|
||||||
|
`4a8db461a311b5df4abd5aa00063e9a347d8b9dba19b30e35684f561d5271549`.
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
| Operation | Median | p95 | Limit | Traced peak | Response |
|
||||||
|
|---|---:|---:|---:|---:|---:|
|
||||||
|
| Cold incremental build | 1,599.760 ms | 1,599.760 ms | 20,000 ms | 68,573,540 B | 14,580 B |
|
||||||
|
| Warm incremental build | 1,244.427 ms | 1,265.387 ms | 20,000 ms | 69,997,166 B | 14,578 B |
|
||||||
|
| Complete/incremental equivalence | 1,897.919 ms | 1,897.919 ms | 30,000 ms | 64,305,919 B | 230 B |
|
||||||
|
| Corrupt extraction-cache recovery | 1,695.406 ms | 1,695.406 ms | 30,000 ms | 69,776,923 B | 14,580 B |
|
||||||
|
| Corrupt index recovery | 1,707.879 ms | 1,707.879 ms | 30,000 ms | 68,561,924 B | 14,779 B |
|
||||||
|
|
||||||
|
Process high-water was 78,798,848 bytes. The regression limits intentionally leave multiple times
|
||||||
|
the measured headroom; they are tripwires, not performance promises.
|
||||||
|
|
||||||
|
## Deterministic graph and Logic evidence
|
||||||
|
|
||||||
|
The candidate produced:
|
||||||
|
|
||||||
|
- 1,002 primary nodes with hash
|
||||||
|
`f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f`.
|
||||||
|
- 1,001 primary edges with hash
|
||||||
|
`00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35`.
|
||||||
|
- 334 Logic projections containing 2,338 Logic nodes and 2,338 Logic edges, with hash
|
||||||
|
`c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c`.
|
||||||
|
- Complete assembly hash
|
||||||
|
`2888182ab765fbffe3ba873c1613345640e8e6d89be74ddfca7452c0a5056345`.
|
||||||
|
- Source hash
|
||||||
|
`30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2`.
|
||||||
|
|
||||||
|
The independently loaded complete assembly and the incremental assembly matched exactly across
|
||||||
|
project identity, revision, source, nodes, edges, and Logic.
|
||||||
|
|
||||||
|
## Work and recovery gates
|
||||||
|
|
||||||
|
The warm build recorded all 334 sources as cache hits, zero reparsed sources, zero `ast.parse`
|
||||||
|
calls, and zero `extract_source` calls. Corrupting the extraction cache forced all sources through
|
||||||
|
extraction and reproduced the same graph and Logic hashes. Corrupting SQLite rebuilt the index
|
||||||
|
entirely from cache hits with zero parsing or extraction and reproduced those hashes.
|
||||||
|
|
||||||
|
This zero-parser evidence applies to the Python benchmark. Focused JavaScript and TypeScript tests
|
||||||
|
prove parser-free manifests. The C++ reference manifest uses Tree-sitter for bounded quoted-include
|
||||||
|
discovery and does not make a zero-warm-parser claim.
|
||||||
|
|
||||||
|
## Fresh-wheel and reference evidence
|
||||||
|
|
||||||
|
The offline adoption proof built and installed the base wheel without Tree-sitter packages,
|
||||||
|
built and checked a real Python reference project, started the isolated reference MCP server, and
|
||||||
|
performed bootstrap, search, and exact retrieval over its 21 read tools. Selecting C++ without its
|
||||||
|
extra failed with `optional_dependency_missing` and `docforge[cpp]` remediation.
|
||||||
|
|
||||||
|
Focused fixtures produced:
|
||||||
|
|
||||||
|
- Python: 14 nodes, 13 edges, 5 Logic projections.
|
||||||
|
- JavaScript: 14 nodes, 14 edges, 5 Logic projections.
|
||||||
|
- TypeScript: 13 nodes, 14 edges, 4 Logic projections.
|
||||||
|
- C++: 17 nodes, 16 edges, 5 Logic projections.
|
||||||
|
|
||||||
|
These fixture counts verify implementations; they are not language-wide completeness claims.
|
||||||
83
docs/MILESTONE_4_CLOSEOUT.md
Normal file
83
docs/MILESTONE_4_CLOSEOUT.md
Normal file
|
|
@ -0,0 +1,83 @@
|
||||||
|
# Milestone 4 closeout
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
Milestone 4 is complete. New projects can adopt a public adapter SDK or one of four narrow
|
||||||
|
repository reference integrations, attach a fixed read-only MCP server, and follow maintained
|
||||||
|
product documentation without reading core implementation.
|
||||||
|
|
||||||
|
Implemented contracts:
|
||||||
|
|
||||||
|
- Stable `docforge.adapter_sdk` authoring imports.
|
||||||
|
- Independent complete primary-graph-plus-Logic oracle and exact incremental equivalence.
|
||||||
|
- Bounded adapter assemblies and version-1 extraction caches.
|
||||||
|
- Base Python, optional JavaScript, optional TypeScript, and optional C++ reference integrations.
|
||||||
|
- Closed `.docforge/reference-adapter.toml` and fixed `docforge.reference_mcp` read-only binding.
|
||||||
|
- Immutable, project-bound, launchable `AdapterLauncherV1` declarations and generated Codex,
|
||||||
|
Claude, and OpenClaw fragments for custom adapters.
|
||||||
|
- Live implementation-derived CLI and MCP reference tables with race-safe publication and drift
|
||||||
|
checking.
|
||||||
|
- Strict documentation graph, link, anchor, H1, reachability, required-page, generated-notice, and
|
||||||
|
documented-reference-config checks.
|
||||||
|
- Offline fresh-wheel adoption and maintained 1,002-node scale/recovery gates.
|
||||||
|
|
||||||
|
## Candidate evidence
|
||||||
|
|
||||||
|
The frozen executable candidate is
|
||||||
|
`95271dcf2e48045b9d3aed9b9ea09c7fc155692c`.
|
||||||
|
|
||||||
|
Its complete executable gate passed:
|
||||||
|
|
||||||
|
- Ruff formatting and Python lint.
|
||||||
|
- HTML, rendered-manual HTML, portable-graph HTML, CSS, and JavaScript checks.
|
||||||
|
- Pyright with zero diagnostics.
|
||||||
|
- Warning-strict compilation.
|
||||||
|
- 142 contract tests and 268 subtests.
|
||||||
|
- 347 complete tests and 402 subtests.
|
||||||
|
- Three Playwright and axe accessibility flows for the manual, portable graph, and live viewer.
|
||||||
|
- Lock and npm dependency-tree checks.
|
||||||
|
- Wheel and source-distribution builds.
|
||||||
|
- Offline fresh-wheel adoption.
|
||||||
|
- Milestone 0, 1, 2, 3, and 4 smoke benchmarks.
|
||||||
|
|
||||||
|
The clean full benchmark passed exact graph-plus-Logic equivalence, warm zero Python parser and
|
||||||
|
extraction work, corrupt-cache recovery, corrupt-index recovery, response, memory, and latency
|
||||||
|
gates. Exact results are in [the Milestone 4 baseline](MILESTONE_4_BASELINE.md) and
|
||||||
|
[`benchmarks/milestone4-2026-07-29.json`](../benchmarks/milestone4-2026-07-29.json).
|
||||||
|
|
||||||
|
Gitleaks 8.30.1 scanned the Milestone 4 commit range and candidate tree with no findings. The SSH
|
||||||
|
remote syntax prevents Gitleaks from constructing finding hyperlinks; it does not affect scanning.
|
||||||
|
|
||||||
|
## Reference scope
|
||||||
|
|
||||||
|
- Python uses the standard-library AST and publishes syntax plus local imports.
|
||||||
|
- JavaScript and TypeScript use distinct optional Tree-sitter grammars and publish syntax plus
|
||||||
|
project-local static relative imports and re-exports.
|
||||||
|
- C++ uses a confined `compile_commands.json` as inert translation-unit inventory and publishes
|
||||||
|
syntax plus directly resolvable project-local quoted includes.
|
||||||
|
- The references do not claim resolved calls, inheritance, types, symbol references, compiler
|
||||||
|
include semantics, macro semantics, runtime behavior, or semantic ownership.
|
||||||
|
|
||||||
|
The C++ reference never executes a compiler or compilation-database command. It is not a Clang
|
||||||
|
semantic adapter.
|
||||||
|
|
||||||
|
## Preserved boundaries
|
||||||
|
|
||||||
|
- The `docforge` distribution, Python package, CLI, MCP executable, and tool names remain.
|
||||||
|
- Generic projects and one-method `load_projection()` adapters remain supported.
|
||||||
|
- Descriptor schema version 1, index schema version 3, effective policy version 1, and no-AST
|
||||||
|
behavior remain.
|
||||||
|
- Heavy language frontends are optional. Base generic and Python operation installs no
|
||||||
|
Tree-sitter distribution.
|
||||||
|
- Reference MCP is read-only. Proposal and application remain explicit project-owned gates.
|
||||||
|
- No language adapter was separately published.
|
||||||
|
- No WorldForge, ScrapeStation, legacy-repository, production-binding, storage, or self-hosting
|
||||||
|
change was made.
|
||||||
|
- No tag or Forgejo release was created for Milestone 4.
|
||||||
|
|
||||||
|
## Later work
|
||||||
|
|
||||||
|
Milestone 5 owns stabilization and the first DocForge2 release. Release identity, compatibility
|
||||||
|
matrix, migration and recovery proofs, comparative real-task evidence, versioning, tagging, and
|
||||||
|
publication must be validated there. Remote adapters, render farms, third-party renderers,
|
||||||
|
cross-project graphs, storage replacement, and self-hosting remain deferred without measured need.
|
||||||
|
|
@ -1,6 +1,276 @@
|
||||||
# DocForge setup moved to the user manual
|
# New-project quickstart
|
||||||
|
|
||||||
The complete installation, project setup, visualization, CLI, MCP, application, adapter, and
|
This guide takes a new installation from an empty project binding to a validated generic manual or
|
||||||
troubleshooting reference now lives in the [DocForge user manual](USER_MANUAL.md).
|
one of DocForge's fixed reference source graphs. For the complete operating reference, see the
|
||||||
|
[user manual](USER_MANUAL.md).
|
||||||
|
|
||||||
This file remains only so existing bookmarks and links continue to resolve.
|
## 1. Install DocForge
|
||||||
|
|
||||||
|
DocForge requires Python 3.12 or newer. A base installation includes the generic project service,
|
||||||
|
the public adapter SDK, the Python reference integration, CLI, MCP server, renderers, and viewer
|
||||||
|
assets. It does not install Tree-sitter.
|
||||||
|
|
||||||
|
From a source checkout:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone <repository-url> /absolute/path/DocForge
|
||||||
|
cd /absolute/path/DocForge
|
||||||
|
uv sync --group dev
|
||||||
|
|
||||||
|
DOCFORGE=/absolute/path/DocForge/.venv/bin/docforge
|
||||||
|
DOCFORGE_MCP=/absolute/path/DocForge/.venv/bin/docforge-mcp
|
||||||
|
DOCFORGE_PYTHON=/absolute/path/DocForge/.venv/bin/python
|
||||||
|
```
|
||||||
|
|
||||||
|
For an isolated consumer environment, install the checkout or a built wheel with `uv pip install`.
|
||||||
|
Add only the language extras that project needs:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv venv /absolute/path/docforge-env --python 3.12
|
||||||
|
uv pip install --python /absolute/path/docforge-env/bin/python /absolute/path/DocForge
|
||||||
|
uv pip install --python /absolute/path/docforge-env/bin/python \
|
||||||
|
"/absolute/path/DocForge[javascript]"
|
||||||
|
uv pip install --python /absolute/path/docforge-env/bin/python \
|
||||||
|
"/absolute/path/DocForge[typescript]"
|
||||||
|
uv pip install --python /absolute/path/docforge-env/bin/python \
|
||||||
|
"/absolute/path/DocForge[cpp]"
|
||||||
|
```
|
||||||
|
|
||||||
|
The `languages` extra installs all three optional grammar families. JavaScript and TypeScript are
|
||||||
|
separate extras because they use distinct grammars. The C++ extra supplies a syntax grammar, not a
|
||||||
|
compiler or Clang semantic frontend.
|
||||||
|
|
||||||
|
Confirm the installation:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
"$DOCFORGE" --help
|
||||||
|
"$DOCFORGE_MCP" --help
|
||||||
|
```
|
||||||
|
|
||||||
|
## 2. Choose a project route
|
||||||
|
|
||||||
|
Use a generic project when Markdown or TOML documentation is canonical. Use a fixed reference
|
||||||
|
adapter when you want a bounded source inventory and syntax-level graph for Python, JavaScript,
|
||||||
|
TypeScript, or C++. Use a project-owned adapter when the production contract must supply richer
|
||||||
|
semantics.
|
||||||
|
|
||||||
|
- Generic manual: continue with [Create a generic project](#create-a-generic-project).
|
||||||
|
- Fixed source example: continue with [Use a reference adapter](#use-a-reference-adapter).
|
||||||
|
- Production language frontend: follow the [Adapter authoring guide](ADAPTER_AUTHORING_GUIDE.md).
|
||||||
|
|
||||||
|
## Create a generic project
|
||||||
|
|
||||||
|
Set the absolute project root and assess it without writing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
PROJECT=/absolute/path/MyProject
|
||||||
|
"$DOCFORGE" --project-root "$PROJECT" onboard
|
||||||
|
```
|
||||||
|
|
||||||
|
The assessment reports detected languages, build evidence, documentation candidates, current
|
||||||
|
configuration, and available capabilities. Detection never invents a source graph.
|
||||||
|
|
||||||
|
Create a generic starter explicitly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
"$DOCFORGE" --project-root "$PROJECT" onboard \
|
||||||
|
--scaffold \
|
||||||
|
--project-id my-project \
|
||||||
|
--title "My Project"
|
||||||
|
```
|
||||||
|
|
||||||
|
Scaffolding is create-only and refuses existing target files. It creates:
|
||||||
|
|
||||||
|
- `.docforge/project.toml`, the generic project descriptor;
|
||||||
|
- `docs/docforge/content/architecture-overview.md`, one canonical node;
|
||||||
|
- `.docforge/templates/manual.html`, one built-in-renderer template;
|
||||||
|
- derived index, receipt, and rendered output below `.docforge`.
|
||||||
|
|
||||||
|
If a language was detected, source graph status remains `adapter_required`. The generic starter
|
||||||
|
does not claim source semantics.
|
||||||
|
|
||||||
|
Validate and inspect it:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
"$DOCFORGE" --project-root "$PROJECT" validate
|
||||||
|
"$DOCFORGE" --project-root "$PROJECT" reindex
|
||||||
|
"$DOCFORGE" --project-root "$PROJECT" search architecture
|
||||||
|
"$DOCFORGE" --project-root "$PROJECT" show architecture.overview
|
||||||
|
"$DOCFORGE" --project-root "$PROJECT" render-status
|
||||||
|
```
|
||||||
|
|
||||||
|
The descriptor is explained field by field in [Project descriptor](PROJECT_DESCRIPTOR.md). Add
|
||||||
|
canonical nodes only after choosing their authority, stable IDs, families, statuses, and allowed
|
||||||
|
relationships; [Core concepts and authority](CORE_CONCEPTS_AND_AUTHORITY.md) defines those terms.
|
||||||
|
|
||||||
|
### Start a generic MCP binding
|
||||||
|
|
||||||
|
Start read-only first:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
"$DOCFORGE_MCP" \
|
||||||
|
--project-root "$PROJECT" \
|
||||||
|
--capability-mode read
|
||||||
|
```
|
||||||
|
|
||||||
|
Call `docforge_bootstrap` before other tools. It reports the exact project identity, generation,
|
||||||
|
effective policy, projection policy, available capabilities, and recommended first read.
|
||||||
|
|
||||||
|
To enable proposals, the descriptor must declare the writer and the process must select it:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
"$DOCFORGE_MCP" \
|
||||||
|
--project-root "$PROJECT" \
|
||||||
|
--capability-mode proposal \
|
||||||
|
--proposal-writer project-editor
|
||||||
|
```
|
||||||
|
|
||||||
|
Canonical application is a separate startup gate. Do not add it to a read-only client:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
"$DOCFORGE_MCP" \
|
||||||
|
--project-root "$PROJECT" \
|
||||||
|
--capability-mode application \
|
||||||
|
--proposal-writer project-editor \
|
||||||
|
--canonical-applier project-editor
|
||||||
|
```
|
||||||
|
|
||||||
|
Application accepts one exact reviewed changeset hash. It does not commit, push, build, deploy, or
|
||||||
|
publish the project.
|
||||||
|
|
||||||
|
### Generate a generic client fragment
|
||||||
|
|
||||||
|
Preview is side-effect free:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
"$DOCFORGE" configure codex --project "$PROJECT"
|
||||||
|
"$DOCFORGE" configure claude --project "$PROJECT"
|
||||||
|
"$DOCFORGE" configure openclaw --project "$PROJECT"
|
||||||
|
```
|
||||||
|
|
||||||
|
Add `--output /absolute/path/new-fragment` to create one new private standalone file. Generation
|
||||||
|
does not merge with or replace a different existing file. Diagnose an installed binding with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
"$DOCFORGE" doctor --client codex --project "$PROJECT"
|
||||||
|
```
|
||||||
|
|
||||||
|
Doctor is a bounded configuration inspector, not a connection test. See [Agent
|
||||||
|
integration](AGENT_INTEGRATION.md) for client-specific layouts and limitations.
|
||||||
|
|
||||||
|
## Use a reference adapter
|
||||||
|
|
||||||
|
Reference adapters read one fixed descriptor:
|
||||||
|
`.docforge/reference-adapter.toml`. The path is not selectable.
|
||||||
|
|
||||||
|
Create a Python project configuration:
|
||||||
|
|
||||||
|
```toml reference-adapter
|
||||||
|
schema_version = 1
|
||||||
|
project_id = "my-python-project"
|
||||||
|
title = "My Python Project"
|
||||||
|
language = "python"
|
||||||
|
source_roots = ["src"]
|
||||||
|
```
|
||||||
|
|
||||||
|
JavaScript uses `language = "javascript"` and the `javascript` extra. TypeScript uses
|
||||||
|
`language = "typescript"` and the `typescript` extra.
|
||||||
|
|
||||||
|
C++ additionally requires a confined compilation database:
|
||||||
|
|
||||||
|
```toml reference-adapter
|
||||||
|
schema_version = 1
|
||||||
|
project_id = "my-cpp-project"
|
||||||
|
title = "My C++ Project"
|
||||||
|
language = "cpp"
|
||||||
|
source_roots = ["src", "include"]
|
||||||
|
compilation_database = "compile_commands.json"
|
||||||
|
```
|
||||||
|
|
||||||
|
The C++ integration reads `compile_commands.json` only as bounded translation-unit inventory and
|
||||||
|
fingerprint evidence. It never executes a recorded command or compiler.
|
||||||
|
|
||||||
|
Run the fixed read-only server through the same installed Python interpreter:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
"$DOCFORGE_PYTHON" -I -m docforge.reference_mcp \
|
||||||
|
--project-root "$PROJECT" \
|
||||||
|
--capability-mode read
|
||||||
|
```
|
||||||
|
|
||||||
|
The binding chooses one in-package provider from the descriptor language. It accepts no provider
|
||||||
|
module, command, argument list, working directory, environment, discovery rule, proposal writer,
|
||||||
|
or canonical applier. Its MCP surface is the read subset documented in the [generated command
|
||||||
|
reference](COMMAND_REFERENCE.md).
|
||||||
|
|
||||||
|
### Generate a reference-adapter client fragment
|
||||||
|
|
||||||
|
Generic `docforge configure` intentionally refuses custom adapters. Construct the adapter project
|
||||||
|
and immutable launcher, then call the custom-adapter generator:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from docforge.adapter_launcher import AdapterLauncherV1
|
||||||
|
from docforge.client_config import generate_adapter_client_configuration
|
||||||
|
from docforge.reference_mcp import REFERENCE_MCP_MODULE, create_reference_project
|
||||||
|
|
||||||
|
root = Path("/absolute/path/MyProject").resolve(strict=True)
|
||||||
|
project = create_reference_project(root)
|
||||||
|
launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE)
|
||||||
|
|
||||||
|
preview = generate_adapter_client_configuration(
|
||||||
|
project,
|
||||||
|
launcher,
|
||||||
|
"codex",
|
||||||
|
capability_mode="read",
|
||||||
|
)
|
||||||
|
print(preview["artifact"]["content"])
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `client="claude"` or `client="openclaw"` for those formats. Pass an absolute `output` path only
|
||||||
|
when creating a new standalone private fragment. The generated launch is bound to the selected
|
||||||
|
project, descriptor hash, adapter identity, installed module, isolated interpreter, policy, and
|
||||||
|
source availability evidence.
|
||||||
|
|
||||||
|
Read [Reference adapters](REFERENCE_ADAPTERS.md) before relying on the graph. The Python example
|
||||||
|
publishes local imports, the JavaScript and TypeScript examples publish project-local static
|
||||||
|
relative imports and re-exports, and the C++ example publishes directly resolvable project-local
|
||||||
|
quoted includes. None is a complete semantic compiler frontend.
|
||||||
|
|
||||||
|
## 3. Add visualization only when needed
|
||||||
|
|
||||||
|
Install the per-user viewer manager once:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
/absolute/path/DocForge/.venv/bin/docforge-viewer-manager install-user-service
|
||||||
|
```
|
||||||
|
|
||||||
|
Then start a project-bound snapshot:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
"$DOCFORGE" --project-root "$PROJECT" visualize
|
||||||
|
"$DOCFORGE" --project-root "$PROJECT" visualization-status
|
||||||
|
```
|
||||||
|
|
||||||
|
The listener is loopback-only and tokenized. The viewer is a derived snapshot, not canonical
|
||||||
|
authority and not a continuously monitored filesystem view. Read [Rendering and
|
||||||
|
visualization](RENDERING_AND_VISUALIZATION.md) for manual, portable, and live-viewer differences.
|
||||||
|
|
||||||
|
## 4. Verify the maintained checkout
|
||||||
|
|
||||||
|
Contributors can run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make command-reference-check
|
||||||
|
make docs-check
|
||||||
|
make adoption-m4
|
||||||
|
make benchmark-m4-smoke
|
||||||
|
```
|
||||||
|
|
||||||
|
Run `make gate` before a release candidate. `benchmark-m4-smoke` is routine coverage;
|
||||||
|
`benchmark-m4` is the maintained full adapter workload.
|
||||||
|
|
||||||
|
Continue with [Project onboarding](PROJECT_ONBOARDING.md) for production integration,
|
||||||
|
[Policy precedence](POLICY_PRECEDENCE.md) before widening a binding, and [Security](SECURITY.md)
|
||||||
|
before exposing any MCP process.
|
||||||
|
|
|
||||||
202
docs/POLICY_PRECEDENCE.md
Normal file
202
docs/POLICY_PRECEDENCE.md
Normal file
|
|
@ -0,0 +1,202 @@
|
||||||
|
# Policy precedence
|
||||||
|
|
||||||
|
DocForge composes an immutable effective policy for each project-bound process. It resolves
|
||||||
|
restrictions in a fixed order:
|
||||||
|
|
||||||
|
```text
|
||||||
|
core safety
|
||||||
|
> explicit binding
|
||||||
|
> no-AST shorthand
|
||||||
|
> resource availability
|
||||||
|
```
|
||||||
|
|
||||||
|
A lower layer can make a requested operation unavailable; it cannot override a higher-layer
|
||||||
|
prohibition. Bootstrap and generated client evidence report the composed result and hashes so a
|
||||||
|
client does not need to infer policy from command-line arguments.
|
||||||
|
|
||||||
|
## 1. Core safety
|
||||||
|
|
||||||
|
Core safety is unconditional. No capability mode exposes:
|
||||||
|
|
||||||
|
- arbitrary file access or project switching;
|
||||||
|
- arbitrary renderer, module, command, shell, argument, working-directory, or environment
|
||||||
|
selection;
|
||||||
|
- Git mutation;
|
||||||
|
- project builds or compiler execution;
|
||||||
|
- deployment or publication.
|
||||||
|
|
||||||
|
Canonical application is limited to DocForge's validated serializer boundary. Adapter launch is
|
||||||
|
limited to the immutable launcher contract. Rendering is limited to declared views and fixed
|
||||||
|
built-in workers. An indexed instruction cannot alter any of these rules.
|
||||||
|
|
||||||
|
## 2. Explicit binding
|
||||||
|
|
||||||
|
A process is bound at startup to one project root, descriptor, adapter, capability mode, writer and
|
||||||
|
applier identities when present, no-AST selection, diagnostics selection, and projection modes.
|
||||||
|
The binding does not change during the process lifetime.
|
||||||
|
|
||||||
|
Capability modes are:
|
||||||
|
|
||||||
|
- `read`: register only the read surface;
|
||||||
|
- `proposal`: add proposal tools when a valid selected writer is available;
|
||||||
|
- `application`: require a startup-bound canonical applier and expose exact-hash application;
|
||||||
|
- `operator`: reserved; it currently adds no tools.
|
||||||
|
|
||||||
|
Mode describes the maximum registered surface. Actual authority can be narrower. A descriptor must
|
||||||
|
declare the selected writer, including allowed families and operation types. Application requires
|
||||||
|
the matching configured writer, changeset creator, and canonical-applier identity. A mode name
|
||||||
|
cannot create a missing descriptor grant.
|
||||||
|
|
||||||
|
Generic generated client fragments default to read mode. Other construction paths preserve their
|
||||||
|
documented compatible factory defaults. Treat `docforge_bootstrap.session_contract` and its actual
|
||||||
|
capabilities as authoritative for a running server.
|
||||||
|
|
||||||
|
## 3. No-AST shorthand
|
||||||
|
|
||||||
|
`--no-ast` is a restrictive compatibility shorthand. It composes:
|
||||||
|
|
||||||
|
- adapter evolution `preserve`;
|
||||||
|
- AST analysis `forbidden`;
|
||||||
|
- Logic indexing `off`;
|
||||||
|
- `docforge_get_logic` blocked;
|
||||||
|
- prohibitions on AST, Tree-sitter, compiler-AST, and function-Logic upgrades.
|
||||||
|
|
||||||
|
Non-AST source fingerprinting and incremental caching remain allowed. Existing one-method adapters
|
||||||
|
continue to use `load_projection()`.
|
||||||
|
|
||||||
|
The shorthand does not inspect how an existing adapter was implemented, sandbox its filesystem
|
||||||
|
reads, or transform an AST-based adapter into a no-AST adapter. Do not run the Python,
|
||||||
|
JavaScript/TypeScript, or C++ syntax reference integrations and then describe the binding as a
|
||||||
|
proved no-AST integration. See [Legacy and no-AST operation](LEGACY_AND_NO_AST.md).
|
||||||
|
|
||||||
|
## 4. Resource availability
|
||||||
|
|
||||||
|
Even an allowed policy cannot create a missing resource:
|
||||||
|
|
||||||
|
- application mode requires a startup-bound canonical applier;
|
||||||
|
- manual `explicit` or `auto` requires declared manual render configuration;
|
||||||
|
- manual `auto` also requires canonical application in the current operation or server;
|
||||||
|
- portable graph `explicit` requires declared portable graph configuration;
|
||||||
|
- live viewer `on-demand` requires the viewer runtime;
|
||||||
|
- an optional reference language requires its installed extra;
|
||||||
|
- a project-owned launcher module must be installed, isolated, project-owned, and unchanged.
|
||||||
|
|
||||||
|
Unavailable requested modes fail before hidden work. Missing optional language grammars return an
|
||||||
|
actionable install target such as `docforge[typescript]` or `docforge[cpp]`; DocForge does not
|
||||||
|
silently downgrade to a different frontend.
|
||||||
|
|
||||||
|
## Effective process policy
|
||||||
|
|
||||||
|
The version-1 effective policy reports:
|
||||||
|
|
||||||
|
- capability mode and whether it came from a factory default or explicit selection;
|
||||||
|
- adapter evolution, AST analysis, and Logic indexing;
|
||||||
|
- automatic synchronization and validated integrity;
|
||||||
|
- the compatible manual-render projection;
|
||||||
|
- profiling state;
|
||||||
|
- blocked tools and prohibitions;
|
||||||
|
- the exact precedence list.
|
||||||
|
|
||||||
|
This version-1 projection preserves compatibility. It is not the complete version-2 rendering
|
||||||
|
policy; portable graph and live-viewer choices are reported separately.
|
||||||
|
|
||||||
|
## Independent projection policy
|
||||||
|
|
||||||
|
Manual rendering, portable graph publication, and live visualization use a separate immutable
|
||||||
|
version-2 policy:
|
||||||
|
|
||||||
|
```text
|
||||||
|
manual: auto | explicit | disabled
|
||||||
|
portable_graph: explicit | disabled
|
||||||
|
live_viewer: on-demand | disabled
|
||||||
|
```
|
||||||
|
|
||||||
|
When a selector is omitted, composition uses availability-aware defaults:
|
||||||
|
|
||||||
|
- manual is `auto` only when manual configuration and canonical application are both available;
|
||||||
|
otherwise it is `explicit` when configured, or `disabled`;
|
||||||
|
- portable graph is `explicit` when configured, otherwise `disabled`;
|
||||||
|
- live viewer is `on-demand` when the runtime is available, otherwise `disabled`.
|
||||||
|
|
||||||
|
An explicit non-disabled selection for a missing resource returns
|
||||||
|
`projection_policy_unavailable`. An active operation prohibited by the selected policy returns
|
||||||
|
`projection_policy_forbids_operation` before planning, rendering, or viewer startup.
|
||||||
|
|
||||||
|
Status remains intentionally narrower than active work. Manual and portable receipt-only status
|
||||||
|
are available when their active operations are disabled. Viewer status and explicit stop remain
|
||||||
|
available when viewer start is disabled.
|
||||||
|
|
||||||
|
Ordinary standalone CLI rendering uses manual `explicit`. Manual `auto` belongs to a canonical
|
||||||
|
application operation that owns automatic regeneration.
|
||||||
|
|
||||||
|
## Diagnostics do not grant authority
|
||||||
|
|
||||||
|
`--diagnostics` enables bounded request-local stage timings and compiler-work counters. It does not
|
||||||
|
enable tools, broaden paths, retain project content, or displace a primary MCP result that already
|
||||||
|
needs the response budget.
|
||||||
|
|
||||||
|
## Descriptor policy and process policy
|
||||||
|
|
||||||
|
The project descriptor is canonical project configuration. The process policy is a runtime
|
||||||
|
restriction. They compose by intersection:
|
||||||
|
|
||||||
|
```text
|
||||||
|
operation is available
|
||||||
|
only if core permits it
|
||||||
|
and the startup binding registers it
|
||||||
|
and no-AST permits it
|
||||||
|
and required resources exist
|
||||||
|
and the descriptor grants the requested project authority
|
||||||
|
and current graph/hash preconditions validate
|
||||||
|
```
|
||||||
|
|
||||||
|
Changing a descriptor does not retarget a running project-owned process. Descriptor, adapter
|
||||||
|
implementation, or launcher drift requires a fresh process.
|
||||||
|
|
||||||
|
## Common decisions
|
||||||
|
|
||||||
|
For an agent that only reads documentation:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docforge-mcp \
|
||||||
|
--project-root /absolute/path/MyProject \
|
||||||
|
--capability-mode read
|
||||||
|
```
|
||||||
|
|
||||||
|
For an agent that may prepare reviewable proposals:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docforge-mcp \
|
||||||
|
--project-root /absolute/path/MyProject \
|
||||||
|
--capability-mode proposal \
|
||||||
|
--proposal-writer project-editor
|
||||||
|
```
|
||||||
|
|
||||||
|
For a tightly controlled application process:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docforge-mcp \
|
||||||
|
--project-root /absolute/path/MyProject \
|
||||||
|
--capability-mode application \
|
||||||
|
--proposal-writer project-editor \
|
||||||
|
--canonical-applier project-editor
|
||||||
|
```
|
||||||
|
|
||||||
|
For a read binding with every active output projection disabled:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docforge-mcp \
|
||||||
|
--project-root /absolute/path/MyProject \
|
||||||
|
--capability-mode read \
|
||||||
|
--manual-render-policy disabled \
|
||||||
|
--portable-graph-policy disabled \
|
||||||
|
--live-viewer-policy disabled
|
||||||
|
```
|
||||||
|
|
||||||
|
Prefer the narrowest binding that completes the workflow. Call `docforge_bootstrap` first and use
|
||||||
|
the returned effective policy, projection policy, actual capabilities, and prohibitions rather
|
||||||
|
than assumptions based on client configuration.
|
||||||
|
|
||||||
|
See the [MCP contract](MCP_CONTRACT.md), [Security](SECURITY.md), [Project
|
||||||
|
descriptor](PROJECT_DESCRIPTOR.md), and [Rendering and
|
||||||
|
visualization](RENDERING_AND_VISUALIZATION.md).
|
||||||
279
docs/PROJECT_DESCRIPTOR.md
Normal file
279
docs/PROJECT_DESCRIPTOR.md
Normal file
|
|
@ -0,0 +1,279 @@
|
||||||
|
# Project descriptor
|
||||||
|
|
||||||
|
A generic DocForge project is selected by one fixed file:
|
||||||
|
`.docforge/project.toml` beneath an explicit project root. The descriptor is schema version 1.
|
||||||
|
DocForge validates both the JSON-schema shape in `schemas/project.schema.json` and runtime
|
||||||
|
invariants that schema alone cannot prove.
|
||||||
|
|
||||||
|
Reference integrations use a different fixed descriptor,
|
||||||
|
`.docforge/reference-adapter.toml`; see [Reference adapters](REFERENCE_ADAPTERS.md).
|
||||||
|
Project-owned adapters construct the same runtime `ProjectDescriptor` contract through the public
|
||||||
|
adapter SDK.
|
||||||
|
|
||||||
|
## Complete generic example
|
||||||
|
|
||||||
|
```toml
|
||||||
|
schema_version = 1
|
||||||
|
project_id = "my-project"
|
||||||
|
title = "My Project"
|
||||||
|
adapter = "generic"
|
||||||
|
|
||||||
|
[sources]
|
||||||
|
content_roots = ["docs/docforge/content"]
|
||||||
|
authority_files = []
|
||||||
|
|
||||||
|
[derived]
|
||||||
|
cache_root = ".docforge/cache"
|
||||||
|
index = ".docforge/cache/index.sqlite3"
|
||||||
|
|
||||||
|
[changesets]
|
||||||
|
root = ".docforge/changesets"
|
||||||
|
|
||||||
|
[[changesets.writers]]
|
||||||
|
id = "project-editor"
|
||||||
|
families = ["architecture", "operations", "system"]
|
||||||
|
operations = ["create", "update", "move", "delete"]
|
||||||
|
|
||||||
|
[render]
|
||||||
|
template_root = ".docforge/templates"
|
||||||
|
preview_root = ".docforge/previews"
|
||||||
|
|
||||||
|
[[render.views]]
|
||||||
|
id = "manual"
|
||||||
|
renderer = "generic_html"
|
||||||
|
template = "manual.html"
|
||||||
|
output = ".docforge/rendered/manual.html"
|
||||||
|
title = "My Project Manual"
|
||||||
|
families = ["architecture", "operations", "system"]
|
||||||
|
|
||||||
|
[graph_render]
|
||||||
|
output_root = ".docforge/portable-graph"
|
||||||
|
|
||||||
|
[[graph_render.views]]
|
||||||
|
id = "architecture"
|
||||||
|
renderer = "portable_graph_html"
|
||||||
|
output = "architecture.html"
|
||||||
|
title = "Architecture"
|
||||||
|
root = "architecture.overview"
|
||||||
|
initial_mode = "web"
|
||||||
|
depth = 3
|
||||||
|
max_nodes = 250
|
||||||
|
max_edges = 1000
|
||||||
|
max_work = 100000
|
||||||
|
families = ["architecture", "system"]
|
||||||
|
relations = ["depends_on", "owns", "calls", "reads", "writes", "tested_by", "relates_to"]
|
||||||
|
authorities = []
|
||||||
|
statuses = ["current", "active", "verified"]
|
||||||
|
tags = []
|
||||||
|
include_logic = false
|
||||||
|
|
||||||
|
[graph]
|
||||||
|
allowed_relations = [
|
||||||
|
"calls",
|
||||||
|
"depends_on",
|
||||||
|
"owns",
|
||||||
|
"reads",
|
||||||
|
"relates_to",
|
||||||
|
"tested_by",
|
||||||
|
"writes",
|
||||||
|
]
|
||||||
|
|
||||||
|
[limits]
|
||||||
|
max_source_bytes = 500000
|
||||||
|
max_nodes = 10000
|
||||||
|
max_query_chars = 500
|
||||||
|
max_results = 100
|
||||||
|
max_traversal_depth = 6
|
||||||
|
max_context_tokens = 12000
|
||||||
|
max_tool_output_chars = 200000
|
||||||
|
max_changesets = 100
|
||||||
|
max_changeset_operations = 100
|
||||||
|
max_changeset_bytes = 1000000
|
||||||
|
max_render_views = 20
|
||||||
|
max_template_bytes = 1000000
|
||||||
|
max_render_bytes = 1000000
|
||||||
|
|
||||||
|
[[profiles]]
|
||||||
|
id = "development"
|
||||||
|
families = ["architecture", "operations", "system"]
|
||||||
|
statuses = ["current", "active", "verified"]
|
||||||
|
required_nodes = ["architecture.overview"]
|
||||||
|
token_budget = 8000
|
||||||
|
dependency_depth = 3
|
||||||
|
```
|
||||||
|
|
||||||
|
Rendering sections are optional. Profiles and limits may also be omitted; runtime defaults then
|
||||||
|
apply. The required top-level fields are `schema_version`, `project_id`, `title`, `adapter`,
|
||||||
|
`sources`, `derived`, `changesets`, and `graph`.
|
||||||
|
|
||||||
|
## Identity fields
|
||||||
|
|
||||||
|
`schema_version` must be `1`.
|
||||||
|
|
||||||
|
`project_id` is the stable machine identity. It is lowercase and may contain digits, dots,
|
||||||
|
underscores, and hyphens after its first character. Do not derive it from a mutable display title.
|
||||||
|
|
||||||
|
`title` is the human-readable project name.
|
||||||
|
|
||||||
|
`adapter` is `generic` for this file format. Project-owned adapters publish a validated
|
||||||
|
`adapter_id@adapter_version` identity through their runtime descriptor; changing adapter identity
|
||||||
|
invalidates incompatible derived state.
|
||||||
|
|
||||||
|
The descriptor's byte content contributes to a descriptor hash. Client fragments, launchers, index
|
||||||
|
evidence, and policy results use that hash to detect drift.
|
||||||
|
|
||||||
|
## Canonical sources
|
||||||
|
|
||||||
|
`sources.content_roots` lists the project-relative directories containing generic Markdown and TOML
|
||||||
|
nodes. Each path must resolve beneath the project root. Canonical content roots may not overlap the
|
||||||
|
derived cache.
|
||||||
|
|
||||||
|
`sources.authority_files` lists additional project-relative regular files whose content belongs to
|
||||||
|
the canonical project identity. They are not automatically parsed as nodes.
|
||||||
|
|
||||||
|
A Markdown node contains one TOML metadata block followed by Markdown content:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
+++
|
||||||
|
schema_version = 1
|
||||||
|
id = "architecture.overview"
|
||||||
|
title = "Architecture overview"
|
||||||
|
family = "architecture"
|
||||||
|
authority = "authoritative"
|
||||||
|
status = "current"
|
||||||
|
tags = ["architecture"]
|
||||||
|
summary = "Defines the top-level architecture and ownership."
|
||||||
|
+++
|
||||||
|
|
||||||
|
# Architecture overview
|
||||||
|
|
||||||
|
Describe systems, ownership, runtime flow, failure behavior, and proof.
|
||||||
|
```
|
||||||
|
|
||||||
|
Every node ID is project-wide and stable. Every relationship target must resolve. A TOML source may
|
||||||
|
contain multiple `[[nodes]]` records; proposal-enabled multi-node files need stable
|
||||||
|
`source_anchor` values where creation or movement requires an exact record boundary.
|
||||||
|
|
||||||
|
## Derived state
|
||||||
|
|
||||||
|
`derived.cache_root` owns disposable indexes, attestations, extraction caches, render receipts,
|
||||||
|
projection artifacts, and viewer registry state.
|
||||||
|
|
||||||
|
`derived.index` must be inside `derived.cache_root`. Canonical content and cache paths must not
|
||||||
|
overlap.
|
||||||
|
|
||||||
|
Derived state is not a backup. If it is deleted or rejected as corrupt, DocForge rebuilds it from
|
||||||
|
validated canonical sources.
|
||||||
|
|
||||||
|
## Changesets and writers
|
||||||
|
|
||||||
|
`changesets.root` is the confined proposal store. It must not overlap canonical content or the
|
||||||
|
derived cache.
|
||||||
|
|
||||||
|
Each `changesets.writers` entry declares:
|
||||||
|
|
||||||
|
- a stable writer `id`;
|
||||||
|
- the node `families` that writer may change;
|
||||||
|
- allowed `operations`: `create`, `update`, `move`, and/or `delete`.
|
||||||
|
|
||||||
|
The descriptor grant is necessary but not sufficient. A process must also select that writer at
|
||||||
|
startup, and canonical application requires a separately bound matching applier. Capability mode
|
||||||
|
does not broaden the descriptor grant. See [Policy precedence](POLICY_PRECEDENCE.md).
|
||||||
|
|
||||||
|
## Allowed relationships
|
||||||
|
|
||||||
|
`graph.allowed_relations` is the exact project vocabulary accepted on edges. It must be nonempty.
|
||||||
|
Relationship names are stable IDs. DocForge rejects relationships that are not declared and gives
|
||||||
|
`depends_on` additional cycle validation.
|
||||||
|
|
||||||
|
The core does not reinterpret a custom relationship just because its spelling resembles a known
|
||||||
|
term. Task-context retrieval classifies only the documented versioned aliases and preserves
|
||||||
|
unknown allowed relationships as `unclassified`.
|
||||||
|
|
||||||
|
## Context profiles
|
||||||
|
|
||||||
|
Each `profiles` entry defines one bounded context compilation:
|
||||||
|
|
||||||
|
- `id` selects the profile;
|
||||||
|
- `families` and `statuses` filter eligible nodes;
|
||||||
|
- `required_nodes` names stable nodes that must be present;
|
||||||
|
- `token_budget` limits compiled content;
|
||||||
|
- `dependency_depth` bounds relationship expansion.
|
||||||
|
|
||||||
|
Profiles choose derived retrieval scope. They do not change node authority or writer permissions.
|
||||||
|
|
||||||
|
## Limits
|
||||||
|
|
||||||
|
Positive limits bound input, graph, retrieval, proposal, and rendering work. Current defaults are:
|
||||||
|
|
||||||
|
- `max_source_bytes = 1000000`
|
||||||
|
- `max_nodes = 10000`
|
||||||
|
- `max_query_chars = 500`
|
||||||
|
- `max_results = 100`
|
||||||
|
- `max_traversal_depth = 8`
|
||||||
|
- `max_context_tokens = 32000`
|
||||||
|
- `max_tool_output_chars = 200000`
|
||||||
|
- `max_changesets = 1000`
|
||||||
|
- `max_changeset_operations = 100`
|
||||||
|
- `max_changeset_bytes = 1000000`
|
||||||
|
- `max_render_views = 100`
|
||||||
|
- `max_template_bytes = 1000000`
|
||||||
|
- `max_render_bytes = 1000000`
|
||||||
|
|
||||||
|
Smaller project limits are useful policy. They cannot widen fixed internal worker, package,
|
||||||
|
response, or cache ceilings. In particular, detached renderer transfer has its own fixed boundary
|
||||||
|
even if a compatibility descriptor retains a larger `max_render_bytes`.
|
||||||
|
|
||||||
|
## Manual rendering
|
||||||
|
|
||||||
|
The optional `render` section declares:
|
||||||
|
|
||||||
|
- one confined `template_root`;
|
||||||
|
- one isolated `preview_root`;
|
||||||
|
- one or more stable views.
|
||||||
|
|
||||||
|
Each view uses the built-in `generic_html` renderer, a template beneath `template_root`, one
|
||||||
|
declared output path, a title, and a family filter. Paths may not overlap canonical content,
|
||||||
|
authority files, changesets, cache, templates, or previews in unsafe ways.
|
||||||
|
|
||||||
|
Templates are inert UTF-8 files with a fixed token vocabulary. They cannot select executable
|
||||||
|
renderers or commands. See [Rendering and visualization](RENDERING_AND_VISUALIZATION.md).
|
||||||
|
|
||||||
|
## Portable graph rendering
|
||||||
|
|
||||||
|
The optional `graph_render` section declares an output root and one or more
|
||||||
|
`portable_graph_html` views. Each view selects exactly one:
|
||||||
|
|
||||||
|
- `root`, an exact stable node ID; or
|
||||||
|
- `query`, a bounded metadata-only lexical seed.
|
||||||
|
|
||||||
|
It may then restrict families, relations, authorities, statuses, and tags, plus depth, node, edge,
|
||||||
|
and work limits. `initial_mode` is `nodes`, `flow`, or `web`. Portable graph contract version 1
|
||||||
|
requires `include_logic = false`.
|
||||||
|
|
||||||
|
The declared `output` is relative to `graph_render.output_root`.
|
||||||
|
|
||||||
|
## Paths and confinement
|
||||||
|
|
||||||
|
Descriptor paths are project-relative. Absolute paths and parent traversal are rejected. Runtime
|
||||||
|
validation also rejects symlink escapes, unexpected file types, unsafe overlap, changing path
|
||||||
|
identity during sensitive reads or publication, and derived outputs outside their declared roots.
|
||||||
|
|
||||||
|
The explicit CLI `--project-root` is the only project selector. The MCP process binds it at startup
|
||||||
|
and exposes no project-switching tool.
|
||||||
|
|
||||||
|
## Validate changes safely
|
||||||
|
|
||||||
|
After editing the descriptor:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docforge --project-root /absolute/path/MyProject validate
|
||||||
|
docforge --project-root /absolute/path/MyProject reindex
|
||||||
|
docforge --project-root /absolute/path/MyProject check
|
||||||
|
```
|
||||||
|
|
||||||
|
Descriptor or adapter implementation drift makes a project-owned running process fail closed; start
|
||||||
|
a fresh process after changing those boundaries.
|
||||||
|
|
||||||
|
For first-time creation, prefer the create-only [New-project
|
||||||
|
quickstart](NEW_PROJECT_QUICKSTART.md). For full invariants, read the [Core contract](CONTRACT.md).
|
||||||
|
|
@ -5,6 +5,11 @@ DocForge onboarding has two separate outcomes:
|
||||||
1. A generic manual can be configured, indexed, rendered, visualized, and exposed through the MCP.
|
1. A generic manual can be configured, indexed, rendered, visualized, and exposed through the MCP.
|
||||||
2. A source graph additionally requires one validated language frontend per source language.
|
2. A source graph additionally requires one validated language frontend per source language.
|
||||||
|
|
||||||
|
DocForge ships narrow fixed references for Python, JavaScript, TypeScript, and C++. They are useful
|
||||||
|
for syntax-scoped projects and adoption proof, but production semantic requirements may still
|
||||||
|
require a project-owned compiler or language-service adapter. Review
|
||||||
|
[Reference Adapters](REFERENCE_ADAPTERS.md) before selecting a frontend.
|
||||||
|
|
||||||
The onboarding command never claims that source semantics exist merely because it found source
|
The onboarding command never claims that source semantics exist merely because it found source
|
||||||
files. It reports each detected language as `adapter_required` until a project integration supplies
|
files. It reports each detected language as `adapter_required` until a project integration supplies
|
||||||
and proves that frontend.
|
and proves that frontend.
|
||||||
|
|
@ -64,6 +69,32 @@ immediately usable through the generic CLI, viewer, and MCP.
|
||||||
The starter overview records detected languages and states that the source graph is unavailable
|
The starter overview records detected languages and states that the source graph is unavailable
|
||||||
until a language frontend passes the adapter proof. That limitation is deliberate.
|
until a language frontend passes the adapter proof. That limitation is deliberate.
|
||||||
|
|
||||||
|
## Configure a fixed reference adapter
|
||||||
|
|
||||||
|
The generic onboarding scaffold and fixed reference configuration are separate project routes.
|
||||||
|
For a syntax-scoped reference project, create `.docforge/reference-adapter.toml`:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
schema_version = 1
|
||||||
|
project_id = "my-python-project"
|
||||||
|
title = "My Python project"
|
||||||
|
language = "python"
|
||||||
|
source_roots = ["src"]
|
||||||
|
```
|
||||||
|
|
||||||
|
Then start the fixed read-only binding:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -I -m docforge.reference_mcp \
|
||||||
|
--project-root /absolute/path/MyProject \
|
||||||
|
--capability-mode read
|
||||||
|
```
|
||||||
|
|
||||||
|
The configuration selects only a fixed in-repository provider and cannot name a command or custom
|
||||||
|
module. JavaScript, TypeScript, and C++ require their respective optional extras; C++ also requires
|
||||||
|
`compilation_database = "compile_commands.json"`. The complete configuration and supported-fact
|
||||||
|
contract are in [Reference Adapters](REFERENCE_ADAPTERS.md).
|
||||||
|
|
||||||
## Complete onboarding checklist
|
## Complete onboarding checklist
|
||||||
|
|
||||||
### 1. Repository assessment
|
### 1. Repository assessment
|
||||||
|
|
@ -118,6 +149,13 @@ For every source language:
|
||||||
- [ ] Define dependency discovery.
|
- [ ] Define dependency discovery.
|
||||||
- [ ] State unsupported semantic facts explicitly.
|
- [ ] State unsupported semantic facts explicitly.
|
||||||
|
|
||||||
|
Decide whether the project needs production semantic evidence or the narrower syntax-only
|
||||||
|
reference scope. The Python reference publishes only project-local imports. JavaScript and
|
||||||
|
TypeScript publish only project-local static relative imports and re-exports. The C++ reference
|
||||||
|
publishes only directly resolvable project-local quoted includes and does not run a compiler. None
|
||||||
|
of those references resolves calls, inheritance, types, symbols, runtime behavior, or semantic
|
||||||
|
ownership.
|
||||||
|
|
||||||
All frontends emit the same DocForge contracts:
|
All frontends emit the same DocForge contracts:
|
||||||
|
|
||||||
- `AdapterManifest` inventories fingerprinted extraction units and dependencies.
|
- `AdapterManifest` inventories fingerprinted extraction units and dependencies.
|
||||||
|
|
@ -151,6 +189,11 @@ this checklist summarizes.
|
||||||
- [ ] Assign shared symbols to one deterministic source contribution.
|
- [ ] Assign shared symbols to one deterministic source contribution.
|
||||||
- [ ] Record compiler-derived project include dependencies.
|
- [ ] Record compiler-derived project include dependencies.
|
||||||
|
|
||||||
|
These are production semantic-adapter expectations. The built-in C++ reference uses
|
||||||
|
`compile_commands.json` only as bounded translation-unit inventory and fingerprint evidence. It
|
||||||
|
parses commands and arguments as inert data, executes no command or compiler, and does not claim
|
||||||
|
compiler include semantics, symbol ownership, or a Clang-derived graph.
|
||||||
|
|
||||||
#### Rust
|
#### Rust
|
||||||
|
|
||||||
- [ ] Read the Cargo workspace and package graph.
|
- [ ] Read the Cargo workspace and package graph.
|
||||||
|
|
@ -240,6 +283,10 @@ from filenames.
|
||||||
Done when a new session can identify and retrieve the correct project without being told its file
|
Done when a new session can identify and retrieve the correct project without being told its file
|
||||||
layout.
|
layout.
|
||||||
|
|
||||||
|
The fixed reference server is read-only. Generic and custom-adapter client generation, including
|
||||||
|
the immutable custom launcher boundary, is documented in
|
||||||
|
[Agent Integration](AGENT_INTEGRATION.md).
|
||||||
|
|
||||||
### 10. Operating guide and maintenance
|
### 10. Operating guide and maintenance
|
||||||
|
|
||||||
- [ ] Record the authority and progressive-reading order.
|
- [ ] Record the authority and progressive-reading order.
|
||||||
|
|
@ -286,14 +333,18 @@ to one configured project root.
|
||||||
DocForge does not let an MCP call install dependencies, run project builds, modify Git, deploy, or
|
DocForge does not let an MCP call install dependencies, run project builds, modify Git, deploy, or
|
||||||
publish. A project integration may use its own normal development workflow for those actions.
|
publish. A project integration may use its own normal development workflow for those actions.
|
||||||
|
|
||||||
## Frontend packaging direction
|
## Reference and production frontend boundaries
|
||||||
|
|
||||||
Reusable language frontends should be separate packages or project-owned adapters over the public
|
The public authoring namespace is `docforge.adapter_sdk`. Project-owned and separately distributed
|
||||||
DocForge contracts. They must not put language-specific rules into the graph, index, viewer, or MCP
|
production frontends should build on that contract without putting language-specific rules into
|
||||||
core.
|
the graph, index, viewer, or MCP core.
|
||||||
|
|
||||||
Worldforge is the first complete C++ reference integration. A reusable C++ package should be
|
The in-repository Python, JavaScript, TypeScript, and C++ adapters are reference implementations.
|
||||||
extracted only after that integration proves stable ownership, compiler dependency invalidation,
|
Their syntax-scoped behavior is useful without becoming a claim that every project in those
|
||||||
and complete/incremental equivalence. Rust and Java frontends should then implement the same
|
languages has complete semantic coverage. A compiler-backed production C++ adapter may resolve
|
||||||
contract using their authoritative build and language tooling rather than copying C++ extraction
|
build flags, calls, types, inheritance, include semantics, and ownership when it can prove those
|
||||||
rules.
|
facts. The reference C++ adapter does none of that and is not a Clang semantic adapter.
|
||||||
|
|
||||||
|
New Rust, Java, or other frontends should use their authoritative build and language tooling and
|
||||||
|
must pass the graph-plus-Logic complete/incremental contract in the
|
||||||
|
[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md).
|
||||||
|
|
|
||||||
86
docs/RECOVERY_AND_PERFORMANCE.md
Normal file
86
docs/RECOVERY_AND_PERFORMANCE.md
Normal file
|
|
@ -0,0 +1,86 @@
|
||||||
|
# Recovery and performance
|
||||||
|
|
||||||
|
DocForge keeps canonical project evidence separate from disposable indexes, caches, receipts, and
|
||||||
|
rendered projections. Recovery rebuilds derived state from current authority; it does not rewrite
|
||||||
|
canonical content to make a cache look valid.
|
||||||
|
|
||||||
|
For the underlying boundaries, see [core authority](CORE_CONCEPTS_AND_AUTHORITY.md) and the
|
||||||
|
[security model](SECURITY.md).
|
||||||
|
|
||||||
|
## Recovery order
|
||||||
|
|
||||||
|
Use the narrowest verified recovery:
|
||||||
|
|
||||||
|
1. Run `docforge check` or the corresponding MCP status operation.
|
||||||
|
2. Run `docforge sync` to repair a missing or stale disposable index when safe.
|
||||||
|
3. Run `docforge reindex` for an explicit complete rebuild.
|
||||||
|
4. Restart a long-running MCP or viewer binding after descriptor or adapter implementation drift.
|
||||||
|
5. Regenerate a derived client fragment, command reference, preview, manual, or portable graph
|
||||||
|
from current project evidence.
|
||||||
|
|
||||||
|
Never delete or rewrite canonical sources, active changesets, or reviewed hashes as cache cleanup.
|
||||||
|
|
||||||
|
## Extraction-cache recovery
|
||||||
|
|
||||||
|
Incremental adapter contributions are stored in a version-1 disposable extraction cache. Reads are
|
||||||
|
no-follow, regular-file-only, identity-checked, bounded to 64,000,000 bytes and 10,000 source
|
||||||
|
records, and fail to a cache miss on malformed or incompatible data. Publication is atomic.
|
||||||
|
|
||||||
|
After a miss, current manifest sources are extracted again and the complete assembly is validated.
|
||||||
|
The active SQLite generation remains authoritative for reads until a new verified index is
|
||||||
|
published. An extraction cache may therefore be safely ahead of the last index; the two files do
|
||||||
|
not pretend to be one transaction.
|
||||||
|
|
||||||
|
## Index and receipt recovery
|
||||||
|
|
||||||
|
A validated SQLite index is a generation-pinned derived snapshot. Missing, corrupt, unattested, or
|
||||||
|
stale indexes rebuild from the complete project or adapter oracle. Generation receipts and render
|
||||||
|
receipts are post-publication evidence. Failure to write a receipt after a committed artifact is
|
||||||
|
reported as degraded committed success, not as permission to repeat a mutation.
|
||||||
|
|
||||||
|
The live viewer pins one validated index identity. Index replacement makes the running snapshot
|
||||||
|
stale and causes a later visualize request to start a fresh worker.
|
||||||
|
|
||||||
|
## Proposal and application recovery
|
||||||
|
|
||||||
|
Hash or base conflicts are not cache failures. Retrieve the current changeset and diff, then
|
||||||
|
review the new exact hash. Rebase is allowed only when every touched node, relationship, source,
|
||||||
|
permission, and graph invariant still matches. A content conflict requires a new proposal.
|
||||||
|
|
||||||
|
If a canonical serializer fails its round-trip check, use its reported rollback state. Do not
|
||||||
|
reapply a changeset whose application may already have committed. See
|
||||||
|
[migrating from v1](MIGRATING_FROM_V1.md) for rollback planning.
|
||||||
|
|
||||||
|
## Milestone 4 scale evidence
|
||||||
|
|
||||||
|
The maintained Python reference benchmark creates 334 Python source files and produces:
|
||||||
|
|
||||||
|
- 1,002 primary nodes and 1,001 primary edges.
|
||||||
|
- 334 Logic projections with 2,338 Logic nodes and 2,338 Logic edges.
|
||||||
|
- Exact complete/incremental primary graph and Logic equality.
|
||||||
|
- Exact output after corrupt extraction-cache recovery and corrupt-index recovery.
|
||||||
|
- Zero `ast.parse` calls and zero `extract_source` calls during a warm build.
|
||||||
|
|
||||||
|
On the frozen Milestone 4 candidate, operation p95 values were 1.266 to 1.898 seconds. Per-operation
|
||||||
|
traced peaks were about 61 to 67 MiB, and process high-water was 78,798,848 bytes. These are
|
||||||
|
regression measurements from one machine, not universal latency promises. The machine-readable
|
||||||
|
record is `../benchmarks/milestone4-2026-07-29.json`; detailed method and hashes are in
|
||||||
|
[the Milestone 4 baseline](MILESTONE_4_BASELINE.md).
|
||||||
|
|
||||||
|
The zero-parser claim is deliberately Python-only. Focused JavaScript and TypeScript tests prove
|
||||||
|
their manifests avoid Tree-sitter. The C++ reference manifest currently uses Tree-sitter while
|
||||||
|
discovering bounded quoted includes, so a warm C++ cache hit is not evidence of zero parser work.
|
||||||
|
|
||||||
|
## Maintained gates
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make gate
|
||||||
|
make adoption-m4
|
||||||
|
make benchmark-m4-full
|
||||||
|
make docs-check
|
||||||
|
```
|
||||||
|
|
||||||
|
The main gate includes smoke benchmarks. Full milestone evidence is recorded separately from a
|
||||||
|
clean candidate so smoke or dirty-tree results cannot become release claims.
|
||||||
149
docs/REFERENCE_ADAPTERS.md
Normal file
149
docs/REFERENCE_ADAPTERS.md
Normal file
|
|
@ -0,0 +1,149 @@
|
||||||
|
# Reference adapters
|
||||||
|
|
||||||
|
DocForge includes fixed, syntax-scoped reference adapters for Python, JavaScript, TypeScript, and
|
||||||
|
C++. They demonstrate the public adapter contract, deterministic complete and incremental
|
||||||
|
publication, function Logic, cache recovery, and a runnable read-only MCP binding. They are not
|
||||||
|
compiler or language-service replacements.
|
||||||
|
|
||||||
|
Use a reference adapter when its deliberately narrow graph is sufficient or when proving a fresh
|
||||||
|
DocForge integration. Use the [Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md) for a
|
||||||
|
production adapter that needs resolved symbols, calls, inheritance, types, build semantics, or
|
||||||
|
semantic ownership.
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
The Python reference adapter uses the standard library and is available in the base wheel. The
|
||||||
|
other frontends are separate optional extras:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m pip install docforge
|
||||||
|
python -m pip install "docforge[javascript]"
|
||||||
|
python -m pip install "docforge[typescript]"
|
||||||
|
python -m pip install "docforge[cpp]"
|
||||||
|
```
|
||||||
|
|
||||||
|
Install `docforge[languages]` only when one environment intentionally needs all three Tree-sitter
|
||||||
|
frontends. JavaScript and TypeScript use distinct grammar packages and distinct extras.
|
||||||
|
|
||||||
|
## Fixed project configuration
|
||||||
|
|
||||||
|
The runnable reference binding reads exactly
|
||||||
|
`.docforge/reference-adapter.toml` below the selected project root. The configuration is data only:
|
||||||
|
it cannot name a provider, Python module, command, arguments, working directory, environment, or
|
||||||
|
discovery rule.
|
||||||
|
|
||||||
|
Reference adapter configuration `.docforge/reference-adapter.toml`:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
schema_version = 1
|
||||||
|
project_id = "example-python"
|
||||||
|
title = "Example Python project"
|
||||||
|
language = "python"
|
||||||
|
source_roots = ["src"]
|
||||||
|
```
|
||||||
|
|
||||||
|
`language` is one of `python`, `javascript`, `typescript`, or `cpp`. Source roots must be sorted,
|
||||||
|
unique, non-overlapping project-relative directories outside `.docforge`. The configuration,
|
||||||
|
roots, and inventoried source files must be regular confined paths without symlink traversal. The
|
||||||
|
configuration is limited to 65,536 bytes, at most 64 source roots, and 65,536 examined inventory
|
||||||
|
entries.
|
||||||
|
|
||||||
|
C++ additionally requires one explicit project-confined compilation database:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
schema_version = 1
|
||||||
|
project_id = "example-cpp"
|
||||||
|
title = "Example C++ project"
|
||||||
|
language = "cpp"
|
||||||
|
source_roots = ["include", "src"]
|
||||||
|
compilation_database = "compile_commands.json"
|
||||||
|
```
|
||||||
|
|
||||||
|
`compilation_database` is forbidden for the other three languages.
|
||||||
|
|
||||||
|
## Published scope
|
||||||
|
|
||||||
|
All four adapters publish deterministic file, module or translation-unit, class or struct, and
|
||||||
|
function nodes where their syntax supports those categories. They publish lexical `contains`
|
||||||
|
relationships, a narrow set of project-local `depends_on` relationships, and function-scoped
|
||||||
|
Logic. Calls that appear inside Logic are syntax steps, not resolved symbol relationships.
|
||||||
|
|
||||||
|
### Python
|
||||||
|
|
||||||
|
Python uses `ast` from the standard library for source extraction and Logic. Manifest construction
|
||||||
|
does not build a Python AST; it tokenizes imports only far enough to publish dependencies that
|
||||||
|
resolve to another module in the declared source inventory.
|
||||||
|
|
||||||
|
It does not import or execute project code. It does not resolve dynamic imports, calls,
|
||||||
|
inheritance, imported symbols, types, overloads, re-exports, decorators, metaclasses, descriptors,
|
||||||
|
or runtime-generated behavior.
|
||||||
|
|
||||||
|
### JavaScript and TypeScript
|
||||||
|
|
||||||
|
JavaScript uses the optional `tree-sitter-javascript` grammar. TypeScript uses the distinct
|
||||||
|
optional `tree-sitter-typescript` grammar. Their focused incremental tests prove that manifest
|
||||||
|
construction and an unchanged warm build do not invoke the Tree-sitter extraction parser.
|
||||||
|
|
||||||
|
Only static relative imports and re-exports that resolve to another inventoried source file become
|
||||||
|
dependencies. Dynamic `import()`, `require()`, bare package specifiers, aliases, `tsconfig` paths,
|
||||||
|
loader hooks, types, interfaces, overloads, calls, inheritance, symbols, and runtime behavior are
|
||||||
|
not resolved. The adapters never import, compile, transpile, or execute project code.
|
||||||
|
|
||||||
|
### C++
|
||||||
|
|
||||||
|
C++ treats `compile_commands.json` as the authoritative bounded translation-unit inventory and as
|
||||||
|
fingerprint evidence. Commands, arguments, directories, and output fields are parsed as inert
|
||||||
|
data. The adapter executes no compiler, build tool, command, project binary, or project code.
|
||||||
|
|
||||||
|
The reference C++ manifest uses the optional `tree-sitter-cpp` grammar to parse inventoried sources
|
||||||
|
and discover quoted includes. It publishes a dependency only when that quoted include resolves
|
||||||
|
directly to a real project-local header in the declared roots. Angle-bracket includes, compiler
|
||||||
|
include paths, frameworks, generated headers, compiler-provided headers, conditional compilation,
|
||||||
|
and macro expansion are omitted.
|
||||||
|
|
||||||
|
This is not a Clang semantic adapter. It does not claim resolved calls, types, templates, aliases,
|
||||||
|
concepts, references, inheritance, overload ownership, out-of-line semantic ownership, macro
|
||||||
|
semantics, or compiler include semantics. A warm C++ extraction-cache hit is therefore not evidence
|
||||||
|
that manifest construction performed zero parser work.
|
||||||
|
|
||||||
|
## Complete and incremental proof
|
||||||
|
|
||||||
|
Each reference adapter implements:
|
||||||
|
|
||||||
|
- `load_projection()` for the complete primary graph compatibility oracle;
|
||||||
|
- `load_complete_assembly()` for the complete graph-plus-Logic oracle;
|
||||||
|
- `load_manifest()` and `extract_source()` for incremental extraction; and
|
||||||
|
- deterministic assembly of the complete current contribution set.
|
||||||
|
|
||||||
|
The maintained fixtures prove complete determinism, exact complete/incremental graph and Logic
|
||||||
|
parity, reverse-dependency invalidation, additions and deletions, corrupt extraction-cache
|
||||||
|
recovery, confinement, and exact unsupported-fact inventories. See
|
||||||
|
[Incremental Adapter Indexing](INCREMENTAL_INDEXING.md) for the cache contract and
|
||||||
|
[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for compatibility and policy limits.
|
||||||
|
|
||||||
|
Current fixture evidence is:
|
||||||
|
|
||||||
|
| Language | Primary nodes | Relationships | Logic projections |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| Python | 14 | 13 | 5 |
|
||||||
|
| JavaScript | 14 | 14 | 5 |
|
||||||
|
| TypeScript | 13 | 14 | 4 |
|
||||||
|
| C++ | 17 | 16 | 5 |
|
||||||
|
|
||||||
|
These are regression-fixture shapes, not promises for arbitrary repositories.
|
||||||
|
|
||||||
|
## Read-only reference server
|
||||||
|
|
||||||
|
Start the fixed server with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -I -m docforge.reference_mcp \
|
||||||
|
--project-root /absolute/path/to/project \
|
||||||
|
--capability-mode read
|
||||||
|
```
|
||||||
|
|
||||||
|
The server selects one of the four in-repository providers solely from the validated fixed
|
||||||
|
configuration. Its cache stays below `.docforge/cache/reference-adapter/<language>`. It registers
|
||||||
|
the 21-tool read surface and no proposal or application tools. See
|
||||||
|
[Agent Integration](AGENT_INTEGRATION.md) for generated client fragments and
|
||||||
|
[MCP Boundary](MCP_CONTRACT.md) for the exact tool contract.
|
||||||
233
docs/RENDERING_AND_VISUALIZATION.md
Normal file
233
docs/RENDERING_AND_VISUALIZATION.md
Normal file
|
|
@ -0,0 +1,233 @@
|
||||||
|
# Rendering and visualization
|
||||||
|
|
||||||
|
DocForge has three independent ways to present one validated graph generation:
|
||||||
|
|
||||||
|
```text
|
||||||
|
validated generation
|
||||||
|
├── manual plan → immutable package → detached HTML renderer → declared manual output
|
||||||
|
├── graph plan → immutable package → detached graph renderer → portable static artifact
|
||||||
|
└── pinned index → managed loopback viewer → interactive Nodes, Flow, Web, and lazy Logic
|
||||||
|
```
|
||||||
|
|
||||||
|
They share validated facts but not authority, publication, or lifecycle. A manual render does not
|
||||||
|
publish a portable graph. A portable graph does not start the live viewer. None is canonical
|
||||||
|
project content.
|
||||||
|
|
||||||
|
## Manual rendering
|
||||||
|
|
||||||
|
A generic descriptor may declare a manual template root, isolated preview root, and stable views:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[render]
|
||||||
|
template_root = ".docforge/templates"
|
||||||
|
preview_root = ".docforge/previews"
|
||||||
|
|
||||||
|
[[render.views]]
|
||||||
|
id = "manual"
|
||||||
|
renderer = "generic_html"
|
||||||
|
template = "manual.html"
|
||||||
|
output = ".docforge/rendered/manual.html"
|
||||||
|
title = "My Project Manual"
|
||||||
|
families = ["architecture", "operations", "system"]
|
||||||
|
```
|
||||||
|
|
||||||
|
The renderer name is fixed to `generic_html`. Templates are confined UTF-8 assets with a fixed
|
||||||
|
token vocabulary. They cannot name commands, Python modules, executable renderers, or arbitrary
|
||||||
|
publication paths. Raw HTML in canonical Markdown is disabled by the pinned CommonMark path.
|
||||||
|
|
||||||
|
Plan and render identity covers the canonical source hash, optional changeset hash, selected nodes
|
||||||
|
and relationships, view configuration, template hash, renderer contract, and parser version.
|
||||||
|
|
||||||
|
Use:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docforge --project-root "$PROJECT" render-status
|
||||||
|
docforge --project-root "$PROJECT" render-status manual --deep
|
||||||
|
docforge --project-root "$PROJECT" render manual
|
||||||
|
docforge --project-root "$PROJECT" preview CHANGESET_ID manual
|
||||||
|
```
|
||||||
|
|
||||||
|
Normal status verifies bounded source, configuration, template, output, renderer, and receipt
|
||||||
|
identities without reconstructing output. `--deep` explicitly runs the side-effect-free full-render
|
||||||
|
oracle. A changeset preview writes only to the isolated preview root.
|
||||||
|
|
||||||
|
Manual `auto` means that a successful canonical application owns regeneration of every declared
|
||||||
|
manual view. It does not mean background rendering, and ordinary standalone CLI render cannot
|
||||||
|
select `auto`; use `explicit`.
|
||||||
|
|
||||||
|
## Portable graph rendering
|
||||||
|
|
||||||
|
Portable graph configuration is separate:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[graph_render]
|
||||||
|
output_root = ".docforge/portable-graph"
|
||||||
|
|
||||||
|
[[graph_render.views]]
|
||||||
|
id = "architecture"
|
||||||
|
renderer = "portable_graph_html"
|
||||||
|
output = "architecture.html"
|
||||||
|
title = "Architecture"
|
||||||
|
root = "architecture.overview"
|
||||||
|
initial_mode = "web"
|
||||||
|
depth = 3
|
||||||
|
max_nodes = 250
|
||||||
|
max_edges = 1000
|
||||||
|
max_work = 100000
|
||||||
|
families = ["architecture", "system"]
|
||||||
|
relations = ["depends_on", "owns", "calls", "reads", "writes", "tested_by", "relates_to"]
|
||||||
|
authorities = []
|
||||||
|
statuses = ["current", "active", "verified"]
|
||||||
|
tags = []
|
||||||
|
include_logic = false
|
||||||
|
```
|
||||||
|
|
||||||
|
A view selects exactly one exact `root` node or bounded metadata-only lexical `query`. Exact
|
||||||
|
filters and node, edge, depth, and work limits close the selection. The initial mode is `nodes`,
|
||||||
|
`flow`, or `web`. Portable graph contract version 1 excludes function-scoped Logic.
|
||||||
|
|
||||||
|
Use:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docforge --project-root "$PROJECT" graph-plan architecture
|
||||||
|
docforge --project-root "$PROJECT" graph-render architecture
|
||||||
|
docforge --project-root "$PROJECT" graph-render-status architecture
|
||||||
|
```
|
||||||
|
|
||||||
|
`graph-plan` validates and returns the generation-pinned plan without publication. `graph-render`
|
||||||
|
is an explicit local CLI publication action. It commits a content-addressed artifact, renderer
|
||||||
|
receipt, and bounded generation/view manifest in that order; the manifest is the publication
|
||||||
|
commit. `graph-render-status` checks only bounded committed evidence and never plans or renders.
|
||||||
|
|
||||||
|
MCP may plan and inspect portable graph evidence, but it does not publish the portable artifact.
|
||||||
|
Use the explicit project-local CLI command for publication.
|
||||||
|
|
||||||
|
Portable output is a complete static artifact. JavaScript is progressive enhancement, not a
|
||||||
|
requirement for the graph facts to be present.
|
||||||
|
|
||||||
|
## Live viewer
|
||||||
|
|
||||||
|
The live viewer is a managed, read-only browser over one generation-pinned validated SQLite file.
|
||||||
|
Install the per-user manager once:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docforge-viewer-manager install-user-service
|
||||||
|
```
|
||||||
|
|
||||||
|
Then operate a project viewer:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docforge --project-root "$PROJECT" visualize
|
||||||
|
docforge --project-root "$PROJECT" visualize --node architecture.overview
|
||||||
|
docforge --project-root "$PROJECT" visualize --query persistence
|
||||||
|
docforge --project-root "$PROJECT" visualization-status
|
||||||
|
docforge --project-root "$PROJECT" visualization-stop
|
||||||
|
```
|
||||||
|
|
||||||
|
`visualize` accepts only an optional stable node ID or lexical query, bounded traversal depth, and
|
||||||
|
the local `--no-open` presentation choice. It does not accept a project root override, database
|
||||||
|
path, SQL, template, command, renderer, bind address, or module.
|
||||||
|
|
||||||
|
The HTTP listener binds to `127.0.0.1` on an operating-system-selected port. A random token is part
|
||||||
|
of every accepted path. The server supports only `GET` and `HEAD`, sets no-store and restrictive
|
||||||
|
browser security headers, and has no write, project-selection, arbitrary-query, or static
|
||||||
|
filesystem endpoint.
|
||||||
|
|
||||||
|
The viewer is a snapshot. Index replacement or alteration makes that snapshot fail closed; start a
|
||||||
|
new visualization to use a new validated generation. Source inspection reads only project-confined
|
||||||
|
source evidence bound to the pinned generation.
|
||||||
|
|
||||||
|
## Nodes, Flow, Web, and Logic
|
||||||
|
|
||||||
|
The interactive viewer offers four complementary projections:
|
||||||
|
|
||||||
|
- **Nodes** shows a bounded relation-neutral incoming and outgoing neighborhood.
|
||||||
|
- **Flow** presents semantic contributors toward the focus. Prerequisite-style stored
|
||||||
|
relationships may be reversed for presentation without changing stored direction.
|
||||||
|
- **Web** expands the convergence picture with contributors, callers, containers, members, and
|
||||||
|
contextual relationships.
|
||||||
|
- **Logic** loads a function-scoped control-flow projection only when requested.
|
||||||
|
|
||||||
|
Logic supports the explicit control paths published by Python, JavaScript, TypeScript, and C++
|
||||||
|
integrations. It stays outside the primary graph, portable graph version 1, search, and
|
||||||
|
generation-diff receipts.
|
||||||
|
|
||||||
|
Hiding a node is presentation-only. In Flow and Web, ancestors without another path to the focus
|
||||||
|
are removed. In Logic, an omitted-path bridge preserves downstream readability. Restore reverses
|
||||||
|
the presentation change; neither action mutates the index.
|
||||||
|
|
||||||
|
## Immutable plan and worker boundary
|
||||||
|
|
||||||
|
Manual and graph plans are versioned, canonical JSON with deterministic ordering and fixed
|
||||||
|
structural and byte limits. They contain selected graph facts and bounded content, but no live
|
||||||
|
project object, SQLite handle, absolute project or index path, arbitrary query, command,
|
||||||
|
executable path, or caller-selected renderer module.
|
||||||
|
|
||||||
|
A projection package binds one plan to inert assets, fixed component versions, a closed built-in
|
||||||
|
renderer identity, and an exact artifact inventory. The detached worker:
|
||||||
|
|
||||||
|
- runs one fixed private module through isolated Python;
|
||||||
|
- uses a trusted working directory and sanitized environment;
|
||||||
|
- accepts one canonical newline-terminated JSON request;
|
||||||
|
- returns one bounded canonical JSON response;
|
||||||
|
- has a fixed renderer allowlist and timeout;
|
||||||
|
- cannot select graph facts, read project state, choose output paths, or mutate canonical files.
|
||||||
|
|
||||||
|
The package contract is bounded, and actual artifact transfer has a fixed 20,000,000-byte ceiling.
|
||||||
|
A larger descriptor `max_render_bytes` compatibility value does not widen that worker boundary.
|
||||||
|
|
||||||
|
## Manual fragment reuse
|
||||||
|
|
||||||
|
Manual fragments are disposable semantic cache records, not publication authority. A cold record
|
||||||
|
is accepted only after byte-exact comparison against a full detached render. On a warm hit, the
|
||||||
|
worker independently recomputes the expected fragment before reuse.
|
||||||
|
|
||||||
|
Corrupt, forged, stale, incompatible, individually oversized, or aggregate-oversized records fall
|
||||||
|
back to the complete full-render oracle. The current cache is bounded to 10,000 records and
|
||||||
|
64,000,000 bytes.
|
||||||
|
|
||||||
|
Fragment reuse is a correctness and recovery boundary. Do not promise a speedup without current
|
||||||
|
measurements.
|
||||||
|
|
||||||
|
## Projection policy
|
||||||
|
|
||||||
|
The version-2 projection policy is independent of capability mode:
|
||||||
|
|
||||||
|
```text
|
||||||
|
manual: auto | explicit | disabled
|
||||||
|
portable_graph: explicit | disabled
|
||||||
|
live_viewer: on-demand | disabled
|
||||||
|
```
|
||||||
|
|
||||||
|
Set global CLI flags before the subcommand:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docforge --project-root "$PROJECT" --manual-render-policy disabled render manual
|
||||||
|
docforge --project-root "$PROJECT" --portable-graph-policy disabled graph-plan architecture
|
||||||
|
docforge --project-root "$PROJECT" --live-viewer-policy disabled visualize
|
||||||
|
```
|
||||||
|
|
||||||
|
An active operation blocked by policy fails before hidden work. Receipt-only manual and portable
|
||||||
|
status remain available. Viewer status and explicit stop remain available when viewer startup is
|
||||||
|
disabled.
|
||||||
|
|
||||||
|
A non-disabled selection also requires its resource: declared manual configuration, declared
|
||||||
|
portable graph configuration, or viewer runtime. Manual `auto` additionally requires canonical
|
||||||
|
application. Read [Policy precedence](POLICY_PRECEDENCE.md) for exact composition.
|
||||||
|
|
||||||
|
## Failure and recovery
|
||||||
|
|
||||||
|
Render input drift detected before replacement fails without publishing a current receipt for stale
|
||||||
|
output. Portable graph publication uses content-addressed evidence and a final manifest commit.
|
||||||
|
Status never repairs implicitly.
|
||||||
|
|
||||||
|
Recovery is explicit:
|
||||||
|
|
||||||
|
- rerun a manual render from validated canonical state;
|
||||||
|
- use deep manual status for the full, side-effect-free equivalence oracle;
|
||||||
|
- rerun portable graph publication, or repair only from validated content-addressed evidence;
|
||||||
|
- stop and restart a stale live viewer against a current validated index.
|
||||||
|
|
||||||
|
Do not recover a derived output by editing its receipt or treating it as canonical. See [Recovery
|
||||||
|
and performance](RECOVERY_AND_PERFORMANCE.md), [Security](SECURITY.md), and the [Viewer
|
||||||
|
manager](VIEWER_MANAGER.md).
|
||||||
101
docs/SECURITY.md
Normal file
101
docs/SECURITY.md
Normal file
|
|
@ -0,0 +1,101 @@
|
||||||
|
# Security model
|
||||||
|
|
||||||
|
DocForge is a project-bound knowledge compiler. Its security boundary is an explicit project root,
|
||||||
|
closed configuration, bounded data, and exact identities. It is not a general process sandbox.
|
||||||
|
|
||||||
|
Start with [core authority](CORE_CONCEPTS_AND_AUTHORITY.md), then use
|
||||||
|
[policy precedence](POLICY_PRECEDENCE.md) to decide which capabilities a server should expose.
|
||||||
|
|
||||||
|
## Project and path confinement
|
||||||
|
|
||||||
|
Descriptors, reference-adapter configurations, canonical sources, authority files, templates,
|
||||||
|
changesets, caches, indexes, previews, and declared outputs are resolved against one project root.
|
||||||
|
DocForge rejects absolute paths where only project-relative paths are allowed, parent traversal,
|
||||||
|
symbolic-link escapes, unsafe file types, and protected-root overlap. Important reads use
|
||||||
|
no-follow file descriptors and compare file identity before and after reading.
|
||||||
|
|
||||||
|
Confinement protects DocForge operations. It does not stop another process with repository access
|
||||||
|
from changing files. Long-running bindings revalidate descriptor and adapter implementation
|
||||||
|
identity and require a restart after drift.
|
||||||
|
|
||||||
|
## Untrusted project content
|
||||||
|
|
||||||
|
Documentation, source text, templates, adapter metadata, compiler-database entries, and changeset
|
||||||
|
content are data. They cannot redefine policy or instruct DocForge to execute a command. Generic
|
||||||
|
manual rendering disables raw HTML, accepts a fixed template vocabulary, and rejects script-like
|
||||||
|
content. Portable graph and manual workers accept validated inert packages and fixed built-in
|
||||||
|
renderer identities.
|
||||||
|
|
||||||
|
The C++ reference adapter reads `compile_commands.json` only as bounded translation-unit inventory
|
||||||
|
and fingerprint evidence. It never executes the recorded command, compiler, response file, or
|
||||||
|
project program.
|
||||||
|
|
||||||
|
## Adapter launcher boundary
|
||||||
|
|
||||||
|
`AdapterLauncherV1` contains one Python module name and project identity. It contains no command,
|
||||||
|
shell string, arbitrary argument list, working directory, environment, discovery rule, or callable
|
||||||
|
selector.
|
||||||
|
|
||||||
|
Custom modules must be installed top-level modules. An isolated `python -I` probe resolves the
|
||||||
|
module without importing it and requires its regular-file origin to remain inside the bound
|
||||||
|
project. The sole trusted dotted exception is the packaged `docforge.reference_mcp` module. Client
|
||||||
|
fragments use the exact validated interpreter and canonical fixed arguments with an empty
|
||||||
|
environment.
|
||||||
|
|
||||||
|
This proves that the declared module is resolvable and project-bound. It does not make arbitrary
|
||||||
|
module code safe. Project owners remain responsible for the implementation they install.
|
||||||
|
|
||||||
|
## Mutation boundary
|
||||||
|
|
||||||
|
Normal MCP and the fixed reference server are read-only. Proposal tools exist only when a
|
||||||
|
startup-bound writer is authorized by the descriptor. Canonical application exists only when a
|
||||||
|
matching applier is explicitly configured.
|
||||||
|
|
||||||
|
Every proposal append, rebase, abandonment, and application is hash-bound. Application requires
|
||||||
|
the exact changeset hash that was reviewed. Source identity, content hashes, permissions,
|
||||||
|
conflicts, graph validity, and serializer round trips are checked before success. DocForge never
|
||||||
|
turns prose approval into a fuzzy merge.
|
||||||
|
|
||||||
|
## Derived state and publication
|
||||||
|
|
||||||
|
SQLite indexes, source-generation receipts, extraction caches, render fragments, previews, and
|
||||||
|
portable artifacts are disposable. Corrupt, stale, foreign, oversized, or mismatched derived
|
||||||
|
state is rejected or rebuilt from current project evidence.
|
||||||
|
|
||||||
|
Generated command-reference publication serializes cooperating writers and uses no-clobber or
|
||||||
|
compare-and-swap publication. A raced target is restored or retained for recovery instead of being
|
||||||
|
silently discarded. Projection publication records when an artifact was committed but later
|
||||||
|
receipt verification degraded, so a completed mutation is never reported as an ordinary failure.
|
||||||
|
|
||||||
|
## Limits and denial-of-service resistance
|
||||||
|
|
||||||
|
Inputs, results, traversal, context, changesets, renders, worker protocols, manifests, adapter
|
||||||
|
assemblies, and extraction caches have explicit count and byte limits. Incremental extraction
|
||||||
|
caches are capped at 10,000 sources and 64,000,000 bytes. Adapter primary nodes use the project
|
||||||
|
`max_nodes` limit; edges and Logic have deterministic multipliers over that limit.
|
||||||
|
|
||||||
|
Limits reduce accidental and adversarial amplification. An in-process adapter can still allocate
|
||||||
|
memory before returning data, so only trusted project-owned adapter code should run in the server
|
||||||
|
process.
|
||||||
|
|
||||||
|
## Secrets and network behavior
|
||||||
|
|
||||||
|
DocForge does not copy the parent environment into generated client fragments or detached
|
||||||
|
projection workers. Doctor checks never return environment values. The live viewer binds to
|
||||||
|
loopback, uses an unguessable URL token, supports read-only methods, and serves no arbitrary
|
||||||
|
filesystem tree.
|
||||||
|
|
||||||
|
Project secrets must not be placed in canonical documentation, adapter configuration, compiler
|
||||||
|
databases, templates, or changesets. Repository release gates include secret scanning, but that
|
||||||
|
scan is not a substitute for credential hygiene.
|
||||||
|
|
||||||
|
## No-AST boundary
|
||||||
|
|
||||||
|
`--no-ast` is a binding policy that preserves the selected adapter and prohibits Logic publication
|
||||||
|
and retrieval. It is not a parser detector, filesystem sandbox, or promise that unrelated
|
||||||
|
processes cannot parse source. See [legacy and no-AST operation](LEGACY_AND_NO_AST.md).
|
||||||
|
|
||||||
|
## Reporting and recovery
|
||||||
|
|
||||||
|
Do not bypass a confinement, identity, policy, hash, or limit error. Preserve the failing evidence,
|
||||||
|
stop the affected binding, and follow [recovery and performance](RECOVERY_AND_PERFORMANCE.md).
|
||||||
|
|
@ -35,6 +35,8 @@ incremental methods while retaining the full loader as a fallback.
|
||||||
relationship keys, source inspection, branch-aware node hiding, panel resizing, zooming, and
|
relationship keys, source inspection, branch-aware node hiding, panel resizing, zooming, and
|
||||||
managed idle shutdown.
|
managed idle shutdown.
|
||||||
- A generic Markdown/TOML adapter plus contracts for deterministic project-owned adapters.
|
- A generic Markdown/TOML adapter plus contracts for deterministic project-owned adapters.
|
||||||
|
- A public adapter SDK, optional Python/JavaScript/TypeScript/C++ reference integrations, and one
|
||||||
|
fixed read-only reference MCP binding.
|
||||||
|
|
||||||
DocForge does not run shell commands from documentation, mutate Git, build an application, deploy,
|
DocForge does not run shell commands from documentation, mutate Git, build an application, deploy,
|
||||||
publish, choose a project globally, or cross project boundaries.
|
publish, choose a project globally, or cross project boundaries.
|
||||||
|
|
@ -100,6 +102,49 @@ uv run pytest -q
|
||||||
Use the executables under `/absolute/path/DocForge/.venv/bin/` when DocForge is not installed into
|
Use the executables under `/absolute/path/DocForge/.venv/bin/` when DocForge is not installed into
|
||||||
the active shell environment.
|
the active shell environment.
|
||||||
|
|
||||||
|
For an installed distribution, choose only the language extras the project needs:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m pip install docforge
|
||||||
|
python -m pip install 'docforge[javascript]'
|
||||||
|
python -m pip install 'docforge[typescript]'
|
||||||
|
python -m pip install 'docforge[cpp]'
|
||||||
|
```
|
||||||
|
|
||||||
|
The base wheel contains the Python reference adapter and no Tree-sitter distribution. JavaScript,
|
||||||
|
TypeScript, and C++ require their matching optional extras. `docforge[languages]` installs all
|
||||||
|
three optional frontend groups.
|
||||||
|
|
||||||
|
### Configure a reference source project
|
||||||
|
|
||||||
|
Reference adapters are a narrow alternative to the generic documentation descriptor. Create
|
||||||
|
`.docforge/reference-adapter.toml`:
|
||||||
|
|
||||||
|
```toml reference-adapter
|
||||||
|
schema_version = 1
|
||||||
|
project_id = "my-python-project"
|
||||||
|
title = "My Python Project"
|
||||||
|
language = "python"
|
||||||
|
source_roots = ["src"]
|
||||||
|
```
|
||||||
|
|
||||||
|
Start the fixed read-only server:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -I -m docforge.reference_mcp \
|
||||||
|
--project-root /absolute/path/MyProject \
|
||||||
|
--capability-mode read
|
||||||
|
```
|
||||||
|
|
||||||
|
For C++, set `language = "cpp"` and add a project-relative
|
||||||
|
`compilation_database = "compile_commands.json"`. The database is inert bounded inventory; the
|
||||||
|
reference adapter does not execute its commands or compiler.
|
||||||
|
|
||||||
|
The reference integrations publish syntax and local static relationships only. They do not claim
|
||||||
|
resolved calls, types, inheritance, macro behavior, compiler include semantics, runtime behavior,
|
||||||
|
or semantic ownership. See [reference adapters](REFERENCE_ADAPTERS.md) for exact evidence and
|
||||||
|
limitations.
|
||||||
|
|
||||||
### Assess and onboard an unconfigured project
|
### Assess and onboard an unconfigured project
|
||||||
|
|
||||||
Run a read-only assessment before writing configuration:
|
Run a read-only assessment before writing configuration:
|
||||||
|
|
@ -478,6 +523,10 @@ Every command emits deterministic JSON:
|
||||||
docforge --project-root /absolute/path/MyProject <command>
|
docforge --project-root /absolute/path/MyProject <command>
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The sections below group common workflows. The implementation-derived list of all 28 current
|
||||||
|
commands, exact invocations, 36 generic MCP tools, arguments, and input-schema hashes is the
|
||||||
|
[generated command reference](COMMAND_REFERENCE.md).
|
||||||
|
|
||||||
### Project and index commands
|
### Project and index commands
|
||||||
|
|
||||||
```text
|
```text
|
||||||
|
|
@ -590,6 +639,31 @@ declares the named writer, and application requires the same writer/applier iden
|
||||||
`--no-ast` to preserve the no-AST binding. Generic CLI generation refuses project-owned adapters
|
`--no-ast` to preserve the no-AST binding. Generic CLI generation refuses project-owned adapters
|
||||||
because it cannot safely reconstruct their composition.
|
because it cannot safely reconstruct their composition.
|
||||||
|
|
||||||
|
A custom adapter owner supplies the already constructed project and immutable launcher through the
|
||||||
|
Python API:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from docforge.adapter_launcher import AdapterLauncherV1
|
||||||
|
from docforge.client_config import generate_adapter_client_configuration
|
||||||
|
|
||||||
|
launcher = AdapterLauncherV1.for_project(
|
||||||
|
project,
|
||||||
|
module="my_project_docforge",
|
||||||
|
)
|
||||||
|
fragment = generate_adapter_client_configuration(
|
||||||
|
project,
|
||||||
|
launcher,
|
||||||
|
"codex",
|
||||||
|
capability_mode="read",
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
The top-level module must be installed for the exact isolated Python environment and resolve to a
|
||||||
|
regular file inside the project root. The fixed `docforge.reference_mcp` module is the only trusted
|
||||||
|
dotted exception. Generation probes resolution without importing the custom module, binds current
|
||||||
|
source availability and policy, and emits no arbitrary command, arguments, working directory, or
|
||||||
|
environment.
|
||||||
|
|
||||||
Select projection behavior independently:
|
Select projection behavior independently:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|
@ -684,6 +758,10 @@ docforge-mcp \
|
||||||
|
|
||||||
Omit `--proposal-writer` when the MCP client should not create or append proposals.
|
Omit `--proposal-writer` when the MCP client should not create or append proposals.
|
||||||
|
|
||||||
|
For `.docforge/reference-adapter.toml`, use the fixed `docforge.reference_mcp` command shown in
|
||||||
|
[setup](#configure-a-reference-source-project). It exposes exactly the 21 read tools and never
|
||||||
|
registers proposal or application tools.
|
||||||
|
|
||||||
Select the session's declared surface explicitly when useful:
|
Select the session's declared surface explicitly when useful:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|
@ -986,6 +1064,28 @@ changes outside this process boundary.
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
|
### `optional_dependency_missing`
|
||||||
|
|
||||||
|
Install the exact extra named in the error into the same Python environment that starts DocForge:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m pip install 'docforge[javascript]'
|
||||||
|
python -m pip install 'docforge[typescript]'
|
||||||
|
python -m pip install 'docforge[cpp]'
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not install every frontend merely to suppress the check. A missing optional parser is a closed,
|
||||||
|
actionable capability error and does not affect base generic or Python reference operation.
|
||||||
|
|
||||||
|
### `adapter_launcher_unavailable` or `invalid_adapter_launcher`
|
||||||
|
|
||||||
|
Use one installed top-level Python module whose resolved regular-file origin is inside the project
|
||||||
|
root, or use the fixed `docforge.reference_mcp` binding. Arbitrary dotted modules, packages,
|
||||||
|
stdlib modules, missing modules, commands, argument strings, working directories, and environment
|
||||||
|
injection are rejected. Test the exact generated fragment rather than editing its command by hand.
|
||||||
|
|
||||||
|
See [agent integration](AGENT_INTEGRATION.md) and the [security model](SECURITY.md).
|
||||||
|
|
||||||
### `adapter_restart_required`
|
### `adapter_restart_required`
|
||||||
|
|
||||||
The project-local adapter code, its declared descriptor, or another implementation file changed
|
The project-local adapter code, its declared descriptor, or another implementation file changed
|
||||||
|
|
@ -1128,8 +1228,11 @@ make gate
|
||||||
Use `make benchmark` for the historical Milestone 0 baseline, `make benchmark-m1` for the
|
Use `make benchmark` for the historical Milestone 0 baseline, `make benchmark-m1` for the
|
||||||
counter-gated warm-operation benchmark, `make benchmark-m2` for agent workflow gates, and
|
counter-gated warm-operation benchmark, `make benchmark-m2` for agent workflow gates, and
|
||||||
`make benchmark-m3-full` for the ten-sample 1,000-node projection, worker, fragment, status,
|
`make benchmark-m3-full` for the ten-sample 1,000-node projection, worker, fragment, status,
|
||||||
equivalence, response-size, and memory gates. `make accessibility` runs the generated manual,
|
equivalence, response-size, and memory gates. `make benchmark-m4-full` runs the 1,002-node adapter
|
||||||
portable graph, and live viewer axe and keyboard flows.
|
and recovery benchmark. `make adoption-m4` performs the offline fresh-wheel proof.
|
||||||
|
`make command-reference-check` rejects command-reference drift, and `make docs-check` validates
|
||||||
|
the maintained documentation graph. `make accessibility` runs the generated manual, portable
|
||||||
|
graph, and live viewer axe and keyboard flows.
|
||||||
|
|
||||||
Project-specific vocabulary, extraction rules, and serialization belong in the project adapter.
|
Project-specific vocabulary, extraction rules, and serialization belong in the project adapter.
|
||||||
Generic core behavior must remain deterministic, project-bound, and recoverable.
|
Generic core behavior must remain deterministic, project-bound, and recoverable.
|
||||||
|
|
|
||||||
|
|
@ -1,13 +1 @@
|
||||||
# Remove each entry in the same commit that authors the required page.
|
# Milestone 4 required-page inventory is complete.
|
||||||
docs/AGENT_INTEGRATION.md
|
|
||||||
docs/CORE_CONCEPTS_AND_AUTHORITY.md
|
|
||||||
docs/LEGACY_AND_NO_AST.md
|
|
||||||
docs/MIGRATING_FROM_V1.md
|
|
||||||
docs/MILESTONE_4_BASELINE.md
|
|
||||||
docs/MILESTONE_4_CLOSEOUT.md
|
|
||||||
docs/POLICY_PRECEDENCE.md
|
|
||||||
docs/PROJECT_DESCRIPTOR.md
|
|
||||||
docs/RECOVERY_AND_PERFORMANCE.md
|
|
||||||
docs/REFERENCE_ADAPTERS.md
|
|
||||||
docs/RENDERING_AND_VISUALIZATION.md
|
|
||||||
docs/SECURITY.md
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue