administrator/DocForge2
1
0
Fork 0
Code Issues Pull requests Projects Releases 2 Packages Wiki Activity Actions Pages

Close Milestone 4 with adapter adoption evidence

Browse source
This commit is contained in:
Andraxion 2026-07-29 15:34:25 -04:00
parent 95271dcf2e
commit 6d06195950
27 changed files with 2870 additions and 325 deletions
Download patch file Download diff file Expand all files Collapse all files

409
README.md
View file

@ -1,88 +1,107 @@
# DocForge
DocForge is a project-scoped documentation graph for people and AI agents. It validates canonical
documentation, builds a disposable search and relationship index, compiles bounded context, renders
declared manuals, visualizes project structure, and manages reviewable documentation changesets.
DocForge is a project-scoped documentation and source graph for people and AI agents. Canonical
project files remain authoritative; DocForge validates them, builds disposable search and
relationship indexes, compiles bounded context, manages reviewable changesets, renders declared
manuals and portable graph views, and serves one project-bound MCP surface.
## What it does
DocForge never treats indexed text as instructions. It does not run project build commands,
compilers, Git operations, deployments, or arbitrary renderers, and it does not select projects
globally.
- Validates stable Markdown/TOML nodes and typed relationships.
- Builds a deterministic SQLite search and graph index.
- Exposes project-bound CLI and MCP query surfaces.
- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, and bounded
continuation.
- Records one bounded, versioned latest-generation graph transition without creating a history
database.
- Generates deterministic project-bound Codex, Claude, and OpenClaw client fragments without
copying ambient secrets.
- Diagnoses one client binding through bounded read-only checks without starting MCP or rebuilding
project state.
- Automatically synchronizes disposable indexes before MCP work.
- Creates, validates, diffs, and previews isolated changesets.
- Registers complete proposals atomically without caller-managed hash chaining.
- Applies one explicitly approved changeset hash through CLI or gated MCP.
- Supports opt-in incremental adapters with reverse-dependency invalidation and full-build
equivalence checks.
- Detects project-local adapter implementation and configuration changes and requires a fresh
project-bound process before any further MCP work.
- Keeps function-scoped control-flow projections separate from the primary architecture graph.
- Compiles manuals and portable graph artifacts from separate versioned, generation-pinned plans
and immutable packages.
- Runs built-in manual and graph renderers in fixed detached workers with validated receipts,
bounded transfer, and no project-path authority.
- Enforces independent manual, portable-graph, and live-viewer policy while keeping status
receipt-only.
- Runs a managed loopback graph browser with neighborhood, semantic Flow, convergence Web,
function-scoped Logic, source inspection, and branch-aware node hiding.
- Supports generic documentation projects and project-owned source adapters.
DocForge 1.0.0 remains the first stable release in this product line. This checkout is the
post-1.0 development line (`1.3.0.dev0`) and contains the Milestone 4 adapter-platform work.
Milestone 4 is not a new product release, and Milestone 5 release and publication work remains
separate.
DocForge never treats indexed text as instructions. It does not run shell commands, mutate Git,
build applications, deploy, publish, or select projects globally.
## Start here
For implementation projects, the recommended cadence is to read canonical documentation during
intake, keep it read-only through implementation and focused testing, freeze and validate a release
candidate, then perform one atomic documentation closeout before the final commit and tag. This
keeps the manual authoritative without using it as an implementation notebook.
- New installation or first project: [New-project quickstart](docs/NEW_PROJECT_QUICKSTART.md)
- Mental model and authority: [Core concepts and authority](docs/CORE_CONCEPTS_AND_AUTHORITY.md)
- Complete configuration shape: [Project descriptor](docs/PROJECT_DESCRIPTOR.md)
- Fixed runnable examples: [Reference adapters](docs/REFERENCE_ADAPTERS.md)
- Agent and client setup: [Agent integration](docs/AGENT_INTEGRATION.md)
- Exact live command and tool inventory: [Generated command reference](docs/COMMAND_REFERENCE.md)
- Task-oriented operating guide: [User manual](docs/USER_MANUAL.md)
## Release 1
## Current capabilities
DocForge 1.0.0 is the first stable product release. It combines the project-scoped graph, CLI and
MCP query surfaces, reviewable hash-approved changesets, generic and project-owned adapters,
declared rendering, and the complete Nodes/Flow/Web visualization model in one supported release.
- Validates Markdown and TOML nodes, stable IDs, typed relationships, project limits, and confined
paths.
- Builds a deterministic, disposable SQLite graph and search index with integrity and generation
evidence.
- Exposes project-bound CLI and MCP read, proposal, application, rendering, and visualization
surfaces according to the startup policy.
- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, bounded
output, and deterministic continuation.
- Creates isolated documentation changesets, validates complete projected graphs, and applies only
one explicitly approved changeset hash through a separately bound canonical applier.
- Supports complete-projection adapters and opt-in incremental adapters with reverse-dependency
invalidation, bounded extraction caches, and a clean full-build equivalence oracle.
- Publishes function-scoped Logic separately from the primary graph for Python, JavaScript,
TypeScript, and C++ integrations that provide it.
- Compiles manual and portable graph plans into immutable packages for fixed detached renderers,
then records bounded receipts.
- Runs a managed loopback graph viewer with Nodes, Flow, Web, lazy Logic, source inspection, and
branch-aware hiding.
- Generates deterministic Codex, Claude, and OpenClaw client fragments without copying ambient
environment values or secrets.
The post-1.0 incremental compiler is a backward-compatible, optional enhancement. Existing Release
1 adapters that implement only `load_projection()` continue to use the original complete-projection
path without modification. Adapters gain incremental performance only when they additionally
implement the source manifest and extraction methods. Incremental adapters must retain
`load_projection()` as their clean-rebuild fallback and equivalence oracle.
## Adapter platform
MCP bindings can additionally select `--no-ast` when an owner wants to preserve an existing
non-AST adapter. The binding advertises that policy to clients, forbids adapter rewrites that add
AST, Tree-sitter, compiler-AST, or function-Logic extraction, blocks the Logic tool, and rejects
nonempty Logic publication. Complete-projection adapters continue unchanged, and non-AST
incremental fingerprinting and caching remain allowed.
Adapter authors use the public `docforge.adapter_sdk` surface for graph types, complete and
incremental contracts, validation helpers, and `verify_adapter_conformance()`. Existing adapters
that implement only `load_projection()` remain supported. Incremental adapters add a manifest and
source extraction while retaining `load_projection()` as the complete graph oracle. An adapter
that publishes Logic incrementally also supplies `load_complete_assembly()` so the complete oracle
covers both graph and Logic.
DocForge2 bindings may also declare
`--capability-mode read|proposal|application|operator`. Bootstrap returns one versioned effective
policy and the actual startup-gated capabilities. Existing tool surfaces and the legacy no-AST
payload remain compatible.
The repository includes bounded reference integrations for:
`docforge_get_task_context` is an additive read tool for `change`, `implementation`, `failure`,
`ownership`, `test`, `operation`, and `release` work. It derives a closed version-1 retrieval plan,
executes it against one immutable index generation, and returns a hash-bound context capsule.
Project relation names remain authoritative. DocForge classifies only its versioned alias set and
preserves every unknown relation as `unclassified` instead of guessing semantics.
- Python, using the standard-library AST and publishing only project-local imports;
- JavaScript and TypeScript, using their distinct optional Tree-sitter grammars and publishing only
project-local static relative imports and re-exports;
- C++, using `compile_commands.json` as translation-unit inventory and fingerprint evidence, without
executing its commands or a compiler, and publishing only directly resolvable project-local
quoted includes.
`docforge_get_generation_diff` reports the latest verified primary-graph transition through one
bounded disposable receipt. It includes exact node and edge change counts, hash-bound retained
details, and explicit truncation. Paged results use one top-level cursor and a versioned
`receipt_header`; `stored_receipt_hash` identifies the complete persisted receipt. The read never
exposes Logic details, loads canonical source, repairs derived state, or invents history.
These integrations demonstrate the adapter contract; they do not claim resolved calls,
inheritance, types, runtime behavior, macro expansion, compiler include semantics, or semantic
ownership. Python works from the base wheel. Install the `javascript`, `typescript`, or `cpp`
extra for the corresponding grammar, or `languages` for all three:
## Independent projections
```bash
uv pip install "/absolute/path/DocForge[javascript]"
uv pip install "/absolute/path/DocForge[typescript]"
uv pip install "/absolute/path/DocForge[cpp]"
uv pip install "/absolute/path/DocForge[languages]"
```
Manual compilation, portable graph rendering, and the live viewer consume the same validated graph
generation through separate boundaries:
Reference projects use the fixed `.docforge/reference-adapter.toml` descriptor and the installed
`docforge.reference_mcp` module. That server is read-only and exposes no proposal or application
surface.
Project-owned adapters use `AdapterLauncherV1` and
`generate_adapter_client_configuration()` to produce a client fragment from an explicitly
constructed adapter project. The launcher is immutable and contains no command, caller arguments,
working directory, environment, discovery rule, or callable selector. It resolves through isolated
Python to one installed, project-owned top-level module; the only trusted dotted exception is the
fixed `docforge.reference_mcp` binding. Generic `docforge configure` intentionally refuses custom
adapters because it cannot safely reconstruct project-owned composition.
See the [Adapter authoring guide](docs/ADAPTER_AUTHORING_GUIDE.md), [Reference
adapters](docs/REFERENCE_ADAPTERS.md), and [Agent integration](docs/AGENT_INTEGRATION.md) for the
supported routes.
## Authority and projections
Canonical Markdown, TOML, adapter-declared sources, and descriptor files own project facts. The
SQLite index, extraction cache, render packages, previews, portable artifacts, receipts, client
fragments, and viewer processes are derived and replaceable.
The primary graph contains project nodes and relationships. Logic is a separate, lazy,
function-scoped control-flow projection. Manual output, portable graph output, and the live viewer
are independent consumers of one validated generation:
```text
validated generation
@ -91,190 +110,142 @@ validated generation
└── pinned index → managed read-only live viewer
```
Plans, packages, and receipts are canonical, versioned, hash-identified, bounded, and contain no
project object, SQLite handle, absolute project path, command, or caller-selected renderer module.
Renderers cannot select graph facts, crawl canonical sources, choose publication paths, or mutate
project state.
The version-2 projection policy selects each consumer independently:
Declare portable graph output separately from manual views:
```toml
[graph_render]
output_root = ".docforge/portable-graph"
[[graph_render.views]]
id = "architecture"
renderer = "portable_graph_html"
output = "architecture.html"
title = "Architecture"
root = "architecture.overview"
initial_mode = "web"
depth = 3
max_nodes = 250
max_edges = 1000
max_work = 100000
include_logic = false
```text
manual: auto | explicit | disabled
portable_graph: explicit | disabled
live_viewer: on-demand | disabled
```
Plan, publish, and inspect it explicitly:
The process capability policy is separate. Capability mode controls the registered read,
proposal, application, or reserved operator surface; descriptor writers and the startup-bound
canonical applier determine whether mutations are actually authorized. `--no-ast` is a restrictive
binding policy over adapter evolution, Logic publication, and Logic retrieval. It is not a parser
inspection mechanism or a filesystem sandbox, and the AST/Tree-sitter reference integrations
should not be presented as no-AST adapters.
Read [Policy precedence](docs/POLICY_PRECEDENCE.md), [Rendering and
visualization](docs/RENDERING_AND_VISUALIZATION.md), and [Legacy and no-AST
operation](docs/LEGACY_AND_NO_AST.md) before changing those boundaries.
## Five-minute generic project
Requirements are Python 3.12 or newer and an installed DocForge environment. From a development
checkout, `uv sync --group dev` creates `.venv`:
```bash
.venv/bin/docforge --project-root "$PROJECT" graph-plan architecture
.venv/bin/docforge --project-root "$PROJECT" graph-render architecture
.venv/bin/docforge --project-root "$PROJECT" graph-render-status architecture
```
The version-2 projection policy independently selects manual
`auto|explicit|disabled`, portable graph `explicit|disabled`, and live viewer
`on-demand|disabled`. Use `--manual-render-policy`, `--portable-graph-policy`, and
`--live-viewer-policy` on CLI/MCP startup or generated client configuration. Status remains
available when the corresponding active operation is disabled.
Manual fragment reuse is disposable. Cold record publication is guarded by byte-exact comparison
with a full detached render; warm records are independently recomputed and validated inside the
worker. Full rendering remains the recovery and equivalence oracle. Portable artifacts commit
content-addressed output and renderer evidence before one bounded generation/view manifest; status
does not plan or render.
## Graph views
The browser presents the primary architecture graph through three complementary views and loads a
fourth function-scoped view only when requested:
- **Nodes** shows a bounded, relation-neutral neighborhood around the focus. It is the broad
inspection view for seeing stored incoming and outgoing relationships without changing their
direction. Semantic cards distinguish structure, behavior, dependencies, execution, data,
evidence, context, and other relationships.
- **Flow** shows semantic origin-to-destination paths that terminate at the focus. DocForge
reverses prerequisite-style relationships for presentation, so imports, dependencies, reads,
inheritance, definitions, and tests flow toward the thing they help create or exercise.
- **Web** shows the larger convergence picture: Flow contributors plus contextual relationships,
callers, containers, and direct members or execution dependencies owned by the focus.
- **Logic** shows the possible static control paths inside a focused Python, JavaScript, or C++
function or method. Entry, decisions, actions, loops, convergence points, returns, and
exceptions connect through explicit
`TRUE`, `FALSE`, `NEXT`, `CASE`, `LOOP`, `RETURN`, and `RAISE` paths. Logic is stored separately
and does not add statement-level noise to Nodes, Flow, Web, or search.
Graph cards show the node's readable leaf name and kind without clipping either value. The full
qualified identity remains available in the tooltip, compact descriptor, and full inspector.
The left browser panel can combine text, family, node-kind, language, and capability filters.
Quick presets expose Logic-ready nodes, Python callables, tests, routes, and documentation without
requiring users to know stable IDs. Selecting a canvas node emphasizes its directly connected
neighbors and edges while muting unrelated paths.
**Hide node** removes noise without changing the index. In Flow and Web, hiding a contributor also
removes upstream ancestors that no longer have a path to the focus. Nodes between the hidden
contributor and the focus stay visible, and alternate ancestor paths remain intact. In Logic,
hiding a step inserts an explicit omitted-path bridge so downstream control flow remains readable.
**Restore hidden** restores the presentation.
## Five-minute start
Requirements are Python 3.12+, `uv`, and Node.js/npm.
```bash
git clone forgejo@repo.andraxion.net:administrator/DocForge2.git /absolute/path/DocForge2
cd /absolute/path/DocForge2
git clone <repository-url> /absolute/path/DocForge
cd /absolute/path/DocForge
uv sync --group dev
npm ci
PROJECT=/absolute/path/MyProject
.venv/bin/docforge --project-root "$PROJECT" onboard
.venv/bin/docforge --project-root "$PROJECT" onboard \
--scaffold \
--project-id my-project \
--title "My Project"
```
The first command is read-only. Scaffolding is explicit and create-only: it writes a generic
descriptor, one canonical overview node, and a built-in manual template, then indexes and renders
them. Detected source languages remain `adapter_required` until a validated frontend is selected.
Operate the configured project:
```bash
.venv/bin/docforge --project-root "$PROJECT" validate
.venv/bin/docforge --project-root "$PROJECT" reindex
.venv/bin/docforge --project-root "$PROJECT" search architecture
.venv/bin/docforge --project-root "$PROJECT" visualize
```
Install the persistent per-user graph viewer once:
Install the persistent per-user graph viewer manager once when using the live viewer:
```bash
.venv/bin/docforge-viewer-manager install-user-service
```
Start an MCP server for one project:
Start one generic, read-only MCP server:
```bash
.venv/bin/docforge-mcp \
--project-root "$PROJECT" \
--proposal-writer project-editor
--capability-mode read
```
Add `--canonical-applier project-editor` only when that MCP integration should expose the
hash-bound `docforge_apply_changeset` tool.
Preview a read-only Codex fragment and diagnose an installed binding:
```bash
.venv/bin/docforge configure codex --project "$PROJECT"
.venv/bin/docforge doctor --client codex --project "$PROJECT"
```
Pass `--output /absolute/path/docforge.toml` only when creating a standalone fragment. DocForge
never replaces or merges an existing different client file.
For an unconfigured codebase, begin with a read-only language and documentation assessment:
```bash
.venv/bin/docforge --project-root /absolute/path/MyProject onboard
```
Add `--scaffold`, a stable project ID, and a title to create, index, and render a generic starter
manual. Source files are reported separately and require a validated language frontend before
DocForge describes them as a source graph.
Add a descriptor-authorized proposal writer only when the client should create proposals. Add a
matching `--canonical-applier` only when that integration should expose exact-hash application.
See the [quickstart](docs/NEW_PROJECT_QUICKSTART.md) for the reference-adapter and generated-client
paths.
## Documentation
- [User manual](docs/USER_MANUAL.md) — features, setup, visualization, CLI, MCP, apply, adapters,
and troubleshooting.
- [Core contract](docs/CONTRACT.md) — invariants and security boundary.
- [Milestone 0 compatibility](docs/COMPATIBILITY.md) — preserved package, CLI, MCP, adapter,
schema, changeset, rendering, and no-AST guarantees.
- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) — validation evidence, cold and warm
performance, memory, rendering and response sizes, bottlenecks, and missing coverage.
- [Milestone 0 closeout](docs/MILESTONE_0_CLOSEOUT.md) — lineage, migration, security scan,
repository state, and fresh-clone proof.
- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) — warm operation latency, structured work,
status, retrieval, and memory measurements.
- [Milestone 1 closeout](docs/MILESTONE_1_CLOSEOUT.md) — fast-core contracts, adversarial
validation, compatibility boundaries, and exact candidate evidence.
- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) — task context, generation diff, client
configuration, doctor, response-size, counter, and memory measurements.
- [Milestone 2 closeout](docs/MILESTONE_2_CLOSEOUT.md) — implemented contracts, adversarial
validation, exclusions, and exact candidate evidence.
- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) — manual, fragment, worker, portable graph,
status, equivalence, response-size, and memory measurements.
- [Milestone 3 closeout](docs/MILESTONE_3_CLOSEOUT.md) — independent projection contracts,
adversarial validation, compatibility boundaries, and exact candidate evidence.
- [MCP contract](docs/MCP_CONTRACT.md) — exact tool and process boundary.
- [Viewer manager](docs/VIEWER_MANAGER.md) — native service setup and lifecycle.
- [Adapter decision](docs/APPLICATION_DECISION.md) — why custom adapters own canonical
serialization.
- [Incremental adapter indexing](docs/INCREMENTAL_INDEXING.md) — source-scoped extraction,
invalidation, equivalence, relationship changes, and the lazy Logic boundary.
- [Project onboarding](docs/PROJECT_ONBOARDING.md) — repository assessment, safe manual
scaffolding, language frontends, source/manual integration, proof, and MCP activation.
- [Language adapter authoring](docs/ADAPTER_AUTHORING_GUIDE.md) — implementation sequence,
stable identities, overlap ownership, normalization, incremental equivalence, troubleshooting,
and the complete adapter proof matrix.
### Learn and operate
- [New-project quickstart](docs/NEW_PROJECT_QUICKSTART.md)
- [Core concepts and authority](docs/CORE_CONCEPTS_AND_AUTHORITY.md)
- [Project descriptor](docs/PROJECT_DESCRIPTOR.md)
- [Policy precedence](docs/POLICY_PRECEDENCE.md)
- [Reference adapters](docs/REFERENCE_ADAPTERS.md)
- [Agent integration](docs/AGENT_INTEGRATION.md)
- [User manual](docs/USER_MANUAL.md)
- [Generated command reference](docs/COMMAND_REFERENCE.md)
- [Project onboarding](docs/PROJECT_ONBOARDING.md)
- [Rendering and visualization](docs/RENDERING_AND_VISUALIZATION.md)
- [Recovery and performance](docs/RECOVERY_AND_PERFORMANCE.md)
- [Security](docs/SECURITY.md)
### Contracts and compatibility
- [Core contract](docs/CONTRACT.md)
- [MCP contract](docs/MCP_CONTRACT.md)
- [Compatibility contract](docs/COMPATIBILITY.md)
- [Adapter authoring guide](docs/ADAPTER_AUTHORING_GUIDE.md)
- [Incremental indexing](docs/INCREMENTAL_INDEXING.md)
- [Adapter application decision](docs/APPLICATION_DECISION.md)
- [Legacy and no-AST operation](docs/LEGACY_AND_NO_AST.md)
- [Migrating from version 1](docs/MIGRATING_FROM_V1.md)
- [Viewer manager](docs/VIEWER_MANAGER.md)
### Milestone evidence
- [Milestone 4 baseline](docs/MILESTONE_4_BASELINE.md)
- [Milestone 4 closeout](docs/MILESTONE_4_CLOSEOUT.md)
- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) and [closeout](docs/MILESTONE_3_CLOSEOUT.md)
- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) and [closeout](docs/MILESTONE_2_CLOSEOUT.md)
- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) and [closeout](docs/MILESTONE_1_CLOSEOUT.md)
- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) and [closeout](docs/MILESTONE_0_CLOSEOUT.md)
Historical milestone records preserve the facts and dependency observations of their frozen
candidates. Use the current guides and contracts for present behavior.
## Development
Run the complete repository-native gate:
Run the repository-native gate:
```bash
make gate
```
Focused entry points are available as `make contract`, `make test`, `make type`,
`make benchmark-smoke`, `make benchmark`, `make benchmark-m1-smoke`, and
`make benchmark-m1`. Milestone 2 adds `make benchmark-m2-smoke` and `make benchmark-m2`.
Milestone 3 adds `make accessibility`, `make benchmark-m3-smoke`, `make benchmark-m3`, and
`make benchmark-m3-full`.
Milestone 4 maintenance entry points include:
The committed 1,000-node baseline and its measurement method are under `benchmarks/`.
```bash
make adoption-m4
make benchmark-m4-smoke
make benchmark-m4
make benchmark-m4-full
make command-reference-check
make docs-check
```
Pass `--diagnostics` to `docforge` or `docforge-mcp` to attach bounded request-local stage timings
and compiler-work counters. Diagnostics are disabled by default and are dropped before primary MCP
results when the configured output budget is tight.
`adoption-m4` builds and exercises a fresh base wheel without Tree-sitter packages and proves the
Python reference plus a real isolated read-only MCP retrieval. The optional-language integrations
have their own focused tests and extras. `benchmark-m4` runs the maintained full adapter workload;
the smoke target is for routine gate coverage, not final performance evidence.
See [AGENTS.md](AGENTS.md) before changing core boundaries.
Pass `--diagnostics` to `docforge` or `docforge-mcp` for bounded request-local timings and compiler
work counters. Diagnostics are disabled by default and do not displace a primary result when the
configured output budget is tight.
Read [AGENTS.md](AGENTS.md) before changing core boundaries.