Compare commits
No commits in common. "6d06195950d33bcd2d712f8819bbfb3d6652ad03" and "d6d9f47672c513c4927c5c42ad97a175ed651a74" have entirely different histories.
6d06195950
...
d6d9f47672
83 changed files with 312 additions and 14959 deletions
|
|
@ -1,15 +1,13 @@
|
||||||
# Milestone state
|
# Milestone state
|
||||||
|
|
||||||
```text
|
```text
|
||||||
Last completed milestone: 4 — adapter SDK and product documentation.
|
Last completed milestone: 3 — independent projections
|
||||||
Evidence: Frozen executable candidate 95271dcf2e48045b9d3aed9b9ea09c7fc155692c; 347 tests and 402 subtests; 3 accessibility flows; offline fresh-wheel adoption; clean 1,002-node graph-plus-Logic benchmark; no secret-scan findings.
|
Outcome: Manual output, portable graph artifacts, and the live viewer are independent generation-pinned consumers of the validated graph.
|
||||||
|
Evidence: Clean candidate f5dccb5e1c312121f1af63780162f593d9363b98; 281 tests and 272 subtests; 3 accessibility flows; clean 1,000-node ten-sample benchmark; isolated wheel proof; no secret-scan findings.
|
||||||
Active milestone: None.
|
Active milestone: None.
|
||||||
Outcome: New projects can adopt DocForge incrementally, and users can understand the CLI, MCP server, policies, adapters, renderers, and agent integration without reading implementation code.
|
Next directional milestone: 4 — adapter SDK and product documentation.
|
||||||
Status: Milestone 4 is closed. Milestone 5 has not started.
|
Status: Milestone 3 is closed. Milestone 4 has not started.
|
||||||
Next directional milestone: 5 — stabilization and first DocForge2 release.
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Milestone 5 remains directional context. Do not begin release stabilization, versioning, tagging,
|
Milestones 4–5 remain directional context. Do not begin Milestone 4 without a new active-slice
|
||||||
or publication without a new active-slice contract based on the Milestone 4 evidence.
|
contract.
|
||||||
|
|
||||||
The completed Milestone 4 contract and its exclusions are preserved in `SLICE_HISTORY.md`.
|
|
||||||
|
|
|
||||||
|
|
@ -784,108 +784,3 @@ proof. Gitleaks 8.30.1 found no findings across the six Milestone 3 commits or c
|
||||||
Milestone 3 is complete. No tag, release, production integration repointing, WorldForge change,
|
Milestone 3 is complete. No tag, release, production integration repointing, WorldForge change,
|
||||||
ScrapeStation change, storage rewrite, or self-hosting dependency was introduced. Milestone 4
|
ScrapeStation change, storage rewrite, or self-hosting dependency was introduced. Milestone 4
|
||||||
remains directional and has not started.
|
remains directional and has not started.
|
||||||
|
|
||||||
## Milestone 4 — complete: adapter SDK and product documentation
|
|
||||||
|
|
||||||
Milestone 4 activated from the verified Milestone 3 closeout. Its scope was deliberately additive:
|
|
||||||
freeze an adapter-authoring boundary, prove narrow language references, make heavy frontends
|
|
||||||
optional, generate command and client integration evidence, and document adoption. WorldForge,
|
|
||||||
ScrapeStation, the legacy repository, production bindings, self-hosting, storage replacement, and
|
|
||||||
release publication remained excluded.
|
|
||||||
|
|
||||||
### Adapter SDK and conformance
|
|
||||||
|
|
||||||
`docforge.adapter_sdk` is now the public authoring facade for typed projections, manifests, source
|
|
||||||
contributions, complete assemblies, incremental loaders, project bindings, core graph models, and
|
|
||||||
conformance reports. Complete evidence includes primary graph and function Logic. An incremental
|
|
||||||
adapter that publishes Logic must provide an independent `load_complete_assembly()` oracle.
|
|
||||||
|
|
||||||
The conformance helper repeats complete loading for determinism, compares the complete assembly
|
|
||||||
with `load_projection()`, and proves exact complete/incremental graph-plus-Logic parity. It does not
|
|
||||||
pretend to replace separate confinement, restart, no-AST, corrupt-cache, or retrieval tests.
|
|
||||||
|
|
||||||
An adversarial review found that extraction caches and adapter assemblies lacked aggregate
|
|
||||||
boundaries. Version-1 cache reads and writes are now regular-file-only, identity-checked, capped at
|
|
||||||
64,000,000 bytes and 10,000 sources, and fail safely to a miss. Adapter projects enforce primary
|
|
||||||
node limits and deterministic edge and Logic multipliers before publication.
|
|
||||||
|
|
||||||
### Reference language evidence
|
|
||||||
|
|
||||||
The Python reference uses the standard-library AST and publishes files, modules, classes,
|
|
||||||
functions, arguments, local static imports, and function Logic. Its manifest uses tokenization
|
|
||||||
rather than AST parsing, so unchanged warm builds perform no syntax parse.
|
|
||||||
|
|
||||||
JavaScript and TypeScript use separate pinned optional Tree-sitter grammars. They publish files,
|
|
||||||
modules, classes, functions/methods, static project-relative imports/re-exports, and function
|
|
||||||
Logic. Their manifests use a closed comment/string-aware ESM scan and do not load Tree-sitter.
|
|
||||||
|
|
||||||
C++ uses explicit non-overlapping source roots and one confined `compile_commands.json` as
|
|
||||||
translation-unit inventory and fingerprint evidence. It validates entries but never executes a
|
|
||||||
command, compiler, response file, or project program. It publishes syntax and directly resolvable
|
|
||||||
project-local quoted includes. Its manifest currently parses for include discovery, so no
|
|
||||||
zero-warm-parser claim is made.
|
|
||||||
|
|
||||||
None of the references claim compiler-resolved calls, inheritance, types, symbol references,
|
|
||||||
compiler include semantics, macro semantics, runtime behavior, or semantic ownership.
|
|
||||||
|
|
||||||
The first packaging proof exposed that language parsers were inherited as mandatory dependencies.
|
|
||||||
The base wheel now depends only on Markdown and MCP packages. Python works in the base wheel;
|
|
||||||
JavaScript, TypeScript, C++, and all-language extras install their respective Tree-sitter
|
|
||||||
frontends. Missing extras return a closed error with the exact install target.
|
|
||||||
|
|
||||||
### Fixed reference MCP and launchers
|
|
||||||
|
|
||||||
A strict `.docforge/reference-adapter.toml` selects one fixed reference language, project
|
|
||||||
identity, and explicit source roots. C++ additionally requires the compilation database. The fixed
|
|
||||||
`python -m docforge.reference_mcp` binding constructs only repository-owned providers and exposes
|
|
||||||
the 21 read tools.
|
|
||||||
|
|
||||||
The initial launcher design generated `python -I -m` for arbitrary project module names without
|
|
||||||
proving that isolated Python could resolve them. The corrected `AdapterLauncherV1` accepts only an
|
|
||||||
installed project-confined top-level module or the exact trusted `docforge.reference_mcp`
|
|
||||||
exception. A fixed bounded `find_spec` probe runs under isolated Python without importing project
|
|
||||||
code. Tests then launch both a real project module and the real reference MCP process.
|
|
||||||
|
|
||||||
Generated Codex, Claude, and OpenClaw fragments bind the exact launcher, descriptor, source
|
|
||||||
availability, effective and projection policy, interpreter, canonical arguments, empty
|
|
||||||
environment, artifact bytes, and hashes. There is no arbitrary command, argument list, working
|
|
||||||
directory, environment, discovery, or callable selector.
|
|
||||||
|
|
||||||
### Generated reference and documentation gates
|
|
||||||
|
|
||||||
CLI tables are derived from the real argparse subcommands. MCP tables are derived from a real
|
|
||||||
application-enabled server's `list_tools()` registrations, including capability surface,
|
|
||||||
arguments, descriptions, and input-schema hashes. The generated artifact contains 28 CLI rows and
|
|
||||||
36 MCP rows.
|
|
||||||
|
|
||||||
An adversarial review found a target-change race between the generator's identity check and
|
|
||||||
replacement. Publication now locks cooperating generators, uses no-clobber publication for a
|
|
||||||
missing target, and uses Linux atomic exchange plus displaced-byte and file-identity verification
|
|
||||||
for an existing target. Raced data is restored or retained for recovery rather than discarded.
|
|
||||||
|
|
||||||
`docs-check` validates generated drift, local files and anchors, one H1 per page, README-rooted
|
|
||||||
reachability, required Milestone 4 inventory, the generated notice, and every documented
|
|
||||||
reference-adapter TOML example against the packaged schema.
|
|
||||||
|
|
||||||
### Candidate, adoption, and scale evidence
|
|
||||||
|
|
||||||
Frozen executable candidate `95271dcf2e48045b9d3aed9b9ea09c7fc155692c` passed formatting,
|
|
||||||
Python and web lint, strict Pyright, compilation, 142 contract tests and 268 subtests, 347 total
|
|
||||||
tests and 402 subtests, three accessibility flows, lock and dependency checks, package builds,
|
|
||||||
offline wheel adoption, and every maintained smoke benchmark.
|
|
||||||
|
|
||||||
The offline adoption proof installed the base wheel from the lock without network access. It
|
|
||||||
proved no Tree-sitter package or module was installed, built and checked a real Python reference
|
|
||||||
project, performed isolated MCP bootstrap/search/get-node over the exact 21 read tools, and proved
|
|
||||||
that base-only C++ fails with `optional_dependency_missing` and `docforge[cpp]` remediation.
|
|
||||||
|
|
||||||
The clean 334-source benchmark produced 1,002 primary nodes, 1,001 primary edges, 334 Logic
|
|
||||||
projections, 2,338 Logic nodes, and 2,338 Logic edges. Complete and incremental output matched
|
|
||||||
exactly. Warm build p95 was 1,265.387 ms with zero `ast.parse` and zero `extract_source` calls.
|
|
||||||
Complete equivalence, corrupt cache recovery, and corrupt index recovery were all below 1.9
|
|
||||||
seconds. The largest per-operation traced peak was 69,997,166 bytes and process high-water was
|
|
||||||
78,798,848 bytes. Exact hashes and method are recorded in
|
|
||||||
`benchmarks/milestone4-2026-07-29.json` and `docs/MILESTONE_4_BASELINE.md`.
|
|
||||||
|
|
||||||
Milestone 4 is complete. No tag or release was created. Milestone 5 remains unstarted until its own
|
|
||||||
active-slice contract is frozen.
|
|
||||||
|
|
|
||||||
24
Makefile
24
Makefile
|
|
@ -5,7 +5,7 @@ NPM := npm
|
||||||
PYTHONPYCACHEPREFIX := /tmp/docforge-quality-pycache
|
PYTHONPYCACHEPREFIX := /tmp/docforge-quality-pycache
|
||||||
PYTEST_BASETEMP := /tmp/docforge-quality-pytest
|
PYTEST_BASETEMP := /tmp/docforge-quality-pytest
|
||||||
|
|
||||||
.PHONY: accessibility adoption-m4 benchmark benchmark-m1 benchmark-m1-smoke benchmark-m2 benchmark-m2-smoke benchmark-m3 benchmark-m3-full benchmark-m3-smoke benchmark-m4 benchmark-m4-full benchmark-m4-smoke benchmark-smoke build command-reference-check compile contract dependencies docs-check format-check gate lint lock test type
|
.PHONY: accessibility benchmark benchmark-m1 benchmark-m1-smoke benchmark-m2 benchmark-m2-smoke benchmark-m3 benchmark-m3-full benchmark-m3-smoke benchmark-smoke build compile contract dependencies format-check gate lint lock test type
|
||||||
|
|
||||||
accessibility:
|
accessibility:
|
||||||
$(NPM) run test:accessibility
|
$(NPM) run test:accessibility
|
||||||
|
|
@ -59,17 +59,6 @@ dependencies:
|
||||||
build:
|
build:
|
||||||
$(UV) build
|
$(UV) build
|
||||||
|
|
||||||
adoption-m4:
|
|
||||||
$(PYTHON) tools/milestone4_adoption.py
|
|
||||||
|
|
||||||
command-reference-check:
|
|
||||||
$(PYTHON) tools/generate_command_reference.py \
|
|
||||||
--output docs/COMMAND_REFERENCE.md --check > /dev/null
|
|
||||||
|
|
||||||
docs-check: command-reference-check
|
|
||||||
$(PYTHON) tools/check_documentation.py \
|
|
||||||
--pending-inventory tools/docs_pending_m4_pages.txt
|
|
||||||
|
|
||||||
benchmark-smoke:
|
benchmark-smoke:
|
||||||
$(PYTHON) tools/milestone0_baseline.py --nodes 25 --samples 1 --cold-samples 1 \
|
$(PYTHON) tools/milestone0_baseline.py --nodes 25 --samples 1 --cold-samples 1 \
|
||||||
--output /tmp/docforge-milestone0-smoke.json > /dev/null
|
--output /tmp/docforge-milestone0-smoke.json > /dev/null
|
||||||
|
|
@ -100,13 +89,4 @@ benchmark-m3:
|
||||||
|
|
||||||
benchmark-m3-full: benchmark-m3
|
benchmark-m3-full: benchmark-m3
|
||||||
|
|
||||||
benchmark-m4-smoke:
|
gate: format-check lint type compile contract test accessibility lock dependencies build benchmark-smoke benchmark-m1-smoke benchmark-m2-smoke benchmark-m3-smoke
|
||||||
$(PYTHON) tools/milestone4_benchmark.py --mode smoke \
|
|
||||||
--output /tmp/docforge-milestone4-smoke.json > /dev/null
|
|
||||||
|
|
||||||
benchmark-m4:
|
|
||||||
$(PYTHON) tools/milestone4_benchmark.py --mode full
|
|
||||||
|
|
||||||
benchmark-m4-full: benchmark-m4
|
|
||||||
|
|
||||||
gate: format-check lint type compile contract test accessibility lock dependencies build docs-check benchmark-smoke benchmark-m1-smoke benchmark-m2-smoke benchmark-m3-smoke benchmark-m4-smoke
|
|
||||||
|
|
|
||||||
409
README.md
409
README.md
|
|
@ -1,107 +1,88 @@
|
||||||
# DocForge
|
# DocForge
|
||||||
|
|
||||||
DocForge is a project-scoped documentation and source graph for people and AI agents. Canonical
|
DocForge is a project-scoped documentation graph for people and AI agents. It validates canonical
|
||||||
project files remain authoritative; DocForge validates them, builds disposable search and
|
documentation, builds a disposable search and relationship index, compiles bounded context, renders
|
||||||
relationship indexes, compiles bounded context, manages reviewable changesets, renders declared
|
declared manuals, visualizes project structure, and manages reviewable documentation changesets.
|
||||||
manuals and portable graph views, and serves one project-bound MCP surface.
|
|
||||||
|
|
||||||
DocForge never treats indexed text as instructions. It does not run project build commands,
|
## What it does
|
||||||
compilers, Git operations, deployments, or arbitrary renderers, and it does not select projects
|
|
||||||
globally.
|
|
||||||
|
|
||||||
DocForge 1.0.0 remains the first stable release in this product line. This checkout is the
|
- Validates stable Markdown/TOML nodes and typed relationships.
|
||||||
post-1.0 development line (`1.3.0.dev0`) and contains the Milestone 4 adapter-platform work.
|
- Builds a deterministic SQLite search and graph index.
|
||||||
Milestone 4 is not a new product release, and Milestone 5 release and publication work remains
|
- Exposes project-bound CLI and MCP query surfaces.
|
||||||
separate.
|
- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, and bounded
|
||||||
|
continuation.
|
||||||
|
- Records one bounded, versioned latest-generation graph transition without creating a history
|
||||||
|
database.
|
||||||
|
- Generates deterministic project-bound Codex, Claude, and OpenClaw client fragments without
|
||||||
|
copying ambient secrets.
|
||||||
|
- Diagnoses one client binding through bounded read-only checks without starting MCP or rebuilding
|
||||||
|
project state.
|
||||||
|
- Automatically synchronizes disposable indexes before MCP work.
|
||||||
|
- Creates, validates, diffs, and previews isolated changesets.
|
||||||
|
- Registers complete proposals atomically without caller-managed hash chaining.
|
||||||
|
- Applies one explicitly approved changeset hash through CLI or gated MCP.
|
||||||
|
- Supports opt-in incremental adapters with reverse-dependency invalidation and full-build
|
||||||
|
equivalence checks.
|
||||||
|
- Detects project-local adapter implementation and configuration changes and requires a fresh
|
||||||
|
project-bound process before any further MCP work.
|
||||||
|
- Keeps function-scoped control-flow projections separate from the primary architecture graph.
|
||||||
|
- Compiles manuals and portable graph artifacts from separate versioned, generation-pinned plans
|
||||||
|
and immutable packages.
|
||||||
|
- Runs built-in manual and graph renderers in fixed detached workers with validated receipts,
|
||||||
|
bounded transfer, and no project-path authority.
|
||||||
|
- Enforces independent manual, portable-graph, and live-viewer policy while keeping status
|
||||||
|
receipt-only.
|
||||||
|
- Runs a managed loopback graph browser with neighborhood, semantic Flow, convergence Web,
|
||||||
|
function-scoped Logic, source inspection, and branch-aware node hiding.
|
||||||
|
- Supports generic documentation projects and project-owned source adapters.
|
||||||
|
|
||||||
## Start here
|
DocForge never treats indexed text as instructions. It does not run shell commands, mutate Git,
|
||||||
|
build applications, deploy, publish, or select projects globally.
|
||||||
|
|
||||||
- New installation or first project: [New-project quickstart](docs/NEW_PROJECT_QUICKSTART.md)
|
For implementation projects, the recommended cadence is to read canonical documentation during
|
||||||
- Mental model and authority: [Core concepts and authority](docs/CORE_CONCEPTS_AND_AUTHORITY.md)
|
intake, keep it read-only through implementation and focused testing, freeze and validate a release
|
||||||
- Complete configuration shape: [Project descriptor](docs/PROJECT_DESCRIPTOR.md)
|
candidate, then perform one atomic documentation closeout before the final commit and tag. This
|
||||||
- Fixed runnable examples: [Reference adapters](docs/REFERENCE_ADAPTERS.md)
|
keeps the manual authoritative without using it as an implementation notebook.
|
||||||
- Agent and client setup: [Agent integration](docs/AGENT_INTEGRATION.md)
|
|
||||||
- Exact live command and tool inventory: [Generated command reference](docs/COMMAND_REFERENCE.md)
|
|
||||||
- Task-oriented operating guide: [User manual](docs/USER_MANUAL.md)
|
|
||||||
|
|
||||||
## Current capabilities
|
## Release 1
|
||||||
|
|
||||||
- Validates Markdown and TOML nodes, stable IDs, typed relationships, project limits, and confined
|
DocForge 1.0.0 is the first stable product release. It combines the project-scoped graph, CLI and
|
||||||
paths.
|
MCP query surfaces, reviewable hash-approved changesets, generic and project-owned adapters,
|
||||||
- Builds a deterministic, disposable SQLite graph and search index with integrity and generation
|
declared rendering, and the complete Nodes/Flow/Web visualization model in one supported release.
|
||||||
evidence.
|
|
||||||
- Exposes project-bound CLI and MCP read, proposal, application, rendering, and visualization
|
|
||||||
surfaces according to the startup policy.
|
|
||||||
- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, bounded
|
|
||||||
output, and deterministic continuation.
|
|
||||||
- Creates isolated documentation changesets, validates complete projected graphs, and applies only
|
|
||||||
one explicitly approved changeset hash through a separately bound canonical applier.
|
|
||||||
- Supports complete-projection adapters and opt-in incremental adapters with reverse-dependency
|
|
||||||
invalidation, bounded extraction caches, and a clean full-build equivalence oracle.
|
|
||||||
- Publishes function-scoped Logic separately from the primary graph for Python, JavaScript,
|
|
||||||
TypeScript, and C++ integrations that provide it.
|
|
||||||
- Compiles manual and portable graph plans into immutable packages for fixed detached renderers,
|
|
||||||
then records bounded receipts.
|
|
||||||
- Runs a managed loopback graph viewer with Nodes, Flow, Web, lazy Logic, source inspection, and
|
|
||||||
branch-aware hiding.
|
|
||||||
- Generates deterministic Codex, Claude, and OpenClaw client fragments without copying ambient
|
|
||||||
environment values or secrets.
|
|
||||||
|
|
||||||
## Adapter platform
|
The post-1.0 incremental compiler is a backward-compatible, optional enhancement. Existing Release
|
||||||
|
1 adapters that implement only `load_projection()` continue to use the original complete-projection
|
||||||
|
path without modification. Adapters gain incremental performance only when they additionally
|
||||||
|
implement the source manifest and extraction methods. Incremental adapters must retain
|
||||||
|
`load_projection()` as their clean-rebuild fallback and equivalence oracle.
|
||||||
|
|
||||||
Adapter authors use the public `docforge.adapter_sdk` surface for graph types, complete and
|
MCP bindings can additionally select `--no-ast` when an owner wants to preserve an existing
|
||||||
incremental contracts, validation helpers, and `verify_adapter_conformance()`. Existing adapters
|
non-AST adapter. The binding advertises that policy to clients, forbids adapter rewrites that add
|
||||||
that implement only `load_projection()` remain supported. Incremental adapters add a manifest and
|
AST, Tree-sitter, compiler-AST, or function-Logic extraction, blocks the Logic tool, and rejects
|
||||||
source extraction while retaining `load_projection()` as the complete graph oracle. An adapter
|
nonempty Logic publication. Complete-projection adapters continue unchanged, and non-AST
|
||||||
that publishes Logic incrementally also supplies `load_complete_assembly()` so the complete oracle
|
incremental fingerprinting and caching remain allowed.
|
||||||
covers both graph and Logic.
|
|
||||||
|
|
||||||
The repository includes bounded reference integrations for:
|
DocForge2 bindings may also declare
|
||||||
|
`--capability-mode read|proposal|application|operator`. Bootstrap returns one versioned effective
|
||||||
|
policy and the actual startup-gated capabilities. Existing tool surfaces and the legacy no-AST
|
||||||
|
payload remain compatible.
|
||||||
|
|
||||||
- Python, using the standard-library AST and publishing only project-local imports;
|
`docforge_get_task_context` is an additive read tool for `change`, `implementation`, `failure`,
|
||||||
- JavaScript and TypeScript, using their distinct optional Tree-sitter grammars and publishing only
|
`ownership`, `test`, `operation`, and `release` work. It derives a closed version-1 retrieval plan,
|
||||||
project-local static relative imports and re-exports;
|
executes it against one immutable index generation, and returns a hash-bound context capsule.
|
||||||
- C++, using `compile_commands.json` as translation-unit inventory and fingerprint evidence, without
|
Project relation names remain authoritative. DocForge classifies only its versioned alias set and
|
||||||
executing its commands or a compiler, and publishing only directly resolvable project-local
|
preserves every unknown relation as `unclassified` instead of guessing semantics.
|
||||||
quoted includes.
|
|
||||||
|
|
||||||
These integrations demonstrate the adapter contract; they do not claim resolved calls,
|
`docforge_get_generation_diff` reports the latest verified primary-graph transition through one
|
||||||
inheritance, types, runtime behavior, macro expansion, compiler include semantics, or semantic
|
bounded disposable receipt. It includes exact node and edge change counts, hash-bound retained
|
||||||
ownership. Python works from the base wheel. Install the `javascript`, `typescript`, or `cpp`
|
details, and explicit truncation. Paged results use one top-level cursor and a versioned
|
||||||
extra for the corresponding grammar, or `languages` for all three:
|
`receipt_header`; `stored_receipt_hash` identifies the complete persisted receipt. The read never
|
||||||
|
exposes Logic details, loads canonical source, repairs derived state, or invents history.
|
||||||
|
|
||||||
```bash
|
## Independent projections
|
||||||
uv pip install "/absolute/path/DocForge[javascript]"
|
|
||||||
uv pip install "/absolute/path/DocForge[typescript]"
|
|
||||||
uv pip install "/absolute/path/DocForge[cpp]"
|
|
||||||
uv pip install "/absolute/path/DocForge[languages]"
|
|
||||||
```
|
|
||||||
|
|
||||||
Reference projects use the fixed `.docforge/reference-adapter.toml` descriptor and the installed
|
Manual compilation, portable graph rendering, and the live viewer consume the same validated graph
|
||||||
`docforge.reference_mcp` module. That server is read-only and exposes no proposal or application
|
generation through separate boundaries:
|
||||||
surface.
|
|
||||||
|
|
||||||
Project-owned adapters use `AdapterLauncherV1` and
|
|
||||||
`generate_adapter_client_configuration()` to produce a client fragment from an explicitly
|
|
||||||
constructed adapter project. The launcher is immutable and contains no command, caller arguments,
|
|
||||||
working directory, environment, discovery rule, or callable selector. It resolves through isolated
|
|
||||||
Python to one installed, project-owned top-level module; the only trusted dotted exception is the
|
|
||||||
fixed `docforge.reference_mcp` binding. Generic `docforge configure` intentionally refuses custom
|
|
||||||
adapters because it cannot safely reconstruct project-owned composition.
|
|
||||||
|
|
||||||
See the [Adapter authoring guide](docs/ADAPTER_AUTHORING_GUIDE.md), [Reference
|
|
||||||
adapters](docs/REFERENCE_ADAPTERS.md), and [Agent integration](docs/AGENT_INTEGRATION.md) for the
|
|
||||||
supported routes.
|
|
||||||
|
|
||||||
## Authority and projections
|
|
||||||
|
|
||||||
Canonical Markdown, TOML, adapter-declared sources, and descriptor files own project facts. The
|
|
||||||
SQLite index, extraction cache, render packages, previews, portable artifacts, receipts, client
|
|
||||||
fragments, and viewer processes are derived and replaceable.
|
|
||||||
|
|
||||||
The primary graph contains project nodes and relationships. Logic is a separate, lazy,
|
|
||||||
function-scoped control-flow projection. Manual output, portable graph output, and the live viewer
|
|
||||||
are independent consumers of one validated generation:
|
|
||||||
|
|
||||||
```text
|
```text
|
||||||
validated generation
|
validated generation
|
||||||
|
|
@ -110,142 +91,190 @@ validated generation
|
||||||
└── pinned index → managed read-only live viewer
|
└── pinned index → managed read-only live viewer
|
||||||
```
|
```
|
||||||
|
|
||||||
The version-2 projection policy selects each consumer independently:
|
Plans, packages, and receipts are canonical, versioned, hash-identified, bounded, and contain no
|
||||||
|
project object, SQLite handle, absolute project path, command, or caller-selected renderer module.
|
||||||
|
Renderers cannot select graph facts, crawl canonical sources, choose publication paths, or mutate
|
||||||
|
project state.
|
||||||
|
|
||||||
```text
|
Declare portable graph output separately from manual views:
|
||||||
manual: auto | explicit | disabled
|
|
||||||
portable_graph: explicit | disabled
|
```toml
|
||||||
live_viewer: on-demand | disabled
|
[graph_render]
|
||||||
|
output_root = ".docforge/portable-graph"
|
||||||
|
|
||||||
|
[[graph_render.views]]
|
||||||
|
id = "architecture"
|
||||||
|
renderer = "portable_graph_html"
|
||||||
|
output = "architecture.html"
|
||||||
|
title = "Architecture"
|
||||||
|
root = "architecture.overview"
|
||||||
|
initial_mode = "web"
|
||||||
|
depth = 3
|
||||||
|
max_nodes = 250
|
||||||
|
max_edges = 1000
|
||||||
|
max_work = 100000
|
||||||
|
include_logic = false
|
||||||
```
|
```
|
||||||
|
|
||||||
The process capability policy is separate. Capability mode controls the registered read,
|
Plan, publish, and inspect it explicitly:
|
||||||
proposal, application, or reserved operator surface; descriptor writers and the startup-bound
|
|
||||||
canonical applier determine whether mutations are actually authorized. `--no-ast` is a restrictive
|
|
||||||
binding policy over adapter evolution, Logic publication, and Logic retrieval. It is not a parser
|
|
||||||
inspection mechanism or a filesystem sandbox, and the AST/Tree-sitter reference integrations
|
|
||||||
should not be presented as no-AST adapters.
|
|
||||||
|
|
||||||
Read [Policy precedence](docs/POLICY_PRECEDENCE.md), [Rendering and
|
|
||||||
visualization](docs/RENDERING_AND_VISUALIZATION.md), and [Legacy and no-AST
|
|
||||||
operation](docs/LEGACY_AND_NO_AST.md) before changing those boundaries.
|
|
||||||
|
|
||||||
## Five-minute generic project
|
|
||||||
|
|
||||||
Requirements are Python 3.12 or newer and an installed DocForge environment. From a development
|
|
||||||
checkout, `uv sync --group dev` creates `.venv`:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone <repository-url> /absolute/path/DocForge
|
.venv/bin/docforge --project-root "$PROJECT" graph-plan architecture
|
||||||
cd /absolute/path/DocForge
|
.venv/bin/docforge --project-root "$PROJECT" graph-render architecture
|
||||||
|
.venv/bin/docforge --project-root "$PROJECT" graph-render-status architecture
|
||||||
|
```
|
||||||
|
|
||||||
|
The version-2 projection policy independently selects manual
|
||||||
|
`auto|explicit|disabled`, portable graph `explicit|disabled`, and live viewer
|
||||||
|
`on-demand|disabled`. Use `--manual-render-policy`, `--portable-graph-policy`, and
|
||||||
|
`--live-viewer-policy` on CLI/MCP startup or generated client configuration. Status remains
|
||||||
|
available when the corresponding active operation is disabled.
|
||||||
|
|
||||||
|
Manual fragment reuse is disposable. Cold record publication is guarded by byte-exact comparison
|
||||||
|
with a full detached render; warm records are independently recomputed and validated inside the
|
||||||
|
worker. Full rendering remains the recovery and equivalence oracle. Portable artifacts commit
|
||||||
|
content-addressed output and renderer evidence before one bounded generation/view manifest; status
|
||||||
|
does not plan or render.
|
||||||
|
|
||||||
|
## Graph views
|
||||||
|
|
||||||
|
The browser presents the primary architecture graph through three complementary views and loads a
|
||||||
|
fourth function-scoped view only when requested:
|
||||||
|
|
||||||
|
- **Nodes** shows a bounded, relation-neutral neighborhood around the focus. It is the broad
|
||||||
|
inspection view for seeing stored incoming and outgoing relationships without changing their
|
||||||
|
direction. Semantic cards distinguish structure, behavior, dependencies, execution, data,
|
||||||
|
evidence, context, and other relationships.
|
||||||
|
- **Flow** shows semantic origin-to-destination paths that terminate at the focus. DocForge
|
||||||
|
reverses prerequisite-style relationships for presentation, so imports, dependencies, reads,
|
||||||
|
inheritance, definitions, and tests flow toward the thing they help create or exercise.
|
||||||
|
- **Web** shows the larger convergence picture: Flow contributors plus contextual relationships,
|
||||||
|
callers, containers, and direct members or execution dependencies owned by the focus.
|
||||||
|
- **Logic** shows the possible static control paths inside a focused Python, JavaScript, or C++
|
||||||
|
function or method. Entry, decisions, actions, loops, convergence points, returns, and
|
||||||
|
exceptions connect through explicit
|
||||||
|
`TRUE`, `FALSE`, `NEXT`, `CASE`, `LOOP`, `RETURN`, and `RAISE` paths. Logic is stored separately
|
||||||
|
and does not add statement-level noise to Nodes, Flow, Web, or search.
|
||||||
|
|
||||||
|
Graph cards show the node's readable leaf name and kind without clipping either value. The full
|
||||||
|
qualified identity remains available in the tooltip, compact descriptor, and full inspector.
|
||||||
|
The left browser panel can combine text, family, node-kind, language, and capability filters.
|
||||||
|
Quick presets expose Logic-ready nodes, Python callables, tests, routes, and documentation without
|
||||||
|
requiring users to know stable IDs. Selecting a canvas node emphasizes its directly connected
|
||||||
|
neighbors and edges while muting unrelated paths.
|
||||||
|
|
||||||
|
**Hide node** removes noise without changing the index. In Flow and Web, hiding a contributor also
|
||||||
|
removes upstream ancestors that no longer have a path to the focus. Nodes between the hidden
|
||||||
|
contributor and the focus stay visible, and alternate ancestor paths remain intact. In Logic,
|
||||||
|
hiding a step inserts an explicit omitted-path bridge so downstream control flow remains readable.
|
||||||
|
**Restore hidden** restores the presentation.
|
||||||
|
|
||||||
|
## Five-minute start
|
||||||
|
|
||||||
|
Requirements are Python 3.12+, `uv`, and Node.js/npm.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone forgejo@repo.andraxion.net:administrator/DocForge2.git /absolute/path/DocForge2
|
||||||
|
cd /absolute/path/DocForge2
|
||||||
uv sync --group dev
|
uv sync --group dev
|
||||||
|
npm ci
|
||||||
|
|
||||||
PROJECT=/absolute/path/MyProject
|
PROJECT=/absolute/path/MyProject
|
||||||
.venv/bin/docforge --project-root "$PROJECT" onboard
|
|
||||||
.venv/bin/docforge --project-root "$PROJECT" onboard \
|
|
||||||
--scaffold \
|
|
||||||
--project-id my-project \
|
|
||||||
--title "My Project"
|
|
||||||
```
|
|
||||||
|
|
||||||
The first command is read-only. Scaffolding is explicit and create-only: it writes a generic
|
|
||||||
descriptor, one canonical overview node, and a built-in manual template, then indexes and renders
|
|
||||||
them. Detected source languages remain `adapter_required` until a validated frontend is selected.
|
|
||||||
|
|
||||||
Operate the configured project:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
.venv/bin/docforge --project-root "$PROJECT" validate
|
.venv/bin/docforge --project-root "$PROJECT" validate
|
||||||
.venv/bin/docforge --project-root "$PROJECT" reindex
|
.venv/bin/docforge --project-root "$PROJECT" reindex
|
||||||
.venv/bin/docforge --project-root "$PROJECT" search architecture
|
|
||||||
.venv/bin/docforge --project-root "$PROJECT" visualize
|
.venv/bin/docforge --project-root "$PROJECT" visualize
|
||||||
```
|
```
|
||||||
|
|
||||||
Install the persistent per-user graph viewer manager once when using the live viewer:
|
Install the persistent per-user graph viewer once:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
.venv/bin/docforge-viewer-manager install-user-service
|
.venv/bin/docforge-viewer-manager install-user-service
|
||||||
```
|
```
|
||||||
|
|
||||||
Start one generic, read-only MCP server:
|
Start an MCP server for one project:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
.venv/bin/docforge-mcp \
|
.venv/bin/docforge-mcp \
|
||||||
--project-root "$PROJECT" \
|
--project-root "$PROJECT" \
|
||||||
--capability-mode read
|
--proposal-writer project-editor
|
||||||
```
|
```
|
||||||
|
|
||||||
Add a descriptor-authorized proposal writer only when the client should create proposals. Add a
|
Add `--canonical-applier project-editor` only when that MCP integration should expose the
|
||||||
matching `--canonical-applier` only when that integration should expose exact-hash application.
|
hash-bound `docforge_apply_changeset` tool.
|
||||||
See the [quickstart](docs/NEW_PROJECT_QUICKSTART.md) for the reference-adapter and generated-client
|
|
||||||
paths.
|
Preview a read-only Codex fragment and diagnose an installed binding:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/docforge configure codex --project "$PROJECT"
|
||||||
|
.venv/bin/docforge doctor --client codex --project "$PROJECT"
|
||||||
|
```
|
||||||
|
|
||||||
|
Pass `--output /absolute/path/docforge.toml` only when creating a standalone fragment. DocForge
|
||||||
|
never replaces or merges an existing different client file.
|
||||||
|
|
||||||
|
For an unconfigured codebase, begin with a read-only language and documentation assessment:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/docforge --project-root /absolute/path/MyProject onboard
|
||||||
|
```
|
||||||
|
|
||||||
|
Add `--scaffold`, a stable project ID, and a title to create, index, and render a generic starter
|
||||||
|
manual. Source files are reported separately and require a validated language frontend before
|
||||||
|
DocForge describes them as a source graph.
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
### Learn and operate
|
- [User manual](docs/USER_MANUAL.md) — features, setup, visualization, CLI, MCP, apply, adapters,
|
||||||
|
and troubleshooting.
|
||||||
- [New-project quickstart](docs/NEW_PROJECT_QUICKSTART.md)
|
- [Core contract](docs/CONTRACT.md) — invariants and security boundary.
|
||||||
- [Core concepts and authority](docs/CORE_CONCEPTS_AND_AUTHORITY.md)
|
- [Milestone 0 compatibility](docs/COMPATIBILITY.md) — preserved package, CLI, MCP, adapter,
|
||||||
- [Project descriptor](docs/PROJECT_DESCRIPTOR.md)
|
schema, changeset, rendering, and no-AST guarantees.
|
||||||
- [Policy precedence](docs/POLICY_PRECEDENCE.md)
|
- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) — validation evidence, cold and warm
|
||||||
- [Reference adapters](docs/REFERENCE_ADAPTERS.md)
|
performance, memory, rendering and response sizes, bottlenecks, and missing coverage.
|
||||||
- [Agent integration](docs/AGENT_INTEGRATION.md)
|
- [Milestone 0 closeout](docs/MILESTONE_0_CLOSEOUT.md) — lineage, migration, security scan,
|
||||||
- [User manual](docs/USER_MANUAL.md)
|
repository state, and fresh-clone proof.
|
||||||
- [Generated command reference](docs/COMMAND_REFERENCE.md)
|
- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) — warm operation latency, structured work,
|
||||||
- [Project onboarding](docs/PROJECT_ONBOARDING.md)
|
status, retrieval, and memory measurements.
|
||||||
- [Rendering and visualization](docs/RENDERING_AND_VISUALIZATION.md)
|
- [Milestone 1 closeout](docs/MILESTONE_1_CLOSEOUT.md) — fast-core contracts, adversarial
|
||||||
- [Recovery and performance](docs/RECOVERY_AND_PERFORMANCE.md)
|
validation, compatibility boundaries, and exact candidate evidence.
|
||||||
- [Security](docs/SECURITY.md)
|
- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) — task context, generation diff, client
|
||||||
|
configuration, doctor, response-size, counter, and memory measurements.
|
||||||
### Contracts and compatibility
|
- [Milestone 2 closeout](docs/MILESTONE_2_CLOSEOUT.md) — implemented contracts, adversarial
|
||||||
|
validation, exclusions, and exact candidate evidence.
|
||||||
- [Core contract](docs/CONTRACT.md)
|
- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) — manual, fragment, worker, portable graph,
|
||||||
- [MCP contract](docs/MCP_CONTRACT.md)
|
status, equivalence, response-size, and memory measurements.
|
||||||
- [Compatibility contract](docs/COMPATIBILITY.md)
|
- [Milestone 3 closeout](docs/MILESTONE_3_CLOSEOUT.md) — independent projection contracts,
|
||||||
- [Adapter authoring guide](docs/ADAPTER_AUTHORING_GUIDE.md)
|
adversarial validation, compatibility boundaries, and exact candidate evidence.
|
||||||
- [Incremental indexing](docs/INCREMENTAL_INDEXING.md)
|
- [MCP contract](docs/MCP_CONTRACT.md) — exact tool and process boundary.
|
||||||
- [Adapter application decision](docs/APPLICATION_DECISION.md)
|
- [Viewer manager](docs/VIEWER_MANAGER.md) — native service setup and lifecycle.
|
||||||
- [Legacy and no-AST operation](docs/LEGACY_AND_NO_AST.md)
|
- [Adapter decision](docs/APPLICATION_DECISION.md) — why custom adapters own canonical
|
||||||
- [Migrating from version 1](docs/MIGRATING_FROM_V1.md)
|
serialization.
|
||||||
- [Viewer manager](docs/VIEWER_MANAGER.md)
|
- [Incremental adapter indexing](docs/INCREMENTAL_INDEXING.md) — source-scoped extraction,
|
||||||
|
invalidation, equivalence, relationship changes, and the lazy Logic boundary.
|
||||||
### Milestone evidence
|
- [Project onboarding](docs/PROJECT_ONBOARDING.md) — repository assessment, safe manual
|
||||||
|
scaffolding, language frontends, source/manual integration, proof, and MCP activation.
|
||||||
- [Milestone 4 baseline](docs/MILESTONE_4_BASELINE.md)
|
- [Language adapter authoring](docs/ADAPTER_AUTHORING_GUIDE.md) — implementation sequence,
|
||||||
- [Milestone 4 closeout](docs/MILESTONE_4_CLOSEOUT.md)
|
stable identities, overlap ownership, normalization, incremental equivalence, troubleshooting,
|
||||||
- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) and [closeout](docs/MILESTONE_3_CLOSEOUT.md)
|
and the complete adapter proof matrix.
|
||||||
- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) and [closeout](docs/MILESTONE_2_CLOSEOUT.md)
|
|
||||||
- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) and [closeout](docs/MILESTONE_1_CLOSEOUT.md)
|
|
||||||
- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) and [closeout](docs/MILESTONE_0_CLOSEOUT.md)
|
|
||||||
|
|
||||||
Historical milestone records preserve the facts and dependency observations of their frozen
|
|
||||||
candidates. Use the current guides and contracts for present behavior.
|
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
Run the repository-native gate:
|
Run the complete repository-native gate:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
make gate
|
make gate
|
||||||
```
|
```
|
||||||
|
|
||||||
Milestone 4 maintenance entry points include:
|
Focused entry points are available as `make contract`, `make test`, `make type`,
|
||||||
|
`make benchmark-smoke`, `make benchmark`, `make benchmark-m1-smoke`, and
|
||||||
|
`make benchmark-m1`. Milestone 2 adds `make benchmark-m2-smoke` and `make benchmark-m2`.
|
||||||
|
Milestone 3 adds `make accessibility`, `make benchmark-m3-smoke`, `make benchmark-m3`, and
|
||||||
|
`make benchmark-m3-full`.
|
||||||
|
|
||||||
```bash
|
The committed 1,000-node baseline and its measurement method are under `benchmarks/`.
|
||||||
make adoption-m4
|
|
||||||
make benchmark-m4-smoke
|
|
||||||
make benchmark-m4
|
|
||||||
make benchmark-m4-full
|
|
||||||
make command-reference-check
|
|
||||||
make docs-check
|
|
||||||
```
|
|
||||||
|
|
||||||
`adoption-m4` builds and exercises a fresh base wheel without Tree-sitter packages and proves the
|
Pass `--diagnostics` to `docforge` or `docforge-mcp` to attach bounded request-local stage timings
|
||||||
Python reference plus a real isolated read-only MCP retrieval. The optional-language integrations
|
and compiler-work counters. Diagnostics are disabled by default and are dropped before primary MCP
|
||||||
have their own focused tests and extras. `benchmark-m4` runs the maintained full adapter workload;
|
results when the configured output budget is tight.
|
||||||
the smoke target is for routine gate coverage, not final performance evidence.
|
|
||||||
|
|
||||||
Pass `--diagnostics` to `docforge` or `docforge-mcp` for bounded request-local timings and compiler
|
See [AGENTS.md](AGENTS.md) before changing core boundaries.
|
||||||
work counters. Diagnostics are disabled by default and do not displace a primary result when the
|
|
||||||
configured output budget is tight.
|
|
||||||
|
|
||||||
Read [AGENTS.md](AGENTS.md) before changing core boundaries.
|
|
||||||
|
|
|
||||||
|
|
@ -1,50 +1,5 @@
|
||||||
# Completed slices
|
# Completed slices
|
||||||
|
|
||||||
## DocForge2 Milestone 4 adapter SDK and product documentation
|
|
||||||
|
|
||||||
### Changed
|
|
||||||
|
|
||||||
- Added stable `docforge.adapter_sdk` authoring imports and a conformance helper for deterministic
|
|
||||||
complete and exact complete/incremental primary graph plus Logic equivalence.
|
|
||||||
- Added bounded Python, JavaScript, TypeScript, and C++ reference integrations with explicit
|
|
||||||
unsupported-fact reports and optional heavy frontends.
|
|
||||||
- Added a closed `.docforge/reference-adapter.toml` contract and fixed read-only reference MCP
|
|
||||||
server.
|
|
||||||
- Added immutable project-bound launcher declarations and deterministic Codex, Claude, and OpenClaw
|
|
||||||
configuration generation for custom adapters.
|
|
||||||
- Generated CLI and MCP references from live implementation metadata and added race-safe
|
|
||||||
publication plus repository drift checks.
|
|
||||||
- Added strict documentation-graph and fresh-wheel adoption gates and dedicated onboarding,
|
|
||||||
authority, descriptor, policy, adapter, legacy/no-AST, rendering, agent-integration, security,
|
|
||||||
recovery/performance, and v1 migration guides.
|
|
||||||
|
|
||||||
### Verification
|
|
||||||
|
|
||||||
- The executable candidate passed 347 tests and 402 subtests, 142 contract tests and 268 subtests,
|
|
||||||
three accessibility flows, lint, strict types, compilation, dependencies, builds, adoption, and
|
|
||||||
every maintained smoke benchmark.
|
|
||||||
- The offline base wheel contained no Tree-sitter distribution and ran a real Python reference
|
|
||||||
build/check and isolated 21-tool MCP retrieval flow. Missing C++ support failed with the exact
|
|
||||||
`docforge[cpp]` remediation.
|
|
||||||
- The clean 334-source benchmark produced 1,002 nodes and exact full/incremental graph and Logic
|
|
||||||
evidence, warm zero Python parsing/extraction, and exact corrupt-cache and corrupt-index
|
|
||||||
recovery below every latency, memory, and response gate.
|
|
||||||
- Gitleaks found no findings in the Milestone 4 commit range or candidate tree.
|
|
||||||
|
|
||||||
### Limits
|
|
||||||
|
|
||||||
- The reference integrations publish syntax and local static relationships. They do not claim
|
|
||||||
compiler-resolved calls, types, inheritance, macros, runtime behavior, or semantic ownership.
|
|
||||||
- C++ compilation-database commands are inert and never executed.
|
|
||||||
- C++ manifest include discovery uses Tree-sitter and makes no zero-warm-parser claim.
|
|
||||||
- No WorldForge, ScrapeStation, legacy repository, production binding, language distribution,
|
|
||||||
self-hosting, tag, or release was changed.
|
|
||||||
|
|
||||||
### Next gate
|
|
||||||
|
|
||||||
Milestone 5 remains directional. Create a new active-slice contract before stabilization,
|
|
||||||
versioning, tagging, or publication.
|
|
||||||
|
|
||||||
## DocForge2 Milestone 3 independent projections
|
## DocForge2 Milestone 3 independent projections
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|
|
||||||
|
|
@ -42,14 +42,6 @@ make benchmark-m3
|
||||||
make benchmark-m3-full
|
make benchmark-m3-full
|
||||||
```
|
```
|
||||||
|
|
||||||
Run the Milestone 4 adapter SDK smoke, maintained full, and fresh-wheel adoption gates:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
make benchmark-m4-smoke
|
|
||||||
make benchmark-m4-full
|
|
||||||
make adoption-m4
|
|
||||||
```
|
|
||||||
|
|
||||||
The benchmark creates canonical sources, derived state, changesets, rendered output, and caches
|
The benchmark creates canonical sources, derived state, changesets, rendered output, and caches
|
||||||
only in a disposable temporary directory. It does not read another project, self-host DocForge, or
|
only in a disposable temporary directory. It does not read another project, self-host DocForge, or
|
||||||
mutate repository content.
|
mutate repository content.
|
||||||
|
|
@ -79,13 +71,6 @@ sweeps, add/change/delete/reorder equivalence, portable publication, receipt-onl
|
||||||
memory, detached worker peak memory, and response size. Its interpretation is in
|
memory, detached worker peak memory, and response size. Its interpretation is in
|
||||||
[`docs/MILESTONE_3_BASELINE.md`](../docs/MILESTONE_3_BASELINE.md).
|
[`docs/MILESTONE_3_BASELINE.md`](../docs/MILESTONE_3_BASELINE.md).
|
||||||
|
|
||||||
`milestone4-2026-07-29.json` is the clean-tree adapter SDK and reference-Python baseline captured
|
|
||||||
from commit `95271dcf2e48045b9d3aed9b9ea09c7fc155692c`. It measures cold and warm
|
|
||||||
incremental builds, complete/incremental graph-plus-Logic equivalence, corrupt extraction-cache
|
|
||||||
recovery, corrupt-index recovery, parser and extraction counters, traced and process memory, and
|
|
||||||
bounded response size. Its interpretation is in
|
|
||||||
[`docs/MILESTONE_4_BASELINE.md`](../docs/MILESTONE_4_BASELINE.md).
|
|
||||||
|
|
||||||
The generic fixtures expose whole-source and projection scaling. They do not replace incremental
|
The generic fixtures expose whole-source and projection scaling. They do not replace incremental
|
||||||
adapter equivalence tests. Milestone 3's portable fixture contains 1,000 Nodes/Flow/Web nodes and
|
adapter equivalence tests. Milestone 3's portable fixture contains 1,000 Nodes/Flow/Web nodes and
|
||||||
999 edges; portable version 1 deliberately excludes Logic.
|
999 edges; portable version 1 deliberately excludes Logic.
|
||||||
|
|
@ -104,5 +89,4 @@ backlinks, both traversal directions, paged context, render receipt states, and
|
||||||
freshness. The reported p95 uses the nearest-rank method; with ten samples it is the maximum.
|
freshness. The reported p95 uses the nearest-rank method; with ten samples it is the maximum.
|
||||||
`process_peak_rss_kib` is the cumulative main-process `RUSAGE_SELF` high-water mark and excludes the
|
`process_peak_rss_kib` is the cumulative main-process `RUSAGE_SELF` high-water mark and excludes the
|
||||||
detached viewer worker. It is diagnostic and not operation-local. Milestone 3 memory gates use
|
detached viewer worker. It is diagnostic and not operation-local. Milestone 3 memory gates use
|
||||||
per-operation `tracemalloc` peaks and detached worker receipt peaks instead. Milestone 4 uses
|
per-operation `tracemalloc` peaks and detached worker receipt peaks instead.
|
||||||
per-operation `tracemalloc` peaks and a cumulative benchmark-process high-water gate.
|
|
||||||
|
|
|
||||||
|
|
@ -1,205 +0,0 @@
|
||||||
{
|
|
||||||
"benchmark": "docforge2_milestone4",
|
|
||||||
"environment": {
|
|
||||||
"implementation": "CPython",
|
|
||||||
"machine": "x86_64",
|
|
||||||
"platform": "Linux-7.1.3-200.nobara.fc44.x86_64-x86_64-with-glibc2.43",
|
|
||||||
"python": "3.14.6"
|
|
||||||
},
|
|
||||||
"evidence": {
|
|
||||||
"cache_recovery_exact": true,
|
|
||||||
"complete_assembly_hash": "2888182ab765fbffe3ba873c1613345640e8e6d89be74ddfca7452c0a5056345",
|
|
||||||
"edge_count": 1001,
|
|
||||||
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
|
|
||||||
"full_incremental_graph_and_logic_exact": true,
|
|
||||||
"index_recovery_exact": true,
|
|
||||||
"logic_edge_count": 2338,
|
|
||||||
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
|
|
||||||
"logic_node_count": 2338,
|
|
||||||
"logic_projection_count": 334,
|
|
||||||
"node_count": 1002,
|
|
||||||
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
|
|
||||||
"source_count": 334,
|
|
||||||
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
|
|
||||||
"warm_zero_ast_parse": true,
|
|
||||||
"warm_zero_source_extraction": true
|
|
||||||
},
|
|
||||||
"evidence_sha256": "4a8db461a311b5df4abd5aa00063e9a347d8b9dba19b30e35684f561d5271549",
|
|
||||||
"fixture": {
|
|
||||||
"expected_node_count": 1002,
|
|
||||||
"kind": "synthetic_python_reference_adapter",
|
|
||||||
"nodes_per_source": 3,
|
|
||||||
"source_count": 334
|
|
||||||
},
|
|
||||||
"memory": {
|
|
||||||
"process_peak_bytes": 78798848,
|
|
||||||
"process_peak_limit_bytes": 536870912
|
|
||||||
},
|
|
||||||
"method": {
|
|
||||||
"ast_instrumentation": "temporary counter around stdlib ast.parse",
|
|
||||||
"clock": "time.perf_counter_ns",
|
|
||||||
"extraction_instrumentation": "reference adapter extract_source entry counter",
|
|
||||||
"full_mode_source_requirement": 334,
|
|
||||||
"in_process_peak_memory": "tracemalloc per measured invocation",
|
|
||||||
"process_peak_memory": "resource.getrusage(RUSAGE_SELF).ru_maxrss",
|
|
||||||
"response_size": "UTF-8 bytes of compact sorted JSON",
|
|
||||||
"samples": 3,
|
|
||||||
"threshold_basis": "Regression tripwires leave several times the Milestone 3 1,000-node allowances for AST, Logic, extraction-cache, and SQLite work."
|
|
||||||
},
|
|
||||||
"mode": "full",
|
|
||||||
"operations": {
|
|
||||||
"cold_incremental_build": {
|
|
||||||
"max_ms": 1599.76,
|
|
||||||
"maximum_response_bytes": 14580,
|
|
||||||
"maximum_traced_peak_bytes": 68573540,
|
|
||||||
"median_ms": 1599.76,
|
|
||||||
"min_ms": 1599.76,
|
|
||||||
"p95_limit_ms": 30000.0,
|
|
||||||
"p95_ms": 1599.76,
|
|
||||||
"response_limit_bytes": 262144,
|
|
||||||
"samples": 1,
|
|
||||||
"stable_result": {
|
|
||||||
"ast_parse_calls": 668,
|
|
||||||
"cache_hits": 0,
|
|
||||||
"deleted_sources": 0,
|
|
||||||
"edge_count": 1001,
|
|
||||||
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
|
|
||||||
"extraction_calls": 334,
|
|
||||||
"invalidated_sources": 334,
|
|
||||||
"logic_edge_count": 2338,
|
|
||||||
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
|
|
||||||
"logic_node_count": 2338,
|
|
||||||
"logic_projection_count": 334,
|
|
||||||
"node_count": 1002,
|
|
||||||
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
|
|
||||||
"reparsed_sources": 334,
|
|
||||||
"revision": "30ef23aa4106",
|
|
||||||
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
|
|
||||||
"status": "ok",
|
|
||||||
"total_sources": 334
|
|
||||||
},
|
|
||||||
"traced_peak_limit_bytes": 268435456
|
|
||||||
},
|
|
||||||
"corrupt_extraction_cache_recovery": {
|
|
||||||
"max_ms": 1695.406,
|
|
||||||
"maximum_response_bytes": 14580,
|
|
||||||
"maximum_traced_peak_bytes": 69776923,
|
|
||||||
"median_ms": 1695.406,
|
|
||||||
"min_ms": 1695.406,
|
|
||||||
"p95_limit_ms": 30000.0,
|
|
||||||
"p95_ms": 1695.406,
|
|
||||||
"response_limit_bytes": 262144,
|
|
||||||
"samples": 1,
|
|
||||||
"stable_result": {
|
|
||||||
"ast_parse_calls": 668,
|
|
||||||
"cache_hits": 0,
|
|
||||||
"deleted_sources": 0,
|
|
||||||
"edge_count": 1001,
|
|
||||||
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
|
|
||||||
"extraction_calls": 334,
|
|
||||||
"invalidated_sources": 334,
|
|
||||||
"logic_edge_count": 2338,
|
|
||||||
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
|
|
||||||
"logic_node_count": 2338,
|
|
||||||
"logic_projection_count": 334,
|
|
||||||
"node_count": 1002,
|
|
||||||
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
|
|
||||||
"reparsed_sources": 334,
|
|
||||||
"revision": "30ef23aa4106",
|
|
||||||
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
|
|
||||||
"status": "ok",
|
|
||||||
"total_sources": 334
|
|
||||||
},
|
|
||||||
"traced_peak_limit_bytes": 268435456
|
|
||||||
},
|
|
||||||
"corrupt_index_recovery": {
|
|
||||||
"max_ms": 1707.879,
|
|
||||||
"maximum_response_bytes": 14779,
|
|
||||||
"maximum_traced_peak_bytes": 68561924,
|
|
||||||
"median_ms": 1707.879,
|
|
||||||
"min_ms": 1707.879,
|
|
||||||
"p95_limit_ms": 30000.0,
|
|
||||||
"p95_ms": 1707.879,
|
|
||||||
"response_limit_bytes": 262144,
|
|
||||||
"samples": 1,
|
|
||||||
"stable_result": {
|
|
||||||
"action": "rebuilt",
|
|
||||||
"ast_parse_calls": 0,
|
|
||||||
"cache_hits": 334,
|
|
||||||
"edge_count": 1001,
|
|
||||||
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
|
|
||||||
"extraction_calls": 0,
|
|
||||||
"initial_error_code": "invalid_index",
|
|
||||||
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
|
|
||||||
"logic_projection_count": 334,
|
|
||||||
"node_count": 1002,
|
|
||||||
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
|
|
||||||
"reparsed_sources": 0,
|
|
||||||
"revision": "30ef23aa4106",
|
|
||||||
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
|
|
||||||
"status": "ok"
|
|
||||||
},
|
|
||||||
"traced_peak_limit_bytes": 268435456
|
|
||||||
},
|
|
||||||
"full_incremental_equivalence": {
|
|
||||||
"max_ms": 1897.919,
|
|
||||||
"maximum_response_bytes": 230,
|
|
||||||
"maximum_traced_peak_bytes": 64305919,
|
|
||||||
"median_ms": 1897.919,
|
|
||||||
"min_ms": 1897.919,
|
|
||||||
"p95_limit_ms": 30000.0,
|
|
||||||
"p95_ms": 1897.919,
|
|
||||||
"response_limit_bytes": 262144,
|
|
||||||
"samples": 1,
|
|
||||||
"stable_result": {
|
|
||||||
"ast_parse_calls": 1336,
|
|
||||||
"edge_count": 1001,
|
|
||||||
"extraction_calls": 668,
|
|
||||||
"logic_projection_count": 334,
|
|
||||||
"node_count": 1002,
|
|
||||||
"project_id": "milestone4-python-benchmark",
|
|
||||||
"revision": "30ef23aa4106",
|
|
||||||
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
|
|
||||||
"status": "ok"
|
|
||||||
},
|
|
||||||
"traced_peak_limit_bytes": 268435456
|
|
||||||
},
|
|
||||||
"warm_incremental_build": {
|
|
||||||
"max_ms": 1265.387,
|
|
||||||
"maximum_response_bytes": 14578,
|
|
||||||
"maximum_traced_peak_bytes": 69997166,
|
|
||||||
"median_ms": 1244.427,
|
|
||||||
"min_ms": 1239.941,
|
|
||||||
"p95_limit_ms": 20000.0,
|
|
||||||
"p95_ms": 1265.387,
|
|
||||||
"response_limit_bytes": 262144,
|
|
||||||
"samples": 3,
|
|
||||||
"stable_result": {
|
|
||||||
"ast_parse_calls": 0,
|
|
||||||
"cache_hits": 334,
|
|
||||||
"deleted_sources": 0,
|
|
||||||
"edge_count": 1001,
|
|
||||||
"edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35",
|
|
||||||
"extraction_calls": 0,
|
|
||||||
"invalidated_sources": 0,
|
|
||||||
"logic_edge_count": 2338,
|
|
||||||
"logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c",
|
|
||||||
"logic_node_count": 2338,
|
|
||||||
"logic_projection_count": 334,
|
|
||||||
"node_count": 1002,
|
|
||||||
"node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f",
|
|
||||||
"reparsed_sources": 0,
|
|
||||||
"revision": "30ef23aa4106",
|
|
||||||
"source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2",
|
|
||||||
"status": "ok",
|
|
||||||
"total_sources": 334
|
|
||||||
},
|
|
||||||
"traced_peak_limit_bytes": 268435456
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"schema_version": 1,
|
|
||||||
"source": {
|
|
||||||
"dirty": false,
|
|
||||||
"revision": "95271dcf2e48045b9d3aed9b9ea09c7fc155692c"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -8,30 +8,6 @@ Use this guide after the repository assessment in
|
||||||
[Project Onboarding](PROJECT_ONBOARDING.md). The onboarding checklist decides whether an adapter
|
[Project Onboarding](PROJECT_ONBOARDING.md). The onboarding checklist decides whether an adapter
|
||||||
is needed. This guide defines how to build and prove one.
|
is needed. This guide defines how to build and prove one.
|
||||||
|
|
||||||
## Public authoring surface
|
|
||||||
|
|
||||||
Adapter authors should import typed contracts, validation helpers, graph models, and the
|
|
||||||
conformance helper from `docforge.adapter_sdk`:
|
|
||||||
|
|
||||||
```python
|
|
||||||
from docforge.adapter_sdk import (
|
|
||||||
AdapterAssembly,
|
|
||||||
AdapterImplementation,
|
|
||||||
AdapterManifest,
|
|
||||||
AdapterProject,
|
|
||||||
AdapterProjectSettings,
|
|
||||||
AdapterProjection,
|
|
||||||
AdapterSource,
|
|
||||||
AdapterSourceProjection,
|
|
||||||
LogicProjection,
|
|
||||||
verify_adapter_conformance,
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
Language-specific modules under `docforge.adapters` are repository reference implementations, not
|
|
||||||
the general authoring namespace. Their exact, deliberately narrow behavior is documented in
|
|
||||||
[Reference Adapters](REFERENCE_ADAPTERS.md).
|
|
||||||
|
|
||||||
## Required outcome
|
## Required outcome
|
||||||
|
|
||||||
A production adapter must provide one reproducible public graph from authoritative project
|
A production adapter must provide one reproducible public graph from authoritative project
|
||||||
|
|
@ -75,39 +51,11 @@ class MyAdapter:
|
||||||
manifest: AdapterManifest,
|
manifest: AdapterManifest,
|
||||||
contributions: tuple[AdapterSourceProjection, ...],
|
contributions: tuple[AdapterSourceProjection, ...],
|
||||||
) -> AdapterAssembly: ...
|
) -> AdapterAssembly: ...
|
||||||
def load_complete_assembly(self) -> AdapterAssembly: ...
|
|
||||||
def load_projection(self) -> AdapterProjection: ...
|
def load_projection(self) -> AdapterProjection: ...
|
||||||
```
|
```
|
||||||
|
|
||||||
`assemble_projection()` is optional only when extraction units already have disjoint ownership.
|
`assemble_projection()` is optional only when extraction units already have disjoint ownership.
|
||||||
`load_projection()` is always required. It is the clean primary-graph compatibility oracle.
|
`load_projection()` is always required. It is the clean rebuild and equivalence oracle.
|
||||||
`load_complete_assembly()` supplies the cache-independent complete graph-plus-Logic oracle. It is
|
|
||||||
required when incremental contributions publish Logic; without it, DocForge cannot prove Logic
|
|
||||||
parity.
|
|
||||||
|
|
||||||
## What the conformance helper proves
|
|
||||||
|
|
||||||
Run the public helper against a fresh confined cache root:
|
|
||||||
|
|
||||||
```python
|
|
||||||
report = verify_adapter_conformance(
|
|
||||||
MyAdapter(project_root),
|
|
||||||
cache_root=project_root / ".docforge" / "cache" / "adapter-conformance",
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
`verify_adapter_conformance()` proves:
|
|
||||||
|
|
||||||
- two repeated complete assemblies are exactly deterministic, including Logic;
|
|
||||||
- when `load_complete_assembly()` exists, its primary graph exactly matches
|
|
||||||
`load_projection()`; and
|
|
||||||
- when the incremental methods exist, the assembled incremental graph and Logic exactly match the
|
|
||||||
independent complete oracle.
|
|
||||||
|
|
||||||
The report records stable identity, counts, and an assembly hash. This helper does not by itself
|
|
||||||
prove process confinement, implementation restart behavior, corrupt-cache recovery, warm zero
|
|
||||||
parsing, retrieval, or every case in the proof matrix below. Keep those as separate focused and
|
|
||||||
integration tests.
|
|
||||||
|
|
||||||
## Step 1: define authority before parsing
|
## Step 1: define authority before parsing
|
||||||
|
|
||||||
|
|
@ -376,25 +324,6 @@ The manifest remains a current source snapshot, not a Git-index snapshot. A Git-
|
||||||
omit a deleted source whether its deletion is unstaged or staged. Staging is never a required
|
omit a deleted source whether its deletion is unstaged or staged. Staging is never a required
|
||||||
DocForge synchronization step.
|
DocForge synchronization step.
|
||||||
|
|
||||||
### Current aggregate bounds
|
|
||||||
|
|
||||||
DocForge limits one extraction-cache generation to 10,000 source contributions and 64,000,000
|
|
||||||
encoded bytes. Malformed, incompatible, missing, oversized, or unsafe cache data is treated as a
|
|
||||||
cache miss.
|
|
||||||
|
|
||||||
The assembled graph is bounded by the effective project `Limits.max_nodes`. When adapter settings
|
|
||||||
do not override limits, DocForge selects at least 10,000 nodes and raises that bound to the
|
|
||||||
manifest's `estimated_nodes` when larger. Aggregate assembly ceilings are:
|
|
||||||
|
|
||||||
- nodes: `max_nodes`;
|
|
||||||
- relationships: `max_nodes * 32`;
|
|
||||||
- Logic nodes across all functions: `max_nodes * 32`; and
|
|
||||||
- Logic edges across all functions: `max_nodes * 64`.
|
|
||||||
|
|
||||||
Adapters should set realistic limits and fail before retaining unbounded frontend evidence. The
|
|
||||||
cache and assembly bounds do not replace each adapter's own bounded source, command, parser, or
|
|
||||||
per-function limits.
|
|
||||||
|
|
||||||
## Step 9: keep the complete path independent
|
## Step 9: keep the complete path independent
|
||||||
|
|
||||||
The full rebuild must not read the incremental extraction cache. Otherwise equivalence compares
|
The full rebuild must not read the incremental extraction cache. Otherwise equivalence compares
|
||||||
|
|
@ -541,8 +470,7 @@ For every such change:
|
||||||
- [ ] Two independent complete builds match exactly.
|
- [ ] Two independent complete builds match exactly.
|
||||||
- [ ] Incremental extraction uses authoritative dependencies.
|
- [ ] Incremental extraction uses authoritative dependencies.
|
||||||
- [ ] The complete oracle is independent of the cache.
|
- [ ] The complete oracle is independent of the cache.
|
||||||
- [ ] Complete and incremental public graph and Logic projections match exactly.
|
- [ ] Complete and incremental public projections match exactly.
|
||||||
- [ ] `verify_adapter_conformance()` passes, and the separate proof-matrix cases also pass.
|
|
||||||
- [ ] Corrupt, missing, and interrupted cache cases fail safely.
|
- [ ] Corrupt, missing, and interrupted cache cases fail safely.
|
||||||
- [ ] Session composition and family isolation are proven.
|
- [ ] Session composition and family isolation are proven.
|
||||||
- [ ] Viewer, query, context, and Logic retrieval are proven.
|
- [ ] Viewer, query, context, and Logic retrieval are proven.
|
||||||
|
|
|
||||||
|
|
@ -1,105 +0,0 @@
|
||||||
# Agent integration
|
|
||||||
|
|
||||||
DocForge generates deterministic, project-bound MCP client fragments for Codex, Claude, and
|
|
||||||
OpenClaw. Generic projects and custom adapters use different configuration APIs, but both produce
|
|
||||||
fixed standard-input/output bindings with empty generated environments and explicit capability
|
|
||||||
policy.
|
|
||||||
|
|
||||||
## Fixed reference binding
|
|
||||||
|
|
||||||
Configure one of the in-repository adapters with
|
|
||||||
`.docforge/reference-adapter.toml` as described in
|
|
||||||
[Reference Adapters](REFERENCE_ADAPTERS.md), then start:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -I -m docforge.reference_mcp \
|
|
||||||
--project-root /absolute/path/to/project \
|
|
||||||
--capability-mode read
|
|
||||||
```
|
|
||||||
|
|
||||||
`docforge.reference_mcp` is a fixed trusted module in the installed DocForge distribution. It
|
|
||||||
selects a provider only from the validated reference configuration and reports binding metadata
|
|
||||||
containing:
|
|
||||||
|
|
||||||
- `server_module = "docforge.reference_mcp"`;
|
|
||||||
- `adapter_mode = "reference"`;
|
|
||||||
- the selected `reference_language`; and
|
|
||||||
- the exact `reference_config_hash`.
|
|
||||||
|
|
||||||
The reference binding registers exactly the 21 read tools listed in
|
|
||||||
[MCP Boundary](MCP_CONTRACT.md#read-tools). It has no proposal or canonical-application surface.
|
|
||||||
|
|
||||||
## Generate a reference client fragment
|
|
||||||
|
|
||||||
The public launcher and generator APIs are:
|
|
||||||
|
|
||||||
```python
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from docforge.adapter_launcher import AdapterLauncherV1
|
|
||||||
from docforge.client_config import generate_adapter_client_configuration
|
|
||||||
from docforge.reference_mcp import REFERENCE_MCP_MODULE, create_reference_project
|
|
||||||
|
|
||||||
root = Path("/absolute/path/to/project")
|
|
||||||
project = create_reference_project(root)
|
|
||||||
launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE)
|
|
||||||
|
|
||||||
plan = generate_adapter_client_configuration(
|
|
||||||
project,
|
|
||||||
launcher,
|
|
||||||
"codex", # "codex", "claude", or "openclaw"
|
|
||||||
capability_mode="read",
|
|
||||||
)
|
|
||||||
print(plan["artifact"]["content"])
|
|
||||||
```
|
|
||||||
|
|
||||||
Pass `output=Path(...)` only when the caller has selected an exact destination. Publication is an
|
|
||||||
atomic create-or-exact-match operation; it does not merge or replace different existing content.
|
|
||||||
The result includes the launcher, source-availability, project, policy, artifact, and configuration
|
|
||||||
hashes needed to inspect the binding before use.
|
|
||||||
|
|
||||||
The fixed reference module accepts read mode only. Do not request proposal or application mode for
|
|
||||||
it.
|
|
||||||
|
|
||||||
## Custom project-owned adapter launchers
|
|
||||||
|
|
||||||
Generic `docforge configure` intentionally refuses a custom adapter project. Construct the
|
|
||||||
project-owned `ProjectService`, then use `AdapterLauncherV1` and
|
|
||||||
`generate_adapter_client_configuration()` as the custom-adapter route.
|
|
||||||
|
|
||||||
`AdapterLauncherV1` schema version 1 binds:
|
|
||||||
|
|
||||||
- one project ID, canonical absolute project root, adapter identity, and descriptor hash;
|
|
||||||
- `entry_point = "python-module"`; and
|
|
||||||
- one installed project-owned top-level Python module.
|
|
||||||
|
|
||||||
The only trusted dotted-module exception is the fixed `docforge.reference_mcp` binding. A
|
|
||||||
project-owned module is resolved through isolated Python without importing or executing it during
|
|
||||||
the probe, and it must resolve to one canonical regular `.py` file inside the project root.
|
|
||||||
|
|
||||||
The launcher contract has no arbitrary command, command arguments, shell string, working
|
|
||||||
directory, environment, callable selector, discovery rule, or module-reload mechanism. Generated
|
|
||||||
bindings invoke the current Python executable as `python -I -m <module>` with only DocForge's
|
|
||||||
validated project, capability, render-policy, authority, and no-AST options. Source identity and
|
|
||||||
the project binding are revalidated before publication; drift fails closed.
|
|
||||||
|
|
||||||
Proposal and application modes are available only to a custom module that implements those fixed
|
|
||||||
server arguments and only when the project descriptor declares the matching writer and canonical
|
|
||||||
applier authority. Generating a mode does not manufacture that authority.
|
|
||||||
|
|
||||||
## Session workflow
|
|
||||||
|
|
||||||
After registering the generated fragment in the selected client:
|
|
||||||
|
|
||||||
1. Start a new MCP process or client session.
|
|
||||||
2. Call `docforge_bootstrap`.
|
|
||||||
3. Verify project ID, root fingerprint, adapter identity, revision, source hash, binding metadata,
|
|
||||||
effective policy, and registered tools.
|
|
||||||
4. Retrieve exact or bounded project evidence through the fixed read tools.
|
|
||||||
5. Restart the process if `adapter_restart_required` reports an implementation or configuration
|
|
||||||
change.
|
|
||||||
|
|
||||||
Document text returned by DocForge is untrusted project content. It never overrides client, user,
|
|
||||||
or project authority. See [MCP Boundary](MCP_CONTRACT.md) for synchronization, pagination,
|
|
||||||
retrieval, proposal, and application rules, and
|
|
||||||
[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) before selecting `no_ast=True`.
|
|
||||||
|
|
@ -1,79 +0,0 @@
|
||||||
# DocForge command reference
|
|
||||||
|
|
||||||
> Generated from the live CLI parser and MCP registrations. Do not edit this file by hand.
|
|
||||||
|
|
||||||
Global CLI options are documented in `docforge --help` and are not repeated in each command row. The MCP table is the complete generic application-enabled surface; the fixed reference-adapter server exposes only its read rows.
|
|
||||||
|
|
||||||
## CLI commands
|
|
||||||
|
|
||||||
| Command | Invocation |
|
|
||||||
|---|---|
|
|
||||||
| `apply` | `docforge apply [-h] --changeset-hash CHANGESET_HASH --applier APPLIER changeset_id` |
|
|
||||||
| `backlinks` | `docforge backlinks [-h] [--relation RELATION] [--limit LIMIT] node_id` |
|
|
||||||
| `build` | `docforge build [-h]` |
|
|
||||||
| `check` | `docforge check [-h]` |
|
|
||||||
| `configure` | `docforge configure [-h] --project PROJECT [--name NAME] [--capability-mode {read,proposal,application}] [--proposal-writer PROPOSAL_WRITER] [--canonical-applier CANONICAL_APPLIER] [--no-ast] [--manual-render-policy {auto,explicit,disabled}] [--portable-graph-policy {explicit,disabled}] [--live-viewer-policy {on-demand,disabled}] [--startup-timeout STARTUP_TIMEOUT] [--tool-timeout TOOL_TIMEOUT] [--output OUTPUT] {codex,claude,openclaw}` |
|
|
||||||
| `context` | `docforge context [-h] [--budget BUDGET] [--limit LIMIT] [--cursor CURSOR] profile` |
|
|
||||||
| `dependencies` | `docforge dependencies [-h] [--depth DEPTH] [--limit LIMIT] node_id` |
|
|
||||||
| `doctor` | `docforge doctor [-h] --client {codex,claude,openclaw} [--project PROJECT] [--config CONFIG] [--server-name SERVER_NAME]` |
|
|
||||||
| `filter` | `docforge filter [-h] [--family FAMILY] [--authority AUTHORITY] [--status STATUS] [--tag TAG] [--limit LIMIT]` |
|
|
||||||
| `generation-diff` | `docforge generation-diff [-h] [--limit LIMIT] [--cursor CURSOR]` |
|
|
||||||
| `graph-plan` | `docforge graph-plan [-h] view_id` |
|
|
||||||
| `graph-render` | `docforge graph-render [-h] view_id` |
|
|
||||||
| `graph-render-status` | `docforge graph-render-status [-h] [view_id]` |
|
|
||||||
| `impact` | `docforge impact [-h] [--depth DEPTH] [--limit LIMIT] node_id` |
|
|
||||||
| `info` | `docforge info [-h]` |
|
|
||||||
| `onboard` | `docforge onboard [-h] [--language LANGUAGE] [--scaffold] [--project-id PROJECT_ID] [--title TITLE] [--content-root CONTENT_ROOT]` |
|
|
||||||
| `preview` | `docforge preview [-h] changeset_id view_id` |
|
|
||||||
| `reindex` | `docforge reindex [-h]` |
|
|
||||||
| `render` | `docforge render [-h] view_id` |
|
|
||||||
| `render-status` | `docforge render-status [-h] [--deep] [view_id]` |
|
|
||||||
| `search` | `docforge search [-h] [--limit LIMIT] query` |
|
|
||||||
| `show` | `docforge show [-h] node_id` |
|
|
||||||
| `sync` | `docforge sync [-h]` |
|
|
||||||
| `validate` | `docforge validate [-h]` |
|
|
||||||
| `validate-index` | `docforge validate-index [-h]` |
|
|
||||||
| `visualization-status` | `docforge visualization-status [-h]` |
|
|
||||||
| `visualization-stop` | `docforge visualization-stop [-h]` |
|
|
||||||
| `visualize` | `docforge visualize [-h] [--node NODE \| --query QUERY] [--depth DEPTH] [--no-open]` |
|
|
||||||
|
|
||||||
## MCP tools
|
|
||||||
|
|
||||||
| Surface | Tool | Required arguments | Optional arguments | Input schema SHA-256 | Description |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| read | `docforge_backlinks` | `node_id` | `limit`, `relation` | `bf9a5936e2d1ee70897dc0795ec9d716be63c9447e6fdcd3890d5dd92f45f690` | Return bounded incoming relationships for one exact stable node. |
|
|
||||||
| read | `docforge_bootstrap` | — | — | `af6c17a59a7713cfb4497f0f66b05efc091abbfe109573bb86d673e39b9b7413` | Synchronize and report the complete fixed project binding and workflow. |
|
|
||||||
| read | `docforge_dependencies` | `node_id` | `depth`, `limit` | `1283a8a030cf9369493a13e2aa87317d371470904a397910bc0b52997b3ec3b8` | Traverse declared depends_on relationships within the configured depth limit. |
|
|
||||||
| read | `docforge_filter_nodes` | — | `authority`, `family`, `limit`, `status`, `tag` | `2a51c4d8f4d76d89a2e7f90b3b06d14c7e16303b9c81e62af6747286b58cf68b` | Filter current nodes deterministically by validated metadata. |
|
|
||||||
| read | `docforge_get_context` | `profile` | `budget`, `cursor`, `limit` | `2f164828c2891a88d0ea7173790a72cc6f3da1279f14da2ca495e8989880b298` | Compile bounded cited context from one configured profile with explicit omissions. |
|
|
||||||
| read | `docforge_get_contract` | — | — | `19eb6785523ed9db645d17f61df2b8b584ac10ad308199e9a96e6e176a87169e` | Report canonical and derived boundaries plus allowed and excluded operations. |
|
|
||||||
| read | `docforge_get_generation_diff` | — | `cursor`, `limit` | `0a2fccfea4bf5d78481d40b49e7c812ded9fbe9c93cb79296ede78f22b259fcf` | Return the latest bounded primary-graph generation transition. |
|
|
||||||
| read | `docforge_get_logic` | `owner_node_id` | — | `56909ee4d72143b470528ea2b3544325e1e1920dbcc758f3caee4a4924f904a1` | Return the lazy control-flow projection owned by one function or method. |
|
|
||||||
| read | `docforge_get_node` | `node_id` | — | `52408b0c3a297e7b5cf96b293fd6d4625f05c4b090c02bfe806d25ce6c2d1c68` | Return one exact stable node from the current validated project index. |
|
|
||||||
| read | `docforge_get_task_context` | `task`, `task_kind` | `budget`, `cursor`, `focus_node_id`, `limit` | `0ffbc030bbe5cca2ae24920ec812ac976be9090da4fa2887d87526aba64a21f7` | Return one bounded task-shaped context capsule with explicit evidence gaps. |
|
|
||||||
| read | `docforge_graph_plan` | `view_id` | — | `7a014ba692f57badca9ca2098b88f82f4749ada6d71f5a3b8bc141abc86645f9` | Plan one declared portable graph without publishing derived output. |
|
|
||||||
| read | `docforge_graph_render_status` | — | `view_id` | `1d2f3604b0151ba2e228cc6ce1829d7f4244193b53d85df46eb6a3e608c59564` | Report portable-graph publication state without rendering. |
|
|
||||||
| read | `docforge_impact` | `node_id` | `depth`, `limit` | `1c71db5bb4e1afe180d98dd5000a757b7e3fd1a048b5f68af13d9b7ac56e2ce2` | Traverse bounded incoming relationships and report exact paths. |
|
|
||||||
| read | `docforge_project_info` | — | — | `a42973543d5e6f4147953d175f6aec43d228cd6bd1a63f3ca4e84a51ad7e8a64` | Report the fixed project identity, revision, source hash, and index health. |
|
|
||||||
| read | `docforge_render_status` | — | `deep`, `view_id` | `5ea5683bd74e685d4cabb1c960a66eff23a001bd2c68e1af350c6e46e936b9fd` | Report receipt state, or explicitly recompute the side-effect-free render oracle. |
|
|
||||||
| read | `docforge_search` | `query` | `limit` | `735fff8bd82b20730b0907db8bf61eb0de9fff240ef4fa239ce555779ae49047` | Run bounded lexical search over the current validated project index. |
|
|
||||||
| read | `docforge_stop_visualization` | — | — | `066a6e34019f4abcd3523f24d9e447775e1479adbea1134341f434a6bb5f8d35` | Explicitly stop this project's persistent read-only graph browser. |
|
|
||||||
| read | `docforge_sync` | — | — | `a2ac50c03b9b0e08b271216f76920279563ed264bef06d6791695c68dcb3b2a2` | Ensure the disposable project index matches current canonical sources. |
|
|
||||||
| read | `docforge_validate_project` | — | — | `3192ba97a6d7fff13008c5ed3bfa549d1eeea95332579592d96b8fd326ad06f7` | Validate current canonical sources and graph without writing any project file. |
|
|
||||||
| read | `docforge_visualization_status` | — | — | `03751f891e0a9318355a6b28be551eec6009c0f96f05ff99536e2f401b76f87f` | Report this project's managed graph browser lifecycle state. |
|
|
||||||
| read | `docforge_visualize` | — | `depth`, `node_id`, `query` | `cdc9669ba934e591379ea684934d99e2e80ba3f181d6c58867774a8064c32f4b` | Start the fixed read-only graph browser for this configured project. |
|
|
||||||
| proposal | `docforge_abandon_changeset` | `changeset_id`, `expected_changeset_hash`, `reason` | — | `8813e40ffc12b1ae3d9abe73db8de214085a4a8042323146283fffa8f7476a62` | Mark one proposal abandoned while preserving its audit record. |
|
|
||||||
| proposal | `docforge_create_changeset` | `changeset_id` | — | `0de93025b9caaa9e33b4b2b3bcfabdde087c944daba48dd77ae3c8f757c52ab8` | Create an empty hash-bound proposal under the configured isolated changeset root. |
|
|
||||||
| proposal | `docforge_get_changeset` | `changeset_id` | `cursor`, `limit` | `919737a8e732cf95305a2964cddf98f1fdce9ce2cfb52ff659436c244a38495d` | Inspect a stored proposal even when its canonical base has become stale. |
|
|
||||||
| proposal | `docforge_get_changeset_diff` | `changeset_id` | `cursor`, `limit` | `5cc1d2b035f37e3866766cfa0c3541b144762b764bf455a641ab459df03f8a59` | Return a deterministic structured and textual diff without applying the proposal. |
|
|
||||||
| proposal | `docforge_list_changesets` | — | `cursor`, `include_history`, `limit`, `status` | `b96519477781cb20acbfc9502a4b7d64a9119e0bd8d71e27290e90677aaf9ced` | List active proposals by default, with optional lifecycle history. |
|
|
||||||
| proposal | `docforge_preview_changeset` | `changeset_id`, `view_id` | — | `d84e1710a5cac52607037a7e581fcb221c42d3776f510eefe42fb6779fee7fb3` | Render one validated changeset through a declared view into its isolated preview path. |
|
|
||||||
| proposal | `docforge_propose_node_create` | `changeset_id`, `content`, `expected_changeset_hash`, `metadata`, `node_id`, `rationale`, `relationship_changes`, `target_source` | — | `8953b502b9d8d9c097935bae51fce4e830286eace9df3cb6c37b7e0542be5a4e` | Append one validated node creation without writing its canonical target. |
|
|
||||||
| proposal | `docforge_propose_node_delete` | `changeset_id`, `expected_changeset_hash`, `expected_content_hash`, `node_id`, `rationale`, `relationship_changes` | — | `8fb6a22e5c1ead3793384652e4b61e75cc23c2725906542533a121401f7faaee` | Append one validated deletion with explicit incident relationship removals. |
|
|
||||||
| proposal | `docforge_propose_node_move` | `changeset_id`, `expected_changeset_hash`, `expected_content_hash`, `node_id`, `rationale`, `target_source` | — | `32ef6fc4b9795a6a8e30f4f81f63d84d7350e983773f796524382d71cb870a67` | Append one validated same-format node move without moving a canonical file. |
|
|
||||||
| proposal | `docforge_propose_node_update` | `changeset_id`, `content`, `expected_changeset_hash`, `expected_content_hash`, `metadata`, `node_id`, `rationale`, `relationship_changes` | — | `6a29a73b9e50c9fe22cb6d056bbcdacef30c93c0822c8a9276ef33f9e51e7728` | Append one validated node update without changing canonical content. |
|
|
||||||
| proposal | `docforge_propose_relationship_update` | `changeset_id`, `expected_changeset_hash`, `expected_content_hash`, `node_id`, `rationale`, `relationship_changes` | — | `8e21fc7ba35289940dd7f12294b14e8553163ab901002dcd4a3998cdb33556c3` | Queue hash-bound relationship changes without rewriting node content. |
|
|
||||||
| proposal | `docforge_rebase_changeset` | `changeset_id`, `expected_changeset_hash` | — | `505434b2f9f611649221b95d0251db347c44752d1c73e5d69e1abfd916966788` | Safely rebase a proposal when every touched fact remains unchanged. |
|
|
||||||
| proposal | `docforge_register_changes` | `changeset_id`, `operations` | — | `c5a7d6ac07d3d3b95c13b0c57ad9b69360f2f063b1c8e22dd5f64f6d3a83dc5b` | Atomically register and validate a complete hash-bound proposal. |
|
|
||||||
| proposal | `docforge_validate_changeset` | `changeset_id` | `cursor`, `limit` | `cb9df5104dbefc3382900ec812ea581a43612d1577619071f1f03674041652c7` | Validate a proposal against its exact canonical base and other active proposals. |
|
|
||||||
| application | `docforge_apply_changeset` | `changeset_id`, `expected_changeset_hash` | — | `80259f5001f36b14f960c10bd94751d1431077b72e081fb5a5c806b79073b085` | Apply one exact validated changeset and refresh declared derived state. |
|
|
||||||
|
|
@ -31,31 +31,16 @@ model, index, rendering, application, and MCP factory names imported from these
|
||||||
remain supported:
|
remain supported:
|
||||||
|
|
||||||
- `docforge.adapter_contract`
|
- `docforge.adapter_contract`
|
||||||
- `docforge.adapter_sdk`
|
|
||||||
- `docforge.adapter_launcher`
|
|
||||||
- `docforge.application`
|
- `docforge.application`
|
||||||
- `docforge.client_config`
|
|
||||||
- `docforge.index`
|
- `docforge.index`
|
||||||
- `docforge.mcp_server`
|
- `docforge.mcp_server`
|
||||||
- `docforge.models`
|
- `docforge.models`
|
||||||
- `docforge.policy`
|
- `docforge.policy`
|
||||||
- `docforge.render_contract`
|
- `docforge.render_contract`
|
||||||
- `docforge.reference_config`
|
|
||||||
- `docforge.reference_mcp`
|
|
||||||
|
|
||||||
Names beginning with an underscore are implementation details. New public names may be added
|
Names beginning with an underscore are implementation details. New public names may be added
|
||||||
without breaking this contract.
|
without breaking this contract.
|
||||||
|
|
||||||
The repository Python, JavaScript/TypeScript, and C++ adapters are supported reference
|
|
||||||
implementations. Their documented configuration, evidence limits, and unsupported-fact reports are
|
|
||||||
compatibility surfaces; their internal parser helpers are not adapter-authoring imports.
|
|
||||||
|
|
||||||
Milestone 4 adds three version-1 schemas without changing existing descriptor or result schemas:
|
|
||||||
|
|
||||||
- `schemas/adapter-launcher.schema.json`
|
|
||||||
- `schemas/adapter-client-configuration.schema.json`
|
|
||||||
- `schemas/reference-adapter.schema.json`
|
|
||||||
|
|
||||||
## CLI and MCP surfaces
|
## CLI and MCP surfaces
|
||||||
|
|
||||||
Existing `docforge` command names and arguments remain supported. Existing `docforge-mcp` tool
|
Existing `docforge` command names and arguments remain supported. Existing `docforge-mcp` tool
|
||||||
|
|
@ -248,11 +233,6 @@ Milestone 3 adds `ManualRenderPlanV1`, `GraphViewPlanV1`, projection package and
|
||||||
and projection policy version 2. These are additive submodule and schema contracts. They do not
|
and projection policy version 2. These are additive submodule and schema contracts. They do not
|
||||||
change the legacy task-context, adapter, changeset, or effective-policy contracts described above.
|
change the legacy task-context, adapter, changeset, or effective-policy contracts described above.
|
||||||
|
|
||||||
Milestone 4 adds `docforge.adapter_sdk`, the fixed reference configuration and read-only MCP
|
|
||||||
binding, immutable adapter launchers, and generated adapter client fragments. A legacy adapter with
|
|
||||||
only `load_projection()` remains first-class and need not adopt incremental extraction, Logic, a
|
|
||||||
reference configuration, or launcher metadata.
|
|
||||||
|
|
||||||
## Safety boundary
|
## Safety boundary
|
||||||
|
|
||||||
DocForge remains bound to one explicit project root. It rejects absolute paths, root escapes, and
|
DocForge remains bound to one explicit project root. It rejects absolute paths, root escapes, and
|
||||||
|
|
@ -265,9 +245,8 @@ publication, or project switching.
|
||||||
Milestone 0 records rather than redesigns these areas:
|
Milestone 0 records rather than redesigns these areas:
|
||||||
|
|
||||||
- Generic warm reads still repeat whole-project discovery, parsing, and validation.
|
- Generic warm reads still repeat whole-project discovery, parsing, and validation.
|
||||||
- The base wheel intentionally omits Tree-sitter. JavaScript, TypeScript, and C++ syntax evidence
|
- Tree-sitter and the JavaScript and C++ grammars remain mandatory installation dependencies even
|
||||||
requires the matching `docforge[javascript]`, `docforge[typescript]`, or `docforge[cpp]` extra.
|
when their runtime modules are unused.
|
||||||
Python reference evidence uses the standard library and remains available in the base wheel.
|
|
||||||
- Several version strings and defaults remain duplicated.
|
- Several version strings and defaults remain duplicated.
|
||||||
- One individually oversized context entry is represented as explicit bounded omission evidence;
|
- One individually oversized context entry is represented as explicit bounded omission evidence;
|
||||||
callers use targeted retrieval for that node.
|
callers use targeted retrieval for that node.
|
||||||
|
|
|
||||||
|
|
@ -29,9 +29,6 @@ commit when Git is available; it cannot change repository state.
|
||||||
- Projection package: `schemas/projection-package.schema.json`, version 1.
|
- Projection package: `schemas/projection-package.schema.json`, version 1.
|
||||||
- Projection receipt: `schemas/projection-receipt.schema.json`, version 1.
|
- Projection receipt: `schemas/projection-receipt.schema.json`, version 1.
|
||||||
- Independent projection policy: `schemas/projection-policy.schema.json`, version 2.
|
- Independent projection policy: `schemas/projection-policy.schema.json`, version 2.
|
||||||
- Adapter launcher: `schemas/adapter-launcher.schema.json`, version 1.
|
|
||||||
- Adapter client configuration: `schemas/adapter-client-configuration.schema.json`, version 1.
|
|
||||||
- Reference adapter configuration: `schemas/reference-adapter.schema.json`, version 1.
|
|
||||||
- Index schema: version 3, disposable and reproducible.
|
- Index schema: version 3, disposable and reproducible.
|
||||||
- Index attestation: schema version 1, disposable and reproducible.
|
- Index attestation: schema version 1, disposable and reproducible.
|
||||||
- Core, CLI, and MCP server: version 1.3.0.dev0.
|
- Core, CLI, and MCP server: version 1.3.0.dev0.
|
||||||
|
|
@ -106,42 +103,6 @@ complete projection, opens SQLite, starts MCP, executes the configured command,
|
||||||
builds, renders, starts a viewer, or writes configuration. Unprovable client behavior is a warning,
|
builds, renders, starts a viewer, or writes configuration. Unprovable client behavior is a warning,
|
||||||
not an invented success.
|
not an invented success.
|
||||||
|
|
||||||
## Public adapter SDK and reference binding
|
|
||||||
|
|
||||||
`docforge.adapter_sdk` is the stable adapter-authoring import boundary. It exposes the typed
|
|
||||||
projection, manifest, source contribution, complete assembly, project wrapper, graph model, and
|
|
||||||
conformance contracts needed by an adapter without requiring authors to import core implementation
|
|
||||||
modules.
|
|
||||||
|
|
||||||
Complete evidence includes the primary graph and function Logic. An incremental adapter that
|
|
||||||
publishes Logic must implement `load_complete_assembly()` as an independent complete oracle.
|
|
||||||
`verify_adapter_conformance()` proves repeated complete determinism, equality between the complete
|
|
||||||
assembly and `load_projection()`, and exact complete/incremental graph-plus-Logic parity. Separate
|
|
||||||
tests remain responsible for confinement, restart behavior, no-AST behavior, cache recovery, and
|
|
||||||
retrieval.
|
|
||||||
|
|
||||||
Adapter assemblies are bounded before publication. Primary nodes use the descriptor `max_nodes`
|
|
||||||
limit. Primary edges, Logic nodes, and Logic edges use fixed deterministic multipliers over that
|
|
||||||
limit. Version-1 extraction caches are regular-file-only, bounded to 10,000 sources and
|
|
||||||
64,000,000 bytes, and are treated as misses when corrupt, oversized, foreign, or incompatible.
|
|
||||||
|
|
||||||
`.docforge/reference-adapter.toml` is a closed version-1 selection among `python`, `javascript`,
|
|
||||||
`typescript`, and `cpp`. It declares one project identity and explicit non-overlapping source
|
|
||||||
roots. C++ additionally requires a confined `compile_commands.json`. It cannot declare a command,
|
|
||||||
module, environment, writer, applier, or remote endpoint.
|
|
||||||
|
|
||||||
`python -m docforge.reference_mcp --project-root ROOT` constructs only the selected fixed
|
|
||||||
repository reference adapter and exposes the read surface. It never registers proposal or
|
|
||||||
application tools.
|
|
||||||
|
|
||||||
`AdapterLauncherV1` is an immutable project-bound Python-module declaration. It accepts no
|
|
||||||
arbitrary command, arguments, working directory, environment, discovery, callable selector, or
|
|
||||||
module reload. Custom launchers resolve one installed top-level module through isolated Python and
|
|
||||||
require its origin inside the project root. The fixed `docforge.reference_mcp` module is the only
|
|
||||||
trusted dotted exception. `generate_adapter_client_configuration()` binds generated Codex,
|
|
||||||
Claude, and OpenClaw fragments to that launcher, current source availability, effective policy,
|
|
||||||
descriptor, interpreter, and exact artifact bytes.
|
|
||||||
|
|
||||||
## Isolated proposal model
|
## Isolated proposal model
|
||||||
|
|
||||||
Create, update, move, and delete are ordered node operations inside an isolated changeset. Every
|
Create, update, move, and delete are ordered node operations inside an isolated changeset. Every
|
||||||
|
|
|
||||||
|
|
@ -1,165 +0,0 @@
|
||||||
# Core concepts and authority
|
|
||||||
|
|
||||||
DocForge is a project-bound knowledge compiler. It turns explicit canonical project facts into
|
|
||||||
validated graphs and bounded derived views without transferring authority to the index, an agent,
|
|
||||||
or a renderer.
|
|
||||||
|
|
||||||
## One project, one explicit root
|
|
||||||
|
|
||||||
Every operation is bound to one canonical real project directory. Paths in descriptors and adapter
|
|
||||||
configuration are project-relative and confined beneath that root. A CLI or MCP process does not
|
|
||||||
discover or switch projects after startup.
|
|
||||||
|
|
||||||
The project root determines:
|
|
||||||
|
|
||||||
- which descriptor and canonical sources may be read;
|
|
||||||
- where derived cache and changeset roots may exist;
|
|
||||||
- which project identity, revision, source hash, and generation appear in results;
|
|
||||||
- which writer, applier, rendering, and viewer policies can be selected.
|
|
||||||
|
|
||||||
Generated client fragments preserve that binding. They are machine-local configuration
|
|
||||||
projections, not portable project authority.
|
|
||||||
|
|
||||||
## Canonical facts and derived evidence
|
|
||||||
|
|
||||||
Canonical inputs own the facts:
|
|
||||||
|
|
||||||
- generic Markdown or TOML node sources;
|
|
||||||
- authority files named by a generic descriptor;
|
|
||||||
- an adapter's declared canonical sources and implementation boundary;
|
|
||||||
- `.docforge/project.toml` for a generic project;
|
|
||||||
- `.docforge/reference-adapter.toml` for a fixed reference integration.
|
|
||||||
|
|
||||||
Everything DocForge builds from those inputs is derived:
|
|
||||||
|
|
||||||
- SQLite indexes, attestations, generation receipts, and generation diffs;
|
|
||||||
- incremental extraction caches;
|
|
||||||
- task-context capsules and query responses;
|
|
||||||
- changeset previews;
|
|
||||||
- render plans, immutable packages, fragments, artifacts, and receipts;
|
|
||||||
- portable graph publications and live-viewer processes;
|
|
||||||
- generated Codex, Claude, and OpenClaw fragments.
|
|
||||||
|
|
||||||
Derived state may be discarded and rebuilt. A derived artifact can prove what it was bound to, but
|
|
||||||
it cannot override current canonical content.
|
|
||||||
|
|
||||||
## Nodes, relationships, and Logic
|
|
||||||
|
|
||||||
The primary graph contains nodes and directed relationships.
|
|
||||||
|
|
||||||
A node has a stable project-wide ID, title, family, authority, status, tags, summary, content,
|
|
||||||
source identity, and content hash. A generic Markdown file contains one node beneath a TOML
|
|
||||||
metadata block; a generic TOML source may contain multiple nodes. Adapter nodes use the same public
|
|
||||||
graph contract.
|
|
||||||
|
|
||||||
A relationship is an exact `(source, relation, target)` triple. The project descriptor defines the
|
|
||||||
allowed relation names. DocForge gives `depends_on` special acyclic validation, but it does not
|
|
||||||
invent domain meaning for a project's other relation names. Retrieval recognizes only a versioned
|
|
||||||
alias set for task planning and reports unknown allowed relations as `unclassified`.
|
|
||||||
|
|
||||||
Logic is deliberately separate. It is a lazy function-scoped control-flow projection owned by one
|
|
||||||
primary node. Decisions, actions, loops, convergence points, returns, and exceptions connect
|
|
||||||
through explicit branch edges. Logic does not add statement-level nodes to ordinary Nodes, Flow,
|
|
||||||
Web, search, or generation-diff results. Python, JavaScript, TypeScript, and C++ integrations may
|
|
||||||
publish Logic when their adapter contract supports it.
|
|
||||||
|
|
||||||
## Authority, status, family, and tags
|
|
||||||
|
|
||||||
These fields answer different questions:
|
|
||||||
|
|
||||||
- `authority` describes the role of the content. The generic vocabulary is `authoritative`,
|
|
||||||
`approved_plan`, `derived`, `proposal`, and `historical`.
|
|
||||||
- `status` is project-defined lifecycle state such as `current`, `active`, or `verified`.
|
|
||||||
- `family` is a project-defined content grouping used for filtering, profiles, rendering, and
|
|
||||||
writer permissions.
|
|
||||||
- `tags` are exact project labels for retrieval and presentation.
|
|
||||||
|
|
||||||
An `authoritative` node can still become stale; authority is not a freshness claim. A `derived`
|
|
||||||
node is still canonical if it is stored in a declared canonical source; the label describes its
|
|
||||||
role, not whether DocForge may silently regenerate it. Status and authority never grant an MCP
|
|
||||||
writer permission.
|
|
||||||
|
|
||||||
## Validation, synchronization, and generations
|
|
||||||
|
|
||||||
Validation loads the complete canonical graph and rejects unsafe paths, invalid source formats,
|
|
||||||
duplicate IDs, unresolved relationships, prohibited cycles, violated project limits, and
|
|
||||||
adapter-specific contract failures.
|
|
||||||
|
|
||||||
The disposable index is published atomically only after the complete graph and SQLite integrity
|
|
||||||
checks pass. Its attestation binds the whole index file. A successful replacement is the derived
|
|
||||||
publication commit point; later receipt-writing trouble is reported as degraded evidence rather
|
|
||||||
than as a false claim that the replacement failed.
|
|
||||||
|
|
||||||
MCP operations automatically synchronize derived state under a project lock before normal work.
|
|
||||||
A graph generation identifies one validated indexed snapshot. Generation-pinned retrieval and
|
|
||||||
rendering do not reopen mutable sources behind an older snapshot.
|
|
||||||
|
|
||||||
The latest generation diff is one bounded primary-graph transition, not a history database. It
|
|
||||||
contains no Logic details or source text.
|
|
||||||
|
|
||||||
## Complete and incremental adapters
|
|
||||||
|
|
||||||
`load_projection()` is the compatibility baseline and clean graph oracle. An incremental adapter
|
|
||||||
adds:
|
|
||||||
|
|
||||||
- `load_manifest()` for cheap project identity, source inventory, fingerprints, and dependencies;
|
|
||||||
- `extract_source()` for one cacheable source contribution;
|
|
||||||
- optionally `assemble_projection()` to normalize overlapping contributions.
|
|
||||||
|
|
||||||
When incremental contributions contain Logic, `load_complete_assembly()` supplies a
|
|
||||||
cache-independent complete graph-plus-Logic oracle. Warm cache behavior is an optimization, never a
|
|
||||||
different authority path. Corrupt or incompatible extraction caches are treated as misses, and a
|
|
||||||
clean complete build remains the equivalence and recovery boundary.
|
|
||||||
|
|
||||||
The public types, validators, and conformance helper are exported from `docforge.adapter_sdk`. See
|
|
||||||
the [Adapter authoring guide](ADAPTER_AUTHORING_GUIDE.md) and [Incremental
|
|
||||||
indexing](INCREMENTAL_INDEXING.md).
|
|
||||||
|
|
||||||
## Proposals are not canonical changes
|
|
||||||
|
|
||||||
A changeset is an isolated, ordered proposal over an exact canonical base. Each operation names
|
|
||||||
preconditions, and the final changeset has a content-derived hash. Validation projects the complete
|
|
||||||
resulting graph before application.
|
|
||||||
|
|
||||||
Canonical application requires:
|
|
||||||
|
|
||||||
1. a writer declared in the descriptor;
|
|
||||||
2. a process started with the matching proposal and application authority;
|
|
||||||
3. one explicit final changeset hash;
|
|
||||||
4. unchanged source, relationship, permission, and graph preconditions.
|
|
||||||
|
|
||||||
Application does not perform Git mutation, build, deployment, or publication. Until exact-hash
|
|
||||||
application succeeds, canonical project files remain unchanged.
|
|
||||||
|
|
||||||
## Three independent output projections
|
|
||||||
|
|
||||||
Manual rendering, portable graph publication, and the live viewer are separate:
|
|
||||||
|
|
||||||
- a manual is a declared derived HTML view over selected nodes;
|
|
||||||
- a portable graph is a content-addressed static Nodes, Flow, or Web artifact;
|
|
||||||
- the live viewer is a managed loopback process pinned to one validated index generation and can
|
|
||||||
request lazy Logic.
|
|
||||||
|
|
||||||
Their policies compose independently. Disabling one does not transfer its authority to another.
|
|
||||||
None is canonical documentation or a retrieval authority. See [Rendering and
|
|
||||||
visualization](RENDERING_AND_VISUALIZATION.md).
|
|
||||||
|
|
||||||
## Binding policy is not project truth
|
|
||||||
|
|
||||||
Capability mode, no-AST preservation, diagnostics, and projection modes describe one running
|
|
||||||
process or generated client binding. They do not rewrite the descriptor or canonical graph.
|
|
||||||
|
|
||||||
`--no-ast` forbids AST-family adapter evolution and Logic publication/retrieval for that binding.
|
|
||||||
It does not inspect parser implementation, sandbox the filesystem, or convert an existing
|
|
||||||
AST/Tree-sitter adapter into a no-AST adapter. Read [Policy precedence](POLICY_PRECEDENCE.md) and
|
|
||||||
[Legacy and no-AST operation](LEGACY_AND_NO_AST.md).
|
|
||||||
|
|
||||||
## Trust the narrowest evidence
|
|
||||||
|
|
||||||
DocForge reports stable IDs, hashes, generations, omissions, truncation, and provenance limits so a
|
|
||||||
consumer can distinguish proof from inference. A source inventory is not a semantic graph; a
|
|
||||||
syntax-level relationship is not compiler resolution; a configuration doctor is not a connection
|
|
||||||
test; a viewer snapshot is not continuous monitoring; and a benchmark is not a release.
|
|
||||||
|
|
||||||
Continue with the [Project descriptor](PROJECT_DESCRIPTOR.md), [New-project
|
|
||||||
quickstart](NEW_PROJECT_QUICKSTART.md), or [Core contract](CONTRACT.md).
|
|
||||||
|
|
@ -4,10 +4,6 @@ DocForge Release 1 adapters return one complete immutable projection. That contr
|
||||||
supported. The incremental compiler adds an opt-in source-scoped contract that avoids reparsing
|
supported. The incremental compiler adds an opt-in source-scoped contract that avoids reparsing
|
||||||
unchanged files while preserving the same validated, atomically published graph.
|
unchanged files while preserving the same validated, atomically published graph.
|
||||||
|
|
||||||
Import these contracts from the public `docforge.adapter_sdk` facade. See
|
|
||||||
[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for the preserved one-method contract and
|
|
||||||
[Reference Adapters](REFERENCE_ADAPTERS.md) for the four maintained implementations.
|
|
||||||
|
|
||||||
## Release 1 compatibility
|
## Release 1 compatibility
|
||||||
|
|
||||||
The incremental interface is additive:
|
The incremental interface is additive:
|
||||||
|
|
@ -38,26 +34,18 @@ Incremental indexing is an extraction optimization. It does not weaken publicati
|
||||||
An interrupted extraction never replaces the last validated SQLite index. A malformed,
|
An interrupted extraction never replaces the last validated SQLite index. A malformed,
|
||||||
incompatible, or missing cache is a cache miss, not a partial graph.
|
incompatible, or missing cache is a cache miss, not a partial graph.
|
||||||
|
|
||||||
One cache generation is bounded to 10,000 source contributions and 64,000,000 encoded bytes.
|
|
||||||
Aggregate graph and Logic assembly limits are described in the
|
|
||||||
[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md#current-aggregate-bounds).
|
|
||||||
|
|
||||||
## Adapter contract
|
## Adapter contract
|
||||||
|
|
||||||
An incremental loader implements the first, second, and fourth methods. It implements
|
An incremental loader implements all three methods:
|
||||||
`load_complete_assembly()` as well when it publishes Logic:
|
|
||||||
|
|
||||||
```python
|
```python
|
||||||
class MyAdapter:
|
class MyAdapter:
|
||||||
def load_manifest(self) -> AdapterManifest: ...
|
def load_manifest(self) -> AdapterManifest: ...
|
||||||
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ...
|
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ...
|
||||||
def load_complete_assembly(self) -> AdapterAssembly: ...
|
|
||||||
def load_projection(self) -> AdapterProjection: ...
|
def load_projection(self) -> AdapterProjection: ...
|
||||||
```
|
```
|
||||||
|
|
||||||
`load_projection()` remains the deterministic full-rebuild primary-graph oracle. An incremental
|
`load_projection()` remains the deterministic full-rebuild fallback and equivalence oracle.
|
||||||
adapter that publishes Logic must additionally implement `load_complete_assembly()` as the
|
|
||||||
cache-independent complete graph-plus-Logic oracle.
|
|
||||||
|
|
||||||
Each `AdapterSource` declares:
|
Each `AdapterSource` declares:
|
||||||
|
|
||||||
|
|
@ -149,8 +137,7 @@ fresh process must import and validate the current adapter.
|
||||||
```
|
```
|
||||||
|
|
||||||
Adapters can call `AdapterProject.verify_incremental_equivalence()` in release and contract tests.
|
Adapters can call `AdapterProject.verify_incremental_equivalence()` in release and contract tests.
|
||||||
The check compares project identity, revision, source hash, nodes, relationships, and Logic against
|
The check compares project identity, revision, source hash, nodes, and relationships against
|
||||||
the independent complete assembly. It also requires the complete assembly's primary graph to match
|
|
||||||
`load_projection()`.
|
`load_projection()`.
|
||||||
|
|
||||||
## Manual changes and relationships
|
## Manual changes and relationships
|
||||||
|
|
@ -183,29 +170,12 @@ logic empty until they implement a language analyzer.
|
||||||
This boundary prevents thousands of boolean expressions and basic blocks from polluting Nodes,
|
This boundary prevents thousands of boolean expressions and basic blocks from polluting Nodes,
|
||||||
Flow, Web, ordinary search, or architectural traversal. The Logic tab and `docforge_get_logic`
|
Flow, Web, ordinary search, or architectural traversal. The Logic tab and `docforge_get_logic`
|
||||||
request one function-scoped projection on demand. The built-in analyzers cover Python,
|
request one function-scoped projection on demand. The built-in analyzers cover Python,
|
||||||
JavaScript, TypeScript, and C++. Python uses the standard-library AST. JavaScript, TypeScript, and
|
JavaScript, and C++. Python uses the standard-library AST. JavaScript and C++ share pinned
|
||||||
C++ use distinct optional Tree-sitter grammars with thin language-aware control-flow profiles.
|
Tree-sitter infrastructure with thin language-aware control-flow profiles. Parsers run only while
|
||||||
Ordinary graph reads use stored projections and do not load or execute these parsers. A grammar
|
extracting a changed source contribution; ordinary graph reads do not load or execute them. A
|
||||||
alone supplies syntax, not control-flow meaning, so each new language still needs a profile for
|
grammar alone supplies syntax, not control-flow meaning, so each new language still needs a small
|
||||||
its branch, loop, case, exception, and termination constructs. All analyzers report possible
|
semantic profile for its branch, loop, case, exception, and termination constructs. All analyzers
|
||||||
static paths; they do not claim runtime branch outcomes.
|
report possible static paths; they do not claim runtime branch outcomes.
|
||||||
|
|
||||||
## Manifest and warm-parser scope
|
|
||||||
|
|
||||||
Parser work is language-specific and must be measured at the correct boundary:
|
|
||||||
|
|
||||||
- Python manifest construction fingerprints source and tokenizes local imports without calling
|
|
||||||
`ast.parse`.
|
|
||||||
- JavaScript and TypeScript manifest construction lexes static relative module specifiers without
|
|
||||||
invoking their distinct Tree-sitter extraction parsers. Focused tests prove this behavior on an
|
|
||||||
unchanged warm build.
|
|
||||||
- The C++ reference manifest parses inventoried sources with `tree-sitter-cpp` to discover quoted
|
|
||||||
include dependencies. A warm C++ extraction-cache hit is not a zero-parser claim.
|
|
||||||
|
|
||||||
The maintained Python benchmark instruments the unchanged warm path and requires zero
|
|
||||||
`ast.parse` calls and zero `extract_source` calls. That exact zero-parser benchmark claim is Python
|
|
||||||
only. JavaScript and TypeScript retain focused parser-free-manifest tests; C++ deliberately does
|
|
||||||
not.
|
|
||||||
|
|
||||||
## Full rebuilds
|
## Full rebuilds
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,80 +0,0 @@
|
||||||
# Legacy adapters and no-AST policy
|
|
||||||
|
|
||||||
DocForge preserves the original one-method adapter contract while offering incremental extraction,
|
|
||||||
complete graph-plus-Logic assemblies, and an independently selectable no-AST MCP policy. These are
|
|
||||||
separate compatibility boundaries.
|
|
||||||
|
|
||||||
## One-method adapters remain valid
|
|
||||||
|
|
||||||
An existing adapter that implements only:
|
|
||||||
|
|
||||||
```python
|
|
||||||
def load_projection(self) -> AdapterProjection: ...
|
|
||||||
```
|
|
||||||
|
|
||||||
continues to work. It receives the same projection validation, indexing, querying, visualization,
|
|
||||||
and MCP behavior as before. It does not receive extraction-cache speedups and publishes no Logic
|
|
||||||
through the complete assembly contract.
|
|
||||||
|
|
||||||
Incremental adoption is additive. Implement `load_manifest()` and `extract_source()` while keeping
|
|
||||||
`load_projection()` as the independent complete graph oracle. If incremental contributions publish
|
|
||||||
Logic, also implement `load_complete_assembly()` so complete and incremental graph-plus-Logic
|
|
||||||
output can be compared exactly. The public authoring surface is `docforge.adapter_sdk`; see the
|
|
||||||
[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md).
|
|
||||||
|
|
||||||
## What no-AST means
|
|
||||||
|
|
||||||
`--no-ast` is an immutable policy on one MCP process binding. It preserves the configured adapter
|
|
||||||
strategy but forbids AST, Tree-sitter, compiler-AST, and function-Logic evolution for that binding.
|
|
||||||
It also rejects nonempty Logic before index publication and rejects a pre-existing index that
|
|
||||||
contains Logic before reads or live visualization.
|
|
||||||
|
|
||||||
Start the generic server with:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docforge-mcp --project-root /absolute/path/to/project --no-ast
|
|
||||||
```
|
|
||||||
|
|
||||||
Project-owned integrations select the same policy with
|
|
||||||
`create_project_server(..., no_ast=True)` or `create_read_only_server(..., no_ast=True)`.
|
|
||||||
`docforge_bootstrap` and `docforge_get_contract` report the effective policy, and
|
|
||||||
`docforge_get_logic` returns `adapter_policy_forbids_logic`.
|
|
||||||
|
|
||||||
Changing this policy requires a new process. It does not rewrite or reload an adapter in place.
|
|
||||||
|
|
||||||
## What no-AST does not mean
|
|
||||||
|
|
||||||
No-AST is not:
|
|
||||||
|
|
||||||
- an inspection mechanism that proves which parser an arbitrary adapter uses;
|
|
||||||
- a filesystem, Python-import, or process sandbox;
|
|
||||||
- a promise that the adapter performs no non-AST fingerprinting, dependency discovery, caching, or
|
|
||||||
complete-projection loading; or
|
|
||||||
- permission to change project files, adapter code, or the configured capability surface.
|
|
||||||
|
|
||||||
Repository permissions and project instructions remain responsible for processes that have direct
|
|
||||||
filesystem access. The MCP boundary still excludes shell execution, arbitrary file operations,
|
|
||||||
Git mutation, builds, deployment, publication, project switching, and cross-project retrieval.
|
|
||||||
|
|
||||||
## Reference adapters are not no-AST adapters
|
|
||||||
|
|
||||||
The built-in Python reference extracts syntax and Logic with the standard-library AST. The
|
|
||||||
JavaScript, TypeScript, and C++ references extract syntax and Logic with their optional
|
|
||||||
Tree-sitter grammars. Their normal projections contain Logic, so they must not be presented or
|
|
||||||
configured as no-AST adapters.
|
|
||||||
|
|
||||||
The narrower performance statements in [Reference Adapters](REFERENCE_ADAPTERS.md) concern
|
|
||||||
specific manifest and warm-cache paths. A parser-free manifest does not turn an adapter that
|
|
||||||
publishes AST-derived Logic into a no-AST adapter.
|
|
||||||
|
|
||||||
## Migration choices
|
|
||||||
|
|
||||||
Keep a one-method adapter when its complete rebuild is acceptable and no function Logic is needed.
|
|
||||||
Adopt incremental extraction when measured source parsing dominates synchronization. Adopt the
|
|
||||||
complete assembly contract when Logic is published or overlapping raw evidence needs deterministic
|
|
||||||
ownership.
|
|
||||||
|
|
||||||
Choose no-AST only when the project binding must forbid AST-derived publication and future adapter
|
|
||||||
upgrades of that kind. It is a binding policy, not a substitute for adapter confinement,
|
|
||||||
conformance tests, restart detection, cache recovery, or the full proof matrix in
|
|
||||||
[Incremental Adapter Indexing](INCREMENTAL_INDEXING.md).
|
|
||||||
|
|
@ -28,30 +28,6 @@ Bootstrap and contract results include `effective_policy` schema version 1 plus
|
||||||
registered surface and startup-bound proposal/application access. The project descriptor remains
|
registered surface and startup-bound proposal/application access. The project descriptor remains
|
||||||
schema version 1 and does not silently acquire machine-specific process policy.
|
schema version 1 and does not silently acquire machine-specific process policy.
|
||||||
|
|
||||||
## Fixed reference binding
|
|
||||||
|
|
||||||
`docforge.reference_mcp` is the fixed runnable binding for the in-repository Python, JavaScript,
|
|
||||||
TypeScript, and C++ reference adapters:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -I -m docforge.reference_mcp \
|
|
||||||
--project-root /absolute/project \
|
|
||||||
--capability-mode read
|
|
||||||
```
|
|
||||||
|
|
||||||
It loads only `.docforge/reference-adapter.toml`, selects only the fixed provider for the declared
|
|
||||||
language, and constructs the same project-bound read surface through `create_read_only_server()`.
|
|
||||||
Bootstrap binding metadata includes `server_module = "docforge.reference_mcp"`,
|
|
||||||
`adapter_mode = "reference"`, the selected `reference_language`, and the exact
|
|
||||||
`reference_config_hash`.
|
|
||||||
|
|
||||||
The reference command accepts read capability mode only. It registers all 21 read tools below and
|
|
||||||
no isolated proposal or canonical-application tools. The configuration cannot select a provider,
|
|
||||||
module, command, arguments, working directory, environment, or arbitrary discovery behavior. See
|
|
||||||
[Reference Adapters](REFERENCE_ADAPTERS.md) for language scope and
|
|
||||||
[Agent Integration](AGENT_INTEGRATION.md) for the immutable launcher and generated client
|
|
||||||
fragments.
|
|
||||||
|
|
||||||
## Read tools
|
## Read tools
|
||||||
|
|
||||||
- `docforge_bootstrap`
|
- `docforge_bootstrap`
|
||||||
|
|
@ -69,8 +45,6 @@ fragments.
|
||||||
- `docforge_get_task_context`
|
- `docforge_get_task_context`
|
||||||
- `docforge_validate_project`
|
- `docforge_validate_project`
|
||||||
- `docforge_render_status`
|
- `docforge_render_status`
|
||||||
- `docforge_graph_plan`
|
|
||||||
- `docforge_graph_render_status`
|
|
||||||
- `docforge_visualize`
|
- `docforge_visualize`
|
||||||
- `docforge_stop_visualization`
|
- `docforge_stop_visualization`
|
||||||
- `docforge_visualization_status`
|
- `docforge_visualization_status`
|
||||||
|
|
@ -375,6 +349,3 @@ permissions and project instructions remain responsible for that broader boundar
|
||||||
The legacy `adapter_policy` object remains byte-compatible. It is now a projection of the
|
The legacy `adapter_policy` object remains byte-compatible. It is now a projection of the
|
||||||
versioned `effective_policy`; `--no-ast` restrictively overrides adapter evolution, AST analysis,
|
versioned `effective_policy`; `--no-ast` restrictively overrides adapter evolution, AST analysis,
|
||||||
and Logic indexing without widening any other capability.
|
and Logic indexing without widening any other capability.
|
||||||
|
|
||||||
See [Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for the compatibility boundary and
|
|
||||||
why the AST and Tree-sitter reference adapters are not no-AST adapters.
|
|
||||||
|
|
|
||||||
|
|
@ -1,79 +0,0 @@
|
||||||
# Migrating from DocForge v1
|
|
||||||
|
|
||||||
DocForge2 preserves the `docforge` distribution, Python package, CLI command, MCP tool prefix,
|
|
||||||
generic project descriptor version, and legacy adapter entry point. Migration is an additive
|
|
||||||
validation exercise, not a canonical-content rewrite.
|
|
||||||
|
|
||||||
Read [compatibility](COMPATIBILITY.md), [legacy and no-AST operation](LEGACY_AND_NO_AST.md), and
|
|
||||||
[recovery](RECOVERY_AND_PERFORMANCE.md) before changing a production binding.
|
|
||||||
|
|
||||||
## What remains compatible
|
|
||||||
|
|
||||||
- Generic Markdown and TOML projects keep `.docforge/project.toml` schema version 1.
|
|
||||||
- A custom adapter implementing only `load_projection()` remains valid.
|
|
||||||
- Existing canonical nodes, stable IDs, source paths, relationship vocabulary, changesets, and
|
|
||||||
reviewed hashes are not silently rewritten.
|
|
||||||
- Existing CLI and MCP names remain available. New fields, commands, tools, plans, policies, and
|
|
||||||
reference integrations are additive.
|
|
||||||
- `docforge-mcp --no-ast` remains the shorthand for the preserved no-AST binding.
|
|
||||||
|
|
||||||
Disposable index and cache schemas may change. Rebuild them rather than copying them as authority.
|
|
||||||
|
|
||||||
## Recommended migration
|
|
||||||
|
|
||||||
1. Record the v1 package version, adapter identity, project descriptor, canonical source hash,
|
|
||||||
active changesets, generated client configuration, and current rendered outputs.
|
|
||||||
2. Back up canonical sources and active proposal files. Derived `.docforge/cache`, preview,
|
|
||||||
portable-graph, and viewer state need not be authoritative backups.
|
|
||||||
3. Install the DocForge2 candidate in a separate environment. Do not repoint the production MCP
|
|
||||||
binding yet.
|
|
||||||
4. Run the existing adapter through complete loading and `ProjectIndex.check()`. For incremental
|
|
||||||
adapters, verify complete/incremental equivalence. Logic-producing incremental adapters must
|
|
||||||
implement the independent complete assembly oracle.
|
|
||||||
5. Compare project ID, adapter, revision, source hash, primary node and edge hashes, Logic policy,
|
|
||||||
and retrieval results with the v1 evidence.
|
|
||||||
6. Recompose effective capability and projection policy. Do not assume that a new default grants
|
|
||||||
proposal, application, rendering, or viewer access.
|
|
||||||
7. Rebuild disposable indexes, extraction caches, fragments, receipts, previews, and portable
|
|
||||||
artifacts from the validated candidate.
|
|
||||||
8. Generate a new client fragment. Generic projects use `docforge configure`; custom adapters use
|
|
||||||
`AdapterLauncherV1` with `generate_adapter_client_configuration()`.
|
|
||||||
9. Run doctor, real MCP bootstrap, representative retrieval, rendering status, recovery, and
|
|
||||||
exact-hash proposal/application checks in a disposable or shadow environment.
|
|
||||||
10. Repoint one production binding only after the candidate and rollback procedure pass.
|
|
||||||
|
|
||||||
## Adopting reference adapters
|
|
||||||
|
|
||||||
The repository reference adapters are narrow syntax evidence, not automatic semantic replacements
|
|
||||||
for mature v1 integrations. Python publishes modules, classes, functions, and local imports.
|
|
||||||
JavaScript and TypeScript publish syntax plus static project-relative imports and re-exports.
|
|
||||||
C++ uses a confined `compile_commands.json` as inert translation-unit inventory and publishes
|
|
||||||
directly resolvable project-local quoted includes.
|
|
||||||
|
|
||||||
Do not replace a compiler-, language-server-, or project-owned semantic adapter if the task depends
|
|
||||||
on resolved calls, types, inheritance, macros, compiler include semantics, runtime facts, or
|
|
||||||
ownership. Use [reference adapters](REFERENCE_ADAPTERS.md) as a measured starting point.
|
|
||||||
|
|
||||||
## Changesets
|
|
||||||
|
|
||||||
Existing changesets remain bound to their original project, root, base revision, source hash,
|
|
||||||
writer, operations, and exact hash. Do not edit a changeset to make it look current. Retrieve and
|
|
||||||
validate it through the candidate. Rebase only when every precondition remains true; otherwise
|
|
||||||
create and review a new proposal.
|
|
||||||
|
|
||||||
Before enabling canonical application, prove the exact serializer round trip and recovery behavior
|
|
||||||
for the selected adapter. The fixed reference MCP server is intentionally read-only.
|
|
||||||
|
|
||||||
## Rollback
|
|
||||||
|
|
||||||
Keep the v1 environment, configuration fragment, and service definition until the DocForge2
|
|
||||||
binding has passed live validation. To roll back:
|
|
||||||
|
|
||||||
1. Stop the DocForge2 MCP/viewer processes.
|
|
||||||
2. Restore the previous client or service binding.
|
|
||||||
3. Restore canonical sources only if a verified application receipt says they changed and the
|
|
||||||
adapter-specific rollback requires it.
|
|
||||||
4. Discard DocForge2-derived indexes, receipts, fragments, previews, and portable artifacts.
|
|
||||||
5. Restart v1 and verify its recorded project/source identity.
|
|
||||||
|
|
||||||
Never move a published tag or reuse a version identity for a corrected release.
|
|
||||||
|
|
@ -1,95 +0,0 @@
|
||||||
# Milestone 4 baseline
|
|
||||||
|
|
||||||
## Scope and method
|
|
||||||
|
|
||||||
This baseline records the adapter SDK, Python reference adapter, incremental equivalence, and
|
|
||||||
recovery behavior completed in Milestone 4. It was captured on 2026-07-29 from clean executable
|
|
||||||
candidate `95271dcf2e48045b9d3aed9b9ea09c7fc155692c`.
|
|
||||||
|
|
||||||
The maintained command was:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
.venv/bin/python tools/milestone4_benchmark.py \
|
|
||||||
--mode full \
|
|
||||||
--output benchmarks/milestone4-2026-07-29.json
|
|
||||||
```
|
|
||||||
|
|
||||||
The synthetic project contains 334 Python files. Each file contributes one module, one function,
|
|
||||||
and one argument node, for 1,002 primary nodes. Imports form a deterministic chain. Durations use
|
|
||||||
`time.perf_counter_ns()` and nearest-rank p95. Each ordinary result is serialized as compact sorted
|
|
||||||
JSON for its response-size gate. Per-operation memory uses `tracemalloc`; cumulative process
|
|
||||||
high-water uses `RUSAGE_SELF`.
|
|
||||||
|
|
||||||
Environment:
|
|
||||||
|
|
||||||
- Linux 7.1.3-200.nobara.fc44.x86_64 with glibc 2.43.
|
|
||||||
- CPython 3.14.6.
|
|
||||||
- x86_64.
|
|
||||||
- Three warm samples; cold, equivalence, and recovery operations run once.
|
|
||||||
- Per-operation traced-memory ceiling: 268,435,456 bytes.
|
|
||||||
- Process high-water ceiling: 536,870,912 bytes.
|
|
||||||
- Response ceiling: 524,288 bytes.
|
|
||||||
|
|
||||||
The complete machine-readable result is
|
|
||||||
[`benchmarks/milestone4-2026-07-29.json`](../benchmarks/milestone4-2026-07-29.json).
|
|
||||||
Its SHA-256 is `b6a871dde730a119fc0a138c47bd25f2c533ed3171a9a3d098075aef08173600`.
|
|
||||||
The stable evidence payload SHA-256 is
|
|
||||||
`4a8db461a311b5df4abd5aa00063e9a347d8b9dba19b30e35684f561d5271549`.
|
|
||||||
|
|
||||||
## Results
|
|
||||||
|
|
||||||
| Operation | Median | p95 | Limit | Traced peak | Response |
|
|
||||||
|---|---:|---:|---:|---:|---:|
|
|
||||||
| Cold incremental build | 1,599.760 ms | 1,599.760 ms | 20,000 ms | 68,573,540 B | 14,580 B |
|
|
||||||
| Warm incremental build | 1,244.427 ms | 1,265.387 ms | 20,000 ms | 69,997,166 B | 14,578 B |
|
|
||||||
| Complete/incremental equivalence | 1,897.919 ms | 1,897.919 ms | 30,000 ms | 64,305,919 B | 230 B |
|
|
||||||
| Corrupt extraction-cache recovery | 1,695.406 ms | 1,695.406 ms | 30,000 ms | 69,776,923 B | 14,580 B |
|
|
||||||
| Corrupt index recovery | 1,707.879 ms | 1,707.879 ms | 30,000 ms | 68,561,924 B | 14,779 B |
|
|
||||||
|
|
||||||
Process high-water was 78,798,848 bytes. The regression limits intentionally leave multiple times
|
|
||||||
the measured headroom; they are tripwires, not performance promises.
|
|
||||||
|
|
||||||
## Deterministic graph and Logic evidence
|
|
||||||
|
|
||||||
The candidate produced:
|
|
||||||
|
|
||||||
- 1,002 primary nodes with hash
|
|
||||||
`f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f`.
|
|
||||||
- 1,001 primary edges with hash
|
|
||||||
`00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35`.
|
|
||||||
- 334 Logic projections containing 2,338 Logic nodes and 2,338 Logic edges, with hash
|
|
||||||
`c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c`.
|
|
||||||
- Complete assembly hash
|
|
||||||
`2888182ab765fbffe3ba873c1613345640e8e6d89be74ddfca7452c0a5056345`.
|
|
||||||
- Source hash
|
|
||||||
`30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2`.
|
|
||||||
|
|
||||||
The independently loaded complete assembly and the incremental assembly matched exactly across
|
|
||||||
project identity, revision, source, nodes, edges, and Logic.
|
|
||||||
|
|
||||||
## Work and recovery gates
|
|
||||||
|
|
||||||
The warm build recorded all 334 sources as cache hits, zero reparsed sources, zero `ast.parse`
|
|
||||||
calls, and zero `extract_source` calls. Corrupting the extraction cache forced all sources through
|
|
||||||
extraction and reproduced the same graph and Logic hashes. Corrupting SQLite rebuilt the index
|
|
||||||
entirely from cache hits with zero parsing or extraction and reproduced those hashes.
|
|
||||||
|
|
||||||
This zero-parser evidence applies to the Python benchmark. Focused JavaScript and TypeScript tests
|
|
||||||
prove parser-free manifests. The C++ reference manifest uses Tree-sitter for bounded quoted-include
|
|
||||||
discovery and does not make a zero-warm-parser claim.
|
|
||||||
|
|
||||||
## Fresh-wheel and reference evidence
|
|
||||||
|
|
||||||
The offline adoption proof built and installed the base wheel without Tree-sitter packages,
|
|
||||||
built and checked a real Python reference project, started the isolated reference MCP server, and
|
|
||||||
performed bootstrap, search, and exact retrieval over its 21 read tools. Selecting C++ without its
|
|
||||||
extra failed with `optional_dependency_missing` and `docforge[cpp]` remediation.
|
|
||||||
|
|
||||||
Focused fixtures produced:
|
|
||||||
|
|
||||||
- Python: 14 nodes, 13 edges, 5 Logic projections.
|
|
||||||
- JavaScript: 14 nodes, 14 edges, 5 Logic projections.
|
|
||||||
- TypeScript: 13 nodes, 14 edges, 4 Logic projections.
|
|
||||||
- C++: 17 nodes, 16 edges, 5 Logic projections.
|
|
||||||
|
|
||||||
These fixture counts verify implementations; they are not language-wide completeness claims.
|
|
||||||
|
|
@ -1,83 +0,0 @@
|
||||||
# Milestone 4 closeout
|
|
||||||
|
|
||||||
## Outcome
|
|
||||||
|
|
||||||
Milestone 4 is complete. New projects can adopt a public adapter SDK or one of four narrow
|
|
||||||
repository reference integrations, attach a fixed read-only MCP server, and follow maintained
|
|
||||||
product documentation without reading core implementation.
|
|
||||||
|
|
||||||
Implemented contracts:
|
|
||||||
|
|
||||||
- Stable `docforge.adapter_sdk` authoring imports.
|
|
||||||
- Independent complete primary-graph-plus-Logic oracle and exact incremental equivalence.
|
|
||||||
- Bounded adapter assemblies and version-1 extraction caches.
|
|
||||||
- Base Python, optional JavaScript, optional TypeScript, and optional C++ reference integrations.
|
|
||||||
- Closed `.docforge/reference-adapter.toml` and fixed `docforge.reference_mcp` read-only binding.
|
|
||||||
- Immutable, project-bound, launchable `AdapterLauncherV1` declarations and generated Codex,
|
|
||||||
Claude, and OpenClaw fragments for custom adapters.
|
|
||||||
- Live implementation-derived CLI and MCP reference tables with race-safe publication and drift
|
|
||||||
checking.
|
|
||||||
- Strict documentation graph, link, anchor, H1, reachability, required-page, generated-notice, and
|
|
||||||
documented-reference-config checks.
|
|
||||||
- Offline fresh-wheel adoption and maintained 1,002-node scale/recovery gates.
|
|
||||||
|
|
||||||
## Candidate evidence
|
|
||||||
|
|
||||||
The frozen executable candidate is
|
|
||||||
`95271dcf2e48045b9d3aed9b9ea09c7fc155692c`.
|
|
||||||
|
|
||||||
Its complete executable gate passed:
|
|
||||||
|
|
||||||
- Ruff formatting and Python lint.
|
|
||||||
- HTML, rendered-manual HTML, portable-graph HTML, CSS, and JavaScript checks.
|
|
||||||
- Pyright with zero diagnostics.
|
|
||||||
- Warning-strict compilation.
|
|
||||||
- 142 contract tests and 268 subtests.
|
|
||||||
- 347 complete tests and 402 subtests.
|
|
||||||
- Three Playwright and axe accessibility flows for the manual, portable graph, and live viewer.
|
|
||||||
- Lock and npm dependency-tree checks.
|
|
||||||
- Wheel and source-distribution builds.
|
|
||||||
- Offline fresh-wheel adoption.
|
|
||||||
- Milestone 0, 1, 2, 3, and 4 smoke benchmarks.
|
|
||||||
|
|
||||||
The clean full benchmark passed exact graph-plus-Logic equivalence, warm zero Python parser and
|
|
||||||
extraction work, corrupt-cache recovery, corrupt-index recovery, response, memory, and latency
|
|
||||||
gates. Exact results are in [the Milestone 4 baseline](MILESTONE_4_BASELINE.md) and
|
|
||||||
[`benchmarks/milestone4-2026-07-29.json`](../benchmarks/milestone4-2026-07-29.json).
|
|
||||||
|
|
||||||
Gitleaks 8.30.1 scanned the Milestone 4 commit range and candidate tree with no findings. The SSH
|
|
||||||
remote syntax prevents Gitleaks from constructing finding hyperlinks; it does not affect scanning.
|
|
||||||
|
|
||||||
## Reference scope
|
|
||||||
|
|
||||||
- Python uses the standard-library AST and publishes syntax plus local imports.
|
|
||||||
- JavaScript and TypeScript use distinct optional Tree-sitter grammars and publish syntax plus
|
|
||||||
project-local static relative imports and re-exports.
|
|
||||||
- C++ uses a confined `compile_commands.json` as inert translation-unit inventory and publishes
|
|
||||||
syntax plus directly resolvable project-local quoted includes.
|
|
||||||
- The references do not claim resolved calls, inheritance, types, symbol references, compiler
|
|
||||||
include semantics, macro semantics, runtime behavior, or semantic ownership.
|
|
||||||
|
|
||||||
The C++ reference never executes a compiler or compilation-database command. It is not a Clang
|
|
||||||
semantic adapter.
|
|
||||||
|
|
||||||
## Preserved boundaries
|
|
||||||
|
|
||||||
- The `docforge` distribution, Python package, CLI, MCP executable, and tool names remain.
|
|
||||||
- Generic projects and one-method `load_projection()` adapters remain supported.
|
|
||||||
- Descriptor schema version 1, index schema version 3, effective policy version 1, and no-AST
|
|
||||||
behavior remain.
|
|
||||||
- Heavy language frontends are optional. Base generic and Python operation installs no
|
|
||||||
Tree-sitter distribution.
|
|
||||||
- Reference MCP is read-only. Proposal and application remain explicit project-owned gates.
|
|
||||||
- No language adapter was separately published.
|
|
||||||
- No WorldForge, ScrapeStation, legacy-repository, production-binding, storage, or self-hosting
|
|
||||||
change was made.
|
|
||||||
- No tag or Forgejo release was created for Milestone 4.
|
|
||||||
|
|
||||||
## Later work
|
|
||||||
|
|
||||||
Milestone 5 owns stabilization and the first DocForge2 release. Release identity, compatibility
|
|
||||||
matrix, migration and recovery proofs, comparative real-task evidence, versioning, tagging, and
|
|
||||||
publication must be validated there. Remote adapters, render farms, third-party renderers,
|
|
||||||
cross-project graphs, storage replacement, and self-hosting remain deferred without measured need.
|
|
||||||
|
|
@ -1,276 +1,6 @@
|
||||||
# New-project quickstart
|
# DocForge setup moved to the user manual
|
||||||
|
|
||||||
This guide takes a new installation from an empty project binding to a validated generic manual or
|
The complete installation, project setup, visualization, CLI, MCP, application, adapter, and
|
||||||
one of DocForge's fixed reference source graphs. For the complete operating reference, see the
|
troubleshooting reference now lives in the [DocForge user manual](USER_MANUAL.md).
|
||||||
[user manual](USER_MANUAL.md).
|
|
||||||
|
|
||||||
## 1. Install DocForge
|
This file remains only so existing bookmarks and links continue to resolve.
|
||||||
|
|
||||||
DocForge requires Python 3.12 or newer. A base installation includes the generic project service,
|
|
||||||
the public adapter SDK, the Python reference integration, CLI, MCP server, renderers, and viewer
|
|
||||||
assets. It does not install Tree-sitter.
|
|
||||||
|
|
||||||
From a source checkout:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git clone <repository-url> /absolute/path/DocForge
|
|
||||||
cd /absolute/path/DocForge
|
|
||||||
uv sync --group dev
|
|
||||||
|
|
||||||
DOCFORGE=/absolute/path/DocForge/.venv/bin/docforge
|
|
||||||
DOCFORGE_MCP=/absolute/path/DocForge/.venv/bin/docforge-mcp
|
|
||||||
DOCFORGE_PYTHON=/absolute/path/DocForge/.venv/bin/python
|
|
||||||
```
|
|
||||||
|
|
||||||
For an isolated consumer environment, install the checkout or a built wheel with `uv pip install`.
|
|
||||||
Add only the language extras that project needs:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
uv venv /absolute/path/docforge-env --python 3.12
|
|
||||||
uv pip install --python /absolute/path/docforge-env/bin/python /absolute/path/DocForge
|
|
||||||
uv pip install --python /absolute/path/docforge-env/bin/python \
|
|
||||||
"/absolute/path/DocForge[javascript]"
|
|
||||||
uv pip install --python /absolute/path/docforge-env/bin/python \
|
|
||||||
"/absolute/path/DocForge[typescript]"
|
|
||||||
uv pip install --python /absolute/path/docforge-env/bin/python \
|
|
||||||
"/absolute/path/DocForge[cpp]"
|
|
||||||
```
|
|
||||||
|
|
||||||
The `languages` extra installs all three optional grammar families. JavaScript and TypeScript are
|
|
||||||
separate extras because they use distinct grammars. The C++ extra supplies a syntax grammar, not a
|
|
||||||
compiler or Clang semantic frontend.
|
|
||||||
|
|
||||||
Confirm the installation:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
"$DOCFORGE" --help
|
|
||||||
"$DOCFORGE_MCP" --help
|
|
||||||
```
|
|
||||||
|
|
||||||
## 2. Choose a project route
|
|
||||||
|
|
||||||
Use a generic project when Markdown or TOML documentation is canonical. Use a fixed reference
|
|
||||||
adapter when you want a bounded source inventory and syntax-level graph for Python, JavaScript,
|
|
||||||
TypeScript, or C++. Use a project-owned adapter when the production contract must supply richer
|
|
||||||
semantics.
|
|
||||||
|
|
||||||
- Generic manual: continue with [Create a generic project](#create-a-generic-project).
|
|
||||||
- Fixed source example: continue with [Use a reference adapter](#use-a-reference-adapter).
|
|
||||||
- Production language frontend: follow the [Adapter authoring guide](ADAPTER_AUTHORING_GUIDE.md).
|
|
||||||
|
|
||||||
## Create a generic project
|
|
||||||
|
|
||||||
Set the absolute project root and assess it without writing:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
PROJECT=/absolute/path/MyProject
|
|
||||||
"$DOCFORGE" --project-root "$PROJECT" onboard
|
|
||||||
```
|
|
||||||
|
|
||||||
The assessment reports detected languages, build evidence, documentation candidates, current
|
|
||||||
configuration, and available capabilities. Detection never invents a source graph.
|
|
||||||
|
|
||||||
Create a generic starter explicitly:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
"$DOCFORGE" --project-root "$PROJECT" onboard \
|
|
||||||
--scaffold \
|
|
||||||
--project-id my-project \
|
|
||||||
--title "My Project"
|
|
||||||
```
|
|
||||||
|
|
||||||
Scaffolding is create-only and refuses existing target files. It creates:
|
|
||||||
|
|
||||||
- `.docforge/project.toml`, the generic project descriptor;
|
|
||||||
- `docs/docforge/content/architecture-overview.md`, one canonical node;
|
|
||||||
- `.docforge/templates/manual.html`, one built-in-renderer template;
|
|
||||||
- derived index, receipt, and rendered output below `.docforge`.
|
|
||||||
|
|
||||||
If a language was detected, source graph status remains `adapter_required`. The generic starter
|
|
||||||
does not claim source semantics.
|
|
||||||
|
|
||||||
Validate and inspect it:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
"$DOCFORGE" --project-root "$PROJECT" validate
|
|
||||||
"$DOCFORGE" --project-root "$PROJECT" reindex
|
|
||||||
"$DOCFORGE" --project-root "$PROJECT" search architecture
|
|
||||||
"$DOCFORGE" --project-root "$PROJECT" show architecture.overview
|
|
||||||
"$DOCFORGE" --project-root "$PROJECT" render-status
|
|
||||||
```
|
|
||||||
|
|
||||||
The descriptor is explained field by field in [Project descriptor](PROJECT_DESCRIPTOR.md). Add
|
|
||||||
canonical nodes only after choosing their authority, stable IDs, families, statuses, and allowed
|
|
||||||
relationships; [Core concepts and authority](CORE_CONCEPTS_AND_AUTHORITY.md) defines those terms.
|
|
||||||
|
|
||||||
### Start a generic MCP binding
|
|
||||||
|
|
||||||
Start read-only first:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
"$DOCFORGE_MCP" \
|
|
||||||
--project-root "$PROJECT" \
|
|
||||||
--capability-mode read
|
|
||||||
```
|
|
||||||
|
|
||||||
Call `docforge_bootstrap` before other tools. It reports the exact project identity, generation,
|
|
||||||
effective policy, projection policy, available capabilities, and recommended first read.
|
|
||||||
|
|
||||||
To enable proposals, the descriptor must declare the writer and the process must select it:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
"$DOCFORGE_MCP" \
|
|
||||||
--project-root "$PROJECT" \
|
|
||||||
--capability-mode proposal \
|
|
||||||
--proposal-writer project-editor
|
|
||||||
```
|
|
||||||
|
|
||||||
Canonical application is a separate startup gate. Do not add it to a read-only client:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
"$DOCFORGE_MCP" \
|
|
||||||
--project-root "$PROJECT" \
|
|
||||||
--capability-mode application \
|
|
||||||
--proposal-writer project-editor \
|
|
||||||
--canonical-applier project-editor
|
|
||||||
```
|
|
||||||
|
|
||||||
Application accepts one exact reviewed changeset hash. It does not commit, push, build, deploy, or
|
|
||||||
publish the project.
|
|
||||||
|
|
||||||
### Generate a generic client fragment
|
|
||||||
|
|
||||||
Preview is side-effect free:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
"$DOCFORGE" configure codex --project "$PROJECT"
|
|
||||||
"$DOCFORGE" configure claude --project "$PROJECT"
|
|
||||||
"$DOCFORGE" configure openclaw --project "$PROJECT"
|
|
||||||
```
|
|
||||||
|
|
||||||
Add `--output /absolute/path/new-fragment` to create one new private standalone file. Generation
|
|
||||||
does not merge with or replace a different existing file. Diagnose an installed binding with:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
"$DOCFORGE" doctor --client codex --project "$PROJECT"
|
|
||||||
```
|
|
||||||
|
|
||||||
Doctor is a bounded configuration inspector, not a connection test. See [Agent
|
|
||||||
integration](AGENT_INTEGRATION.md) for client-specific layouts and limitations.
|
|
||||||
|
|
||||||
## Use a reference adapter
|
|
||||||
|
|
||||||
Reference adapters read one fixed descriptor:
|
|
||||||
`.docforge/reference-adapter.toml`. The path is not selectable.
|
|
||||||
|
|
||||||
Create a Python project configuration:
|
|
||||||
|
|
||||||
```toml reference-adapter
|
|
||||||
schema_version = 1
|
|
||||||
project_id = "my-python-project"
|
|
||||||
title = "My Python Project"
|
|
||||||
language = "python"
|
|
||||||
source_roots = ["src"]
|
|
||||||
```
|
|
||||||
|
|
||||||
JavaScript uses `language = "javascript"` and the `javascript` extra. TypeScript uses
|
|
||||||
`language = "typescript"` and the `typescript` extra.
|
|
||||||
|
|
||||||
C++ additionally requires a confined compilation database:
|
|
||||||
|
|
||||||
```toml reference-adapter
|
|
||||||
schema_version = 1
|
|
||||||
project_id = "my-cpp-project"
|
|
||||||
title = "My C++ Project"
|
|
||||||
language = "cpp"
|
|
||||||
source_roots = ["src", "include"]
|
|
||||||
compilation_database = "compile_commands.json"
|
|
||||||
```
|
|
||||||
|
|
||||||
The C++ integration reads `compile_commands.json` only as bounded translation-unit inventory and
|
|
||||||
fingerprint evidence. It never executes a recorded command or compiler.
|
|
||||||
|
|
||||||
Run the fixed read-only server through the same installed Python interpreter:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
"$DOCFORGE_PYTHON" -I -m docforge.reference_mcp \
|
|
||||||
--project-root "$PROJECT" \
|
|
||||||
--capability-mode read
|
|
||||||
```
|
|
||||||
|
|
||||||
The binding chooses one in-package provider from the descriptor language. It accepts no provider
|
|
||||||
module, command, argument list, working directory, environment, discovery rule, proposal writer,
|
|
||||||
or canonical applier. Its MCP surface is the read subset documented in the [generated command
|
|
||||||
reference](COMMAND_REFERENCE.md).
|
|
||||||
|
|
||||||
### Generate a reference-adapter client fragment
|
|
||||||
|
|
||||||
Generic `docforge configure` intentionally refuses custom adapters. Construct the adapter project
|
|
||||||
and immutable launcher, then call the custom-adapter generator:
|
|
||||||
|
|
||||||
```python
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from docforge.adapter_launcher import AdapterLauncherV1
|
|
||||||
from docforge.client_config import generate_adapter_client_configuration
|
|
||||||
from docforge.reference_mcp import REFERENCE_MCP_MODULE, create_reference_project
|
|
||||||
|
|
||||||
root = Path("/absolute/path/MyProject").resolve(strict=True)
|
|
||||||
project = create_reference_project(root)
|
|
||||||
launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE)
|
|
||||||
|
|
||||||
preview = generate_adapter_client_configuration(
|
|
||||||
project,
|
|
||||||
launcher,
|
|
||||||
"codex",
|
|
||||||
capability_mode="read",
|
|
||||||
)
|
|
||||||
print(preview["artifact"]["content"])
|
|
||||||
```
|
|
||||||
|
|
||||||
Use `client="claude"` or `client="openclaw"` for those formats. Pass an absolute `output` path only
|
|
||||||
when creating a new standalone private fragment. The generated launch is bound to the selected
|
|
||||||
project, descriptor hash, adapter identity, installed module, isolated interpreter, policy, and
|
|
||||||
source availability evidence.
|
|
||||||
|
|
||||||
Read [Reference adapters](REFERENCE_ADAPTERS.md) before relying on the graph. The Python example
|
|
||||||
publishes local imports, the JavaScript and TypeScript examples publish project-local static
|
|
||||||
relative imports and re-exports, and the C++ example publishes directly resolvable project-local
|
|
||||||
quoted includes. None is a complete semantic compiler frontend.
|
|
||||||
|
|
||||||
## 3. Add visualization only when needed
|
|
||||||
|
|
||||||
Install the per-user viewer manager once:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
/absolute/path/DocForge/.venv/bin/docforge-viewer-manager install-user-service
|
|
||||||
```
|
|
||||||
|
|
||||||
Then start a project-bound snapshot:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
"$DOCFORGE" --project-root "$PROJECT" visualize
|
|
||||||
"$DOCFORGE" --project-root "$PROJECT" visualization-status
|
|
||||||
```
|
|
||||||
|
|
||||||
The listener is loopback-only and tokenized. The viewer is a derived snapshot, not canonical
|
|
||||||
authority and not a continuously monitored filesystem view. Read [Rendering and
|
|
||||||
visualization](RENDERING_AND_VISUALIZATION.md) for manual, portable, and live-viewer differences.
|
|
||||||
|
|
||||||
## 4. Verify the maintained checkout
|
|
||||||
|
|
||||||
Contributors can run:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
make command-reference-check
|
|
||||||
make docs-check
|
|
||||||
make adoption-m4
|
|
||||||
make benchmark-m4-smoke
|
|
||||||
```
|
|
||||||
|
|
||||||
Run `make gate` before a release candidate. `benchmark-m4-smoke` is routine coverage;
|
|
||||||
`benchmark-m4` is the maintained full adapter workload.
|
|
||||||
|
|
||||||
Continue with [Project onboarding](PROJECT_ONBOARDING.md) for production integration,
|
|
||||||
[Policy precedence](POLICY_PRECEDENCE.md) before widening a binding, and [Security](SECURITY.md)
|
|
||||||
before exposing any MCP process.
|
|
||||||
|
|
|
||||||
|
|
@ -1,202 +0,0 @@
|
||||||
# Policy precedence
|
|
||||||
|
|
||||||
DocForge composes an immutable effective policy for each project-bound process. It resolves
|
|
||||||
restrictions in a fixed order:
|
|
||||||
|
|
||||||
```text
|
|
||||||
core safety
|
|
||||||
> explicit binding
|
|
||||||
> no-AST shorthand
|
|
||||||
> resource availability
|
|
||||||
```
|
|
||||||
|
|
||||||
A lower layer can make a requested operation unavailable; it cannot override a higher-layer
|
|
||||||
prohibition. Bootstrap and generated client evidence report the composed result and hashes so a
|
|
||||||
client does not need to infer policy from command-line arguments.
|
|
||||||
|
|
||||||
## 1. Core safety
|
|
||||||
|
|
||||||
Core safety is unconditional. No capability mode exposes:
|
|
||||||
|
|
||||||
- arbitrary file access or project switching;
|
|
||||||
- arbitrary renderer, module, command, shell, argument, working-directory, or environment
|
|
||||||
selection;
|
|
||||||
- Git mutation;
|
|
||||||
- project builds or compiler execution;
|
|
||||||
- deployment or publication.
|
|
||||||
|
|
||||||
Canonical application is limited to DocForge's validated serializer boundary. Adapter launch is
|
|
||||||
limited to the immutable launcher contract. Rendering is limited to declared views and fixed
|
|
||||||
built-in workers. An indexed instruction cannot alter any of these rules.
|
|
||||||
|
|
||||||
## 2. Explicit binding
|
|
||||||
|
|
||||||
A process is bound at startup to one project root, descriptor, adapter, capability mode, writer and
|
|
||||||
applier identities when present, no-AST selection, diagnostics selection, and projection modes.
|
|
||||||
The binding does not change during the process lifetime.
|
|
||||||
|
|
||||||
Capability modes are:
|
|
||||||
|
|
||||||
- `read`: register only the read surface;
|
|
||||||
- `proposal`: add proposal tools when a valid selected writer is available;
|
|
||||||
- `application`: require a startup-bound canonical applier and expose exact-hash application;
|
|
||||||
- `operator`: reserved; it currently adds no tools.
|
|
||||||
|
|
||||||
Mode describes the maximum registered surface. Actual authority can be narrower. A descriptor must
|
|
||||||
declare the selected writer, including allowed families and operation types. Application requires
|
|
||||||
the matching configured writer, changeset creator, and canonical-applier identity. A mode name
|
|
||||||
cannot create a missing descriptor grant.
|
|
||||||
|
|
||||||
Generic generated client fragments default to read mode. Other construction paths preserve their
|
|
||||||
documented compatible factory defaults. Treat `docforge_bootstrap.session_contract` and its actual
|
|
||||||
capabilities as authoritative for a running server.
|
|
||||||
|
|
||||||
## 3. No-AST shorthand
|
|
||||||
|
|
||||||
`--no-ast` is a restrictive compatibility shorthand. It composes:
|
|
||||||
|
|
||||||
- adapter evolution `preserve`;
|
|
||||||
- AST analysis `forbidden`;
|
|
||||||
- Logic indexing `off`;
|
|
||||||
- `docforge_get_logic` blocked;
|
|
||||||
- prohibitions on AST, Tree-sitter, compiler-AST, and function-Logic upgrades.
|
|
||||||
|
|
||||||
Non-AST source fingerprinting and incremental caching remain allowed. Existing one-method adapters
|
|
||||||
continue to use `load_projection()`.
|
|
||||||
|
|
||||||
The shorthand does not inspect how an existing adapter was implemented, sandbox its filesystem
|
|
||||||
reads, or transform an AST-based adapter into a no-AST adapter. Do not run the Python,
|
|
||||||
JavaScript/TypeScript, or C++ syntax reference integrations and then describe the binding as a
|
|
||||||
proved no-AST integration. See [Legacy and no-AST operation](LEGACY_AND_NO_AST.md).
|
|
||||||
|
|
||||||
## 4. Resource availability
|
|
||||||
|
|
||||||
Even an allowed policy cannot create a missing resource:
|
|
||||||
|
|
||||||
- application mode requires a startup-bound canonical applier;
|
|
||||||
- manual `explicit` or `auto` requires declared manual render configuration;
|
|
||||||
- manual `auto` also requires canonical application in the current operation or server;
|
|
||||||
- portable graph `explicit` requires declared portable graph configuration;
|
|
||||||
- live viewer `on-demand` requires the viewer runtime;
|
|
||||||
- an optional reference language requires its installed extra;
|
|
||||||
- a project-owned launcher module must be installed, isolated, project-owned, and unchanged.
|
|
||||||
|
|
||||||
Unavailable requested modes fail before hidden work. Missing optional language grammars return an
|
|
||||||
actionable install target such as `docforge[typescript]` or `docforge[cpp]`; DocForge does not
|
|
||||||
silently downgrade to a different frontend.
|
|
||||||
|
|
||||||
## Effective process policy
|
|
||||||
|
|
||||||
The version-1 effective policy reports:
|
|
||||||
|
|
||||||
- capability mode and whether it came from a factory default or explicit selection;
|
|
||||||
- adapter evolution, AST analysis, and Logic indexing;
|
|
||||||
- automatic synchronization and validated integrity;
|
|
||||||
- the compatible manual-render projection;
|
|
||||||
- profiling state;
|
|
||||||
- blocked tools and prohibitions;
|
|
||||||
- the exact precedence list.
|
|
||||||
|
|
||||||
This version-1 projection preserves compatibility. It is not the complete version-2 rendering
|
|
||||||
policy; portable graph and live-viewer choices are reported separately.
|
|
||||||
|
|
||||||
## Independent projection policy
|
|
||||||
|
|
||||||
Manual rendering, portable graph publication, and live visualization use a separate immutable
|
|
||||||
version-2 policy:
|
|
||||||
|
|
||||||
```text
|
|
||||||
manual: auto | explicit | disabled
|
|
||||||
portable_graph: explicit | disabled
|
|
||||||
live_viewer: on-demand | disabled
|
|
||||||
```
|
|
||||||
|
|
||||||
When a selector is omitted, composition uses availability-aware defaults:
|
|
||||||
|
|
||||||
- manual is `auto` only when manual configuration and canonical application are both available;
|
|
||||||
otherwise it is `explicit` when configured, or `disabled`;
|
|
||||||
- portable graph is `explicit` when configured, otherwise `disabled`;
|
|
||||||
- live viewer is `on-demand` when the runtime is available, otherwise `disabled`.
|
|
||||||
|
|
||||||
An explicit non-disabled selection for a missing resource returns
|
|
||||||
`projection_policy_unavailable`. An active operation prohibited by the selected policy returns
|
|
||||||
`projection_policy_forbids_operation` before planning, rendering, or viewer startup.
|
|
||||||
|
|
||||||
Status remains intentionally narrower than active work. Manual and portable receipt-only status
|
|
||||||
are available when their active operations are disabled. Viewer status and explicit stop remain
|
|
||||||
available when viewer start is disabled.
|
|
||||||
|
|
||||||
Ordinary standalone CLI rendering uses manual `explicit`. Manual `auto` belongs to a canonical
|
|
||||||
application operation that owns automatic regeneration.
|
|
||||||
|
|
||||||
## Diagnostics do not grant authority
|
|
||||||
|
|
||||||
`--diagnostics` enables bounded request-local stage timings and compiler-work counters. It does not
|
|
||||||
enable tools, broaden paths, retain project content, or displace a primary MCP result that already
|
|
||||||
needs the response budget.
|
|
||||||
|
|
||||||
## Descriptor policy and process policy
|
|
||||||
|
|
||||||
The project descriptor is canonical project configuration. The process policy is a runtime
|
|
||||||
restriction. They compose by intersection:
|
|
||||||
|
|
||||||
```text
|
|
||||||
operation is available
|
|
||||||
only if core permits it
|
|
||||||
and the startup binding registers it
|
|
||||||
and no-AST permits it
|
|
||||||
and required resources exist
|
|
||||||
and the descriptor grants the requested project authority
|
|
||||||
and current graph/hash preconditions validate
|
|
||||||
```
|
|
||||||
|
|
||||||
Changing a descriptor does not retarget a running project-owned process. Descriptor, adapter
|
|
||||||
implementation, or launcher drift requires a fresh process.
|
|
||||||
|
|
||||||
## Common decisions
|
|
||||||
|
|
||||||
For an agent that only reads documentation:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docforge-mcp \
|
|
||||||
--project-root /absolute/path/MyProject \
|
|
||||||
--capability-mode read
|
|
||||||
```
|
|
||||||
|
|
||||||
For an agent that may prepare reviewable proposals:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docforge-mcp \
|
|
||||||
--project-root /absolute/path/MyProject \
|
|
||||||
--capability-mode proposal \
|
|
||||||
--proposal-writer project-editor
|
|
||||||
```
|
|
||||||
|
|
||||||
For a tightly controlled application process:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docforge-mcp \
|
|
||||||
--project-root /absolute/path/MyProject \
|
|
||||||
--capability-mode application \
|
|
||||||
--proposal-writer project-editor \
|
|
||||||
--canonical-applier project-editor
|
|
||||||
```
|
|
||||||
|
|
||||||
For a read binding with every active output projection disabled:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docforge-mcp \
|
|
||||||
--project-root /absolute/path/MyProject \
|
|
||||||
--capability-mode read \
|
|
||||||
--manual-render-policy disabled \
|
|
||||||
--portable-graph-policy disabled \
|
|
||||||
--live-viewer-policy disabled
|
|
||||||
```
|
|
||||||
|
|
||||||
Prefer the narrowest binding that completes the workflow. Call `docforge_bootstrap` first and use
|
|
||||||
the returned effective policy, projection policy, actual capabilities, and prohibitions rather
|
|
||||||
than assumptions based on client configuration.
|
|
||||||
|
|
||||||
See the [MCP contract](MCP_CONTRACT.md), [Security](SECURITY.md), [Project
|
|
||||||
descriptor](PROJECT_DESCRIPTOR.md), and [Rendering and
|
|
||||||
visualization](RENDERING_AND_VISUALIZATION.md).
|
|
||||||
|
|
@ -1,279 +0,0 @@
|
||||||
# Project descriptor
|
|
||||||
|
|
||||||
A generic DocForge project is selected by one fixed file:
|
|
||||||
`.docforge/project.toml` beneath an explicit project root. The descriptor is schema version 1.
|
|
||||||
DocForge validates both the JSON-schema shape in `schemas/project.schema.json` and runtime
|
|
||||||
invariants that schema alone cannot prove.
|
|
||||||
|
|
||||||
Reference integrations use a different fixed descriptor,
|
|
||||||
`.docforge/reference-adapter.toml`; see [Reference adapters](REFERENCE_ADAPTERS.md).
|
|
||||||
Project-owned adapters construct the same runtime `ProjectDescriptor` contract through the public
|
|
||||||
adapter SDK.
|
|
||||||
|
|
||||||
## Complete generic example
|
|
||||||
|
|
||||||
```toml
|
|
||||||
schema_version = 1
|
|
||||||
project_id = "my-project"
|
|
||||||
title = "My Project"
|
|
||||||
adapter = "generic"
|
|
||||||
|
|
||||||
[sources]
|
|
||||||
content_roots = ["docs/docforge/content"]
|
|
||||||
authority_files = []
|
|
||||||
|
|
||||||
[derived]
|
|
||||||
cache_root = ".docforge/cache"
|
|
||||||
index = ".docforge/cache/index.sqlite3"
|
|
||||||
|
|
||||||
[changesets]
|
|
||||||
root = ".docforge/changesets"
|
|
||||||
|
|
||||||
[[changesets.writers]]
|
|
||||||
id = "project-editor"
|
|
||||||
families = ["architecture", "operations", "system"]
|
|
||||||
operations = ["create", "update", "move", "delete"]
|
|
||||||
|
|
||||||
[render]
|
|
||||||
template_root = ".docforge/templates"
|
|
||||||
preview_root = ".docforge/previews"
|
|
||||||
|
|
||||||
[[render.views]]
|
|
||||||
id = "manual"
|
|
||||||
renderer = "generic_html"
|
|
||||||
template = "manual.html"
|
|
||||||
output = ".docforge/rendered/manual.html"
|
|
||||||
title = "My Project Manual"
|
|
||||||
families = ["architecture", "operations", "system"]
|
|
||||||
|
|
||||||
[graph_render]
|
|
||||||
output_root = ".docforge/portable-graph"
|
|
||||||
|
|
||||||
[[graph_render.views]]
|
|
||||||
id = "architecture"
|
|
||||||
renderer = "portable_graph_html"
|
|
||||||
output = "architecture.html"
|
|
||||||
title = "Architecture"
|
|
||||||
root = "architecture.overview"
|
|
||||||
initial_mode = "web"
|
|
||||||
depth = 3
|
|
||||||
max_nodes = 250
|
|
||||||
max_edges = 1000
|
|
||||||
max_work = 100000
|
|
||||||
families = ["architecture", "system"]
|
|
||||||
relations = ["depends_on", "owns", "calls", "reads", "writes", "tested_by", "relates_to"]
|
|
||||||
authorities = []
|
|
||||||
statuses = ["current", "active", "verified"]
|
|
||||||
tags = []
|
|
||||||
include_logic = false
|
|
||||||
|
|
||||||
[graph]
|
|
||||||
allowed_relations = [
|
|
||||||
"calls",
|
|
||||||
"depends_on",
|
|
||||||
"owns",
|
|
||||||
"reads",
|
|
||||||
"relates_to",
|
|
||||||
"tested_by",
|
|
||||||
"writes",
|
|
||||||
]
|
|
||||||
|
|
||||||
[limits]
|
|
||||||
max_source_bytes = 500000
|
|
||||||
max_nodes = 10000
|
|
||||||
max_query_chars = 500
|
|
||||||
max_results = 100
|
|
||||||
max_traversal_depth = 6
|
|
||||||
max_context_tokens = 12000
|
|
||||||
max_tool_output_chars = 200000
|
|
||||||
max_changesets = 100
|
|
||||||
max_changeset_operations = 100
|
|
||||||
max_changeset_bytes = 1000000
|
|
||||||
max_render_views = 20
|
|
||||||
max_template_bytes = 1000000
|
|
||||||
max_render_bytes = 1000000
|
|
||||||
|
|
||||||
[[profiles]]
|
|
||||||
id = "development"
|
|
||||||
families = ["architecture", "operations", "system"]
|
|
||||||
statuses = ["current", "active", "verified"]
|
|
||||||
required_nodes = ["architecture.overview"]
|
|
||||||
token_budget = 8000
|
|
||||||
dependency_depth = 3
|
|
||||||
```
|
|
||||||
|
|
||||||
Rendering sections are optional. Profiles and limits may also be omitted; runtime defaults then
|
|
||||||
apply. The required top-level fields are `schema_version`, `project_id`, `title`, `adapter`,
|
|
||||||
`sources`, `derived`, `changesets`, and `graph`.
|
|
||||||
|
|
||||||
## Identity fields
|
|
||||||
|
|
||||||
`schema_version` must be `1`.
|
|
||||||
|
|
||||||
`project_id` is the stable machine identity. It is lowercase and may contain digits, dots,
|
|
||||||
underscores, and hyphens after its first character. Do not derive it from a mutable display title.
|
|
||||||
|
|
||||||
`title` is the human-readable project name.
|
|
||||||
|
|
||||||
`adapter` is `generic` for this file format. Project-owned adapters publish a validated
|
|
||||||
`adapter_id@adapter_version` identity through their runtime descriptor; changing adapter identity
|
|
||||||
invalidates incompatible derived state.
|
|
||||||
|
|
||||||
The descriptor's byte content contributes to a descriptor hash. Client fragments, launchers, index
|
|
||||||
evidence, and policy results use that hash to detect drift.
|
|
||||||
|
|
||||||
## Canonical sources
|
|
||||||
|
|
||||||
`sources.content_roots` lists the project-relative directories containing generic Markdown and TOML
|
|
||||||
nodes. Each path must resolve beneath the project root. Canonical content roots may not overlap the
|
|
||||||
derived cache.
|
|
||||||
|
|
||||||
`sources.authority_files` lists additional project-relative regular files whose content belongs to
|
|
||||||
the canonical project identity. They are not automatically parsed as nodes.
|
|
||||||
|
|
||||||
A Markdown node contains one TOML metadata block followed by Markdown content:
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
+++
|
|
||||||
schema_version = 1
|
|
||||||
id = "architecture.overview"
|
|
||||||
title = "Architecture overview"
|
|
||||||
family = "architecture"
|
|
||||||
authority = "authoritative"
|
|
||||||
status = "current"
|
|
||||||
tags = ["architecture"]
|
|
||||||
summary = "Defines the top-level architecture and ownership."
|
|
||||||
+++
|
|
||||||
|
|
||||||
# Architecture overview
|
|
||||||
|
|
||||||
Describe systems, ownership, runtime flow, failure behavior, and proof.
|
|
||||||
```
|
|
||||||
|
|
||||||
Every node ID is project-wide and stable. Every relationship target must resolve. A TOML source may
|
|
||||||
contain multiple `[[nodes]]` records; proposal-enabled multi-node files need stable
|
|
||||||
`source_anchor` values where creation or movement requires an exact record boundary.
|
|
||||||
|
|
||||||
## Derived state
|
|
||||||
|
|
||||||
`derived.cache_root` owns disposable indexes, attestations, extraction caches, render receipts,
|
|
||||||
projection artifacts, and viewer registry state.
|
|
||||||
|
|
||||||
`derived.index` must be inside `derived.cache_root`. Canonical content and cache paths must not
|
|
||||||
overlap.
|
|
||||||
|
|
||||||
Derived state is not a backup. If it is deleted or rejected as corrupt, DocForge rebuilds it from
|
|
||||||
validated canonical sources.
|
|
||||||
|
|
||||||
## Changesets and writers
|
|
||||||
|
|
||||||
`changesets.root` is the confined proposal store. It must not overlap canonical content or the
|
|
||||||
derived cache.
|
|
||||||
|
|
||||||
Each `changesets.writers` entry declares:
|
|
||||||
|
|
||||||
- a stable writer `id`;
|
|
||||||
- the node `families` that writer may change;
|
|
||||||
- allowed `operations`: `create`, `update`, `move`, and/or `delete`.
|
|
||||||
|
|
||||||
The descriptor grant is necessary but not sufficient. A process must also select that writer at
|
|
||||||
startup, and canonical application requires a separately bound matching applier. Capability mode
|
|
||||||
does not broaden the descriptor grant. See [Policy precedence](POLICY_PRECEDENCE.md).
|
|
||||||
|
|
||||||
## Allowed relationships
|
|
||||||
|
|
||||||
`graph.allowed_relations` is the exact project vocabulary accepted on edges. It must be nonempty.
|
|
||||||
Relationship names are stable IDs. DocForge rejects relationships that are not declared and gives
|
|
||||||
`depends_on` additional cycle validation.
|
|
||||||
|
|
||||||
The core does not reinterpret a custom relationship just because its spelling resembles a known
|
|
||||||
term. Task-context retrieval classifies only the documented versioned aliases and preserves
|
|
||||||
unknown allowed relationships as `unclassified`.
|
|
||||||
|
|
||||||
## Context profiles
|
|
||||||
|
|
||||||
Each `profiles` entry defines one bounded context compilation:
|
|
||||||
|
|
||||||
- `id` selects the profile;
|
|
||||||
- `families` and `statuses` filter eligible nodes;
|
|
||||||
- `required_nodes` names stable nodes that must be present;
|
|
||||||
- `token_budget` limits compiled content;
|
|
||||||
- `dependency_depth` bounds relationship expansion.
|
|
||||||
|
|
||||||
Profiles choose derived retrieval scope. They do not change node authority or writer permissions.
|
|
||||||
|
|
||||||
## Limits
|
|
||||||
|
|
||||||
Positive limits bound input, graph, retrieval, proposal, and rendering work. Current defaults are:
|
|
||||||
|
|
||||||
- `max_source_bytes = 1000000`
|
|
||||||
- `max_nodes = 10000`
|
|
||||||
- `max_query_chars = 500`
|
|
||||||
- `max_results = 100`
|
|
||||||
- `max_traversal_depth = 8`
|
|
||||||
- `max_context_tokens = 32000`
|
|
||||||
- `max_tool_output_chars = 200000`
|
|
||||||
- `max_changesets = 1000`
|
|
||||||
- `max_changeset_operations = 100`
|
|
||||||
- `max_changeset_bytes = 1000000`
|
|
||||||
- `max_render_views = 100`
|
|
||||||
- `max_template_bytes = 1000000`
|
|
||||||
- `max_render_bytes = 1000000`
|
|
||||||
|
|
||||||
Smaller project limits are useful policy. They cannot widen fixed internal worker, package,
|
|
||||||
response, or cache ceilings. In particular, detached renderer transfer has its own fixed boundary
|
|
||||||
even if a compatibility descriptor retains a larger `max_render_bytes`.
|
|
||||||
|
|
||||||
## Manual rendering
|
|
||||||
|
|
||||||
The optional `render` section declares:
|
|
||||||
|
|
||||||
- one confined `template_root`;
|
|
||||||
- one isolated `preview_root`;
|
|
||||||
- one or more stable views.
|
|
||||||
|
|
||||||
Each view uses the built-in `generic_html` renderer, a template beneath `template_root`, one
|
|
||||||
declared output path, a title, and a family filter. Paths may not overlap canonical content,
|
|
||||||
authority files, changesets, cache, templates, or previews in unsafe ways.
|
|
||||||
|
|
||||||
Templates are inert UTF-8 files with a fixed token vocabulary. They cannot select executable
|
|
||||||
renderers or commands. See [Rendering and visualization](RENDERING_AND_VISUALIZATION.md).
|
|
||||||
|
|
||||||
## Portable graph rendering
|
|
||||||
|
|
||||||
The optional `graph_render` section declares an output root and one or more
|
|
||||||
`portable_graph_html` views. Each view selects exactly one:
|
|
||||||
|
|
||||||
- `root`, an exact stable node ID; or
|
|
||||||
- `query`, a bounded metadata-only lexical seed.
|
|
||||||
|
|
||||||
It may then restrict families, relations, authorities, statuses, and tags, plus depth, node, edge,
|
|
||||||
and work limits. `initial_mode` is `nodes`, `flow`, or `web`. Portable graph contract version 1
|
|
||||||
requires `include_logic = false`.
|
|
||||||
|
|
||||||
The declared `output` is relative to `graph_render.output_root`.
|
|
||||||
|
|
||||||
## Paths and confinement
|
|
||||||
|
|
||||||
Descriptor paths are project-relative. Absolute paths and parent traversal are rejected. Runtime
|
|
||||||
validation also rejects symlink escapes, unexpected file types, unsafe overlap, changing path
|
|
||||||
identity during sensitive reads or publication, and derived outputs outside their declared roots.
|
|
||||||
|
|
||||||
The explicit CLI `--project-root` is the only project selector. The MCP process binds it at startup
|
|
||||||
and exposes no project-switching tool.
|
|
||||||
|
|
||||||
## Validate changes safely
|
|
||||||
|
|
||||||
After editing the descriptor:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docforge --project-root /absolute/path/MyProject validate
|
|
||||||
docforge --project-root /absolute/path/MyProject reindex
|
|
||||||
docforge --project-root /absolute/path/MyProject check
|
|
||||||
```
|
|
||||||
|
|
||||||
Descriptor or adapter implementation drift makes a project-owned running process fail closed; start
|
|
||||||
a fresh process after changing those boundaries.
|
|
||||||
|
|
||||||
For first-time creation, prefer the create-only [New-project
|
|
||||||
quickstart](NEW_PROJECT_QUICKSTART.md). For full invariants, read the [Core contract](CONTRACT.md).
|
|
||||||
|
|
@ -5,11 +5,6 @@ DocForge onboarding has two separate outcomes:
|
||||||
1. A generic manual can be configured, indexed, rendered, visualized, and exposed through the MCP.
|
1. A generic manual can be configured, indexed, rendered, visualized, and exposed through the MCP.
|
||||||
2. A source graph additionally requires one validated language frontend per source language.
|
2. A source graph additionally requires one validated language frontend per source language.
|
||||||
|
|
||||||
DocForge ships narrow fixed references for Python, JavaScript, TypeScript, and C++. They are useful
|
|
||||||
for syntax-scoped projects and adoption proof, but production semantic requirements may still
|
|
||||||
require a project-owned compiler or language-service adapter. Review
|
|
||||||
[Reference Adapters](REFERENCE_ADAPTERS.md) before selecting a frontend.
|
|
||||||
|
|
||||||
The onboarding command never claims that source semantics exist merely because it found source
|
The onboarding command never claims that source semantics exist merely because it found source
|
||||||
files. It reports each detected language as `adapter_required` until a project integration supplies
|
files. It reports each detected language as `adapter_required` until a project integration supplies
|
||||||
and proves that frontend.
|
and proves that frontend.
|
||||||
|
|
@ -69,32 +64,6 @@ immediately usable through the generic CLI, viewer, and MCP.
|
||||||
The starter overview records detected languages and states that the source graph is unavailable
|
The starter overview records detected languages and states that the source graph is unavailable
|
||||||
until a language frontend passes the adapter proof. That limitation is deliberate.
|
until a language frontend passes the adapter proof. That limitation is deliberate.
|
||||||
|
|
||||||
## Configure a fixed reference adapter
|
|
||||||
|
|
||||||
The generic onboarding scaffold and fixed reference configuration are separate project routes.
|
|
||||||
For a syntax-scoped reference project, create `.docforge/reference-adapter.toml`:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
schema_version = 1
|
|
||||||
project_id = "my-python-project"
|
|
||||||
title = "My Python project"
|
|
||||||
language = "python"
|
|
||||||
source_roots = ["src"]
|
|
||||||
```
|
|
||||||
|
|
||||||
Then start the fixed read-only binding:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -I -m docforge.reference_mcp \
|
|
||||||
--project-root /absolute/path/MyProject \
|
|
||||||
--capability-mode read
|
|
||||||
```
|
|
||||||
|
|
||||||
The configuration selects only a fixed in-repository provider and cannot name a command or custom
|
|
||||||
module. JavaScript, TypeScript, and C++ require their respective optional extras; C++ also requires
|
|
||||||
`compilation_database = "compile_commands.json"`. The complete configuration and supported-fact
|
|
||||||
contract are in [Reference Adapters](REFERENCE_ADAPTERS.md).
|
|
||||||
|
|
||||||
## Complete onboarding checklist
|
## Complete onboarding checklist
|
||||||
|
|
||||||
### 1. Repository assessment
|
### 1. Repository assessment
|
||||||
|
|
@ -149,13 +118,6 @@ For every source language:
|
||||||
- [ ] Define dependency discovery.
|
- [ ] Define dependency discovery.
|
||||||
- [ ] State unsupported semantic facts explicitly.
|
- [ ] State unsupported semantic facts explicitly.
|
||||||
|
|
||||||
Decide whether the project needs production semantic evidence or the narrower syntax-only
|
|
||||||
reference scope. The Python reference publishes only project-local imports. JavaScript and
|
|
||||||
TypeScript publish only project-local static relative imports and re-exports. The C++ reference
|
|
||||||
publishes only directly resolvable project-local quoted includes and does not run a compiler. None
|
|
||||||
of those references resolves calls, inheritance, types, symbols, runtime behavior, or semantic
|
|
||||||
ownership.
|
|
||||||
|
|
||||||
All frontends emit the same DocForge contracts:
|
All frontends emit the same DocForge contracts:
|
||||||
|
|
||||||
- `AdapterManifest` inventories fingerprinted extraction units and dependencies.
|
- `AdapterManifest` inventories fingerprinted extraction units and dependencies.
|
||||||
|
|
@ -189,11 +151,6 @@ this checklist summarizes.
|
||||||
- [ ] Assign shared symbols to one deterministic source contribution.
|
- [ ] Assign shared symbols to one deterministic source contribution.
|
||||||
- [ ] Record compiler-derived project include dependencies.
|
- [ ] Record compiler-derived project include dependencies.
|
||||||
|
|
||||||
These are production semantic-adapter expectations. The built-in C++ reference uses
|
|
||||||
`compile_commands.json` only as bounded translation-unit inventory and fingerprint evidence. It
|
|
||||||
parses commands and arguments as inert data, executes no command or compiler, and does not claim
|
|
||||||
compiler include semantics, symbol ownership, or a Clang-derived graph.
|
|
||||||
|
|
||||||
#### Rust
|
#### Rust
|
||||||
|
|
||||||
- [ ] Read the Cargo workspace and package graph.
|
- [ ] Read the Cargo workspace and package graph.
|
||||||
|
|
@ -283,10 +240,6 @@ from filenames.
|
||||||
Done when a new session can identify and retrieve the correct project without being told its file
|
Done when a new session can identify and retrieve the correct project without being told its file
|
||||||
layout.
|
layout.
|
||||||
|
|
||||||
The fixed reference server is read-only. Generic and custom-adapter client generation, including
|
|
||||||
the immutable custom launcher boundary, is documented in
|
|
||||||
[Agent Integration](AGENT_INTEGRATION.md).
|
|
||||||
|
|
||||||
### 10. Operating guide and maintenance
|
### 10. Operating guide and maintenance
|
||||||
|
|
||||||
- [ ] Record the authority and progressive-reading order.
|
- [ ] Record the authority and progressive-reading order.
|
||||||
|
|
@ -333,18 +286,14 @@ to one configured project root.
|
||||||
DocForge does not let an MCP call install dependencies, run project builds, modify Git, deploy, or
|
DocForge does not let an MCP call install dependencies, run project builds, modify Git, deploy, or
|
||||||
publish. A project integration may use its own normal development workflow for those actions.
|
publish. A project integration may use its own normal development workflow for those actions.
|
||||||
|
|
||||||
## Reference and production frontend boundaries
|
## Frontend packaging direction
|
||||||
|
|
||||||
The public authoring namespace is `docforge.adapter_sdk`. Project-owned and separately distributed
|
Reusable language frontends should be separate packages or project-owned adapters over the public
|
||||||
production frontends should build on that contract without putting language-specific rules into
|
DocForge contracts. They must not put language-specific rules into the graph, index, viewer, or MCP
|
||||||
the graph, index, viewer, or MCP core.
|
core.
|
||||||
|
|
||||||
The in-repository Python, JavaScript, TypeScript, and C++ adapters are reference implementations.
|
Worldforge is the first complete C++ reference integration. A reusable C++ package should be
|
||||||
Their syntax-scoped behavior is useful without becoming a claim that every project in those
|
extracted only after that integration proves stable ownership, compiler dependency invalidation,
|
||||||
languages has complete semantic coverage. A compiler-backed production C++ adapter may resolve
|
and complete/incremental equivalence. Rust and Java frontends should then implement the same
|
||||||
build flags, calls, types, inheritance, include semantics, and ownership when it can prove those
|
contract using their authoritative build and language tooling rather than copying C++ extraction
|
||||||
facts. The reference C++ adapter does none of that and is not a Clang semantic adapter.
|
rules.
|
||||||
|
|
||||||
New Rust, Java, or other frontends should use their authoritative build and language tooling and
|
|
||||||
must pass the graph-plus-Logic complete/incremental contract in the
|
|
||||||
[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md).
|
|
||||||
|
|
|
||||||
|
|
@ -1,86 +0,0 @@
|
||||||
# Recovery and performance
|
|
||||||
|
|
||||||
DocForge keeps canonical project evidence separate from disposable indexes, caches, receipts, and
|
|
||||||
rendered projections. Recovery rebuilds derived state from current authority; it does not rewrite
|
|
||||||
canonical content to make a cache look valid.
|
|
||||||
|
|
||||||
For the underlying boundaries, see [core authority](CORE_CONCEPTS_AND_AUTHORITY.md) and the
|
|
||||||
[security model](SECURITY.md).
|
|
||||||
|
|
||||||
## Recovery order
|
|
||||||
|
|
||||||
Use the narrowest verified recovery:
|
|
||||||
|
|
||||||
1. Run `docforge check` or the corresponding MCP status operation.
|
|
||||||
2. Run `docforge sync` to repair a missing or stale disposable index when safe.
|
|
||||||
3. Run `docforge reindex` for an explicit complete rebuild.
|
|
||||||
4. Restart a long-running MCP or viewer binding after descriptor or adapter implementation drift.
|
|
||||||
5. Regenerate a derived client fragment, command reference, preview, manual, or portable graph
|
|
||||||
from current project evidence.
|
|
||||||
|
|
||||||
Never delete or rewrite canonical sources, active changesets, or reviewed hashes as cache cleanup.
|
|
||||||
|
|
||||||
## Extraction-cache recovery
|
|
||||||
|
|
||||||
Incremental adapter contributions are stored in a version-1 disposable extraction cache. Reads are
|
|
||||||
no-follow, regular-file-only, identity-checked, bounded to 64,000,000 bytes and 10,000 source
|
|
||||||
records, and fail to a cache miss on malformed or incompatible data. Publication is atomic.
|
|
||||||
|
|
||||||
After a miss, current manifest sources are extracted again and the complete assembly is validated.
|
|
||||||
The active SQLite generation remains authoritative for reads until a new verified index is
|
|
||||||
published. An extraction cache may therefore be safely ahead of the last index; the two files do
|
|
||||||
not pretend to be one transaction.
|
|
||||||
|
|
||||||
## Index and receipt recovery
|
|
||||||
|
|
||||||
A validated SQLite index is a generation-pinned derived snapshot. Missing, corrupt, unattested, or
|
|
||||||
stale indexes rebuild from the complete project or adapter oracle. Generation receipts and render
|
|
||||||
receipts are post-publication evidence. Failure to write a receipt after a committed artifact is
|
|
||||||
reported as degraded committed success, not as permission to repeat a mutation.
|
|
||||||
|
|
||||||
The live viewer pins one validated index identity. Index replacement makes the running snapshot
|
|
||||||
stale and causes a later visualize request to start a fresh worker.
|
|
||||||
|
|
||||||
## Proposal and application recovery
|
|
||||||
|
|
||||||
Hash or base conflicts are not cache failures. Retrieve the current changeset and diff, then
|
|
||||||
review the new exact hash. Rebase is allowed only when every touched node, relationship, source,
|
|
||||||
permission, and graph invariant still matches. A content conflict requires a new proposal.
|
|
||||||
|
|
||||||
If a canonical serializer fails its round-trip check, use its reported rollback state. Do not
|
|
||||||
reapply a changeset whose application may already have committed. See
|
|
||||||
[migrating from v1](MIGRATING_FROM_V1.md) for rollback planning.
|
|
||||||
|
|
||||||
## Milestone 4 scale evidence
|
|
||||||
|
|
||||||
The maintained Python reference benchmark creates 334 Python source files and produces:
|
|
||||||
|
|
||||||
- 1,002 primary nodes and 1,001 primary edges.
|
|
||||||
- 334 Logic projections with 2,338 Logic nodes and 2,338 Logic edges.
|
|
||||||
- Exact complete/incremental primary graph and Logic equality.
|
|
||||||
- Exact output after corrupt extraction-cache recovery and corrupt-index recovery.
|
|
||||||
- Zero `ast.parse` calls and zero `extract_source` calls during a warm build.
|
|
||||||
|
|
||||||
On the frozen Milestone 4 candidate, operation p95 values were 1.266 to 1.898 seconds. Per-operation
|
|
||||||
traced peaks were about 61 to 67 MiB, and process high-water was 78,798,848 bytes. These are
|
|
||||||
regression measurements from one machine, not universal latency promises. The machine-readable
|
|
||||||
record is `../benchmarks/milestone4-2026-07-29.json`; detailed method and hashes are in
|
|
||||||
[the Milestone 4 baseline](MILESTONE_4_BASELINE.md).
|
|
||||||
|
|
||||||
The zero-parser claim is deliberately Python-only. Focused JavaScript and TypeScript tests prove
|
|
||||||
their manifests avoid Tree-sitter. The C++ reference manifest currently uses Tree-sitter while
|
|
||||||
discovering bounded quoted includes, so a warm C++ cache hit is not evidence of zero parser work.
|
|
||||||
|
|
||||||
## Maintained gates
|
|
||||||
|
|
||||||
Run:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
make gate
|
|
||||||
make adoption-m4
|
|
||||||
make benchmark-m4-full
|
|
||||||
make docs-check
|
|
||||||
```
|
|
||||||
|
|
||||||
The main gate includes smoke benchmarks. Full milestone evidence is recorded separately from a
|
|
||||||
clean candidate so smoke or dirty-tree results cannot become release claims.
|
|
||||||
|
|
@ -1,149 +0,0 @@
|
||||||
# Reference adapters
|
|
||||||
|
|
||||||
DocForge includes fixed, syntax-scoped reference adapters for Python, JavaScript, TypeScript, and
|
|
||||||
C++. They demonstrate the public adapter contract, deterministic complete and incremental
|
|
||||||
publication, function Logic, cache recovery, and a runnable read-only MCP binding. They are not
|
|
||||||
compiler or language-service replacements.
|
|
||||||
|
|
||||||
Use a reference adapter when its deliberately narrow graph is sufficient or when proving a fresh
|
|
||||||
DocForge integration. Use the [Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md) for a
|
|
||||||
production adapter that needs resolved symbols, calls, inheritance, types, build semantics, or
|
|
||||||
semantic ownership.
|
|
||||||
|
|
||||||
## Installation
|
|
||||||
|
|
||||||
The Python reference adapter uses the standard library and is available in the base wheel. The
|
|
||||||
other frontends are separate optional extras:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m pip install docforge
|
|
||||||
python -m pip install "docforge[javascript]"
|
|
||||||
python -m pip install "docforge[typescript]"
|
|
||||||
python -m pip install "docforge[cpp]"
|
|
||||||
```
|
|
||||||
|
|
||||||
Install `docforge[languages]` only when one environment intentionally needs all three Tree-sitter
|
|
||||||
frontends. JavaScript and TypeScript use distinct grammar packages and distinct extras.
|
|
||||||
|
|
||||||
## Fixed project configuration
|
|
||||||
|
|
||||||
The runnable reference binding reads exactly
|
|
||||||
`.docforge/reference-adapter.toml` below the selected project root. The configuration is data only:
|
|
||||||
it cannot name a provider, Python module, command, arguments, working directory, environment, or
|
|
||||||
discovery rule.
|
|
||||||
|
|
||||||
Reference adapter configuration `.docforge/reference-adapter.toml`:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
schema_version = 1
|
|
||||||
project_id = "example-python"
|
|
||||||
title = "Example Python project"
|
|
||||||
language = "python"
|
|
||||||
source_roots = ["src"]
|
|
||||||
```
|
|
||||||
|
|
||||||
`language` is one of `python`, `javascript`, `typescript`, or `cpp`. Source roots must be sorted,
|
|
||||||
unique, non-overlapping project-relative directories outside `.docforge`. The configuration,
|
|
||||||
roots, and inventoried source files must be regular confined paths without symlink traversal. The
|
|
||||||
configuration is limited to 65,536 bytes, at most 64 source roots, and 65,536 examined inventory
|
|
||||||
entries.
|
|
||||||
|
|
||||||
C++ additionally requires one explicit project-confined compilation database:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
schema_version = 1
|
|
||||||
project_id = "example-cpp"
|
|
||||||
title = "Example C++ project"
|
|
||||||
language = "cpp"
|
|
||||||
source_roots = ["include", "src"]
|
|
||||||
compilation_database = "compile_commands.json"
|
|
||||||
```
|
|
||||||
|
|
||||||
`compilation_database` is forbidden for the other three languages.
|
|
||||||
|
|
||||||
## Published scope
|
|
||||||
|
|
||||||
All four adapters publish deterministic file, module or translation-unit, class or struct, and
|
|
||||||
function nodes where their syntax supports those categories. They publish lexical `contains`
|
|
||||||
relationships, a narrow set of project-local `depends_on` relationships, and function-scoped
|
|
||||||
Logic. Calls that appear inside Logic are syntax steps, not resolved symbol relationships.
|
|
||||||
|
|
||||||
### Python
|
|
||||||
|
|
||||||
Python uses `ast` from the standard library for source extraction and Logic. Manifest construction
|
|
||||||
does not build a Python AST; it tokenizes imports only far enough to publish dependencies that
|
|
||||||
resolve to another module in the declared source inventory.
|
|
||||||
|
|
||||||
It does not import or execute project code. It does not resolve dynamic imports, calls,
|
|
||||||
inheritance, imported symbols, types, overloads, re-exports, decorators, metaclasses, descriptors,
|
|
||||||
or runtime-generated behavior.
|
|
||||||
|
|
||||||
### JavaScript and TypeScript
|
|
||||||
|
|
||||||
JavaScript uses the optional `tree-sitter-javascript` grammar. TypeScript uses the distinct
|
|
||||||
optional `tree-sitter-typescript` grammar. Their focused incremental tests prove that manifest
|
|
||||||
construction and an unchanged warm build do not invoke the Tree-sitter extraction parser.
|
|
||||||
|
|
||||||
Only static relative imports and re-exports that resolve to another inventoried source file become
|
|
||||||
dependencies. Dynamic `import()`, `require()`, bare package specifiers, aliases, `tsconfig` paths,
|
|
||||||
loader hooks, types, interfaces, overloads, calls, inheritance, symbols, and runtime behavior are
|
|
||||||
not resolved. The adapters never import, compile, transpile, or execute project code.
|
|
||||||
|
|
||||||
### C++
|
|
||||||
|
|
||||||
C++ treats `compile_commands.json` as the authoritative bounded translation-unit inventory and as
|
|
||||||
fingerprint evidence. Commands, arguments, directories, and output fields are parsed as inert
|
|
||||||
data. The adapter executes no compiler, build tool, command, project binary, or project code.
|
|
||||||
|
|
||||||
The reference C++ manifest uses the optional `tree-sitter-cpp` grammar to parse inventoried sources
|
|
||||||
and discover quoted includes. It publishes a dependency only when that quoted include resolves
|
|
||||||
directly to a real project-local header in the declared roots. Angle-bracket includes, compiler
|
|
||||||
include paths, frameworks, generated headers, compiler-provided headers, conditional compilation,
|
|
||||||
and macro expansion are omitted.
|
|
||||||
|
|
||||||
This is not a Clang semantic adapter. It does not claim resolved calls, types, templates, aliases,
|
|
||||||
concepts, references, inheritance, overload ownership, out-of-line semantic ownership, macro
|
|
||||||
semantics, or compiler include semantics. A warm C++ extraction-cache hit is therefore not evidence
|
|
||||||
that manifest construction performed zero parser work.
|
|
||||||
|
|
||||||
## Complete and incremental proof
|
|
||||||
|
|
||||||
Each reference adapter implements:
|
|
||||||
|
|
||||||
- `load_projection()` for the complete primary graph compatibility oracle;
|
|
||||||
- `load_complete_assembly()` for the complete graph-plus-Logic oracle;
|
|
||||||
- `load_manifest()` and `extract_source()` for incremental extraction; and
|
|
||||||
- deterministic assembly of the complete current contribution set.
|
|
||||||
|
|
||||||
The maintained fixtures prove complete determinism, exact complete/incremental graph and Logic
|
|
||||||
parity, reverse-dependency invalidation, additions and deletions, corrupt extraction-cache
|
|
||||||
recovery, confinement, and exact unsupported-fact inventories. See
|
|
||||||
[Incremental Adapter Indexing](INCREMENTAL_INDEXING.md) for the cache contract and
|
|
||||||
[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for compatibility and policy limits.
|
|
||||||
|
|
||||||
Current fixture evidence is:
|
|
||||||
|
|
||||||
| Language | Primary nodes | Relationships | Logic projections |
|
|
||||||
|---|---:|---:|---:|
|
|
||||||
| Python | 14 | 13 | 5 |
|
|
||||||
| JavaScript | 14 | 14 | 5 |
|
|
||||||
| TypeScript | 13 | 14 | 4 |
|
|
||||||
| C++ | 17 | 16 | 5 |
|
|
||||||
|
|
||||||
These are regression-fixture shapes, not promises for arbitrary repositories.
|
|
||||||
|
|
||||||
## Read-only reference server
|
|
||||||
|
|
||||||
Start the fixed server with:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -I -m docforge.reference_mcp \
|
|
||||||
--project-root /absolute/path/to/project \
|
|
||||||
--capability-mode read
|
|
||||||
```
|
|
||||||
|
|
||||||
The server selects one of the four in-repository providers solely from the validated fixed
|
|
||||||
configuration. Its cache stays below `.docforge/cache/reference-adapter/<language>`. It registers
|
|
||||||
the 21-tool read surface and no proposal or application tools. See
|
|
||||||
[Agent Integration](AGENT_INTEGRATION.md) for generated client fragments and
|
|
||||||
[MCP Boundary](MCP_CONTRACT.md) for the exact tool contract.
|
|
||||||
|
|
@ -1,233 +0,0 @@
|
||||||
# Rendering and visualization
|
|
||||||
|
|
||||||
DocForge has three independent ways to present one validated graph generation:
|
|
||||||
|
|
||||||
```text
|
|
||||||
validated generation
|
|
||||||
├── manual plan → immutable package → detached HTML renderer → declared manual output
|
|
||||||
├── graph plan → immutable package → detached graph renderer → portable static artifact
|
|
||||||
└── pinned index → managed loopback viewer → interactive Nodes, Flow, Web, and lazy Logic
|
|
||||||
```
|
|
||||||
|
|
||||||
They share validated facts but not authority, publication, or lifecycle. A manual render does not
|
|
||||||
publish a portable graph. A portable graph does not start the live viewer. None is canonical
|
|
||||||
project content.
|
|
||||||
|
|
||||||
## Manual rendering
|
|
||||||
|
|
||||||
A generic descriptor may declare a manual template root, isolated preview root, and stable views:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
[render]
|
|
||||||
template_root = ".docforge/templates"
|
|
||||||
preview_root = ".docforge/previews"
|
|
||||||
|
|
||||||
[[render.views]]
|
|
||||||
id = "manual"
|
|
||||||
renderer = "generic_html"
|
|
||||||
template = "manual.html"
|
|
||||||
output = ".docforge/rendered/manual.html"
|
|
||||||
title = "My Project Manual"
|
|
||||||
families = ["architecture", "operations", "system"]
|
|
||||||
```
|
|
||||||
|
|
||||||
The renderer name is fixed to `generic_html`. Templates are confined UTF-8 assets with a fixed
|
|
||||||
token vocabulary. They cannot name commands, Python modules, executable renderers, or arbitrary
|
|
||||||
publication paths. Raw HTML in canonical Markdown is disabled by the pinned CommonMark path.
|
|
||||||
|
|
||||||
Plan and render identity covers the canonical source hash, optional changeset hash, selected nodes
|
|
||||||
and relationships, view configuration, template hash, renderer contract, and parser version.
|
|
||||||
|
|
||||||
Use:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docforge --project-root "$PROJECT" render-status
|
|
||||||
docforge --project-root "$PROJECT" render-status manual --deep
|
|
||||||
docforge --project-root "$PROJECT" render manual
|
|
||||||
docforge --project-root "$PROJECT" preview CHANGESET_ID manual
|
|
||||||
```
|
|
||||||
|
|
||||||
Normal status verifies bounded source, configuration, template, output, renderer, and receipt
|
|
||||||
identities without reconstructing output. `--deep` explicitly runs the side-effect-free full-render
|
|
||||||
oracle. A changeset preview writes only to the isolated preview root.
|
|
||||||
|
|
||||||
Manual `auto` means that a successful canonical application owns regeneration of every declared
|
|
||||||
manual view. It does not mean background rendering, and ordinary standalone CLI render cannot
|
|
||||||
select `auto`; use `explicit`.
|
|
||||||
|
|
||||||
## Portable graph rendering
|
|
||||||
|
|
||||||
Portable graph configuration is separate:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
[graph_render]
|
|
||||||
output_root = ".docforge/portable-graph"
|
|
||||||
|
|
||||||
[[graph_render.views]]
|
|
||||||
id = "architecture"
|
|
||||||
renderer = "portable_graph_html"
|
|
||||||
output = "architecture.html"
|
|
||||||
title = "Architecture"
|
|
||||||
root = "architecture.overview"
|
|
||||||
initial_mode = "web"
|
|
||||||
depth = 3
|
|
||||||
max_nodes = 250
|
|
||||||
max_edges = 1000
|
|
||||||
max_work = 100000
|
|
||||||
families = ["architecture", "system"]
|
|
||||||
relations = ["depends_on", "owns", "calls", "reads", "writes", "tested_by", "relates_to"]
|
|
||||||
authorities = []
|
|
||||||
statuses = ["current", "active", "verified"]
|
|
||||||
tags = []
|
|
||||||
include_logic = false
|
|
||||||
```
|
|
||||||
|
|
||||||
A view selects exactly one exact `root` node or bounded metadata-only lexical `query`. Exact
|
|
||||||
filters and node, edge, depth, and work limits close the selection. The initial mode is `nodes`,
|
|
||||||
`flow`, or `web`. Portable graph contract version 1 excludes function-scoped Logic.
|
|
||||||
|
|
||||||
Use:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docforge --project-root "$PROJECT" graph-plan architecture
|
|
||||||
docforge --project-root "$PROJECT" graph-render architecture
|
|
||||||
docforge --project-root "$PROJECT" graph-render-status architecture
|
|
||||||
```
|
|
||||||
|
|
||||||
`graph-plan` validates and returns the generation-pinned plan without publication. `graph-render`
|
|
||||||
is an explicit local CLI publication action. It commits a content-addressed artifact, renderer
|
|
||||||
receipt, and bounded generation/view manifest in that order; the manifest is the publication
|
|
||||||
commit. `graph-render-status` checks only bounded committed evidence and never plans or renders.
|
|
||||||
|
|
||||||
MCP may plan and inspect portable graph evidence, but it does not publish the portable artifact.
|
|
||||||
Use the explicit project-local CLI command for publication.
|
|
||||||
|
|
||||||
Portable output is a complete static artifact. JavaScript is progressive enhancement, not a
|
|
||||||
requirement for the graph facts to be present.
|
|
||||||
|
|
||||||
## Live viewer
|
|
||||||
|
|
||||||
The live viewer is a managed, read-only browser over one generation-pinned validated SQLite file.
|
|
||||||
Install the per-user manager once:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docforge-viewer-manager install-user-service
|
|
||||||
```
|
|
||||||
|
|
||||||
Then operate a project viewer:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docforge --project-root "$PROJECT" visualize
|
|
||||||
docforge --project-root "$PROJECT" visualize --node architecture.overview
|
|
||||||
docforge --project-root "$PROJECT" visualize --query persistence
|
|
||||||
docforge --project-root "$PROJECT" visualization-status
|
|
||||||
docforge --project-root "$PROJECT" visualization-stop
|
|
||||||
```
|
|
||||||
|
|
||||||
`visualize` accepts only an optional stable node ID or lexical query, bounded traversal depth, and
|
|
||||||
the local `--no-open` presentation choice. It does not accept a project root override, database
|
|
||||||
path, SQL, template, command, renderer, bind address, or module.
|
|
||||||
|
|
||||||
The HTTP listener binds to `127.0.0.1` on an operating-system-selected port. A random token is part
|
|
||||||
of every accepted path. The server supports only `GET` and `HEAD`, sets no-store and restrictive
|
|
||||||
browser security headers, and has no write, project-selection, arbitrary-query, or static
|
|
||||||
filesystem endpoint.
|
|
||||||
|
|
||||||
The viewer is a snapshot. Index replacement or alteration makes that snapshot fail closed; start a
|
|
||||||
new visualization to use a new validated generation. Source inspection reads only project-confined
|
|
||||||
source evidence bound to the pinned generation.
|
|
||||||
|
|
||||||
## Nodes, Flow, Web, and Logic
|
|
||||||
|
|
||||||
The interactive viewer offers four complementary projections:
|
|
||||||
|
|
||||||
- **Nodes** shows a bounded relation-neutral incoming and outgoing neighborhood.
|
|
||||||
- **Flow** presents semantic contributors toward the focus. Prerequisite-style stored
|
|
||||||
relationships may be reversed for presentation without changing stored direction.
|
|
||||||
- **Web** expands the convergence picture with contributors, callers, containers, members, and
|
|
||||||
contextual relationships.
|
|
||||||
- **Logic** loads a function-scoped control-flow projection only when requested.
|
|
||||||
|
|
||||||
Logic supports the explicit control paths published by Python, JavaScript, TypeScript, and C++
|
|
||||||
integrations. It stays outside the primary graph, portable graph version 1, search, and
|
|
||||||
generation-diff receipts.
|
|
||||||
|
|
||||||
Hiding a node is presentation-only. In Flow and Web, ancestors without another path to the focus
|
|
||||||
are removed. In Logic, an omitted-path bridge preserves downstream readability. Restore reverses
|
|
||||||
the presentation change; neither action mutates the index.
|
|
||||||
|
|
||||||
## Immutable plan and worker boundary
|
|
||||||
|
|
||||||
Manual and graph plans are versioned, canonical JSON with deterministic ordering and fixed
|
|
||||||
structural and byte limits. They contain selected graph facts and bounded content, but no live
|
|
||||||
project object, SQLite handle, absolute project or index path, arbitrary query, command,
|
|
||||||
executable path, or caller-selected renderer module.
|
|
||||||
|
|
||||||
A projection package binds one plan to inert assets, fixed component versions, a closed built-in
|
|
||||||
renderer identity, and an exact artifact inventory. The detached worker:
|
|
||||||
|
|
||||||
- runs one fixed private module through isolated Python;
|
|
||||||
- uses a trusted working directory and sanitized environment;
|
|
||||||
- accepts one canonical newline-terminated JSON request;
|
|
||||||
- returns one bounded canonical JSON response;
|
|
||||||
- has a fixed renderer allowlist and timeout;
|
|
||||||
- cannot select graph facts, read project state, choose output paths, or mutate canonical files.
|
|
||||||
|
|
||||||
The package contract is bounded, and actual artifact transfer has a fixed 20,000,000-byte ceiling.
|
|
||||||
A larger descriptor `max_render_bytes` compatibility value does not widen that worker boundary.
|
|
||||||
|
|
||||||
## Manual fragment reuse
|
|
||||||
|
|
||||||
Manual fragments are disposable semantic cache records, not publication authority. A cold record
|
|
||||||
is accepted only after byte-exact comparison against a full detached render. On a warm hit, the
|
|
||||||
worker independently recomputes the expected fragment before reuse.
|
|
||||||
|
|
||||||
Corrupt, forged, stale, incompatible, individually oversized, or aggregate-oversized records fall
|
|
||||||
back to the complete full-render oracle. The current cache is bounded to 10,000 records and
|
|
||||||
64,000,000 bytes.
|
|
||||||
|
|
||||||
Fragment reuse is a correctness and recovery boundary. Do not promise a speedup without current
|
|
||||||
measurements.
|
|
||||||
|
|
||||||
## Projection policy
|
|
||||||
|
|
||||||
The version-2 projection policy is independent of capability mode:
|
|
||||||
|
|
||||||
```text
|
|
||||||
manual: auto | explicit | disabled
|
|
||||||
portable_graph: explicit | disabled
|
|
||||||
live_viewer: on-demand | disabled
|
|
||||||
```
|
|
||||||
|
|
||||||
Set global CLI flags before the subcommand:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docforge --project-root "$PROJECT" --manual-render-policy disabled render manual
|
|
||||||
docforge --project-root "$PROJECT" --portable-graph-policy disabled graph-plan architecture
|
|
||||||
docforge --project-root "$PROJECT" --live-viewer-policy disabled visualize
|
|
||||||
```
|
|
||||||
|
|
||||||
An active operation blocked by policy fails before hidden work. Receipt-only manual and portable
|
|
||||||
status remain available. Viewer status and explicit stop remain available when viewer startup is
|
|
||||||
disabled.
|
|
||||||
|
|
||||||
A non-disabled selection also requires its resource: declared manual configuration, declared
|
|
||||||
portable graph configuration, or viewer runtime. Manual `auto` additionally requires canonical
|
|
||||||
application. Read [Policy precedence](POLICY_PRECEDENCE.md) for exact composition.
|
|
||||||
|
|
||||||
## Failure and recovery
|
|
||||||
|
|
||||||
Render input drift detected before replacement fails without publishing a current receipt for stale
|
|
||||||
output. Portable graph publication uses content-addressed evidence and a final manifest commit.
|
|
||||||
Status never repairs implicitly.
|
|
||||||
|
|
||||||
Recovery is explicit:
|
|
||||||
|
|
||||||
- rerun a manual render from validated canonical state;
|
|
||||||
- use deep manual status for the full, side-effect-free equivalence oracle;
|
|
||||||
- rerun portable graph publication, or repair only from validated content-addressed evidence;
|
|
||||||
- stop and restart a stale live viewer against a current validated index.
|
|
||||||
|
|
||||||
Do not recover a derived output by editing its receipt or treating it as canonical. See [Recovery
|
|
||||||
and performance](RECOVERY_AND_PERFORMANCE.md), [Security](SECURITY.md), and the [Viewer
|
|
||||||
manager](VIEWER_MANAGER.md).
|
|
||||||
101
docs/SECURITY.md
101
docs/SECURITY.md
|
|
@ -1,101 +0,0 @@
|
||||||
# Security model
|
|
||||||
|
|
||||||
DocForge is a project-bound knowledge compiler. Its security boundary is an explicit project root,
|
|
||||||
closed configuration, bounded data, and exact identities. It is not a general process sandbox.
|
|
||||||
|
|
||||||
Start with [core authority](CORE_CONCEPTS_AND_AUTHORITY.md), then use
|
|
||||||
[policy precedence](POLICY_PRECEDENCE.md) to decide which capabilities a server should expose.
|
|
||||||
|
|
||||||
## Project and path confinement
|
|
||||||
|
|
||||||
Descriptors, reference-adapter configurations, canonical sources, authority files, templates,
|
|
||||||
changesets, caches, indexes, previews, and declared outputs are resolved against one project root.
|
|
||||||
DocForge rejects absolute paths where only project-relative paths are allowed, parent traversal,
|
|
||||||
symbolic-link escapes, unsafe file types, and protected-root overlap. Important reads use
|
|
||||||
no-follow file descriptors and compare file identity before and after reading.
|
|
||||||
|
|
||||||
Confinement protects DocForge operations. It does not stop another process with repository access
|
|
||||||
from changing files. Long-running bindings revalidate descriptor and adapter implementation
|
|
||||||
identity and require a restart after drift.
|
|
||||||
|
|
||||||
## Untrusted project content
|
|
||||||
|
|
||||||
Documentation, source text, templates, adapter metadata, compiler-database entries, and changeset
|
|
||||||
content are data. They cannot redefine policy or instruct DocForge to execute a command. Generic
|
|
||||||
manual rendering disables raw HTML, accepts a fixed template vocabulary, and rejects script-like
|
|
||||||
content. Portable graph and manual workers accept validated inert packages and fixed built-in
|
|
||||||
renderer identities.
|
|
||||||
|
|
||||||
The C++ reference adapter reads `compile_commands.json` only as bounded translation-unit inventory
|
|
||||||
and fingerprint evidence. It never executes the recorded command, compiler, response file, or
|
|
||||||
project program.
|
|
||||||
|
|
||||||
## Adapter launcher boundary
|
|
||||||
|
|
||||||
`AdapterLauncherV1` contains one Python module name and project identity. It contains no command,
|
|
||||||
shell string, arbitrary argument list, working directory, environment, discovery rule, or callable
|
|
||||||
selector.
|
|
||||||
|
|
||||||
Custom modules must be installed top-level modules. An isolated `python -I` probe resolves the
|
|
||||||
module without importing it and requires its regular-file origin to remain inside the bound
|
|
||||||
project. The sole trusted dotted exception is the packaged `docforge.reference_mcp` module. Client
|
|
||||||
fragments use the exact validated interpreter and canonical fixed arguments with an empty
|
|
||||||
environment.
|
|
||||||
|
|
||||||
This proves that the declared module is resolvable and project-bound. It does not make arbitrary
|
|
||||||
module code safe. Project owners remain responsible for the implementation they install.
|
|
||||||
|
|
||||||
## Mutation boundary
|
|
||||||
|
|
||||||
Normal MCP and the fixed reference server are read-only. Proposal tools exist only when a
|
|
||||||
startup-bound writer is authorized by the descriptor. Canonical application exists only when a
|
|
||||||
matching applier is explicitly configured.
|
|
||||||
|
|
||||||
Every proposal append, rebase, abandonment, and application is hash-bound. Application requires
|
|
||||||
the exact changeset hash that was reviewed. Source identity, content hashes, permissions,
|
|
||||||
conflicts, graph validity, and serializer round trips are checked before success. DocForge never
|
|
||||||
turns prose approval into a fuzzy merge.
|
|
||||||
|
|
||||||
## Derived state and publication
|
|
||||||
|
|
||||||
SQLite indexes, source-generation receipts, extraction caches, render fragments, previews, and
|
|
||||||
portable artifacts are disposable. Corrupt, stale, foreign, oversized, or mismatched derived
|
|
||||||
state is rejected or rebuilt from current project evidence.
|
|
||||||
|
|
||||||
Generated command-reference publication serializes cooperating writers and uses no-clobber or
|
|
||||||
compare-and-swap publication. A raced target is restored or retained for recovery instead of being
|
|
||||||
silently discarded. Projection publication records when an artifact was committed but later
|
|
||||||
receipt verification degraded, so a completed mutation is never reported as an ordinary failure.
|
|
||||||
|
|
||||||
## Limits and denial-of-service resistance
|
|
||||||
|
|
||||||
Inputs, results, traversal, context, changesets, renders, worker protocols, manifests, adapter
|
|
||||||
assemblies, and extraction caches have explicit count and byte limits. Incremental extraction
|
|
||||||
caches are capped at 10,000 sources and 64,000,000 bytes. Adapter primary nodes use the project
|
|
||||||
`max_nodes` limit; edges and Logic have deterministic multipliers over that limit.
|
|
||||||
|
|
||||||
Limits reduce accidental and adversarial amplification. An in-process adapter can still allocate
|
|
||||||
memory before returning data, so only trusted project-owned adapter code should run in the server
|
|
||||||
process.
|
|
||||||
|
|
||||||
## Secrets and network behavior
|
|
||||||
|
|
||||||
DocForge does not copy the parent environment into generated client fragments or detached
|
|
||||||
projection workers. Doctor checks never return environment values. The live viewer binds to
|
|
||||||
loopback, uses an unguessable URL token, supports read-only methods, and serves no arbitrary
|
|
||||||
filesystem tree.
|
|
||||||
|
|
||||||
Project secrets must not be placed in canonical documentation, adapter configuration, compiler
|
|
||||||
databases, templates, or changesets. Repository release gates include secret scanning, but that
|
|
||||||
scan is not a substitute for credential hygiene.
|
|
||||||
|
|
||||||
## No-AST boundary
|
|
||||||
|
|
||||||
`--no-ast` is a binding policy that preserves the selected adapter and prohibits Logic publication
|
|
||||||
and retrieval. It is not a parser detector, filesystem sandbox, or promise that unrelated
|
|
||||||
processes cannot parse source. See [legacy and no-AST operation](LEGACY_AND_NO_AST.md).
|
|
||||||
|
|
||||||
## Reporting and recovery
|
|
||||||
|
|
||||||
Do not bypass a confinement, identity, policy, hash, or limit error. Preserve the failing evidence,
|
|
||||||
stop the affected binding, and follow [recovery and performance](RECOVERY_AND_PERFORMANCE.md).
|
|
||||||
|
|
@ -35,8 +35,6 @@ incremental methods while retaining the full loader as a fallback.
|
||||||
relationship keys, source inspection, branch-aware node hiding, panel resizing, zooming, and
|
relationship keys, source inspection, branch-aware node hiding, panel resizing, zooming, and
|
||||||
managed idle shutdown.
|
managed idle shutdown.
|
||||||
- A generic Markdown/TOML adapter plus contracts for deterministic project-owned adapters.
|
- A generic Markdown/TOML adapter plus contracts for deterministic project-owned adapters.
|
||||||
- A public adapter SDK, optional Python/JavaScript/TypeScript/C++ reference integrations, and one
|
|
||||||
fixed read-only reference MCP binding.
|
|
||||||
|
|
||||||
DocForge does not run shell commands from documentation, mutate Git, build an application, deploy,
|
DocForge does not run shell commands from documentation, mutate Git, build an application, deploy,
|
||||||
publish, choose a project globally, or cross project boundaries.
|
publish, choose a project globally, or cross project boundaries.
|
||||||
|
|
@ -102,49 +100,6 @@ uv run pytest -q
|
||||||
Use the executables under `/absolute/path/DocForge/.venv/bin/` when DocForge is not installed into
|
Use the executables under `/absolute/path/DocForge/.venv/bin/` when DocForge is not installed into
|
||||||
the active shell environment.
|
the active shell environment.
|
||||||
|
|
||||||
For an installed distribution, choose only the language extras the project needs:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m pip install docforge
|
|
||||||
python -m pip install 'docforge[javascript]'
|
|
||||||
python -m pip install 'docforge[typescript]'
|
|
||||||
python -m pip install 'docforge[cpp]'
|
|
||||||
```
|
|
||||||
|
|
||||||
The base wheel contains the Python reference adapter and no Tree-sitter distribution. JavaScript,
|
|
||||||
TypeScript, and C++ require their matching optional extras. `docforge[languages]` installs all
|
|
||||||
three optional frontend groups.
|
|
||||||
|
|
||||||
### Configure a reference source project
|
|
||||||
|
|
||||||
Reference adapters are a narrow alternative to the generic documentation descriptor. Create
|
|
||||||
`.docforge/reference-adapter.toml`:
|
|
||||||
|
|
||||||
```toml reference-adapter
|
|
||||||
schema_version = 1
|
|
||||||
project_id = "my-python-project"
|
|
||||||
title = "My Python Project"
|
|
||||||
language = "python"
|
|
||||||
source_roots = ["src"]
|
|
||||||
```
|
|
||||||
|
|
||||||
Start the fixed read-only server:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -I -m docforge.reference_mcp \
|
|
||||||
--project-root /absolute/path/MyProject \
|
|
||||||
--capability-mode read
|
|
||||||
```
|
|
||||||
|
|
||||||
For C++, set `language = "cpp"` and add a project-relative
|
|
||||||
`compilation_database = "compile_commands.json"`. The database is inert bounded inventory; the
|
|
||||||
reference adapter does not execute its commands or compiler.
|
|
||||||
|
|
||||||
The reference integrations publish syntax and local static relationships only. They do not claim
|
|
||||||
resolved calls, types, inheritance, macro behavior, compiler include semantics, runtime behavior,
|
|
||||||
or semantic ownership. See [reference adapters](REFERENCE_ADAPTERS.md) for exact evidence and
|
|
||||||
limitations.
|
|
||||||
|
|
||||||
### Assess and onboard an unconfigured project
|
### Assess and onboard an unconfigured project
|
||||||
|
|
||||||
Run a read-only assessment before writing configuration:
|
Run a read-only assessment before writing configuration:
|
||||||
|
|
@ -523,10 +478,6 @@ Every command emits deterministic JSON:
|
||||||
docforge --project-root /absolute/path/MyProject <command>
|
docforge --project-root /absolute/path/MyProject <command>
|
||||||
```
|
```
|
||||||
|
|
||||||
The sections below group common workflows. The implementation-derived list of all 28 current
|
|
||||||
commands, exact invocations, 36 generic MCP tools, arguments, and input-schema hashes is the
|
|
||||||
[generated command reference](COMMAND_REFERENCE.md).
|
|
||||||
|
|
||||||
### Project and index commands
|
### Project and index commands
|
||||||
|
|
||||||
```text
|
```text
|
||||||
|
|
@ -639,31 +590,6 @@ declares the named writer, and application requires the same writer/applier iden
|
||||||
`--no-ast` to preserve the no-AST binding. Generic CLI generation refuses project-owned adapters
|
`--no-ast` to preserve the no-AST binding. Generic CLI generation refuses project-owned adapters
|
||||||
because it cannot safely reconstruct their composition.
|
because it cannot safely reconstruct their composition.
|
||||||
|
|
||||||
A custom adapter owner supplies the already constructed project and immutable launcher through the
|
|
||||||
Python API:
|
|
||||||
|
|
||||||
```python
|
|
||||||
from docforge.adapter_launcher import AdapterLauncherV1
|
|
||||||
from docforge.client_config import generate_adapter_client_configuration
|
|
||||||
|
|
||||||
launcher = AdapterLauncherV1.for_project(
|
|
||||||
project,
|
|
||||||
module="my_project_docforge",
|
|
||||||
)
|
|
||||||
fragment = generate_adapter_client_configuration(
|
|
||||||
project,
|
|
||||||
launcher,
|
|
||||||
"codex",
|
|
||||||
capability_mode="read",
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
The top-level module must be installed for the exact isolated Python environment and resolve to a
|
|
||||||
regular file inside the project root. The fixed `docforge.reference_mcp` module is the only trusted
|
|
||||||
dotted exception. Generation probes resolution without importing the custom module, binds current
|
|
||||||
source availability and policy, and emits no arbitrary command, arguments, working directory, or
|
|
||||||
environment.
|
|
||||||
|
|
||||||
Select projection behavior independently:
|
Select projection behavior independently:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|
@ -758,10 +684,6 @@ docforge-mcp \
|
||||||
|
|
||||||
Omit `--proposal-writer` when the MCP client should not create or append proposals.
|
Omit `--proposal-writer` when the MCP client should not create or append proposals.
|
||||||
|
|
||||||
For `.docforge/reference-adapter.toml`, use the fixed `docforge.reference_mcp` command shown in
|
|
||||||
[setup](#configure-a-reference-source-project). It exposes exactly the 21 read tools and never
|
|
||||||
registers proposal or application tools.
|
|
||||||
|
|
||||||
Select the session's declared surface explicitly when useful:
|
Select the session's declared surface explicitly when useful:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|
@ -1064,28 +986,6 @@ changes outside this process boundary.
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
### `optional_dependency_missing`
|
|
||||||
|
|
||||||
Install the exact extra named in the error into the same Python environment that starts DocForge:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python -m pip install 'docforge[javascript]'
|
|
||||||
python -m pip install 'docforge[typescript]'
|
|
||||||
python -m pip install 'docforge[cpp]'
|
|
||||||
```
|
|
||||||
|
|
||||||
Do not install every frontend merely to suppress the check. A missing optional parser is a closed,
|
|
||||||
actionable capability error and does not affect base generic or Python reference operation.
|
|
||||||
|
|
||||||
### `adapter_launcher_unavailable` or `invalid_adapter_launcher`
|
|
||||||
|
|
||||||
Use one installed top-level Python module whose resolved regular-file origin is inside the project
|
|
||||||
root, or use the fixed `docforge.reference_mcp` binding. Arbitrary dotted modules, packages,
|
|
||||||
stdlib modules, missing modules, commands, argument strings, working directories, and environment
|
|
||||||
injection are rejected. Test the exact generated fragment rather than editing its command by hand.
|
|
||||||
|
|
||||||
See [agent integration](AGENT_INTEGRATION.md) and the [security model](SECURITY.md).
|
|
||||||
|
|
||||||
### `adapter_restart_required`
|
### `adapter_restart_required`
|
||||||
|
|
||||||
The project-local adapter code, its declared descriptor, or another implementation file changed
|
The project-local adapter code, its declared descriptor, or another implementation file changed
|
||||||
|
|
@ -1228,11 +1128,8 @@ make gate
|
||||||
Use `make benchmark` for the historical Milestone 0 baseline, `make benchmark-m1` for the
|
Use `make benchmark` for the historical Milestone 0 baseline, `make benchmark-m1` for the
|
||||||
counter-gated warm-operation benchmark, `make benchmark-m2` for agent workflow gates, and
|
counter-gated warm-operation benchmark, `make benchmark-m2` for agent workflow gates, and
|
||||||
`make benchmark-m3-full` for the ten-sample 1,000-node projection, worker, fragment, status,
|
`make benchmark-m3-full` for the ten-sample 1,000-node projection, worker, fragment, status,
|
||||||
equivalence, response-size, and memory gates. `make benchmark-m4-full` runs the 1,002-node adapter
|
equivalence, response-size, and memory gates. `make accessibility` runs the generated manual,
|
||||||
and recovery benchmark. `make adoption-m4` performs the offline fresh-wheel proof.
|
portable graph, and live viewer axe and keyboard flows.
|
||||||
`make command-reference-check` rejects command-reference drift, and `make docs-check` validates
|
|
||||||
the maintained documentation graph. `make accessibility` runs the generated manual, portable
|
|
||||||
graph, and live viewer axe and keyboard flows.
|
|
||||||
|
|
||||||
Project-specific vocabulary, extraction rules, and serialization belong in the project adapter.
|
Project-specific vocabulary, extraction rules, and serialization belong in the project adapter.
|
||||||
Generic core behavior must remain deterministic, project-bound, and recoverable.
|
Generic core behavior must remain deterministic, project-bound, and recoverable.
|
||||||
|
|
|
||||||
|
|
@ -13,26 +13,9 @@ authors = [{ name = "Worldforge contributors" }]
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"markdown-it-py>=4.2,<5",
|
"markdown-it-py>=4.2,<5",
|
||||||
"mcp>=1.28,<2",
|
"mcp>=1.28,<2",
|
||||||
]
|
|
||||||
|
|
||||||
[project.optional-dependencies]
|
|
||||||
javascript = [
|
|
||||||
"tree-sitter>=0.25,<0.26",
|
|
||||||
"tree-sitter-javascript>=0.25,<0.26",
|
|
||||||
]
|
|
||||||
typescript = [
|
|
||||||
"tree-sitter>=0.25,<0.26",
|
|
||||||
"tree-sitter-typescript>=0.23,<0.24",
|
|
||||||
]
|
|
||||||
cpp = [
|
|
||||||
"tree-sitter>=0.25,<0.26",
|
|
||||||
"tree-sitter-cpp>=0.23,<0.24",
|
|
||||||
]
|
|
||||||
languages = [
|
|
||||||
"tree-sitter>=0.25,<0.26",
|
"tree-sitter>=0.25,<0.26",
|
||||||
"tree-sitter-cpp>=0.23,<0.24",
|
"tree-sitter-cpp>=0.23,<0.24",
|
||||||
"tree-sitter-javascript>=0.25,<0.26",
|
"tree-sitter-javascript>=0.25,<0.26",
|
||||||
"tree-sitter-typescript>=0.23,<0.24",
|
|
||||||
]
|
]
|
||||||
|
|
||||||
[dependency-groups]
|
[dependency-groups]
|
||||||
|
|
@ -40,10 +23,6 @@ dev = [
|
||||||
"jsonschema>=4.25,<5",
|
"jsonschema>=4.25,<5",
|
||||||
"pytest>=9.1,<10",
|
"pytest>=9.1,<10",
|
||||||
"ruff>=0.15,<1",
|
"ruff>=0.15,<1",
|
||||||
"tree-sitter>=0.25,<0.26",
|
|
||||||
"tree-sitter-cpp>=0.23,<0.24",
|
|
||||||
"tree-sitter-javascript>=0.25,<0.26",
|
|
||||||
"tree-sitter-typescript>=0.23,<0.24",
|
|
||||||
]
|
]
|
||||||
|
|
||||||
[project.scripts]
|
[project.scripts]
|
||||||
|
|
|
||||||
|
|
@ -1,358 +0,0 @@
|
||||||
{
|
|
||||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
||||||
"$id": "https://docforge.local/schema/adapter-client-configuration-v1.json",
|
|
||||||
"title": "DocForge project-owned adapter client configuration",
|
|
||||||
"$defs": {
|
|
||||||
"sha256": {
|
|
||||||
"type": "string",
|
|
||||||
"pattern": "^[0-9a-f]{64}$"
|
|
||||||
},
|
|
||||||
"project": {
|
|
||||||
"type": "object",
|
|
||||||
"required": [
|
|
||||||
"project_id",
|
|
||||||
"project_root",
|
|
||||||
"project_root_fingerprint",
|
|
||||||
"adapter",
|
|
||||||
"descriptor_hash"
|
|
||||||
],
|
|
||||||
"properties": {
|
|
||||||
"project_id": {
|
|
||||||
"type": "string",
|
|
||||||
"pattern": "^[a-z0-9][a-z0-9._-]{1,127}$"
|
|
||||||
},
|
|
||||||
"project_root": {
|
|
||||||
"type": "string",
|
|
||||||
"minLength": 1,
|
|
||||||
"maxLength": 4096
|
|
||||||
},
|
|
||||||
"project_root_fingerprint": {
|
|
||||||
"type": "string",
|
|
||||||
"pattern": "^[0-9a-f]{16}$"
|
|
||||||
},
|
|
||||||
"adapter": {
|
|
||||||
"type": "string",
|
|
||||||
"pattern": "^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}@[A-Za-z0-9][A-Za-z0-9_.+-]{0,63}$"
|
|
||||||
},
|
|
||||||
"descriptor_hash": { "$ref": "#/$defs/sha256" }
|
|
||||||
},
|
|
||||||
"additionalProperties": false
|
|
||||||
},
|
|
||||||
"source_availability": {
|
|
||||||
"type": "object",
|
|
||||||
"required": [
|
|
||||||
"schema_version",
|
|
||||||
"status",
|
|
||||||
"method",
|
|
||||||
"revision",
|
|
||||||
"source_hash"
|
|
||||||
],
|
|
||||||
"properties": {
|
|
||||||
"schema_version": { "const": 1 },
|
|
||||||
"status": { "const": "available" },
|
|
||||||
"method": {
|
|
||||||
"enum": ["incremental-state", "complete-projection"]
|
|
||||||
},
|
|
||||||
"revision": {
|
|
||||||
"type": "string",
|
|
||||||
"minLength": 1,
|
|
||||||
"maxLength": 4096
|
|
||||||
},
|
|
||||||
"source_hash": { "$ref": "#/$defs/sha256" }
|
|
||||||
},
|
|
||||||
"additionalProperties": false
|
|
||||||
},
|
|
||||||
"binding": {
|
|
||||||
"type": "object",
|
|
||||||
"required": [
|
|
||||||
"transport",
|
|
||||||
"capability_mode",
|
|
||||||
"adapter_policy",
|
|
||||||
"render_policy",
|
|
||||||
"command",
|
|
||||||
"args",
|
|
||||||
"environment",
|
|
||||||
"timeouts",
|
|
||||||
"launcher_hash"
|
|
||||||
],
|
|
||||||
"properties": {
|
|
||||||
"transport": { "const": "stdio" },
|
|
||||||
"capability_mode": {
|
|
||||||
"enum": ["read", "proposal", "application"]
|
|
||||||
},
|
|
||||||
"adapter_policy": {
|
|
||||||
"$ref": "https://docforge.local/schema/client-configuration-v1.json#/$defs/adapter_policy"
|
|
||||||
},
|
|
||||||
"render_policy": {
|
|
||||||
"type": "object",
|
|
||||||
"required": ["manual", "graph", "live_viewer"],
|
|
||||||
"properties": {
|
|
||||||
"manual": { "enum": ["auto", "explicit", "disabled"] },
|
|
||||||
"graph": { "const": "disabled" },
|
|
||||||
"live_viewer": { "const": "on-demand" }
|
|
||||||
},
|
|
||||||
"additionalProperties": false
|
|
||||||
},
|
|
||||||
"command": {
|
|
||||||
"type": "string",
|
|
||||||
"minLength": 1,
|
|
||||||
"maxLength": 4096
|
|
||||||
},
|
|
||||||
"args": {
|
|
||||||
"type": "array",
|
|
||||||
"minItems": 7,
|
|
||||||
"maxItems": 64,
|
|
||||||
"prefixItems": [
|
|
||||||
{ "const": "-I" },
|
|
||||||
{ "const": "-m" },
|
|
||||||
{
|
|
||||||
"type": "string",
|
|
||||||
"minLength": 1,
|
|
||||||
"maxLength": 255,
|
|
||||||
"pattern": "^[A-Za-z_][A-Za-z0-9_]*(\\.[A-Za-z_][A-Za-z0-9_]*){0,31}$",
|
|
||||||
"not": { "const": "docforge.mcp_server" }
|
|
||||||
},
|
|
||||||
{ "const": "--project-root" },
|
|
||||||
{
|
|
||||||
"type": "string",
|
|
||||||
"minLength": 1,
|
|
||||||
"maxLength": 4096
|
|
||||||
},
|
|
||||||
{ "const": "--capability-mode" },
|
|
||||||
{ "enum": ["read", "proposal", "application"] }
|
|
||||||
],
|
|
||||||
"items": {
|
|
||||||
"type": "string",
|
|
||||||
"minLength": 1,
|
|
||||||
"maxLength": 4096
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"environment": {
|
|
||||||
"type": "object",
|
|
||||||
"maxProperties": 0
|
|
||||||
},
|
|
||||||
"timeouts": {
|
|
||||||
"type": "object",
|
|
||||||
"required": ["startup_seconds", "tool_seconds"],
|
|
||||||
"properties": {
|
|
||||||
"startup_seconds": {
|
|
||||||
"type": "integer",
|
|
||||||
"minimum": 1,
|
|
||||||
"maximum": 3600
|
|
||||||
},
|
|
||||||
"tool_seconds": {
|
|
||||||
"type": "integer",
|
|
||||||
"minimum": 1,
|
|
||||||
"maximum": 86400
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false
|
|
||||||
},
|
|
||||||
"launcher_hash": { "$ref": "#/$defs/sha256" }
|
|
||||||
},
|
|
||||||
"additionalProperties": false
|
|
||||||
},
|
|
||||||
"projection_availability": {
|
|
||||||
"type": "object",
|
|
||||||
"required": [
|
|
||||||
"manual_configured",
|
|
||||||
"portable_graph_configured",
|
|
||||||
"application_enabled",
|
|
||||||
"live_viewer_available"
|
|
||||||
],
|
|
||||||
"properties": {
|
|
||||||
"manual_configured": { "type": "boolean" },
|
|
||||||
"portable_graph_configured": { "type": "boolean" },
|
|
||||||
"application_enabled": { "type": "boolean" },
|
|
||||||
"live_viewer_available": { "const": true }
|
|
||||||
},
|
|
||||||
"additionalProperties": false
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"type": "object",
|
|
||||||
"required": [
|
|
||||||
"status",
|
|
||||||
"schema_version",
|
|
||||||
"operation",
|
|
||||||
"action",
|
|
||||||
"client",
|
|
||||||
"server_name",
|
|
||||||
"project",
|
|
||||||
"launcher",
|
|
||||||
"launcher_hash",
|
|
||||||
"source_availability",
|
|
||||||
"source_availability_hash",
|
|
||||||
"binding",
|
|
||||||
"effective_policy",
|
|
||||||
"projection_policy",
|
|
||||||
"projection_policy_hash",
|
|
||||||
"projection_availability",
|
|
||||||
"artifact",
|
|
||||||
"configuration_hash",
|
|
||||||
"warnings"
|
|
||||||
],
|
|
||||||
"properties": {
|
|
||||||
"status": { "const": "ok" },
|
|
||||||
"schema_version": { "const": 1 },
|
|
||||||
"operation": { "const": "adapter_client.configure" },
|
|
||||||
"action": { "enum": ["preview", "write"] },
|
|
||||||
"client": { "enum": ["codex", "claude", "openclaw"] },
|
|
||||||
"server_name": {
|
|
||||||
"type": "string",
|
|
||||||
"pattern": "^[a-z0-9][a-z0-9_-]{0,63}$"
|
|
||||||
},
|
|
||||||
"project": { "$ref": "#/$defs/project" },
|
|
||||||
"launcher": {
|
|
||||||
"$ref": "https://docforge.local/schema/adapter-launcher-v1.json"
|
|
||||||
},
|
|
||||||
"launcher_hash": { "$ref": "#/$defs/sha256" },
|
|
||||||
"source_availability": {
|
|
||||||
"$ref": "#/$defs/source_availability"
|
|
||||||
},
|
|
||||||
"source_availability_hash": { "$ref": "#/$defs/sha256" },
|
|
||||||
"binding": { "$ref": "#/$defs/binding" },
|
|
||||||
"effective_policy": {
|
|
||||||
"$ref": "https://docforge.local/schema/client-configuration-v1.json#/$defs/effective_policy"
|
|
||||||
},
|
|
||||||
"projection_policy": {
|
|
||||||
"$ref": "https://docforge.local/schema/client-configuration-v1.json#/$defs/projection_policy"
|
|
||||||
},
|
|
||||||
"projection_policy_hash": { "$ref": "#/$defs/sha256" },
|
|
||||||
"projection_availability": {
|
|
||||||
"$ref": "#/$defs/projection_availability"
|
|
||||||
},
|
|
||||||
"artifact": {
|
|
||||||
"$ref": "https://docforge.local/schema/client-configuration-v1.json#/properties/artifact"
|
|
||||||
},
|
|
||||||
"configuration_hash": { "$ref": "#/$defs/sha256" },
|
|
||||||
"warnings": {
|
|
||||||
"$ref": "https://docforge.local/schema/client-configuration-v1.json#/properties/warnings"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"allOf": [
|
|
||||||
{
|
|
||||||
"if": {
|
|
||||||
"properties": { "client": { "const": "codex" } },
|
|
||||||
"required": ["client"]
|
|
||||||
},
|
|
||||||
"then": {
|
|
||||||
"properties": {
|
|
||||||
"artifact": {
|
|
||||||
"properties": {
|
|
||||||
"format": { "const": "codex-toml-fragment-v1" }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"if": {
|
|
||||||
"properties": { "client": { "const": "claude" } },
|
|
||||||
"required": ["client"]
|
|
||||||
},
|
|
||||||
"then": {
|
|
||||||
"properties": {
|
|
||||||
"artifact": {
|
|
||||||
"properties": {
|
|
||||||
"format": { "const": "claude-json-fragment-v1" }
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"warnings": {
|
|
||||||
"contains": {
|
|
||||||
"properties": {
|
|
||||||
"code": { "const": "timeout_format_unverified" }
|
|
||||||
},
|
|
||||||
"required": ["code"]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"if": {
|
|
||||||
"properties": { "client": { "const": "openclaw" } },
|
|
||||||
"required": ["client"]
|
|
||||||
},
|
|
||||||
"then": {
|
|
||||||
"properties": {
|
|
||||||
"artifact": {
|
|
||||||
"properties": {
|
|
||||||
"format": { "const": "openclaw-json-fragment-v1" }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"if": {
|
|
||||||
"properties": { "action": { "const": "preview" } },
|
|
||||||
"required": ["action"]
|
|
||||||
},
|
|
||||||
"then": {
|
|
||||||
"properties": {
|
|
||||||
"artifact": {
|
|
||||||
"properties": {
|
|
||||||
"output_path": { "type": "null" },
|
|
||||||
"write_state": { "const": "not_requested" },
|
|
||||||
"durability": { "const": "not_applicable" }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"else": {
|
|
||||||
"properties": {
|
|
||||||
"artifact": {
|
|
||||||
"properties": {
|
|
||||||
"write_state": { "enum": ["created", "unchanged"] }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"if": {
|
|
||||||
"properties": {
|
|
||||||
"binding": {
|
|
||||||
"properties": {
|
|
||||||
"adapter_policy": {
|
|
||||||
"properties": {
|
|
||||||
"mode": { "const": "preserve-no-ast" }
|
|
||||||
},
|
|
||||||
"required": ["mode"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": ["adapter_policy"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": ["binding"]
|
|
||||||
},
|
|
||||||
"then": {
|
|
||||||
"properties": {
|
|
||||||
"binding": {
|
|
||||||
"properties": {
|
|
||||||
"args": {
|
|
||||||
"contains": { "const": "--no-ast" },
|
|
||||||
"minContains": 1,
|
|
||||||
"maxContains": 1
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"else": {
|
|
||||||
"properties": {
|
|
||||||
"binding": {
|
|
||||||
"properties": {
|
|
||||||
"args": {
|
|
||||||
"not": {
|
|
||||||
"contains": { "const": "--no-ast" }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"additionalProperties": false
|
|
||||||
}
|
|
||||||
|
|
@ -1,46 +0,0 @@
|
||||||
{
|
|
||||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
||||||
"$id": "https://docforge.local/schema/adapter-launcher-v1.json",
|
|
||||||
"title": "DocForge project-owned adapter launcher",
|
|
||||||
"type": "object",
|
|
||||||
"required": [
|
|
||||||
"schema_version",
|
|
||||||
"entry_point",
|
|
||||||
"project_id",
|
|
||||||
"project_root",
|
|
||||||
"adapter",
|
|
||||||
"descriptor_hash",
|
|
||||||
"module"
|
|
||||||
],
|
|
||||||
"properties": {
|
|
||||||
"schema_version": { "const": 1 },
|
|
||||||
"entry_point": { "const": "python-module" },
|
|
||||||
"project_id": {
|
|
||||||
"type": "string",
|
|
||||||
"pattern": "^[a-z0-9][a-z0-9._-]{1,127}$"
|
|
||||||
},
|
|
||||||
"project_root": {
|
|
||||||
"type": "string",
|
|
||||||
"minLength": 1,
|
|
||||||
"maxLength": 4096
|
|
||||||
},
|
|
||||||
"adapter": {
|
|
||||||
"type": "string",
|
|
||||||
"pattern": "^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}@[A-Za-z0-9][A-Za-z0-9_.+-]{0,63}$"
|
|
||||||
},
|
|
||||||
"descriptor_hash": {
|
|
||||||
"type": "string",
|
|
||||||
"pattern": "^[0-9a-f]{64}$"
|
|
||||||
},
|
|
||||||
"module": {
|
|
||||||
"type": "string",
|
|
||||||
"minLength": 1,
|
|
||||||
"maxLength": 255,
|
|
||||||
"anyOf": [
|
|
||||||
{ "pattern": "^[A-Za-z_][A-Za-z0-9_]*$" },
|
|
||||||
{ "const": "docforge.reference_mcp" }
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"additionalProperties": false
|
|
||||||
}
|
|
||||||
|
|
@ -1,78 +0,0 @@
|
||||||
{
|
|
||||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
||||||
"$id": "https://andraxion.net/docforge/schemas/reference-adapter.schema.json",
|
|
||||||
"title": "DocForge Reference Adapter Configuration",
|
|
||||||
"type": "object",
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": [
|
|
||||||
"schema_version",
|
|
||||||
"project_id",
|
|
||||||
"title",
|
|
||||||
"language",
|
|
||||||
"source_roots"
|
|
||||||
],
|
|
||||||
"properties": {
|
|
||||||
"schema_version": {
|
|
||||||
"const": 1
|
|
||||||
},
|
|
||||||
"project_id": {
|
|
||||||
"type": "string",
|
|
||||||
"pattern": "^[a-z0-9][a-z0-9._-]{1,127}$"
|
|
||||||
},
|
|
||||||
"title": {
|
|
||||||
"type": "string",
|
|
||||||
"minLength": 1,
|
|
||||||
"maxLength": 256
|
|
||||||
},
|
|
||||||
"language": {
|
|
||||||
"enum": [
|
|
||||||
"python",
|
|
||||||
"javascript",
|
|
||||||
"typescript",
|
|
||||||
"cpp"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"source_roots": {
|
|
||||||
"type": "array",
|
|
||||||
"minItems": 1,
|
|
||||||
"maxItems": 64,
|
|
||||||
"uniqueItems": true,
|
|
||||||
"items": {
|
|
||||||
"type": "string",
|
|
||||||
"minLength": 1,
|
|
||||||
"pattern": "^(?!/)(?!.*(?:^|/)\\.\\.(?:/|$)).+$"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"compilation_database": {
|
|
||||||
"type": "string",
|
|
||||||
"minLength": 1,
|
|
||||||
"pattern": "^(?!/)(?!.*(?:^|/)\\.\\.(?:/|$)).+$"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"allOf": [
|
|
||||||
{
|
|
||||||
"if": {
|
|
||||||
"properties": {
|
|
||||||
"language": {
|
|
||||||
"const": "cpp"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": [
|
|
||||||
"language"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"then": {
|
|
||||||
"required": [
|
|
||||||
"compilation_database"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"else": {
|
|
||||||
"not": {
|
|
||||||
"required": [
|
|
||||||
"compilation_database"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
|
|
@ -21,8 +21,6 @@ from .adapter_validation import (
|
||||||
from .config_validation import ID_PATTERN
|
from .config_validation import ID_PATTERN
|
||||||
from .errors import DocForgeError
|
from .errors import DocForgeError
|
||||||
from .incremental import (
|
from .incremental import (
|
||||||
MAX_EXTRACTION_CACHE_BYTES,
|
|
||||||
MAX_EXTRACTION_CACHE_SOURCES,
|
|
||||||
CachedSource,
|
CachedSource,
|
||||||
ExtractionCache,
|
ExtractionCache,
|
||||||
affected_sources,
|
affected_sources,
|
||||||
|
|
@ -42,10 +40,6 @@ from .models import (
|
||||||
)
|
)
|
||||||
from .telemetry import increment, stage
|
from .telemetry import increment, stage
|
||||||
|
|
||||||
MAX_ADAPTER_EDGE_MULTIPLIER = 32
|
|
||||||
MAX_ADAPTER_LOGIC_NODE_MULTIPLIER = 32
|
|
||||||
MAX_ADAPTER_LOGIC_EDGE_MULTIPLIER = 64
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class AdapterNode:
|
class AdapterNode:
|
||||||
|
|
@ -153,13 +147,6 @@ class AdapterAssembly:
|
||||||
logic: tuple[LogicProjection, ...] = ()
|
logic: tuple[LogicProjection, ...] = ()
|
||||||
|
|
||||||
|
|
||||||
@runtime_checkable
|
|
||||||
class CompleteAdapterAssemblyLoader(Protocol):
|
|
||||||
"""Load one cache-independent complete graph and Logic equivalence oracle."""
|
|
||||||
|
|
||||||
def load_complete_assembly(self) -> AdapterAssembly: ...
|
|
||||||
|
|
||||||
|
|
||||||
class AdapterLoader(Protocol):
|
class AdapterLoader(Protocol):
|
||||||
"""Load one current, deterministic, project-confined adapter projection."""
|
"""Load one current, deterministic, project-confined adapter projection."""
|
||||||
|
|
||||||
|
|
@ -214,14 +201,6 @@ def _extract_adapter_source(
|
||||||
return loader.extract_source(source)
|
return loader.extract_source(source)
|
||||||
|
|
||||||
|
|
||||||
def _load_complete_adapter_assembly(
|
|
||||||
loader: CompleteAdapterAssemblyLoader,
|
|
||||||
) -> AdapterAssembly:
|
|
||||||
increment("adapter_projection_loads")
|
|
||||||
with stage("adapter.projection"):
|
|
||||||
return loader.load_complete_assembly()
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class AdapterImplementation:
|
class AdapterImplementation:
|
||||||
"""One confined implementation boundary that must remain stable for a process."""
|
"""One confined implementation boundary that must remain stable for a process."""
|
||||||
|
|
@ -253,38 +232,6 @@ class _ImplementationSnapshot:
|
||||||
files: tuple[tuple[str, str], ...]
|
files: tuple[tuple[str, str], ...]
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class AdapterConformanceReport:
|
|
||||||
"""Stable evidence that one adapter passed its complete and incremental contracts."""
|
|
||||||
|
|
||||||
schema_version: int
|
|
||||||
project_id: str
|
|
||||||
adapter_id: str
|
|
||||||
adapter_version: str
|
|
||||||
revision: str
|
|
||||||
source_hash: str
|
|
||||||
assembly_hash: str
|
|
||||||
node_count: int
|
|
||||||
edge_count: int
|
|
||||||
logic_projection_count: int
|
|
||||||
incremental: bool
|
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, object]:
|
|
||||||
return {
|
|
||||||
"schema_version": self.schema_version,
|
|
||||||
"project_id": self.project_id,
|
|
||||||
"adapter_id": self.adapter_id,
|
|
||||||
"adapter_version": self.adapter_version,
|
|
||||||
"revision": self.revision,
|
|
||||||
"source_hash": self.source_hash,
|
|
||||||
"assembly_hash": self.assembly_hash,
|
|
||||||
"node_count": self.node_count,
|
|
||||||
"edge_count": self.edge_count,
|
|
||||||
"logic_projection_count": self.logic_projection_count,
|
|
||||||
"incremental": self.incremental,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
class AdapterProject:
|
class AdapterProject:
|
||||||
"""Expose a validated adapter projection through the standard index boundary."""
|
"""Expose a validated adapter projection through the standard index boundary."""
|
||||||
|
|
||||||
|
|
@ -470,9 +417,6 @@ class AdapterProject:
|
||||||
canonical_sources,
|
canonical_sources,
|
||||||
captured,
|
captured,
|
||||||
)
|
)
|
||||||
self._enforce_assembly_limits(
|
|
||||||
AdapterAssembly(projection=projection, logic=self._last_logic)
|
|
||||||
)
|
|
||||||
return ProjectSnapshot(
|
return ProjectSnapshot(
|
||||||
descriptor=self.descriptor,
|
descriptor=self.descriptor,
|
||||||
nodes=projection.core_nodes(),
|
nodes=projection.core_nodes(),
|
||||||
|
|
@ -579,57 +523,43 @@ class AdapterProject:
|
||||||
return self._last_logic
|
return self._last_logic
|
||||||
|
|
||||||
def verify_incremental_equivalence(self) -> dict[str, object]:
|
def verify_incremental_equivalence(self) -> dict[str, object]:
|
||||||
"""Prove incremental graph and Logic output matches an independent complete oracle."""
|
"""Prove the incremental and full loader contracts produce the same graph."""
|
||||||
|
|
||||||
self.validate_runtime()
|
self.validate_runtime()
|
||||||
loader = self._incremental_loader
|
if self._incremental_loader is None:
|
||||||
if loader is None:
|
|
||||||
raise DocForgeError(
|
raise DocForgeError(
|
||||||
"incremental_disabled", "Adapter does not implement incremental extraction"
|
"incremental_disabled", "Adapter does not implement incremental extraction"
|
||||||
)
|
)
|
||||||
incremental = self._load_incremental_assembly()
|
incremental = self._load_incremental()
|
||||||
full = _load_adapter_projection(self.loader)
|
full = _load_adapter_projection(self.loader)
|
||||||
validate_projection(full)
|
validate_projection(full)
|
||||||
complete = AdapterAssembly(full)
|
fields = {
|
||||||
if isinstance(loader, CompleteAdapterAssemblyLoader):
|
"project_id": incremental.project_id == full.project_id,
|
||||||
complete = _load_complete_adapter_assembly(loader)
|
"adapter_id": incremental.adapter_id == full.adapter_id,
|
||||||
_validate_adapter_assembly(complete)
|
"adapter_version": incremental.adapter_version == full.adapter_version,
|
||||||
oracle_fields = _projection_equivalence_fields(complete.projection, full)
|
"revision": incremental.revision == full.revision,
|
||||||
oracle_mismatches = [field for field, matches in oracle_fields.items() if not matches]
|
"source_hash": incremental.source_hash == full.source_hash,
|
||||||
if oracle_mismatches:
|
"nodes": incremental.nodes == full.nodes,
|
||||||
raise DocForgeError(
|
"edges": incremental.edges == full.edges,
|
||||||
"complete_oracle_mismatch",
|
}
|
||||||
"Complete adapter assembly does not match load_projection()",
|
|
||||||
fields=oracle_mismatches,
|
|
||||||
)
|
|
||||||
elif incremental.logic:
|
|
||||||
raise DocForgeError(
|
|
||||||
"complete_logic_oracle_required",
|
|
||||||
"Incremental Logic publication requires load_complete_assembly()",
|
|
||||||
)
|
|
||||||
fields = _assembly_equivalence_fields(incremental, complete)
|
|
||||||
mismatches = [field for field, matches in fields.items() if not matches]
|
mismatches = [field for field, matches in fields.items() if not matches]
|
||||||
if mismatches:
|
if mismatches:
|
||||||
raise DocForgeError(
|
raise DocForgeError(
|
||||||
"incremental_mismatch",
|
"incremental_mismatch",
|
||||||
"Incremental extraction does not match a complete adapter assembly",
|
"Incremental extraction does not match a full adapter projection",
|
||||||
fields=mismatches,
|
fields=mismatches,
|
||||||
)
|
)
|
||||||
self.validate_runtime()
|
self.validate_runtime()
|
||||||
return {
|
return {
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"project_id": incremental.projection.project_id,
|
"project_id": incremental.project_id,
|
||||||
"revision": incremental.projection.revision,
|
"revision": incremental.revision,
|
||||||
"source_hash": incremental.projection.source_hash,
|
"source_hash": incremental.source_hash,
|
||||||
"node_count": len(incremental.projection.nodes),
|
"node_count": len(incremental.nodes),
|
||||||
"edge_count": len(incremental.projection.edges),
|
"edge_count": len(incremental.edges),
|
||||||
"logic_projection_count": len(incremental.logic),
|
|
||||||
}
|
}
|
||||||
|
|
||||||
def _load_incremental(self) -> AdapterProjection:
|
def _load_incremental(self) -> AdapterProjection:
|
||||||
return self._load_incremental_assembly().projection
|
|
||||||
|
|
||||||
def _load_incremental_assembly(self) -> AdapterAssembly:
|
|
||||||
loader = self._incremental_loader
|
loader = self._incremental_loader
|
||||||
if loader is None:
|
if loader is None:
|
||||||
raise DocForgeError("incremental_disabled", "Incremental adapter is not configured")
|
raise DocForgeError("incremental_disabled", "Incremental adapter is not configured")
|
||||||
|
|
@ -637,14 +567,11 @@ class AdapterProject:
|
||||||
validate_manifest(manifest)
|
validate_manifest(manifest)
|
||||||
if manifest.identity() != self._identity:
|
if manifest.identity() != self._identity:
|
||||||
raise DocForgeError("adapter_changed", "Adapter identity changed during the operation")
|
raise DocForgeError("adapter_changed", "Adapter identity changed during the operation")
|
||||||
self._enforce_manifest_limits(manifest)
|
|
||||||
cache = load_extraction_cache(
|
cache = load_extraction_cache(
|
||||||
self._cache_path,
|
self._cache_path,
|
||||||
project_id=manifest.project_id,
|
project_id=manifest.project_id,
|
||||||
adapter_id=manifest.adapter_id,
|
adapter_id=manifest.adapter_id,
|
||||||
adapter_version=manifest.adapter_version,
|
adapter_version=manifest.adapter_version,
|
||||||
max_bytes=MAX_EXTRACTION_CACHE_BYTES,
|
|
||||||
max_sources=MAX_EXTRACTION_CACHE_SOURCES,
|
|
||||||
)
|
)
|
||||||
cached = {source.source_id: source for source in cache.sources} if cache else {}
|
cached = {source.source_id: source for source in cache.sources} if cache else {}
|
||||||
current = {source.source_id: source for source in manifest.sources}
|
current = {source.source_id: source for source in manifest.sources}
|
||||||
|
|
@ -731,7 +658,6 @@ class AdapterProject:
|
||||||
key=lambda projection: projection.owner_node_id,
|
key=lambda projection: projection.owner_node_id,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
assembly = AdapterAssembly(projection=projection, logic=logic_projections)
|
|
||||||
identity = (
|
identity = (
|
||||||
projection.project_id,
|
projection.project_id,
|
||||||
projection.adapter_id,
|
projection.adapter_id,
|
||||||
|
|
@ -748,8 +674,21 @@ class AdapterProject:
|
||||||
"invalid_adapter",
|
"invalid_adapter",
|
||||||
"Incremental assembly changed the manifest-bound project identity",
|
"Incremental assembly changed the manifest-bound project identity",
|
||||||
)
|
)
|
||||||
_validate_adapter_assembly(assembly)
|
validate_projection(projection)
|
||||||
self._enforce_assembly_limits(assembly)
|
owners = [projection.owner_node_id for projection in logic_projections]
|
||||||
|
if len(owners) != len(set(owners)):
|
||||||
|
raise DocForgeError(
|
||||||
|
"invalid_adapter", "A primary graph node may own only one logic projection"
|
||||||
|
)
|
||||||
|
node_ids = {item.node.node_id for item in projection.nodes}
|
||||||
|
for logic_projection in logic_projections:
|
||||||
|
validate_logic_projection(logic_projection)
|
||||||
|
if logic_projection.owner_node_id not in node_ids:
|
||||||
|
raise DocForgeError(
|
||||||
|
"invalid_adapter",
|
||||||
|
"A Logic projection owner must exist in the assembled primary graph",
|
||||||
|
owner_node_id=logic_projection.owner_node_id,
|
||||||
|
)
|
||||||
stable = loader.load_manifest()
|
stable = loader.load_manifest()
|
||||||
validate_manifest(stable)
|
validate_manifest(stable)
|
||||||
if stable != manifest:
|
if stable != manifest:
|
||||||
|
|
@ -765,8 +704,6 @@ class AdapterProject:
|
||||||
adapter_version=manifest.adapter_version,
|
adapter_version=manifest.adapter_version,
|
||||||
sources=tuple(cache_records),
|
sources=tuple(cache_records),
|
||||||
),
|
),
|
||||||
max_bytes=MAX_EXTRACTION_CACHE_BYTES,
|
|
||||||
max_sources=MAX_EXTRACTION_CACHE_SOURCES,
|
|
||||||
)
|
)
|
||||||
self._last_logic = logic_projections
|
self._last_logic = logic_projections
|
||||||
self._last_build_report = {
|
self._last_build_report = {
|
||||||
|
|
@ -779,41 +716,7 @@ class AdapterProject:
|
||||||
"cache_hit_ids": hits,
|
"cache_hit_ids": hits,
|
||||||
"reparsed_source_ids": reparsed,
|
"reparsed_source_ids": reparsed,
|
||||||
}
|
}
|
||||||
return assembly
|
return projection
|
||||||
|
|
||||||
def _enforce_manifest_limits(self, manifest: AdapterManifest) -> None:
|
|
||||||
if len(manifest.sources) > MAX_EXTRACTION_CACHE_SOURCES:
|
|
||||||
raise DocForgeError(
|
|
||||||
"adapter_limit",
|
|
||||||
"Adapter manifest exceeds the bounded source limit",
|
|
||||||
maximum=MAX_EXTRACTION_CACHE_SOURCES,
|
|
||||||
actual=len(manifest.sources),
|
|
||||||
)
|
|
||||||
|
|
||||||
def _enforce_assembly_limits(self, assembly: AdapterAssembly) -> None:
|
|
||||||
maximum_nodes = self.descriptor.limits.max_nodes
|
|
||||||
maximum_edges = maximum_nodes * MAX_ADAPTER_EDGE_MULTIPLIER
|
|
||||||
maximum_logic_nodes = maximum_nodes * MAX_ADAPTER_LOGIC_NODE_MULTIPLIER
|
|
||||||
maximum_logic_edges = maximum_nodes * MAX_ADAPTER_LOGIC_EDGE_MULTIPLIER
|
|
||||||
actual_nodes = len(assembly.projection.nodes)
|
|
||||||
actual_edges = len(assembly.projection.edges)
|
|
||||||
actual_logic_nodes = sum(len(projection.nodes) for projection in assembly.logic)
|
|
||||||
actual_logic_edges = sum(len(projection.edges) for projection in assembly.logic)
|
|
||||||
limits = (
|
|
||||||
("nodes", actual_nodes, maximum_nodes),
|
|
||||||
("edges", actual_edges, maximum_edges),
|
|
||||||
("logic_nodes", actual_logic_nodes, maximum_logic_nodes),
|
|
||||||
("logic_edges", actual_logic_edges, maximum_logic_edges),
|
|
||||||
)
|
|
||||||
for label, actual, maximum in limits:
|
|
||||||
if actual > maximum:
|
|
||||||
raise DocForgeError(
|
|
||||||
"adapter_limit",
|
|
||||||
f"Adapter assembly exceeds the configured {label.replace('_', ' ')} limit",
|
|
||||||
kind=label,
|
|
||||||
maximum=maximum,
|
|
||||||
actual=actual,
|
|
||||||
)
|
|
||||||
|
|
||||||
def canonical_source_paths(self) -> tuple[Path, ...]:
|
def canonical_source_paths(self) -> tuple[Path, ...]:
|
||||||
"""Adapters validate their own source sets before producing a projection."""
|
"""Adapters validate their own source sets before producing a projection."""
|
||||||
|
|
@ -1129,141 +1032,6 @@ class AdapterProject:
|
||||||
outputs.add(output)
|
outputs.add(output)
|
||||||
|
|
||||||
|
|
||||||
def _validate_adapter_assembly(assembly: AdapterAssembly) -> None:
|
|
||||||
validate_projection(assembly.projection)
|
|
||||||
ordered_logic = tuple(sorted(assembly.logic, key=lambda projection: projection.owner_node_id))
|
|
||||||
if assembly.logic != ordered_logic:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter",
|
|
||||||
"Complete Logic projections must be deterministically ordered",
|
|
||||||
)
|
|
||||||
owners = [projection.owner_node_id for projection in assembly.logic]
|
|
||||||
if len(owners) != len(set(owners)):
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter", "A primary graph node may own only one logic projection"
|
|
||||||
)
|
|
||||||
node_ids = {item.node.node_id for item in assembly.projection.nodes}
|
|
||||||
for logic_projection in assembly.logic:
|
|
||||||
validate_logic_projection(logic_projection)
|
|
||||||
if logic_projection.owner_node_id not in node_ids:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter",
|
|
||||||
"A Logic projection owner must exist in the assembled primary graph",
|
|
||||||
owner_node_id=logic_projection.owner_node_id,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _projection_equivalence_fields(
|
|
||||||
candidate: AdapterProjection,
|
|
||||||
reference: AdapterProjection,
|
|
||||||
) -> dict[str, bool]:
|
|
||||||
return {
|
|
||||||
"project_id": candidate.project_id == reference.project_id,
|
|
||||||
"title": candidate.title == reference.title,
|
|
||||||
"adapter_id": candidate.adapter_id == reference.adapter_id,
|
|
||||||
"adapter_version": candidate.adapter_version == reference.adapter_version,
|
|
||||||
"root": candidate.root == reference.root,
|
|
||||||
"revision": candidate.revision == reference.revision,
|
|
||||||
"source_hash": candidate.source_hash == reference.source_hash,
|
|
||||||
"nodes": candidate.nodes == reference.nodes,
|
|
||||||
"edges": candidate.edges == reference.edges,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _assembly_equivalence_fields(
|
|
||||||
candidate: AdapterAssembly,
|
|
||||||
reference: AdapterAssembly,
|
|
||||||
) -> dict[str, bool]:
|
|
||||||
return {
|
|
||||||
**_projection_equivalence_fields(candidate.projection, reference.projection),
|
|
||||||
"logic": candidate.logic == reference.logic,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _assembly_hash(assembly: AdapterAssembly) -> str:
|
|
||||||
projection = assembly.projection
|
|
||||||
payload = {
|
|
||||||
"projection": {
|
|
||||||
"project_id": projection.project_id,
|
|
||||||
"title": projection.title,
|
|
||||||
"adapter_id": projection.adapter_id,
|
|
||||||
"adapter_version": projection.adapter_version,
|
|
||||||
"root": str(projection.root),
|
|
||||||
"revision": projection.revision,
|
|
||||||
"source_hash": projection.source_hash,
|
|
||||||
"nodes": [item.as_dict() for item in projection.nodes],
|
|
||||||
"edges": [item.as_dict() for item in projection.edges],
|
|
||||||
},
|
|
||||||
"logic": [logic_projection.as_dict() for logic_projection in assembly.logic],
|
|
||||||
}
|
|
||||||
return hashlib.sha256(
|
|
||||||
json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
|
|
||||||
).hexdigest()
|
|
||||||
|
|
||||||
|
|
||||||
def _complete_reference_assembly(loader: AdapterLoader) -> AdapterAssembly:
|
|
||||||
if isinstance(loader, CompleteAdapterAssemblyLoader):
|
|
||||||
assembly = _load_complete_adapter_assembly(loader)
|
|
||||||
else:
|
|
||||||
assembly = AdapterAssembly(_load_adapter_projection(loader))
|
|
||||||
_validate_adapter_assembly(assembly)
|
|
||||||
return assembly
|
|
||||||
|
|
||||||
|
|
||||||
def verify_adapter_conformance(
|
|
||||||
loader: AdapterLoader,
|
|
||||||
*,
|
|
||||||
cache_root: Path,
|
|
||||||
settings: AdapterProjectSettings | None = None,
|
|
||||||
) -> AdapterConformanceReport:
|
|
||||||
"""Prove deterministic complete output and, when supported, incremental parity."""
|
|
||||||
|
|
||||||
project = AdapterProject(loader, cache_root=cache_root, settings=settings)
|
|
||||||
project.validate_runtime()
|
|
||||||
first = _complete_reference_assembly(loader)
|
|
||||||
second = _complete_reference_assembly(loader)
|
|
||||||
deterministic_fields = _assembly_equivalence_fields(first, second)
|
|
||||||
nondeterministic = [field for field, matches in deterministic_fields.items() if not matches]
|
|
||||||
if nondeterministic:
|
|
||||||
raise DocForgeError(
|
|
||||||
"nondeterministic_adapter",
|
|
||||||
"Repeated complete adapter assemblies do not match",
|
|
||||||
fields=nondeterministic,
|
|
||||||
)
|
|
||||||
|
|
||||||
if isinstance(loader, CompleteAdapterAssemblyLoader):
|
|
||||||
graph_oracle = _load_adapter_projection(loader)
|
|
||||||
validate_projection(graph_oracle)
|
|
||||||
oracle_fields = _projection_equivalence_fields(first.projection, graph_oracle)
|
|
||||||
oracle_mismatches = [field for field, matches in oracle_fields.items() if not matches]
|
|
||||||
if oracle_mismatches:
|
|
||||||
raise DocForgeError(
|
|
||||||
"complete_oracle_mismatch",
|
|
||||||
"Complete adapter assembly does not match load_projection()",
|
|
||||||
fields=oracle_mismatches,
|
|
||||||
)
|
|
||||||
|
|
||||||
incremental = isinstance(loader, IncrementalAdapterLoader)
|
|
||||||
if incremental:
|
|
||||||
project.verify_incremental_equivalence()
|
|
||||||
|
|
||||||
project.validate_runtime()
|
|
||||||
projection = first.projection
|
|
||||||
return AdapterConformanceReport(
|
|
||||||
schema_version=1,
|
|
||||||
project_id=projection.project_id,
|
|
||||||
adapter_id=projection.adapter_id,
|
|
||||||
adapter_version=projection.adapter_version,
|
|
||||||
revision=projection.revision,
|
|
||||||
source_hash=projection.source_hash,
|
|
||||||
assembly_hash=_assembly_hash(first),
|
|
||||||
node_count=len(projection.nodes),
|
|
||||||
edge_count=len(projection.edges),
|
|
||||||
logic_projection_count=len(first.logic),
|
|
||||||
incremental=incremental,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class ShadowArtifact:
|
class ShadowArtifact:
|
||||||
"""One named deterministic byte artifact used by a shadow comparison."""
|
"""One named deterministic byte artifact used by a shadow comparison."""
|
||||||
|
|
|
||||||
|
|
@ -1,392 +0,0 @@
|
||||||
"""Strict, data-only launch contracts for project-owned adapter MCP modules."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
import stat
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Literal, cast
|
|
||||||
|
|
||||||
from .changeset_contract import document_hash
|
|
||||||
from .config_validation import ID_PATTERN
|
|
||||||
from .errors import DocForgeError
|
|
||||||
from .models import IncrementalStateProject, ProjectService, RuntimeValidatedProject
|
|
||||||
|
|
||||||
ADAPTER_LAUNCHER_SCHEMA_VERSION = 1
|
|
||||||
_PYTHON_MODULE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*){0,31}")
|
|
||||||
_TOP_LEVEL_PYTHON_MODULE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*")
|
|
||||||
_ADAPTER_IDENTITY = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}@[A-Za-z0-9][A-Za-z0-9_.+-]{0,63}")
|
|
||||||
_SHA256 = re.compile(r"[0-9a-f]{64}")
|
|
||||||
_TRUSTED_DOTTED_MODULE = "docforge.reference_mcp"
|
|
||||||
_MODULE_PROBE_TIMEOUT_SECONDS = 5
|
|
||||||
_MAX_MODULE_PROBE_BYTES = 8_192
|
|
||||||
_MODULE_PROBE = (
|
|
||||||
"import importlib.util,json,sys;"
|
|
||||||
"spec=importlib.util.find_spec(sys.argv[1]);"
|
|
||||||
"result=None if spec is None else {"
|
|
||||||
"'has_loader':spec.loader is not None,"
|
|
||||||
"'is_package':spec.submodule_search_locations is not None,"
|
|
||||||
"'origin':spec.origin};"
|
|
||||||
"print(json.dumps(result,sort_keys=True,separators=(',',':')))"
|
|
||||||
)
|
|
||||||
_run_module_probe = subprocess.run
|
|
||||||
__all__ = [
|
|
||||||
"ADAPTER_LAUNCHER_SCHEMA_VERSION",
|
|
||||||
"AdapterLauncherV1",
|
|
||||||
"AdapterSourceAvailabilityV1",
|
|
||||||
"adapter_source_availability",
|
|
||||||
"validate_adapter_launcher",
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class AdapterLauncherV1:
|
|
||||||
"""One immutable, project-bound Python module launch declaration.
|
|
||||||
|
|
||||||
The contract intentionally has no command, shell string, working directory,
|
|
||||||
environment, arbitrary arguments, discovery rule, or callable selector.
|
|
||||||
"""
|
|
||||||
|
|
||||||
schema_version: int
|
|
||||||
entry_point: Literal["python-module"]
|
|
||||||
project_id: str
|
|
||||||
project_root: Path
|
|
||||||
adapter: str
|
|
||||||
descriptor_hash: str
|
|
||||||
module: str
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
_validate_launcher_fields(self)
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def for_project(
|
|
||||||
cls,
|
|
||||||
project: ProjectService,
|
|
||||||
*,
|
|
||||||
module: str,
|
|
||||||
) -> AdapterLauncherV1:
|
|
||||||
"""Bind a structural module entry point to one already constructed adapter project."""
|
|
||||||
|
|
||||||
descriptor = project.descriptor
|
|
||||||
if descriptor.adapter == "generic":
|
|
||||||
raise DocForgeError(
|
|
||||||
"adapter_launcher_unavailable",
|
|
||||||
"Project-owned launcher configuration requires a custom adapter",
|
|
||||||
)
|
|
||||||
launcher = cls(
|
|
||||||
schema_version=ADAPTER_LAUNCHER_SCHEMA_VERSION,
|
|
||||||
entry_point="python-module",
|
|
||||||
project_id=descriptor.project_id,
|
|
||||||
project_root=descriptor.root,
|
|
||||||
adapter=descriptor.adapter,
|
|
||||||
descriptor_hash=descriptor.descriptor_hash,
|
|
||||||
module=module,
|
|
||||||
)
|
|
||||||
validate_adapter_launcher(project, launcher)
|
|
||||||
return launcher
|
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, object]:
|
|
||||||
return {
|
|
||||||
"schema_version": self.schema_version,
|
|
||||||
"entry_point": self.entry_point,
|
|
||||||
"project_id": self.project_id,
|
|
||||||
"project_root": str(self.project_root),
|
|
||||||
"adapter": self.adapter,
|
|
||||||
"descriptor_hash": self.descriptor_hash,
|
|
||||||
"module": self.module,
|
|
||||||
}
|
|
||||||
|
|
||||||
@property
|
|
||||||
def launcher_hash(self) -> str:
|
|
||||||
return document_hash(self.as_dict())
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class AdapterSourceAvailabilityV1:
|
|
||||||
"""One bounded source identity observed without executing the launcher module."""
|
|
||||||
|
|
||||||
schema_version: int
|
|
||||||
status: Literal["available"]
|
|
||||||
method: Literal["incremental-state", "complete-projection"]
|
|
||||||
revision: str
|
|
||||||
source_hash: str
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
revision = _runtime_value(self.revision)
|
|
||||||
if (
|
|
||||||
self.schema_version != 1
|
|
||||||
or self.status != "available"
|
|
||||||
or self.method not in {"incremental-state", "complete-projection"}
|
|
||||||
or not isinstance(revision, str)
|
|
||||||
or not revision
|
|
||||||
or _SHA256.fullmatch(self.source_hash) is None
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter_launcher",
|
|
||||||
"Adapter source availability evidence is invalid",
|
|
||||||
)
|
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, object]:
|
|
||||||
return {
|
|
||||||
"schema_version": self.schema_version,
|
|
||||||
"status": self.status,
|
|
||||||
"method": self.method,
|
|
||||||
"revision": self.revision,
|
|
||||||
"source_hash": self.source_hash,
|
|
||||||
}
|
|
||||||
|
|
||||||
@property
|
|
||||||
def availability_hash(self) -> str:
|
|
||||||
return document_hash(self.as_dict())
|
|
||||||
|
|
||||||
|
|
||||||
def validate_adapter_launcher(
|
|
||||||
project: ProjectService,
|
|
||||||
launcher: AdapterLauncherV1,
|
|
||||||
) -> None:
|
|
||||||
"""Require one launcher to match the exact current adapter project binding."""
|
|
||||||
|
|
||||||
_validate_launcher_fields(launcher)
|
|
||||||
descriptor = project.descriptor
|
|
||||||
_validate_project_root(descriptor.root)
|
|
||||||
if descriptor.adapter == "generic":
|
|
||||||
raise DocForgeError(
|
|
||||||
"adapter_launcher_unavailable",
|
|
||||||
"Project-owned launcher configuration requires a custom adapter",
|
|
||||||
)
|
|
||||||
if (
|
|
||||||
launcher.project_id != descriptor.project_id
|
|
||||||
or launcher.project_root != descriptor.root
|
|
||||||
or launcher.adapter != descriptor.adapter
|
|
||||||
or launcher.descriptor_hash != descriptor.descriptor_hash
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"adapter_launcher_mismatch",
|
|
||||||
"Adapter launcher does not match the selected project binding",
|
|
||||||
project_id=descriptor.project_id,
|
|
||||||
adapter=descriptor.adapter,
|
|
||||||
)
|
|
||||||
if isinstance(project, RuntimeValidatedProject):
|
|
||||||
project.validate_runtime()
|
|
||||||
|
|
||||||
|
|
||||||
def adapter_source_availability(
|
|
||||||
project: ProjectService,
|
|
||||||
launcher: AdapterLauncherV1,
|
|
||||||
) -> AdapterSourceAvailabilityV1:
|
|
||||||
"""Capture current source identity without importing or executing the launcher module."""
|
|
||||||
|
|
||||||
validate_adapter_launcher(project, launcher)
|
|
||||||
state = project.incremental_state() if isinstance(project, IncrementalStateProject) else None
|
|
||||||
if state is not None:
|
|
||||||
method: Literal["incremental-state", "complete-projection"] = "incremental-state"
|
|
||||||
revision = state.revision
|
|
||||||
source_hash = state.source_hash
|
|
||||||
else:
|
|
||||||
snapshot = project.load()
|
|
||||||
descriptor = snapshot.descriptor
|
|
||||||
if (
|
|
||||||
descriptor.project_id != launcher.project_id
|
|
||||||
or descriptor.root != launcher.project_root
|
|
||||||
or descriptor.adapter != launcher.adapter
|
|
||||||
or descriptor.descriptor_hash != launcher.descriptor_hash
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"adapter_launcher_mismatch",
|
|
||||||
"Loaded adapter projection drifted from its launcher binding",
|
|
||||||
)
|
|
||||||
method = "complete-projection"
|
|
||||||
revision = snapshot.revision
|
|
||||||
source_hash = snapshot.source_hash
|
|
||||||
validate_adapter_launcher(project, launcher)
|
|
||||||
return AdapterSourceAvailabilityV1(
|
|
||||||
schema_version=1,
|
|
||||||
status="available",
|
|
||||||
method=method,
|
|
||||||
revision=revision,
|
|
||||||
source_hash=source_hash,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _validate_launcher_fields(launcher: AdapterLauncherV1) -> None:
|
|
||||||
if type(launcher.schema_version) is not int or launcher.schema_version != 1:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter_launcher",
|
|
||||||
"Adapter launcher schema version is unsupported",
|
|
||||||
supported=1,
|
|
||||||
)
|
|
||||||
if launcher.entry_point != "python-module":
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter_launcher",
|
|
||||||
"Adapter launcher entry point must be one isolated Python module",
|
|
||||||
)
|
|
||||||
project_id = _runtime_value(launcher.project_id)
|
|
||||||
project_root = _runtime_value(launcher.project_root)
|
|
||||||
adapter = _runtime_value(launcher.adapter)
|
|
||||||
descriptor_hash = _runtime_value(launcher.descriptor_hash)
|
|
||||||
module = _runtime_value(launcher.module)
|
|
||||||
if not isinstance(project_id, str) or ID_PATTERN.fullmatch(project_id) is None:
|
|
||||||
raise DocForgeError("invalid_adapter_launcher", "Adapter launcher project ID is invalid")
|
|
||||||
if not isinstance(project_root, Path):
|
|
||||||
raise DocForgeError("invalid_adapter_launcher", "Adapter launcher project root is invalid")
|
|
||||||
_validate_project_root(project_root)
|
|
||||||
if not isinstance(adapter, str) or _ADAPTER_IDENTITY.fullmatch(adapter) is None:
|
|
||||||
raise DocForgeError("invalid_adapter_launcher", "Adapter launcher identity is invalid")
|
|
||||||
if not isinstance(descriptor_hash, str) or _SHA256.fullmatch(descriptor_hash) is None:
|
|
||||||
raise DocForgeError("invalid_adapter_launcher", "Adapter descriptor hash is invalid")
|
|
||||||
if (
|
|
||||||
not isinstance(module, str)
|
|
||||||
or len(module) > 255
|
|
||||||
or _PYTHON_MODULE.fullmatch(module) is None
|
|
||||||
or module == "docforge.mcp_server"
|
|
||||||
or (module != _TRUSTED_DOTTED_MODULE and _TOP_LEVEL_PYTHON_MODULE.fullmatch(module) is None)
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter_launcher",
|
|
||||||
(
|
|
||||||
"Adapter launcher module must be one installed project-owned top-level "
|
|
||||||
"Python module or the fixed DocForge reference binding"
|
|
||||||
),
|
|
||||||
)
|
|
||||||
_validate_isolated_module(project_root, module)
|
|
||||||
|
|
||||||
|
|
||||||
def _validate_isolated_module(project_root: Path, module: str) -> None:
|
|
||||||
"""Prove ``python -I -m`` can resolve one module without importing project code."""
|
|
||||||
|
|
||||||
executable = Path(sys.executable)
|
|
||||||
try:
|
|
||||||
executable_status = executable.stat()
|
|
||||||
except OSError as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"adapter_launcher_unavailable",
|
|
||||||
"The isolated Python executable is unavailable",
|
|
||||||
) from error
|
|
||||||
if not stat.S_ISREG(executable_status.st_mode):
|
|
||||||
raise DocForgeError(
|
|
||||||
"adapter_launcher_unavailable",
|
|
||||||
"The isolated Python executable is not a regular file",
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
completed = _run_module_probe(
|
|
||||||
[str(executable), "-I", "-c", _MODULE_PROBE, module],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
timeout=_MODULE_PROBE_TIMEOUT_SECONDS,
|
|
||||||
check=False,
|
|
||||||
)
|
|
||||||
except (OSError, subprocess.TimeoutExpired) as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"adapter_launcher_unavailable",
|
|
||||||
"The isolated adapter launcher probe could not complete",
|
|
||||||
module=module,
|
|
||||||
) from error
|
|
||||||
if (
|
|
||||||
completed.returncode != 0
|
|
||||||
or len(completed.stdout.encode("utf-8")) > _MAX_MODULE_PROBE_BYTES
|
|
||||||
or len(completed.stderr.encode("utf-8")) > _MAX_MODULE_PROBE_BYTES
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"adapter_launcher_unavailable",
|
|
||||||
"The isolated adapter launcher module could not be resolved",
|
|
||||||
module=module,
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
result: object = json.loads(completed.stdout)
|
|
||||||
except (UnicodeError, json.JSONDecodeError) as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"adapter_launcher_unavailable",
|
|
||||||
"The isolated adapter launcher probe returned invalid evidence",
|
|
||||||
module=module,
|
|
||||||
) from error
|
|
||||||
if result is None:
|
|
||||||
raise DocForgeError(
|
|
||||||
"adapter_launcher_unavailable",
|
|
||||||
"The isolated adapter launcher module is not installed",
|
|
||||||
module=module,
|
|
||||||
)
|
|
||||||
if not isinstance(result, dict):
|
|
||||||
raise DocForgeError(
|
|
||||||
"adapter_launcher_unavailable",
|
|
||||||
"The isolated adapter launcher probe returned invalid evidence",
|
|
||||||
module=module,
|
|
||||||
)
|
|
||||||
evidence = cast(dict[object, object], result)
|
|
||||||
if (
|
|
||||||
set(evidence) != {"has_loader", "is_package", "origin"}
|
|
||||||
or evidence["has_loader"] is not True
|
|
||||||
or evidence["is_package"] is not False
|
|
||||||
or not isinstance(evidence["origin"], str)
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter_launcher",
|
|
||||||
"The isolated adapter launcher must resolve to one executable module file",
|
|
||||||
module=module,
|
|
||||||
)
|
|
||||||
origin = Path(evidence["origin"])
|
|
||||||
try:
|
|
||||||
origin_status = origin.lstat()
|
|
||||||
resolved_origin = origin.resolve(strict=True)
|
|
||||||
except OSError as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"adapter_launcher_unavailable",
|
|
||||||
"The isolated adapter launcher module origin is unavailable",
|
|
||||||
module=module,
|
|
||||||
) from error
|
|
||||||
if (
|
|
||||||
not origin.is_absolute()
|
|
||||||
or stat.S_ISLNK(origin_status.st_mode)
|
|
||||||
or not stat.S_ISREG(origin_status.st_mode)
|
|
||||||
or resolved_origin != origin
|
|
||||||
or origin.suffix != ".py"
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter_launcher",
|
|
||||||
"The isolated adapter launcher must be one canonical Python module file",
|
|
||||||
module=module,
|
|
||||||
)
|
|
||||||
if module == _TRUSTED_DOTTED_MODULE:
|
|
||||||
expected = Path(__file__).with_name("reference_mcp.py").resolve(strict=True)
|
|
||||||
if origin != expected:
|
|
||||||
raise DocForgeError(
|
|
||||||
"adapter_launcher_mismatch",
|
|
||||||
"The fixed reference launcher resolved outside this DocForge installation",
|
|
||||||
module=module,
|
|
||||||
)
|
|
||||||
elif not origin.is_relative_to(project_root):
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter_launcher",
|
|
||||||
"The installed adapter launcher module is not owned by the selected project",
|
|
||||||
module=module,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _validate_project_root(root: Path) -> None:
|
|
||||||
try:
|
|
||||||
status = root.lstat()
|
|
||||||
resolved = root.resolve(strict=True)
|
|
||||||
except OSError as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter_launcher",
|
|
||||||
"Adapter launcher project root is unavailable",
|
|
||||||
) from error
|
|
||||||
if (
|
|
||||||
not root.is_absolute()
|
|
||||||
or stat.S_ISLNK(status.st_mode)
|
|
||||||
or not stat.S_ISDIR(status.st_mode)
|
|
||||||
or resolved != root
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter_launcher",
|
|
||||||
"Adapter launcher project root must be one canonical real directory",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _runtime_value(value: object) -> object:
|
|
||||||
"""Keep runtime validation explicit even when static callers are typed."""
|
|
||||||
|
|
||||||
return value
|
|
||||||
|
|
@ -1,69 +0,0 @@
|
||||||
"""Public typed surface for project adapter implementations and conformance checks."""
|
|
||||||
|
|
||||||
from .adapter_contract import (
|
|
||||||
AdapterAssembly,
|
|
||||||
AdapterConformanceReport,
|
|
||||||
AdapterEdge,
|
|
||||||
AdapterImplementation,
|
|
||||||
AdapterLoader,
|
|
||||||
AdapterManifest,
|
|
||||||
AdapterNode,
|
|
||||||
AdapterProject,
|
|
||||||
AdapterProjection,
|
|
||||||
AdapterProjectSettings,
|
|
||||||
AdapterSource,
|
|
||||||
AdapterSourceProjection,
|
|
||||||
CompleteAdapterAssemblyLoader,
|
|
||||||
IncrementalAdapterAssembler,
|
|
||||||
IncrementalAdapterLoader,
|
|
||||||
verify_adapter_conformance,
|
|
||||||
)
|
|
||||||
from .adapter_validation import (
|
|
||||||
validate_logic_projection,
|
|
||||||
validate_manifest,
|
|
||||||
validate_projection,
|
|
||||||
validate_source_projection,
|
|
||||||
)
|
|
||||||
from .models import (
|
|
||||||
Edge,
|
|
||||||
Limits,
|
|
||||||
LogicEdge,
|
|
||||||
LogicNode,
|
|
||||||
LogicProjection,
|
|
||||||
Node,
|
|
||||||
ProposalWriter,
|
|
||||||
RenderConfig,
|
|
||||||
RenderView,
|
|
||||||
)
|
|
||||||
|
|
||||||
__all__ = [
|
|
||||||
"AdapterAssembly",
|
|
||||||
"AdapterConformanceReport",
|
|
||||||
"AdapterEdge",
|
|
||||||
"AdapterImplementation",
|
|
||||||
"AdapterLoader",
|
|
||||||
"AdapterManifest",
|
|
||||||
"AdapterNode",
|
|
||||||
"AdapterProject",
|
|
||||||
"AdapterProjection",
|
|
||||||
"AdapterProjectSettings",
|
|
||||||
"AdapterSource",
|
|
||||||
"AdapterSourceProjection",
|
|
||||||
"CompleteAdapterAssemblyLoader",
|
|
||||||
"Edge",
|
|
||||||
"IncrementalAdapterAssembler",
|
|
||||||
"IncrementalAdapterLoader",
|
|
||||||
"Limits",
|
|
||||||
"LogicEdge",
|
|
||||||
"LogicNode",
|
|
||||||
"LogicProjection",
|
|
||||||
"Node",
|
|
||||||
"ProposalWriter",
|
|
||||||
"RenderConfig",
|
|
||||||
"RenderView",
|
|
||||||
"validate_logic_projection",
|
|
||||||
"validate_manifest",
|
|
||||||
"validate_projection",
|
|
||||||
"validate_source_projection",
|
|
||||||
"verify_adapter_conformance",
|
|
||||||
]
|
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
"""Repository-owned reference adapters built on the public adapter SDK."""
|
|
||||||
|
|
||||||
from .javascript import (
|
|
||||||
JAVASCRIPT_ADAPTER_VERSION,
|
|
||||||
JAVASCRIPT_EXTRACTOR_VERSION,
|
|
||||||
JavaScriptReferenceAdapter,
|
|
||||||
JavaScriptUnsupportedFact,
|
|
||||||
)
|
|
||||||
from .python import (
|
|
||||||
PYTHON_ADAPTER_VERSION,
|
|
||||||
PYTHON_EXTRACTOR_VERSION,
|
|
||||||
PythonReferenceAdapter,
|
|
||||||
PythonUnsupportedFact,
|
|
||||||
)
|
|
||||||
|
|
||||||
__all__ = [
|
|
||||||
"JAVASCRIPT_ADAPTER_VERSION",
|
|
||||||
"JAVASCRIPT_EXTRACTOR_VERSION",
|
|
||||||
"PYTHON_ADAPTER_VERSION",
|
|
||||||
"PYTHON_EXTRACTOR_VERSION",
|
|
||||||
"JavaScriptReferenceAdapter",
|
|
||||||
"JavaScriptUnsupportedFact",
|
|
||||||
"PythonReferenceAdapter",
|
|
||||||
"PythonUnsupportedFact",
|
|
||||||
]
|
|
||||||
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
|
|
@ -1,948 +0,0 @@
|
||||||
"""Deterministic stdlib-AST reference adapter for explicitly confined Python roots.
|
|
||||||
|
|
||||||
The adapter parses Python source as untrusted data. It never imports or executes
|
|
||||||
project code. Its intentionally narrow dependency model publishes only imports
|
|
||||||
that resolve to another module in the same declared source inventory.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import ast
|
|
||||||
import hashlib
|
|
||||||
import io
|
|
||||||
import os
|
|
||||||
import stat
|
|
||||||
import tokenize
|
|
||||||
from collections.abc import Iterable, Sequence
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from pathlib import Path, PurePosixPath
|
|
||||||
|
|
||||||
from ..adapter_sdk import (
|
|
||||||
AdapterAssembly,
|
|
||||||
AdapterEdge,
|
|
||||||
AdapterManifest,
|
|
||||||
AdapterNode,
|
|
||||||
AdapterProjection,
|
|
||||||
AdapterSource,
|
|
||||||
AdapterSourceProjection,
|
|
||||||
Edge,
|
|
||||||
Node,
|
|
||||||
)
|
|
||||||
from ..errors import DocForgeError
|
|
||||||
from ..python_logic import PythonLogicOwner, analyze_python_source
|
|
||||||
|
|
||||||
PYTHON_ADAPTER_ID = "docforge.reference.python"
|
|
||||||
PYTHON_ADAPTER_VERSION = "1"
|
|
||||||
PYTHON_EXTRACTOR_VERSION = "stdlib-ast@1"
|
|
||||||
PYTHON_IDENTITY_VERSION = "python-reference-id@1"
|
|
||||||
PYTHON_SUPPORT_SCHEMA_VERSION = 1
|
|
||||||
|
|
||||||
_SUPPORTED_FACTS = (
|
|
||||||
"python_file",
|
|
||||||
"python_module",
|
|
||||||
"python_class",
|
|
||||||
"python_function",
|
|
||||||
"lexical_containment",
|
|
||||||
"local_import_dependency",
|
|
||||||
"function_logic",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class PythonUnsupportedFact:
|
|
||||||
"""One semantic fact this syntax-only reference adapter does not claim."""
|
|
||||||
|
|
||||||
code: str
|
|
||||||
description: str
|
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, str]:
|
|
||||||
return {"code": self.code, "description": self.description}
|
|
||||||
|
|
||||||
|
|
||||||
_UNSUPPORTED_FACTS = (
|
|
||||||
PythonUnsupportedFact(
|
|
||||||
"call_resolution",
|
|
||||||
"Calls are represented only inside function Logic and are not resolved to symbols.",
|
|
||||||
),
|
|
||||||
PythonUnsupportedFact(
|
|
||||||
"dynamic_import_resolution",
|
|
||||||
"Imports performed through runtime calls are not dependency evidence.",
|
|
||||||
),
|
|
||||||
PythonUnsupportedFact(
|
|
||||||
"inheritance_resolution",
|
|
||||||
"Class bases are syntax only and are not resolved to local or external types.",
|
|
||||||
),
|
|
||||||
PythonUnsupportedFact(
|
|
||||||
"runtime_generated_facts",
|
|
||||||
"Decorators, metaclasses, descriptors, and executed module code are never evaluated.",
|
|
||||||
),
|
|
||||||
PythonUnsupportedFact(
|
|
||||||
"symbol_reference_resolution",
|
|
||||||
"Imported names, variable references, types, overloads, and re-exports are not resolved.",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class _SourceRecord:
|
|
||||||
source_id: str
|
|
||||||
source_path: str
|
|
||||||
module_name: str
|
|
||||||
fingerprint: str
|
|
||||||
raw: bytes
|
|
||||||
text: str
|
|
||||||
tree: ast.Module | None
|
|
||||||
dependencies: tuple[str, ...] = ()
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class _Definition:
|
|
||||||
kind: str
|
|
||||||
qualified_name: str
|
|
||||||
parent_node_id: str
|
|
||||||
node: ast.ClassDef | ast.FunctionDef | ast.AsyncFunctionDef
|
|
||||||
node_id: str
|
|
||||||
|
|
||||||
|
|
||||||
class PythonReferenceAdapter:
|
|
||||||
"""Reference Python adapter over explicit, non-overlapping source roots."""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
root: Path,
|
|
||||||
*,
|
|
||||||
source_roots: Iterable[str | Path],
|
|
||||||
project_id: str = "python-reference",
|
|
||||||
title: str = "Python reference project",
|
|
||||||
max_sources: int = 4_096,
|
|
||||||
max_source_bytes: int = 1_000_000,
|
|
||||||
max_logic_nodes_per_function: int = 2_000,
|
|
||||||
) -> None:
|
|
||||||
resolved_root = root.resolve(strict=True)
|
|
||||||
if not resolved_root.is_dir():
|
|
||||||
raise DocForgeError("invalid_adapter", "Python adapter root must be a directory")
|
|
||||||
if max_sources < 1 or max_source_bytes < 1 or max_logic_nodes_per_function < 2:
|
|
||||||
raise ValueError("Python adapter limits must be positive")
|
|
||||||
self.root = resolved_root
|
|
||||||
self.project_id = project_id
|
|
||||||
self.title = title
|
|
||||||
self.max_sources = max_sources
|
|
||||||
self.max_source_bytes = max_source_bytes
|
|
||||||
self.max_logic_nodes_per_function = max_logic_nodes_per_function
|
|
||||||
self.source_roots = self._normalize_source_roots(source_roots)
|
|
||||||
|
|
||||||
def support_report(self) -> dict[str, object]:
|
|
||||||
"""Return deterministic, machine-readable scope and limitation evidence."""
|
|
||||||
|
|
||||||
return {
|
|
||||||
"schema_version": PYTHON_SUPPORT_SCHEMA_VERSION,
|
|
||||||
"adapter_id": PYTHON_ADAPTER_ID,
|
|
||||||
"adapter_version": PYTHON_ADAPTER_VERSION,
|
|
||||||
"extractor_version": PYTHON_EXTRACTOR_VERSION,
|
|
||||||
"identity_version": PYTHON_IDENTITY_VERSION,
|
|
||||||
"frontend": "python-stdlib-ast",
|
|
||||||
"imports_project_code": False,
|
|
||||||
"executes_project_code": False,
|
|
||||||
"source_roots": list(self.source_roots),
|
|
||||||
"supported_facts": list(_SUPPORTED_FACTS),
|
|
||||||
"unsupported_facts": [fact.as_dict() for fact in _UNSUPPORTED_FACTS],
|
|
||||||
}
|
|
||||||
|
|
||||||
def unsupported_facts(self) -> tuple[PythonUnsupportedFact, ...]:
|
|
||||||
"""Return the adapter's fixed unsupported semantic fact inventory."""
|
|
||||||
|
|
||||||
return _UNSUPPORTED_FACTS
|
|
||||||
|
|
||||||
def load_manifest(self) -> AdapterManifest:
|
|
||||||
records = self._inventory(parse=False)
|
|
||||||
module_sources = {record.module_name: record.source_id for record in records}
|
|
||||||
if len(module_sources) != len(records):
|
|
||||||
raise DocForgeError(
|
|
||||||
"ambiguous_python_module",
|
|
||||||
"Declared Python source roots produce duplicate module names",
|
|
||||||
)
|
|
||||||
sources = tuple(
|
|
||||||
sorted(
|
|
||||||
(
|
|
||||||
AdapterSource(
|
|
||||||
source_id=record.source_id,
|
|
||||||
source_path=record.source_path,
|
|
||||||
fingerprint=record.fingerprint,
|
|
||||||
extractor_version=PYTHON_EXTRACTOR_VERSION,
|
|
||||||
dependencies=self._local_dependencies(record, module_sources),
|
|
||||||
)
|
|
||||||
for record in records
|
|
||||||
),
|
|
||||||
key=lambda source: source.source_id,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
source_hash = self._source_hash(sources)
|
|
||||||
return AdapterManifest(
|
|
||||||
project_id=self.project_id,
|
|
||||||
title=self.title,
|
|
||||||
adapter_id=PYTHON_ADAPTER_ID,
|
|
||||||
adapter_version=PYTHON_ADAPTER_VERSION,
|
|
||||||
root=self.root,
|
|
||||||
revision=source_hash[:12],
|
|
||||||
source_hash=source_hash,
|
|
||||||
families=("code",),
|
|
||||||
allowed_relations=("contains", "depends_on"),
|
|
||||||
sources=sources,
|
|
||||||
estimated_nodes=max(1, len(sources) * 8),
|
|
||||||
)
|
|
||||||
|
|
||||||
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection:
|
|
||||||
record = self._read_source(source.source_path)
|
|
||||||
if record.source_id != source.source_id or record.fingerprint != source.fingerprint:
|
|
||||||
raise DocForgeError(
|
|
||||||
"stale_adapter_source",
|
|
||||||
"Python source changed after its manifest was captured",
|
|
||||||
source=source.source_path,
|
|
||||||
)
|
|
||||||
return self._extract_record(record, source)
|
|
||||||
|
|
||||||
def assemble_projection(
|
|
||||||
self,
|
|
||||||
manifest: AdapterManifest,
|
|
||||||
contributions: tuple[AdapterSourceProjection, ...],
|
|
||||||
) -> AdapterAssembly:
|
|
||||||
expected = tuple(source.source_id for source in manifest.sources)
|
|
||||||
actual = tuple(sorted(contribution.source_id for contribution in contributions))
|
|
||||||
if actual != expected:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter",
|
|
||||||
"Python assembly contributions do not match the current manifest",
|
|
||||||
)
|
|
||||||
nodes = tuple(
|
|
||||||
sorted(
|
|
||||||
(node for contribution in contributions for node in contribution.nodes),
|
|
||||||
key=lambda item: item.node.node_id,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
edges = tuple(
|
|
||||||
sorted(
|
|
||||||
(edge for contribution in contributions for edge in contribution.edges),
|
|
||||||
key=lambda item: (
|
|
||||||
item.edge.source_id,
|
|
||||||
item.edge.relation,
|
|
||||||
item.edge.target_id,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
logic = tuple(
|
|
||||||
sorted(
|
|
||||||
(projection for contribution in contributions for projection in contribution.logic),
|
|
||||||
key=lambda projection: projection.owner_node_id,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return AdapterAssembly(
|
|
||||||
projection=AdapterProjection(
|
|
||||||
project_id=manifest.project_id,
|
|
||||||
title=manifest.title,
|
|
||||||
adapter_id=manifest.adapter_id,
|
|
||||||
adapter_version=manifest.adapter_version,
|
|
||||||
root=manifest.root,
|
|
||||||
revision=manifest.revision,
|
|
||||||
source_hash=manifest.source_hash,
|
|
||||||
nodes=nodes,
|
|
||||||
edges=edges,
|
|
||||||
),
|
|
||||||
logic=logic,
|
|
||||||
)
|
|
||||||
|
|
||||||
def load_assembly(self) -> AdapterAssembly:
|
|
||||||
"""Load one cache-independent complete graph and Logic assembly."""
|
|
||||||
|
|
||||||
manifest = self.load_manifest()
|
|
||||||
contributions = tuple(self.extract_source(source) for source in manifest.sources)
|
|
||||||
stable = self.load_manifest()
|
|
||||||
if stable != manifest:
|
|
||||||
raise DocForgeError(
|
|
||||||
"source_changed",
|
|
||||||
"Python sources changed during complete adapter extraction",
|
|
||||||
)
|
|
||||||
return self.assemble_projection(manifest, contributions)
|
|
||||||
|
|
||||||
def load_complete_assembly(self) -> AdapterAssembly:
|
|
||||||
"""Implement the SDK's independent complete-assembly oracle."""
|
|
||||||
|
|
||||||
return self.load_assembly()
|
|
||||||
|
|
||||||
def load_projection(self) -> AdapterProjection:
|
|
||||||
"""Preserve the first-class legacy complete-projection contract."""
|
|
||||||
|
|
||||||
return self.load_assembly().projection
|
|
||||||
|
|
||||||
def _normalize_source_roots(self, source_roots: Iterable[str | Path]) -> tuple[str, ...]:
|
|
||||||
normalized: list[str] = []
|
|
||||||
for value in source_roots:
|
|
||||||
candidate = PurePosixPath(Path(value).as_posix())
|
|
||||||
if (
|
|
||||||
candidate.is_absolute()
|
|
||||||
or not candidate.parts
|
|
||||||
or ".." in candidate.parts
|
|
||||||
or str(candidate) in {"", "."}
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"path_escape",
|
|
||||||
"Python source roots must be explicit project-relative directories",
|
|
||||||
)
|
|
||||||
relative = candidate.as_posix()
|
|
||||||
absolute = self.root.joinpath(*candidate.parts)
|
|
||||||
if absolute.is_symlink():
|
|
||||||
raise DocForgeError("path_escape", "Python source root cannot be a symbolic link")
|
|
||||||
try:
|
|
||||||
resolved = absolute.resolve(strict=True)
|
|
||||||
except OSError as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter",
|
|
||||||
"Python source root does not exist",
|
|
||||||
source_root=relative,
|
|
||||||
) from error
|
|
||||||
if (
|
|
||||||
resolved != absolute
|
|
||||||
or not resolved.is_dir()
|
|
||||||
or not resolved.is_relative_to(self.root)
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"path_escape",
|
|
||||||
"Python source root must be a confined real directory",
|
|
||||||
source_root=relative,
|
|
||||||
)
|
|
||||||
normalized.append(relative)
|
|
||||||
result = tuple(sorted(set(normalized)))
|
|
||||||
if not result:
|
|
||||||
raise DocForgeError("invalid_adapter", "At least one Python source root is required")
|
|
||||||
if len(result) != len(normalized):
|
|
||||||
raise DocForgeError("invalid_adapter", "Python source roots must be unique")
|
|
||||||
paths = [PurePosixPath(value) for value in result]
|
|
||||||
for index, left in enumerate(paths):
|
|
||||||
for right in paths[index + 1 :]:
|
|
||||||
if left in right.parents or right in left.parents:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter",
|
|
||||||
"Python source roots must not overlap",
|
|
||||||
)
|
|
||||||
return result
|
|
||||||
|
|
||||||
def _inventory(self, *, parse: bool = True) -> tuple[_SourceRecord, ...]:
|
|
||||||
paths: list[str] = []
|
|
||||||
for source_root in self.source_roots:
|
|
||||||
absolute_root = self.root.joinpath(*PurePosixPath(source_root).parts)
|
|
||||||
if (
|
|
||||||
absolute_root.is_symlink()
|
|
||||||
or not absolute_root.is_dir()
|
|
||||||
or absolute_root.resolve(strict=True) != absolute_root
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"path_escape",
|
|
||||||
"Python source root changed after adapter binding",
|
|
||||||
source_root=source_root,
|
|
||||||
)
|
|
||||||
for directory, directory_names, file_names in os.walk(
|
|
||||||
absolute_root,
|
|
||||||
topdown=True,
|
|
||||||
followlinks=False,
|
|
||||||
):
|
|
||||||
current = Path(directory)
|
|
||||||
for name in tuple(directory_names):
|
|
||||||
child = current / name
|
|
||||||
if child.is_symlink():
|
|
||||||
raise DocForgeError(
|
|
||||||
"path_escape",
|
|
||||||
"Python source inventory contains a symbolic-link directory",
|
|
||||||
source=child.relative_to(self.root).as_posix(),
|
|
||||||
)
|
|
||||||
directory_names[:] = sorted(
|
|
||||||
name for name in directory_names if name != "__pycache__"
|
|
||||||
)
|
|
||||||
for name in sorted(file_names):
|
|
||||||
if not name.endswith(".py"):
|
|
||||||
continue
|
|
||||||
child = current / name
|
|
||||||
relative = child.relative_to(self.root).as_posix()
|
|
||||||
if child.is_symlink():
|
|
||||||
raise DocForgeError(
|
|
||||||
"path_escape",
|
|
||||||
"Python source inventory contains a symbolic-link file",
|
|
||||||
source=relative,
|
|
||||||
)
|
|
||||||
paths.append(relative)
|
|
||||||
if len(paths) > self.max_sources:
|
|
||||||
raise DocForgeError(
|
|
||||||
"adapter_too_large",
|
|
||||||
"Python source inventory exceeds the configured limit",
|
|
||||||
maximum=self.max_sources,
|
|
||||||
)
|
|
||||||
records = tuple(self._read_source(path, parse=parse) for path in sorted(paths))
|
|
||||||
modules = [record.module_name for record in records]
|
|
||||||
if len(modules) != len(set(modules)):
|
|
||||||
duplicates = sorted(module for module in set(modules) if modules.count(module) > 1)
|
|
||||||
raise DocForgeError(
|
|
||||||
"ambiguous_python_module",
|
|
||||||
"Declared Python source roots produce duplicate module names",
|
|
||||||
modules=duplicates,
|
|
||||||
)
|
|
||||||
return records
|
|
||||||
|
|
||||||
def _read_source(self, relative: str, *, parse: bool = True) -> _SourceRecord:
|
|
||||||
path = PurePosixPath(relative)
|
|
||||||
if path.is_absolute() or ".." in path.parts or path.suffix != ".py":
|
|
||||||
raise DocForgeError("path_escape", "Python source path is unsafe")
|
|
||||||
source_root = self._source_root_for(path)
|
|
||||||
candidate = self.root.joinpath(*path.parts)
|
|
||||||
try:
|
|
||||||
before = candidate.lstat()
|
|
||||||
except OSError as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"stale_adapter_source",
|
|
||||||
"Python source is unavailable",
|
|
||||||
source=relative,
|
|
||||||
) from error
|
|
||||||
if (
|
|
||||||
stat.S_ISLNK(before.st_mode)
|
|
||||||
or not stat.S_ISREG(before.st_mode)
|
|
||||||
or candidate.resolve(strict=True) != candidate
|
|
||||||
or not candidate.is_relative_to(self.root)
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"path_escape",
|
|
||||||
"Python source must be a confined regular file",
|
|
||||||
source=relative,
|
|
||||||
)
|
|
||||||
if before.st_size > self.max_source_bytes:
|
|
||||||
raise DocForgeError(
|
|
||||||
"source_too_large",
|
|
||||||
"Python source exceeds the configured adapter limit",
|
|
||||||
source=relative,
|
|
||||||
maximum=self.max_source_bytes,
|
|
||||||
)
|
|
||||||
descriptor = -1
|
|
||||||
try:
|
|
||||||
flags = os.O_RDONLY | getattr(os, "O_BINARY", 0)
|
|
||||||
flags |= getattr(os, "O_NOFOLLOW", 0)
|
|
||||||
descriptor = os.open(candidate, flags)
|
|
||||||
opened = os.fstat(descriptor)
|
|
||||||
if not stat.S_ISREG(opened.st_mode) or (opened.st_dev, opened.st_ino) != (
|
|
||||||
before.st_dev,
|
|
||||||
before.st_ino,
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"path_escape",
|
|
||||||
"Python source identity changed before it was opened",
|
|
||||||
source=relative,
|
|
||||||
)
|
|
||||||
with os.fdopen(descriptor, "rb") as handle:
|
|
||||||
descriptor = -1
|
|
||||||
raw = handle.read(self.max_source_bytes + 1)
|
|
||||||
after_open = os.fstat(handle.fileno())
|
|
||||||
after = candidate.lstat()
|
|
||||||
except DocForgeError:
|
|
||||||
raise
|
|
||||||
except OSError as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"stale_adapter_source",
|
|
||||||
"Python source changed while being read",
|
|
||||||
source=relative,
|
|
||||||
) from error
|
|
||||||
finally:
|
|
||||||
if descriptor >= 0:
|
|
||||||
os.close(descriptor)
|
|
||||||
identity_before = (
|
|
||||||
before.st_dev,
|
|
||||||
before.st_ino,
|
|
||||||
before.st_size,
|
|
||||||
before.st_mtime_ns,
|
|
||||||
)
|
|
||||||
identity_after = (
|
|
||||||
after.st_dev,
|
|
||||||
after.st_ino,
|
|
||||||
after.st_size,
|
|
||||||
after.st_mtime_ns,
|
|
||||||
)
|
|
||||||
identity_opened = (
|
|
||||||
after_open.st_dev,
|
|
||||||
after_open.st_ino,
|
|
||||||
after_open.st_size,
|
|
||||||
after_open.st_mtime_ns,
|
|
||||||
)
|
|
||||||
if (
|
|
||||||
len(raw) > self.max_source_bytes
|
|
||||||
or identity_before != identity_opened
|
|
||||||
or identity_before != identity_after
|
|
||||||
or len(raw) != after.st_size
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"stale_adapter_source",
|
|
||||||
"Python source changed while being read",
|
|
||||||
source=relative,
|
|
||||||
)
|
|
||||||
text = self._decode_source(raw, relative)
|
|
||||||
tree: ast.Module | None = None
|
|
||||||
if parse:
|
|
||||||
try:
|
|
||||||
tree = ast.parse(text, filename=relative, type_comments=True)
|
|
||||||
except SyntaxError as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_python_source",
|
|
||||||
"Python source cannot be parsed by the reference adapter",
|
|
||||||
source=relative,
|
|
||||||
line=error.lineno,
|
|
||||||
) from error
|
|
||||||
module_name = self._module_name(path, PurePosixPath(source_root))
|
|
||||||
return _SourceRecord(
|
|
||||||
source_id=self._source_id(relative),
|
|
||||||
source_path=relative,
|
|
||||||
module_name=module_name,
|
|
||||||
fingerprint=hashlib.sha256(raw).hexdigest(),
|
|
||||||
raw=raw,
|
|
||||||
text=text,
|
|
||||||
tree=tree,
|
|
||||||
)
|
|
||||||
|
|
||||||
def _source_root_for(self, path: PurePosixPath) -> str:
|
|
||||||
matches = [
|
|
||||||
source_root
|
|
||||||
for source_root in self.source_roots
|
|
||||||
if path == PurePosixPath(source_root) or PurePosixPath(source_root) in path.parents
|
|
||||||
]
|
|
||||||
if len(matches) != 1:
|
|
||||||
raise DocForgeError(
|
|
||||||
"path_escape",
|
|
||||||
"Python source is outside the declared source roots",
|
|
||||||
source=path.as_posix(),
|
|
||||||
)
|
|
||||||
return matches[0]
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _decode_source(raw: bytes, source_path: str) -> str:
|
|
||||||
try:
|
|
||||||
encoding, _ = tokenize.detect_encoding(io.BytesIO(raw).readline)
|
|
||||||
return raw.decode(encoding)
|
|
||||||
except (LookupError, SyntaxError, UnicodeDecodeError) as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_python_source",
|
|
||||||
"Python source encoding is invalid",
|
|
||||||
source=source_path,
|
|
||||||
) from error
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _module_name(path: PurePosixPath, source_root: PurePosixPath) -> str:
|
|
||||||
local = path.relative_to(source_root)
|
|
||||||
parts = list(local.with_suffix("").parts)
|
|
||||||
if parts and parts[-1] == "__init__":
|
|
||||||
parts.pop()
|
|
||||||
if not parts:
|
|
||||||
parts = list(source_root.parts)
|
|
||||||
return ".".join(parts)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _source_id(relative: str) -> str:
|
|
||||||
digest = hashlib.sha256(
|
|
||||||
f"{PYTHON_IDENTITY_VERSION}\0source\0{relative}".encode()
|
|
||||||
).hexdigest()[:24]
|
|
||||||
return f"python.source.{digest}"
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _file_node_id(source_id: str) -> str:
|
|
||||||
return source_id.replace("python.source.", "python.file.", 1)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _module_node_id(source_id: str) -> str:
|
|
||||||
return source_id.replace("python.source.", "python.module.", 1)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _symbol_node_id(kind: str, source_id: str, qualified_name: str) -> str:
|
|
||||||
digest = hashlib.sha256(
|
|
||||||
(f"{PYTHON_IDENTITY_VERSION}\0{kind}\0{source_id}\0{qualified_name}").encode()
|
|
||||||
).hexdigest()[:24]
|
|
||||||
return f"python.{kind}.{digest}"
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _source_hash(sources: Sequence[AdapterSource]) -> str:
|
|
||||||
digest = hashlib.sha256()
|
|
||||||
digest.update(PYTHON_ADAPTER_VERSION.encode())
|
|
||||||
digest.update(PYTHON_EXTRACTOR_VERSION.encode())
|
|
||||||
digest.update(PYTHON_IDENTITY_VERSION.encode())
|
|
||||||
for source in sources:
|
|
||||||
digest.update(b"\0source\0")
|
|
||||||
digest.update(source.source_id.encode())
|
|
||||||
digest.update(b"\0path\0")
|
|
||||||
digest.update(source.source_path.encode())
|
|
||||||
digest.update(b"\0fingerprint\0")
|
|
||||||
digest.update(source.fingerprint.encode())
|
|
||||||
for dependency in source.dependencies:
|
|
||||||
digest.update(b"\0dependency\0")
|
|
||||||
digest.update(dependency.encode())
|
|
||||||
return digest.hexdigest()
|
|
||||||
|
|
||||||
def _local_dependencies(
|
|
||||||
self,
|
|
||||||
record: _SourceRecord,
|
|
||||||
module_sources: dict[str, str],
|
|
||||||
) -> tuple[str, ...]:
|
|
||||||
dependencies: set[str] = set()
|
|
||||||
package = (
|
|
||||||
record.module_name
|
|
||||||
if record.source_path.endswith("/__init__.py")
|
|
||||||
else record.module_name.rpartition(".")[0]
|
|
||||||
)
|
|
||||||
for level, module, imported_names in self._lexical_imports(record):
|
|
||||||
candidates: list[str] = []
|
|
||||||
if not imported_names and level == 0 and module is not None:
|
|
||||||
candidates.append(module)
|
|
||||||
else:
|
|
||||||
base = self._import_from_base(package, level, module)
|
|
||||||
if base:
|
|
||||||
candidates.append(base)
|
|
||||||
candidates.extend(f"{base}.{name}" for name in imported_names if name != "*")
|
|
||||||
for module_name in candidates:
|
|
||||||
target = module_sources.get(module_name)
|
|
||||||
if target is not None and target != record.source_id:
|
|
||||||
dependencies.add(target)
|
|
||||||
return tuple(sorted(dependencies))
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _lexical_imports(
|
|
||||||
record: _SourceRecord,
|
|
||||||
) -> tuple[tuple[int, str | None, tuple[str, ...]], ...]:
|
|
||||||
"""Inventory import dependencies without constructing a Python AST."""
|
|
||||||
|
|
||||||
try:
|
|
||||||
tokens = tuple(tokenize.generate_tokens(io.StringIO(record.text).readline))
|
|
||||||
except (IndentationError, tokenize.TokenError) as error:
|
|
||||||
line = error.args[1][0] if len(error.args) > 1 else None
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_python_source",
|
|
||||||
"Python source cannot be tokenized by the reference adapter",
|
|
||||||
source=record.source_path,
|
|
||||||
line=line,
|
|
||||||
) from error
|
|
||||||
|
|
||||||
statements: list[list[tokenize.TokenInfo]] = []
|
|
||||||
current: list[tokenize.TokenInfo] = []
|
|
||||||
nesting = 0
|
|
||||||
ignored = {
|
|
||||||
tokenize.ENCODING,
|
|
||||||
tokenize.INDENT,
|
|
||||||
tokenize.DEDENT,
|
|
||||||
tokenize.NL,
|
|
||||||
tokenize.COMMENT,
|
|
||||||
}
|
|
||||||
for token in tokens:
|
|
||||||
if token.type in ignored:
|
|
||||||
continue
|
|
||||||
if token.type == tokenize.OP:
|
|
||||||
if token.string in "([{":
|
|
||||||
nesting += 1
|
|
||||||
elif token.string in ")]}":
|
|
||||||
nesting = max(0, nesting - 1)
|
|
||||||
elif token.string == ";" and nesting == 0:
|
|
||||||
if current:
|
|
||||||
statements.append(current)
|
|
||||||
current = []
|
|
||||||
continue
|
|
||||||
if token.type in {tokenize.NEWLINE, tokenize.ENDMARKER} and nesting == 0:
|
|
||||||
if current:
|
|
||||||
statements.append(current)
|
|
||||||
current = []
|
|
||||||
continue
|
|
||||||
current.append(token)
|
|
||||||
|
|
||||||
imports: list[tuple[int, str | None, tuple[str, ...]]] = []
|
|
||||||
for statement in statements:
|
|
||||||
for position, token in enumerate(statement):
|
|
||||||
if token.type != tokenize.NAME or token.string not in {"import", "from"}:
|
|
||||||
continue
|
|
||||||
if token.string == "import":
|
|
||||||
names = PythonReferenceAdapter._imported_names(statement[position + 1 :])
|
|
||||||
imports.extend((0, name, ()) for name in names)
|
|
||||||
break
|
|
||||||
import_position = next(
|
|
||||||
(
|
|
||||||
index
|
|
||||||
for index in range(position + 1, len(statement))
|
|
||||||
if statement[index].type == tokenize.NAME
|
|
||||||
and statement[index].string == "import"
|
|
||||||
),
|
|
||||||
None,
|
|
||||||
)
|
|
||||||
if import_position is None:
|
|
||||||
break
|
|
||||||
prefix = statement[position + 1 : import_position]
|
|
||||||
level = 0
|
|
||||||
for item in prefix:
|
|
||||||
if item.type == tokenize.NAME:
|
|
||||||
break
|
|
||||||
if item.type == tokenize.OP and set(item.string) == {"."}:
|
|
||||||
level += len(item.string)
|
|
||||||
module_parts = [item.string for item in prefix if item.type == tokenize.NAME]
|
|
||||||
module = ".".join(module_parts) or None
|
|
||||||
names = PythonReferenceAdapter._imported_names(statement[import_position + 1 :])
|
|
||||||
imports.append((level, module, names))
|
|
||||||
break
|
|
||||||
return tuple(imports)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _imported_names(tokens: Sequence[tokenize.TokenInfo]) -> tuple[str, ...]:
|
|
||||||
names: list[str] = []
|
|
||||||
current: list[str] = []
|
|
||||||
skip_alias = False
|
|
||||||
for token in tokens:
|
|
||||||
if token.type == tokenize.NAME and token.string == "as":
|
|
||||||
skip_alias = True
|
|
||||||
continue
|
|
||||||
if token.type == tokenize.OP and token.string == ",":
|
|
||||||
if current:
|
|
||||||
names.append(".".join(current))
|
|
||||||
current = []
|
|
||||||
skip_alias = False
|
|
||||||
continue
|
|
||||||
if token.type == tokenize.OP and token.string == "*":
|
|
||||||
if not skip_alias:
|
|
||||||
current.append("*")
|
|
||||||
continue
|
|
||||||
if token.type == tokenize.NAME and not skip_alias:
|
|
||||||
current.append(token.string)
|
|
||||||
if current:
|
|
||||||
names.append(".".join(current))
|
|
||||||
return tuple(name for name in names if name)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _import_from_base(package: str, level: int, module: str | None) -> str:
|
|
||||||
if level == 0:
|
|
||||||
return module or ""
|
|
||||||
package_parts = package.split(".") if package else []
|
|
||||||
keep = len(package_parts) - (level - 1)
|
|
||||||
if keep < 0:
|
|
||||||
return ""
|
|
||||||
prefix = package_parts[:keep]
|
|
||||||
if module:
|
|
||||||
prefix.extend(module.split("."))
|
|
||||||
return ".".join(prefix)
|
|
||||||
|
|
||||||
def _extract_record(
|
|
||||||
self,
|
|
||||||
record: _SourceRecord,
|
|
||||||
source: AdapterSource,
|
|
||||||
) -> AdapterSourceProjection:
|
|
||||||
tree = record.tree
|
|
||||||
if tree is None:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter",
|
|
||||||
"Python extraction requires a parsed source record",
|
|
||||||
source=record.source_path,
|
|
||||||
)
|
|
||||||
file_id = self._file_node_id(source.source_id)
|
|
||||||
module_id = self._module_node_id(source.source_id)
|
|
||||||
nodes: list[AdapterNode] = [
|
|
||||||
self._node(
|
|
||||||
node_id=file_id,
|
|
||||||
title=record.source_path,
|
|
||||||
kind="file",
|
|
||||||
qualified_name=record.source_path,
|
|
||||||
content=record.text,
|
|
||||||
source_path=record.source_path,
|
|
||||||
anchor="L1",
|
|
||||||
),
|
|
||||||
self._node(
|
|
||||||
node_id=module_id,
|
|
||||||
title=record.module_name,
|
|
||||||
kind="module",
|
|
||||||
qualified_name=record.module_name,
|
|
||||||
content=(
|
|
||||||
ast.get_docstring(tree, clean=False) or f"Python module {record.module_name}."
|
|
||||||
),
|
|
||||||
source_path=record.source_path,
|
|
||||||
anchor="L1",
|
|
||||||
),
|
|
||||||
]
|
|
||||||
edges: list[AdapterEdge] = [self._edge(file_id, "contains", module_id, "syntax")]
|
|
||||||
definitions = self._definitions(record, module_id)
|
|
||||||
owners: list[PythonLogicOwner] = []
|
|
||||||
for definition in definitions:
|
|
||||||
content = ast.get_source_segment(record.text, definition.node)
|
|
||||||
if content is None or not content.strip():
|
|
||||||
content = f"Python {definition.kind} {definition.qualified_name}."
|
|
||||||
anchor = f"L{definition.node.lineno}"
|
|
||||||
nodes.append(
|
|
||||||
self._node(
|
|
||||||
node_id=definition.node_id,
|
|
||||||
title=definition.qualified_name,
|
|
||||||
kind=definition.kind,
|
|
||||||
qualified_name=f"{record.module_name}.{definition.qualified_name}",
|
|
||||||
content=content,
|
|
||||||
source_path=record.source_path,
|
|
||||||
anchor=anchor,
|
|
||||||
asynchronous=isinstance(definition.node, ast.AsyncFunctionDef),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
edges.append(
|
|
||||||
self._edge(
|
|
||||||
definition.parent_node_id,
|
|
||||||
"contains",
|
|
||||||
definition.node_id,
|
|
||||||
"syntax",
|
|
||||||
anchor=anchor,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
if definition.kind == "function":
|
|
||||||
owners.append(
|
|
||||||
PythonLogicOwner(
|
|
||||||
owner_node_id=definition.node_id,
|
|
||||||
qualified_name=definition.qualified_name,
|
|
||||||
line=definition.node.lineno,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
for dependency in source.dependencies:
|
|
||||||
edges.append(
|
|
||||||
self._edge(
|
|
||||||
module_id,
|
|
||||||
"depends_on",
|
|
||||||
self._module_node_id(dependency),
|
|
||||||
"local_import",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
logic = analyze_python_source(
|
|
||||||
record.text,
|
|
||||||
source_id=source.source_id,
|
|
||||||
owners=owners,
|
|
||||||
filename=record.source_path,
|
|
||||||
max_nodes_per_function=self.max_logic_nodes_per_function,
|
|
||||||
)
|
|
||||||
return AdapterSourceProjection(
|
|
||||||
source_id=source.source_id,
|
|
||||||
fingerprint=source.fingerprint,
|
|
||||||
nodes=tuple(sorted(nodes, key=lambda item: item.node.node_id)),
|
|
||||||
edges=tuple(
|
|
||||||
sorted(
|
|
||||||
edges,
|
|
||||||
key=lambda item: (
|
|
||||||
item.edge.source_id,
|
|
||||||
item.edge.relation,
|
|
||||||
item.edge.target_id,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
),
|
|
||||||
logic=logic,
|
|
||||||
)
|
|
||||||
|
|
||||||
def _definitions(
|
|
||||||
self,
|
|
||||||
record: _SourceRecord,
|
|
||||||
module_node_id: str,
|
|
||||||
) -> tuple[_Definition, ...]:
|
|
||||||
tree = record.tree
|
|
||||||
if tree is None:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter",
|
|
||||||
"Python definition extraction requires a parsed source record",
|
|
||||||
source=record.source_path,
|
|
||||||
)
|
|
||||||
definitions: list[_Definition] = []
|
|
||||||
qualified_names: set[str] = set()
|
|
||||||
adapter = self
|
|
||||||
|
|
||||||
class Collector(ast.NodeVisitor):
|
|
||||||
def __init__(self) -> None:
|
|
||||||
self.scope_names: list[str] = []
|
|
||||||
self.scope_node_ids: list[str] = [module_node_id]
|
|
||||||
|
|
||||||
def _definition(
|
|
||||||
self,
|
|
||||||
kind: str,
|
|
||||||
node: ast.ClassDef | ast.FunctionDef | ast.AsyncFunctionDef,
|
|
||||||
) -> None:
|
|
||||||
qualified_name = ".".join((*self.scope_names, node.name))
|
|
||||||
if qualified_name in qualified_names:
|
|
||||||
raise DocForgeError(
|
|
||||||
"ambiguous_python_symbol",
|
|
||||||
"Python source repeats a class or function identity",
|
|
||||||
source=record.source_path,
|
|
||||||
qualified_name=qualified_name,
|
|
||||||
)
|
|
||||||
qualified_names.add(qualified_name)
|
|
||||||
node_id = adapter._symbol_node_id(
|
|
||||||
kind,
|
|
||||||
record.source_id,
|
|
||||||
qualified_name,
|
|
||||||
)
|
|
||||||
definitions.append(
|
|
||||||
_Definition(
|
|
||||||
kind=kind,
|
|
||||||
qualified_name=qualified_name,
|
|
||||||
parent_node_id=self.scope_node_ids[-1],
|
|
||||||
node=node,
|
|
||||||
node_id=node_id,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
self.scope_names.append(node.name)
|
|
||||||
self.scope_node_ids.append(node_id)
|
|
||||||
self.generic_visit(node)
|
|
||||||
self.scope_node_ids.pop()
|
|
||||||
self.scope_names.pop()
|
|
||||||
|
|
||||||
def visit_ClassDef(self, node: ast.ClassDef) -> None:
|
|
||||||
self._definition("class", node)
|
|
||||||
|
|
||||||
def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
|
|
||||||
self._definition("function", node)
|
|
||||||
|
|
||||||
def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None:
|
|
||||||
self._definition("function", node)
|
|
||||||
|
|
||||||
Collector().visit(tree)
|
|
||||||
return tuple(sorted(definitions, key=lambda item: item.node_id))
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _node(
|
|
||||||
*,
|
|
||||||
node_id: str,
|
|
||||||
title: str,
|
|
||||||
kind: str,
|
|
||||||
qualified_name: str,
|
|
||||||
content: str,
|
|
||||||
source_path: str,
|
|
||||||
anchor: str,
|
|
||||||
asynchronous: bool = False,
|
|
||||||
) -> AdapterNode:
|
|
||||||
normalized = content.strip() or f"Python {kind} {qualified_name}."
|
|
||||||
tags = tuple(sorted({"python", kind, *(("async",) if asynchronous else ())}))
|
|
||||||
return AdapterNode(
|
|
||||||
node=Node(
|
|
||||||
node_id=node_id,
|
|
||||||
title=title,
|
|
||||||
family="code",
|
|
||||||
authority="derived",
|
|
||||||
status="active",
|
|
||||||
tags=tags,
|
|
||||||
summary=f"Python {kind} fact for {qualified_name}.",
|
|
||||||
content=normalized,
|
|
||||||
source_path=source_path,
|
|
||||||
source_anchor=anchor,
|
|
||||||
content_hash=hashlib.sha256(normalized.encode()).hexdigest(),
|
|
||||||
),
|
|
||||||
metadata=(
|
|
||||||
("extractor", PYTHON_EXTRACTOR_VERSION),
|
|
||||||
("identity", PYTHON_IDENTITY_VERSION),
|
|
||||||
("kind", kind),
|
|
||||||
("qualified_name", qualified_name),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _edge(
|
|
||||||
source_id: str,
|
|
||||||
relation: str,
|
|
||||||
target_id: str,
|
|
||||||
evidence: str,
|
|
||||||
*,
|
|
||||||
anchor: str | None = None,
|
|
||||||
) -> AdapterEdge:
|
|
||||||
metadata = [("evidence", evidence)]
|
|
||||||
if anchor is not None:
|
|
||||||
metadata.append(("source_anchor", anchor))
|
|
||||||
return AdapterEdge(
|
|
||||||
edge=Edge(source_id, relation, target_id),
|
|
||||||
metadata=tuple(metadata),
|
|
||||||
)
|
|
||||||
|
|
@ -16,12 +16,6 @@ from dataclasses import dataclass
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Literal, cast
|
from typing import Literal, cast
|
||||||
|
|
||||||
from .adapter_launcher import (
|
|
||||||
AdapterLauncherV1,
|
|
||||||
AdapterSourceAvailabilityV1,
|
|
||||||
adapter_source_availability,
|
|
||||||
validate_adapter_launcher,
|
|
||||||
)
|
|
||||||
from .changeset_contract import document_hash
|
from .changeset_contract import document_hash
|
||||||
from .errors import DocForgeError
|
from .errors import DocForgeError
|
||||||
from .models import ProjectDescriptor, ProjectService
|
from .models import ProjectDescriptor, ProjectService
|
||||||
|
|
@ -1085,450 +1079,3 @@ def generate_client_configuration(
|
||||||
}
|
}
|
||||||
_validate_configuration_result(result, trusted_descriptor=descriptor)
|
_validate_configuration_result(result, trusted_descriptor=descriptor)
|
||||||
return result
|
return result
|
||||||
|
|
||||||
|
|
||||||
def generate_adapter_client_configuration(
|
|
||||||
project: ProjectService,
|
|
||||||
launcher: AdapterLauncherV1,
|
|
||||||
client: str,
|
|
||||||
*,
|
|
||||||
server_name: str | None = None,
|
|
||||||
capability_mode: str = "read",
|
|
||||||
proposal_writer: str | None = None,
|
|
||||||
canonical_applier: str | None = None,
|
|
||||||
no_ast: bool = False,
|
|
||||||
manual_render_policy: str | None = None,
|
|
||||||
portable_graph_policy: str | None = None,
|
|
||||||
live_viewer_policy: str | None = None,
|
|
||||||
startup_timeout: int = 30,
|
|
||||||
tool_timeout: int = 300,
|
|
||||||
output: Path | None = None,
|
|
||||||
) -> dict[str, object]:
|
|
||||||
"""Generate one client fragment for an explicitly constructed project-owned adapter."""
|
|
||||||
|
|
||||||
validate_adapter_launcher(project, launcher)
|
|
||||||
selected_client = _client_name(client)
|
|
||||||
selected_mode = _capability_mode(capability_mode)
|
|
||||||
startup_seconds = _bounded_seconds(
|
|
||||||
startup_timeout,
|
|
||||||
field="startup_timeout",
|
|
||||||
maximum=3_600,
|
|
||||||
)
|
|
||||||
tool_seconds = _bounded_seconds(
|
|
||||||
tool_timeout,
|
|
||||||
field="tool_timeout",
|
|
||||||
maximum=86_400,
|
|
||||||
)
|
|
||||||
descriptor = project.descriptor
|
|
||||||
_validate_adapter_authority(
|
|
||||||
descriptor,
|
|
||||||
selected_mode=selected_mode,
|
|
||||||
proposal_writer=proposal_writer,
|
|
||||||
canonical_applier=canonical_applier,
|
|
||||||
)
|
|
||||||
source_availability = adapter_source_availability(project, launcher)
|
|
||||||
fingerprint = project_root_fingerprint(descriptor.root)
|
|
||||||
selected_name = _validated_server_name(
|
|
||||||
server_name,
|
|
||||||
project_id=descriptor.project_id,
|
|
||||||
fingerprint=fingerprint,
|
|
||||||
)
|
|
||||||
executable = Path(os.path.abspath(sys.executable))
|
|
||||||
try:
|
|
||||||
executable_status = executable.stat()
|
|
||||||
except OSError as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"client_configuration_unavailable",
|
|
||||||
"Current Python executable cannot be inspected",
|
|
||||||
) from error
|
|
||||||
if not stat.S_ISREG(executable_status.st_mode) or not os.access(executable, os.X_OK):
|
|
||||||
raise DocForgeError(
|
|
||||||
"client_configuration_unavailable",
|
|
||||||
"Current Python executable is not a runnable regular file",
|
|
||||||
)
|
|
||||||
|
|
||||||
policy = compose_effective_policy(
|
|
||||||
selected_mode=selected_mode,
|
|
||||||
capability_source="explicit",
|
|
||||||
no_ast=no_ast,
|
|
||||||
diagnostics=False,
|
|
||||||
render_configured=descriptor.render is not None,
|
|
||||||
application_enabled=canonical_applier is not None,
|
|
||||||
)
|
|
||||||
projection_policy = compose_projection_policy(
|
|
||||||
manual=manual_render_policy,
|
|
||||||
portable_graph=portable_graph_policy,
|
|
||||||
live_viewer=live_viewer_policy,
|
|
||||||
manual_configured=descriptor.render is not None,
|
|
||||||
portable_graph_configured=descriptor.graph_render is not None,
|
|
||||||
application_enabled=canonical_applier is not None,
|
|
||||||
)
|
|
||||||
default_projection_policy = compose_projection_policy(
|
|
||||||
manual_configured=descriptor.render is not None,
|
|
||||||
portable_graph_configured=descriptor.graph_render is not None,
|
|
||||||
application_enabled=canonical_applier is not None,
|
|
||||||
)
|
|
||||||
arguments = [
|
|
||||||
"-I",
|
|
||||||
"-m",
|
|
||||||
launcher.module,
|
|
||||||
"--project-root",
|
|
||||||
str(descriptor.root),
|
|
||||||
"--capability-mode",
|
|
||||||
selected_mode,
|
|
||||||
]
|
|
||||||
if proposal_writer is not None:
|
|
||||||
arguments.extend(("--proposal-writer", proposal_writer))
|
|
||||||
if canonical_applier is not None:
|
|
||||||
arguments.extend(("--canonical-applier", canonical_applier))
|
|
||||||
for option, selected, default in (
|
|
||||||
(
|
|
||||||
"--manual-render-policy",
|
|
||||||
projection_policy.manual,
|
|
||||||
default_projection_policy.manual,
|
|
||||||
),
|
|
||||||
(
|
|
||||||
"--portable-graph-policy",
|
|
||||||
projection_policy.portable_graph,
|
|
||||||
default_projection_policy.portable_graph,
|
|
||||||
),
|
|
||||||
(
|
|
||||||
"--live-viewer-policy",
|
|
||||||
projection_policy.live_viewer,
|
|
||||||
default_projection_policy.live_viewer,
|
|
||||||
),
|
|
||||||
):
|
|
||||||
if selected != default:
|
|
||||||
arguments.extend((option, selected))
|
|
||||||
if no_ast:
|
|
||||||
arguments.append("--no-ast")
|
|
||||||
|
|
||||||
artifact_format, content, warning = _artifact(
|
|
||||||
selected_client,
|
|
||||||
server_name=selected_name,
|
|
||||||
command=str(executable),
|
|
||||||
arguments=arguments,
|
|
||||||
startup_timeout=startup_seconds,
|
|
||||||
tool_timeout=tool_seconds,
|
|
||||||
)
|
|
||||||
|
|
||||||
def validate_current_binding() -> None:
|
|
||||||
validate_adapter_launcher(project, launcher)
|
|
||||||
current = adapter_source_availability(project, launcher)
|
|
||||||
if current != source_availability:
|
|
||||||
raise DocForgeError(
|
|
||||||
"source_changed",
|
|
||||||
"Adapter sources changed during client configuration generation",
|
|
||||||
)
|
|
||||||
|
|
||||||
if output is None:
|
|
||||||
validate_current_binding()
|
|
||||||
write_state = "not_requested"
|
|
||||||
durability = "not_applicable"
|
|
||||||
publication_warning = None
|
|
||||||
output_path = None
|
|
||||||
else:
|
|
||||||
validate_current_binding()
|
|
||||||
write_state, durability, publication_warning, published_path = _atomic_write(
|
|
||||||
output,
|
|
||||||
content,
|
|
||||||
validate_binding=validate_current_binding,
|
|
||||||
)
|
|
||||||
output_path = str(published_path) if published_path is not None else None
|
|
||||||
artifact = {
|
|
||||||
"format": artifact_format,
|
|
||||||
"content": content,
|
|
||||||
"content_sha256": hashlib.sha256(content.encode("utf-8")).hexdigest(),
|
|
||||||
"output_path": output_path,
|
|
||||||
"write_state": write_state,
|
|
||||||
"durability": durability,
|
|
||||||
}
|
|
||||||
binding = {
|
|
||||||
"transport": "stdio",
|
|
||||||
"capability_mode": selected_mode,
|
|
||||||
"adapter_policy": policy.adapter_policy(),
|
|
||||||
"render_policy": {
|
|
||||||
"manual": policy.manual_render,
|
|
||||||
"graph": policy.graph_render,
|
|
||||||
"live_viewer": policy.live_viewer,
|
|
||||||
},
|
|
||||||
"command": str(executable),
|
|
||||||
"args": arguments,
|
|
||||||
"environment": {},
|
|
||||||
"timeouts": {
|
|
||||||
"startup_seconds": startup_seconds,
|
|
||||||
"tool_seconds": tool_seconds,
|
|
||||||
},
|
|
||||||
"launcher_hash": launcher.launcher_hash,
|
|
||||||
}
|
|
||||||
project_binding = {
|
|
||||||
"project_id": descriptor.project_id,
|
|
||||||
"project_root": str(descriptor.root),
|
|
||||||
"project_root_fingerprint": fingerprint,
|
|
||||||
"adapter": descriptor.adapter,
|
|
||||||
"descriptor_hash": descriptor.descriptor_hash,
|
|
||||||
}
|
|
||||||
projection_availability = {
|
|
||||||
"manual_configured": descriptor.render is not None,
|
|
||||||
"portable_graph_configured": descriptor.graph_render is not None,
|
|
||||||
"application_enabled": canonical_applier is not None,
|
|
||||||
"live_viewer_available": True,
|
|
||||||
}
|
|
||||||
policy_payload = policy.as_dict()
|
|
||||||
result: dict[str, object] = {
|
|
||||||
"status": "ok",
|
|
||||||
"schema_version": 1,
|
|
||||||
"operation": "adapter_client.configure",
|
|
||||||
"action": "write" if output is not None else "preview",
|
|
||||||
"client": selected_client,
|
|
||||||
"server_name": selected_name,
|
|
||||||
"project": project_binding,
|
|
||||||
"launcher": launcher.as_dict(),
|
|
||||||
"launcher_hash": launcher.launcher_hash,
|
|
||||||
"source_availability": source_availability.as_dict(),
|
|
||||||
"source_availability_hash": source_availability.availability_hash,
|
|
||||||
"binding": binding,
|
|
||||||
"effective_policy": policy_payload,
|
|
||||||
"projection_policy": projection_policy.as_dict(),
|
|
||||||
"projection_policy_hash": projection_policy.policy_hash,
|
|
||||||
"projection_availability": projection_availability,
|
|
||||||
"artifact": artifact,
|
|
||||||
"warnings": [
|
|
||||||
*([] if warning is None else [{"code": "timeout_format_unverified"}]),
|
|
||||||
*([] if publication_warning is None else [{"code": publication_warning}]),
|
|
||||||
],
|
|
||||||
}
|
|
||||||
result["configuration_hash"] = document_hash(_adapter_configuration_hash_payload(result))
|
|
||||||
_validate_adapter_configuration_result(
|
|
||||||
result,
|
|
||||||
project=project,
|
|
||||||
launcher=launcher,
|
|
||||||
source_availability=source_availability,
|
|
||||||
)
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
def _validate_adapter_authority(
|
|
||||||
descriptor: ProjectDescriptor,
|
|
||||||
*,
|
|
||||||
selected_mode: CapabilityMode,
|
|
||||||
proposal_writer: str | None,
|
|
||||||
canonical_applier: str | None,
|
|
||||||
) -> None:
|
|
||||||
writer_ids = {writer.writer_id for writer in descriptor.proposal_writers}
|
|
||||||
if selected_mode == "read":
|
|
||||||
if proposal_writer is not None or canonical_applier is not None:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_capability_binding",
|
|
||||||
"Read configuration cannot bind proposal or application authority",
|
|
||||||
)
|
|
||||||
elif selected_mode == "proposal":
|
|
||||||
if proposal_writer is None or proposal_writer not in writer_ids:
|
|
||||||
raise DocForgeError(
|
|
||||||
"capability_unavailable",
|
|
||||||
"Proposal configuration requires a descriptor-declared writer",
|
|
||||||
required="proposal_writer",
|
|
||||||
)
|
|
||||||
if canonical_applier is not None:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_capability_binding",
|
|
||||||
"Proposal configuration cannot bind a canonical applier",
|
|
||||||
)
|
|
||||||
elif (
|
|
||||||
proposal_writer is None
|
|
||||||
or canonical_applier is None
|
|
||||||
or proposal_writer != canonical_applier
|
|
||||||
or proposal_writer not in writer_ids
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"capability_unavailable",
|
|
||||||
"Application configuration requires one declared writer/applier identity",
|
|
||||||
required="matching_declared_writer_and_applier",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _adapter_configuration_hash_payload(result: dict[str, object]) -> dict[str, object]:
|
|
||||||
artifact = cast(dict[str, object], result["artifact"])
|
|
||||||
return {
|
|
||||||
"schema_version": 1,
|
|
||||||
"client": result["client"],
|
|
||||||
"server_name": result["server_name"],
|
|
||||||
"project": result["project"],
|
|
||||||
"launcher": result["launcher"],
|
|
||||||
"launcher_hash": result["launcher_hash"],
|
|
||||||
"source_availability": result["source_availability"],
|
|
||||||
"source_availability_hash": result["source_availability_hash"],
|
|
||||||
"binding": result["binding"],
|
|
||||||
"effective_policy": result["effective_policy"],
|
|
||||||
"projection_policy": result["projection_policy"],
|
|
||||||
"projection_policy_hash": result["projection_policy_hash"],
|
|
||||||
"projection_availability": result["projection_availability"],
|
|
||||||
"artifact_format": artifact["format"],
|
|
||||||
"artifact_content_sha256": artifact["content_sha256"],
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _validate_adapter_configuration_result(
|
|
||||||
result: dict[str, object],
|
|
||||||
*,
|
|
||||||
project: ProjectService,
|
|
||||||
launcher: AdapterLauncherV1,
|
|
||||||
source_availability: AdapterSourceAvailabilityV1,
|
|
||||||
) -> None:
|
|
||||||
validate_adapter_launcher(project, launcher)
|
|
||||||
descriptor = project.descriptor
|
|
||||||
project_binding = cast(dict[str, object], result["project"])
|
|
||||||
if (
|
|
||||||
result["launcher"] != launcher.as_dict()
|
|
||||||
or result["launcher_hash"] != launcher.launcher_hash
|
|
||||||
or result["source_availability"] != source_availability.as_dict()
|
|
||||||
or result["source_availability_hash"] != source_availability.availability_hash
|
|
||||||
or project_binding
|
|
||||||
!= {
|
|
||||||
"project_id": descriptor.project_id,
|
|
||||||
"project_root": str(descriptor.root),
|
|
||||||
"project_root_fingerprint": project_root_fingerprint(descriptor.root),
|
|
||||||
"adapter": descriptor.adapter,
|
|
||||||
"descriptor_hash": descriptor.descriptor_hash,
|
|
||||||
}
|
|
||||||
):
|
|
||||||
raise AssertionError("Generated adapter client identity drifted")
|
|
||||||
|
|
||||||
artifact = cast(dict[str, object], result["artifact"])
|
|
||||||
binding = cast(dict[str, object], result["binding"])
|
|
||||||
content = cast(str, artifact["content"])
|
|
||||||
if artifact["content_sha256"] != hashlib.sha256(content.encode("utf-8")).hexdigest():
|
|
||||||
raise AssertionError("Generated adapter client content hash drifted")
|
|
||||||
timeouts = cast(dict[str, object], binding["timeouts"])
|
|
||||||
expected_format, expected_content, _ = _artifact(
|
|
||||||
cast(ClientName, result["client"]),
|
|
||||||
server_name=cast(str, result["server_name"]),
|
|
||||||
command=cast(str, binding["command"]),
|
|
||||||
arguments=cast(list[str], binding["args"]),
|
|
||||||
startup_timeout=cast(int, timeouts["startup_seconds"]),
|
|
||||||
tool_timeout=cast(int, timeouts["tool_seconds"]),
|
|
||||||
)
|
|
||||||
if artifact["format"] != expected_format or content != expected_content:
|
|
||||||
raise AssertionError("Generated adapter client artifact drifted from its binding")
|
|
||||||
if binding["environment"] != {} or binding["launcher_hash"] != launcher.launcher_hash:
|
|
||||||
raise AssertionError("Generated adapter launch boundary drifted")
|
|
||||||
|
|
||||||
adapter_policy = cast(dict[str, object], binding["adapter_policy"])
|
|
||||||
render_policy = cast(dict[str, object], binding["render_policy"])
|
|
||||||
projection_policy = cast(dict[str, object], result["projection_policy"])
|
|
||||||
projection_availability = cast(dict[str, object], result["projection_availability"])
|
|
||||||
arguments = cast(list[str], binding["args"])
|
|
||||||
prefix = [
|
|
||||||
"-I",
|
|
||||||
"-m",
|
|
||||||
launcher.module,
|
|
||||||
"--project-root",
|
|
||||||
str(descriptor.root),
|
|
||||||
"--capability-mode",
|
|
||||||
cast(str, binding["capability_mode"]),
|
|
||||||
]
|
|
||||||
if arguments[:7] != prefix:
|
|
||||||
raise AssertionError("Generated adapter client arguments drifted from their binding")
|
|
||||||
remaining = arguments[7:]
|
|
||||||
no_ast_argument = "--no-ast" in arguments
|
|
||||||
if no_ast_argument:
|
|
||||||
if remaining[-1:] != ["--no-ast"] or arguments.count("--no-ast") != 1:
|
|
||||||
raise AssertionError("Generated adapter no-AST argument layout drifted")
|
|
||||||
remaining = remaining[:-1]
|
|
||||||
projection_arguments: dict[str, str] = {}
|
|
||||||
authority_arguments: list[str] = []
|
|
||||||
projection_options = {
|
|
||||||
"--manual-render-policy": "manual",
|
|
||||||
"--portable-graph-policy": "portable_graph",
|
|
||||||
"--live-viewer-policy": "live_viewer",
|
|
||||||
}
|
|
||||||
position = 0
|
|
||||||
while position < len(remaining):
|
|
||||||
option = remaining[position]
|
|
||||||
field = projection_options.get(option)
|
|
||||||
if field is None:
|
|
||||||
authority_arguments.append(option)
|
|
||||||
position += 1
|
|
||||||
continue
|
|
||||||
if position + 1 >= len(remaining) or option in projection_arguments:
|
|
||||||
raise AssertionError("Generated adapter projection argument layout drifted")
|
|
||||||
value = remaining[position + 1]
|
|
||||||
projection_arguments[option] = value
|
|
||||||
if projection_policy[field] != value:
|
|
||||||
raise AssertionError("Generated adapter projection policy argument drifted")
|
|
||||||
position += 2
|
|
||||||
mode = binding["capability_mode"]
|
|
||||||
if (
|
|
||||||
(mode == "read" and authority_arguments)
|
|
||||||
or (
|
|
||||||
mode == "proposal"
|
|
||||||
and (
|
|
||||||
len(authority_arguments) != 2
|
|
||||||
or authority_arguments[0] != "--proposal-writer"
|
|
||||||
or not authority_arguments[1]
|
|
||||||
)
|
|
||||||
)
|
|
||||||
or (
|
|
||||||
mode == "application"
|
|
||||||
and (
|
|
||||||
len(authority_arguments) != 4
|
|
||||||
or authority_arguments[0] != "--proposal-writer"
|
|
||||||
or authority_arguments[2] != "--canonical-applier"
|
|
||||||
or not authority_arguments[1]
|
|
||||||
or authority_arguments[1] != authority_arguments[3]
|
|
||||||
)
|
|
||||||
)
|
|
||||||
):
|
|
||||||
raise AssertionError("Generated adapter authority argument layout drifted")
|
|
||||||
|
|
||||||
expected_projection_policy = compose_projection_policy(
|
|
||||||
manual=projection_arguments.get("--manual-render-policy"),
|
|
||||||
portable_graph=projection_arguments.get("--portable-graph-policy"),
|
|
||||||
live_viewer=projection_arguments.get("--live-viewer-policy"),
|
|
||||||
manual_configured=cast(bool, projection_availability["manual_configured"]),
|
|
||||||
portable_graph_configured=cast(
|
|
||||||
bool,
|
|
||||||
projection_availability["portable_graph_configured"],
|
|
||||||
),
|
|
||||||
application_enabled=cast(
|
|
||||||
bool,
|
|
||||||
projection_availability["application_enabled"],
|
|
||||||
),
|
|
||||||
live_viewer_available=cast(
|
|
||||||
bool,
|
|
||||||
projection_availability["live_viewer_available"],
|
|
||||||
),
|
|
||||||
)
|
|
||||||
policy = cast(dict[str, object], result["effective_policy"])
|
|
||||||
composed_policy = compose_effective_policy(
|
|
||||||
selected_mode=cast(CapabilityMode, mode),
|
|
||||||
capability_source="explicit",
|
|
||||||
no_ast=adapter_policy["mode"] == "preserve-no-ast",
|
|
||||||
diagnostics=False,
|
|
||||||
render_configured=render_policy["manual"] != "disabled",
|
|
||||||
application_enabled=mode == "application",
|
|
||||||
)
|
|
||||||
if (
|
|
||||||
projection_policy != expected_projection_policy.as_dict()
|
|
||||||
or projection_availability["manual_configured"] != (descriptor.render is not None)
|
|
||||||
or projection_availability["portable_graph_configured"]
|
|
||||||
!= (descriptor.graph_render is not None)
|
|
||||||
or projection_availability["application_enabled"] != (mode == "application")
|
|
||||||
or projection_availability["live_viewer_available"] is not True
|
|
||||||
or policy != composed_policy.as_dict()
|
|
||||||
or adapter_policy != composed_policy.adapter_policy()
|
|
||||||
or no_ast_argument != (adapter_policy["mode"] == "preserve-no-ast")
|
|
||||||
or binding["launcher_hash"] != result["launcher_hash"]
|
|
||||||
):
|
|
||||||
raise AssertionError("Generated adapter client policy drifted")
|
|
||||||
if result["projection_policy_hash"] != hashlib.sha256(
|
|
||||||
json.dumps(
|
|
||||||
projection_policy,
|
|
||||||
sort_keys=True,
|
|
||||||
separators=(",", ":"),
|
|
||||||
ensure_ascii=False,
|
|
||||||
).encode("utf-8")
|
|
||||||
).hexdigest() or result["configuration_hash"] != document_hash(
|
|
||||||
_adapter_configuration_hash_payload(result)
|
|
||||||
):
|
|
||||||
raise AssertionError("Generated adapter client hash drifted")
|
|
||||||
|
|
|
||||||
|
|
@ -1,242 +0,0 @@
|
||||||
"""Deterministic command references derived from live CLI and MCP registrations."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
from collections.abc import Collection, Iterable, Mapping
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from typing import Any, cast
|
|
||||||
|
|
||||||
from mcp.types import Tool
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class CliCommandReference:
|
|
||||||
"""One CLI command and the exact normalized usage emitted by argparse."""
|
|
||||||
|
|
||||||
name: str
|
|
||||||
invocation: str
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class McpToolReference:
|
|
||||||
"""One registered MCP tool with arguments derived from its input schema."""
|
|
||||||
|
|
||||||
surface: str
|
|
||||||
name: str
|
|
||||||
required_arguments: tuple[str, ...]
|
|
||||||
optional_arguments: tuple[str, ...]
|
|
||||||
description: str
|
|
||||||
input_schema_hash: str
|
|
||||||
|
|
||||||
|
|
||||||
def cli_command_references(
|
|
||||||
parser: argparse.ArgumentParser | None = None,
|
|
||||||
) -> tuple[CliCommandReference, ...]:
|
|
||||||
"""Read CLI reference rows from the parser used by ``docforge``."""
|
|
||||||
|
|
||||||
effective_parser = parser or _docforge_parser()
|
|
||||||
commands = _command_parsers(effective_parser)
|
|
||||||
return tuple(
|
|
||||||
CliCommandReference(
|
|
||||||
name=name,
|
|
||||||
invocation=_normalize_usage(command_parser.format_usage()),
|
|
||||||
)
|
|
||||||
for name, command_parser in sorted(commands.items())
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def mcp_tool_references(
|
|
||||||
tools: Iterable[Tool],
|
|
||||||
*,
|
|
||||||
expected_names: Collection[str] | None = None,
|
|
||||||
) -> tuple[McpToolReference, ...]:
|
|
||||||
"""Read MCP reference rows from registered tools returned by ``list_tools``.
|
|
||||||
|
|
||||||
``expected_names`` makes documentation generation fail closed when the selected
|
|
||||||
server surface is incomplete or has drifted.
|
|
||||||
"""
|
|
||||||
|
|
||||||
surface_by_name = _mcp_surface_by_name()
|
|
||||||
references: list[McpToolReference] = []
|
|
||||||
observed: set[str] = set()
|
|
||||||
for tool in tools:
|
|
||||||
if tool.name in observed:
|
|
||||||
raise ValueError(f"MCP tool registration repeats {tool.name!r}")
|
|
||||||
observed.add(tool.name)
|
|
||||||
try:
|
|
||||||
surface = surface_by_name[tool.name]
|
|
||||||
except KeyError as error:
|
|
||||||
raise ValueError(
|
|
||||||
f"MCP tool {tool.name!r} has no declared capability surface"
|
|
||||||
) from error
|
|
||||||
schema = tool.inputSchema
|
|
||||||
properties = _schema_properties(schema)
|
|
||||||
required = _schema_required(schema, properties)
|
|
||||||
references.append(
|
|
||||||
McpToolReference(
|
|
||||||
surface=surface,
|
|
||||||
name=tool.name,
|
|
||||||
required_arguments=tuple(sorted(required)),
|
|
||||||
optional_arguments=tuple(sorted(set(properties) - required)),
|
|
||||||
description=_normalize_text(tool.description or ""),
|
|
||||||
input_schema_hash=_canonical_hash(schema),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
if expected_names is not None:
|
|
||||||
expected = set(expected_names)
|
|
||||||
if observed != expected:
|
|
||||||
missing = sorted(expected - observed)
|
|
||||||
unexpected = sorted(observed - expected)
|
|
||||||
raise ValueError(
|
|
||||||
"Registered MCP tools do not match the requested reference surface: "
|
|
||||||
f"missing={missing!r}, unexpected={unexpected!r}"
|
|
||||||
)
|
|
||||||
surface_rank = {"read": 0, "proposal": 1, "application": 2}
|
|
||||||
return tuple(sorted(references, key=lambda item: (surface_rank[item.surface], item.name)))
|
|
||||||
|
|
||||||
|
|
||||||
def render_cli_reference_markdown(references: Iterable[CliCommandReference]) -> str:
|
|
||||||
"""Render a deterministic Markdown table for CLI commands."""
|
|
||||||
|
|
||||||
rows = sorted(references, key=lambda item: item.name)
|
|
||||||
lines = [
|
|
||||||
"| Command | Invocation |",
|
|
||||||
"|---|---|",
|
|
||||||
]
|
|
||||||
lines.extend(
|
|
||||||
f"| `{_escape_markdown(item.name)}` | `{_escape_markdown(item.invocation)}` |"
|
|
||||||
for item in rows
|
|
||||||
)
|
|
||||||
return "\n".join(lines) + "\n"
|
|
||||||
|
|
||||||
|
|
||||||
def render_mcp_reference_markdown(references: Iterable[McpToolReference]) -> str:
|
|
||||||
"""Render a deterministic Markdown table for registered MCP tools."""
|
|
||||||
|
|
||||||
surface_rank = {"read": 0, "proposal": 1, "application": 2}
|
|
||||||
rows = sorted(references, key=lambda item: (surface_rank[item.surface], item.name))
|
|
||||||
lines = [
|
|
||||||
"| Surface | Tool | Required arguments | Optional arguments | "
|
|
||||||
"Input schema SHA-256 | Description |",
|
|
||||||
"|---|---|---|---|---|---|",
|
|
||||||
]
|
|
||||||
lines.extend(
|
|
||||||
"| "
|
|
||||||
f"{_escape_markdown(item.surface)} | "
|
|
||||||
f"`{_escape_markdown(item.name)}` | "
|
|
||||||
f"{_argument_list(item.required_arguments)} | "
|
|
||||||
f"{_argument_list(item.optional_arguments)} | "
|
|
||||||
f"`{item.input_schema_hash}` | "
|
|
||||||
f"{_escape_markdown(item.description) or '—'} |"
|
|
||||||
for item in rows
|
|
||||||
)
|
|
||||||
return "\n".join(lines) + "\n"
|
|
||||||
|
|
||||||
|
|
||||||
def render_command_reference_markdown(
|
|
||||||
cli_references: Iterable[CliCommandReference],
|
|
||||||
mcp_references: Iterable[McpToolReference],
|
|
||||||
) -> str:
|
|
||||||
"""Render the complete deterministic CLI and MCP command reference."""
|
|
||||||
|
|
||||||
return (
|
|
||||||
"# DocForge command reference\n\n"
|
|
||||||
"> Generated from the live CLI parser and MCP registrations. Do not edit this file "
|
|
||||||
"by hand.\n\n"
|
|
||||||
"Global CLI options are documented in `docforge --help` and are not repeated in each "
|
|
||||||
"command row. The MCP table is the complete generic application-enabled surface; "
|
|
||||||
"the fixed reference-adapter server exposes only its read rows.\n\n"
|
|
||||||
"## CLI commands\n\n"
|
|
||||||
f"{render_cli_reference_markdown(cli_references)}"
|
|
||||||
"\n## MCP tools\n\n"
|
|
||||||
f"{render_mcp_reference_markdown(mcp_references)}"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _docforge_parser() -> argparse.ArgumentParser:
|
|
||||||
from .cli import _parser # pyright: ignore[reportPrivateUsage]
|
|
||||||
|
|
||||||
return _parser()
|
|
||||||
|
|
||||||
|
|
||||||
def _command_parsers(
|
|
||||||
parser: argparse.ArgumentParser,
|
|
||||||
) -> Mapping[str, argparse.ArgumentParser]:
|
|
||||||
actions = parser._actions # pyright: ignore[reportPrivateUsage]
|
|
||||||
subparsers = [
|
|
||||||
action
|
|
||||||
for action in actions
|
|
||||||
if action.dest == "command" and isinstance(action.choices, dict)
|
|
||||||
]
|
|
||||||
if len(subparsers) != 1:
|
|
||||||
raise ValueError("CLI parser must define exactly one command subparser")
|
|
||||||
return cast(Mapping[str, argparse.ArgumentParser], subparsers[0].choices)
|
|
||||||
|
|
||||||
|
|
||||||
def _mcp_surface_by_name() -> dict[str, str]:
|
|
||||||
from .mcp_server import APPLICATION_TOOLS, PROPOSAL_TOOLS, READ_TOOLS
|
|
||||||
|
|
||||||
groups = {
|
|
||||||
"read": READ_TOOLS,
|
|
||||||
"proposal": PROPOSAL_TOOLS,
|
|
||||||
"application": APPLICATION_TOOLS,
|
|
||||||
}
|
|
||||||
surface_by_name: dict[str, str] = {}
|
|
||||||
for surface, names in groups.items():
|
|
||||||
for name in names:
|
|
||||||
if name in surface_by_name:
|
|
||||||
raise ValueError(f"MCP tool surface declaration repeats {name!r}")
|
|
||||||
surface_by_name[name] = surface
|
|
||||||
return surface_by_name
|
|
||||||
|
|
||||||
|
|
||||||
def _normalize_usage(usage: str) -> str:
|
|
||||||
return _normalize_text(usage.removeprefix("usage: "))
|
|
||||||
|
|
||||||
|
|
||||||
def _normalize_text(value: str) -> str:
|
|
||||||
return " ".join(value.split())
|
|
||||||
|
|
||||||
|
|
||||||
def _schema_properties(schema: Mapping[str, Any]) -> dict[str, Any]:
|
|
||||||
raw_properties: object = schema.get("properties", {})
|
|
||||||
if not isinstance(raw_properties, dict):
|
|
||||||
raise ValueError("MCP tool input schema properties must be a string-keyed object")
|
|
||||||
properties: dict[str, Any] = {}
|
|
||||||
for name, value in cast(dict[object, object], raw_properties).items():
|
|
||||||
if not isinstance(name, str):
|
|
||||||
raise ValueError("MCP tool input schema properties must be a string-keyed object")
|
|
||||||
properties[name] = value
|
|
||||||
return properties
|
|
||||||
|
|
||||||
|
|
||||||
def _schema_required(schema: Mapping[str, Any], properties: Mapping[str, Any]) -> set[str]:
|
|
||||||
raw_required: object = schema.get("required", [])
|
|
||||||
if not isinstance(raw_required, list):
|
|
||||||
raise ValueError("MCP tool input schema required arguments must be strings")
|
|
||||||
required_names: set[str] = set()
|
|
||||||
for name in cast(list[object], raw_required):
|
|
||||||
if not isinstance(name, str):
|
|
||||||
raise ValueError("MCP tool input schema required arguments must be strings")
|
|
||||||
required_names.add(name)
|
|
||||||
if not required_names <= set(properties):
|
|
||||||
raise ValueError("MCP tool input schema requires an undeclared argument")
|
|
||||||
return required_names
|
|
||||||
|
|
||||||
|
|
||||||
def _canonical_hash(payload: Mapping[str, Any]) -> str:
|
|
||||||
encoded = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
|
|
||||||
return hashlib.sha256(encoded).hexdigest()
|
|
||||||
|
|
||||||
|
|
||||||
def _argument_list(arguments: tuple[str, ...]) -> str:
|
|
||||||
if not arguments:
|
|
||||||
return "—"
|
|
||||||
return ", ".join(f"`{_escape_markdown(argument)}`" for argument in arguments)
|
|
||||||
|
|
||||||
|
|
||||||
def _escape_markdown(value: str) -> str:
|
|
||||||
return value.replace("\\", "\\\\").replace("|", "\\|").replace("`", "\\`")
|
|
||||||
|
|
@ -4,7 +4,6 @@ from __future__ import annotations
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import stat
|
|
||||||
import tempfile
|
import tempfile
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
@ -13,8 +12,6 @@ from typing import Any, cast
|
||||||
from .errors import DocForgeError
|
from .errors import DocForgeError
|
||||||
|
|
||||||
EXTRACTION_CACHE_SCHEMA_VERSION = 1
|
EXTRACTION_CACHE_SCHEMA_VERSION = 1
|
||||||
MAX_EXTRACTION_CACHE_BYTES = 64_000_000
|
|
||||||
MAX_EXTRACTION_CACHE_SOURCES = 10_000
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
|
|
@ -45,39 +42,13 @@ def load_extraction_cache(
|
||||||
project_id: str,
|
project_id: str,
|
||||||
adapter_id: str,
|
adapter_id: str,
|
||||||
adapter_version: str,
|
adapter_version: str,
|
||||||
max_bytes: int = MAX_EXTRACTION_CACHE_BYTES,
|
|
||||||
max_sources: int = MAX_EXTRACTION_CACHE_SOURCES,
|
|
||||||
) -> ExtractionCache | None:
|
) -> ExtractionCache | None:
|
||||||
"""Read a cache generation, treating malformed or incompatible data as a miss."""
|
"""Read a cache generation, treating malformed or incompatible data as a miss."""
|
||||||
|
|
||||||
if max_bytes < 1 or max_sources < 1:
|
if not path.is_file() or path.is_symlink():
|
||||||
raise ValueError("Extraction cache limits must be positive")
|
|
||||||
try:
|
|
||||||
descriptor = os.open(
|
|
||||||
path,
|
|
||||||
os.O_RDONLY
|
|
||||||
| getattr(os, "O_CLOEXEC", 0)
|
|
||||||
| getattr(os, "O_NOFOLLOW", 0)
|
|
||||||
| getattr(os, "O_NONBLOCK", 0),
|
|
||||||
)
|
|
||||||
except OSError:
|
|
||||||
return None
|
return None
|
||||||
try:
|
try:
|
||||||
with os.fdopen(descriptor, "rb") as handle:
|
raw = json.loads(path.read_text(encoding="utf-8"))
|
||||||
before = os.fstat(handle.fileno())
|
|
||||||
if not stat.S_ISREG(before.st_mode) or before.st_size > max_bytes:
|
|
||||||
return None
|
|
||||||
encoded = handle.read(max_bytes + 1)
|
|
||||||
after = os.fstat(handle.fileno())
|
|
||||||
if (
|
|
||||||
len(encoded) > max_bytes
|
|
||||||
or before.st_dev != after.st_dev
|
|
||||||
or before.st_ino != after.st_ino
|
|
||||||
or before.st_size != after.st_size
|
|
||||||
or before.st_mtime_ns != after.st_mtime_ns
|
|
||||||
):
|
|
||||||
return None
|
|
||||||
raw = json.loads(encoded.decode("utf-8"))
|
|
||||||
if not isinstance(raw, dict):
|
if not isinstance(raw, dict):
|
||||||
return None
|
return None
|
||||||
document = cast(dict[str, Any], raw)
|
document = cast(dict[str, Any], raw)
|
||||||
|
|
@ -91,11 +62,8 @@ def load_extraction_cache(
|
||||||
raw_sources = document.get("sources")
|
raw_sources = document.get("sources")
|
||||||
if not isinstance(raw_sources, list):
|
if not isinstance(raw_sources, list):
|
||||||
return None
|
return None
|
||||||
source_items = cast(list[object], raw_sources)
|
|
||||||
if len(source_items) > max_sources:
|
|
||||||
return None
|
|
||||||
sources: list[CachedSource] = []
|
sources: list[CachedSource] = []
|
||||||
for raw_source in source_items:
|
for raw_source in cast(list[object], raw_sources):
|
||||||
if not isinstance(raw_source, dict):
|
if not isinstance(raw_source, dict):
|
||||||
return None
|
return None
|
||||||
item = cast(dict[str, object], raw_source)
|
item = cast(dict[str, object], raw_source)
|
||||||
|
|
@ -142,24 +110,9 @@ def load_extraction_cache(
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
def write_extraction_cache(
|
def write_extraction_cache(path: Path, cache: ExtractionCache) -> None:
|
||||||
path: Path,
|
|
||||||
cache: ExtractionCache,
|
|
||||||
*,
|
|
||||||
max_bytes: int = MAX_EXTRACTION_CACHE_BYTES,
|
|
||||||
max_sources: int = MAX_EXTRACTION_CACHE_SOURCES,
|
|
||||||
) -> None:
|
|
||||||
"""Atomically publish one validated extraction-cache generation."""
|
"""Atomically publish one validated extraction-cache generation."""
|
||||||
|
|
||||||
if max_bytes < 1 or max_sources < 1:
|
|
||||||
raise ValueError("Extraction cache limits must be positive")
|
|
||||||
if len(cache.sources) > max_sources:
|
|
||||||
raise DocForgeError(
|
|
||||||
"cache_limit",
|
|
||||||
"Incremental extraction cache exceeds its source limit",
|
|
||||||
maximum=max_sources,
|
|
||||||
actual=len(cache.sources),
|
|
||||||
)
|
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
document = {
|
document = {
|
||||||
"schema_version": EXTRACTION_CACHE_SCHEMA_VERSION,
|
"schema_version": EXTRACTION_CACHE_SCHEMA_VERSION,
|
||||||
|
|
@ -178,23 +131,17 @@ def write_extraction_cache(
|
||||||
for source in cache.sources
|
for source in cache.sources
|
||||||
],
|
],
|
||||||
}
|
}
|
||||||
encoded = json.dumps(document, sort_keys=True, separators=(",", ":")).encode("utf-8") + b"\n"
|
|
||||||
if len(encoded) > max_bytes:
|
|
||||||
raise DocForgeError(
|
|
||||||
"cache_limit",
|
|
||||||
"Incremental extraction cache exceeds its byte limit",
|
|
||||||
maximum=max_bytes,
|
|
||||||
actual=len(encoded),
|
|
||||||
)
|
|
||||||
with tempfile.NamedTemporaryFile(
|
with tempfile.NamedTemporaryFile(
|
||||||
mode="wb",
|
mode="w",
|
||||||
|
encoding="utf-8",
|
||||||
prefix="extractions-",
|
prefix="extractions-",
|
||||||
suffix=".json",
|
suffix=".json",
|
||||||
dir=path.parent,
|
dir=path.parent,
|
||||||
delete=False,
|
delete=False,
|
||||||
) as descriptor:
|
) as descriptor:
|
||||||
temporary = Path(descriptor.name)
|
temporary = Path(descriptor.name)
|
||||||
descriptor.write(encoded)
|
json.dump(document, descriptor, sort_keys=True, separators=(",", ":"))
|
||||||
|
descriptor.write("\n")
|
||||||
descriptor.flush()
|
descriptor.flush()
|
||||||
os.fsync(descriptor.fileno())
|
os.fsync(descriptor.fileno())
|
||||||
try:
|
try:
|
||||||
|
|
|
||||||
|
|
@ -1,428 +0,0 @@
|
||||||
"""Strict, data-only configuration for the runnable reference adapters."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import os
|
|
||||||
import stat
|
|
||||||
import tomllib
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from pathlib import Path, PurePosixPath
|
|
||||||
from typing import Literal, cast
|
|
||||||
|
|
||||||
from .config_validation import ID_PATTERN
|
|
||||||
from .errors import DocForgeError
|
|
||||||
|
|
||||||
REFERENCE_ADAPTER_SCHEMA_VERSION = 1
|
|
||||||
REFERENCE_ADAPTER_CONFIG = PurePosixPath(".docforge/reference-adapter.toml")
|
|
||||||
MAX_REFERENCE_CONFIG_BYTES = 65_536
|
|
||||||
MAX_REFERENCE_INVENTORY_ENTRIES = 65_536
|
|
||||||
|
|
||||||
ReferenceLanguage = Literal["python", "javascript", "typescript", "cpp"]
|
|
||||||
REFERENCE_LANGUAGES: tuple[ReferenceLanguage, ...] = (
|
|
||||||
"python",
|
|
||||||
"javascript",
|
|
||||||
"typescript",
|
|
||||||
"cpp",
|
|
||||||
)
|
|
||||||
|
|
||||||
_CONFIG_FIELDS = frozenset(
|
|
||||||
{
|
|
||||||
"schema_version",
|
|
||||||
"project_id",
|
|
||||||
"title",
|
|
||||||
"language",
|
|
||||||
"source_roots",
|
|
||||||
"compilation_database",
|
|
||||||
}
|
|
||||||
)
|
|
||||||
_LANGUAGE_SUFFIXES: dict[ReferenceLanguage, frozenset[str]] = {
|
|
||||||
"python": frozenset({".py"}),
|
|
||||||
"javascript": frozenset({".cjs", ".js", ".mjs"}),
|
|
||||||
"typescript": frozenset({".cts", ".mts", ".ts"}),
|
|
||||||
"cpp": frozenset({".c", ".cc", ".cpp", ".cxx", ".h", ".hh", ".hpp", ".hxx"}),
|
|
||||||
}
|
|
||||||
_KNOWN_SOURCE_SUFFIXES = frozenset(
|
|
||||||
suffix for suffixes in _LANGUAGE_SUFFIXES.values() for suffix in suffixes
|
|
||||||
)
|
|
||||||
|
|
||||||
__all__ = [
|
|
||||||
"MAX_REFERENCE_CONFIG_BYTES",
|
|
||||||
"MAX_REFERENCE_INVENTORY_ENTRIES",
|
|
||||||
"REFERENCE_ADAPTER_CONFIG",
|
|
||||||
"REFERENCE_ADAPTER_SCHEMA_VERSION",
|
|
||||||
"REFERENCE_LANGUAGES",
|
|
||||||
"ReferenceAdapterConfigV1",
|
|
||||||
"ReferenceLanguage",
|
|
||||||
"load_reference_adapter_config",
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class ReferenceAdapterConfigV1:
|
|
||||||
"""One immutable, project-confined reference adapter selection."""
|
|
||||||
|
|
||||||
schema_version: Literal[1]
|
|
||||||
project_id: str
|
|
||||||
title: str
|
|
||||||
language: ReferenceLanguage
|
|
||||||
source_roots: tuple[Path, ...]
|
|
||||||
compilation_database: Path | None
|
|
||||||
project_root: Path
|
|
||||||
config_path: Path
|
|
||||||
config_hash: str
|
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, object]:
|
|
||||||
document: dict[str, object] = {
|
|
||||||
"schema_version": self.schema_version,
|
|
||||||
"project_id": self.project_id,
|
|
||||||
"title": self.title,
|
|
||||||
"language": self.language,
|
|
||||||
"source_roots": [
|
|
||||||
path.relative_to(self.project_root).as_posix() for path in self.source_roots
|
|
||||||
],
|
|
||||||
}
|
|
||||||
if self.compilation_database is not None:
|
|
||||||
document["compilation_database"] = self.compilation_database.relative_to(
|
|
||||||
self.project_root
|
|
||||||
).as_posix()
|
|
||||||
return document
|
|
||||||
|
|
||||||
|
|
||||||
def load_reference_adapter_config(project_root: str | Path) -> ReferenceAdapterConfigV1:
|
|
||||||
"""Load the fixed reference-adapter descriptor without executing project content."""
|
|
||||||
|
|
||||||
root = _project_root(project_root)
|
|
||||||
config_path = root.joinpath(*REFERENCE_ADAPTER_CONFIG.parts)
|
|
||||||
try:
|
|
||||||
resolved_config = config_path.resolve(strict=True)
|
|
||||||
except (OSError, ValueError) as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"missing_path",
|
|
||||||
"Reference adapter configuration does not exist",
|
|
||||||
path=str(config_path),
|
|
||||||
) from error
|
|
||||||
if resolved_config != config_path or not resolved_config.is_relative_to(root):
|
|
||||||
raise DocForgeError(
|
|
||||||
"path_escape",
|
|
||||||
"Reference adapter configuration must not traverse symbolic links",
|
|
||||||
)
|
|
||||||
raw = _read_regular_file(
|
|
||||||
config_path,
|
|
||||||
field="reference adapter configuration",
|
|
||||||
maximum_bytes=MAX_REFERENCE_CONFIG_BYTES,
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
document = cast(dict[str, object], tomllib.loads(raw.decode("utf-8")))
|
|
||||||
except UnicodeDecodeError as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
"Reference adapter configuration must be UTF-8",
|
|
||||||
) from error
|
|
||||||
except tomllib.TOMLDecodeError as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
f"Invalid reference adapter configuration: {error}",
|
|
||||||
) from error
|
|
||||||
|
|
||||||
unknown = sorted(set(document) - _CONFIG_FIELDS)
|
|
||||||
if unknown:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
"Reference adapter configuration has unknown fields",
|
|
||||||
fields=unknown,
|
|
||||||
)
|
|
||||||
if document.get("schema_version") != REFERENCE_ADAPTER_SCHEMA_VERSION:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
"Reference adapter schema_version must be 1",
|
|
||||||
)
|
|
||||||
project_id = _string(document, "project_id")
|
|
||||||
if ID_PATTERN.fullmatch(project_id) is None:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
"Reference adapter project_id is not a stable ID",
|
|
||||||
project_id=project_id,
|
|
||||||
)
|
|
||||||
title = _string(document, "title")
|
|
||||||
if len(title) > 256:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
"Reference adapter title exceeds its size limit",
|
|
||||||
maximum_characters=256,
|
|
||||||
)
|
|
||||||
raw_language = _string(document, "language")
|
|
||||||
if raw_language not in REFERENCE_LANGUAGES:
|
|
||||||
raise DocForgeError(
|
|
||||||
"unsupported_reference_language",
|
|
||||||
"Reference adapter language is not supported",
|
|
||||||
language=raw_language,
|
|
||||||
supported=list(REFERENCE_LANGUAGES),
|
|
||||||
)
|
|
||||||
language = raw_language
|
|
||||||
source_roots = _source_roots(root, document.get("source_roots"))
|
|
||||||
|
|
||||||
raw_compilation_database = document.get("compilation_database")
|
|
||||||
compilation_database: Path | None = None
|
|
||||||
if language == "cpp":
|
|
||||||
if raw_compilation_database is None:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
"C++ reference adapters require compilation_database",
|
|
||||||
)
|
|
||||||
compilation_database = _confined_path(
|
|
||||||
root,
|
|
||||||
raw_compilation_database,
|
|
||||||
field="compilation_database",
|
|
||||||
expected="file",
|
|
||||||
)
|
|
||||||
elif raw_compilation_database is not None:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
"compilation_database is allowed only for the C++ reference adapter",
|
|
||||||
language=language,
|
|
||||||
)
|
|
||||||
|
|
||||||
_validate_language_inventory(source_roots, language)
|
|
||||||
return ReferenceAdapterConfigV1(
|
|
||||||
schema_version=1,
|
|
||||||
project_id=project_id,
|
|
||||||
title=title,
|
|
||||||
language=language,
|
|
||||||
source_roots=source_roots,
|
|
||||||
compilation_database=compilation_database,
|
|
||||||
project_root=root,
|
|
||||||
config_path=config_path,
|
|
||||||
config_hash=hashlib.sha256(raw).hexdigest(),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _project_root(raw: str | Path) -> Path:
|
|
||||||
path = Path(raw)
|
|
||||||
if path.is_symlink():
|
|
||||||
raise DocForgeError("path_escape", "Reference project root must not be a symlink")
|
|
||||||
try:
|
|
||||||
root = path.resolve(strict=True)
|
|
||||||
except (OSError, ValueError) as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"missing_path",
|
|
||||||
"Reference project root does not exist",
|
|
||||||
path=str(path),
|
|
||||||
) from error
|
|
||||||
if not root.is_dir():
|
|
||||||
raise DocForgeError("invalid_path", "Reference project root must be a directory")
|
|
||||||
return root
|
|
||||||
|
|
||||||
|
|
||||||
def _read_regular_file(path: Path, *, field: str, maximum_bytes: int) -> bytes:
|
|
||||||
try:
|
|
||||||
before = path.lstat()
|
|
||||||
except OSError as error:
|
|
||||||
raise DocForgeError("missing_path", f"{field} does not exist", path=str(path)) from error
|
|
||||||
if stat.S_ISLNK(before.st_mode) or not stat.S_ISREG(before.st_mode):
|
|
||||||
raise DocForgeError("invalid_path", f"{field} must be a regular non-symlink file")
|
|
||||||
if before.st_size > maximum_bytes:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
f"{field} exceeds its size limit",
|
|
||||||
maximum_bytes=maximum_bytes,
|
|
||||||
)
|
|
||||||
|
|
||||||
def identity(value: os.stat_result) -> tuple[int, int, int, int, int, int]:
|
|
||||||
return (
|
|
||||||
value.st_dev,
|
|
||||||
value.st_ino,
|
|
||||||
value.st_mode,
|
|
||||||
value.st_size,
|
|
||||||
value.st_mtime_ns,
|
|
||||||
value.st_ctime_ns,
|
|
||||||
)
|
|
||||||
|
|
||||||
flags = os.O_RDONLY | getattr(os, "O_CLOEXEC", 0) | getattr(os, "O_NOFOLLOW", 0)
|
|
||||||
try:
|
|
||||||
descriptor = os.open(path, flags)
|
|
||||||
except OSError as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"reference_config_changed",
|
|
||||||
f"{field} changed while it was opened",
|
|
||||||
) from error
|
|
||||||
try:
|
|
||||||
opened = os.fstat(descriptor)
|
|
||||||
chunks: list[bytes] = []
|
|
||||||
remaining = maximum_bytes + 1
|
|
||||||
while remaining:
|
|
||||||
chunk = os.read(descriptor, min(remaining, 65_536))
|
|
||||||
if not chunk:
|
|
||||||
break
|
|
||||||
chunks.append(chunk)
|
|
||||||
remaining -= len(chunk)
|
|
||||||
raw = b"".join(chunks)
|
|
||||||
opened_after = os.fstat(descriptor)
|
|
||||||
except OSError as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"reference_config_changed",
|
|
||||||
f"{field} changed while it was read",
|
|
||||||
) from error
|
|
||||||
finally:
|
|
||||||
os.close(descriptor)
|
|
||||||
try:
|
|
||||||
after = path.lstat()
|
|
||||||
except OSError as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"reference_config_changed",
|
|
||||||
f"{field} changed while it was read",
|
|
||||||
) from error
|
|
||||||
|
|
||||||
if (
|
|
||||||
identity(before) != identity(opened)
|
|
||||||
or identity(opened) != identity(opened_after)
|
|
||||||
or identity(opened_after) != identity(after)
|
|
||||||
or len(raw) != opened_after.st_size
|
|
||||||
or len(raw) > maximum_bytes
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"reference_config_changed",
|
|
||||||
f"{field} changed while it was read",
|
|
||||||
)
|
|
||||||
return raw
|
|
||||||
|
|
||||||
|
|
||||||
def _string(document: dict[str, object], field: str) -> str:
|
|
||||||
value = document.get(field)
|
|
||||||
if not isinstance(value, str) or not value or value != value.strip():
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
f"Reference adapter {field} must be a non-empty normalized string",
|
|
||||||
)
|
|
||||||
return value
|
|
||||||
|
|
||||||
|
|
||||||
def _source_roots(root: Path, raw: object) -> tuple[Path, ...]:
|
|
||||||
if not isinstance(raw, list) or not raw:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
"Reference adapter source_roots must be a non-empty string list",
|
|
||||||
)
|
|
||||||
values = cast(list[object], raw)
|
|
||||||
if len(values) > 64:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
"Reference adapter source_roots exceeds its item limit",
|
|
||||||
maximum_items=64,
|
|
||||||
)
|
|
||||||
paths = tuple(
|
|
||||||
_confined_path(root, value, field="source_roots", expected="directory") for value in values
|
|
||||||
)
|
|
||||||
if len(paths) != len(set(paths)):
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
"Reference adapter source_roots contains duplicates",
|
|
||||||
)
|
|
||||||
ordered = tuple(sorted(paths))
|
|
||||||
if paths != ordered:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
"Reference adapter source_roots must be sorted",
|
|
||||||
)
|
|
||||||
private_root = root / ".docforge"
|
|
||||||
if any(
|
|
||||||
path == private_root
|
|
||||||
or path.is_relative_to(private_root)
|
|
||||||
or any(
|
|
||||||
path.is_relative_to(other) or other.is_relative_to(path)
|
|
||||||
for other in ordered
|
|
||||||
if other != path
|
|
||||||
)
|
|
||||||
for path in ordered
|
|
||||||
):
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
"Reference adapter source_roots must be non-overlapping and outside .docforge",
|
|
||||||
)
|
|
||||||
return ordered
|
|
||||||
|
|
||||||
|
|
||||||
def _confined_path(root: Path, raw: object, *, field: str, expected: str) -> Path:
|
|
||||||
if not isinstance(raw, str) or not raw or raw != raw.strip():
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
f"{field} must be a non-empty normalized relative path",
|
|
||||||
)
|
|
||||||
relative = Path(raw)
|
|
||||||
if relative.is_absolute() or ".." in relative.parts or relative == Path("."):
|
|
||||||
raise DocForgeError("path_escape", f"{field} must stay inside the project root", path=raw)
|
|
||||||
candidate = root / relative
|
|
||||||
if candidate.is_symlink():
|
|
||||||
raise DocForgeError("path_escape", f"{field} must not be a symlink", path=raw)
|
|
||||||
try:
|
|
||||||
resolved = candidate.resolve(strict=True)
|
|
||||||
except (OSError, ValueError) as error:
|
|
||||||
raise DocForgeError("missing_path", f"{field} does not exist", path=raw) from error
|
|
||||||
if resolved != candidate or not resolved.is_relative_to(root):
|
|
||||||
raise DocForgeError(
|
|
||||||
"path_escape",
|
|
||||||
f"{field} must not traverse symlinks or leave the project root",
|
|
||||||
path=raw,
|
|
||||||
)
|
|
||||||
if expected == "directory" and not resolved.is_dir():
|
|
||||||
raise DocForgeError("invalid_path", f"{field} must identify a directory", path=raw)
|
|
||||||
if expected == "file" and (not resolved.is_file() or resolved.is_symlink()):
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_path",
|
|
||||||
f"{field} must identify a regular non-symlink file",
|
|
||||||
path=raw,
|
|
||||||
)
|
|
||||||
return resolved
|
|
||||||
|
|
||||||
|
|
||||||
def _validate_language_inventory(
|
|
||||||
source_roots: tuple[Path, ...],
|
|
||||||
language: ReferenceLanguage,
|
|
||||||
) -> None:
|
|
||||||
expected = _LANGUAGE_SUFFIXES[language]
|
|
||||||
selected = 0
|
|
||||||
examined = 0
|
|
||||||
other_languages: set[str] = set()
|
|
||||||
for source_root in source_roots:
|
|
||||||
for directory, names, files in os.walk(
|
|
||||||
source_root,
|
|
||||||
followlinks=False,
|
|
||||||
onerror=_inventory_error,
|
|
||||||
):
|
|
||||||
base = Path(directory)
|
|
||||||
for name in (*names, *files):
|
|
||||||
examined += 1
|
|
||||||
if examined > MAX_REFERENCE_INVENTORY_ENTRIES:
|
|
||||||
raise DocForgeError(
|
|
||||||
"reference_inventory_too_large",
|
|
||||||
"Reference source inventory exceeds its entry limit",
|
|
||||||
maximum_entries=MAX_REFERENCE_INVENTORY_ENTRIES,
|
|
||||||
)
|
|
||||||
candidate = base / name
|
|
||||||
if candidate.is_symlink():
|
|
||||||
raise DocForgeError(
|
|
||||||
"path_escape",
|
|
||||||
"Reference source roots must not contain symlinks",
|
|
||||||
path=candidate.as_posix(),
|
|
||||||
)
|
|
||||||
for name in files:
|
|
||||||
suffix = Path(name).suffix
|
|
||||||
if suffix in expected:
|
|
||||||
selected += 1
|
|
||||||
elif suffix in _KNOWN_SOURCE_SUFFIXES:
|
|
||||||
other_languages.add(suffix)
|
|
||||||
if selected == 0:
|
|
||||||
raise DocForgeError(
|
|
||||||
"reference_language_mismatch",
|
|
||||||
"Declared source roots contain no sources for the selected language",
|
|
||||||
language=language,
|
|
||||||
observed_suffixes=sorted(other_languages),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _inventory_error(error: OSError) -> None:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_path",
|
|
||||||
"Reference source inventory cannot be read safely",
|
|
||||||
path=error.filename,
|
|
||||||
) from error
|
|
||||||
|
|
@ -1,196 +0,0 @@
|
||||||
"""Runnable, read-only MCP binding for fixed in-repository reference adapters."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import importlib
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import cast
|
|
||||||
|
|
||||||
from mcp.server.fastmcp import FastMCP
|
|
||||||
|
|
||||||
from .adapter_sdk import (
|
|
||||||
AdapterImplementation,
|
|
||||||
AdapterLoader,
|
|
||||||
AdapterProject,
|
|
||||||
AdapterProjectSettings,
|
|
||||||
)
|
|
||||||
from .errors import DocForgeError
|
|
||||||
from .mcp_server import create_read_only_server
|
|
||||||
from .reference_config import ReferenceAdapterConfigV1, load_reference_adapter_config
|
|
||||||
|
|
||||||
REFERENCE_MCP_MODULE = "docforge.reference_mcp"
|
|
||||||
_FIXED_PROVIDERS = {
|
|
||||||
"python": ("docforge.adapters.python", "PythonReferenceAdapter"),
|
|
||||||
"javascript": ("docforge.adapters.javascript", "JavaScriptReferenceAdapter"),
|
|
||||||
"typescript": ("docforge.adapters.javascript", "JavaScriptReferenceAdapter"),
|
|
||||||
"cpp": ("docforge.adapters.cpp", "CppReferenceAdapter"),
|
|
||||||
}
|
|
||||||
|
|
||||||
__all__ = [
|
|
||||||
"REFERENCE_MCP_MODULE",
|
|
||||||
"create_reference_project",
|
|
||||||
"create_reference_server",
|
|
||||||
"main",
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def create_reference_project(project_root: str | Path) -> AdapterProject:
|
|
||||||
"""Construct one configured adapter project from fixed internal providers."""
|
|
||||||
|
|
||||||
config = load_reference_adapter_config(project_root)
|
|
||||||
return _project_from_config(config)
|
|
||||||
|
|
||||||
|
|
||||||
def create_reference_server(
|
|
||||||
project_root: str | Path,
|
|
||||||
*,
|
|
||||||
no_ast: bool = False,
|
|
||||||
diagnostics: bool = False,
|
|
||||||
capability_mode: str | None = None,
|
|
||||||
manual_projection_policy: str | None = None,
|
|
||||||
portable_graph_policy: str | None = None,
|
|
||||||
live_viewer_policy: str | None = None,
|
|
||||||
) -> FastMCP:
|
|
||||||
"""Create the fixed read-only MCP surface for one reference project."""
|
|
||||||
|
|
||||||
config = load_reference_adapter_config(project_root)
|
|
||||||
project = _project_from_config(config)
|
|
||||||
return create_read_only_server(
|
|
||||||
project,
|
|
||||||
binding_metadata={
|
|
||||||
"server_module": REFERENCE_MCP_MODULE,
|
|
||||||
"adapter_mode": "reference",
|
|
||||||
"reference_language": config.language,
|
|
||||||
"reference_config_hash": config.config_hash,
|
|
||||||
},
|
|
||||||
no_ast=no_ast,
|
|
||||||
diagnostics=diagnostics,
|
|
||||||
capability_mode=capability_mode,
|
|
||||||
manual_projection_policy=manual_projection_policy,
|
|
||||||
portable_graph_policy=portable_graph_policy,
|
|
||||||
live_viewer_policy=live_viewer_policy,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _project_from_config(config: ReferenceAdapterConfigV1) -> AdapterProject:
|
|
||||||
loader = _adapter_loader(config)
|
|
||||||
cache_root = config.project_root / ".docforge" / "cache" / "reference-adapter" / config.language
|
|
||||||
return AdapterProject(
|
|
||||||
loader,
|
|
||||||
cache_root=cache_root,
|
|
||||||
settings=AdapterProjectSettings(
|
|
||||||
descriptor_path=config.config_path,
|
|
||||||
implementation=AdapterImplementation(files=(config.config_path,)),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _adapter_loader(config: ReferenceAdapterConfigV1) -> AdapterLoader:
|
|
||||||
module_name, class_name = _FIXED_PROVIDERS[config.language]
|
|
||||||
try:
|
|
||||||
module = importlib.import_module(module_name)
|
|
||||||
except ModuleNotFoundError as error:
|
|
||||||
if error.name != module_name:
|
|
||||||
raise DocForgeError(
|
|
||||||
"optional_dependency_missing",
|
|
||||||
"The selected reference adapter dependency is not installed",
|
|
||||||
language=config.language,
|
|
||||||
missing_module=error.name,
|
|
||||||
install=_install_target(config.language),
|
|
||||||
) from error
|
|
||||||
raise DocForgeError(
|
|
||||||
"reference_adapter_unavailable",
|
|
||||||
"The selected fixed reference adapter is not available",
|
|
||||||
language=config.language,
|
|
||||||
) from error
|
|
||||||
constructor = getattr(module, class_name, None)
|
|
||||||
if not callable(constructor):
|
|
||||||
raise DocForgeError(
|
|
||||||
"reference_adapter_unavailable",
|
|
||||||
"The selected fixed reference adapter is not available",
|
|
||||||
language=config.language,
|
|
||||||
)
|
|
||||||
arguments: dict[str, object] = {
|
|
||||||
"source_roots": tuple(
|
|
||||||
path.relative_to(config.project_root).as_posix() for path in config.source_roots
|
|
||||||
),
|
|
||||||
"project_id": config.project_id,
|
|
||||||
"title": config.title,
|
|
||||||
}
|
|
||||||
if config.language == "cpp":
|
|
||||||
if config.compilation_database is None:
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_reference_config",
|
|
||||||
"C++ reference adapters require compilation_database",
|
|
||||||
)
|
|
||||||
arguments["compilation_database"] = config.compilation_database.relative_to(
|
|
||||||
config.project_root
|
|
||||||
).as_posix()
|
|
||||||
loader = constructor(config.project_root, **arguments)
|
|
||||||
if not callable(getattr(loader, "load_projection", None)):
|
|
||||||
raise DocForgeError(
|
|
||||||
"invalid_adapter",
|
|
||||||
"The selected fixed provider does not implement the adapter contract",
|
|
||||||
language=config.language,
|
|
||||||
)
|
|
||||||
return cast(AdapterLoader, loader)
|
|
||||||
|
|
||||||
|
|
||||||
def _install_target(language: str) -> str:
|
|
||||||
if language in {"javascript", "typescript"}:
|
|
||||||
return f"docforge[{language}]"
|
|
||||||
if language == "cpp":
|
|
||||||
return "docforge[cpp]"
|
|
||||||
return "docforge"
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
"""Run the fixed reference binding over stdio."""
|
|
||||||
|
|
||||||
parser = argparse.ArgumentParser(prog="python -m docforge.reference_mcp")
|
|
||||||
parser.add_argument("--project-root", type=Path, required=True)
|
|
||||||
parser.add_argument(
|
|
||||||
"--no-ast",
|
|
||||||
action="store_true",
|
|
||||||
help=(
|
|
||||||
"Preserve the existing adapter and forbid AST, Tree-sitter, compiler-AST, "
|
|
||||||
"and function-Logic extraction changes"
|
|
||||||
),
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"--diagnostics",
|
|
||||||
action="store_true",
|
|
||||||
help="Attach bounded request-local stage timings and counters",
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"--capability-mode",
|
|
||||||
choices=("read",),
|
|
||||||
help="Expose the fixed read-only project-bound capability surface",
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"--manual-render-policy",
|
|
||||||
choices=("auto", "explicit", "disabled"),
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"--portable-graph-policy",
|
|
||||||
choices=("explicit", "disabled"),
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"--live-viewer-policy",
|
|
||||||
choices=("on-demand", "disabled"),
|
|
||||||
)
|
|
||||||
arguments = parser.parse_args()
|
|
||||||
create_reference_server(
|
|
||||||
arguments.project_root,
|
|
||||||
no_ast=arguments.no_ast,
|
|
||||||
diagnostics=arguments.diagnostics,
|
|
||||||
capability_mode=arguments.capability_mode,
|
|
||||||
manual_projection_policy=arguments.manual_render_policy,
|
|
||||||
portable_graph_policy=arguments.portable_graph_policy,
|
|
||||||
live_viewer_policy=arguments.live_viewer_policy,
|
|
||||||
).run(transport="stdio")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
|
|
@ -9,20 +9,18 @@ data; it is never imported, compiled, or executed.
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import hashlib
|
import hashlib
|
||||||
import importlib
|
|
||||||
from collections.abc import Iterable
|
from collections.abc import Iterable
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from functools import lru_cache
|
from functools import lru_cache
|
||||||
from typing import TYPE_CHECKING
|
|
||||||
|
import tree_sitter_cpp
|
||||||
|
import tree_sitter_javascript
|
||||||
|
from tree_sitter import Language, Node, Parser
|
||||||
|
|
||||||
from .errors import DocForgeError
|
from .errors import DocForgeError
|
||||||
from .models import LogicEdge, LogicNode, LogicProjection
|
from .models import LogicEdge, LogicNode, LogicProjection
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
|
||||||
from tree_sitter import Language, Node
|
|
||||||
|
|
||||||
_COMMENT_NODE_TYPES = frozenset({"comment"})
|
_COMMENT_NODE_TYPES = frozenset({"comment"})
|
||||||
_SCRIPT_LANGUAGES = frozenset({"javascript", "typescript"})
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
|
|
@ -81,34 +79,7 @@ class _LanguageProfile:
|
||||||
def _javascript_profile() -> _LanguageProfile:
|
def _javascript_profile() -> _LanguageProfile:
|
||||||
return _LanguageProfile(
|
return _LanguageProfile(
|
||||||
name="javascript",
|
name="javascript",
|
||||||
language=_optional_language(
|
language=Language(tree_sitter_javascript.language()),
|
||||||
grammar_module="tree_sitter_javascript",
|
|
||||||
grammar_function="language",
|
|
||||||
extra="javascript",
|
|
||||||
),
|
|
||||||
root_type="program",
|
|
||||||
block_types=frozenset({"program", "statement_block"}),
|
|
||||||
function_types=frozenset(
|
|
||||||
{"function_declaration", "generator_function_declaration", "method_definition"}
|
|
||||||
),
|
|
||||||
loop_types=frozenset(
|
|
||||||
{"while_statement", "do_statement", "for_statement", "for_in_statement"}
|
|
||||||
),
|
|
||||||
return_types=frozenset({"return_statement"}),
|
|
||||||
raise_types=frozenset({"throw_statement"}),
|
|
||||||
switch_case_types=frozenset({"switch_case", "switch_default"}),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@lru_cache(maxsize=1)
|
|
||||||
def _typescript_profile() -> _LanguageProfile:
|
|
||||||
return _LanguageProfile(
|
|
||||||
name="typescript",
|
|
||||||
language=_optional_language(
|
|
||||||
grammar_module="tree_sitter_typescript",
|
|
||||||
grammar_function="language_typescript",
|
|
||||||
extra="typescript",
|
|
||||||
),
|
|
||||||
root_type="program",
|
root_type="program",
|
||||||
block_types=frozenset({"program", "statement_block"}),
|
block_types=frozenset({"program", "statement_block"}),
|
||||||
function_types=frozenset(
|
function_types=frozenset(
|
||||||
|
|
@ -127,11 +98,7 @@ def _typescript_profile() -> _LanguageProfile:
|
||||||
def _cpp_profile() -> _LanguageProfile:
|
def _cpp_profile() -> _LanguageProfile:
|
||||||
return _LanguageProfile(
|
return _LanguageProfile(
|
||||||
name="cpp",
|
name="cpp",
|
||||||
language=_optional_language(
|
language=Language(tree_sitter_cpp.language()),
|
||||||
grammar_module="tree_sitter_cpp",
|
|
||||||
grammar_function="language",
|
|
||||||
extra="cpp",
|
|
||||||
),
|
|
||||||
root_type="translation_unit",
|
root_type="translation_unit",
|
||||||
block_types=frozenset({"translation_unit", "compound_statement"}),
|
block_types=frozenset({"translation_unit", "compound_statement"}),
|
||||||
function_types=frozenset({"function_definition"}),
|
function_types=frozenset({"function_definition"}),
|
||||||
|
|
@ -149,28 +116,6 @@ def _cpp_profile() -> _LanguageProfile:
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def _optional_language(
|
|
||||||
*,
|
|
||||||
grammar_module: str,
|
|
||||||
grammar_function: str,
|
|
||||||
extra: str,
|
|
||||||
) -> Language:
|
|
||||||
try:
|
|
||||||
tree_sitter = importlib.import_module("tree_sitter")
|
|
||||||
grammar = importlib.import_module(grammar_module)
|
|
||||||
except ModuleNotFoundError as error:
|
|
||||||
raise DocForgeError(
|
|
||||||
"optional_dependency_missing",
|
|
||||||
"The requested language frontend is not installed",
|
|
||||||
extra=extra,
|
|
||||||
install=f"docforge[{extra}]",
|
|
||||||
missing_module=error.name,
|
|
||||||
) from error
|
|
||||||
language_type = tree_sitter.Language
|
|
||||||
language_factory = getattr(grammar, grammar_function)
|
|
||||||
return language_type(language_factory())
|
|
||||||
|
|
||||||
|
|
||||||
def analyze_javascript_source(
|
def analyze_javascript_source(
|
||||||
source: str,
|
source: str,
|
||||||
*,
|
*,
|
||||||
|
|
@ -191,38 +136,12 @@ def analyze_javascript_source(
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def analyze_typescript_source(
|
|
||||||
source: str,
|
|
||||||
*,
|
|
||||||
source_id: str,
|
|
||||||
owners: Iterable[TreeSitterLogicOwner],
|
|
||||||
filename: str = "<typescript-source>",
|
|
||||||
max_nodes_per_function: int = 2_000,
|
|
||||||
) -> tuple[LogicProjection, ...]:
|
|
||||||
"""Build control-flow projections for named TypeScript functions and methods."""
|
|
||||||
|
|
||||||
return _analyze_tree_sitter_source(
|
|
||||||
source,
|
|
||||||
source_id=source_id,
|
|
||||||
owners=owners,
|
|
||||||
filename=filename,
|
|
||||||
profile=_typescript_profile(),
|
|
||||||
max_nodes_per_function=max_nodes_per_function,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def discover_javascript_functions(source: str) -> tuple[DiscoveredFunction, ...]:
|
def discover_javascript_functions(source: str) -> tuple[DiscoveredFunction, ...]:
|
||||||
"""Return named JavaScript functions, methods, and assigned arrow functions."""
|
"""Return named JavaScript functions, methods, and assigned arrow functions."""
|
||||||
|
|
||||||
return _discover_functions(source, _javascript_profile())
|
return _discover_functions(source, _javascript_profile())
|
||||||
|
|
||||||
|
|
||||||
def discover_typescript_functions(source: str) -> tuple[DiscoveredFunction, ...]:
|
|
||||||
"""Return named TypeScript functions, methods, and assigned arrow functions."""
|
|
||||||
|
|
||||||
return _discover_functions(source, _typescript_profile())
|
|
||||||
|
|
||||||
|
|
||||||
def discover_cpp_functions(source: str) -> tuple[DiscoveredFunction, ...]:
|
def discover_cpp_functions(source: str) -> tuple[DiscoveredFunction, ...]:
|
||||||
"""Return named C++ functions and methods."""
|
"""Return named C++ functions and methods."""
|
||||||
|
|
||||||
|
|
@ -253,8 +172,6 @@ def _discover_functions(
|
||||||
source: str,
|
source: str,
|
||||||
profile: _LanguageProfile,
|
profile: _LanguageProfile,
|
||||||
) -> tuple[DiscoveredFunction, ...]:
|
) -> tuple[DiscoveredFunction, ...]:
|
||||||
from tree_sitter import Parser
|
|
||||||
|
|
||||||
raw = source.encode("utf-8")
|
raw = source.encode("utf-8")
|
||||||
parser = Parser(profile.language)
|
parser = Parser(profile.language)
|
||||||
tree = parser.parse(raw)
|
tree = parser.parse(raw)
|
||||||
|
|
@ -295,8 +212,6 @@ def _analyze_tree_sitter_source(
|
||||||
profile: _LanguageProfile,
|
profile: _LanguageProfile,
|
||||||
max_nodes_per_function: int,
|
max_nodes_per_function: int,
|
||||||
) -> tuple[LogicProjection, ...]:
|
) -> tuple[LogicProjection, ...]:
|
||||||
from tree_sitter import Parser
|
|
||||||
|
|
||||||
if max_nodes_per_function < 2:
|
if max_nodes_per_function < 2:
|
||||||
raise ValueError("max_nodes_per_function must allow entry and exit nodes")
|
raise ValueError("max_nodes_per_function must allow entry and exit nodes")
|
||||||
raw = source.encode("utf-8")
|
raw = source.encode("utf-8")
|
||||||
|
|
@ -378,7 +293,7 @@ def _function_definitions(
|
||||||
|
|
||||||
|
|
||||||
def _scope_name(node: Node, raw: bytes, profile: _LanguageProfile) -> str | None:
|
def _scope_name(node: Node, raw: bytes, profile: _LanguageProfile) -> str | None:
|
||||||
if profile.name in _SCRIPT_LANGUAGES and node.type in {"class_declaration", "class"}:
|
if profile.name == "javascript" and node.type in {"class_declaration", "class"}:
|
||||||
return _field_text(node, "name", raw)
|
return _field_text(node, "name", raw)
|
||||||
if profile.name == "cpp" and node.type in {
|
if profile.name == "cpp" and node.type in {
|
||||||
"namespace_definition",
|
"namespace_definition",
|
||||||
|
|
@ -392,11 +307,11 @@ def _scope_name(node: Node, raw: bytes, profile: _LanguageProfile) -> str | None
|
||||||
|
|
||||||
def _function_name(node: Node, raw: bytes, profile: _LanguageProfile) -> str | None:
|
def _function_name(node: Node, raw: bytes, profile: _LanguageProfile) -> str | None:
|
||||||
if node.type in profile.function_types:
|
if node.type in profile.function_types:
|
||||||
if profile.name in _SCRIPT_LANGUAGES:
|
if profile.name == "javascript":
|
||||||
return _field_text(node, "name", raw)
|
return _field_text(node, "name", raw)
|
||||||
declarator = node.child_by_field_name("declarator")
|
declarator = node.child_by_field_name("declarator")
|
||||||
return _declarator_name(declarator, raw) if declarator is not None else None
|
return _declarator_name(declarator, raw) if declarator is not None else None
|
||||||
if profile.name not in _SCRIPT_LANGUAGES or node.type != "variable_declarator":
|
if profile.name != "javascript" or node.type != "variable_declarator":
|
||||||
return None
|
return None
|
||||||
value = node.child_by_field_name("value")
|
value = node.child_by_field_name("value")
|
||||||
if value is None or value.type not in {"arrow_function", "function_expression"}:
|
if value is None or value.type not in {"arrow_function", "function_expression"}:
|
||||||
|
|
|
||||||
|
|
@ -1,19 +0,0 @@
|
||||||
[
|
|
||||||
{
|
|
||||||
"arguments": [
|
|
||||||
"c++",
|
|
||||||
"-std=c++20",
|
|
||||||
"-Iinclude",
|
|
||||||
"-c",
|
|
||||||
"src/main.cpp"
|
|
||||||
],
|
|
||||||
"directory": ".",
|
|
||||||
"file": "src/main.cpp"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"command": "c++ -std=c++20 -Iinclude -c src/worker.cpp",
|
|
||||||
"directory": ".",
|
|
||||||
"file": "src/worker.cpp",
|
|
||||||
"output": "worker.o"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
15
tests/fixtures/reference-cpp/include/config.hpp
vendored
15
tests/fixtures/reference-cpp/include/config.hpp
vendored
|
|
@ -1,15 +0,0 @@
|
||||||
#pragma once
|
|
||||||
|
|
||||||
#include "detail.hpp"
|
|
||||||
|
|
||||||
namespace demo {
|
|
||||||
|
|
||||||
struct Config {
|
|
||||||
int limit;
|
|
||||||
};
|
|
||||||
|
|
||||||
inline int effective_limit(const Config& config) {
|
|
||||||
return normalize(config.limit);
|
|
||||||
}
|
|
||||||
|
|
||||||
} // namespace demo
|
|
||||||
12
tests/fixtures/reference-cpp/include/detail.hpp
vendored
12
tests/fixtures/reference-cpp/include/detail.hpp
vendored
|
|
@ -1,12 +0,0 @@
|
||||||
#pragma once
|
|
||||||
|
|
||||||
namespace demo {
|
|
||||||
|
|
||||||
inline int normalize(int value) {
|
|
||||||
if (value < 0) {
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
return value;
|
|
||||||
}
|
|
||||||
|
|
||||||
} // namespace demo
|
|
||||||
17
tests/fixtures/reference-cpp/src/main.cpp
vendored
17
tests/fixtures/reference-cpp/src/main.cpp
vendored
|
|
@ -1,17 +0,0 @@
|
||||||
#include "../include/config.hpp"
|
|
||||||
|
|
||||||
namespace demo {
|
|
||||||
|
|
||||||
int execute(int value) {
|
|
||||||
Config config{value};
|
|
||||||
if (value == 0) {
|
|
||||||
return effective_limit(config);
|
|
||||||
}
|
|
||||||
return normalize(value);
|
|
||||||
}
|
|
||||||
|
|
||||||
} // namespace demo
|
|
||||||
|
|
||||||
int main() {
|
|
||||||
return demo::execute(3);
|
|
||||||
}
|
|
||||||
17
tests/fixtures/reference-cpp/src/worker.cpp
vendored
17
tests/fixtures/reference-cpp/src/worker.cpp
vendored
|
|
@ -1,17 +0,0 @@
|
||||||
#include "../include/config.hpp"
|
|
||||||
|
|
||||||
namespace demo {
|
|
||||||
|
|
||||||
class Worker {
|
|
||||||
public:
|
|
||||||
int run(int value) {
|
|
||||||
for (int attempt = 0; attempt < 2; ++attempt) {
|
|
||||||
if (value > attempt) {
|
|
||||||
return effective_limit(Config{value});
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
} // namespace demo
|
|
||||||
|
|
@ -1,8 +0,0 @@
|
||||||
import { Service } from "./service.mjs";
|
|
||||||
export { clamp } from "./shared.js";
|
|
||||||
|
|
||||||
throw new Error("The reference adapter must never execute project code");
|
|
||||||
|
|
||||||
export function applicationName() {
|
|
||||||
return Service.name;
|
|
||||||
}
|
|
||||||
|
|
@ -1,14 +0,0 @@
|
||||||
import { clamp } from "./shared.js";
|
|
||||||
|
|
||||||
export class Service {
|
|
||||||
run(value) {
|
|
||||||
return clamp(value);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
export const buildService = (enabled) => {
|
|
||||||
if (enabled) {
|
|
||||||
return new Service();
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
};
|
|
||||||
|
|
@ -1,11 +0,0 @@
|
||||||
export const DEFAULT_LIMIT = 3;
|
|
||||||
|
|
||||||
export function clamp(value, limit = DEFAULT_LIMIT) {
|
|
||||||
if (value < 0) {
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
if (value > limit) {
|
|
||||||
return limit;
|
|
||||||
}
|
|
||||||
return value;
|
|
||||||
}
|
|
||||||
|
|
@ -1,5 +0,0 @@
|
||||||
export { Service } from "./service.mjs";
|
|
||||||
|
|
||||||
export function execute(worker, value) {
|
|
||||||
return worker.run(value);
|
|
||||||
}
|
|
||||||
|
|
@ -1,7 +0,0 @@
|
||||||
"""Small multi-file package for the Python reference adapter."""
|
|
||||||
|
|
||||||
from .service import Service
|
|
||||||
|
|
||||||
|
|
||||||
def package_name() -> str:
|
|
||||||
return Service.__name__
|
|
||||||
|
|
@ -1,18 +0,0 @@
|
||||||
"""Service behavior with class, method, function, and local import evidence."""
|
|
||||||
|
|
||||||
import json
|
|
||||||
|
|
||||||
from .shared import clamp
|
|
||||||
|
|
||||||
raise RuntimeError("The reference adapter must never execute project code")
|
|
||||||
|
|
||||||
|
|
||||||
class Service:
|
|
||||||
def run(self, value: int) -> int:
|
|
||||||
return clamp(value) if json.__name__ else value
|
|
||||||
|
|
||||||
|
|
||||||
async def build_service(enabled: bool) -> Service | None:
|
|
||||||
if enabled:
|
|
||||||
return Service()
|
|
||||||
return None
|
|
||||||
|
|
@ -1,11 +0,0 @@
|
||||||
"""Shared values used by dependent modules."""
|
|
||||||
|
|
||||||
DEFAULT_LIMIT = 3
|
|
||||||
|
|
||||||
|
|
||||||
def clamp(value: int, limit: int = DEFAULT_LIMIT) -> int:
|
|
||||||
if value < 0:
|
|
||||||
return 0
|
|
||||||
if value > limit:
|
|
||||||
return limit
|
|
||||||
return value
|
|
||||||
|
|
@ -1,8 +0,0 @@
|
||||||
"""A transitive dependent used to prove reverse invalidation."""
|
|
||||||
|
|
||||||
from sample.service import Service
|
|
||||||
|
|
||||||
|
|
||||||
def execute(value: int) -> int:
|
|
||||||
worker = Service()
|
|
||||||
return worker.run(value)
|
|
||||||
|
|
@ -1,8 +0,0 @@
|
||||||
import { Service } from "./service";
|
|
||||||
export type { Choice } from "./types";
|
|
||||||
|
|
||||||
throw new Error("The reference adapter must never execute project code");
|
|
||||||
|
|
||||||
export function applicationName(): string {
|
|
||||||
return Service.name;
|
|
||||||
}
|
|
||||||
|
|
@ -1,17 +0,0 @@
|
||||||
import type { Choice } from "./types";
|
|
||||||
|
|
||||||
export class Service {
|
|
||||||
run(choice: Choice): number {
|
|
||||||
if (choice.enabled) {
|
|
||||||
return choice.value;
|
|
||||||
}
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
export const buildService = (choice: Choice): Service | null => {
|
|
||||||
if (choice.enabled) {
|
|
||||||
return new Service();
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
};
|
|
||||||
|
|
@ -1,6 +0,0 @@
|
||||||
export interface Choice {
|
|
||||||
enabled: boolean;
|
|
||||||
value: number;
|
|
||||||
}
|
|
||||||
|
|
||||||
export const DEFAULT_LIMIT: number = 3;
|
|
||||||
|
|
@ -1,6 +0,0 @@
|
||||||
import { Service } from "./service";
|
|
||||||
import type { Choice } from "./types";
|
|
||||||
|
|
||||||
export function execute(worker: Service, choice: Choice): number {
|
|
||||||
return worker.run(choice);
|
|
||||||
}
|
|
||||||
|
|
@ -39,7 +39,6 @@ from docforge.mcp_server import (
|
||||||
)
|
)
|
||||||
from docforge.models import (
|
from docforge.models import (
|
||||||
Edge,
|
Edge,
|
||||||
Limits,
|
|
||||||
LogicEdge,
|
LogicEdge,
|
||||||
LogicNode,
|
LogicNode,
|
||||||
LogicProjection,
|
LogicProjection,
|
||||||
|
|
@ -249,36 +248,6 @@ class NonLogicIncrementalLoader(IncrementalLoader):
|
||||||
return replace(super().extract_source(source), logic=())
|
return replace(super().extract_source(source), logic=())
|
||||||
|
|
||||||
|
|
||||||
class CompleteLogicIncrementalLoader(IncrementalLoader):
|
|
||||||
"""Supply an independent complete Logic oracle alongside the legacy graph oracle."""
|
|
||||||
|
|
||||||
def __init__(self, root: Path, *, mismatched_logic: bool = False) -> None:
|
|
||||||
super().__init__(root)
|
|
||||||
self.mismatched_logic = mismatched_logic
|
|
||||||
|
|
||||||
def load_complete_assembly(self) -> AdapterAssembly:
|
|
||||||
projection = super().load_projection()
|
|
||||||
if "guide.workflow" not in self.sources:
|
|
||||||
return AdapterAssembly(projection)
|
|
||||||
manifest = self.load_manifest()
|
|
||||||
workflow = next(
|
|
||||||
source for source in manifest.sources if source.source_id == "guide.workflow"
|
|
||||||
)
|
|
||||||
logic = self.extract_source(workflow).logic
|
|
||||||
if self.mismatched_logic:
|
|
||||||
current = logic[0]
|
|
||||||
logic = (
|
|
||||||
replace(
|
|
||||||
current,
|
|
||||||
nodes=(
|
|
||||||
current.nodes[0],
|
|
||||||
replace(current.nodes[1], label="Different complete return"),
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
return AdapterAssembly(projection, logic)
|
|
||||||
|
|
||||||
|
|
||||||
class AdapterContractTests(unittest.TestCase):
|
class AdapterContractTests(unittest.TestCase):
|
||||||
def projection(self, root: Path) -> AdapterProjection:
|
def projection(self, root: Path) -> AdapterProjection:
|
||||||
foundation = Node(
|
foundation = Node(
|
||||||
|
|
@ -342,23 +311,6 @@ class AdapterContractTests(unittest.TestCase):
|
||||||
self.assertEqual("Workflow", index.get_node("guide.workflow")["node"]["title"])
|
self.assertEqual("Workflow", index.get_node("guide.workflow")["node"]["title"])
|
||||||
self.assertEqual(projection.identity(), projection.identity())
|
self.assertEqual(projection.identity(), projection.identity())
|
||||||
|
|
||||||
def test_adapter_project_enforces_graph_bounds_before_publication(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = Path(directory).resolve()
|
|
||||||
project = AdapterProject(
|
|
||||||
Loader(self.projection(root)),
|
|
||||||
cache_root=root / ".cache" / "bounded",
|
|
||||||
settings=AdapterProjectSettings(limits=Limits(max_nodes=1)),
|
|
||||||
)
|
|
||||||
|
|
||||||
with self.assertRaises(DocForgeError) as captured:
|
|
||||||
project.load()
|
|
||||||
|
|
||||||
self.assertEqual("adapter_limit", captured.exception.code)
|
|
||||||
self.assertEqual("nodes", captured.exception.details["kind"])
|
|
||||||
self.assertEqual(1, captured.exception.details["maximum"])
|
|
||||||
self.assertEqual(2, captured.exception.details["actual"])
|
|
||||||
|
|
||||||
def test_projection_rejects_unsorted_metadata_graph_and_identity_changes(self) -> None:
|
def test_projection_rejects_unsorted_metadata_graph_and_identity_changes(self) -> None:
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
with tempfile.TemporaryDirectory() as directory:
|
||||||
root = Path(directory).resolve()
|
root = Path(directory).resolve()
|
||||||
|
|
@ -830,46 +782,6 @@ class AdapterContractTests(unittest.TestCase):
|
||||||
self.assertEqual("ok", equivalent["status"])
|
self.assertEqual("ok", equivalent["status"])
|
||||||
self.assertEqual(1, equivalent["node_count"])
|
self.assertEqual(1, equivalent["node_count"])
|
||||||
|
|
||||||
def test_logic_equivalence_requires_an_independent_complete_oracle(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = Path(directory).resolve()
|
|
||||||
project = AdapterProject(
|
|
||||||
IncrementalLoader(root),
|
|
||||||
cache_root=root / ".cache" / "logic-without-oracle",
|
|
||||||
)
|
|
||||||
|
|
||||||
with self.assertRaises(DocForgeError) as captured:
|
|
||||||
project.verify_incremental_equivalence()
|
|
||||||
|
|
||||||
self.assertEqual("complete_logic_oracle_required", captured.exception.code)
|
|
||||||
|
|
||||||
def test_logic_equivalence_uses_the_complete_assembly_oracle(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = Path(directory).resolve()
|
|
||||||
project = AdapterProject(
|
|
||||||
CompleteLogicIncrementalLoader(root),
|
|
||||||
cache_root=root / ".cache" / "logic-oracle",
|
|
||||||
)
|
|
||||||
|
|
||||||
result = project.verify_incremental_equivalence()
|
|
||||||
|
|
||||||
self.assertEqual("ok", result["status"])
|
|
||||||
self.assertEqual(1, result["logic_projection_count"])
|
|
||||||
|
|
||||||
def test_logic_equivalence_reports_exact_mismatch_field(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = Path(directory).resolve()
|
|
||||||
project = AdapterProject(
|
|
||||||
CompleteLogicIncrementalLoader(root, mismatched_logic=True),
|
|
||||||
cache_root=root / ".cache" / "logic-mismatch",
|
|
||||||
)
|
|
||||||
|
|
||||||
with self.assertRaises(DocForgeError) as captured:
|
|
||||||
project.verify_incremental_equivalence()
|
|
||||||
|
|
||||||
self.assertEqual("incremental_mismatch", captured.exception.code)
|
|
||||||
self.assertEqual(["logic"], captured.exception.details["fields"])
|
|
||||||
|
|
||||||
def test_incremental_adapter_can_assemble_overlapping_source_evidence(self) -> None:
|
def test_incremental_adapter_can_assemble_overlapping_source_evidence(self) -> None:
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
with tempfile.TemporaryDirectory() as directory:
|
||||||
root = Path(directory).resolve()
|
root = Path(directory).resolve()
|
||||||
|
|
|
||||||
|
|
@ -1,475 +0,0 @@
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import shutil
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
import tempfile
|
|
||||||
import tomllib
|
|
||||||
import unittest
|
|
||||||
import venv
|
|
||||||
from dataclasses import replace
|
|
||||||
from pathlib import Path
|
|
||||||
from unittest import mock
|
|
||||||
|
|
||||||
from jsonschema import Draft202012Validator
|
|
||||||
from mcp import ClientSession, StdioServerParameters
|
|
||||||
from mcp.client.stdio import stdio_client
|
|
||||||
from referencing import Registry, Resource
|
|
||||||
|
|
||||||
from docforge.adapter_contract import AdapterNode, AdapterProject, AdapterProjection
|
|
||||||
from docforge.adapter_launcher import AdapterLauncherV1
|
|
||||||
from docforge.client_config import (
|
|
||||||
_validate_adapter_configuration_result,
|
|
||||||
generate_adapter_client_configuration,
|
|
||||||
generate_client_configuration,
|
|
||||||
)
|
|
||||||
from docforge.errors import DocForgeError
|
|
||||||
from docforge.models import Node
|
|
||||||
from docforge.project import Project
|
|
||||||
from docforge.reference_mcp import (
|
|
||||||
REFERENCE_MCP_MODULE,
|
|
||||||
create_reference_project,
|
|
||||||
)
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
FIXTURES = ROOT / "tests" / "fixtures"
|
|
||||||
SCHEMAS = ROOT / "schemas"
|
|
||||||
LAUNCHER_SCHEMA = json.loads((SCHEMAS / "adapter-launcher.schema.json").read_text(encoding="utf-8"))
|
|
||||||
ADAPTER_CONFIGURATION_SCHEMA = json.loads(
|
|
||||||
(SCHEMAS / "adapter-client-configuration.schema.json").read_text(encoding="utf-8")
|
|
||||||
)
|
|
||||||
GENERIC_CONFIGURATION_SCHEMA = json.loads(
|
|
||||||
(SCHEMAS / "client-configuration.schema.json").read_text(encoding="utf-8")
|
|
||||||
)
|
|
||||||
SCHEMA_REGISTRY = Registry().with_resources(
|
|
||||||
(
|
|
||||||
(LAUNCHER_SCHEMA["$id"], Resource.from_contents(LAUNCHER_SCHEMA)),
|
|
||||||
(
|
|
||||||
GENERIC_CONFIGURATION_SCHEMA["$id"],
|
|
||||||
Resource.from_contents(GENERIC_CONFIGURATION_SCHEMA),
|
|
||||||
),
|
|
||||||
(
|
|
||||||
ADAPTER_CONFIGURATION_SCHEMA["$id"],
|
|
||||||
Resource.from_contents(ADAPTER_CONFIGURATION_SCHEMA),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
ADAPTER_CONFIGURATION_VALIDATOR = Draft202012Validator(
|
|
||||||
ADAPTER_CONFIGURATION_SCHEMA,
|
|
||||||
registry=SCHEMA_REGISTRY,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class Loader:
|
|
||||||
def __init__(self, projection: AdapterProjection) -> None:
|
|
||||||
self.projection = projection
|
|
||||||
self.load_calls = 0
|
|
||||||
|
|
||||||
def load_projection(self) -> AdapterProjection:
|
|
||||||
self.load_calls += 1
|
|
||||||
return self.projection
|
|
||||||
|
|
||||||
|
|
||||||
class DriftingLoader(Loader):
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
projection: AdapterProjection,
|
|
||||||
changed: AdapterProjection,
|
|
||||||
) -> None:
|
|
||||||
super().__init__(projection)
|
|
||||||
self.changed = changed
|
|
||||||
|
|
||||||
def load_projection(self) -> AdapterProjection:
|
|
||||||
self.load_calls += 1
|
|
||||||
return self.projection if self.load_calls < 3 else self.changed
|
|
||||||
|
|
||||||
|
|
||||||
class AdapterLauncherTests(unittest.TestCase):
|
|
||||||
def copy_fixture(self, destination: Path) -> Path:
|
|
||||||
root = destination / "alpha"
|
|
||||||
shutil.copytree(FIXTURES / "alpha", root)
|
|
||||||
return root
|
|
||||||
|
|
||||||
def copy_reference_fixture(self, destination: Path) -> Path:
|
|
||||||
root = destination / "reference-python"
|
|
||||||
shutil.copytree(FIXTURES / "reference-python", root)
|
|
||||||
config = root / ".docforge" / "reference-adapter.toml"
|
|
||||||
config.parent.mkdir(parents=True)
|
|
||||||
config.write_text(
|
|
||||||
"\n".join(
|
|
||||||
(
|
|
||||||
"schema_version = 1",
|
|
||||||
'project_id = "reference-python"',
|
|
||||||
'title = "Runnable Python reference"',
|
|
||||||
'language = "python"',
|
|
||||||
'source_roots = ["src"]',
|
|
||||||
"",
|
|
||||||
)
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
return root.resolve()
|
|
||||||
|
|
||||||
def projection(
|
|
||||||
self,
|
|
||||||
root: Path,
|
|
||||||
*,
|
|
||||||
source_hash: str = "3" * 64,
|
|
||||||
) -> AdapterProjection:
|
|
||||||
node = Node(
|
|
||||||
node_id="source.entry",
|
|
||||||
title="Entry",
|
|
||||||
family="source",
|
|
||||||
authority="derived",
|
|
||||||
status="active",
|
|
||||||
tags=("python",),
|
|
||||||
summary="Reference adapter entry.",
|
|
||||||
content="Reference adapter content.",
|
|
||||||
source_path="src/entry.py",
|
|
||||||
source_anchor="L1",
|
|
||||||
content_hash="2" * 64,
|
|
||||||
)
|
|
||||||
return AdapterProjection(
|
|
||||||
project_id="adapter-client-fixture",
|
|
||||||
title="Adapter client fixture",
|
|
||||||
adapter_id="fixture-client",
|
|
||||||
adapter_version="1",
|
|
||||||
root=root,
|
|
||||||
revision="fixture-revision",
|
|
||||||
source_hash=source_hash,
|
|
||||||
nodes=(AdapterNode(node),),
|
|
||||||
edges=(),
|
|
||||||
)
|
|
||||||
|
|
||||||
def project(
|
|
||||||
self,
|
|
||||||
root: Path,
|
|
||||||
*,
|
|
||||||
loader: Loader | None = None,
|
|
||||||
) -> tuple[AdapterProject, Loader]:
|
|
||||||
effective_loader = loader or Loader(self.projection(root))
|
|
||||||
return (
|
|
||||||
AdapterProject(
|
|
||||||
effective_loader,
|
|
||||||
cache_root=root / ".docforge-cache" / "adapter-client",
|
|
||||||
),
|
|
||||||
effective_loader,
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_launcher_and_all_client_fragments_are_deterministic_and_schema_valid(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = Path(directory).resolve()
|
|
||||||
project, _ = self.project(root)
|
|
||||||
launcher = AdapterLauncherV1.for_project(
|
|
||||||
project,
|
|
||||||
module=REFERENCE_MCP_MODULE,
|
|
||||||
)
|
|
||||||
Draft202012Validator(LAUNCHER_SCHEMA).validate(launcher.as_dict())
|
|
||||||
previous = os.environ.get("DOCFORGE_ADAPTER_LAUNCHER_SECRET")
|
|
||||||
os.environ["DOCFORGE_ADAPTER_LAUNCHER_SECRET"] = "must-not-appear"
|
|
||||||
try:
|
|
||||||
with mock.patch("docforge.client_config.subprocess.run") as executed:
|
|
||||||
results: dict[str, dict[str, object]] = {}
|
|
||||||
for client in ("codex", "claude", "openclaw"):
|
|
||||||
first = generate_adapter_client_configuration(
|
|
||||||
project,
|
|
||||||
launcher,
|
|
||||||
client,
|
|
||||||
no_ast=True,
|
|
||||||
)
|
|
||||||
second = generate_adapter_client_configuration(
|
|
||||||
project,
|
|
||||||
launcher,
|
|
||||||
client,
|
|
||||||
no_ast=True,
|
|
||||||
)
|
|
||||||
self.assertEqual(first, second)
|
|
||||||
ADAPTER_CONFIGURATION_VALIDATOR.validate(first)
|
|
||||||
self.assertEqual(launcher.launcher_hash, first["launcher_hash"])
|
|
||||||
self.assertEqual(
|
|
||||||
launcher.launcher_hash,
|
|
||||||
first["binding"]["launcher_hash"],
|
|
||||||
)
|
|
||||||
self.assertEqual(
|
|
||||||
[
|
|
||||||
"-I",
|
|
||||||
"-m",
|
|
||||||
REFERENCE_MCP_MODULE,
|
|
||||||
"--project-root",
|
|
||||||
str(root),
|
|
||||||
"--capability-mode",
|
|
||||||
"read",
|
|
||||||
"--no-ast",
|
|
||||||
],
|
|
||||||
first["binding"]["args"],
|
|
||||||
)
|
|
||||||
self.assertEqual({}, first["binding"]["environment"])
|
|
||||||
self.assertNotIn("cwd", first["binding"])
|
|
||||||
self.assertNotIn(
|
|
||||||
"must-not-appear",
|
|
||||||
json.dumps(first, sort_keys=True),
|
|
||||||
)
|
|
||||||
results[client] = first
|
|
||||||
executed.assert_not_called()
|
|
||||||
finally:
|
|
||||||
if previous is None:
|
|
||||||
os.environ.pop("DOCFORGE_ADAPTER_LAUNCHER_SECRET", None)
|
|
||||||
else:
|
|
||||||
os.environ["DOCFORGE_ADAPTER_LAUNCHER_SECRET"] = previous
|
|
||||||
|
|
||||||
self.assertIn(
|
|
||||||
"mcp_servers",
|
|
||||||
tomllib.loads(results["codex"]["artifact"]["content"]),
|
|
||||||
)
|
|
||||||
self.assertIn(
|
|
||||||
"mcpServers",
|
|
||||||
json.loads(results["claude"]["artifact"]["content"]),
|
|
||||||
)
|
|
||||||
self.assertIn(
|
|
||||||
"mcp",
|
|
||||||
json.loads(results["openclaw"]["artifact"]["content"]),
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_launcher_rejects_module_path_entry_point_and_argument_injection(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = Path(directory).resolve()
|
|
||||||
project, _ = self.project(root)
|
|
||||||
launcher = AdapterLauncherV1.for_project(
|
|
||||||
project,
|
|
||||||
module=REFERENCE_MCP_MODULE,
|
|
||||||
)
|
|
||||||
for module in (
|
|
||||||
"-m",
|
|
||||||
"fixture_adapter.mcp_server",
|
|
||||||
"fixture_adapter.mcp_server --debug",
|
|
||||||
"fixture_adapter:mcp_server",
|
|
||||||
"fixture_adapter/mcp_server",
|
|
||||||
".fixture_adapter",
|
|
||||||
"docforge.mcp_server",
|
|
||||||
"fixture_adapter.$server",
|
|
||||||
):
|
|
||||||
with self.subTest(module=module), self.assertRaises(DocForgeError) as captured:
|
|
||||||
replace(launcher, module=module)
|
|
||||||
self.assertEqual("invalid_adapter_launcher", captured.exception.code)
|
|
||||||
|
|
||||||
with self.assertRaises(DocForgeError):
|
|
||||||
replace(launcher, project_root=Path("relative/project"))
|
|
||||||
with self.assertRaises(DocForgeError):
|
|
||||||
replace(launcher, entry_point="console-script") # type: ignore[arg-type]
|
|
||||||
payload = launcher.as_dict()
|
|
||||||
payload["arguments"] = ["--shell", "command"]
|
|
||||||
with self.assertRaises(TypeError):
|
|
||||||
AdapterLauncherV1(**payload) # type: ignore[arg-type]
|
|
||||||
|
|
||||||
result = generate_adapter_client_configuration(project, launcher, "codex")
|
|
||||||
result["binding"]["args"].append("--arbitrary")
|
|
||||||
with self.assertRaisesRegex(AssertionError, "Generated adapter client"):
|
|
||||||
_validate_adapter_configuration_result(
|
|
||||||
result,
|
|
||||||
project=project,
|
|
||||||
launcher=launcher,
|
|
||||||
source_availability=project_source_availability(result),
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_wrong_project_descriptor_and_source_drift_fail_closed(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
parent = Path(directory)
|
|
||||||
first_root = (parent / "first").resolve()
|
|
||||||
second_root = (parent / "second").resolve()
|
|
||||||
first_root.mkdir()
|
|
||||||
second_root.mkdir()
|
|
||||||
first, _ = self.project(first_root)
|
|
||||||
second, _ = self.project(second_root)
|
|
||||||
launcher = AdapterLauncherV1.for_project(
|
|
||||||
first,
|
|
||||||
module=REFERENCE_MCP_MODULE,
|
|
||||||
)
|
|
||||||
|
|
||||||
with self.assertRaises(DocForgeError) as wrong_project:
|
|
||||||
generate_adapter_client_configuration(second, launcher, "codex")
|
|
||||||
self.assertEqual("adapter_launcher_mismatch", wrong_project.exception.code)
|
|
||||||
with self.assertRaises(DocForgeError) as descriptor_drift:
|
|
||||||
generate_adapter_client_configuration(
|
|
||||||
first,
|
|
||||||
replace(launcher, descriptor_hash="0" * 64),
|
|
||||||
"codex",
|
|
||||||
)
|
|
||||||
self.assertEqual("adapter_launcher_mismatch", descriptor_drift.exception.code)
|
|
||||||
|
|
||||||
base = self.projection(first_root)
|
|
||||||
changed = replace(base, source_hash="4" * 64)
|
|
||||||
drifting, _ = self.project(
|
|
||||||
first_root,
|
|
||||||
loader=DriftingLoader(base, changed),
|
|
||||||
)
|
|
||||||
drifting_launcher = AdapterLauncherV1.for_project(
|
|
||||||
drifting,
|
|
||||||
module=REFERENCE_MCP_MODULE,
|
|
||||||
)
|
|
||||||
with self.assertRaises(DocForgeError) as source_drift:
|
|
||||||
generate_adapter_client_configuration(
|
|
||||||
drifting,
|
|
||||||
drifting_launcher,
|
|
||||||
"openclaw",
|
|
||||||
)
|
|
||||||
self.assertEqual("source_changed", source_drift.exception.code)
|
|
||||||
|
|
||||||
def test_generic_configuration_remains_separate_and_compatible(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
generic = Project.open(root)
|
|
||||||
result = generate_client_configuration(generic, "codex", no_ast=True)
|
|
||||||
Draft202012Validator(GENERIC_CONFIGURATION_SCHEMA).validate(result)
|
|
||||||
self.assertEqual("client.configure", result["operation"])
|
|
||||||
self.assertNotIn("launcher", result)
|
|
||||||
with self.assertRaises(DocForgeError) as generic_launcher:
|
|
||||||
AdapterLauncherV1.for_project(
|
|
||||||
generic,
|
|
||||||
module=REFERENCE_MCP_MODULE,
|
|
||||||
)
|
|
||||||
self.assertEqual("adapter_launcher_unavailable", generic_launcher.exception.code)
|
|
||||||
|
|
||||||
custom_root = (Path(directory) / "custom").resolve()
|
|
||||||
custom_root.mkdir()
|
|
||||||
custom, _ = self.project(custom_root)
|
|
||||||
with self.assertRaises(DocForgeError) as generic_api:
|
|
||||||
generate_client_configuration(custom, "codex")
|
|
||||||
self.assertEqual("missing_config", generic_api.exception.code)
|
|
||||||
|
|
||||||
def test_installed_project_owned_top_level_module_launches_without_probe_execution(
|
|
||||||
self,
|
|
||||||
) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = Path(directory).resolve()
|
|
||||||
environment = root / ".launcher-venv"
|
|
||||||
venv.EnvBuilder(with_pip=False).create(environment)
|
|
||||||
executable = environment / "bin" / "python"
|
|
||||||
site_packages = Path(
|
|
||||||
subprocess.run(
|
|
||||||
[
|
|
||||||
str(executable),
|
|
||||||
"-I",
|
|
||||||
"-c",
|
|
||||||
"import site; print(site.getsitepackages()[0])",
|
|
||||||
],
|
|
||||||
check=True,
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
).stdout.strip()
|
|
||||||
)
|
|
||||||
(site_packages / "fixture-adapter.pth").write_text(
|
|
||||||
f"{root}\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
sentinel = root / "fixture-adapter-executed"
|
|
||||||
(root / "fixture_adapter.py").write_text(
|
|
||||||
"\n".join(
|
|
||||||
(
|
|
||||||
"import argparse",
|
|
||||||
"import json",
|
|
||||||
"from pathlib import Path",
|
|
||||||
"parser = argparse.ArgumentParser()",
|
|
||||||
'parser.add_argument("--project-root", required=True)',
|
|
||||||
(
|
|
||||||
'parser.add_argument("--capability-mode", '
|
|
||||||
'choices=("read",), required=True)'
|
|
||||||
),
|
|
||||||
"arguments = parser.parse_args()",
|
|
||||||
f"Path({str(sentinel)!r}).write_text('executed', encoding='utf-8')",
|
|
||||||
(
|
|
||||||
"print(json.dumps({'project_root': arguments.project_root}, "
|
|
||||||
"sort_keys=True))"
|
|
||||||
),
|
|
||||||
"",
|
|
||||||
)
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
|
|
||||||
project, _ = self.project(root)
|
|
||||||
with mock.patch.object(sys, "executable", str(executable)):
|
|
||||||
launcher = AdapterLauncherV1.for_project(
|
|
||||||
project,
|
|
||||||
module="fixture_adapter",
|
|
||||||
)
|
|
||||||
result = generate_adapter_client_configuration(
|
|
||||||
project,
|
|
||||||
launcher,
|
|
||||||
"codex",
|
|
||||||
)
|
|
||||||
self.assertFalse(sentinel.exists())
|
|
||||||
Draft202012Validator(LAUNCHER_SCHEMA).validate(launcher.as_dict())
|
|
||||||
binding = result["binding"]
|
|
||||||
completed = subprocess.run(
|
|
||||||
[binding["command"], *binding["args"]],
|
|
||||||
check=True,
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
timeout=10,
|
|
||||||
)
|
|
||||||
self.assertEqual({"project_root": str(root)}, json.loads(completed.stdout))
|
|
||||||
self.assertEqual("executed", sentinel.read_text(encoding="utf-8"))
|
|
||||||
|
|
||||||
def test_uninstalled_packages_and_non_project_modules_fail_closed(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = Path(directory).resolve()
|
|
||||||
project, _ = self.project(root)
|
|
||||||
for module, code in (
|
|
||||||
("fixture_adapter", "adapter_launcher_unavailable"),
|
|
||||||
("os", "adapter_launcher_unavailable"),
|
|
||||||
("json", "invalid_adapter_launcher"),
|
|
||||||
):
|
|
||||||
with self.subTest(module=module), self.assertRaises(DocForgeError) as captured:
|
|
||||||
AdapterLauncherV1.for_project(project, module=module)
|
|
||||||
self.assertEqual(code, captured.exception.code)
|
|
||||||
|
|
||||||
def test_generated_reference_binding_starts_a_real_isolated_mcp_process(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_reference_fixture(Path(directory))
|
|
||||||
project = create_reference_project(root)
|
|
||||||
launcher = AdapterLauncherV1.for_project(
|
|
||||||
project,
|
|
||||||
module=REFERENCE_MCP_MODULE,
|
|
||||||
)
|
|
||||||
result = generate_adapter_client_configuration(
|
|
||||||
project,
|
|
||||||
launcher,
|
|
||||||
"codex",
|
|
||||||
)
|
|
||||||
binding = result["binding"]
|
|
||||||
|
|
||||||
async def inspect() -> tuple[str, ...]:
|
|
||||||
parameters = StdioServerParameters(
|
|
||||||
command=binding["command"],
|
|
||||||
args=binding["args"],
|
|
||||||
env=binding["environment"],
|
|
||||||
)
|
|
||||||
async with (
|
|
||||||
stdio_client(parameters) as streams,
|
|
||||||
ClientSession(*streams) as session,
|
|
||||||
):
|
|
||||||
await session.initialize()
|
|
||||||
return tuple(tool.name for tool in (await session.list_tools()).tools)
|
|
||||||
|
|
||||||
tools = asyncio.run(inspect())
|
|
||||||
self.assertIn("docforge_bootstrap", tools)
|
|
||||||
|
|
||||||
|
|
||||||
def project_source_availability(result: dict[str, object]):
|
|
||||||
from docforge.adapter_launcher import AdapterSourceAvailabilityV1
|
|
||||||
|
|
||||||
payload = result["source_availability"]
|
|
||||||
return AdapterSourceAvailabilityV1(
|
|
||||||
schema_version=payload["schema_version"],
|
|
||||||
status=payload["status"],
|
|
||||||
method=payload["method"],
|
|
||||||
revision=payload["revision"],
|
|
||||||
source_hash=payload["source_hash"],
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
unittest.main()
|
|
||||||
|
|
@ -1,261 +0,0 @@
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import tempfile
|
|
||||||
import unittest
|
|
||||||
from dataclasses import replace
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import docforge.adapter_sdk as sdk
|
|
||||||
from docforge.errors import DocForgeError
|
|
||||||
|
|
||||||
|
|
||||||
class ReferenceIncrementalAdapter:
|
|
||||||
"""Small public-SDK-only adapter used to prove the conformance entry point."""
|
|
||||||
|
|
||||||
def __init__(self, root: Path) -> None:
|
|
||||||
self.root = root
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _hash(value: str) -> str:
|
|
||||||
return hashlib.sha256(value.encode()).hexdigest()
|
|
||||||
|
|
||||||
def _node(self) -> sdk.AdapterNode:
|
|
||||||
content = "A deterministic reference function."
|
|
||||||
return sdk.AdapterNode(
|
|
||||||
sdk.Node(
|
|
||||||
node_id="reference.run",
|
|
||||||
title="Reference run",
|
|
||||||
family="source",
|
|
||||||
authority="derived",
|
|
||||||
status="current",
|
|
||||||
tags=("python",),
|
|
||||||
summary=content,
|
|
||||||
content=content,
|
|
||||||
source_path="src/reference.py",
|
|
||||||
source_anchor="L1",
|
|
||||||
content_hash=self._hash(content),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _logic() -> sdk.LogicProjection:
|
|
||||||
return sdk.LogicProjection(
|
|
||||||
owner_node_id="reference.run",
|
|
||||||
source_id="source.reference",
|
|
||||||
nodes=(
|
|
||||||
sdk.LogicNode("entry", "entry", "Enter run", "L1"),
|
|
||||||
sdk.LogicNode("return", "return", "Return", "L2"),
|
|
||||||
),
|
|
||||||
edges=(sdk.LogicEdge("entry", "return", "return", "RETURN", 0),),
|
|
||||||
)
|
|
||||||
|
|
||||||
def load_manifest(self) -> sdk.AdapterManifest:
|
|
||||||
fingerprint = self._hash("def run(): return True\n")
|
|
||||||
return sdk.AdapterManifest(
|
|
||||||
project_id="sdk-reference",
|
|
||||||
title="SDK reference",
|
|
||||||
adapter_id="python-reference",
|
|
||||||
adapter_version="1",
|
|
||||||
root=self.root,
|
|
||||||
revision=fingerprint[:12],
|
|
||||||
source_hash=fingerprint,
|
|
||||||
families=("source",),
|
|
||||||
allowed_relations=(),
|
|
||||||
sources=(
|
|
||||||
sdk.AdapterSource(
|
|
||||||
source_id="source.reference",
|
|
||||||
source_path="src/reference.py",
|
|
||||||
fingerprint=fingerprint,
|
|
||||||
extractor_version="python-reference@1",
|
|
||||||
),
|
|
||||||
),
|
|
||||||
estimated_nodes=1,
|
|
||||||
)
|
|
||||||
|
|
||||||
def extract_source(self, source: sdk.AdapterSource) -> sdk.AdapterSourceProjection:
|
|
||||||
return sdk.AdapterSourceProjection(
|
|
||||||
source_id=source.source_id,
|
|
||||||
fingerprint=source.fingerprint,
|
|
||||||
nodes=(self._node(),),
|
|
||||||
edges=(),
|
|
||||||
logic=(self._logic(),),
|
|
||||||
)
|
|
||||||
|
|
||||||
def load_projection(self) -> sdk.AdapterProjection:
|
|
||||||
manifest = self.load_manifest()
|
|
||||||
return sdk.AdapterProjection(
|
|
||||||
project_id=manifest.project_id,
|
|
||||||
title=manifest.title,
|
|
||||||
adapter_id=manifest.adapter_id,
|
|
||||||
adapter_version=manifest.adapter_version,
|
|
||||||
root=manifest.root,
|
|
||||||
revision=manifest.revision,
|
|
||||||
source_hash=manifest.source_hash,
|
|
||||||
nodes=(self._node(),),
|
|
||||||
edges=(),
|
|
||||||
)
|
|
||||||
|
|
||||||
def load_complete_assembly(self) -> sdk.AdapterAssembly:
|
|
||||||
return sdk.AdapterAssembly(
|
|
||||||
projection=self.load_projection(),
|
|
||||||
logic=(self._logic(),),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class LegacyAdapter:
|
|
||||||
def __init__(self, projection: sdk.AdapterProjection) -> None:
|
|
||||||
self.projection = projection
|
|
||||||
|
|
||||||
def load_projection(self) -> sdk.AdapterProjection:
|
|
||||||
return self.projection
|
|
||||||
|
|
||||||
|
|
||||||
class LogicWithoutCompleteAdapter:
|
|
||||||
def __init__(self, root: Path) -> None:
|
|
||||||
self.reference = ReferenceIncrementalAdapter(root)
|
|
||||||
|
|
||||||
def load_manifest(self) -> sdk.AdapterManifest:
|
|
||||||
return self.reference.load_manifest()
|
|
||||||
|
|
||||||
def extract_source(self, source: sdk.AdapterSource) -> sdk.AdapterSourceProjection:
|
|
||||||
return self.reference.extract_source(source)
|
|
||||||
|
|
||||||
def load_projection(self) -> sdk.AdapterProjection:
|
|
||||||
return self.reference.load_projection()
|
|
||||||
|
|
||||||
|
|
||||||
class NondeterministicCompleteAdapter(ReferenceIncrementalAdapter):
|
|
||||||
def __init__(self, root: Path) -> None:
|
|
||||||
super().__init__(root)
|
|
||||||
self.complete_calls = 0
|
|
||||||
|
|
||||||
def load_complete_assembly(self) -> sdk.AdapterAssembly:
|
|
||||||
self.complete_calls += 1
|
|
||||||
assembly = super().load_complete_assembly()
|
|
||||||
if self.complete_calls % 2 == 0:
|
|
||||||
logic = assembly.logic[0]
|
|
||||||
return replace(
|
|
||||||
assembly,
|
|
||||||
logic=(
|
|
||||||
replace(
|
|
||||||
logic,
|
|
||||||
nodes=(
|
|
||||||
logic.nodes[0],
|
|
||||||
replace(logic.nodes[1], label="Unstable return"),
|
|
||||||
),
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
return assembly
|
|
||||||
|
|
||||||
|
|
||||||
class AdapterSdkTests(unittest.TestCase):
|
|
||||||
def test_public_facade_contains_the_complete_authoring_surface(self) -> None:
|
|
||||||
expected = {
|
|
||||||
"AdapterAssembly",
|
|
||||||
"AdapterConformanceReport",
|
|
||||||
"AdapterManifest",
|
|
||||||
"AdapterNode",
|
|
||||||
"AdapterProjection",
|
|
||||||
"AdapterSource",
|
|
||||||
"AdapterSourceProjection",
|
|
||||||
"CompleteAdapterAssemblyLoader",
|
|
||||||
"Edge",
|
|
||||||
"LogicEdge",
|
|
||||||
"LogicNode",
|
|
||||||
"LogicProjection",
|
|
||||||
"Node",
|
|
||||||
"verify_adapter_conformance",
|
|
||||||
}
|
|
||||||
|
|
||||||
self.assertLessEqual(expected, set(sdk.__all__))
|
|
||||||
for name in expected:
|
|
||||||
with self.subTest(name=name):
|
|
||||||
self.assertTrue(hasattr(sdk, name))
|
|
||||||
|
|
||||||
def test_conformance_proves_deterministic_complete_graph_and_logic_parity(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = Path(directory).resolve()
|
|
||||||
report = sdk.verify_adapter_conformance(
|
|
||||||
ReferenceIncrementalAdapter(root),
|
|
||||||
cache_root=root / ".cache" / "conformance",
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual(1, report.schema_version)
|
|
||||||
self.assertTrue(report.incremental)
|
|
||||||
self.assertEqual(1, report.node_count)
|
|
||||||
self.assertEqual(0, report.edge_count)
|
|
||||||
self.assertEqual(1, report.logic_projection_count)
|
|
||||||
self.assertEqual(64, len(report.assembly_hash))
|
|
||||||
self.assertEqual(report.assembly_hash, report.as_dict()["assembly_hash"])
|
|
||||||
|
|
||||||
def test_conformance_keeps_load_projection_only_adapters_first_class(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = Path(directory).resolve()
|
|
||||||
content = "Legacy adapter content."
|
|
||||||
projection = sdk.AdapterProjection(
|
|
||||||
project_id="legacy-sdk",
|
|
||||||
title="Legacy SDK",
|
|
||||||
adapter_id="legacy",
|
|
||||||
adapter_version="1",
|
|
||||||
root=root,
|
|
||||||
revision="legacy",
|
|
||||||
source_hash=hashlib.sha256(content.encode()).hexdigest(),
|
|
||||||
nodes=(
|
|
||||||
sdk.AdapterNode(
|
|
||||||
sdk.Node(
|
|
||||||
node_id="legacy.node",
|
|
||||||
title="Legacy node",
|
|
||||||
family="guide",
|
|
||||||
authority="authoritative",
|
|
||||||
status="current",
|
|
||||||
tags=("legacy",),
|
|
||||||
summary=content,
|
|
||||||
content=content,
|
|
||||||
source_path="docs/legacy.md",
|
|
||||||
source_anchor=None,
|
|
||||||
content_hash=hashlib.sha256(content.encode()).hexdigest(),
|
|
||||||
)
|
|
||||||
),
|
|
||||||
),
|
|
||||||
edges=(),
|
|
||||||
)
|
|
||||||
|
|
||||||
report = sdk.verify_adapter_conformance(
|
|
||||||
LegacyAdapter(projection),
|
|
||||||
cache_root=root / ".cache" / "legacy-conformance",
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertFalse(report.incremental)
|
|
||||||
self.assertEqual(0, report.logic_projection_count)
|
|
||||||
self.assertEqual(projection.identity(), projection.identity())
|
|
||||||
|
|
||||||
def test_conformance_requires_a_complete_oracle_for_incremental_logic(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = Path(directory).resolve()
|
|
||||||
|
|
||||||
with self.assertRaises(DocForgeError) as captured:
|
|
||||||
sdk.verify_adapter_conformance(
|
|
||||||
LogicWithoutCompleteAdapter(root),
|
|
||||||
cache_root=root / ".cache" / "missing-logic-oracle",
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual("complete_logic_oracle_required", captured.exception.code)
|
|
||||||
|
|
||||||
def test_conformance_rejects_nondeterministic_complete_logic(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = Path(directory).resolve()
|
|
||||||
|
|
||||||
with self.assertRaises(DocForgeError) as captured:
|
|
||||||
sdk.verify_adapter_conformance(
|
|
||||||
NondeterministicCompleteAdapter(root),
|
|
||||||
cache_root=root / ".cache" / "nondeterministic",
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual("nondeterministic_adapter", captured.exception.code)
|
|
||||||
self.assertEqual(["logic"], captured.exception.details["fields"])
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
unittest.main()
|
|
||||||
|
|
@ -1,141 +0,0 @@
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import shutil
|
|
||||||
import tempfile
|
|
||||||
import unittest
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from mcp.shared.memory import create_connected_server_and_client_session
|
|
||||||
from mcp.types import Tool
|
|
||||||
|
|
||||||
from docforge.cli import _parser
|
|
||||||
from docforge.command_reference import (
|
|
||||||
cli_command_references,
|
|
||||||
mcp_tool_references,
|
|
||||||
render_cli_reference_markdown,
|
|
||||||
render_command_reference_markdown,
|
|
||||||
render_mcp_reference_markdown,
|
|
||||||
)
|
|
||||||
from docforge.index import ProjectIndex
|
|
||||||
from docforge.mcp_server import ALL_TOOLS, APPLICATION_TOOLS, create_server
|
|
||||||
from docforge.project import Project
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
FIXTURES = ROOT / "tests" / "fixtures"
|
|
||||||
|
|
||||||
|
|
||||||
class CommandReferenceTests(unittest.IsolatedAsyncioTestCase):
|
|
||||||
def copy_fixture(self, destination: Path) -> Path:
|
|
||||||
root = destination / "alpha"
|
|
||||||
shutil.copytree(FIXTURES / "alpha", root)
|
|
||||||
return root
|
|
||||||
|
|
||||||
def test_cli_reference_is_derived_from_the_real_parser(self) -> None:
|
|
||||||
parser = _parser()
|
|
||||||
references = cli_command_references(parser)
|
|
||||||
subparsers = next(
|
|
||||||
action
|
|
||||||
for action in parser._actions # pyright: ignore[reportPrivateUsage]
|
|
||||||
if isinstance(
|
|
||||||
action,
|
|
||||||
argparse._SubParsersAction, # pyright: ignore[reportPrivateUsage]
|
|
||||||
)
|
|
||||||
)
|
|
||||||
expected = {
|
|
||||||
name: " ".join(command.format_usage().removeprefix("usage: ").split())
|
|
||||||
for name, command in subparsers.choices.items()
|
|
||||||
}
|
|
||||||
|
|
||||||
self.assertEqual(tuple(sorted(expected)), tuple(item.name for item in references))
|
|
||||||
self.assertEqual(expected, {item.name: item.invocation for item in references})
|
|
||||||
rendered = render_cli_reference_markdown(references)
|
|
||||||
self.assertEqual(rendered, render_cli_reference_markdown(reversed(references)))
|
|
||||||
self.assertIn("| `search` | `docforge search [-h] [--limit LIMIT] query` |", rendered)
|
|
||||||
self.assertIn(
|
|
||||||
"| `graph-render-status` | `docforge graph-render-status [-h] [view_id]` |",
|
|
||||||
rendered,
|
|
||||||
)
|
|
||||||
|
|
||||||
async def test_mcp_reference_is_derived_from_registered_tool_metadata(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
ProjectIndex(Project.open(root)).build()
|
|
||||||
async with create_connected_server_and_client_session(
|
|
||||||
create_server(
|
|
||||||
root,
|
|
||||||
"alpha-editor",
|
|
||||||
canonical_applier_id="alpha-editor",
|
|
||||||
),
|
|
||||||
raise_exceptions=True,
|
|
||||||
) as session:
|
|
||||||
tools = (await session.list_tools()).tools
|
|
||||||
|
|
||||||
expected_names = (*ALL_TOOLS, *APPLICATION_TOOLS)
|
|
||||||
references = mcp_tool_references(tools, expected_names=expected_names)
|
|
||||||
by_name = {item.name: item for item in references}
|
|
||||||
tools_by_name = {tool.name: tool for tool in tools}
|
|
||||||
|
|
||||||
self.assertEqual(set(expected_names), set(by_name))
|
|
||||||
for name, tool in tools_by_name.items():
|
|
||||||
with self.subTest(tool=name):
|
|
||||||
schema_hash = hashlib.sha256(
|
|
||||||
json.dumps(
|
|
||||||
tool.inputSchema,
|
|
||||||
sort_keys=True,
|
|
||||||
separators=(",", ":"),
|
|
||||||
).encode()
|
|
||||||
).hexdigest()
|
|
||||||
properties = set(tool.inputSchema.get("properties", {}))
|
|
||||||
required = set(tool.inputSchema.get("required", []))
|
|
||||||
reference = by_name[name]
|
|
||||||
self.assertEqual(" ".join((tool.description or "").split()), reference.description)
|
|
||||||
self.assertEqual(tuple(sorted(required)), reference.required_arguments)
|
|
||||||
self.assertEqual(
|
|
||||||
tuple(sorted(properties - required)),
|
|
||||||
reference.optional_arguments,
|
|
||||||
)
|
|
||||||
self.assertEqual(schema_hash, reference.input_schema_hash)
|
|
||||||
|
|
||||||
rendered = render_mcp_reference_markdown(references)
|
|
||||||
self.assertEqual(rendered, render_mcp_reference_markdown(reversed(references)))
|
|
||||||
self.assertIn("| read | `docforge_bootstrap` | — | — |", rendered)
|
|
||||||
self.assertIn(
|
|
||||||
"| application | `docforge_apply_changeset` | "
|
|
||||||
"`changeset_id`, `expected_changeset_hash` | — |",
|
|
||||||
rendered,
|
|
||||||
)
|
|
||||||
complete = render_command_reference_markdown(
|
|
||||||
cli_command_references(),
|
|
||||||
references,
|
|
||||||
)
|
|
||||||
self.assertEqual(1, complete.count("## CLI commands"))
|
|
||||||
self.assertEqual(1, complete.count("## MCP tools"))
|
|
||||||
self.assertTrue(complete.startswith("# DocForge command reference\n"))
|
|
||||||
self.assertIn("Do not edit this file by hand.", complete)
|
|
||||||
self.assertIn("fixed reference-adapter server exposes only its read rows", complete)
|
|
||||||
|
|
||||||
def test_mcp_reference_fails_closed_on_registration_drift(self) -> None:
|
|
||||||
known = Tool(
|
|
||||||
name="docforge_bootstrap",
|
|
||||||
description="Bootstrap.",
|
|
||||||
inputSchema={"type": "object", "properties": {}},
|
|
||||||
)
|
|
||||||
unknown = Tool(
|
|
||||||
name="docforge_unregistered",
|
|
||||||
description="Unknown.",
|
|
||||||
inputSchema={"type": "object", "properties": {}},
|
|
||||||
)
|
|
||||||
|
|
||||||
with self.assertRaisesRegex(ValueError, "no declared capability surface"):
|
|
||||||
mcp_tool_references((unknown,))
|
|
||||||
with self.assertRaisesRegex(ValueError, "missing="):
|
|
||||||
mcp_tool_references((known,), expected_names=ALL_TOOLS)
|
|
||||||
with self.assertRaisesRegex(ValueError, "repeats"):
|
|
||||||
mcp_tool_references((known, known))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
unittest.main()
|
|
||||||
|
|
@ -1,404 +0,0 @@
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import contextlib
|
|
||||||
import hashlib
|
|
||||||
import io
|
|
||||||
import os
|
|
||||||
import tempfile
|
|
||||||
import threading
|
|
||||||
import unittest
|
|
||||||
from pathlib import Path
|
|
||||||
from unittest import mock
|
|
||||||
|
|
||||||
from docforge.mcp_server import ALL_TOOLS, APPLICATION_TOOLS
|
|
||||||
from tools import generate_command_reference as command_reference_tool
|
|
||||||
from tools.generate_command_reference import (
|
|
||||||
EXPECTED_CLI_ROWS,
|
|
||||||
EXPECTED_MCP_ROWS,
|
|
||||||
CommandReferenceDrift,
|
|
||||||
CommandReferenceToolError,
|
|
||||||
collect_command_reference_rows,
|
|
||||||
generate_command_reference_bytes,
|
|
||||||
main,
|
|
||||||
publish_or_check_command_reference,
|
|
||||||
)
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
ALPHA = ROOT / "tests" / "fixtures" / "alpha"
|
|
||||||
|
|
||||||
|
|
||||||
def _tree_snapshot(root: Path) -> tuple[tuple[str, str, str], ...]:
|
|
||||||
records: list[tuple[str, str, str]] = []
|
|
||||||
for path in sorted(root.rglob("*")):
|
|
||||||
relative = path.relative_to(root).as_posix()
|
|
||||||
if path.is_symlink():
|
|
||||||
records.append((relative, "symlink", os.readlink(path)))
|
|
||||||
elif path.is_dir():
|
|
||||||
records.append((relative, "directory", ""))
|
|
||||||
else:
|
|
||||||
records.append(
|
|
||||||
(
|
|
||||||
relative,
|
|
||||||
"file",
|
|
||||||
hashlib.sha256(path.read_bytes()).hexdigest(),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return tuple(records)
|
|
||||||
|
|
||||||
|
|
||||||
class CommandReferenceToolTests(unittest.TestCase):
|
|
||||||
def test_generated_bytes_are_stable_and_never_mutate_the_fixture(self) -> None:
|
|
||||||
before = _tree_snapshot(ALPHA)
|
|
||||||
|
|
||||||
first = generate_command_reference_bytes(ALPHA)
|
|
||||||
middle = _tree_snapshot(ALPHA)
|
|
||||||
second = generate_command_reference_bytes(ALPHA)
|
|
||||||
|
|
||||||
self.assertEqual(first, second)
|
|
||||||
self.assertEqual(before, middle)
|
|
||||||
self.assertEqual(before, _tree_snapshot(ALPHA))
|
|
||||||
self.assertTrue(first.endswith(b"\n"))
|
|
||||||
self.assertTrue(first.startswith(b"# DocForge command reference\n"))
|
|
||||||
self.assertIn(b"Do not edit this file by hand.", first)
|
|
||||||
self.assertEqual(1, first.count(b"## CLI commands"))
|
|
||||||
self.assertEqual(1, first.count(b"## MCP tools"))
|
|
||||||
|
|
||||||
def test_registration_collection_is_complete(self) -> None:
|
|
||||||
cli_rows, mcp_rows = asyncio.run(
|
|
||||||
collect_command_reference_rows(
|
|
||||||
ALPHA,
|
|
||||||
proposal_writer="alpha-editor",
|
|
||||||
canonical_applier="alpha-editor",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual(EXPECTED_CLI_ROWS, len(cli_rows))
|
|
||||||
self.assertEqual(EXPECTED_MCP_ROWS, len(mcp_rows))
|
|
||||||
self.assertEqual(len(cli_rows), len({item.name for item in cli_rows}))
|
|
||||||
self.assertEqual(
|
|
||||||
set((*ALL_TOOLS, *APPLICATION_TOOLS)),
|
|
||||||
{item.name for item in mcp_rows},
|
|
||||||
)
|
|
||||||
self.assertEqual(
|
|
||||||
{"read", "proposal", "application"},
|
|
||||||
{item.surface for item in mcp_rows},
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_atomic_write_check_and_drift_detection(self) -> None:
|
|
||||||
content = generate_command_reference_bytes(ALPHA)
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
repository = Path(directory).resolve()
|
|
||||||
(repository / "generated").mkdir()
|
|
||||||
output = Path("generated/command-reference.md")
|
|
||||||
target = repository / output
|
|
||||||
|
|
||||||
self.assertEqual(
|
|
||||||
"written",
|
|
||||||
publish_or_check_command_reference(
|
|
||||||
content,
|
|
||||||
repository_root=repository,
|
|
||||||
project_root=ALPHA,
|
|
||||||
output=output,
|
|
||||||
check=False,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
self.assertEqual(content, target.read_bytes())
|
|
||||||
self.assertEqual(
|
|
||||||
"current",
|
|
||||||
publish_or_check_command_reference(
|
|
||||||
content,
|
|
||||||
repository_root=repository,
|
|
||||||
project_root=ALPHA,
|
|
||||||
output=output,
|
|
||||||
check=True,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
target.write_text("stale\n", encoding="utf-8")
|
|
||||||
with self.assertRaises(CommandReferenceDrift):
|
|
||||||
publish_or_check_command_reference(
|
|
||||||
content,
|
|
||||||
repository_root=repository,
|
|
||||||
project_root=ALPHA,
|
|
||||||
output=output,
|
|
||||||
check=True,
|
|
||||||
)
|
|
||||||
self.assertEqual(b"stale\n", target.read_bytes())
|
|
||||||
self.assertEqual(
|
|
||||||
"written",
|
|
||||||
publish_or_check_command_reference(
|
|
||||||
content,
|
|
||||||
repository_root=repository,
|
|
||||||
project_root=ALPHA,
|
|
||||||
output=output,
|
|
||||||
check=False,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
self.assertEqual(content, target.read_bytes())
|
|
||||||
self.assertEqual(
|
|
||||||
"unchanged",
|
|
||||||
publish_or_check_command_reference(
|
|
||||||
content,
|
|
||||||
repository_root=repository,
|
|
||||||
project_root=ALPHA,
|
|
||||||
output=output,
|
|
||||||
check=False,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_output_confinement_rejects_escapes_and_symlinks(self) -> None:
|
|
||||||
content = b"reference\n"
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
repository = Path(directory).resolve()
|
|
||||||
generated = repository / "generated"
|
|
||||||
generated.mkdir()
|
|
||||||
outside = repository.parent / f"{repository.name}-outside.md"
|
|
||||||
symlink = generated / "reference.md"
|
|
||||||
symlink.symlink_to(outside)
|
|
||||||
|
|
||||||
for output in (Path("../escape.md"), Path("generated/reference.md")):
|
|
||||||
with (
|
|
||||||
self.subTest(output=output),
|
|
||||||
self.assertRaises(CommandReferenceToolError),
|
|
||||||
):
|
|
||||||
publish_or_check_command_reference(
|
|
||||||
content,
|
|
||||||
repository_root=repository,
|
|
||||||
project_root=ALPHA,
|
|
||||||
output=output,
|
|
||||||
check=False,
|
|
||||||
)
|
|
||||||
symlink.unlink()
|
|
||||||
(repository / "linked").symlink_to(generated, target_is_directory=True)
|
|
||||||
with self.assertRaises(CommandReferenceToolError):
|
|
||||||
publish_or_check_command_reference(
|
|
||||||
content,
|
|
||||||
repository_root=repository,
|
|
||||||
project_root=ALPHA,
|
|
||||||
output=Path("linked/reference.md"),
|
|
||||||
check=False,
|
|
||||||
)
|
|
||||||
|
|
||||||
with self.assertRaises(CommandReferenceToolError):
|
|
||||||
publish_or_check_command_reference(
|
|
||||||
content,
|
|
||||||
repository_root=ROOT,
|
|
||||||
project_root=ALPHA,
|
|
||||||
output=Path("tests/fixtures/alpha/generated-reference.md"),
|
|
||||||
check=False,
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_existing_target_race_is_restored_without_overwrite(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
repository = Path(directory).resolve()
|
|
||||||
generated = repository / "generated"
|
|
||||||
generated.mkdir()
|
|
||||||
output = Path("generated/reference.md")
|
|
||||||
target = repository / output
|
|
||||||
target.write_bytes(b"initial\n")
|
|
||||||
original_exchange = command_reference_tool._rename_exchange
|
|
||||||
injected = False
|
|
||||||
|
|
||||||
def exchange_after_race(directory_fd: int, first: str, second: str) -> None:
|
|
||||||
nonlocal injected
|
|
||||||
if not injected:
|
|
||||||
injected = True
|
|
||||||
target.write_bytes(b"concurrent\n")
|
|
||||||
original_exchange(directory_fd, first, second)
|
|
||||||
|
|
||||||
with (
|
|
||||||
mock.patch.object(
|
|
||||||
command_reference_tool,
|
|
||||||
"_rename_exchange",
|
|
||||||
side_effect=exchange_after_race,
|
|
||||||
),
|
|
||||||
self.assertRaisesRegex(
|
|
||||||
CommandReferenceToolError,
|
|
||||||
"changed during atomic publication",
|
|
||||||
),
|
|
||||||
):
|
|
||||||
publish_or_check_command_reference(
|
|
||||||
b"generated\n",
|
|
||||||
repository_root=repository,
|
|
||||||
project_root=ALPHA,
|
|
||||||
output=output,
|
|
||||||
check=False,
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual(b"concurrent\n", target.read_bytes())
|
|
||||||
self.assertEqual(
|
|
||||||
[],
|
|
||||||
list(generated.glob(".reference.md.docforge-command-reference-*")),
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_missing_target_race_is_never_clobbered(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
repository = Path(directory).resolve()
|
|
||||||
generated = repository / "generated"
|
|
||||||
generated.mkdir()
|
|
||||||
output = Path("generated/reference.md")
|
|
||||||
target = repository / output
|
|
||||||
original_link = command_reference_tool._link_no_replace
|
|
||||||
injected = False
|
|
||||||
|
|
||||||
def link_after_race(directory_fd: int, source: str, destination: str) -> None:
|
|
||||||
nonlocal injected
|
|
||||||
if not injected:
|
|
||||||
injected = True
|
|
||||||
target.write_bytes(b"concurrent\n")
|
|
||||||
original_link(directory_fd, source, destination)
|
|
||||||
|
|
||||||
with (
|
|
||||||
mock.patch.object(
|
|
||||||
command_reference_tool,
|
|
||||||
"_link_no_replace",
|
|
||||||
side_effect=link_after_race,
|
|
||||||
),
|
|
||||||
self.assertRaisesRegex(CommandReferenceToolError, "appeared"),
|
|
||||||
):
|
|
||||||
publish_or_check_command_reference(
|
|
||||||
b"generated\n",
|
|
||||||
repository_root=repository,
|
|
||||||
project_root=ALPHA,
|
|
||||||
output=output,
|
|
||||||
check=False,
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual(b"concurrent\n", target.read_bytes())
|
|
||||||
self.assertEqual(
|
|
||||||
[],
|
|
||||||
list(generated.glob(".reference.md.docforge-command-reference-*")),
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_concurrent_generators_are_serialized_around_inspection_and_write(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
repository = Path(directory).resolve()
|
|
||||||
(repository / "generated").mkdir()
|
|
||||||
output = Path("generated/reference.md")
|
|
||||||
target = repository / output
|
|
||||||
target.write_bytes(b"initial\n")
|
|
||||||
first_entered = threading.Event()
|
|
||||||
release_first = threading.Event()
|
|
||||||
second_started = threading.Event()
|
|
||||||
second_entered = threading.Event()
|
|
||||||
results: list[str] = []
|
|
||||||
errors: list[BaseException] = []
|
|
||||||
original_replace = command_reference_tool._atomic_replace
|
|
||||||
invocation_count = 0
|
|
||||||
invocation_lock = threading.Lock()
|
|
||||||
|
|
||||||
def blocking_replace(
|
|
||||||
directory_fd: int,
|
|
||||||
name: str,
|
|
||||||
content: bytes,
|
|
||||||
*,
|
|
||||||
expected: object,
|
|
||||||
expected_content: bytes | None,
|
|
||||||
) -> None:
|
|
||||||
nonlocal invocation_count
|
|
||||||
with invocation_lock:
|
|
||||||
invocation_count += 1
|
|
||||||
invocation = invocation_count
|
|
||||||
if invocation == 1:
|
|
||||||
first_entered.set()
|
|
||||||
if not release_first.wait(timeout=5):
|
|
||||||
raise AssertionError("Timed out waiting to release first generator")
|
|
||||||
else:
|
|
||||||
second_entered.set()
|
|
||||||
original_replace(
|
|
||||||
directory_fd,
|
|
||||||
name,
|
|
||||||
content,
|
|
||||||
expected=expected, # type: ignore[arg-type]
|
|
||||||
expected_content=expected_content,
|
|
||||||
)
|
|
||||||
|
|
||||||
def publish(content: bytes, *, started: threading.Event | None = None) -> None:
|
|
||||||
if started is not None:
|
|
||||||
started.set()
|
|
||||||
try:
|
|
||||||
results.append(
|
|
||||||
publish_or_check_command_reference(
|
|
||||||
content,
|
|
||||||
repository_root=repository,
|
|
||||||
project_root=ALPHA,
|
|
||||||
output=output,
|
|
||||||
check=False,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
except BaseException as error:
|
|
||||||
errors.append(error)
|
|
||||||
|
|
||||||
with mock.patch.object(
|
|
||||||
command_reference_tool,
|
|
||||||
"_atomic_replace",
|
|
||||||
side_effect=blocking_replace,
|
|
||||||
):
|
|
||||||
first = threading.Thread(target=publish, args=(b"first\n",))
|
|
||||||
second = threading.Thread(
|
|
||||||
target=publish,
|
|
||||||
args=(b"second\n",),
|
|
||||||
kwargs={"started": second_started},
|
|
||||||
)
|
|
||||||
first.start()
|
|
||||||
self.assertTrue(first_entered.wait(timeout=5))
|
|
||||||
second.start()
|
|
||||||
self.assertTrue(second_started.wait(timeout=5))
|
|
||||||
self.assertFalse(second_entered.wait(timeout=0.1))
|
|
||||||
release_first.set()
|
|
||||||
first.join(timeout=5)
|
|
||||||
second.join(timeout=5)
|
|
||||||
|
|
||||||
self.assertFalse(first.is_alive())
|
|
||||||
self.assertFalse(second.is_alive())
|
|
||||||
self.assertEqual([], errors)
|
|
||||||
self.assertEqual(["written", "written"], results)
|
|
||||||
self.assertEqual(b"second\n", target.read_bytes())
|
|
||||||
|
|
||||||
def test_command_runs_with_explicit_repository_and_project_roots(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
repository = Path(directory).resolve()
|
|
||||||
(repository / "generated").mkdir()
|
|
||||||
output = Path("generated/reference.md")
|
|
||||||
stdout = io.StringIO()
|
|
||||||
|
|
||||||
with contextlib.redirect_stdout(stdout):
|
|
||||||
status = main(
|
|
||||||
(
|
|
||||||
"--repository-root",
|
|
||||||
str(repository),
|
|
||||||
"--project-root",
|
|
||||||
str(ALPHA),
|
|
||||||
"--output",
|
|
||||||
str(output),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual(0, status)
|
|
||||||
self.assertTrue((repository / output).is_file())
|
|
||||||
self.assertIn('"cli_rows":28', stdout.getvalue())
|
|
||||||
self.assertIn('"mcp_rows":36', stdout.getvalue())
|
|
||||||
|
|
||||||
target = repository / output
|
|
||||||
target.write_text("drift\n", encoding="utf-8")
|
|
||||||
stderr = io.StringIO()
|
|
||||||
with contextlib.redirect_stderr(stderr):
|
|
||||||
check_status = main(
|
|
||||||
(
|
|
||||||
"--repository-root",
|
|
||||||
str(repository),
|
|
||||||
"--project-root",
|
|
||||||
str(ALPHA),
|
|
||||||
"--output",
|
|
||||||
str(output),
|
|
||||||
"--check",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual(1, check_status)
|
|
||||||
self.assertEqual("drift\n", target.read_text(encoding="utf-8"))
|
|
||||||
self.assertIn("missing or stale", stderr.getvalue())
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
unittest.main()
|
|
||||||
|
|
@ -1,486 +0,0 @@
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
|
||||||
import tempfile
|
|
||||||
import unittest
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import cast
|
|
||||||
from unittest.mock import patch
|
|
||||||
|
|
||||||
from docforge.adapter_sdk import (
|
|
||||||
AdapterProject,
|
|
||||||
AdapterSource,
|
|
||||||
AdapterSourceProjection,
|
|
||||||
verify_adapter_conformance,
|
|
||||||
)
|
|
||||||
from docforge.adapters.cpp import (
|
|
||||||
CPP_ADAPTER_ID,
|
|
||||||
CPP_ADAPTER_VERSION,
|
|
||||||
CPP_EXTRACTOR_VERSION,
|
|
||||||
CppReferenceAdapter,
|
|
||||||
)
|
|
||||||
from docforge.errors import DocForgeError
|
|
||||||
from docforge.index import ProjectIndex
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
FIXTURE = ROOT / "tests" / "fixtures" / "reference-cpp"
|
|
||||||
|
|
||||||
|
|
||||||
class RecordingCppAdapter(CppReferenceAdapter):
|
|
||||||
def __init__(self, root: Path) -> None:
|
|
||||||
super().__init__(
|
|
||||||
root,
|
|
||||||
source_roots=("include", "src"),
|
|
||||||
compilation_database="compile_commands.json",
|
|
||||||
project_id="reference-cpp",
|
|
||||||
title="Reference C++ fixture",
|
|
||||||
)
|
|
||||||
self.extracted_paths: list[str] = []
|
|
||||||
|
|
||||||
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection:
|
|
||||||
self.extracted_paths.append(source.source_path)
|
|
||||||
return super().extract_source(source)
|
|
||||||
|
|
||||||
|
|
||||||
class CppReferenceAdapterTests(unittest.TestCase):
|
|
||||||
def copy_fixture(self, parent: Path) -> Path:
|
|
||||||
root = parent / "reference-cpp"
|
|
||||||
shutil.copytree(FIXTURE, root)
|
|
||||||
return root.resolve()
|
|
||||||
|
|
||||||
def adapter(self, root: Path) -> RecordingCppAdapter:
|
|
||||||
return RecordingCppAdapter(root)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def build_metrics(result: dict[str, object]) -> dict[str, object]:
|
|
||||||
return cast(dict[str, object], result["build"])
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def read_database(root: Path) -> list[dict[str, object]]:
|
|
||||||
return cast(
|
|
||||||
list[dict[str, object]],
|
|
||||||
json.loads((root / "compile_commands.json").read_text(encoding="utf-8")),
|
|
||||||
)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def write_database(root: Path, entries: list[dict[str, object]]) -> None:
|
|
||||||
(root / "compile_commands.json").write_text(
|
|
||||||
json.dumps(entries, indent=2, sort_keys=True) + "\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_complete_assembly_is_deterministic_conformant_and_syntax_scoped(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
adapter = self.adapter(root)
|
|
||||||
|
|
||||||
first = adapter.load_assembly()
|
|
||||||
second = adapter.load_complete_assembly()
|
|
||||||
projection = adapter.load_projection()
|
|
||||||
|
|
||||||
self.assertEqual(first, second)
|
|
||||||
self.assertEqual(first.projection, projection)
|
|
||||||
self.assertEqual(CPP_ADAPTER_ID, projection.adapter_id)
|
|
||||||
self.assertEqual(CPP_ADAPTER_VERSION, projection.adapter_version)
|
|
||||||
self.assertEqual(17, len(projection.nodes))
|
|
||||||
self.assertEqual(16, len(projection.edges))
|
|
||||||
self.assertEqual(5, len(first.logic))
|
|
||||||
self.assertEqual(
|
|
||||||
list(projection.nodes),
|
|
||||||
sorted(projection.nodes, key=lambda item: item.node.node_id),
|
|
||||||
)
|
|
||||||
self.assertEqual(
|
|
||||||
{"contains", "depends_on"},
|
|
||||||
{item.edge.relation for item in projection.edges},
|
|
||||||
)
|
|
||||||
self.assertTrue(
|
|
||||||
any(
|
|
||||||
item.node.title == "demo.Worker" and ("kind", "class") in item.metadata
|
|
||||||
for item in projection.nodes
|
|
||||||
)
|
|
||||||
)
|
|
||||||
self.assertTrue(
|
|
||||||
any(
|
|
||||||
item.node.title == "demo.Config" and ("kind", "struct") in item.metadata
|
|
||||||
for item in projection.nodes
|
|
||||||
)
|
|
||||||
)
|
|
||||||
self.assertTrue(
|
|
||||||
any(
|
|
||||||
item.node.title == "demo.Worker.run" and ("kind", "function") in item.metadata
|
|
||||||
for item in projection.nodes
|
|
||||||
)
|
|
||||||
)
|
|
||||||
self.assertEqual(
|
|
||||||
{"include/config.hpp", "include/detail.hpp", "src/main.cpp", "src/worker.cpp"},
|
|
||||||
{item.node.source_path for item in projection.nodes},
|
|
||||||
)
|
|
||||||
self.assertFalse(
|
|
||||||
{"calls", "inherits", "type_of"} & {item.edge.relation for item in projection.edges}
|
|
||||||
)
|
|
||||||
|
|
||||||
report = adapter.support_report()
|
|
||||||
self.assertEqual(1, report["schema_version"])
|
|
||||||
self.assertEqual(CPP_EXTRACTOR_VERSION, report["extractor_version"])
|
|
||||||
self.assertEqual(["include", "src"], report["source_roots"])
|
|
||||||
self.assertTrue(report["compilation_database_is_authoritative"])
|
|
||||||
self.assertFalse(report["executes_compiler"])
|
|
||||||
self.assertFalse(report["executes_project_code"])
|
|
||||||
self.assertEqual(
|
|
||||||
[
|
|
||||||
"call_resolution",
|
|
||||||
"type_resolution",
|
|
||||||
"inheritance_resolution",
|
|
||||||
"overload_ownership",
|
|
||||||
"macro_semantics",
|
|
||||||
"compiler_command_execution",
|
|
||||||
"nonlocal_include_resolution",
|
|
||||||
],
|
|
||||||
[item.code for item in adapter.unsupported_facts()],
|
|
||||||
)
|
|
||||||
self.assertFalse((root / ".cache").exists())
|
|
||||||
|
|
||||||
conformance = verify_adapter_conformance(
|
|
||||||
adapter,
|
|
||||||
cache_root=root / ".cache" / "conformance",
|
|
||||||
)
|
|
||||||
self.assertEqual("reference-cpp", conformance.project_id)
|
|
||||||
self.assertEqual(17, conformance.node_count)
|
|
||||||
self.assertEqual(16, conformance.edge_count)
|
|
||||||
self.assertEqual(5, conformance.logic_projection_count)
|
|
||||||
self.assertTrue(conformance.incremental)
|
|
||||||
|
|
||||||
def test_incremental_cold_warm_source_header_and_flag_invalidation(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
adapter = self.adapter(root)
|
|
||||||
project = AdapterProject(adapter, cache_root=root / ".cache" / "incremental")
|
|
||||||
index = ProjectIndex(project)
|
|
||||||
|
|
||||||
cold = index.build()
|
|
||||||
self.assertEqual(4, self.build_metrics(cold)["reparsed_sources"])
|
|
||||||
self.assertEqual(
|
|
||||||
[
|
|
||||||
"include/config.hpp",
|
|
||||||
"include/detail.hpp",
|
|
||||||
"src/main.cpp",
|
|
||||||
"src/worker.cpp",
|
|
||||||
],
|
|
||||||
sorted(adapter.extracted_paths),
|
|
||||||
)
|
|
||||||
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
warm = index.build()
|
|
||||||
self.assertEqual(4, self.build_metrics(warm)["cache_hits"])
|
|
||||||
self.assertEqual(0, self.build_metrics(warm)["reparsed_sources"])
|
|
||||||
self.assertEqual([], adapter.extracted_paths)
|
|
||||||
|
|
||||||
main = root / "src" / "main.cpp"
|
|
||||||
main.write_text(
|
|
||||||
main.read_text(encoding="utf-8").replace(
|
|
||||||
"return demo::execute(3);",
|
|
||||||
"return demo::execute(4);",
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
source_changed = index.build()
|
|
||||||
self.assertEqual(1, self.build_metrics(source_changed)["invalidated_sources"])
|
|
||||||
self.assertEqual(["src/main.cpp"], adapter.extracted_paths)
|
|
||||||
|
|
||||||
detail = root / "include" / "detail.hpp"
|
|
||||||
detail.write_text(
|
|
||||||
detail.read_text(encoding="utf-8").replace(
|
|
||||||
"return value;",
|
|
||||||
"return value + 0;",
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
header_changed = index.build()
|
|
||||||
self.assertEqual(4, self.build_metrics(header_changed)["invalidated_sources"])
|
|
||||||
self.assertEqual(
|
|
||||||
[
|
|
||||||
"include/config.hpp",
|
|
||||||
"include/detail.hpp",
|
|
||||||
"src/main.cpp",
|
|
||||||
"src/worker.cpp",
|
|
||||||
],
|
|
||||||
sorted(adapter.extracted_paths),
|
|
||||||
)
|
|
||||||
|
|
||||||
entries = self.read_database(root)
|
|
||||||
arguments = cast(list[str], entries[0]["arguments"])
|
|
||||||
arguments.append("-DREFERENCE_FLAG=1")
|
|
||||||
self.write_database(root, entries)
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
flag_changed = index.build()
|
|
||||||
self.assertEqual(1, self.build_metrics(flag_changed)["invalidated_sources"])
|
|
||||||
self.assertEqual(["src/main.cpp"], adapter.extracted_paths)
|
|
||||||
|
|
||||||
equivalent = project.verify_incremental_equivalence()
|
|
||||||
self.assertEqual("ok", equivalent["status"])
|
|
||||||
self.assertEqual(17, equivalent["node_count"])
|
|
||||||
self.assertEqual(16, equivalent["edge_count"])
|
|
||||||
self.assertEqual(5, equivalent["logic_projection_count"])
|
|
||||||
|
|
||||||
def test_add_delete_and_corrupt_cache_recover_without_stale_facts(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
adapter = self.adapter(root)
|
|
||||||
cache_root = root / ".cache" / "incremental"
|
|
||||||
project = AdapterProject(adapter, cache_root=cache_root)
|
|
||||||
index = ProjectIndex(project)
|
|
||||||
index.build()
|
|
||||||
|
|
||||||
extra = root / "src" / "extra.cpp"
|
|
||||||
extra.write_text(
|
|
||||||
'#include "../include/config.hpp"\n\n'
|
|
||||||
"int extra(int value) {\n"
|
|
||||||
" return demo::normalize(value);\n"
|
|
||||||
"}\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
entries = self.read_database(root)
|
|
||||||
entries.append(
|
|
||||||
{
|
|
||||||
"arguments": ["c++", "-std=c++20", "-c", "src/extra.cpp"],
|
|
||||||
"directory": ".",
|
|
||||||
"file": "src/extra.cpp",
|
|
||||||
}
|
|
||||||
)
|
|
||||||
self.write_database(root, entries)
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
added = index.build()
|
|
||||||
self.assertEqual(1, self.build_metrics(added)["reparsed_sources"])
|
|
||||||
self.assertEqual(["src/extra.cpp"], adapter.extracted_paths)
|
|
||||||
self.assertIn(
|
|
||||||
"src/extra.cpp",
|
|
||||||
{node.source_path for node in project.load().nodes},
|
|
||||||
)
|
|
||||||
|
|
||||||
worker = root / "src" / "worker.cpp"
|
|
||||||
entries = [
|
|
||||||
entry for entry in self.read_database(root) if entry["file"] != "src/worker.cpp"
|
|
||||||
]
|
|
||||||
self.write_database(root, entries)
|
|
||||||
worker.unlink()
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
deleted = index.build()
|
|
||||||
self.assertEqual(1, self.build_metrics(deleted)["deleted_sources"])
|
|
||||||
self.assertNotIn(
|
|
||||||
"src/worker.cpp",
|
|
||||||
{node.source_path for node in project.load().nodes},
|
|
||||||
)
|
|
||||||
|
|
||||||
(cache_root / "extractions.json").write_text("{broken", encoding="utf-8")
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
recovered = index.build()
|
|
||||||
self.assertEqual(4, self.build_metrics(recovered)["reparsed_sources"])
|
|
||||||
self.assertEqual(
|
|
||||||
[
|
|
||||||
"include/config.hpp",
|
|
||||||
"include/detail.hpp",
|
|
||||||
"src/extra.cpp",
|
|
||||||
"src/main.cpp",
|
|
||||||
],
|
|
||||||
sorted(adapter.extracted_paths),
|
|
||||||
)
|
|
||||||
self.assertEqual("ok", project.verify_incremental_equivalence()["status"])
|
|
||||||
|
|
||||||
def test_database_shape_bounds_duplicates_and_paths_fail_closed(self) -> None:
|
|
||||||
invalid_documents: list[tuple[str, list[dict[str, object]], str]] = []
|
|
||||||
baseline = cast(
|
|
||||||
list[dict[str, object]],
|
|
||||||
json.loads((FIXTURE / "compile_commands.json").read_text(encoding="utf-8")),
|
|
||||||
)
|
|
||||||
|
|
||||||
both = json.loads(json.dumps(baseline))
|
|
||||||
both[0]["command"] = "c++ -c src/main.cpp"
|
|
||||||
invalid_documents.append(("both command forms", both, "invalid_compilation_database"))
|
|
||||||
|
|
||||||
neither = json.loads(json.dumps(baseline))
|
|
||||||
del neither[1]["command"]
|
|
||||||
invalid_documents.append(("neither command form", neither, "invalid_compilation_database"))
|
|
||||||
|
|
||||||
bad_argument = json.loads(json.dumps(baseline))
|
|
||||||
bad_argument[0]["arguments"] = ["c++", 7]
|
|
||||||
invalid_documents.append(
|
|
||||||
("non-string argument", bad_argument, "invalid_compilation_database")
|
|
||||||
)
|
|
||||||
|
|
||||||
control_command = json.loads(json.dumps(baseline))
|
|
||||||
control_command[1]["command"] = "c++ -c src/worker.cpp\nmalicious"
|
|
||||||
invalid_documents.append(
|
|
||||||
("control command", control_command, "invalid_compilation_database")
|
|
||||||
)
|
|
||||||
|
|
||||||
unknown_key = json.loads(json.dumps(baseline))
|
|
||||||
unknown_key[0]["environment"] = {"PATH": "/tmp"}
|
|
||||||
invalid_documents.append(("unknown key", unknown_key, "invalid_compilation_database"))
|
|
||||||
|
|
||||||
duplicate = json.loads(json.dumps(baseline))
|
|
||||||
duplicate.append(json.loads(json.dumps(duplicate[0])))
|
|
||||||
invalid_documents.append(("duplicate TU", duplicate, "duplicate_translation_unit"))
|
|
||||||
|
|
||||||
escaped_directory = json.loads(json.dumps(baseline))
|
|
||||||
escaped_directory[0]["directory"] = ".."
|
|
||||||
invalid_documents.append(("escaped directory", escaped_directory, "path_escape"))
|
|
||||||
|
|
||||||
escaped_file = json.loads(json.dumps(baseline))
|
|
||||||
escaped_file[0]["file"] = "../outside.cpp"
|
|
||||||
invalid_documents.append(("escaped file", escaped_file, "path_escape"))
|
|
||||||
|
|
||||||
for name, document, code in invalid_documents:
|
|
||||||
with self.subTest(name=name), tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
self.write_database(root, document)
|
|
||||||
adapter = self.adapter(root)
|
|
||||||
with self.assertRaises(DocForgeError) as captured:
|
|
||||||
adapter.load_manifest()
|
|
||||||
self.assertEqual(code, captured.exception.code)
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
parent = Path(directory)
|
|
||||||
root = self.copy_fixture(parent)
|
|
||||||
outside_database = parent / "compile_commands.json"
|
|
||||||
outside_database.write_text("[]\n", encoding="utf-8")
|
|
||||||
database = root / "compile_commands.json"
|
|
||||||
database.unlink()
|
|
||||||
database.symlink_to(outside_database)
|
|
||||||
with self.assertRaises(DocForgeError) as database_link:
|
|
||||||
self.adapter(root)
|
|
||||||
self.assertEqual("path_escape", database_link.exception.code)
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
parent = Path(directory)
|
|
||||||
root = self.copy_fixture(parent)
|
|
||||||
outside_source = parent / "worker.cpp"
|
|
||||||
outside_source.write_text("int outside() { return 1; }\n", encoding="utf-8")
|
|
||||||
worker = root / "src" / "worker.cpp"
|
|
||||||
worker.unlink()
|
|
||||||
worker.symlink_to(outside_source)
|
|
||||||
adapter = self.adapter(root)
|
|
||||||
with self.assertRaises(DocForgeError) as source_link:
|
|
||||||
adapter.load_manifest()
|
|
||||||
self.assertEqual("path_escape", source_link.exception.code)
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
bounded_adapters = (
|
|
||||||
(
|
|
||||||
CppReferenceAdapter(
|
|
||||||
root,
|
|
||||||
source_roots=("include", "src"),
|
|
||||||
compilation_database="compile_commands.json",
|
|
||||||
max_database_bytes=1,
|
|
||||||
),
|
|
||||||
"source_too_large",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
CppReferenceAdapter(
|
|
||||||
root,
|
|
||||||
source_roots=("include", "src"),
|
|
||||||
compilation_database="compile_commands.json",
|
|
||||||
max_source_bytes=1,
|
|
||||||
),
|
|
||||||
"source_too_large",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
CppReferenceAdapter(
|
|
||||||
root,
|
|
||||||
source_roots=("include", "src"),
|
|
||||||
compilation_database="compile_commands.json",
|
|
||||||
max_command_chars=32,
|
|
||||||
),
|
|
||||||
"invalid_compilation_database",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
CppReferenceAdapter(
|
|
||||||
root,
|
|
||||||
source_roots=("include", "src"),
|
|
||||||
compilation_database="compile_commands.json",
|
|
||||||
max_arguments=2,
|
|
||||||
),
|
|
||||||
"invalid_compilation_database",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
CppReferenceAdapter(
|
|
||||||
root,
|
|
||||||
source_roots=("include", "src"),
|
|
||||||
compilation_database="compile_commands.json",
|
|
||||||
max_translation_units=1,
|
|
||||||
),
|
|
||||||
"adapter_too_large",
|
|
||||||
),
|
|
||||||
(
|
|
||||||
CppReferenceAdapter(
|
|
||||||
root,
|
|
||||||
source_roots=("include", "src"),
|
|
||||||
compilation_database="compile_commands.json",
|
|
||||||
max_headers=1,
|
|
||||||
),
|
|
||||||
"adapter_too_large",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
for adapter, code in bounded_adapters:
|
|
||||||
with self.subTest(limit=code), self.assertRaises(DocForgeError) as bounded:
|
|
||||||
adapter.load_manifest()
|
|
||||||
self.assertEqual(code, bounded.exception.code)
|
|
||||||
|
|
||||||
def test_source_roots_no_ast_and_optional_extra_fail_closed(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
with self.assertRaises(DocForgeError) as translation_unit_outside:
|
|
||||||
CppReferenceAdapter(
|
|
||||||
root,
|
|
||||||
source_roots=("include",),
|
|
||||||
compilation_database="compile_commands.json",
|
|
||||||
).load_manifest()
|
|
||||||
self.assertEqual("path_escape", translation_unit_outside.exception.code)
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
other = root / "other"
|
|
||||||
other.mkdir()
|
|
||||||
(other / "outside.hpp").write_text(
|
|
||||||
"#pragma once\nstruct Outside {};\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
main = root / "src" / "main.cpp"
|
|
||||||
main.write_text(
|
|
||||||
'#include "../other/outside.hpp"\n' + main.read_text(encoding="utf-8"),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
with self.assertRaises(DocForgeError) as header_outside:
|
|
||||||
self.adapter(root).load_manifest()
|
|
||||||
self.assertEqual("path_escape", header_outside.exception.code)
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
adapter = self.adapter(root)
|
|
||||||
project = AdapterProject(adapter, cache_root=root / ".cache" / "no-ast")
|
|
||||||
with self.assertRaises(DocForgeError) as no_ast:
|
|
||||||
ProjectIndex(project, allow_logic=False).build()
|
|
||||||
self.assertEqual("adapter_policy_forbids_logic", no_ast.exception.code)
|
|
||||||
self.assertFalse(project.descriptor.index_path.exists())
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
adapter = self.adapter(root)
|
|
||||||
missing = ModuleNotFoundError(
|
|
||||||
"No module named 'tree_sitter'",
|
|
||||||
name="tree_sitter",
|
|
||||||
)
|
|
||||||
with (
|
|
||||||
patch(
|
|
||||||
"docforge.adapters.cpp.importlib.import_module",
|
|
||||||
side_effect=missing,
|
|
||||||
),
|
|
||||||
self.assertRaises(DocForgeError) as optional,
|
|
||||||
):
|
|
||||||
adapter.load_manifest()
|
|
||||||
self.assertEqual("optional_dependency_missing", optional.exception.code)
|
|
||||||
self.assertEqual("cpp", optional.exception.details["extra"])
|
|
||||||
self.assertEqual("docforge[cpp]", optional.exception.details["install"])
|
|
||||||
|
|
@ -1,198 +0,0 @@
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import shutil
|
|
||||||
import tempfile
|
|
||||||
import unittest
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from tools.check_documentation import (
|
|
||||||
COMMAND_REFERENCE_NOTICE,
|
|
||||||
MAX_DIAGNOSTICS,
|
|
||||||
DocumentationPolicy,
|
|
||||||
check_documentation,
|
|
||||||
)
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
SCHEMA = ROOT / "schemas/reference-adapter.schema.json"
|
|
||||||
|
|
||||||
|
|
||||||
class DocumentationCheckTests(unittest.TestCase):
|
|
||||||
def repository(self) -> Path:
|
|
||||||
root = Path(self.enterContext(tempfile.TemporaryDirectory()))
|
|
||||||
(root / "docs").mkdir()
|
|
||||||
(root / "schemas").mkdir()
|
|
||||||
shutil.copyfile(SCHEMA, root / "schemas/reference-adapter.schema.json")
|
|
||||||
(root / "pending.txt").write_text("", encoding="utf-8")
|
|
||||||
return root
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def policy(*required: str) -> DocumentationPolicy:
|
|
||||||
return DocumentationPolicy(
|
|
||||||
required_pages=tuple(Path(item) for item in required),
|
|
||||||
pending_inventory=Path("pending.txt"),
|
|
||||||
)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def command_reference() -> str:
|
|
||||||
return f"# DocForge command reference\n\n{COMMAND_REFERENCE_NOTICE}\n\n## CLI commands\n"
|
|
||||||
|
|
||||||
def test_valid_graph_h1_anchors_and_reference_adapter_toml_pass(self) -> None:
|
|
||||||
root = self.repository()
|
|
||||||
(root / "README.md").write_text(
|
|
||||||
"# Documentation\n\n"
|
|
||||||
"[Guide](docs/GUIDE.md#adapter-setup) and "
|
|
||||||
"[commands](docs/COMMAND_REFERENCE.md). [Top](#documentation)\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
(root / "docs/GUIDE.md").write_text(
|
|
||||||
"# Guide\n\n"
|
|
||||||
"## Adapter *setup*\n\n"
|
|
||||||
"Reference adapter configuration `.docforge/reference-adapter.toml`:\n\n"
|
|
||||||
"```toml\n"
|
|
||||||
"schema_version = 1\n"
|
|
||||||
'project_id = "example-project"\n'
|
|
||||||
'title = "Example"\n'
|
|
||||||
'language = "python"\n'
|
|
||||||
'source_roots = ["src"]\n'
|
|
||||||
"```\n\n"
|
|
||||||
"[Back](../README.md#documentation)\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
(root / "docs/COMMAND_REFERENCE.md").write_text(
|
|
||||||
self.command_reference(),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
|
|
||||||
report = check_documentation(
|
|
||||||
root,
|
|
||||||
policy=self.policy(
|
|
||||||
"README.md",
|
|
||||||
"docs/GUIDE.md",
|
|
||||||
"docs/COMMAND_REFERENCE.md",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertTrue(report.ok, [item.render() for item in report.diagnostics])
|
|
||||||
self.assertEqual(3, report.markdown_pages)
|
|
||||||
self.assertEqual(1, report.reference_adapter_examples)
|
|
||||||
|
|
||||||
def test_links_anchors_h1_and_generated_notice_fail_closed(self) -> None:
|
|
||||||
root = self.repository()
|
|
||||||
(root / "README.md").write_text(
|
|
||||||
"# Documentation\n\n"
|
|
||||||
"[bad anchor](docs/GUIDE.md#missing) "
|
|
||||||
"[missing file](docs/ABSENT.md) "
|
|
||||||
"[commands](docs/COMMAND_REFERENCE.md).\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
(root / "docs/GUIDE.md").write_text(
|
|
||||||
"# Guide\n\n# Duplicate\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
(root / "docs/COMMAND_REFERENCE.md").write_text(
|
|
||||||
"# DocForge command reference\n\n## CLI commands\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
|
|
||||||
report = check_documentation(
|
|
||||||
root,
|
|
||||||
policy=self.policy(
|
|
||||||
"README.md",
|
|
||||||
"docs/GUIDE.md",
|
|
||||||
"docs/COMMAND_REFERENCE.md",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertFalse(report.ok)
|
|
||||||
self.assertEqual(
|
|
||||||
{"DOC016", "DOC017", "DOC020", "DOC028"},
|
|
||||||
{item.code for item in report.diagnostics},
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_only_prior_milestone_records_are_reachability_exempt(self) -> None:
|
|
||||||
root = self.repository()
|
|
||||||
(root / "README.md").write_text("# Documentation\n", encoding="utf-8")
|
|
||||||
(root / "docs/MILESTONE_2_BASELINE.md").write_text(
|
|
||||||
"# Historical evidence\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
(root / "docs/ORPHAN.md").write_text("# Orphan\n", encoding="utf-8")
|
|
||||||
|
|
||||||
report = check_documentation(
|
|
||||||
root,
|
|
||||||
policy=self.policy("README.md"),
|
|
||||||
)
|
|
||||||
|
|
||||||
orphans = [item.path for item in report.diagnostics if item.code == "DOC023"]
|
|
||||||
self.assertEqual(["docs/ORPHAN.md"], orphans)
|
|
||||||
|
|
||||||
def test_pending_page_becomes_a_hard_failure_as_soon_as_it_exists(self) -> None:
|
|
||||||
root = self.repository()
|
|
||||||
(root / "README.md").write_text("# Documentation\n", encoding="utf-8")
|
|
||||||
(root / "pending.txt").write_text("docs/FUTURE.md\n", encoding="utf-8")
|
|
||||||
policy = self.policy("README.md", "docs/FUTURE.md")
|
|
||||||
|
|
||||||
pending = check_documentation(root, policy=policy)
|
|
||||||
|
|
||||||
self.assertTrue(pending.ok)
|
|
||||||
self.assertEqual(1, pending.pending_pages)
|
|
||||||
|
|
||||||
(root / "docs/FUTURE.md").write_text("# Future\n", encoding="utf-8")
|
|
||||||
stale = check_documentation(root, policy=policy)
|
|
||||||
|
|
||||||
self.assertIn("DOC014", {item.code for item in stale.diagnostics})
|
|
||||||
self.assertIn("DOC023", {item.code for item in stale.diagnostics})
|
|
||||||
|
|
||||||
def test_unknown_pending_page_and_invalid_toml_are_reported(self) -> None:
|
|
||||||
root = self.repository()
|
|
||||||
(root / "README.md").write_text(
|
|
||||||
"# Documentation\n\n[Adapter](docs/ADAPTER.md)\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
(root / "docs/ADAPTER.md").write_text(
|
|
||||||
"# Adapter\n\n"
|
|
||||||
"```toml reference-adapter\n"
|
|
||||||
"schema_version = 1\n"
|
|
||||||
'project_id = "cpp-example"\n'
|
|
||||||
'title = "C++"\n'
|
|
||||||
'language = "cpp"\n'
|
|
||||||
'source_roots = ["src"]\n'
|
|
||||||
"```\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
(root / "pending.txt").write_text("docs/UNKNOWN.md\n", encoding="utf-8")
|
|
||||||
|
|
||||||
report = check_documentation(
|
|
||||||
root,
|
|
||||||
policy=self.policy("README.md", "docs/ADAPTER.md"),
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertIn("DOC012", {item.code for item in report.diagnostics})
|
|
||||||
schema_failures = [item for item in report.diagnostics if item.code == "DOC026"]
|
|
||||||
self.assertEqual(1, len(schema_failures))
|
|
||||||
self.assertIn("compilation_database", schema_failures[0].message)
|
|
||||||
|
|
||||||
def test_diagnostics_are_deterministically_bounded(self) -> None:
|
|
||||||
root = self.repository()
|
|
||||||
(root / "README.md").write_text("# Documentation\n", encoding="utf-8")
|
|
||||||
for index in range(MAX_DIAGNOSTICS + 7):
|
|
||||||
(root / "docs" / f"ORPHAN_{index:03d}.md").write_text(
|
|
||||||
f"# Orphan {index}\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
|
|
||||||
report = check_documentation(
|
|
||||||
root,
|
|
||||||
policy=self.policy("README.md"),
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual(MAX_DIAGNOSTICS, len(report.diagnostics))
|
|
||||||
self.assertEqual(7, report.omitted_diagnostics)
|
|
||||||
self.assertEqual(
|
|
||||||
sorted(report.diagnostics),
|
|
||||||
list(report.diagnostics),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
unittest.main()
|
|
||||||
|
|
@ -1,176 +0,0 @@
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import tempfile
|
|
||||||
import unittest
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from tools.check_documentation import (
|
|
||||||
DocumentationPolicy,
|
|
||||||
check_documentation,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class DocumentationCheckTests(unittest.TestCase):
|
|
||||||
def repository(self, parent: Path) -> Path:
|
|
||||||
root = parent / "repository"
|
|
||||||
(root / "docs").mkdir(parents=True)
|
|
||||||
(root / "schemas").mkdir()
|
|
||||||
(root / "schemas" / "reference-adapter.schema.json").write_text(
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
||||||
"type": "object",
|
|
||||||
"additionalProperties": False,
|
|
||||||
"required": [
|
|
||||||
"schema_version",
|
|
||||||
"project_id",
|
|
||||||
"title",
|
|
||||||
"language",
|
|
||||||
"source_roots",
|
|
||||||
],
|
|
||||||
"properties": {
|
|
||||||
"schema_version": {"const": 1},
|
|
||||||
"project_id": {"type": "string"},
|
|
||||||
"title": {"type": "string"},
|
|
||||||
"language": {"enum": ["python", "javascript", "typescript", "cpp"]},
|
|
||||||
"source_roots": {
|
|
||||||
"type": "array",
|
|
||||||
"minItems": 1,
|
|
||||||
"items": {"type": "string"},
|
|
||||||
},
|
|
||||||
"compilation_database": {"type": "string"},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
return root
|
|
||||||
|
|
||||||
def test_linked_pages_h1_anchors_and_reference_example_pass(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.repository(Path(directory))
|
|
||||||
(root / "README.md").write_text(
|
|
||||||
"# Project\n\n[Guide](docs/GUIDE.md#reference-adapter-configuration)\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
(root / "docs" / "GUIDE.md").write_text(
|
|
||||||
"\n".join(
|
|
||||||
(
|
|
||||||
"# Guide",
|
|
||||||
"",
|
|
||||||
"## Reference adapter configuration",
|
|
||||||
"",
|
|
||||||
"`.docforge/reference-adapter.toml`:",
|
|
||||||
"",
|
|
||||||
"```toml",
|
|
||||||
"schema_version = 1",
|
|
||||||
'project_id = "fixture"',
|
|
||||||
'title = "Fixture"',
|
|
||||||
'language = "python"',
|
|
||||||
'source_roots = ["src"]',
|
|
||||||
"```",
|
|
||||||
"",
|
|
||||||
)
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
|
|
||||||
report = check_documentation(
|
|
||||||
root,
|
|
||||||
policy=DocumentationPolicy(
|
|
||||||
required_pages=(Path("README.md"), Path("docs/GUIDE.md")),
|
|
||||||
pending_inventory=None,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertTrue(report.ok, [item.render() for item in report.diagnostics])
|
|
||||||
self.assertEqual(2, report.markdown_pages)
|
|
||||||
self.assertEqual(1, report.reference_adapter_examples)
|
|
||||||
|
|
||||||
def test_missing_anchor_orphan_and_duplicate_h1_fail_deterministically(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.repository(Path(directory))
|
|
||||||
(root / "README.md").write_text(
|
|
||||||
"# Project\n\n[Broken](docs/GUIDE.md#missing)\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
(root / "docs" / "GUIDE.md").write_text(
|
|
||||||
"# Guide\n\n# Duplicate\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
(root / "docs" / "ORPHAN.md").write_text("# Orphan\n", encoding="utf-8")
|
|
||||||
|
|
||||||
report = check_documentation(
|
|
||||||
root,
|
|
||||||
policy=DocumentationPolicy(
|
|
||||||
required_pages=(Path("README.md"), Path("docs/GUIDE.md")),
|
|
||||||
pending_inventory=None,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertFalse(report.ok)
|
|
||||||
codes = [item.code for item in report.diagnostics]
|
|
||||||
self.assertEqual(["DOC016", "DOC017", "DOC023"], sorted(codes))
|
|
||||||
|
|
||||||
def test_pending_inventory_allows_absence_but_rejects_existing_page(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.repository(Path(directory))
|
|
||||||
(root / "README.md").write_text("# Project\n", encoding="utf-8")
|
|
||||||
pending_path = root / "pending.txt"
|
|
||||||
pending_path.write_text("docs/FUTURE.md\n", encoding="utf-8")
|
|
||||||
policy = DocumentationPolicy(
|
|
||||||
required_pages=(Path("README.md"), Path("docs/FUTURE.md")),
|
|
||||||
pending_inventory=Path("pending.txt"),
|
|
||||||
)
|
|
||||||
|
|
||||||
missing = check_documentation(root, policy=policy)
|
|
||||||
self.assertTrue(missing.ok)
|
|
||||||
self.assertEqual(1, missing.pending_pages)
|
|
||||||
|
|
||||||
(root / "docs" / "FUTURE.md").write_text("# Future\n", encoding="utf-8")
|
|
||||||
stale = check_documentation(root, policy=policy)
|
|
||||||
self.assertEqual(["DOC014", "DOC023"], [item.code for item in stale.diagnostics])
|
|
||||||
|
|
||||||
def test_invalid_reference_example_reports_schema_path(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.repository(Path(directory))
|
|
||||||
(root / "README.md").write_text(
|
|
||||||
"# Project\n\n[Guide](docs/GUIDE.md)\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
(root / "docs" / "GUIDE.md").write_text(
|
|
||||||
"\n".join(
|
|
||||||
(
|
|
||||||
"# Guide",
|
|
||||||
"",
|
|
||||||
"Reference adapter configuration:",
|
|
||||||
"",
|
|
||||||
"```toml",
|
|
||||||
'language = "python"',
|
|
||||||
'source_roots = ["src"]',
|
|
||||||
"```",
|
|
||||||
"",
|
|
||||||
)
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
|
|
||||||
report = check_documentation(
|
|
||||||
root,
|
|
||||||
policy=DocumentationPolicy(
|
|
||||||
required_pages=(Path("README.md"), Path("docs/GUIDE.md")),
|
|
||||||
pending_inventory=None,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
schema_errors = [item for item in report.diagnostics if item.code == "DOC026"]
|
|
||||||
self.assertEqual(3, len(schema_errors))
|
|
||||||
messages = " ".join(item.message for item in schema_errors)
|
|
||||||
self.assertIn("schema_version", messages)
|
|
||||||
self.assertIn("project_id", messages)
|
|
||||||
self.assertIn("title", messages)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
unittest.main()
|
|
||||||
|
|
@ -1,102 +0,0 @@
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import os
|
|
||||||
import tempfile
|
|
||||||
import unittest
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from docforge.errors import DocForgeError
|
|
||||||
from docforge.incremental import (
|
|
||||||
CachedSource,
|
|
||||||
ExtractionCache,
|
|
||||||
load_extraction_cache,
|
|
||||||
write_extraction_cache,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class ExtractionCacheBoundsTests(unittest.TestCase):
|
|
||||||
@staticmethod
|
|
||||||
def cache(*, payload: str = "ok", count: int = 1) -> ExtractionCache:
|
|
||||||
sources = tuple(
|
|
||||||
CachedSource(
|
|
||||||
source_id=f"source-{index}",
|
|
||||||
source_path=f"src/{index}.txt",
|
|
||||||
fingerprint=hashlib.sha256(str(index).encode()).hexdigest(),
|
|
||||||
extractor_version="fixture@1",
|
|
||||||
dependencies=(),
|
|
||||||
payload={"content": payload},
|
|
||||||
)
|
|
||||||
for index in range(count)
|
|
||||||
)
|
|
||||||
return ExtractionCache("fixture", "fixture-adapter", "1", sources)
|
|
||||||
|
|
||||||
def test_bounded_round_trip_and_oversized_read_is_a_cache_miss(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
path = Path(directory) / "extractions.json"
|
|
||||||
write_extraction_cache(path, self.cache(), max_bytes=1_000, max_sources=2)
|
|
||||||
loaded = load_extraction_cache(
|
|
||||||
path,
|
|
||||||
project_id="fixture",
|
|
||||||
adapter_id="fixture-adapter",
|
|
||||||
adapter_version="1",
|
|
||||||
max_bytes=1_000,
|
|
||||||
max_sources=2,
|
|
||||||
)
|
|
||||||
self.assertEqual(self.cache(), loaded)
|
|
||||||
|
|
||||||
with path.open("ab") as handle:
|
|
||||||
handle.write(b" " * 1_000)
|
|
||||||
self.assertIsNone(
|
|
||||||
load_extraction_cache(
|
|
||||||
path,
|
|
||||||
project_id="fixture",
|
|
||||||
adapter_id="fixture-adapter",
|
|
||||||
adapter_version="1",
|
|
||||||
max_bytes=1_000,
|
|
||||||
max_sources=2,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_write_rejects_byte_and_source_limits_without_replacing_target(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
path = Path(directory) / "extractions.json"
|
|
||||||
path.write_bytes(b"preserved\n")
|
|
||||||
for cache, maximum_bytes, maximum_sources in (
|
|
||||||
(self.cache(payload="x" * 500), 100, 2),
|
|
||||||
(self.cache(count=2), 1_000, 1),
|
|
||||||
):
|
|
||||||
with self.assertRaises(DocForgeError) as captured:
|
|
||||||
write_extraction_cache(
|
|
||||||
path,
|
|
||||||
cache,
|
|
||||||
max_bytes=maximum_bytes,
|
|
||||||
max_sources=maximum_sources,
|
|
||||||
)
|
|
||||||
self.assertEqual("cache_limit", captured.exception.code)
|
|
||||||
self.assertEqual(b"preserved\n", path.read_bytes())
|
|
||||||
|
|
||||||
def test_symlink_and_non_regular_targets_are_cache_misses(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = Path(directory)
|
|
||||||
real = root / "real.json"
|
|
||||||
write_extraction_cache(real, self.cache(), max_bytes=1_000, max_sources=2)
|
|
||||||
linked = root / "linked.json"
|
|
||||||
linked.symlink_to(real)
|
|
||||||
fifo = root / "fifo"
|
|
||||||
os.mkfifo(fifo)
|
|
||||||
for path in (linked, fifo):
|
|
||||||
self.assertIsNone(
|
|
||||||
load_extraction_cache(
|
|
||||||
path,
|
|
||||||
project_id="fixture",
|
|
||||||
adapter_id="fixture-adapter",
|
|
||||||
adapter_version="1",
|
|
||||||
max_bytes=1_000,
|
|
||||||
max_sources=2,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
unittest.main()
|
|
||||||
|
|
@ -1,267 +0,0 @@
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import shutil
|
|
||||||
import tempfile
|
|
||||||
import unittest
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import cast
|
|
||||||
from unittest import mock
|
|
||||||
|
|
||||||
import docforge.adapters.javascript as javascript_adapter
|
|
||||||
from docforge.adapter_sdk import (
|
|
||||||
AdapterProject,
|
|
||||||
AdapterSource,
|
|
||||||
AdapterSourceProjection,
|
|
||||||
verify_adapter_conformance,
|
|
||||||
)
|
|
||||||
from docforge.adapters.javascript import (
|
|
||||||
JAVASCRIPT_ADAPTER_ID,
|
|
||||||
JAVASCRIPT_ADAPTER_VERSION,
|
|
||||||
JAVASCRIPT_EXTRACTOR_VERSION,
|
|
||||||
JavaScriptReferenceAdapter,
|
|
||||||
)
|
|
||||||
from docforge.errors import DocForgeError
|
|
||||||
from docforge.index import ProjectIndex
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
FIXTURES = ROOT / "tests" / "fixtures"
|
|
||||||
|
|
||||||
|
|
||||||
class RecordingScriptAdapter(JavaScriptReferenceAdapter):
|
|
||||||
def __init__(self, root: Path, project_id: str) -> None:
|
|
||||||
super().__init__(
|
|
||||||
root,
|
|
||||||
source_roots=("src",),
|
|
||||||
project_id=project_id,
|
|
||||||
title=f"{project_id} fixture",
|
|
||||||
)
|
|
||||||
self.extracted_paths: list[str] = []
|
|
||||||
|
|
||||||
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection:
|
|
||||||
self.extracted_paths.append(source.source_path)
|
|
||||||
return super().extract_source(source)
|
|
||||||
|
|
||||||
|
|
||||||
class JavaScriptReferenceAdapterTests(unittest.TestCase):
|
|
||||||
CASES = (
|
|
||||||
("reference-javascript", "javascript", 14, 14, 5, "src/app/shared.js"),
|
|
||||||
("reference-typescript", "typescript", 13, 14, 4, "src/app/types.mts"),
|
|
||||||
)
|
|
||||||
|
|
||||||
def copy_fixture(self, parent: Path, fixture: str) -> Path:
|
|
||||||
root = parent / fixture
|
|
||||||
shutil.copytree(FIXTURES / fixture, root)
|
|
||||||
return root.resolve()
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def build_metrics(result: dict[str, object]) -> dict[str, object]:
|
|
||||||
return cast(dict[str, object], result["build"])
|
|
||||||
|
|
||||||
def test_both_grammars_produce_deterministic_complete_logic_assemblies(self) -> None:
|
|
||||||
for fixture, language, nodes, edges, logic, _shared in self.CASES:
|
|
||||||
with self.subTest(language=language), tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory), fixture)
|
|
||||||
adapter = RecordingScriptAdapter(root, fixture)
|
|
||||||
|
|
||||||
first = adapter.load_assembly()
|
|
||||||
second = adapter.load_complete_assembly()
|
|
||||||
projection = adapter.load_projection()
|
|
||||||
|
|
||||||
self.assertEqual(first, second)
|
|
||||||
self.assertEqual(first.projection, projection)
|
|
||||||
self.assertEqual(JAVASCRIPT_ADAPTER_ID, projection.adapter_id)
|
|
||||||
self.assertEqual(JAVASCRIPT_ADAPTER_VERSION, projection.adapter_version)
|
|
||||||
self.assertEqual(nodes, len(projection.nodes))
|
|
||||||
self.assertEqual(edges, len(projection.edges))
|
|
||||||
self.assertEqual(logic, len(first.logic))
|
|
||||||
self.assertEqual(
|
|
||||||
{language},
|
|
||||||
{dict(item.metadata)["language"] for item in projection.nodes},
|
|
||||||
)
|
|
||||||
self.assertEqual(
|
|
||||||
{"contains", "depends_on"},
|
|
||||||
{item.edge.relation for item in projection.edges},
|
|
||||||
)
|
|
||||||
self.assertTrue(
|
|
||||||
any(
|
|
||||||
item.node.title == "Service.run"
|
|
||||||
and dict(item.metadata)["kind"] == "function"
|
|
||||||
for item in projection.nodes
|
|
||||||
)
|
|
||||||
)
|
|
||||||
self.assertFalse((root / ".cache").exists())
|
|
||||||
|
|
||||||
report = adapter.support_report()
|
|
||||||
self.assertEqual(1, report["schema_version"])
|
|
||||||
self.assertEqual(JAVASCRIPT_EXTRACTOR_VERSION, report["extractor_version"])
|
|
||||||
self.assertFalse(report["imports_project_code"])
|
|
||||||
self.assertFalse(report["executes_project_code"])
|
|
||||||
self.assertEqual(
|
|
||||||
[
|
|
||||||
"call_resolution",
|
|
||||||
"dynamic_module_resolution",
|
|
||||||
"inheritance_resolution",
|
|
||||||
"module_configuration_resolution",
|
|
||||||
"runtime_generated_facts",
|
|
||||||
"type_and_symbol_resolution",
|
|
||||||
],
|
|
||||||
[item.code for item in adapter.unsupported_facts()],
|
|
||||||
)
|
|
||||||
|
|
||||||
conformance = verify_adapter_conformance(
|
|
||||||
adapter,
|
|
||||||
cache_root=root / ".cache" / "conformance",
|
|
||||||
)
|
|
||||||
self.assertEqual(nodes, conformance.node_count)
|
|
||||||
self.assertEqual(edges, conformance.edge_count)
|
|
||||||
self.assertEqual(logic, conformance.logic_projection_count)
|
|
||||||
self.assertTrue(conformance.incremental)
|
|
||||||
|
|
||||||
def test_cold_warm_reverse_dependency_and_exact_equivalence_for_both_grammars(
|
|
||||||
self,
|
|
||||||
) -> None:
|
|
||||||
for fixture, language, nodes, edges, logic, shared_path in self.CASES:
|
|
||||||
with self.subTest(language=language), tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory), fixture)
|
|
||||||
adapter = RecordingScriptAdapter(root, fixture)
|
|
||||||
project = AdapterProject(adapter, cache_root=root / ".cache" / "incremental")
|
|
||||||
index = ProjectIndex(project)
|
|
||||||
|
|
||||||
cold = index.build()
|
|
||||||
self.assertEqual(4, self.build_metrics(cold)["reparsed_sources"])
|
|
||||||
self.assertEqual(4, len(adapter.extracted_paths))
|
|
||||||
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
with mock.patch.object(
|
|
||||||
adapter,
|
|
||||||
"_parse",
|
|
||||||
wraps=adapter._parse, # pyright: ignore[reportPrivateUsage]
|
|
||||||
) as parsed:
|
|
||||||
warm = index.build()
|
|
||||||
self.assertEqual(4, self.build_metrics(warm)["cache_hits"])
|
|
||||||
self.assertEqual(0, self.build_metrics(warm)["reparsed_sources"])
|
|
||||||
self.assertEqual([], adapter.extracted_paths)
|
|
||||||
parsed.assert_not_called()
|
|
||||||
|
|
||||||
shared = root / shared_path
|
|
||||||
shared.write_text(
|
|
||||||
shared.read_text(encoding="utf-8") + "\n// changed dependency evidence\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
with mock.patch.object(
|
|
||||||
adapter,
|
|
||||||
"_parse",
|
|
||||||
wraps=adapter._parse, # pyright: ignore[reportPrivateUsage]
|
|
||||||
) as parsed:
|
|
||||||
changed = index.build()
|
|
||||||
self.assertEqual(4, self.build_metrics(changed)["invalidated_sources"])
|
|
||||||
self.assertEqual(4, len(adapter.extracted_paths))
|
|
||||||
self.assertEqual(4, parsed.call_count)
|
|
||||||
|
|
||||||
equivalent = project.verify_incremental_equivalence()
|
|
||||||
self.assertEqual("ok", equivalent["status"])
|
|
||||||
self.assertEqual(nodes, equivalent["node_count"])
|
|
||||||
self.assertEqual(edges, equivalent["edge_count"])
|
|
||||||
self.assertEqual(logic, equivalent["logic_projection_count"])
|
|
||||||
|
|
||||||
def test_add_delete_and_corrupt_cache_recovery_for_both_grammars(self) -> None:
|
|
||||||
additions = {
|
|
||||||
"javascript": (
|
|
||||||
"src/app/extra.js",
|
|
||||||
"export function extra() { return 'extra'; }\n",
|
|
||||||
),
|
|
||||||
"typescript": (
|
|
||||||
"src/app/extra.ts",
|
|
||||||
"export function extra(): string { return 'extra'; }\n",
|
|
||||||
),
|
|
||||||
}
|
|
||||||
for fixture, language, _nodes, _edges, _logic, _shared_path in self.CASES:
|
|
||||||
with self.subTest(language=language), tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory), fixture)
|
|
||||||
adapter = RecordingScriptAdapter(root, fixture)
|
|
||||||
cache_root = root / ".cache" / "incremental"
|
|
||||||
project = AdapterProject(adapter, cache_root=cache_root)
|
|
||||||
index = ProjectIndex(project)
|
|
||||||
index.build()
|
|
||||||
|
|
||||||
relative, content = additions[language]
|
|
||||||
added_path = root / relative
|
|
||||||
added_path.write_text(content, encoding="utf-8")
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
added = index.build()
|
|
||||||
self.assertEqual(1, self.build_metrics(added)["reparsed_sources"])
|
|
||||||
self.assertEqual([relative], adapter.extracted_paths)
|
|
||||||
self.assertIn(relative, {node.source_path for node in project.load().nodes})
|
|
||||||
|
|
||||||
added_path.unlink()
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
deleted = index.build()
|
|
||||||
self.assertEqual(1, self.build_metrics(deleted)["deleted_sources"])
|
|
||||||
self.assertNotIn(relative, {node.source_path for node in project.load().nodes})
|
|
||||||
|
|
||||||
(cache_root / "extractions.json").write_text("{broken", encoding="utf-8")
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
recovered = index.build()
|
|
||||||
self.assertEqual(4, self.build_metrics(recovered)["reparsed_sources"])
|
|
||||||
self.assertEqual(4, len(adapter.extracted_paths))
|
|
||||||
self.assertEqual("ok", project.verify_incremental_equivalence()["status"])
|
|
||||||
|
|
||||||
def test_confinement_and_no_ast_reject_logic_for_both_grammars(self) -> None:
|
|
||||||
for fixture, language, _nodes, _edges, _logic, _shared_path in self.CASES:
|
|
||||||
with self.subTest(language=language), tempfile.TemporaryDirectory() as directory:
|
|
||||||
parent = Path(directory)
|
|
||||||
root = self.copy_fixture(parent, fixture)
|
|
||||||
|
|
||||||
for source_root in (root / "src", Path("../outside")):
|
|
||||||
with self.assertRaises(DocForgeError) as captured:
|
|
||||||
JavaScriptReferenceAdapter(root, source_roots=(source_root,))
|
|
||||||
self.assertEqual("path_escape", captured.exception.code)
|
|
||||||
|
|
||||||
outside = parent / "outside"
|
|
||||||
outside.mkdir()
|
|
||||||
linked_root = root / "linked"
|
|
||||||
linked_root.symlink_to(outside, target_is_directory=True)
|
|
||||||
with self.assertRaises(DocForgeError) as linked:
|
|
||||||
JavaScriptReferenceAdapter(root, source_roots=("linked",))
|
|
||||||
self.assertEqual("path_escape", linked.exception.code)
|
|
||||||
|
|
||||||
adapter = RecordingScriptAdapter(root, fixture)
|
|
||||||
project = AdapterProject(adapter, cache_root=root / ".cache" / "no-ast")
|
|
||||||
with self.assertRaises(DocForgeError) as no_ast:
|
|
||||||
ProjectIndex(project, allow_logic=False).build()
|
|
||||||
self.assertEqual("adapter_policy_forbids_logic", no_ast.exception.code)
|
|
||||||
self.assertFalse(project.descriptor.index_path.exists())
|
|
||||||
|
|
||||||
def test_missing_optional_frontends_report_exact_install_remediation(self) -> None:
|
|
||||||
for fixture, language, _nodes, _edges, _logic, _shared_path in self.CASES:
|
|
||||||
grammar_module = (
|
|
||||||
"tree_sitter_javascript" if language == "javascript" else "tree_sitter_typescript"
|
|
||||||
)
|
|
||||||
with self.subTest(language=language), tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory), fixture)
|
|
||||||
adapter = RecordingScriptAdapter(root, fixture)
|
|
||||||
missing = ModuleNotFoundError(
|
|
||||||
f"No module named {grammar_module!r}",
|
|
||||||
name=grammar_module,
|
|
||||||
)
|
|
||||||
with (
|
|
||||||
mock.patch.object(
|
|
||||||
javascript_adapter.importlib,
|
|
||||||
"import_module",
|
|
||||||
side_effect=missing,
|
|
||||||
),
|
|
||||||
self.assertRaises(DocForgeError) as captured,
|
|
||||||
):
|
|
||||||
source = adapter.load_manifest().sources[0]
|
|
||||||
adapter.extract_source(source)
|
|
||||||
self.assertEqual("optional_dependency_missing", captured.exception.code)
|
|
||||||
self.assertEqual(language, captured.exception.details["extra"])
|
|
||||||
self.assertEqual(
|
|
||||||
f"docforge[{language}]",
|
|
||||||
captured.exception.details["install"],
|
|
||||||
)
|
|
||||||
self.assertEqual(
|
|
||||||
grammar_module,
|
|
||||||
captured.exception.details["missing_module"],
|
|
||||||
)
|
|
||||||
|
|
@ -1,137 +0,0 @@
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import subprocess
|
|
||||||
import tempfile
|
|
||||||
import unittest
|
|
||||||
from pathlib import Path
|
|
||||||
from unittest import mock
|
|
||||||
|
|
||||||
from tools.milestone4_adoption import (
|
|
||||||
AdoptionProofError,
|
|
||||||
find_wheel,
|
|
||||||
parse_evidence,
|
|
||||||
run_adoption_proof,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _evidence() -> dict[str, object]:
|
|
||||||
return {
|
|
||||||
"schema_version": 1,
|
|
||||||
"base_frontend_modules": [],
|
|
||||||
"base_frontend_distributions": [],
|
|
||||||
"python": {"build_status": "ok", "check_status": "ok", "node_count": 3},
|
|
||||||
"mcp": {
|
|
||||||
"bootstrap_status": "ok",
|
|
||||||
"node_id": "python.class.service",
|
|
||||||
"read_tool_count": 20,
|
|
||||||
},
|
|
||||||
"cpp_without_extra": {
|
|
||||||
"code": "optional_dependency_missing",
|
|
||||||
"install": "docforge[cpp]",
|
|
||||||
"missing_module": "tree_sitter",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
class Milestone4AdoptionTests(unittest.TestCase):
|
|
||||||
def test_evidence_requires_base_isolation_python_mcp_and_actionable_cpp_failure(
|
|
||||||
self,
|
|
||||||
) -> None:
|
|
||||||
self.assertEqual(_evidence(), parse_evidence(json.dumps(_evidence())))
|
|
||||||
|
|
||||||
for mutation in (
|
|
||||||
{"base_frontend_modules": ["tree_sitter"]},
|
|
||||||
{"python": {"build_status": "error", "check_status": "ok"}},
|
|
||||||
{"mcp": {"bootstrap_status": "error"}},
|
|
||||||
{
|
|
||||||
"cpp_without_extra": {
|
|
||||||
"code": "optional_dependency_missing",
|
|
||||||
"install": "docforge[languages]",
|
|
||||||
}
|
|
||||||
},
|
|
||||||
):
|
|
||||||
evidence = {**_evidence(), **mutation}
|
|
||||||
with self.subTest(mutation=mutation), self.assertRaises(AdoptionProofError):
|
|
||||||
parse_evidence(json.dumps(evidence))
|
|
||||||
|
|
||||||
def test_wheel_discovery_rejects_missing_or_ambiguous_artifacts(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = Path(directory)
|
|
||||||
with self.assertRaises(AdoptionProofError):
|
|
||||||
find_wheel(root)
|
|
||||||
first = root / "docforge-1-py3-none-any.whl"
|
|
||||||
first.touch()
|
|
||||||
self.assertEqual(first, find_wheel(root))
|
|
||||||
(root / "docforge-2-py3-none-any.whl").touch()
|
|
||||||
with self.assertRaises(AdoptionProofError):
|
|
||||||
find_wheel(root)
|
|
||||||
|
|
||||||
def test_orchestration_is_offline_locked_and_installs_the_wheel_without_extras(
|
|
||||||
self,
|
|
||||||
) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = Path(directory).resolve()
|
|
||||||
(root / "pyproject.toml").write_text("[project]\n", encoding="utf-8")
|
|
||||||
(root / "uv.lock").write_text("version = 1\n", encoding="utf-8")
|
|
||||||
commands: list[list[str]] = []
|
|
||||||
|
|
||||||
def execute(
|
|
||||||
arguments: list[str],
|
|
||||||
*,
|
|
||||||
cwd: Path,
|
|
||||||
timeout: int,
|
|
||||||
environment: dict[str, str] | None = None,
|
|
||||||
) -> subprocess.CompletedProcess[str]:
|
|
||||||
del cwd, timeout
|
|
||||||
commands.append(arguments)
|
|
||||||
self.assertIsNotNone(environment)
|
|
||||||
assert environment is not None
|
|
||||||
self.assertEqual("1", environment["UV_OFFLINE"])
|
|
||||||
self.assertNotIn("PYTHONPATH", environment)
|
|
||||||
if arguments[1] == "build":
|
|
||||||
output = Path(arguments[arguments.index("--out-dir") + 1])
|
|
||||||
output.mkdir()
|
|
||||||
(output / "docforge-1-py3-none-any.whl").write_bytes(b"wheel")
|
|
||||||
elif arguments[1] == "export":
|
|
||||||
output = Path(arguments[arguments.index("--output-file") + 1])
|
|
||||||
output.write_text("mcp==1\n", encoding="utf-8")
|
|
||||||
elif arguments[1] == "venv":
|
|
||||||
virtual_environment = Path(arguments[-1])
|
|
||||||
(virtual_environment / "bin").mkdir(parents=True)
|
|
||||||
(virtual_environment / "bin" / "python").touch()
|
|
||||||
elif arguments[0].endswith("/bin/python"):
|
|
||||||
return subprocess.CompletedProcess(
|
|
||||||
arguments,
|
|
||||||
0,
|
|
||||||
stdout=json.dumps(_evidence()),
|
|
||||||
stderr="",
|
|
||||||
)
|
|
||||||
return subprocess.CompletedProcess(arguments, 0, stdout="", stderr="")
|
|
||||||
|
|
||||||
with mock.patch(
|
|
||||||
"tools.milestone4_adoption._run_checked",
|
|
||||||
side_effect=execute,
|
|
||||||
):
|
|
||||||
result = run_adoption_proof(root, uv="uv")
|
|
||||||
|
|
||||||
self.assertEqual("offline", result["network"])
|
|
||||||
wheel = result["wheel"]
|
|
||||||
self.assertIsInstance(wheel, dict)
|
|
||||||
assert isinstance(wheel, dict)
|
|
||||||
self.assertEqual("docforge-1-py3-none-any.whl", wheel["filename"])
|
|
||||||
network_commands = (
|
|
||||||
command
|
|
||||||
for command in commands
|
|
||||||
if command[0] == "uv" and command[1] in {"build", "export", "pip"}
|
|
||||||
)
|
|
||||||
self.assertTrue(all("--offline" in command for command in network_commands))
|
|
||||||
export = next(command for command in commands if command[1] == "export")
|
|
||||||
self.assertIn("--frozen", export)
|
|
||||||
self.assertIn("--no-dev", export)
|
|
||||||
wheel_install = next(command for command in commands if command[1:3] == ["pip", "install"])
|
|
||||||
self.assertIn("--no-deps", wheel_install)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
unittest.main()
|
|
||||||
|
|
@ -43,7 +43,6 @@ PUBLIC_IMPORTS = {
|
||||||
),
|
),
|
||||||
"docforge.adapter_contract": (
|
"docforge.adapter_contract": (
|
||||||
"AdapterAssembly",
|
"AdapterAssembly",
|
||||||
"AdapterConformanceReport",
|
|
||||||
"AdapterEdge",
|
"AdapterEdge",
|
||||||
"AdapterImplementation",
|
"AdapterImplementation",
|
||||||
"AdapterLoader",
|
"AdapterLoader",
|
||||||
|
|
@ -54,41 +53,15 @@ PUBLIC_IMPORTS = {
|
||||||
"AdapterProjectSettings",
|
"AdapterProjectSettings",
|
||||||
"AdapterSource",
|
"AdapterSource",
|
||||||
"AdapterSourceProjection",
|
"AdapterSourceProjection",
|
||||||
"CompleteAdapterAssemblyLoader",
|
|
||||||
"IncrementalAdapterAssembler",
|
"IncrementalAdapterAssembler",
|
||||||
"IncrementalAdapterLoader",
|
"IncrementalAdapterLoader",
|
||||||
"verify_adapter_conformance",
|
|
||||||
),
|
|
||||||
"docforge.adapter_sdk": (
|
|
||||||
"AdapterAssembly",
|
|
||||||
"AdapterConformanceReport",
|
|
||||||
"AdapterLoader",
|
|
||||||
"AdapterManifest",
|
|
||||||
"AdapterProject",
|
|
||||||
"AdapterProjection",
|
|
||||||
"AdapterSourceProjection",
|
|
||||||
"CompleteAdapterAssemblyLoader",
|
|
||||||
"IncrementalAdapterLoader",
|
|
||||||
"LogicProjection",
|
|
||||||
"Node",
|
|
||||||
"verify_adapter_conformance",
|
|
||||||
),
|
),
|
||||||
"docforge.application": (
|
"docforge.application": (
|
||||||
"CanonicalApplier",
|
"CanonicalApplier",
|
||||||
"CanonicalApplicationService",
|
"CanonicalApplicationService",
|
||||||
"GenericCanonicalApplier",
|
"GenericCanonicalApplier",
|
||||||
),
|
),
|
||||||
"docforge.adapter_launcher": (
|
"docforge.client_config": ("generate_client_configuration",),
|
||||||
"ADAPTER_LAUNCHER_SCHEMA_VERSION",
|
|
||||||
"AdapterLauncherV1",
|
|
||||||
"AdapterSourceAvailabilityV1",
|
|
||||||
"adapter_source_availability",
|
|
||||||
"validate_adapter_launcher",
|
|
||||||
),
|
|
||||||
"docforge.client_config": (
|
|
||||||
"generate_adapter_client_configuration",
|
|
||||||
"generate_client_configuration",
|
|
||||||
),
|
|
||||||
"docforge.doctor": ("run_doctor",),
|
"docforge.doctor": ("run_doctor",),
|
||||||
"docforge.index": ("ProjectIndex",),
|
"docforge.index": ("ProjectIndex",),
|
||||||
"docforge.graph_projection": (
|
"docforge.graph_projection": (
|
||||||
|
|
@ -145,18 +118,6 @@ PUBLIC_IMPORTS = {
|
||||||
"projection_hash",
|
"projection_hash",
|
||||||
),
|
),
|
||||||
"docforge.projection_worker": ("render_projection_in_worker",),
|
"docforge.projection_worker": ("render_projection_in_worker",),
|
||||||
"docforge.reference_config": (
|
|
||||||
"REFERENCE_ADAPTER_CONFIG",
|
|
||||||
"REFERENCE_ADAPTER_SCHEMA_VERSION",
|
|
||||||
"REFERENCE_LANGUAGES",
|
|
||||||
"ReferenceAdapterConfigV1",
|
|
||||||
"load_reference_adapter_config",
|
|
||||||
),
|
|
||||||
"docforge.reference_mcp": (
|
|
||||||
"REFERENCE_MCP_MODULE",
|
|
||||||
"create_reference_project",
|
|
||||||
"create_reference_server",
|
|
||||||
),
|
|
||||||
"docforge.retrieval": (
|
"docforge.retrieval": (
|
||||||
"ContextCapsuleV1",
|
"ContextCapsuleV1",
|
||||||
"RetrievalPlanV1",
|
"RetrievalPlanV1",
|
||||||
|
|
|
||||||
|
|
@ -1,282 +0,0 @@
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import shutil
|
|
||||||
import tempfile
|
|
||||||
import unittest
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import cast
|
|
||||||
from unittest import mock
|
|
||||||
|
|
||||||
import docforge.adapters.python as python_adapter
|
|
||||||
from docforge.adapter_sdk import (
|
|
||||||
AdapterProject,
|
|
||||||
AdapterSource,
|
|
||||||
AdapterSourceProjection,
|
|
||||||
verify_adapter_conformance,
|
|
||||||
)
|
|
||||||
from docforge.adapters.python import (
|
|
||||||
PYTHON_ADAPTER_ID,
|
|
||||||
PYTHON_ADAPTER_VERSION,
|
|
||||||
PYTHON_EXTRACTOR_VERSION,
|
|
||||||
PythonReferenceAdapter,
|
|
||||||
)
|
|
||||||
from docforge.errors import DocForgeError
|
|
||||||
from docforge.index import ProjectIndex
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
FIXTURE = ROOT / "tests" / "fixtures" / "reference-python"
|
|
||||||
|
|
||||||
|
|
||||||
class RecordingPythonAdapter(PythonReferenceAdapter):
|
|
||||||
def __init__(self, root: Path) -> None:
|
|
||||||
super().__init__(
|
|
||||||
root,
|
|
||||||
source_roots=("src",),
|
|
||||||
project_id="reference-python",
|
|
||||||
title="Reference Python fixture",
|
|
||||||
)
|
|
||||||
self.extracted_paths: list[str] = []
|
|
||||||
|
|
||||||
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection:
|
|
||||||
self.extracted_paths.append(source.source_path)
|
|
||||||
return super().extract_source(source)
|
|
||||||
|
|
||||||
|
|
||||||
class PythonReferenceAdapterTests(unittest.TestCase):
|
|
||||||
def copy_fixture(self, parent: Path) -> Path:
|
|
||||||
root = parent / "reference-python"
|
|
||||||
shutil.copytree(FIXTURE, root)
|
|
||||||
return root.resolve()
|
|
||||||
|
|
||||||
def adapter(self, root: Path) -> RecordingPythonAdapter:
|
|
||||||
return RecordingPythonAdapter(root)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def build_metrics(result: dict[str, object]) -> dict[str, object]:
|
|
||||||
return cast(dict[str, object], result["build"])
|
|
||||||
|
|
||||||
def test_complete_assembly_is_exact_ordered_cache_independent_and_scoped(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
adapter = self.adapter(root)
|
|
||||||
|
|
||||||
first = adapter.load_assembly()
|
|
||||||
second = adapter.load_complete_assembly()
|
|
||||||
projection = adapter.load_projection()
|
|
||||||
|
|
||||||
self.assertEqual(first, second)
|
|
||||||
self.assertEqual(first.projection, projection)
|
|
||||||
self.assertEqual(PYTHON_ADAPTER_ID, projection.adapter_id)
|
|
||||||
self.assertEqual(PYTHON_ADAPTER_VERSION, projection.adapter_version)
|
|
||||||
self.assertEqual(14, len(projection.nodes))
|
|
||||||
self.assertEqual(13, len(projection.edges))
|
|
||||||
self.assertEqual(5, len(first.logic))
|
|
||||||
self.assertEqual(
|
|
||||||
list(projection.nodes),
|
|
||||||
sorted(projection.nodes, key=lambda item: item.node.node_id),
|
|
||||||
)
|
|
||||||
self.assertEqual(
|
|
||||||
{"contains", "depends_on"}, {item.edge.relation for item in projection.edges}
|
|
||||||
)
|
|
||||||
self.assertFalse(any(item.node.title == "json" for item in projection.nodes))
|
|
||||||
self.assertTrue(
|
|
||||||
any(
|
|
||||||
item.node.title == "Service" and ("kind", "class") in item.metadata
|
|
||||||
for item in projection.nodes
|
|
||||||
)
|
|
||||||
)
|
|
||||||
self.assertTrue(
|
|
||||||
any(
|
|
||||||
item.node.title == "Service.run" and ("kind", "function") in item.metadata
|
|
||||||
for item in projection.nodes
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
report = adapter.support_report()
|
|
||||||
self.assertEqual(1, report["schema_version"])
|
|
||||||
self.assertEqual(PYTHON_EXTRACTOR_VERSION, report["extractor_version"])
|
|
||||||
self.assertFalse(report["imports_project_code"])
|
|
||||||
self.assertFalse(report["executes_project_code"])
|
|
||||||
self.assertEqual(
|
|
||||||
[
|
|
||||||
"call_resolution",
|
|
||||||
"dynamic_import_resolution",
|
|
||||||
"inheritance_resolution",
|
|
||||||
"runtime_generated_facts",
|
|
||||||
"symbol_reference_resolution",
|
|
||||||
],
|
|
||||||
[item.code for item in adapter.unsupported_facts()],
|
|
||||||
)
|
|
||||||
self.assertFalse((root / ".cache").exists())
|
|
||||||
|
|
||||||
conformance = verify_adapter_conformance(
|
|
||||||
adapter,
|
|
||||||
cache_root=root / ".cache" / "conformance",
|
|
||||||
)
|
|
||||||
self.assertEqual("reference-python", conformance.project_id)
|
|
||||||
self.assertEqual(14, conformance.node_count)
|
|
||||||
self.assertEqual(13, conformance.edge_count)
|
|
||||||
self.assertEqual(5, conformance.logic_projection_count)
|
|
||||||
self.assertTrue(conformance.incremental)
|
|
||||||
|
|
||||||
def test_incremental_cold_warm_reverse_dependencies_and_logic_equivalence(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
adapter = self.adapter(root)
|
|
||||||
project = AdapterProject(adapter, cache_root=root / ".cache" / "incremental")
|
|
||||||
index = ProjectIndex(project)
|
|
||||||
|
|
||||||
cold = index.build()
|
|
||||||
self.assertEqual(4, self.build_metrics(cold)["reparsed_sources"])
|
|
||||||
self.assertEqual(
|
|
||||||
[
|
|
||||||
"src/sample/__init__.py",
|
|
||||||
"src/sample/service.py",
|
|
||||||
"src/sample/shared.py",
|
|
||||||
"src/sample/worker.py",
|
|
||||||
],
|
|
||||||
sorted(adapter.extracted_paths),
|
|
||||||
)
|
|
||||||
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
original_parse = python_adapter.ast.parse
|
|
||||||
with mock.patch.object(
|
|
||||||
python_adapter.ast,
|
|
||||||
"parse",
|
|
||||||
wraps=original_parse,
|
|
||||||
) as parsed:
|
|
||||||
warm = index.build()
|
|
||||||
self.assertEqual(4, self.build_metrics(warm)["cache_hits"])
|
|
||||||
self.assertEqual(0, self.build_metrics(warm)["reparsed_sources"])
|
|
||||||
self.assertEqual([], adapter.extracted_paths)
|
|
||||||
parsed.assert_not_called()
|
|
||||||
|
|
||||||
shared = root / "src" / "sample" / "shared.py"
|
|
||||||
shared.write_text(
|
|
||||||
shared.read_text(encoding="utf-8").replace(
|
|
||||||
"DEFAULT_LIMIT = 3", "DEFAULT_LIMIT = 5"
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
with mock.patch.object(
|
|
||||||
python_adapter.ast,
|
|
||||||
"parse",
|
|
||||||
wraps=original_parse,
|
|
||||||
) as parsed:
|
|
||||||
changed = index.build()
|
|
||||||
self.assertEqual(4, self.build_metrics(changed)["invalidated_sources"])
|
|
||||||
self.assertGreater(parsed.call_count, 0)
|
|
||||||
self.assertEqual(
|
|
||||||
[
|
|
||||||
"src/sample/__init__.py",
|
|
||||||
"src/sample/service.py",
|
|
||||||
"src/sample/shared.py",
|
|
||||||
"src/sample/worker.py",
|
|
||||||
],
|
|
||||||
sorted(adapter.extracted_paths),
|
|
||||||
)
|
|
||||||
|
|
||||||
equivalent = project.verify_incremental_equivalence()
|
|
||||||
self.assertEqual("ok", equivalent["status"])
|
|
||||||
self.assertEqual(14, equivalent["node_count"])
|
|
||||||
self.assertEqual(13, equivalent["edge_count"])
|
|
||||||
self.assertEqual(5, equivalent["logic_projection_count"])
|
|
||||||
|
|
||||||
def test_add_rename_delete_and_corrupt_cache_recover_without_stale_facts(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
adapter = self.adapter(root)
|
|
||||||
cache_root = root / ".cache" / "incremental"
|
|
||||||
project = AdapterProject(adapter, cache_root=cache_root)
|
|
||||||
index = ProjectIndex(project)
|
|
||||||
index.build()
|
|
||||||
|
|
||||||
extra = root / "src" / "sample" / "extra.py"
|
|
||||||
extra.write_text(
|
|
||||||
'"""Added source."""\n\n\ndef extra() -> str:\n return "extra"\n',
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
added = index.build()
|
|
||||||
self.assertEqual(1, self.build_metrics(added)["reparsed_sources"])
|
|
||||||
self.assertEqual(["src/sample/extra.py"], adapter.extracted_paths)
|
|
||||||
self.assertIn(
|
|
||||||
"src/sample/extra.py",
|
|
||||||
{node.source_path for node in project.load().nodes},
|
|
||||||
)
|
|
||||||
|
|
||||||
worker = root / "src" / "sample" / "worker.py"
|
|
||||||
runner = root / "src" / "sample" / "runner.py"
|
|
||||||
worker.rename(runner)
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
renamed = index.build()
|
|
||||||
self.assertEqual(1, self.build_metrics(renamed)["deleted_sources"])
|
|
||||||
self.assertEqual(1, self.build_metrics(renamed)["reparsed_sources"])
|
|
||||||
renamed_paths = {node.source_path for node in project.load().nodes}
|
|
||||||
self.assertNotIn("src/sample/worker.py", renamed_paths)
|
|
||||||
self.assertIn("src/sample/runner.py", renamed_paths)
|
|
||||||
|
|
||||||
extra.unlink()
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
deleted = index.build()
|
|
||||||
self.assertEqual(1, self.build_metrics(deleted)["deleted_sources"])
|
|
||||||
self.assertNotIn(
|
|
||||||
"src/sample/extra.py",
|
|
||||||
{node.source_path for node in project.load().nodes},
|
|
||||||
)
|
|
||||||
|
|
||||||
(cache_root / "extractions.json").write_text("{broken", encoding="utf-8")
|
|
||||||
adapter.extracted_paths.clear()
|
|
||||||
recovered = index.build()
|
|
||||||
self.assertEqual(4, self.build_metrics(recovered)["reparsed_sources"])
|
|
||||||
self.assertEqual(
|
|
||||||
[
|
|
||||||
"src/sample/__init__.py",
|
|
||||||
"src/sample/runner.py",
|
|
||||||
"src/sample/service.py",
|
|
||||||
"src/sample/shared.py",
|
|
||||||
],
|
|
||||||
sorted(adapter.extracted_paths),
|
|
||||||
)
|
|
||||||
self.assertEqual("ok", project.verify_incremental_equivalence()["status"])
|
|
||||||
|
|
||||||
def test_confinement_and_no_ast_policy_reject_logic_without_importing_code(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
parent = Path(directory)
|
|
||||||
root = self.copy_fixture(parent)
|
|
||||||
|
|
||||||
for source_root in (root / "src", Path("../outside")):
|
|
||||||
with self.subTest(source_root=source_root):
|
|
||||||
with self.assertRaises(DocForgeError) as captured:
|
|
||||||
PythonReferenceAdapter(root, source_roots=(source_root,))
|
|
||||||
self.assertEqual("path_escape", captured.exception.code)
|
|
||||||
|
|
||||||
outside = parent / "outside"
|
|
||||||
outside.mkdir()
|
|
||||||
linked_root = root / "linked"
|
|
||||||
linked_root.symlink_to(outside, target_is_directory=True)
|
|
||||||
with self.assertRaises(DocForgeError) as linked:
|
|
||||||
PythonReferenceAdapter(root, source_roots=("linked",))
|
|
||||||
self.assertEqual("path_escape", linked.exception.code)
|
|
||||||
|
|
||||||
external_source = outside / "external.py"
|
|
||||||
external_source.write_text("def outside():\n return True\n", encoding="utf-8")
|
|
||||||
worker = root / "src" / "sample" / "worker.py"
|
|
||||||
worker.unlink()
|
|
||||||
worker.symlink_to(external_source)
|
|
||||||
adapter = self.adapter(root)
|
|
||||||
with self.assertRaises(DocForgeError) as source_link:
|
|
||||||
adapter.load_manifest()
|
|
||||||
self.assertEqual("path_escape", source_link.exception.code)
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
adapter = self.adapter(root)
|
|
||||||
project = AdapterProject(adapter, cache_root=root / ".cache" / "no-ast")
|
|
||||||
|
|
||||||
with self.assertRaises(DocForgeError) as no_ast:
|
|
||||||
ProjectIndex(project, allow_logic=False).build()
|
|
||||||
|
|
||||||
self.assertEqual("adapter_policy_forbids_logic", no_ast.exception.code)
|
|
||||||
self.assertFalse(project.descriptor.index_path.exists())
|
|
||||||
|
|
@ -1,296 +0,0 @@
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import io
|
|
||||||
import json
|
|
||||||
import shutil
|
|
||||||
import sys
|
|
||||||
import tempfile
|
|
||||||
import unittest
|
|
||||||
from contextlib import redirect_stderr
|
|
||||||
from pathlib import Path
|
|
||||||
from unittest import mock
|
|
||||||
|
|
||||||
from jsonschema import Draft202012Validator
|
|
||||||
from mcp.shared.memory import create_connected_server_and_client_session
|
|
||||||
|
|
||||||
from docforge.adapter_launcher import AdapterLauncherV1
|
|
||||||
from docforge.client_config import generate_adapter_client_configuration
|
|
||||||
from docforge.errors import DocForgeError
|
|
||||||
from docforge.index import ProjectIndex
|
|
||||||
from docforge.mcp_server import READ_TOOLS
|
|
||||||
from docforge.reference_config import (
|
|
||||||
REFERENCE_ADAPTER_CONFIG,
|
|
||||||
load_reference_adapter_config,
|
|
||||||
)
|
|
||||||
from docforge.reference_mcp import (
|
|
||||||
REFERENCE_MCP_MODULE,
|
|
||||||
create_reference_project,
|
|
||||||
create_reference_server,
|
|
||||||
main,
|
|
||||||
)
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
FIXTURE = ROOT / "tests" / "fixtures" / "reference-python"
|
|
||||||
CPP_FIXTURE = ROOT / "tests" / "fixtures" / "reference-cpp"
|
|
||||||
SCHEMA = json.loads(
|
|
||||||
(ROOT / "schemas" / "reference-adapter.schema.json").read_text(encoding="utf-8")
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class ReferenceMcpTests(unittest.IsolatedAsyncioTestCase):
|
|
||||||
def copy_fixture(self, parent: Path) -> Path:
|
|
||||||
root = parent / "reference-python"
|
|
||||||
shutil.copytree(FIXTURE, root)
|
|
||||||
return root.resolve()
|
|
||||||
|
|
||||||
def copy_cpp_fixture(self, parent: Path) -> Path:
|
|
||||||
root = parent / "reference-cpp"
|
|
||||||
shutil.copytree(CPP_FIXTURE, root)
|
|
||||||
return root.resolve()
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def write_config(
|
|
||||||
root: Path,
|
|
||||||
*,
|
|
||||||
language: str = "python",
|
|
||||||
source_roots: tuple[str, ...] = ("src",),
|
|
||||||
extra: str = "",
|
|
||||||
) -> Path:
|
|
||||||
config = root.joinpath(*REFERENCE_ADAPTER_CONFIG.parts)
|
|
||||||
config.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
quoted_roots = ", ".join(json.dumps(item) for item in source_roots)
|
|
||||||
config.write_text(
|
|
||||||
"\n".join(
|
|
||||||
(
|
|
||||||
"schema_version = 1",
|
|
||||||
'project_id = "reference-python"',
|
|
||||||
'title = "Runnable Python reference"',
|
|
||||||
f"language = {json.dumps(language)}",
|
|
||||||
f"source_roots = [{quoted_roots}]",
|
|
||||||
extra,
|
|
||||||
"",
|
|
||||||
)
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
return config
|
|
||||||
|
|
||||||
async def test_python_config_build_check_and_real_mcp_retrieval(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
self.write_config(root)
|
|
||||||
config = load_reference_adapter_config(root)
|
|
||||||
|
|
||||||
Draft202012Validator(SCHEMA).validate(config.as_dict())
|
|
||||||
self.assertEqual("python", config.language)
|
|
||||||
self.assertEqual((root / "src",), config.source_roots)
|
|
||||||
self.assertIsNone(config.compilation_database)
|
|
||||||
self.assertEqual(64, len(config.config_hash))
|
|
||||||
|
|
||||||
project = create_reference_project(root)
|
|
||||||
index = ProjectIndex(project)
|
|
||||||
built = index.build()
|
|
||||||
checked = index.check()
|
|
||||||
self.assertEqual("ok", built["status"])
|
|
||||||
self.assertEqual("ok", checked["status"])
|
|
||||||
self.assertTrue(project.descriptor.index_path.is_relative_to(root / ".docforge"))
|
|
||||||
|
|
||||||
async with create_connected_server_and_client_session(
|
|
||||||
create_reference_server(root, capability_mode="read"),
|
|
||||||
raise_exceptions=True,
|
|
||||||
) as session:
|
|
||||||
tools = tuple(tool.name for tool in (await session.list_tools()).tools)
|
|
||||||
bootstrap = await session.call_tool("docforge_bootstrap", {})
|
|
||||||
search = await session.call_tool(
|
|
||||||
"docforge_search",
|
|
||||||
{"query": "Service", "limit": 5},
|
|
||||||
)
|
|
||||||
node_id = search.structuredContent["results"][0]["node_id"]
|
|
||||||
node = await session.call_tool("docforge_get_node", {"node_id": node_id})
|
|
||||||
|
|
||||||
self.assertEqual(READ_TOOLS, tools)
|
|
||||||
self.assertEqual("ok", bootstrap.structuredContent["status"])
|
|
||||||
self.assertEqual(
|
|
||||||
REFERENCE_MCP_MODULE,
|
|
||||||
bootstrap.structuredContent["binding"]["server_module"],
|
|
||||||
)
|
|
||||||
self.assertEqual(
|
|
||||||
"python",
|
|
||||||
bootstrap.structuredContent["binding"]["reference_language"],
|
|
||||||
)
|
|
||||||
self.assertGreater(search.structuredContent["count"], 0)
|
|
||||||
self.assertEqual(node_id, node.structuredContent["node"]["node_id"])
|
|
||||||
|
|
||||||
async def test_cpp_config_build_check_and_real_mcp_retrieval(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_cpp_fixture(Path(directory))
|
|
||||||
self.write_config(
|
|
||||||
root,
|
|
||||||
language="cpp",
|
|
||||||
source_roots=("include", "src"),
|
|
||||||
extra='compilation_database = "compile_commands.json"',
|
|
||||||
)
|
|
||||||
config = load_reference_adapter_config(root)
|
|
||||||
Draft202012Validator(SCHEMA).validate(config.as_dict())
|
|
||||||
|
|
||||||
project = create_reference_project(root)
|
|
||||||
index = ProjectIndex(project)
|
|
||||||
self.assertEqual("ok", index.build()["status"])
|
|
||||||
self.assertEqual("ok", index.check()["status"])
|
|
||||||
|
|
||||||
async with create_connected_server_and_client_session(
|
|
||||||
create_reference_server(root, capability_mode="read"),
|
|
||||||
raise_exceptions=True,
|
|
||||||
) as session:
|
|
||||||
bootstrap = await session.call_tool("docforge_bootstrap", {})
|
|
||||||
search = await session.call_tool(
|
|
||||||
"docforge_search",
|
|
||||||
{"query": "worker", "limit": 5},
|
|
||||||
)
|
|
||||||
node_id = search.structuredContent["results"][0]["node_id"]
|
|
||||||
node = await session.call_tool("docforge_get_node", {"node_id": node_id})
|
|
||||||
|
|
||||||
self.assertEqual("cpp", config.language)
|
|
||||||
self.assertEqual(root / "compile_commands.json", config.compilation_database)
|
|
||||||
self.assertEqual(
|
|
||||||
"cpp",
|
|
||||||
bootstrap.structuredContent["binding"]["reference_language"],
|
|
||||||
)
|
|
||||||
self.assertEqual(node_id, node.structuredContent["node"]["node_id"])
|
|
||||||
|
|
||||||
def test_launcher_generates_exact_isolated_module_invocation(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
self.write_config(root)
|
|
||||||
project = create_reference_project(root)
|
|
||||||
launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE)
|
|
||||||
|
|
||||||
with mock.patch("docforge.client_config.subprocess.run") as executed:
|
|
||||||
result = generate_adapter_client_configuration(
|
|
||||||
project,
|
|
||||||
launcher,
|
|
||||||
"codex",
|
|
||||||
capability_mode="read",
|
|
||||||
)
|
|
||||||
|
|
||||||
self.assertEqual(
|
|
||||||
[
|
|
||||||
"-I",
|
|
||||||
"-m",
|
|
||||||
REFERENCE_MCP_MODULE,
|
|
||||||
"--project-root",
|
|
||||||
str(root),
|
|
||||||
"--capability-mode",
|
|
||||||
"read",
|
|
||||||
],
|
|
||||||
result["binding"]["args"],
|
|
||||||
)
|
|
||||||
self.assertEqual({}, result["binding"]["environment"])
|
|
||||||
self.assertNotIn("cwd", result["binding"])
|
|
||||||
executed.assert_not_called()
|
|
||||||
|
|
||||||
def test_config_drift_requires_runtime_restart(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
config_path = self.write_config(root)
|
|
||||||
project = create_reference_project(root)
|
|
||||||
project.validate_runtime()
|
|
||||||
|
|
||||||
config_path.write_text(
|
|
||||||
config_path.read_text(encoding="utf-8").replace(
|
|
||||||
"Runnable Python reference",
|
|
||||||
"Changed Python reference",
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
with self.assertRaises(DocForgeError) as captured:
|
|
||||||
project.validate_runtime()
|
|
||||||
self.assertEqual("adapter_restart_required", captured.exception.code)
|
|
||||||
self.assertEqual(
|
|
||||||
[REFERENCE_ADAPTER_CONFIG.as_posix()],
|
|
||||||
captured.exception.details["changed"],
|
|
||||||
)
|
|
||||||
|
|
||||||
def test_config_rejects_unknown_fields_languages_and_language_mismatch(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
root = self.copy_fixture(Path(directory))
|
|
||||||
for language, extra, code in (
|
|
||||||
("ruby", "", "unsupported_reference_language"),
|
|
||||||
("javascript", "", "reference_language_mismatch"),
|
|
||||||
("python", 'command = "python project.py"', "invalid_reference_config"),
|
|
||||||
(
|
|
||||||
"python",
|
|
||||||
'compilation_database = "compile_commands.json"',
|
|
||||||
"invalid_reference_config",
|
|
||||||
),
|
|
||||||
("cpp", "", "invalid_reference_config"),
|
|
||||||
):
|
|
||||||
with self.subTest(language=language, extra=extra):
|
|
||||||
self.write_config(root, language=language, extra=extra)
|
|
||||||
with self.assertRaises(DocForgeError) as captured:
|
|
||||||
load_reference_adapter_config(root)
|
|
||||||
self.assertEqual(code, captured.exception.code)
|
|
||||||
|
|
||||||
def test_config_rejects_path_escape_and_source_symlinks(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
parent = Path(directory)
|
|
||||||
root = self.copy_fixture(parent)
|
|
||||||
outside = parent / "outside"
|
|
||||||
outside.mkdir()
|
|
||||||
|
|
||||||
self.write_config(root, source_roots=("../outside",))
|
|
||||||
with self.assertRaises(DocForgeError) as escaped:
|
|
||||||
load_reference_adapter_config(root)
|
|
||||||
self.assertEqual("path_escape", escaped.exception.code)
|
|
||||||
|
|
||||||
linked = root / "linked"
|
|
||||||
linked.symlink_to(outside, target_is_directory=True)
|
|
||||||
self.write_config(root, source_roots=("linked",))
|
|
||||||
with self.assertRaises(DocForgeError) as linked_root:
|
|
||||||
load_reference_adapter_config(root)
|
|
||||||
self.assertEqual("path_escape", linked_root.exception.code)
|
|
||||||
|
|
||||||
self.write_config(root)
|
|
||||||
external_source = outside / "external.py"
|
|
||||||
external_source.write_text("def external(): pass\n", encoding="utf-8")
|
|
||||||
source_link = root / "src" / "external.py"
|
|
||||||
source_link.symlink_to(external_source)
|
|
||||||
with self.assertRaises(DocForgeError) as linked_source:
|
|
||||||
load_reference_adapter_config(root)
|
|
||||||
self.assertEqual("path_escape", linked_source.exception.code)
|
|
||||||
|
|
||||||
def test_config_file_must_be_a_regular_non_symlink(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as directory:
|
|
||||||
parent = Path(directory)
|
|
||||||
root = self.copy_fixture(parent)
|
|
||||||
config_path = self.write_config(root)
|
|
||||||
outside = parent / "reference-adapter.toml"
|
|
||||||
shutil.copyfile(config_path, outside)
|
|
||||||
config_path.unlink()
|
|
||||||
config_path.symlink_to(outside)
|
|
||||||
|
|
||||||
with self.assertRaises(DocForgeError) as captured:
|
|
||||||
load_reference_adapter_config(root)
|
|
||||||
self.assertEqual("path_escape", captured.exception.code)
|
|
||||||
|
|
||||||
def test_reference_module_cli_advertises_read_mode_only(self) -> None:
|
|
||||||
with (
|
|
||||||
redirect_stderr(io.StringIO()),
|
|
||||||
mock.patch.object(
|
|
||||||
sys,
|
|
||||||
"argv",
|
|
||||||
[
|
|
||||||
REFERENCE_MCP_MODULE,
|
|
||||||
"--project-root",
|
|
||||||
"/irrelevant",
|
|
||||||
"--capability-mode",
|
|
||||||
"proposal",
|
|
||||||
],
|
|
||||||
),
|
|
||||||
mock.patch("docforge.reference_mcp.create_reference_server") as create_server,
|
|
||||||
self.assertRaises(SystemExit) as captured,
|
|
||||||
):
|
|
||||||
main()
|
|
||||||
|
|
||||||
self.assertEqual(2, captured.exception.code)
|
|
||||||
create_server.assert_not_called()
|
|
||||||
|
|
@ -7,8 +7,6 @@ from docforge.treesitter_logic import (
|
||||||
TreeSitterLogicOwner,
|
TreeSitterLogicOwner,
|
||||||
analyze_cpp_source,
|
analyze_cpp_source,
|
||||||
analyze_javascript_source,
|
analyze_javascript_source,
|
||||||
analyze_typescript_source,
|
|
||||||
discover_typescript_functions,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -202,44 +200,5 @@ public:
|
||||||
self.assertEqual(missing.exception.code, "missing_logic_owner")
|
self.assertEqual(missing.exception.code, "missing_logic_owner")
|
||||||
|
|
||||||
|
|
||||||
class TypeScriptLogicTests(unittest.TestCase):
|
|
||||||
def test_typed_function_and_method_use_the_typescript_grammar(self) -> None:
|
|
||||||
source = """
|
|
||||||
interface Choice {
|
|
||||||
enabled: boolean;
|
|
||||||
}
|
|
||||||
|
|
||||||
function choose(choice: Choice): number {
|
|
||||||
if (choice.enabled) {
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
class Worker {
|
|
||||||
run(choice: Choice): number {
|
|
||||||
return choose(choice);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
""".strip()
|
|
||||||
|
|
||||||
discovered = discover_typescript_functions(source)
|
|
||||||
self.assertEqual(
|
|
||||||
[("Worker.run", 13), ("choose", 5)],
|
|
||||||
[(item.qualified_name, item.line) for item in discovered],
|
|
||||||
)
|
|
||||||
projection = analyze_typescript_source(
|
|
||||||
source,
|
|
||||||
source_id="source.typescript",
|
|
||||||
owners=(TreeSitterLogicOwner("ts.symbol.choose", "choose", 5),),
|
|
||||||
)[0]
|
|
||||||
|
|
||||||
self.assertIn("choice.enabled", {node.label for node in projection.nodes})
|
|
||||||
self.assertEqual(
|
|
||||||
{"return"},
|
|
||||||
{edge.relation for edge in projection.edges if edge.relation == "return"},
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|
|
||||||
|
|
@ -1,698 +0,0 @@
|
||||||
"""Validate the maintained DocForge Markdown documentation graph."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import html
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
import stat
|
|
||||||
import sys
|
|
||||||
import tomllib
|
|
||||||
import unicodedata
|
|
||||||
from collections import deque
|
|
||||||
from collections.abc import Iterable, Sequence
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Protocol, cast
|
|
||||||
from urllib.parse import unquote, urlsplit
|
|
||||||
|
|
||||||
from jsonschema import Draft202012Validator
|
|
||||||
from jsonschema.exceptions import ValidationError
|
|
||||||
from markdown_it import MarkdownIt
|
|
||||||
from markdown_it.token import Token
|
|
||||||
|
|
||||||
REPOSITORY_ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
DEFAULT_PENDING_INVENTORY = Path("tools/docs_pending_m4_pages.txt")
|
|
||||||
COMMAND_REFERENCE = Path("docs/COMMAND_REFERENCE.md")
|
|
||||||
COMMAND_REFERENCE_H1 = "DocForge command reference"
|
|
||||||
COMMAND_REFERENCE_NOTICE = (
|
|
||||||
"> Generated from the live CLI parser and MCP registrations. Do not edit this file by hand."
|
|
||||||
)
|
|
||||||
REFERENCE_ADAPTER_SCHEMA = Path("schemas/reference-adapter.schema.json")
|
|
||||||
MAX_MARKDOWN_PAGES = 256
|
|
||||||
MAX_PAGE_BYTES = 2_000_000
|
|
||||||
MAX_TOTAL_BYTES = 20_000_000
|
|
||||||
MAX_DIAGNOSTICS = 100
|
|
||||||
|
|
||||||
REQUIRED_MILESTONE_4_PAGES = (
|
|
||||||
Path("README.md"),
|
|
||||||
Path("docs/ADAPTER_AUTHORING_GUIDE.md"),
|
|
||||||
Path("docs/AGENT_INTEGRATION.md"),
|
|
||||||
Path("docs/APPLICATION_DECISION.md"),
|
|
||||||
Path("docs/COMMAND_REFERENCE.md"),
|
|
||||||
Path("docs/COMPATIBILITY.md"),
|
|
||||||
Path("docs/CONTRACT.md"),
|
|
||||||
Path("docs/CORE_CONCEPTS_AND_AUTHORITY.md"),
|
|
||||||
Path("docs/INCREMENTAL_INDEXING.md"),
|
|
||||||
Path("docs/LEGACY_AND_NO_AST.md"),
|
|
||||||
Path("docs/MCP_CONTRACT.md"),
|
|
||||||
Path("docs/MIGRATING_FROM_V1.md"),
|
|
||||||
Path("docs/MILESTONE_4_BASELINE.md"),
|
|
||||||
Path("docs/MILESTONE_4_CLOSEOUT.md"),
|
|
||||||
Path("docs/NEW_PROJECT_QUICKSTART.md"),
|
|
||||||
Path("docs/POLICY_PRECEDENCE.md"),
|
|
||||||
Path("docs/PROJECT_DESCRIPTOR.md"),
|
|
||||||
Path("docs/PROJECT_ONBOARDING.md"),
|
|
||||||
Path("docs/RECOVERY_AND_PERFORMANCE.md"),
|
|
||||||
Path("docs/REFERENCE_ADAPTERS.md"),
|
|
||||||
Path("docs/RENDERING_AND_VISUALIZATION.md"),
|
|
||||||
Path("docs/SECURITY.md"),
|
|
||||||
Path("docs/USER_MANUAL.md"),
|
|
||||||
Path("docs/VIEWER_MANAGER.md"),
|
|
||||||
)
|
|
||||||
|
|
||||||
_HISTORICAL_RECORD = re.compile(r"^docs/MILESTONE_[0-3]_(?:BASELINE|CLOSEOUT)\.md$")
|
|
||||||
_HTML_ID = re.compile(
|
|
||||||
r"""\bid\s*=\s*(?:"([^"]+)"|'([^']+)')""",
|
|
||||||
re.IGNORECASE,
|
|
||||||
)
|
|
||||||
_REFERENCE_ADAPTER_KEY = re.compile(r"(?m)^(?:language|source_roots|compilation_database)\s*=")
|
|
||||||
_URI_SCHEME = re.compile(r"^[A-Za-z][A-Za-z0-9+.-]*:")
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True, order=True)
|
|
||||||
class Diagnostic:
|
|
||||||
"""One deterministic documentation validation failure."""
|
|
||||||
|
|
||||||
path: str
|
|
||||||
line: int
|
|
||||||
code: str
|
|
||||||
message: str
|
|
||||||
|
|
||||||
def render(self) -> str:
|
|
||||||
"""Render one bounded compiler-style diagnostic."""
|
|
||||||
|
|
||||||
return f"{self.path}:{self.line}: {self.code}: {self.message}"
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class DocumentationPolicy:
|
|
||||||
"""Repository-relative pages and temporary inventory accepted by one run."""
|
|
||||||
|
|
||||||
required_pages: tuple[Path, ...] = REQUIRED_MILESTONE_4_PAGES
|
|
||||||
pending_inventory: Path | None = DEFAULT_PENDING_INVENTORY
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class DocumentationReport:
|
|
||||||
"""Bounded deterministic result from one complete documentation check."""
|
|
||||||
|
|
||||||
diagnostics: tuple[Diagnostic, ...]
|
|
||||||
omitted_diagnostics: int
|
|
||||||
markdown_pages: int
|
|
||||||
pending_pages: int
|
|
||||||
reference_adapter_examples: int
|
|
||||||
|
|
||||||
@property
|
|
||||||
def ok(self) -> bool:
|
|
||||||
"""Return whether every documentation invariant passed."""
|
|
||||||
|
|
||||||
return not self.diagnostics and self.omitted_diagnostics == 0
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class _Page:
|
|
||||||
relative: Path
|
|
||||||
source: str
|
|
||||||
tokens: tuple[Token, ...]
|
|
||||||
anchors: frozenset[str]
|
|
||||||
|
|
||||||
|
|
||||||
class _Diagnostics:
|
|
||||||
def __init__(self) -> None:
|
|
||||||
self._items: list[Diagnostic] = []
|
|
||||||
self._omitted = 0
|
|
||||||
|
|
||||||
def add(self, path: Path | str, line: int, code: str, message: str) -> None:
|
|
||||||
relative = path.as_posix() if isinstance(path, Path) else path
|
|
||||||
item = Diagnostic(relative, max(1, line), code, message)
|
|
||||||
if len(self._items) < MAX_DIAGNOSTICS:
|
|
||||||
self._items.append(item)
|
|
||||||
else:
|
|
||||||
self._omitted += 1
|
|
||||||
|
|
||||||
def result(self) -> tuple[tuple[Diagnostic, ...], int]:
|
|
||||||
return tuple(sorted(self._items)), self._omitted
|
|
||||||
|
|
||||||
|
|
||||||
class _Validator(Protocol):
|
|
||||||
def iter_errors(self, instance: object) -> Iterable[ValidationError]: ...
|
|
||||||
|
|
||||||
|
|
||||||
def check_documentation(
|
|
||||||
repository_root: Path,
|
|
||||||
*,
|
|
||||||
policy: DocumentationPolicy | None = None,
|
|
||||||
) -> DocumentationReport:
|
|
||||||
"""Check one repository's maintained Markdown pages without modifying it."""
|
|
||||||
|
|
||||||
root = repository_root.resolve(strict=True)
|
|
||||||
if not root.is_dir():
|
|
||||||
raise ValueError(f"repository root is not a directory: {root}")
|
|
||||||
effective_policy = policy or DocumentationPolicy()
|
|
||||||
|
|
||||||
diagnostics = _Diagnostics()
|
|
||||||
parser = MarkdownIt("commonmark", {"html": True, "typographer": False})
|
|
||||||
page_paths = _discover_pages(root, diagnostics)
|
|
||||||
pages = _load_pages(root, page_paths, parser, diagnostics)
|
|
||||||
pending = _load_pending_inventory(root, effective_policy, diagnostics)
|
|
||||||
_check_required_inventory(
|
|
||||||
root,
|
|
||||||
effective_policy.required_pages,
|
|
||||||
pending,
|
|
||||||
diagnostics,
|
|
||||||
)
|
|
||||||
_check_h1s(pages, diagnostics)
|
|
||||||
link_graph = _check_links(root, pages, parser, diagnostics)
|
|
||||||
_check_reachability(pages, link_graph, diagnostics)
|
|
||||||
reference_examples = _check_reference_adapter_examples(root, pages, diagnostics)
|
|
||||||
_check_command_reference(pages, diagnostics)
|
|
||||||
items, omitted = diagnostics.result()
|
|
||||||
return DocumentationReport(
|
|
||||||
diagnostics=items,
|
|
||||||
omitted_diagnostics=omitted,
|
|
||||||
markdown_pages=len(pages),
|
|
||||||
pending_pages=len(pending),
|
|
||||||
reference_adapter_examples=reference_examples,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _discover_pages(root: Path, diagnostics: _Diagnostics) -> tuple[Path, ...]:
|
|
||||||
candidates = [Path("README.md")]
|
|
||||||
docs = root / "docs"
|
|
||||||
if not docs.is_dir():
|
|
||||||
diagnostics.add("docs", 1, "DOC001", "maintained documentation directory is missing")
|
|
||||||
return tuple(candidates)
|
|
||||||
|
|
||||||
candidates.extend(
|
|
||||||
path.relative_to(root)
|
|
||||||
for path in docs.rglob("*.md")
|
|
||||||
if not path.is_symlink() and path.is_file()
|
|
||||||
)
|
|
||||||
unique = tuple(sorted(set(candidates), key=Path.as_posix))
|
|
||||||
if len(unique) > MAX_MARKDOWN_PAGES:
|
|
||||||
diagnostics.add(
|
|
||||||
"docs",
|
|
||||||
1,
|
|
||||||
"DOC002",
|
|
||||||
f"found {len(unique)} Markdown pages; limit is {MAX_MARKDOWN_PAGES}",
|
|
||||||
)
|
|
||||||
return unique[:MAX_MARKDOWN_PAGES]
|
|
||||||
return unique
|
|
||||||
|
|
||||||
|
|
||||||
def _load_pages(
|
|
||||||
root: Path,
|
|
||||||
paths: tuple[Path, ...],
|
|
||||||
parser: MarkdownIt,
|
|
||||||
diagnostics: _Diagnostics,
|
|
||||||
) -> dict[Path, _Page]:
|
|
||||||
pages: dict[Path, _Page] = {}
|
|
||||||
total_bytes = 0
|
|
||||||
for relative in paths:
|
|
||||||
absolute = root / relative
|
|
||||||
try:
|
|
||||||
metadata = absolute.lstat()
|
|
||||||
except FileNotFoundError:
|
|
||||||
diagnostics.add(relative, 1, "DOC003", "maintained Markdown page is missing")
|
|
||||||
continue
|
|
||||||
if not stat.S_ISREG(metadata.st_mode):
|
|
||||||
diagnostics.add(relative, 1, "DOC004", "maintained Markdown page is not a regular file")
|
|
||||||
continue
|
|
||||||
if metadata.st_size > MAX_PAGE_BYTES:
|
|
||||||
diagnostics.add(
|
|
||||||
relative,
|
|
||||||
1,
|
|
||||||
"DOC005",
|
|
||||||
f"page is {metadata.st_size} bytes; limit is {MAX_PAGE_BYTES}",
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
total_bytes += metadata.st_size
|
|
||||||
if total_bytes > MAX_TOTAL_BYTES:
|
|
||||||
diagnostics.add(
|
|
||||||
relative,
|
|
||||||
1,
|
|
||||||
"DOC006",
|
|
||||||
f"total Markdown input exceeds {MAX_TOTAL_BYTES} bytes",
|
|
||||||
)
|
|
||||||
break
|
|
||||||
try:
|
|
||||||
source = absolute.read_text(encoding="utf-8")
|
|
||||||
except UnicodeDecodeError:
|
|
||||||
diagnostics.add(relative, 1, "DOC007", "page is not valid UTF-8")
|
|
||||||
continue
|
|
||||||
tokens = tuple(parser.parse(source))
|
|
||||||
pages[relative] = _Page(
|
|
||||||
relative=relative,
|
|
||||||
source=source,
|
|
||||||
tokens=tokens,
|
|
||||||
anchors=_anchors(tokens),
|
|
||||||
)
|
|
||||||
return pages
|
|
||||||
|
|
||||||
|
|
||||||
def _load_pending_inventory(
|
|
||||||
root: Path,
|
|
||||||
policy: DocumentationPolicy,
|
|
||||||
diagnostics: _Diagnostics,
|
|
||||||
) -> frozenset[Path]:
|
|
||||||
inventory = policy.pending_inventory
|
|
||||||
if inventory is None:
|
|
||||||
return frozenset()
|
|
||||||
if inventory.is_absolute():
|
|
||||||
diagnostics.add(
|
|
||||||
inventory.as_posix(),
|
|
||||||
1,
|
|
||||||
"DOC008",
|
|
||||||
"pending inventory path must be repository-relative",
|
|
||||||
)
|
|
||||||
return frozenset()
|
|
||||||
absolute = root / inventory
|
|
||||||
try:
|
|
||||||
lines = absolute.read_text(encoding="utf-8").splitlines()
|
|
||||||
except FileNotFoundError:
|
|
||||||
diagnostics.add(inventory, 1, "DOC009", "pending inventory file is missing")
|
|
||||||
return frozenset()
|
|
||||||
except UnicodeDecodeError:
|
|
||||||
diagnostics.add(inventory, 1, "DOC010", "pending inventory is not valid UTF-8")
|
|
||||||
return frozenset()
|
|
||||||
|
|
||||||
required = set(policy.required_pages)
|
|
||||||
pending: set[Path] = set()
|
|
||||||
for line_number, raw in enumerate(lines, start=1):
|
|
||||||
value = raw.strip()
|
|
||||||
if not value or value.startswith("#"):
|
|
||||||
continue
|
|
||||||
path = Path(value)
|
|
||||||
if path.is_absolute() or ".." in path.parts or path.as_posix() != value:
|
|
||||||
diagnostics.add(
|
|
||||||
inventory,
|
|
||||||
line_number,
|
|
||||||
"DOC011",
|
|
||||||
f"unsafe pending inventory path: {value!r}",
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
if path not in required:
|
|
||||||
diagnostics.add(
|
|
||||||
inventory,
|
|
||||||
line_number,
|
|
||||||
"DOC012",
|
|
||||||
f"pending page is not in the required inventory: {value}",
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
if path in pending:
|
|
||||||
diagnostics.add(
|
|
||||||
inventory,
|
|
||||||
line_number,
|
|
||||||
"DOC013",
|
|
||||||
f"duplicate pending page: {value}",
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
pending.add(path)
|
|
||||||
return frozenset(pending)
|
|
||||||
|
|
||||||
|
|
||||||
def _check_required_inventory(
|
|
||||||
root: Path,
|
|
||||||
required_pages: tuple[Path, ...],
|
|
||||||
pending: frozenset[Path],
|
|
||||||
diagnostics: _Diagnostics,
|
|
||||||
) -> None:
|
|
||||||
for relative in sorted(set(required_pages), key=Path.as_posix):
|
|
||||||
exists = (root / relative).is_file()
|
|
||||||
if relative in pending:
|
|
||||||
if exists:
|
|
||||||
diagnostics.add(
|
|
||||||
relative,
|
|
||||||
1,
|
|
||||||
"DOC014",
|
|
||||||
"page exists but remains in the pending inventory",
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
if not exists:
|
|
||||||
diagnostics.add(relative, 1, "DOC015", "required Milestone 4 page is missing")
|
|
||||||
|
|
||||||
|
|
||||||
def _check_h1s(pages: dict[Path, _Page], diagnostics: _Diagnostics) -> None:
|
|
||||||
for relative, page in pages.items():
|
|
||||||
h1_lines = [
|
|
||||||
_token_line(token)
|
|
||||||
for token in page.tokens
|
|
||||||
if token.type == "heading_open" and token.tag == "h1"
|
|
||||||
]
|
|
||||||
if len(h1_lines) != 1:
|
|
||||||
rendered = ", ".join(str(line) for line in h1_lines) or "none"
|
|
||||||
diagnostics.add(
|
|
||||||
relative,
|
|
||||||
h1_lines[1] if len(h1_lines) > 1 else 1,
|
|
||||||
"DOC016",
|
|
||||||
f"expected exactly one H1; found {len(h1_lines)} (lines: {rendered})",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _check_links(
|
|
||||||
root: Path,
|
|
||||||
pages: dict[Path, _Page],
|
|
||||||
parser: MarkdownIt,
|
|
||||||
diagnostics: _Diagnostics,
|
|
||||||
) -> dict[Path, frozenset[Path]]:
|
|
||||||
graph: dict[Path, frozenset[Path]] = {}
|
|
||||||
anchor_cache = {relative: page.anchors for relative, page in pages.items()}
|
|
||||||
for relative, page in pages.items():
|
|
||||||
destinations: set[Path] = set()
|
|
||||||
for href, line in _page_links(page.tokens):
|
|
||||||
target = _resolve_local_link(root, relative, href, line, diagnostics)
|
|
||||||
if target is None:
|
|
||||||
continue
|
|
||||||
target_relative, fragment = target
|
|
||||||
if target_relative in pages:
|
|
||||||
destinations.add(target_relative)
|
|
||||||
if not fragment:
|
|
||||||
continue
|
|
||||||
anchors = anchor_cache.get(target_relative)
|
|
||||||
if anchors is None:
|
|
||||||
anchors = _load_link_target_anchors(
|
|
||||||
root,
|
|
||||||
target_relative,
|
|
||||||
parser,
|
|
||||||
diagnostics,
|
|
||||||
)
|
|
||||||
if anchors is None:
|
|
||||||
continue
|
|
||||||
anchor_cache[target_relative] = anchors
|
|
||||||
if fragment not in anchors:
|
|
||||||
diagnostics.add(
|
|
||||||
relative,
|
|
||||||
line,
|
|
||||||
"DOC017",
|
|
||||||
f"missing anchor #{fragment} in {target_relative.as_posix()}",
|
|
||||||
)
|
|
||||||
graph[relative] = frozenset(destinations)
|
|
||||||
return graph
|
|
||||||
|
|
||||||
|
|
||||||
def _page_links(tokens: tuple[Token, ...]) -> Iterable[tuple[str, int]]:
|
|
||||||
for token in tokens:
|
|
||||||
children = token.children or []
|
|
||||||
for child in children:
|
|
||||||
attribute = "href" if child.type == "link_open" else "src"
|
|
||||||
if child.type not in {"link_open", "image"}:
|
|
||||||
continue
|
|
||||||
raw = child.attrs.get(attribute)
|
|
||||||
if isinstance(raw, str):
|
|
||||||
yield raw, _token_line(token)
|
|
||||||
|
|
||||||
|
|
||||||
def _resolve_local_link(
|
|
||||||
root: Path,
|
|
||||||
source: Path,
|
|
||||||
href: str,
|
|
||||||
line: int,
|
|
||||||
diagnostics: _Diagnostics,
|
|
||||||
) -> tuple[Path, str] | None:
|
|
||||||
if not href or href.startswith("//") or _URI_SCHEME.match(href):
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
parsed = urlsplit(href)
|
|
||||||
except ValueError as error:
|
|
||||||
diagnostics.add(source, line, "DOC018", f"invalid link {href!r}: {error}")
|
|
||||||
return None
|
|
||||||
if parsed.netloc:
|
|
||||||
return None
|
|
||||||
decoded_path = unquote(parsed.path)
|
|
||||||
fragment = unquote(parsed.fragment)
|
|
||||||
if "\x00" in decoded_path or decoded_path.startswith("/"):
|
|
||||||
diagnostics.add(source, line, "DOC019", f"local link escapes the repository: {href!r}")
|
|
||||||
return None
|
|
||||||
absolute = (root / source if not decoded_path else root / source.parent / decoded_path).resolve(
|
|
||||||
strict=False
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
target = absolute.relative_to(root)
|
|
||||||
except ValueError:
|
|
||||||
diagnostics.add(source, line, "DOC019", f"local link escapes the repository: {href!r}")
|
|
||||||
return None
|
|
||||||
if not absolute.is_file():
|
|
||||||
diagnostics.add(source, line, "DOC020", f"missing local link target: {target.as_posix()}")
|
|
||||||
return None
|
|
||||||
return target, fragment
|
|
||||||
|
|
||||||
|
|
||||||
def _load_link_target_anchors(
|
|
||||||
root: Path,
|
|
||||||
relative: Path,
|
|
||||||
parser: MarkdownIt,
|
|
||||||
diagnostics: _Diagnostics,
|
|
||||||
) -> frozenset[str] | None:
|
|
||||||
if relative.suffix.lower() not in {".md", ".markdown"}:
|
|
||||||
diagnostics.add(
|
|
||||||
relative,
|
|
||||||
1,
|
|
||||||
"DOC021",
|
|
||||||
"link uses an anchor on a non-Markdown target",
|
|
||||||
)
|
|
||||||
return None
|
|
||||||
absolute = root / relative
|
|
||||||
try:
|
|
||||||
metadata = absolute.lstat()
|
|
||||||
if not stat.S_ISREG(metadata.st_mode) or metadata.st_size > MAX_PAGE_BYTES:
|
|
||||||
raise ValueError("target is not a bounded regular Markdown file")
|
|
||||||
source = absolute.read_text(encoding="utf-8")
|
|
||||||
except (OSError, UnicodeDecodeError, ValueError) as error:
|
|
||||||
diagnostics.add(relative, 1, "DOC022", f"cannot inspect linked Markdown anchors: {error}")
|
|
||||||
return None
|
|
||||||
return _anchors(tuple(parser.parse(source)))
|
|
||||||
|
|
||||||
|
|
||||||
def _anchors(tokens: tuple[Token, ...]) -> frozenset[str]:
|
|
||||||
anchors: set[str] = set()
|
|
||||||
slug_counts: dict[str, int] = {}
|
|
||||||
for index, token in enumerate(tokens):
|
|
||||||
if token.type in {"html_block", "html_inline"}:
|
|
||||||
anchors.update(_html_ids(token.content))
|
|
||||||
if token.children:
|
|
||||||
for child in token.children:
|
|
||||||
if child.type == "html_inline":
|
|
||||||
anchors.update(_html_ids(child.content))
|
|
||||||
if token.type != "heading_open" or index + 1 >= len(tokens):
|
|
||||||
continue
|
|
||||||
inline = tokens[index + 1]
|
|
||||||
base = _heading_slug(_inline_text(inline))
|
|
||||||
duplicate = slug_counts.get(base, 0)
|
|
||||||
slug_counts[base] = duplicate + 1
|
|
||||||
anchors.add(base if duplicate == 0 else f"{base}-{duplicate}")
|
|
||||||
return frozenset(anchors)
|
|
||||||
|
|
||||||
|
|
||||||
def _inline_text(token: Token) -> str:
|
|
||||||
if not token.children:
|
|
||||||
return token.content
|
|
||||||
pieces: list[str] = []
|
|
||||||
for child in token.children:
|
|
||||||
if child.type in {"text", "code_inline"}:
|
|
||||||
pieces.append(child.content)
|
|
||||||
elif child.type in {"softbreak", "hardbreak"}:
|
|
||||||
pieces.append(" ")
|
|
||||||
elif child.type == "image":
|
|
||||||
pieces.append(child.content)
|
|
||||||
elif child.type == "html_inline":
|
|
||||||
pieces.append(re.sub(r"<[^>]*>", "", child.content))
|
|
||||||
return "".join(pieces)
|
|
||||||
|
|
||||||
|
|
||||||
def _heading_slug(value: str) -> str:
|
|
||||||
characters: list[str] = []
|
|
||||||
for character in html.unescape(value).casefold().strip():
|
|
||||||
category = unicodedata.category(character)
|
|
||||||
if category[0] in {"L", "M", "N"} or character in {"-", "_"}:
|
|
||||||
characters.append(character)
|
|
||||||
elif character.isspace():
|
|
||||||
characters.append("-")
|
|
||||||
return "".join(characters)
|
|
||||||
|
|
||||||
|
|
||||||
def _html_ids(value: str) -> set[str]:
|
|
||||||
return {first or second for first, second in _HTML_ID.findall(value)}
|
|
||||||
|
|
||||||
|
|
||||||
def _check_reachability(
|
|
||||||
pages: dict[Path, _Page],
|
|
||||||
graph: dict[Path, frozenset[Path]],
|
|
||||||
diagnostics: _Diagnostics,
|
|
||||||
) -> None:
|
|
||||||
root_page = Path("README.md")
|
|
||||||
if root_page not in pages:
|
|
||||||
return
|
|
||||||
reached = {root_page}
|
|
||||||
queue = deque([root_page])
|
|
||||||
while queue:
|
|
||||||
current = queue.popleft()
|
|
||||||
for target in sorted(graph.get(current, frozenset()), key=Path.as_posix):
|
|
||||||
if target not in reached:
|
|
||||||
reached.add(target)
|
|
||||||
queue.append(target)
|
|
||||||
for relative in sorted(set(pages) - reached, key=Path.as_posix):
|
|
||||||
if _HISTORICAL_RECORD.fullmatch(relative.as_posix()):
|
|
||||||
continue
|
|
||||||
diagnostics.add(
|
|
||||||
relative,
|
|
||||||
1,
|
|
||||||
"DOC023",
|
|
||||||
"maintained page is not reachable from README.md",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _check_reference_adapter_examples(
|
|
||||||
root: Path,
|
|
||||||
pages: dict[Path, _Page],
|
|
||||||
diagnostics: _Diagnostics,
|
|
||||||
) -> int:
|
|
||||||
schema_path = root / REFERENCE_ADAPTER_SCHEMA
|
|
||||||
try:
|
|
||||||
schema_document = cast(
|
|
||||||
dict[str, object],
|
|
||||||
json.loads(schema_path.read_text(encoding="utf-8")),
|
|
||||||
)
|
|
||||||
Draft202012Validator.check_schema(schema_document)
|
|
||||||
except (OSError, UnicodeDecodeError, json.JSONDecodeError, TypeError) as error:
|
|
||||||
diagnostics.add(
|
|
||||||
REFERENCE_ADAPTER_SCHEMA,
|
|
||||||
1,
|
|
||||||
"DOC024",
|
|
||||||
f"cannot load reference-adapter schema: {error}",
|
|
||||||
)
|
|
||||||
return 0
|
|
||||||
validator = cast(_Validator, Draft202012Validator(schema_document))
|
|
||||||
count = 0
|
|
||||||
for relative, page in pages.items():
|
|
||||||
previous_inline: Token | None = None
|
|
||||||
for token in page.tokens:
|
|
||||||
if token.type == "inline":
|
|
||||||
previous_inline = token
|
|
||||||
continue
|
|
||||||
if token.type != "fence" or token.info.split(maxsplit=1)[:1] != ["toml"]:
|
|
||||||
continue
|
|
||||||
if not _is_reference_adapter_example(token, previous_inline):
|
|
||||||
continue
|
|
||||||
count += 1
|
|
||||||
line = _token_line(token)
|
|
||||||
try:
|
|
||||||
document = tomllib.loads(token.content)
|
|
||||||
except tomllib.TOMLDecodeError as error:
|
|
||||||
diagnostics.add(
|
|
||||||
relative,
|
|
||||||
line,
|
|
||||||
"DOC025",
|
|
||||||
f"invalid reference-adapter TOML example: {error}",
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
errors = sorted(
|
|
||||||
validator.iter_errors(document),
|
|
||||||
key=lambda error: tuple(str(part) for part in error.absolute_path),
|
|
||||||
)
|
|
||||||
for error in errors:
|
|
||||||
location = ".".join(str(part) for part in error.absolute_path) or "<root>"
|
|
||||||
diagnostics.add(
|
|
||||||
relative,
|
|
||||||
line,
|
|
||||||
"DOC026",
|
|
||||||
f"reference-adapter example {location}: {error.message}",
|
|
||||||
)
|
|
||||||
return count
|
|
||||||
|
|
||||||
|
|
||||||
def _is_reference_adapter_example(token: Token, previous_inline: Token | None) -> bool:
|
|
||||||
info = token.info.casefold()
|
|
||||||
if "reference-adapter" in info or "reference_adapter" in info:
|
|
||||||
return True
|
|
||||||
if previous_inline is not None and previous_inline.map is not None and token.map is not None:
|
|
||||||
adjacent = token.map[0] - previous_inline.map[1] <= 1
|
|
||||||
context = previous_inline.content.casefold()
|
|
||||||
if adjacent and (
|
|
||||||
"reference-adapter.toml" in context or "reference adapter configuration" in context
|
|
||||||
):
|
|
||||||
return True
|
|
||||||
keys = set(_REFERENCE_ADAPTER_KEY.findall(token.content))
|
|
||||||
return {"language", "source_roots"}.issubset(keys)
|
|
||||||
|
|
||||||
|
|
||||||
def _check_command_reference(
|
|
||||||
pages: dict[Path, _Page],
|
|
||||||
diagnostics: _Diagnostics,
|
|
||||||
) -> None:
|
|
||||||
page = pages.get(COMMAND_REFERENCE)
|
|
||||||
if page is None:
|
|
||||||
return
|
|
||||||
lines = page.source.splitlines()
|
|
||||||
if not lines or lines[0] != f"# {COMMAND_REFERENCE_H1}":
|
|
||||||
diagnostics.add(
|
|
||||||
COMMAND_REFERENCE,
|
|
||||||
1,
|
|
||||||
"DOC027",
|
|
||||||
f"generated reference must begin with '# {COMMAND_REFERENCE_H1}'",
|
|
||||||
)
|
|
||||||
if COMMAND_REFERENCE_NOTICE not in lines[:5]:
|
|
||||||
diagnostics.add(
|
|
||||||
COMMAND_REFERENCE,
|
|
||||||
2,
|
|
||||||
"DOC028",
|
|
||||||
"generated reference notice is missing or changed",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _token_line(token: Token) -> int:
|
|
||||||
return token.map[0] + 1 if token.map else 1
|
|
||||||
|
|
||||||
|
|
||||||
def _parser() -> argparse.ArgumentParser:
|
|
||||||
parser = argparse.ArgumentParser(
|
|
||||||
description="Check deterministic DocForge documentation invariants."
|
|
||||||
)
|
|
||||||
parser.add_argument("--repository-root", type=Path, default=REPOSITORY_ROOT)
|
|
||||||
parser.add_argument(
|
|
||||||
"--pending-inventory",
|
|
||||||
type=Path,
|
|
||||||
default=DEFAULT_PENDING_INVENTORY,
|
|
||||||
help="repository-relative list of required pages not authored yet",
|
|
||||||
)
|
|
||||||
return parser
|
|
||||||
|
|
||||||
|
|
||||||
def main(arguments: Sequence[str] | None = None) -> int:
|
|
||||||
parsed = _parser().parse_args(arguments)
|
|
||||||
try:
|
|
||||||
report = check_documentation(
|
|
||||||
parsed.repository_root,
|
|
||||||
policy=DocumentationPolicy(pending_inventory=parsed.pending_inventory),
|
|
||||||
)
|
|
||||||
except Exception as error:
|
|
||||||
print(f"documentation check failed: {error}", file=sys.stderr)
|
|
||||||
return 2
|
|
||||||
if not report.ok:
|
|
||||||
for diagnostic in report.diagnostics:
|
|
||||||
print(diagnostic.render(), file=sys.stderr)
|
|
||||||
if report.omitted_diagnostics:
|
|
||||||
print(
|
|
||||||
f"... {report.omitted_diagnostics} additional diagnostics omitted "
|
|
||||||
f"(limit {MAX_DIAGNOSTICS})",
|
|
||||||
file=sys.stderr,
|
|
||||||
)
|
|
||||||
return 1
|
|
||||||
print(
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"markdown_pages": report.markdown_pages,
|
|
||||||
"pending_pages": report.pending_pages,
|
|
||||||
"reference_adapter_examples": report.reference_adapter_examples,
|
|
||||||
"status": "ok",
|
|
||||||
},
|
|
||||||
sort_keys=True,
|
|
||||||
separators=(",", ":"),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
# Milestone 4 required-page inventory is complete.
|
|
||||||
|
|
@ -1,533 +0,0 @@
|
||||||
"""Generate or verify the deterministic DocForge CLI and MCP command reference."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import asyncio
|
|
||||||
import contextlib
|
|
||||||
import ctypes
|
|
||||||
import errno
|
|
||||||
import fcntl
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import secrets
|
|
||||||
import shutil
|
|
||||||
import stat
|
|
||||||
import sys
|
|
||||||
import tempfile
|
|
||||||
from collections.abc import Sequence
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Literal, Protocol, cast
|
|
||||||
|
|
||||||
from mcp.shared.memory import create_connected_server_and_client_session
|
|
||||||
|
|
||||||
from docforge.command_reference import (
|
|
||||||
CliCommandReference,
|
|
||||||
McpToolReference,
|
|
||||||
cli_command_references,
|
|
||||||
mcp_tool_references,
|
|
||||||
render_command_reference_markdown,
|
|
||||||
)
|
|
||||||
from docforge.index import ProjectIndex
|
|
||||||
from docforge.mcp_server import ALL_TOOLS, APPLICATION_TOOLS, create_server
|
|
||||||
from docforge.project import Project
|
|
||||||
|
|
||||||
REPOSITORY_ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
DEFAULT_PROJECT_ROOT = REPOSITORY_ROOT / "tests" / "fixtures" / "alpha"
|
|
||||||
EXPECTED_CLI_ROWS = 28
|
|
||||||
EXPECTED_MCP_ROWS = 36
|
|
||||||
MAX_REFERENCE_BYTES = 5_000_000
|
|
||||||
RENAME_EXCHANGE = 2
|
|
||||||
|
|
||||||
|
|
||||||
class CommandReferenceToolError(RuntimeError):
|
|
||||||
"""One safe repository-tool validation or publication failure."""
|
|
||||||
|
|
||||||
|
|
||||||
class CommandReferenceDrift(CommandReferenceToolError):
|
|
||||||
"""The checked output does not match current registered command metadata."""
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class _FileIdentity:
|
|
||||||
device: int
|
|
||||||
inode: int
|
|
||||||
mode: int
|
|
||||||
owner: int
|
|
||||||
group: int
|
|
||||||
size: int
|
|
||||||
modified_ns: int
|
|
||||||
changed_ns: int
|
|
||||||
|
|
||||||
|
|
||||||
class _RenameAt2(Protocol):
|
|
||||||
argtypes: list[object]
|
|
||||||
restype: object
|
|
||||||
|
|
||||||
def __call__(
|
|
||||||
self,
|
|
||||||
old_directory_fd: int,
|
|
||||||
old_name: bytes,
|
|
||||||
new_directory_fd: int,
|
|
||||||
new_name: bytes,
|
|
||||||
flags: int,
|
|
||||||
/,
|
|
||||||
) -> int: ...
|
|
||||||
|
|
||||||
|
|
||||||
async def collect_command_reference_rows(
|
|
||||||
project_root: Path,
|
|
||||||
*,
|
|
||||||
proposal_writer: str,
|
|
||||||
canonical_applier: str,
|
|
||||||
) -> tuple[tuple[CliCommandReference, ...], tuple[McpToolReference, ...]]:
|
|
||||||
"""Collect the real CLI parser and full project-bound MCP registration surface."""
|
|
||||||
|
|
||||||
source = _safe_project_root(project_root)
|
|
||||||
with tempfile.TemporaryDirectory(prefix="docforge-command-reference-") as directory:
|
|
||||||
copied = Path(directory) / "project"
|
|
||||||
shutil.copytree(source, copied, symlinks=True)
|
|
||||||
ProjectIndex(Project.open(copied)).build()
|
|
||||||
server = create_server(
|
|
||||||
copied,
|
|
||||||
proposal_writer,
|
|
||||||
canonical_applier_id=canonical_applier,
|
|
||||||
)
|
|
||||||
async with create_connected_server_and_client_session(
|
|
||||||
server,
|
|
||||||
raise_exceptions=True,
|
|
||||||
) as session:
|
|
||||||
tools = tuple((await session.list_tools()).tools)
|
|
||||||
|
|
||||||
cli_rows = cli_command_references()
|
|
||||||
expected_mcp_names = (*ALL_TOOLS, *APPLICATION_TOOLS)
|
|
||||||
mcp_rows = mcp_tool_references(tools, expected_names=expected_mcp_names)
|
|
||||||
if len(cli_rows) != EXPECTED_CLI_ROWS:
|
|
||||||
raise CommandReferenceToolError(
|
|
||||||
f"Expected {EXPECTED_CLI_ROWS} CLI rows, found {len(cli_rows)}"
|
|
||||||
)
|
|
||||||
if len(mcp_rows) != EXPECTED_MCP_ROWS:
|
|
||||||
raise CommandReferenceToolError(
|
|
||||||
f"Expected {EXPECTED_MCP_ROWS} MCP rows, found {len(mcp_rows)}"
|
|
||||||
)
|
|
||||||
return cli_rows, mcp_rows
|
|
||||||
|
|
||||||
|
|
||||||
def generate_command_reference_bytes(
|
|
||||||
project_root: Path,
|
|
||||||
*,
|
|
||||||
proposal_writer: str = "alpha-editor",
|
|
||||||
canonical_applier: str = "alpha-editor",
|
|
||||||
) -> bytes:
|
|
||||||
"""Render one stable Markdown document from an isolated project copy."""
|
|
||||||
|
|
||||||
cli_rows, mcp_rows = asyncio.run(
|
|
||||||
collect_command_reference_rows(
|
|
||||||
project_root,
|
|
||||||
proposal_writer=proposal_writer,
|
|
||||||
canonical_applier=canonical_applier,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
rendered = render_command_reference_markdown(cli_rows, mcp_rows).encode("utf-8")
|
|
||||||
if len(rendered) > MAX_REFERENCE_BYTES:
|
|
||||||
raise CommandReferenceToolError("Generated command reference exceeds its byte limit")
|
|
||||||
return rendered
|
|
||||||
|
|
||||||
|
|
||||||
def publish_or_check_command_reference(
|
|
||||||
content: bytes,
|
|
||||||
*,
|
|
||||||
repository_root: Path,
|
|
||||||
project_root: Path,
|
|
||||||
output: Path,
|
|
||||||
check: bool,
|
|
||||||
) -> Literal["current", "unchanged", "written"]:
|
|
||||||
"""Safely check or atomically publish one repository-confined Markdown file."""
|
|
||||||
|
|
||||||
if len(content) > MAX_REFERENCE_BYTES:
|
|
||||||
raise CommandReferenceToolError("Command reference exceeds its byte limit")
|
|
||||||
root = _safe_repository_root(repository_root)
|
|
||||||
source = _safe_project_root(project_root)
|
|
||||||
relative = _safe_output_relative(root, source, output)
|
|
||||||
parent_parts = relative.parts[:-1]
|
|
||||||
target_name = relative.name
|
|
||||||
parent_fd = _open_relative_directory(root, parent_parts)
|
|
||||||
try:
|
|
||||||
fcntl.flock(parent_fd, fcntl.LOCK_EX)
|
|
||||||
try:
|
|
||||||
initial = _file_identity(parent_fd, target_name)
|
|
||||||
existing = (
|
|
||||||
_read_regular_file(parent_fd, target_name, limit=MAX_REFERENCE_BYTES)
|
|
||||||
if initial is not None
|
|
||||||
else None
|
|
||||||
)
|
|
||||||
if _file_identity(parent_fd, target_name) != initial:
|
|
||||||
raise CommandReferenceToolError("Output target changed during inspection")
|
|
||||||
if check:
|
|
||||||
if existing != content:
|
|
||||||
raise CommandReferenceDrift(
|
|
||||||
f"Command reference is missing or stale: {relative.as_posix()}"
|
|
||||||
)
|
|
||||||
return "current"
|
|
||||||
if existing == content:
|
|
||||||
return "unchanged"
|
|
||||||
_atomic_replace(
|
|
||||||
parent_fd,
|
|
||||||
target_name,
|
|
||||||
content,
|
|
||||||
expected=initial,
|
|
||||||
expected_content=existing,
|
|
||||||
)
|
|
||||||
return "written"
|
|
||||||
finally:
|
|
||||||
fcntl.flock(parent_fd, fcntl.LOCK_UN)
|
|
||||||
finally:
|
|
||||||
os.close(parent_fd)
|
|
||||||
|
|
||||||
|
|
||||||
def _safe_repository_root(path: Path) -> Path:
|
|
||||||
if path.is_symlink():
|
|
||||||
raise CommandReferenceToolError("Repository root must not be a symbolic link")
|
|
||||||
try:
|
|
||||||
resolved = path.resolve(strict=True)
|
|
||||||
except OSError as error:
|
|
||||||
raise CommandReferenceToolError("Repository root does not exist") from error
|
|
||||||
if not resolved.is_dir():
|
|
||||||
raise CommandReferenceToolError("Repository root must be a directory")
|
|
||||||
return resolved
|
|
||||||
|
|
||||||
|
|
||||||
def _safe_project_root(path: Path) -> Path:
|
|
||||||
if path.is_symlink():
|
|
||||||
raise CommandReferenceToolError("Project fixture root must not be a symbolic link")
|
|
||||||
try:
|
|
||||||
resolved = path.resolve(strict=True)
|
|
||||||
except OSError as error:
|
|
||||||
raise CommandReferenceToolError("Project fixture root does not exist") from error
|
|
||||||
if not resolved.is_dir():
|
|
||||||
raise CommandReferenceToolError("Project fixture root must be a directory")
|
|
||||||
return resolved
|
|
||||||
|
|
||||||
|
|
||||||
def _safe_output_relative(root: Path, project_root: Path, output: Path) -> Path:
|
|
||||||
candidate = output if output.is_absolute() else root / output
|
|
||||||
absolute = Path(os.path.abspath(candidate))
|
|
||||||
if absolute == root or not absolute.is_relative_to(root):
|
|
||||||
raise CommandReferenceToolError("Output must remain inside the repository root")
|
|
||||||
if absolute == project_root or absolute.is_relative_to(project_root):
|
|
||||||
raise CommandReferenceToolError("Output must not modify the source project fixture")
|
|
||||||
relative = absolute.relative_to(root)
|
|
||||||
if relative.suffix != ".md":
|
|
||||||
raise CommandReferenceToolError("Output must be one Markdown file")
|
|
||||||
if not relative.name or len(relative.parts) < 2:
|
|
||||||
raise CommandReferenceToolError("Output must be below an existing repository directory")
|
|
||||||
return relative
|
|
||||||
|
|
||||||
|
|
||||||
def _open_relative_directory(root: Path, parts: tuple[str, ...]) -> int:
|
|
||||||
flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW
|
|
||||||
descriptor = os.open(root, flags)
|
|
||||||
try:
|
|
||||||
for part in parts:
|
|
||||||
if part in {"", ".", ".."}:
|
|
||||||
raise CommandReferenceToolError("Output directory is unsafe")
|
|
||||||
try:
|
|
||||||
next_descriptor = os.open(part, flags, dir_fd=descriptor)
|
|
||||||
except OSError as error:
|
|
||||||
raise CommandReferenceToolError(
|
|
||||||
"Output parent must be an existing non-symlink directory"
|
|
||||||
) from error
|
|
||||||
os.close(descriptor)
|
|
||||||
descriptor = next_descriptor
|
|
||||||
return descriptor
|
|
||||||
except Exception:
|
|
||||||
os.close(descriptor)
|
|
||||||
raise
|
|
||||||
|
|
||||||
|
|
||||||
def _file_identity(directory_fd: int, name: str) -> _FileIdentity | None:
|
|
||||||
try:
|
|
||||||
status = os.stat(name, dir_fd=directory_fd, follow_symlinks=False)
|
|
||||||
except FileNotFoundError:
|
|
||||||
return None
|
|
||||||
if not stat.S_ISREG(status.st_mode):
|
|
||||||
raise CommandReferenceToolError("Output target must be a regular file")
|
|
||||||
return _FileIdentity(
|
|
||||||
device=status.st_dev,
|
|
||||||
inode=status.st_ino,
|
|
||||||
mode=status.st_mode,
|
|
||||||
owner=status.st_uid,
|
|
||||||
group=status.st_gid,
|
|
||||||
size=status.st_size,
|
|
||||||
modified_ns=status.st_mtime_ns,
|
|
||||||
changed_ns=status.st_ctime_ns,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _read_regular_file(directory_fd: int, name: str, *, limit: int) -> bytes:
|
|
||||||
try:
|
|
||||||
descriptor = os.open(name, os.O_RDONLY | os.O_NOFOLLOW, dir_fd=directory_fd)
|
|
||||||
except OSError as error:
|
|
||||||
raise CommandReferenceToolError("Output target could not be opened safely") from error
|
|
||||||
try:
|
|
||||||
before = os.fstat(descriptor)
|
|
||||||
if not stat.S_ISREG(before.st_mode) or before.st_size > limit:
|
|
||||||
raise CommandReferenceToolError("Output target is not a bounded regular file")
|
|
||||||
chunks: list[bytes] = []
|
|
||||||
remaining = limit + 1
|
|
||||||
while remaining:
|
|
||||||
chunk = os.read(descriptor, min(remaining, 64 * 1024))
|
|
||||||
if not chunk:
|
|
||||||
break
|
|
||||||
chunks.append(chunk)
|
|
||||||
remaining -= len(chunk)
|
|
||||||
content = b"".join(chunks)
|
|
||||||
after = os.fstat(descriptor)
|
|
||||||
if len(content) > limit or _identity_from_stat(before) != _identity_from_stat(after):
|
|
||||||
raise CommandReferenceToolError("Output target changed during inspection")
|
|
||||||
return content
|
|
||||||
finally:
|
|
||||||
os.close(descriptor)
|
|
||||||
|
|
||||||
|
|
||||||
def _identity_from_stat(status: os.stat_result) -> _FileIdentity:
|
|
||||||
return _FileIdentity(
|
|
||||||
device=status.st_dev,
|
|
||||||
inode=status.st_ino,
|
|
||||||
mode=status.st_mode,
|
|
||||||
owner=status.st_uid,
|
|
||||||
group=status.st_gid,
|
|
||||||
size=status.st_size,
|
|
||||||
modified_ns=status.st_mtime_ns,
|
|
||||||
changed_ns=status.st_ctime_ns,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _atomic_replace(
|
|
||||||
directory_fd: int,
|
|
||||||
name: str,
|
|
||||||
content: bytes,
|
|
||||||
*,
|
|
||||||
expected: _FileIdentity | None,
|
|
||||||
expected_content: bytes | None,
|
|
||||||
) -> None:
|
|
||||||
temporary_name = f".{name}.docforge-command-reference-{secrets.token_hex(12)}"
|
|
||||||
flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW
|
|
||||||
try:
|
|
||||||
descriptor = os.open(temporary_name, flags, 0o644, dir_fd=directory_fd)
|
|
||||||
except OSError as error:
|
|
||||||
raise CommandReferenceToolError("Could not create atomic output temporary") from error
|
|
||||||
try:
|
|
||||||
view = memoryview(content)
|
|
||||||
while view:
|
|
||||||
written = os.write(descriptor, view)
|
|
||||||
if written <= 0:
|
|
||||||
raise CommandReferenceToolError("Could not write command reference output")
|
|
||||||
view = view[written:]
|
|
||||||
os.fsync(descriptor)
|
|
||||||
except Exception:
|
|
||||||
os.close(descriptor)
|
|
||||||
_unlink_at(directory_fd, temporary_name)
|
|
||||||
raise
|
|
||||||
else:
|
|
||||||
os.close(descriptor)
|
|
||||||
temporary_contains_only_staged_content = True
|
|
||||||
try:
|
|
||||||
staged = _file_identity(directory_fd, temporary_name)
|
|
||||||
if staged is None:
|
|
||||||
raise CommandReferenceToolError("Atomic output temporary disappeared")
|
|
||||||
if _file_identity(directory_fd, name) != expected:
|
|
||||||
raise CommandReferenceToolError("Output target changed before atomic publication")
|
|
||||||
if expected is None:
|
|
||||||
_link_no_replace(directory_fd, temporary_name, name)
|
|
||||||
_unlink_at(directory_fd, temporary_name)
|
|
||||||
os.fsync(directory_fd)
|
|
||||||
return
|
|
||||||
if expected_content is None:
|
|
||||||
raise CommandReferenceToolError("Expected output content was not captured")
|
|
||||||
|
|
||||||
_rename_exchange(directory_fd, temporary_name, name)
|
|
||||||
temporary_contains_only_staged_content = False
|
|
||||||
displaced = _file_identity(directory_fd, temporary_name)
|
|
||||||
published = _file_identity(directory_fd, name)
|
|
||||||
displaced_content = _read_regular_file(
|
|
||||||
directory_fd,
|
|
||||||
temporary_name,
|
|
||||||
limit=MAX_REFERENCE_BYTES,
|
|
||||||
)
|
|
||||||
if (
|
|
||||||
displaced is None
|
|
||||||
or published is None
|
|
||||||
or not _same_identity_after_rename(displaced, expected)
|
|
||||||
or displaced_content != expected_content
|
|
||||||
or not _same_identity_after_rename(published, staged)
|
|
||||||
):
|
|
||||||
try:
|
|
||||||
_rename_exchange(directory_fd, temporary_name, name)
|
|
||||||
except Exception as error:
|
|
||||||
raise CommandReferenceToolError(
|
|
||||||
"Output target raced publication; displaced data was retained "
|
|
||||||
f"in {temporary_name}"
|
|
||||||
) from error
|
|
||||||
temporary_contains_only_staged_content = True
|
|
||||||
if not _same_identity_after_rename(
|
|
||||||
_file_identity(directory_fd, name),
|
|
||||||
displaced,
|
|
||||||
) or not _same_identity_after_rename(
|
|
||||||
_file_identity(directory_fd, temporary_name),
|
|
||||||
staged,
|
|
||||||
):
|
|
||||||
raise CommandReferenceToolError(
|
|
||||||
"Output target raced publication and could not be safely restored"
|
|
||||||
)
|
|
||||||
raise CommandReferenceToolError("Output target changed during atomic publication")
|
|
||||||
|
|
||||||
_unlink_at(directory_fd, temporary_name)
|
|
||||||
os.fsync(directory_fd)
|
|
||||||
except Exception:
|
|
||||||
if temporary_contains_only_staged_content:
|
|
||||||
_unlink_at(directory_fd, temporary_name)
|
|
||||||
raise
|
|
||||||
|
|
||||||
|
|
||||||
def _same_identity_after_rename(
|
|
||||||
actual: _FileIdentity | None,
|
|
||||||
expected: _FileIdentity | None,
|
|
||||||
) -> bool:
|
|
||||||
if actual is None or expected is None:
|
|
||||||
return False
|
|
||||||
return (
|
|
||||||
actual.device,
|
|
||||||
actual.inode,
|
|
||||||
actual.mode,
|
|
||||||
actual.owner,
|
|
||||||
actual.group,
|
|
||||||
actual.size,
|
|
||||||
actual.modified_ns,
|
|
||||||
) == (
|
|
||||||
expected.device,
|
|
||||||
expected.inode,
|
|
||||||
expected.mode,
|
|
||||||
expected.owner,
|
|
||||||
expected.group,
|
|
||||||
expected.size,
|
|
||||||
expected.modified_ns,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _link_no_replace(directory_fd: int, source: str, target: str) -> None:
|
|
||||||
try:
|
|
||||||
os.link(
|
|
||||||
source,
|
|
||||||
target,
|
|
||||||
src_dir_fd=directory_fd,
|
|
||||||
dst_dir_fd=directory_fd,
|
|
||||||
follow_symlinks=False,
|
|
||||||
)
|
|
||||||
except FileExistsError as error:
|
|
||||||
raise CommandReferenceToolError(
|
|
||||||
"Output target appeared during atomic publication"
|
|
||||||
) from error
|
|
||||||
except OSError as error:
|
|
||||||
raise CommandReferenceToolError("Could not publish atomic output") from error
|
|
||||||
|
|
||||||
|
|
||||||
def _rename_exchange(directory_fd: int, first: str, second: str) -> None:
|
|
||||||
rename_at2 = _load_rename_at2()
|
|
||||||
ctypes.set_errno(0)
|
|
||||||
result = rename_at2(
|
|
||||||
directory_fd,
|
|
||||||
os.fsencode(first),
|
|
||||||
directory_fd,
|
|
||||||
os.fsencode(second),
|
|
||||||
RENAME_EXCHANGE,
|
|
||||||
)
|
|
||||||
if result == 0:
|
|
||||||
return
|
|
||||||
error_number = ctypes.get_errno()
|
|
||||||
if error_number in {errno.ENOSYS, errno.EINVAL, errno.EOPNOTSUPP}:
|
|
||||||
raise CommandReferenceToolError(
|
|
||||||
"Atomic exchange publication is unavailable on this filesystem"
|
|
||||||
)
|
|
||||||
raise CommandReferenceToolError("Could not exchange atomic output") from OSError(
|
|
||||||
error_number,
|
|
||||||
os.strerror(error_number),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _load_rename_at2() -> _RenameAt2:
|
|
||||||
library = ctypes.CDLL(None, use_errno=True)
|
|
||||||
try:
|
|
||||||
rename_at2 = cast(_RenameAt2, library.renameat2)
|
|
||||||
except AttributeError as error:
|
|
||||||
raise CommandReferenceToolError(
|
|
||||||
"Atomic exchange publication is unavailable on this platform"
|
|
||||||
) from error
|
|
||||||
rename_at2.argtypes = [
|
|
||||||
ctypes.c_int,
|
|
||||||
ctypes.c_char_p,
|
|
||||||
ctypes.c_int,
|
|
||||||
ctypes.c_char_p,
|
|
||||||
ctypes.c_uint,
|
|
||||||
]
|
|
||||||
rename_at2.restype = ctypes.c_int
|
|
||||||
return rename_at2
|
|
||||||
|
|
||||||
|
|
||||||
def _unlink_at(directory_fd: int, name: str) -> None:
|
|
||||||
with contextlib.suppress(FileNotFoundError):
|
|
||||||
os.unlink(name, dir_fd=directory_fd)
|
|
||||||
|
|
||||||
|
|
||||||
def _parser() -> argparse.ArgumentParser:
|
|
||||||
parser = argparse.ArgumentParser(
|
|
||||||
description="Generate or check deterministic DocForge command-reference Markdown."
|
|
||||||
)
|
|
||||||
parser.add_argument("--output", type=Path, required=True)
|
|
||||||
parser.add_argument("--check", action="store_true")
|
|
||||||
parser.add_argument("--repository-root", type=Path, default=REPOSITORY_ROOT)
|
|
||||||
parser.add_argument("--project-root", type=Path, default=DEFAULT_PROJECT_ROOT)
|
|
||||||
parser.add_argument("--proposal-writer", default="alpha-editor")
|
|
||||||
parser.add_argument("--canonical-applier", default="alpha-editor")
|
|
||||||
return parser
|
|
||||||
|
|
||||||
|
|
||||||
def main(arguments: Sequence[str] | None = None) -> int:
|
|
||||||
parsed = _parser().parse_args(arguments)
|
|
||||||
try:
|
|
||||||
content = generate_command_reference_bytes(
|
|
||||||
parsed.project_root,
|
|
||||||
proposal_writer=parsed.proposal_writer,
|
|
||||||
canonical_applier=parsed.canonical_applier,
|
|
||||||
)
|
|
||||||
state = publish_or_check_command_reference(
|
|
||||||
content,
|
|
||||||
repository_root=parsed.repository_root,
|
|
||||||
project_root=parsed.project_root,
|
|
||||||
output=parsed.output,
|
|
||||||
check=parsed.check,
|
|
||||||
)
|
|
||||||
except CommandReferenceDrift as error:
|
|
||||||
print(str(error), file=sys.stderr)
|
|
||||||
return 1
|
|
||||||
except Exception as error:
|
|
||||||
print(f"command-reference generation failed: {error}", file=sys.stderr)
|
|
||||||
return 2
|
|
||||||
print(
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"status": state,
|
|
||||||
"output": str(parsed.output),
|
|
||||||
"cli_rows": EXPECTED_CLI_ROWS,
|
|
||||||
"mcp_rows": EXPECTED_MCP_ROWS,
|
|
||||||
},
|
|
||||||
sort_keys=True,
|
|
||||||
separators=(",", ":"),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
|
|
@ -1,430 +0,0 @@
|
||||||
#!/usr/bin/env python3
|
|
||||||
"""Run the offline Milestone 4 fresh-wheel adoption proof."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import shutil
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
import tempfile
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import cast
|
|
||||||
|
|
||||||
PROOF_SCHEMA_VERSION = 1
|
|
||||||
COMMAND_TIMEOUT_SECONDS = 120
|
|
||||||
VALIDATION_TIMEOUT_SECONDS = 45
|
|
||||||
OPTIONAL_FRONTEND_MODULES = (
|
|
||||||
"tree_sitter",
|
|
||||||
"tree_sitter_cpp",
|
|
||||||
"tree_sitter_javascript",
|
|
||||||
"tree_sitter_typescript",
|
|
||||||
)
|
|
||||||
|
|
||||||
_VALIDATION_PROGRAM = r"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import importlib.metadata
|
|
||||||
import importlib.util
|
|
||||||
import json
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from docforge.errors import DocForgeError
|
|
||||||
from docforge.index import ProjectIndex
|
|
||||||
from docforge.reference_mcp import create_reference_project
|
|
||||||
from mcp import ClientSession, StdioServerParameters
|
|
||||||
from mcp.client.stdio import stdio_client
|
|
||||||
|
|
||||||
|
|
||||||
def write_config(
|
|
||||||
root: Path,
|
|
||||||
*,
|
|
||||||
language: str,
|
|
||||||
source_roots: tuple[str, ...],
|
|
||||||
compilation_database: str | None = None,
|
|
||||||
) -> None:
|
|
||||||
config = root / ".docforge" / "reference-adapter.toml"
|
|
||||||
config.parent.mkdir(parents=True)
|
|
||||||
roots = ", ".join(json.dumps(item) for item in source_roots)
|
|
||||||
lines = [
|
|
||||||
"schema_version = 1",
|
|
||||||
'project_id = "fresh-wheel-adoption"',
|
|
||||||
'title = "Fresh wheel adoption proof"',
|
|
||||||
f"language = {json.dumps(language)}",
|
|
||||||
f"source_roots = [{roots}]",
|
|
||||||
]
|
|
||||||
if compilation_database is not None:
|
|
||||||
lines.append(f"compilation_database = {json.dumps(compilation_database)}")
|
|
||||||
config.write_text("\n".join((*lines, "")), encoding="utf-8")
|
|
||||||
|
|
||||||
|
|
||||||
async def retrieve(python: Path, root: Path) -> dict[str, object]:
|
|
||||||
parameters = StdioServerParameters(
|
|
||||||
command=str(python),
|
|
||||||
args=[
|
|
||||||
"-I",
|
|
||||||
"-m",
|
|
||||||
"docforge.reference_mcp",
|
|
||||||
"--project-root",
|
|
||||||
str(root),
|
|
||||||
"--capability-mode",
|
|
||||||
"read",
|
|
||||||
],
|
|
||||||
env={},
|
|
||||||
)
|
|
||||||
async with (
|
|
||||||
stdio_client(parameters) as streams,
|
|
||||||
ClientSession(*streams) as session,
|
|
||||||
):
|
|
||||||
await session.initialize()
|
|
||||||
tools = tuple(tool.name for tool in (await session.list_tools()).tools)
|
|
||||||
bootstrap = await session.call_tool("docforge_bootstrap", {})
|
|
||||||
search = await session.call_tool(
|
|
||||||
"docforge_search",
|
|
||||||
{"query": "Service", "limit": 5},
|
|
||||||
)
|
|
||||||
if search.isError or not search.structuredContent:
|
|
||||||
raise RuntimeError("fresh-wheel reference MCP search failed")
|
|
||||||
results = search.structuredContent.get("results")
|
|
||||||
if not isinstance(results, list) or not results:
|
|
||||||
raise RuntimeError("fresh-wheel reference MCP search returned no evidence")
|
|
||||||
first = results[0]
|
|
||||||
if not isinstance(first, dict) or not isinstance(first.get("node_id"), str):
|
|
||||||
raise RuntimeError("fresh-wheel reference MCP search returned invalid evidence")
|
|
||||||
node_id = first["node_id"]
|
|
||||||
node = await session.call_tool("docforge_get_node", {"node_id": node_id})
|
|
||||||
if bootstrap.isError or not bootstrap.structuredContent:
|
|
||||||
raise RuntimeError("fresh-wheel reference MCP bootstrap failed")
|
|
||||||
if node.isError or not node.structuredContent:
|
|
||||||
raise RuntimeError("fresh-wheel reference MCP node retrieval failed")
|
|
||||||
retrieved = node.structuredContent.get("node")
|
|
||||||
if not isinstance(retrieved, dict) or retrieved.get("node_id") != node_id:
|
|
||||||
raise RuntimeError("fresh-wheel reference MCP returned the wrong node")
|
|
||||||
return {
|
|
||||||
"bootstrap_status": bootstrap.structuredContent.get("status"),
|
|
||||||
"node_id": node_id,
|
|
||||||
"read_tool_count": len(tools),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
workspace = Path(sys.argv[1]).resolve(strict=True)
|
|
||||||
python = Path(sys.executable).absolute()
|
|
||||||
|
|
||||||
present_modules = [
|
|
||||||
name
|
|
||||||
for name in (
|
|
||||||
"tree_sitter",
|
|
||||||
"tree_sitter_cpp",
|
|
||||||
"tree_sitter_javascript",
|
|
||||||
"tree_sitter_typescript",
|
|
||||||
)
|
|
||||||
if importlib.util.find_spec(name) is not None
|
|
||||||
]
|
|
||||||
installed_frontends = sorted(
|
|
||||||
distribution.metadata["Name"]
|
|
||||||
for distribution in importlib.metadata.distributions()
|
|
||||||
if distribution.metadata["Name"].casefold().startswith("tree-sitter")
|
|
||||||
)
|
|
||||||
if present_modules or installed_frontends:
|
|
||||||
raise RuntimeError(
|
|
||||||
f"base wheel unexpectedly installed language frontends: "
|
|
||||||
f"{present_modules or installed_frontends}"
|
|
||||||
)
|
|
||||||
|
|
||||||
python_root = workspace / "python-project"
|
|
||||||
(python_root / "src" / "sample").mkdir(parents=True)
|
|
||||||
(python_root / "src" / "sample" / "__init__.py").write_text("", encoding="utf-8")
|
|
||||||
(python_root / "src" / "sample" / "service.py").write_text(
|
|
||||||
"class Service:\n"
|
|
||||||
" def execute(self, ready: bool) -> str:\n"
|
|
||||||
" if ready:\n"
|
|
||||||
' return "ready"\n'
|
|
||||||
' return "waiting"\n',
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
write_config(python_root, language="python", source_roots=("src",))
|
|
||||||
|
|
||||||
project = create_reference_project(python_root)
|
|
||||||
index = ProjectIndex(project)
|
|
||||||
built = index.build()
|
|
||||||
checked = index.check()
|
|
||||||
if built.get("status") != "ok" or checked.get("status") != "ok":
|
|
||||||
raise RuntimeError("fresh-wheel Python reference build/check failed")
|
|
||||||
mcp = asyncio.run(retrieve(python, python_root))
|
|
||||||
if mcp["bootstrap_status"] != "ok":
|
|
||||||
raise RuntimeError("fresh-wheel reference MCP bootstrap did not report ok")
|
|
||||||
|
|
||||||
cpp_root = workspace / "cpp-project"
|
|
||||||
(cpp_root / "src").mkdir(parents=True)
|
|
||||||
(cpp_root / "src" / "main.cpp").write_text("int main() { return 0; }\n", encoding="utf-8")
|
|
||||||
(cpp_root / "compile_commands.json").write_text(
|
|
||||||
json.dumps(
|
|
||||||
[
|
|
||||||
{
|
|
||||||
"directory": str(cpp_root),
|
|
||||||
"file": "src/main.cpp",
|
|
||||||
"arguments": ["c++", "-c", "src/main.cpp"],
|
|
||||||
}
|
|
||||||
],
|
|
||||||
sort_keys=True,
|
|
||||||
)
|
|
||||||
+ "\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
write_config(
|
|
||||||
cpp_root,
|
|
||||||
language="cpp",
|
|
||||||
source_roots=("src",),
|
|
||||||
compilation_database="compile_commands.json",
|
|
||||||
)
|
|
||||||
|
|
||||||
try:
|
|
||||||
ProjectIndex(create_reference_project(cpp_root)).build()
|
|
||||||
except DocForgeError as error:
|
|
||||||
if error.code != "optional_dependency_missing":
|
|
||||||
raise
|
|
||||||
if error.details.get("install") != "docforge[cpp]":
|
|
||||||
raise RuntimeError("C++ missing-extra failure was not actionable") from error
|
|
||||||
cpp_failure = {
|
|
||||||
"code": error.code,
|
|
||||||
"install": error.details["install"],
|
|
||||||
"missing_module": error.details.get("missing_module"),
|
|
||||||
}
|
|
||||||
else:
|
|
||||||
raise RuntimeError("C++ reference build unexpectedly worked without docforge[cpp]")
|
|
||||||
|
|
||||||
print(
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"schema_version": 1,
|
|
||||||
"base_frontend_modules": present_modules,
|
|
||||||
"base_frontend_distributions": installed_frontends,
|
|
||||||
"python": {
|
|
||||||
"build_status": built["status"],
|
|
||||||
"check_status": checked["status"],
|
|
||||||
"node_count": checked["node_count"],
|
|
||||||
},
|
|
||||||
"mcp": mcp,
|
|
||||||
"cpp_without_extra": cpp_failure,
|
|
||||||
},
|
|
||||||
sort_keys=True,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
class AdoptionProofError(RuntimeError):
|
|
||||||
"""The disposable adoption proof could not be completed."""
|
|
||||||
|
|
||||||
|
|
||||||
def _run_checked(
|
|
||||||
arguments: list[str],
|
|
||||||
*,
|
|
||||||
cwd: Path,
|
|
||||||
timeout: int,
|
|
||||||
environment: dict[str, str] | None = None,
|
|
||||||
) -> subprocess.CompletedProcess[str]:
|
|
||||||
result = subprocess.run(
|
|
||||||
arguments,
|
|
||||||
cwd=cwd,
|
|
||||||
env=environment,
|
|
||||||
text=True,
|
|
||||||
capture_output=True,
|
|
||||||
check=False,
|
|
||||||
timeout=timeout,
|
|
||||||
)
|
|
||||||
if result.returncode != 0:
|
|
||||||
detail = result.stderr.strip() or result.stdout.strip() or "no command output"
|
|
||||||
raise AdoptionProofError(
|
|
||||||
f"command failed with exit {result.returncode}: {arguments[0]}\n{detail}"
|
|
||||||
)
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
def find_wheel(directory: Path) -> Path:
|
|
||||||
wheels = tuple(sorted(directory.glob("docforge-*.whl")))
|
|
||||||
if len(wheels) != 1:
|
|
||||||
raise AdoptionProofError(f"expected exactly one DocForge wheel, found {len(wheels)}")
|
|
||||||
return wheels[0]
|
|
||||||
|
|
||||||
|
|
||||||
def parse_evidence(raw: str) -> dict[str, object]:
|
|
||||||
try:
|
|
||||||
loaded: object = json.loads(raw)
|
|
||||||
except json.JSONDecodeError as error:
|
|
||||||
raise AdoptionProofError("fresh-wheel validation returned invalid JSON") from error
|
|
||||||
if not isinstance(loaded, dict):
|
|
||||||
raise AdoptionProofError("fresh-wheel validation returned the wrong schema")
|
|
||||||
document = cast(dict[str, object], loaded)
|
|
||||||
if document.get("schema_version") != PROOF_SCHEMA_VERSION:
|
|
||||||
raise AdoptionProofError("fresh-wheel validation returned the wrong schema")
|
|
||||||
if document.get("base_frontend_modules") != []:
|
|
||||||
raise AdoptionProofError("fresh-wheel validation found optional frontend modules")
|
|
||||||
if document.get("base_frontend_distributions") != []:
|
|
||||||
raise AdoptionProofError("fresh-wheel validation found optional frontend distributions")
|
|
||||||
raw_python = document.get("python")
|
|
||||||
raw_mcp = document.get("mcp")
|
|
||||||
raw_cpp = document.get("cpp_without_extra")
|
|
||||||
python = cast(dict[str, object], raw_python) if isinstance(raw_python, dict) else None
|
|
||||||
mcp = cast(dict[str, object], raw_mcp) if isinstance(raw_mcp, dict) else None
|
|
||||||
cpp = cast(dict[str, object], raw_cpp) if isinstance(raw_cpp, dict) else None
|
|
||||||
if (
|
|
||||||
python is None
|
|
||||||
or python.get("build_status") != "ok"
|
|
||||||
or python.get("check_status") != "ok"
|
|
||||||
or mcp is None
|
|
||||||
or mcp.get("bootstrap_status") != "ok"
|
|
||||||
or cpp is None
|
|
||||||
or cpp.get("code") != "optional_dependency_missing"
|
|
||||||
or cpp.get("install") != "docforge[cpp]"
|
|
||||||
):
|
|
||||||
raise AdoptionProofError("fresh-wheel validation evidence is incomplete")
|
|
||||||
return document
|
|
||||||
|
|
||||||
|
|
||||||
def run_adoption_proof(repository_root: Path, *, uv: str = "uv") -> dict[str, object]:
|
|
||||||
"""Build and exercise a base wheel without using the network."""
|
|
||||||
|
|
||||||
root = repository_root.resolve(strict=True)
|
|
||||||
if not root.joinpath("pyproject.toml").is_file() or not root.joinpath("uv.lock").is_file():
|
|
||||||
raise AdoptionProofError("repository root lacks pyproject.toml or uv.lock")
|
|
||||||
|
|
||||||
offline_environment = {
|
|
||||||
**os.environ,
|
|
||||||
"UV_OFFLINE": "1",
|
|
||||||
"UV_PYTHON_DOWNLOADS": "never",
|
|
||||||
}
|
|
||||||
offline_environment.pop("PYTHONPATH", None)
|
|
||||||
with tempfile.TemporaryDirectory(prefix="docforge-m4-adoption-") as directory:
|
|
||||||
workspace = Path(directory).resolve()
|
|
||||||
distributions = workspace / "dist"
|
|
||||||
requirements = workspace / "base-requirements.txt"
|
|
||||||
virtual_environment = workspace / "venv"
|
|
||||||
python = virtual_environment / "bin" / "python"
|
|
||||||
|
|
||||||
_run_checked(
|
|
||||||
[
|
|
||||||
uv,
|
|
||||||
"build",
|
|
||||||
"--wheel",
|
|
||||||
"--offline",
|
|
||||||
"--no-python-downloads",
|
|
||||||
"--out-dir",
|
|
||||||
str(distributions),
|
|
||||||
str(root),
|
|
||||||
],
|
|
||||||
cwd=root,
|
|
||||||
timeout=COMMAND_TIMEOUT_SECONDS,
|
|
||||||
environment=offline_environment,
|
|
||||||
)
|
|
||||||
wheel = find_wheel(distributions)
|
|
||||||
_run_checked(
|
|
||||||
[
|
|
||||||
uv,
|
|
||||||
"export",
|
|
||||||
"--frozen",
|
|
||||||
"--offline",
|
|
||||||
"--no-dev",
|
|
||||||
"--no-emit-project",
|
|
||||||
"--output-file",
|
|
||||||
str(requirements),
|
|
||||||
],
|
|
||||||
cwd=root,
|
|
||||||
timeout=COMMAND_TIMEOUT_SECONDS,
|
|
||||||
environment=offline_environment,
|
|
||||||
)
|
|
||||||
exported = requirements.read_text(encoding="utf-8").casefold()
|
|
||||||
if any(module.replace("_", "-") in exported for module in OPTIONAL_FRONTEND_MODULES):
|
|
||||||
raise AdoptionProofError("locked base dependency export contains a language frontend")
|
|
||||||
_run_checked(
|
|
||||||
[
|
|
||||||
uv,
|
|
||||||
"venv",
|
|
||||||
"--python",
|
|
||||||
sys.executable,
|
|
||||||
"--no-project",
|
|
||||||
"--no-python-downloads",
|
|
||||||
str(virtual_environment),
|
|
||||||
],
|
|
||||||
cwd=workspace,
|
|
||||||
timeout=COMMAND_TIMEOUT_SECONDS,
|
|
||||||
environment=offline_environment,
|
|
||||||
)
|
|
||||||
_run_checked(
|
|
||||||
[
|
|
||||||
uv,
|
|
||||||
"pip",
|
|
||||||
"sync",
|
|
||||||
"--python",
|
|
||||||
str(python),
|
|
||||||
"--offline",
|
|
||||||
"--strict",
|
|
||||||
str(requirements),
|
|
||||||
],
|
|
||||||
cwd=workspace,
|
|
||||||
timeout=COMMAND_TIMEOUT_SECONDS,
|
|
||||||
environment=offline_environment,
|
|
||||||
)
|
|
||||||
_run_checked(
|
|
||||||
[
|
|
||||||
uv,
|
|
||||||
"pip",
|
|
||||||
"install",
|
|
||||||
"--python",
|
|
||||||
str(python),
|
|
||||||
"--offline",
|
|
||||||
"--strict",
|
|
||||||
"--no-deps",
|
|
||||||
str(wheel),
|
|
||||||
],
|
|
||||||
cwd=workspace,
|
|
||||||
timeout=COMMAND_TIMEOUT_SECONDS,
|
|
||||||
environment=offline_environment,
|
|
||||||
)
|
|
||||||
validation = _run_checked(
|
|
||||||
[str(python), "-I", "-c", _VALIDATION_PROGRAM, str(workspace)],
|
|
||||||
cwd=workspace,
|
|
||||||
timeout=VALIDATION_TIMEOUT_SECONDS,
|
|
||||||
environment=offline_environment,
|
|
||||||
)
|
|
||||||
evidence = parse_evidence(validation.stdout.strip())
|
|
||||||
evidence["wheel"] = {
|
|
||||||
"filename": wheel.name,
|
|
||||||
"sha256": hashlib.sha256(wheel.read_bytes()).hexdigest(),
|
|
||||||
}
|
|
||||||
evidence["network"] = "offline"
|
|
||||||
return evidence
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
parser = argparse.ArgumentParser(
|
|
||||||
description="Run the offline Milestone 4 fresh-wheel adoption proof"
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"--repository-root",
|
|
||||||
type=Path,
|
|
||||||
default=Path(__file__).resolve().parents[1],
|
|
||||||
)
|
|
||||||
parser.add_argument("--uv", default=shutil.which("uv") or "uv")
|
|
||||||
parser.add_argument("--output", type=Path)
|
|
||||||
arguments = parser.parse_args()
|
|
||||||
|
|
||||||
try:
|
|
||||||
evidence = run_adoption_proof(arguments.repository_root, uv=arguments.uv)
|
|
||||||
except (AdoptionProofError, OSError, subprocess.SubprocessError) as error:
|
|
||||||
print(json.dumps({"status": "error", "error": str(error)}, sort_keys=True))
|
|
||||||
raise SystemExit(1) from error
|
|
||||||
|
|
||||||
payload = json.dumps({"status": "ok", **evidence}, indent=2, sort_keys=True) + "\n"
|
|
||||||
if arguments.output is not None:
|
|
||||||
arguments.output.write_text(payload, encoding="utf-8")
|
|
||||||
print(payload, end="")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
|
|
@ -1,498 +0,0 @@
|
||||||
"""Milestone 4 Python adapter SDK, incremental, and recovery benchmark gates."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import ast
|
|
||||||
import gc
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import math
|
|
||||||
import platform
|
|
||||||
import resource
|
|
||||||
import statistics
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
import tempfile
|
|
||||||
import time
|
|
||||||
import tracemalloc
|
|
||||||
from collections.abc import Callable, Generator, Mapping
|
|
||||||
from contextlib import contextmanager
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import cast
|
|
||||||
from unittest import mock
|
|
||||||
|
|
||||||
from docforge.adapter_sdk import (
|
|
||||||
AdapterAssembly,
|
|
||||||
AdapterProject,
|
|
||||||
AdapterSource,
|
|
||||||
AdapterSourceProjection,
|
|
||||||
)
|
|
||||||
from docforge.adapters.python import PythonReferenceAdapter
|
|
||||||
from docforge.index import ProjectIndex
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
FULL_SOURCE_COUNT = 334
|
|
||||||
SMOKE_SOURCE_COUNT = 12
|
|
||||||
NODES_PER_SOURCE = 3
|
|
||||||
DEFAULT_FULL_SAMPLES = 3
|
|
||||||
DEFAULT_SMOKE_SAMPLES = 1
|
|
||||||
|
|
||||||
# These are regression tripwires, not performance targets. They deliberately
|
|
||||||
# leave several times the Milestone 3 1,000-node in-process allowances for the
|
|
||||||
# additional AST, Logic, extraction-cache, and SQLite work in this gate.
|
|
||||||
MAX_TRACED_PEAK_BYTES = 256 * 1024 * 1024
|
|
||||||
MAX_PROCESS_PEAK_BYTES = 512 * 1024 * 1024
|
|
||||||
MAX_RESPONSE_BYTES = 256 * 1024
|
|
||||||
COLD_LIMIT_MS = 30_000.0
|
|
||||||
WARM_LIMIT_MS = 20_000.0
|
|
||||||
EQUIVALENCE_LIMIT_MS = 30_000.0
|
|
||||||
RECOVERY_LIMIT_MS = 30_000.0
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class _InstrumentedResult:
|
|
||||||
value: Mapping[str, object]
|
|
||||||
ast_parse_calls: int
|
|
||||||
extraction_calls: int
|
|
||||||
|
|
||||||
|
|
||||||
class _RecordingPythonAdapter(PythonReferenceAdapter):
|
|
||||||
"""Count extraction entry points without weakening the production adapter."""
|
|
||||||
|
|
||||||
def __init__(self, root: Path) -> None:
|
|
||||||
super().__init__(
|
|
||||||
root,
|
|
||||||
source_roots=("src",),
|
|
||||||
project_id="milestone4-python-benchmark",
|
|
||||||
title="Milestone 4 Python adapter benchmark",
|
|
||||||
)
|
|
||||||
self.extraction_calls = 0
|
|
||||||
|
|
||||||
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection:
|
|
||||||
self.extraction_calls += 1
|
|
||||||
return super().extract_source(source)
|
|
||||||
|
|
||||||
|
|
||||||
def _parser() -> argparse.ArgumentParser:
|
|
||||||
parser = argparse.ArgumentParser(
|
|
||||||
description=("Gate DocForge2 Milestone 4 adapter behavior on a disposable Python project.")
|
|
||||||
)
|
|
||||||
parser.add_argument("--mode", choices=("smoke", "full"), default="full")
|
|
||||||
parser.add_argument("--sources", type=int)
|
|
||||||
parser.add_argument("--samples", type=int)
|
|
||||||
parser.add_argument("--output", type=Path)
|
|
||||||
return parser
|
|
||||||
|
|
||||||
|
|
||||||
def encode_report(value: object) -> str:
|
|
||||||
"""Serialize benchmark evidence in one deterministic JSON representation."""
|
|
||||||
|
|
||||||
return json.dumps(value, sort_keys=True, indent=2, ensure_ascii=False) + "\n"
|
|
||||||
|
|
||||||
|
|
||||||
def _compact_json(value: object) -> bytes:
|
|
||||||
return json.dumps(
|
|
||||||
value,
|
|
||||||
sort_keys=True,
|
|
||||||
separators=(",", ":"),
|
|
||||||
ensure_ascii=False,
|
|
||||||
).encode("utf-8")
|
|
||||||
|
|
||||||
|
|
||||||
def _sha256(value: object) -> str:
|
|
||||||
return hashlib.sha256(_compact_json(value)).hexdigest()
|
|
||||||
|
|
||||||
|
|
||||||
def _git(arguments: list[str]) -> str:
|
|
||||||
return subprocess.run(
|
|
||||||
["git", *arguments],
|
|
||||||
cwd=ROOT,
|
|
||||||
check=True,
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
).stdout.strip()
|
|
||||||
|
|
||||||
|
|
||||||
def write_python_fixture(root: Path, source_count: int) -> None:
|
|
||||||
"""Write a deterministic project with three primary nodes per source."""
|
|
||||||
|
|
||||||
if source_count < 2:
|
|
||||||
raise ValueError("The Milestone 4 fixture requires at least two sources")
|
|
||||||
package = root / "src" / "benchmark"
|
|
||||||
package.mkdir(parents=True)
|
|
||||||
for index in range(source_count):
|
|
||||||
dependency = ""
|
|
||||||
call = f"value = {index}"
|
|
||||||
if index:
|
|
||||||
dependency = f"from benchmark.module_{index - 1:04d} import compute_{index - 1:04d}\n\n"
|
|
||||||
call = f"value = compute_{index - 1:04d}(value)"
|
|
||||||
source = (
|
|
||||||
f'"""Synthetic adapter benchmark module {index:04d}."""\n\n'
|
|
||||||
f"{dependency}"
|
|
||||||
f"def compute_{index:04d}(value: int) -> int:\n"
|
|
||||||
f' """Return one deterministic branch result."""\n'
|
|
||||||
f" {call}\n"
|
|
||||||
" if value % 2:\n"
|
|
||||||
f" return value + {index + 1}\n"
|
|
||||||
" return value\n"
|
|
||||||
)
|
|
||||||
(package / f"module_{index:04d}.py").write_text(source, encoding="utf-8")
|
|
||||||
|
|
||||||
|
|
||||||
@contextmanager
|
|
||||||
def _count_ast_parses() -> Generator[mock.MagicMock]:
|
|
||||||
with mock.patch.object(ast, "parse", wraps=ast.parse) as parsed:
|
|
||||||
yield parsed
|
|
||||||
|
|
||||||
|
|
||||||
def _instrument(
|
|
||||||
adapter: _RecordingPythonAdapter,
|
|
||||||
operation: Callable[[], Mapping[str, object]],
|
|
||||||
) -> _InstrumentedResult:
|
|
||||||
adapter.extraction_calls = 0
|
|
||||||
with _count_ast_parses() as counter:
|
|
||||||
value = operation()
|
|
||||||
return _InstrumentedResult(
|
|
||||||
value=value,
|
|
||||||
ast_parse_calls=counter.call_count,
|
|
||||||
extraction_calls=adapter.extraction_calls,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _build_summary(sample: _InstrumentedResult) -> dict[str, object]:
|
|
||||||
value = sample.value
|
|
||||||
build = cast(Mapping[str, object], value["build"])
|
|
||||||
return {
|
|
||||||
"status": value["status"],
|
|
||||||
"revision": value["revision"],
|
|
||||||
"source_hash": value["source_hash"],
|
|
||||||
"node_hash": value["node_hash"],
|
|
||||||
"node_count": value["node_count"],
|
|
||||||
"edge_hash": value["edge_hash"],
|
|
||||||
"edge_count": value["edge_count"],
|
|
||||||
"logic_hash": value["logic_hash"],
|
|
||||||
"logic_projection_count": value["logic_projection_count"],
|
|
||||||
"logic_node_count": value["logic_node_count"],
|
|
||||||
"logic_edge_count": value["logic_edge_count"],
|
|
||||||
"cache_hits": build["cache_hits"],
|
|
||||||
"reparsed_sources": build["reparsed_sources"],
|
|
||||||
"invalidated_sources": build["invalidated_sources"],
|
|
||||||
"deleted_sources": build["deleted_sources"],
|
|
||||||
"total_sources": build["total_sources"],
|
|
||||||
"ast_parse_calls": sample.ast_parse_calls,
|
|
||||||
"extraction_calls": sample.extraction_calls,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _equivalence_summary(sample: _InstrumentedResult) -> dict[str, object]:
|
|
||||||
return {
|
|
||||||
**sample.value,
|
|
||||||
"ast_parse_calls": sample.ast_parse_calls,
|
|
||||||
"extraction_calls": sample.extraction_calls,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _synchronization_summary(sample: _InstrumentedResult) -> dict[str, object]:
|
|
||||||
synchronization = cast(Mapping[str, object], sample.value["synchronization"])
|
|
||||||
build = cast(Mapping[str, object], synchronization["build"])
|
|
||||||
return {
|
|
||||||
"status": sample.value["status"],
|
|
||||||
"revision": sample.value["revision"],
|
|
||||||
"source_hash": sample.value["source_hash"],
|
|
||||||
"node_hash": sample.value["node_hash"],
|
|
||||||
"node_count": sample.value["node_count"],
|
|
||||||
"edge_hash": sample.value["edge_hash"],
|
|
||||||
"edge_count": sample.value["edge_count"],
|
|
||||||
"logic_hash": sample.value["logic_hash"],
|
|
||||||
"logic_projection_count": sample.value["logic_projection_count"],
|
|
||||||
"action": synchronization["action"],
|
|
||||||
"initial_error_code": cast(Mapping[str, object], synchronization["initial_error"])["code"],
|
|
||||||
"cache_hits": build["cache_hits"],
|
|
||||||
"reparsed_sources": build["reparsed_sources"],
|
|
||||||
"ast_parse_calls": sample.ast_parse_calls,
|
|
||||||
"extraction_calls": sample.extraction_calls,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _measure(
|
|
||||||
operation: Callable[[], _InstrumentedResult],
|
|
||||||
*,
|
|
||||||
samples: int,
|
|
||||||
p95_limit_ms: float,
|
|
||||||
summary: Callable[[_InstrumentedResult], Mapping[str, object]],
|
|
||||||
) -> tuple[dict[str, object], _InstrumentedResult]:
|
|
||||||
durations: list[float] = []
|
|
||||||
traced_peaks: list[int] = []
|
|
||||||
response_sizes: list[int] = []
|
|
||||||
stable_summary: dict[str, object] | None = None
|
|
||||||
last: _InstrumentedResult | None = None
|
|
||||||
for _ in range(samples):
|
|
||||||
gc.collect()
|
|
||||||
tracemalloc.start()
|
|
||||||
started = time.perf_counter_ns()
|
|
||||||
try:
|
|
||||||
current = operation()
|
|
||||||
elapsed_ms = (time.perf_counter_ns() - started) / 1_000_000
|
|
||||||
_, traced_peak = tracemalloc.get_traced_memory()
|
|
||||||
finally:
|
|
||||||
tracemalloc.stop()
|
|
||||||
current_summary = dict(summary(current))
|
|
||||||
if stable_summary is None:
|
|
||||||
stable_summary = current_summary
|
|
||||||
elif current_summary != stable_summary:
|
|
||||||
raise RuntimeError("Milestone 4 operation changed its deterministic result")
|
|
||||||
response_bytes = len(_compact_json(current.value))
|
|
||||||
if response_bytes > MAX_RESPONSE_BYTES:
|
|
||||||
raise RuntimeError(
|
|
||||||
"Milestone 4 response exceeded its fixed byte boundary: "
|
|
||||||
f"{response_bytes} > {MAX_RESPONSE_BYTES}"
|
|
||||||
)
|
|
||||||
if traced_peak > MAX_TRACED_PEAK_BYTES:
|
|
||||||
raise RuntimeError(
|
|
||||||
"Milestone 4 operation exceeded its traced-memory boundary: "
|
|
||||||
f"{traced_peak} > {MAX_TRACED_PEAK_BYTES}"
|
|
||||||
)
|
|
||||||
durations.append(elapsed_ms)
|
|
||||||
traced_peaks.append(traced_peak)
|
|
||||||
response_sizes.append(response_bytes)
|
|
||||||
last = current
|
|
||||||
ordered = sorted(durations)
|
|
||||||
p95_index = max(0, math.ceil(len(ordered) * 0.95) - 1)
|
|
||||||
p95 = ordered[p95_index]
|
|
||||||
if p95 > p95_limit_ms:
|
|
||||||
raise RuntimeError(f"Milestone 4 operation p95 {p95:.3f} ms exceeds {p95_limit_ms:.3f} ms")
|
|
||||||
assert stable_summary is not None
|
|
||||||
assert last is not None
|
|
||||||
return (
|
|
||||||
{
|
|
||||||
"samples": samples,
|
|
||||||
"median_ms": round(statistics.median(ordered), 3),
|
|
||||||
"p95_ms": round(p95, 3),
|
|
||||||
"min_ms": round(ordered[0], 3),
|
|
||||||
"max_ms": round(ordered[-1], 3),
|
|
||||||
"p95_limit_ms": p95_limit_ms,
|
|
||||||
"maximum_response_bytes": max(response_sizes),
|
|
||||||
"response_limit_bytes": MAX_RESPONSE_BYTES,
|
|
||||||
"maximum_traced_peak_bytes": max(traced_peaks),
|
|
||||||
"traced_peak_limit_bytes": MAX_TRACED_PEAK_BYTES,
|
|
||||||
"stable_result": stable_summary,
|
|
||||||
},
|
|
||||||
last,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _semantic_assembly_hash(assembly: AdapterAssembly) -> str:
|
|
||||||
projection = assembly.projection
|
|
||||||
return _sha256(
|
|
||||||
{
|
|
||||||
"project_id": projection.project_id,
|
|
||||||
"title": projection.title,
|
|
||||||
"adapter_id": projection.adapter_id,
|
|
||||||
"adapter_version": projection.adapter_version,
|
|
||||||
"revision": projection.revision,
|
|
||||||
"source_hash": projection.source_hash,
|
|
||||||
"nodes": [item.as_dict() for item in projection.nodes],
|
|
||||||
"edges": [item.as_dict() for item in projection.edges],
|
|
||||||
"logic": [item.as_dict() for item in assembly.logic],
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _benchmark(root: Path, source_count: int, samples: int) -> dict[str, object]:
|
|
||||||
write_python_fixture(root, source_count)
|
|
||||||
adapter = _RecordingPythonAdapter(root)
|
|
||||||
cache_root = root / ".docforge" / "benchmark"
|
|
||||||
project = AdapterProject(adapter, cache_root=cache_root)
|
|
||||||
index = ProjectIndex(project)
|
|
||||||
|
|
||||||
cold, cold_sample = _measure(
|
|
||||||
lambda: _instrument(adapter, index.build),
|
|
||||||
samples=1,
|
|
||||||
p95_limit_ms=COLD_LIMIT_MS,
|
|
||||||
summary=_build_summary,
|
|
||||||
)
|
|
||||||
cold_summary = _build_summary(cold_sample)
|
|
||||||
expected_nodes = source_count * NODES_PER_SOURCE
|
|
||||||
if (
|
|
||||||
cold_summary["node_count"] != expected_nodes
|
|
||||||
or cold_summary["logic_projection_count"] != source_count
|
|
||||||
or cold_summary["reparsed_sources"] != source_count
|
|
||||||
or cast(int, cold_summary["ast_parse_calls"]) < source_count
|
|
||||||
or cold_summary["extraction_calls"] != source_count
|
|
||||||
):
|
|
||||||
raise RuntimeError("Cold adapter build did not cover the complete synthetic fixture")
|
|
||||||
|
|
||||||
warm, warm_sample = _measure(
|
|
||||||
lambda: _instrument(adapter, index.build),
|
|
||||||
samples=samples,
|
|
||||||
p95_limit_ms=WARM_LIMIT_MS,
|
|
||||||
summary=_build_summary,
|
|
||||||
)
|
|
||||||
warm_summary = _build_summary(warm_sample)
|
|
||||||
if (
|
|
||||||
warm_summary["cache_hits"] != source_count
|
|
||||||
or warm_summary["reparsed_sources"] != 0
|
|
||||||
or warm_summary["ast_parse_calls"] != 0
|
|
||||||
or warm_summary["extraction_calls"] != 0
|
|
||||||
):
|
|
||||||
raise RuntimeError("Warm adapter build performed forbidden parsing or extraction")
|
|
||||||
|
|
||||||
equivalence, equivalence_sample = _measure(
|
|
||||||
lambda: _instrument(adapter, project.verify_incremental_equivalence),
|
|
||||||
samples=1,
|
|
||||||
p95_limit_ms=EQUIVALENCE_LIMIT_MS,
|
|
||||||
summary=_equivalence_summary,
|
|
||||||
)
|
|
||||||
equivalence_summary = _equivalence_summary(equivalence_sample)
|
|
||||||
if (
|
|
||||||
equivalence_summary["status"] != "ok"
|
|
||||||
or equivalence_summary["node_count"] != expected_nodes
|
|
||||||
or equivalence_summary["logic_projection_count"] != source_count
|
|
||||||
):
|
|
||||||
raise RuntimeError("Full and incremental graph plus Logic output was not equivalent")
|
|
||||||
complete_assembly = adapter.load_complete_assembly()
|
|
||||||
assembly_hash = _semantic_assembly_hash(complete_assembly)
|
|
||||||
|
|
||||||
extraction_cache = cache_root / "extractions.json"
|
|
||||||
extraction_cache.write_bytes(b"{broken")
|
|
||||||
cache_recovery, cache_recovery_sample = _measure(
|
|
||||||
lambda: _instrument(adapter, index.build),
|
|
||||||
samples=1,
|
|
||||||
p95_limit_ms=RECOVERY_LIMIT_MS,
|
|
||||||
summary=_build_summary,
|
|
||||||
)
|
|
||||||
cache_recovery_summary = _build_summary(cache_recovery_sample)
|
|
||||||
if (
|
|
||||||
cache_recovery_summary["reparsed_sources"] != source_count
|
|
||||||
or cast(int, cache_recovery_summary["ast_parse_calls"]) < source_count
|
|
||||||
or cache_recovery_summary["node_hash"] != cold_summary["node_hash"]
|
|
||||||
or cache_recovery_summary["logic_hash"] != cold_summary["logic_hash"]
|
|
||||||
):
|
|
||||||
raise RuntimeError("Corrupt extraction-cache recovery changed adapter output")
|
|
||||||
|
|
||||||
index.path.write_bytes(b"not-a-sqlite-index")
|
|
||||||
index_recovery, index_recovery_sample = _measure(
|
|
||||||
lambda: _instrument(adapter, index.synchronize),
|
|
||||||
samples=1,
|
|
||||||
p95_limit_ms=RECOVERY_LIMIT_MS,
|
|
||||||
summary=_synchronization_summary,
|
|
||||||
)
|
|
||||||
index_recovery_summary = _synchronization_summary(index_recovery_sample)
|
|
||||||
if (
|
|
||||||
index_recovery_summary["action"] != "rebuilt"
|
|
||||||
or index_recovery_summary["cache_hits"] != source_count
|
|
||||||
or index_recovery_summary["reparsed_sources"] != 0
|
|
||||||
or index_recovery_summary["ast_parse_calls"] != 0
|
|
||||||
or index_recovery_summary["extraction_calls"] != 0
|
|
||||||
or index_recovery_summary["node_hash"] != cold_summary["node_hash"]
|
|
||||||
or index_recovery_summary["logic_hash"] != cold_summary["logic_hash"]
|
|
||||||
):
|
|
||||||
raise RuntimeError("Corrupt derived-index recovery changed adapter output")
|
|
||||||
|
|
||||||
evidence = {
|
|
||||||
"source_count": source_count,
|
|
||||||
"node_count": expected_nodes,
|
|
||||||
"edge_count": cold_summary["edge_count"],
|
|
||||||
"logic_projection_count": source_count,
|
|
||||||
"logic_node_count": cold_summary["logic_node_count"],
|
|
||||||
"logic_edge_count": cold_summary["logic_edge_count"],
|
|
||||||
"source_hash": cold_summary["source_hash"],
|
|
||||||
"node_hash": cold_summary["node_hash"],
|
|
||||||
"edge_hash": cold_summary["edge_hash"],
|
|
||||||
"logic_hash": cold_summary["logic_hash"],
|
|
||||||
"complete_assembly_hash": assembly_hash,
|
|
||||||
"full_incremental_graph_and_logic_exact": True,
|
|
||||||
"warm_zero_ast_parse": True,
|
|
||||||
"warm_zero_source_extraction": True,
|
|
||||||
"cache_recovery_exact": True,
|
|
||||||
"index_recovery_exact": True,
|
|
||||||
}
|
|
||||||
return {
|
|
||||||
"fixture": {
|
|
||||||
"kind": "synthetic_python_reference_adapter",
|
|
||||||
"source_count": source_count,
|
|
||||||
"nodes_per_source": NODES_PER_SOURCE,
|
|
||||||
"expected_node_count": expected_nodes,
|
|
||||||
},
|
|
||||||
"operations": {
|
|
||||||
"cold_incremental_build": cold,
|
|
||||||
"warm_incremental_build": warm,
|
|
||||||
"full_incremental_equivalence": equivalence,
|
|
||||||
"corrupt_extraction_cache_recovery": cache_recovery,
|
|
||||||
"corrupt_index_recovery": index_recovery,
|
|
||||||
},
|
|
||||||
"evidence": evidence,
|
|
||||||
"evidence_sha256": _sha256(evidence),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
arguments = _parser().parse_args()
|
|
||||||
default_sources = FULL_SOURCE_COUNT if arguments.mode == "full" else SMOKE_SOURCE_COUNT
|
|
||||||
default_samples = DEFAULT_FULL_SAMPLES if arguments.mode == "full" else DEFAULT_SMOKE_SAMPLES
|
|
||||||
source_count = default_sources if arguments.sources is None else arguments.sources
|
|
||||||
samples = default_samples if arguments.samples is None else arguments.samples
|
|
||||||
if not 2 <= source_count <= FULL_SOURCE_COUNT:
|
|
||||||
raise SystemExit(f"--sources must be between 2 and {FULL_SOURCE_COUNT}")
|
|
||||||
if arguments.mode == "full" and source_count != FULL_SOURCE_COUNT:
|
|
||||||
raise SystemExit(f"--mode full requires exactly {FULL_SOURCE_COUNT} sources")
|
|
||||||
if samples < 1:
|
|
||||||
raise SystemExit("--samples must be positive")
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory(prefix="docforge-milestone4-") as directory:
|
|
||||||
measurement = _benchmark(Path(directory).resolve(), source_count, samples)
|
|
||||||
|
|
||||||
process_peak_bytes = int(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss) * 1024
|
|
||||||
if process_peak_bytes > MAX_PROCESS_PEAK_BYTES:
|
|
||||||
raise RuntimeError(
|
|
||||||
"Milestone 4 process peak exceeded its memory boundary: "
|
|
||||||
f"{process_peak_bytes} > {MAX_PROCESS_PEAK_BYTES}"
|
|
||||||
)
|
|
||||||
status = _git(["status", "--porcelain"])
|
|
||||||
report: dict[str, object] = {
|
|
||||||
"schema_version": 1,
|
|
||||||
"benchmark": "docforge2_milestone4",
|
|
||||||
"mode": arguments.mode,
|
|
||||||
"source": {
|
|
||||||
"revision": _git(["rev-parse", "HEAD"]),
|
|
||||||
"dirty": bool(status),
|
|
||||||
},
|
|
||||||
"environment": {
|
|
||||||
"platform": platform.platform(),
|
|
||||||
"machine": platform.machine(),
|
|
||||||
"python": platform.python_version(),
|
|
||||||
"implementation": platform.python_implementation(),
|
|
||||||
},
|
|
||||||
"method": {
|
|
||||||
"clock": "time.perf_counter_ns",
|
|
||||||
"in_process_peak_memory": "tracemalloc per measured invocation",
|
|
||||||
"process_peak_memory": "resource.getrusage(RUSAGE_SELF).ru_maxrss",
|
|
||||||
"response_size": "UTF-8 bytes of compact sorted JSON",
|
|
||||||
"ast_instrumentation": "temporary counter around stdlib ast.parse",
|
|
||||||
"extraction_instrumentation": "reference adapter extract_source entry counter",
|
|
||||||
"samples": samples,
|
|
||||||
"full_mode_source_requirement": FULL_SOURCE_COUNT,
|
|
||||||
"threshold_basis": (
|
|
||||||
"Regression tripwires leave several times the Milestone 3 1,000-node "
|
|
||||||
"allowances for AST, Logic, extraction-cache, and SQLite work."
|
|
||||||
),
|
|
||||||
},
|
|
||||||
**measurement,
|
|
||||||
"memory": {
|
|
||||||
"process_peak_bytes": process_peak_bytes,
|
|
||||||
"process_peak_limit_bytes": MAX_PROCESS_PEAK_BYTES,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
encoded = encode_report(report)
|
|
||||||
if arguments.output is not None:
|
|
||||||
output = arguments.output.resolve()
|
|
||||||
output.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
output.write_text(encoded, encoding="utf-8")
|
|
||||||
sys.stdout.write(encoded)
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
54
uv.lock
generated
54
uv.lock
generated
|
|
@ -211,26 +211,9 @@ source = { editable = "." }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "markdown-it-py" },
|
{ name = "markdown-it-py" },
|
||||||
{ name = "mcp" },
|
{ name = "mcp" },
|
||||||
]
|
|
||||||
|
|
||||||
[package.optional-dependencies]
|
|
||||||
cpp = [
|
|
||||||
{ name = "tree-sitter" },
|
|
||||||
{ name = "tree-sitter-cpp" },
|
|
||||||
]
|
|
||||||
javascript = [
|
|
||||||
{ name = "tree-sitter" },
|
|
||||||
{ name = "tree-sitter-javascript" },
|
|
||||||
]
|
|
||||||
languages = [
|
|
||||||
{ name = "tree-sitter" },
|
{ name = "tree-sitter" },
|
||||||
{ name = "tree-sitter-cpp" },
|
{ name = "tree-sitter-cpp" },
|
||||||
{ name = "tree-sitter-javascript" },
|
{ name = "tree-sitter-javascript" },
|
||||||
{ name = "tree-sitter-typescript" },
|
|
||||||
]
|
|
||||||
typescript = [
|
|
||||||
{ name = "tree-sitter" },
|
|
||||||
{ name = "tree-sitter-typescript" },
|
|
||||||
]
|
]
|
||||||
|
|
||||||
[package.dev-dependencies]
|
[package.dev-dependencies]
|
||||||
|
|
@ -238,38 +221,22 @@ dev = [
|
||||||
{ name = "jsonschema" },
|
{ name = "jsonschema" },
|
||||||
{ name = "pytest" },
|
{ name = "pytest" },
|
||||||
{ name = "ruff" },
|
{ name = "ruff" },
|
||||||
{ name = "tree-sitter" },
|
|
||||||
{ name = "tree-sitter-cpp" },
|
|
||||||
{ name = "tree-sitter-javascript" },
|
|
||||||
{ name = "tree-sitter-typescript" },
|
|
||||||
]
|
]
|
||||||
|
|
||||||
[package.metadata]
|
[package.metadata]
|
||||||
requires-dist = [
|
requires-dist = [
|
||||||
{ name = "markdown-it-py", specifier = ">=4.2,<5" },
|
{ name = "markdown-it-py", specifier = ">=4.2,<5" },
|
||||||
{ name = "mcp", specifier = ">=1.28,<2" },
|
{ name = "mcp", specifier = ">=1.28,<2" },
|
||||||
{ name = "tree-sitter", marker = "extra == 'cpp'", specifier = ">=0.25,<0.26" },
|
{ name = "tree-sitter", specifier = ">=0.25,<0.26" },
|
||||||
{ name = "tree-sitter", marker = "extra == 'javascript'", specifier = ">=0.25,<0.26" },
|
{ name = "tree-sitter-cpp", specifier = ">=0.23,<0.24" },
|
||||||
{ name = "tree-sitter", marker = "extra == 'languages'", specifier = ">=0.25,<0.26" },
|
{ name = "tree-sitter-javascript", specifier = ">=0.25,<0.26" },
|
||||||
{ name = "tree-sitter", marker = "extra == 'typescript'", specifier = ">=0.25,<0.26" },
|
|
||||||
{ name = "tree-sitter-cpp", marker = "extra == 'cpp'", specifier = ">=0.23,<0.24" },
|
|
||||||
{ name = "tree-sitter-cpp", marker = "extra == 'languages'", specifier = ">=0.23,<0.24" },
|
|
||||||
{ name = "tree-sitter-javascript", marker = "extra == 'javascript'", specifier = ">=0.25,<0.26" },
|
|
||||||
{ name = "tree-sitter-javascript", marker = "extra == 'languages'", specifier = ">=0.25,<0.26" },
|
|
||||||
{ name = "tree-sitter-typescript", marker = "extra == 'languages'", specifier = ">=0.23,<0.24" },
|
|
||||||
{ name = "tree-sitter-typescript", marker = "extra == 'typescript'", specifier = ">=0.23,<0.24" },
|
|
||||||
]
|
]
|
||||||
provides-extras = ["javascript", "typescript", "cpp", "languages"]
|
|
||||||
|
|
||||||
[package.metadata.requires-dev]
|
[package.metadata.requires-dev]
|
||||||
dev = [
|
dev = [
|
||||||
{ name = "jsonschema", specifier = ">=4.25,<5" },
|
{ name = "jsonschema", specifier = ">=4.25,<5" },
|
||||||
{ name = "pytest", specifier = ">=9.1,<10" },
|
{ name = "pytest", specifier = ">=9.1,<10" },
|
||||||
{ name = "ruff", specifier = ">=0.15,<1" },
|
{ name = "ruff", specifier = ">=0.15,<1" },
|
||||||
{ name = "tree-sitter", specifier = ">=0.25,<0.26" },
|
|
||||||
{ name = "tree-sitter-cpp", specifier = ">=0.23,<0.24" },
|
|
||||||
{ name = "tree-sitter-javascript", specifier = ">=0.25,<0.26" },
|
|
||||||
{ name = "tree-sitter-typescript", specifier = ">=0.23,<0.24" },
|
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
|
|
@ -837,21 +804,6 @@ wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/2e/1f/f9eba1038b7d4394410f3c0a6ec2122b590cd7acb03f196e52fa57ebbe72/tree_sitter_javascript-0.25.0-cp310-abi3-win_arm64.whl", hash = "sha256:622a69d677aa7f6ee2931d8c77c981a33f0ebb6d275aa9d43d3397c879a9bb0b", size = 61668, upload-time = "2025-09-01T07:13:43.803Z" },
|
{ url = "https://files.pythonhosted.org/packages/2e/1f/f9eba1038b7d4394410f3c0a6ec2122b590cd7acb03f196e52fa57ebbe72/tree_sitter_javascript-0.25.0-cp310-abi3-win_arm64.whl", hash = "sha256:622a69d677aa7f6ee2931d8c77c981a33f0ebb6d275aa9d43d3397c879a9bb0b", size = 61668, upload-time = "2025-09-01T07:13:43.803Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "tree-sitter-typescript"
|
|
||||||
version = "0.23.2"
|
|
||||||
source = { registry = "https://pypi.org/simple" }
|
|
||||||
sdist = { url = "https://files.pythonhosted.org/packages/1e/fc/bb52958f7e399250aee093751e9373a6311cadbe76b6e0d109b853757f35/tree_sitter_typescript-0.23.2.tar.gz", hash = "sha256:7b167b5827c882261cb7a50dfa0fb567975f9b315e87ed87ad0a0a3aedb3834d", size = 773053, upload-time = "2024-11-11T02:36:11.396Z" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/28/95/4c00680866280e008e81dd621fd4d3f54aa3dad1b76b857a19da1b2cc426/tree_sitter_typescript-0.23.2-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:3cd752d70d8e5371fdac6a9a4df9d8924b63b6998d268586f7d374c9fba2a478", size = 286677, upload-time = "2024-11-11T02:35:58.839Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/8f/2f/1f36fda564518d84593f2740d5905ac127d590baf5c5753cef2a88a89c15/tree_sitter_typescript-0.23.2-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:c7cc1b0ff5d91bac863b0e38b1578d5505e718156c9db577c8baea2557f66de8", size = 302008, upload-time = "2024-11-11T02:36:00.733Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/96/2d/975c2dad292aa9994f982eb0b69cc6fda0223e4b6c4ea714550477d8ec3a/tree_sitter_typescript-0.23.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4b1eed5b0b3a8134e86126b00b743d667ec27c63fc9de1b7bb23168803879e31", size = 351987, upload-time = "2024-11-11T02:36:02.669Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/49/d1/a71c36da6e2b8a4ed5e2970819b86ef13ba77ac40d9e333cb17df6a2c5db/tree_sitter_typescript-0.23.2-cp39-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e96d36b85bcacdeb8ff5c2618d75593ef12ebaf1b4eace3477e2bdb2abb1752c", size = 344960, upload-time = "2024-11-11T02:36:04.443Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/7f/cb/f57b149d7beed1a85b8266d0c60ebe4c46e79c9ba56bc17b898e17daf88e/tree_sitter_typescript-0.23.2-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:8d4f0f9bcb61ad7b7509d49a1565ff2cc363863644a234e1e0fe10960e55aea0", size = 340245, upload-time = "2024-11-11T02:36:06.473Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/8b/ab/dd84f0e2337296a5f09749f7b5483215d75c8fa9e33738522e5ed81f7254/tree_sitter_typescript-0.23.2-cp39-abi3-win_amd64.whl", hash = "sha256:3f730b66396bc3e11811e4465c41ee45d9e9edd6de355a58bbbc49fa770da8f9", size = 278015, upload-time = "2024-11-11T02:36:07.631Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/9f/e4/81f9a935789233cf412a0ed5fe04c883841d2c8fb0b7e075958a35c65032/tree_sitter_typescript-0.23.2-cp39-abi3-win_arm64.whl", hash = "sha256:05db58f70b95ef0ea126db5560f3775692f609589ed6f8dd0af84b7f19f1cbb7", size = 274052, upload-time = "2024-11-11T02:36:09.514Z" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "typing-extensions"
|
name = "typing-extensions"
|
||||||
version = "4.16.0"
|
version = "4.16.0"
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue