diff --git a/ACTIVE_SLICE.md b/ACTIVE_SLICE.md index 4d6b119..22b7caa 100644 --- a/ACTIVE_SLICE.md +++ b/ACTIVE_SLICE.md @@ -1,13 +1,15 @@ # Milestone state ```text -Last completed milestone: 3 — independent projections -Outcome: Manual output, portable graph artifacts, and the live viewer are independent generation-pinned consumers of the validated graph. -Evidence: Clean candidate f5dccb5e1c312121f1af63780162f593d9363b98; 281 tests and 272 subtests; 3 accessibility flows; clean 1,000-node ten-sample benchmark; isolated wheel proof; no secret-scan findings. +Last completed milestone: 4 — adapter SDK and product documentation. +Evidence: Frozen executable candidate 95271dcf2e48045b9d3aed9b9ea09c7fc155692c; 347 tests and 402 subtests; 3 accessibility flows; offline fresh-wheel adoption; clean 1,002-node graph-plus-Logic benchmark; no secret-scan findings. Active milestone: None. -Next directional milestone: 4 — adapter SDK and product documentation. -Status: Milestone 3 is closed. Milestone 4 has not started. +Outcome: New projects can adopt DocForge incrementally, and users can understand the CLI, MCP server, policies, adapters, renderers, and agent integration without reading implementation code. +Status: Milestone 4 is closed. Milestone 5 has not started. +Next directional milestone: 5 — stabilization and first DocForge2 release. ``` -Milestones 4–5 remain directional context. Do not begin Milestone 4 without a new active-slice -contract. +Milestone 5 remains directional context. Do not begin release stabilization, versioning, tagging, +or publication without a new active-slice contract based on the Milestone 4 evidence. + +The completed Milestone 4 contract and its exclusions are preserved in `SLICE_HISTORY.md`. diff --git a/DEVELOPMENT_NOTES.md b/DEVELOPMENT_NOTES.md index 2548012..78f6be8 100644 --- a/DEVELOPMENT_NOTES.md +++ b/DEVELOPMENT_NOTES.md @@ -784,3 +784,108 @@ proof. Gitleaks 8.30.1 found no findings across the six Milestone 3 commits or c Milestone 3 is complete. No tag, release, production integration repointing, WorldForge change, ScrapeStation change, storage rewrite, or self-hosting dependency was introduced. Milestone 4 remains directional and has not started. + +## Milestone 4 — complete: adapter SDK and product documentation + +Milestone 4 activated from the verified Milestone 3 closeout. Its scope was deliberately additive: +freeze an adapter-authoring boundary, prove narrow language references, make heavy frontends +optional, generate command and client integration evidence, and document adoption. WorldForge, +ScrapeStation, the legacy repository, production bindings, self-hosting, storage replacement, and +release publication remained excluded. + +### Adapter SDK and conformance + +`docforge.adapter_sdk` is now the public authoring facade for typed projections, manifests, source +contributions, complete assemblies, incremental loaders, project bindings, core graph models, and +conformance reports. Complete evidence includes primary graph and function Logic. An incremental +adapter that publishes Logic must provide an independent `load_complete_assembly()` oracle. + +The conformance helper repeats complete loading for determinism, compares the complete assembly +with `load_projection()`, and proves exact complete/incremental graph-plus-Logic parity. It does not +pretend to replace separate confinement, restart, no-AST, corrupt-cache, or retrieval tests. + +An adversarial review found that extraction caches and adapter assemblies lacked aggregate +boundaries. Version-1 cache reads and writes are now regular-file-only, identity-checked, capped at +64,000,000 bytes and 10,000 sources, and fail safely to a miss. Adapter projects enforce primary +node limits and deterministic edge and Logic multipliers before publication. + +### Reference language evidence + +The Python reference uses the standard-library AST and publishes files, modules, classes, +functions, arguments, local static imports, and function Logic. Its manifest uses tokenization +rather than AST parsing, so unchanged warm builds perform no syntax parse. + +JavaScript and TypeScript use separate pinned optional Tree-sitter grammars. They publish files, +modules, classes, functions/methods, static project-relative imports/re-exports, and function +Logic. Their manifests use a closed comment/string-aware ESM scan and do not load Tree-sitter. + +C++ uses explicit non-overlapping source roots and one confined `compile_commands.json` as +translation-unit inventory and fingerprint evidence. It validates entries but never executes a +command, compiler, response file, or project program. It publishes syntax and directly resolvable +project-local quoted includes. Its manifest currently parses for include discovery, so no +zero-warm-parser claim is made. + +None of the references claim compiler-resolved calls, inheritance, types, symbol references, +compiler include semantics, macro semantics, runtime behavior, or semantic ownership. + +The first packaging proof exposed that language parsers were inherited as mandatory dependencies. +The base wheel now depends only on Markdown and MCP packages. Python works in the base wheel; +JavaScript, TypeScript, C++, and all-language extras install their respective Tree-sitter +frontends. Missing extras return a closed error with the exact install target. + +### Fixed reference MCP and launchers + +A strict `.docforge/reference-adapter.toml` selects one fixed reference language, project +identity, and explicit source roots. C++ additionally requires the compilation database. The fixed +`python -m docforge.reference_mcp` binding constructs only repository-owned providers and exposes +the 21 read tools. + +The initial launcher design generated `python -I -m` for arbitrary project module names without +proving that isolated Python could resolve them. The corrected `AdapterLauncherV1` accepts only an +installed project-confined top-level module or the exact trusted `docforge.reference_mcp` +exception. A fixed bounded `find_spec` probe runs under isolated Python without importing project +code. Tests then launch both a real project module and the real reference MCP process. + +Generated Codex, Claude, and OpenClaw fragments bind the exact launcher, descriptor, source +availability, effective and projection policy, interpreter, canonical arguments, empty +environment, artifact bytes, and hashes. There is no arbitrary command, argument list, working +directory, environment, discovery, or callable selector. + +### Generated reference and documentation gates + +CLI tables are derived from the real argparse subcommands. MCP tables are derived from a real +application-enabled server's `list_tools()` registrations, including capability surface, +arguments, descriptions, and input-schema hashes. The generated artifact contains 28 CLI rows and +36 MCP rows. + +An adversarial review found a target-change race between the generator's identity check and +replacement. Publication now locks cooperating generators, uses no-clobber publication for a +missing target, and uses Linux atomic exchange plus displaced-byte and file-identity verification +for an existing target. Raced data is restored or retained for recovery rather than discarded. + +`docs-check` validates generated drift, local files and anchors, one H1 per page, README-rooted +reachability, required Milestone 4 inventory, the generated notice, and every documented +reference-adapter TOML example against the packaged schema. + +### Candidate, adoption, and scale evidence + +Frozen executable candidate `95271dcf2e48045b9d3aed9b9ea09c7fc155692c` passed formatting, +Python and web lint, strict Pyright, compilation, 142 contract tests and 268 subtests, 347 total +tests and 402 subtests, three accessibility flows, lock and dependency checks, package builds, +offline wheel adoption, and every maintained smoke benchmark. + +The offline adoption proof installed the base wheel from the lock without network access. It +proved no Tree-sitter package or module was installed, built and checked a real Python reference +project, performed isolated MCP bootstrap/search/get-node over the exact 21 read tools, and proved +that base-only C++ fails with `optional_dependency_missing` and `docforge[cpp]` remediation. + +The clean 334-source benchmark produced 1,002 primary nodes, 1,001 primary edges, 334 Logic +projections, 2,338 Logic nodes, and 2,338 Logic edges. Complete and incremental output matched +exactly. Warm build p95 was 1,265.387 ms with zero `ast.parse` and zero `extract_source` calls. +Complete equivalence, corrupt cache recovery, and corrupt index recovery were all below 1.9 +seconds. The largest per-operation traced peak was 69,997,166 bytes and process high-water was +78,798,848 bytes. Exact hashes and method are recorded in +`benchmarks/milestone4-2026-07-29.json` and `docs/MILESTONE_4_BASELINE.md`. + +Milestone 4 is complete. No tag or release was created. Milestone 5 remains unstarted until its own +active-slice contract is frozen. diff --git a/Makefile b/Makefile index 64d52c8..203ae12 100644 --- a/Makefile +++ b/Makefile @@ -5,7 +5,7 @@ NPM := npm PYTHONPYCACHEPREFIX := /tmp/docforge-quality-pycache PYTEST_BASETEMP := /tmp/docforge-quality-pytest -.PHONY: accessibility benchmark benchmark-m1 benchmark-m1-smoke benchmark-m2 benchmark-m2-smoke benchmark-m3 benchmark-m3-full benchmark-m3-smoke benchmark-smoke build compile contract dependencies format-check gate lint lock test type +.PHONY: accessibility adoption-m4 benchmark benchmark-m1 benchmark-m1-smoke benchmark-m2 benchmark-m2-smoke benchmark-m3 benchmark-m3-full benchmark-m3-smoke benchmark-m4 benchmark-m4-full benchmark-m4-smoke benchmark-smoke build command-reference-check compile contract dependencies docs-check format-check gate lint lock test type accessibility: $(NPM) run test:accessibility @@ -59,6 +59,17 @@ dependencies: build: $(UV) build +adoption-m4: + $(PYTHON) tools/milestone4_adoption.py + +command-reference-check: + $(PYTHON) tools/generate_command_reference.py \ + --output docs/COMMAND_REFERENCE.md --check > /dev/null + +docs-check: command-reference-check + $(PYTHON) tools/check_documentation.py \ + --pending-inventory tools/docs_pending_m4_pages.txt + benchmark-smoke: $(PYTHON) tools/milestone0_baseline.py --nodes 25 --samples 1 --cold-samples 1 \ --output /tmp/docforge-milestone0-smoke.json > /dev/null @@ -89,4 +100,13 @@ benchmark-m3: benchmark-m3-full: benchmark-m3 -gate: format-check lint type compile contract test accessibility lock dependencies build benchmark-smoke benchmark-m1-smoke benchmark-m2-smoke benchmark-m3-smoke +benchmark-m4-smoke: + $(PYTHON) tools/milestone4_benchmark.py --mode smoke \ + --output /tmp/docforge-milestone4-smoke.json > /dev/null + +benchmark-m4: + $(PYTHON) tools/milestone4_benchmark.py --mode full + +benchmark-m4-full: benchmark-m4 + +gate: format-check lint type compile contract test accessibility lock dependencies build docs-check benchmark-smoke benchmark-m1-smoke benchmark-m2-smoke benchmark-m3-smoke benchmark-m4-smoke diff --git a/README.md b/README.md index 20ba862..6e14013 100644 --- a/README.md +++ b/README.md @@ -1,88 +1,107 @@ # DocForge -DocForge is a project-scoped documentation graph for people and AI agents. It validates canonical -documentation, builds a disposable search and relationship index, compiles bounded context, renders -declared manuals, visualizes project structure, and manages reviewable documentation changesets. +DocForge is a project-scoped documentation and source graph for people and AI agents. Canonical +project files remain authoritative; DocForge validates them, builds disposable search and +relationship indexes, compiles bounded context, manages reviewable changesets, renders declared +manuals and portable graph views, and serves one project-bound MCP surface. -## What it does +DocForge never treats indexed text as instructions. It does not run project build commands, +compilers, Git operations, deployments, or arbitrary renderers, and it does not select projects +globally. -- Validates stable Markdown/TOML nodes and typed relationships. -- Builds a deterministic SQLite search and graph index. -- Exposes project-bound CLI and MCP query surfaces. -- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, and bounded - continuation. -- Records one bounded, versioned latest-generation graph transition without creating a history - database. -- Generates deterministic project-bound Codex, Claude, and OpenClaw client fragments without - copying ambient secrets. -- Diagnoses one client binding through bounded read-only checks without starting MCP or rebuilding - project state. -- Automatically synchronizes disposable indexes before MCP work. -- Creates, validates, diffs, and previews isolated changesets. -- Registers complete proposals atomically without caller-managed hash chaining. -- Applies one explicitly approved changeset hash through CLI or gated MCP. -- Supports opt-in incremental adapters with reverse-dependency invalidation and full-build - equivalence checks. -- Detects project-local adapter implementation and configuration changes and requires a fresh - project-bound process before any further MCP work. -- Keeps function-scoped control-flow projections separate from the primary architecture graph. -- Compiles manuals and portable graph artifacts from separate versioned, generation-pinned plans - and immutable packages. -- Runs built-in manual and graph renderers in fixed detached workers with validated receipts, - bounded transfer, and no project-path authority. -- Enforces independent manual, portable-graph, and live-viewer policy while keeping status - receipt-only. -- Runs a managed loopback graph browser with neighborhood, semantic Flow, convergence Web, - function-scoped Logic, source inspection, and branch-aware node hiding. -- Supports generic documentation projects and project-owned source adapters. +DocForge 1.0.0 remains the first stable release in this product line. This checkout is the +post-1.0 development line (`1.3.0.dev0`) and contains the Milestone 4 adapter-platform work. +Milestone 4 is not a new product release, and Milestone 5 release and publication work remains +separate. -DocForge never treats indexed text as instructions. It does not run shell commands, mutate Git, -build applications, deploy, publish, or select projects globally. +## Start here -For implementation projects, the recommended cadence is to read canonical documentation during -intake, keep it read-only through implementation and focused testing, freeze and validate a release -candidate, then perform one atomic documentation closeout before the final commit and tag. This -keeps the manual authoritative without using it as an implementation notebook. +- New installation or first project: [New-project quickstart](docs/NEW_PROJECT_QUICKSTART.md) +- Mental model and authority: [Core concepts and authority](docs/CORE_CONCEPTS_AND_AUTHORITY.md) +- Complete configuration shape: [Project descriptor](docs/PROJECT_DESCRIPTOR.md) +- Fixed runnable examples: [Reference adapters](docs/REFERENCE_ADAPTERS.md) +- Agent and client setup: [Agent integration](docs/AGENT_INTEGRATION.md) +- Exact live command and tool inventory: [Generated command reference](docs/COMMAND_REFERENCE.md) +- Task-oriented operating guide: [User manual](docs/USER_MANUAL.md) -## Release 1 +## Current capabilities -DocForge 1.0.0 is the first stable product release. It combines the project-scoped graph, CLI and -MCP query surfaces, reviewable hash-approved changesets, generic and project-owned adapters, -declared rendering, and the complete Nodes/Flow/Web visualization model in one supported release. +- Validates Markdown and TOML nodes, stable IDs, typed relationships, project limits, and confined + paths. +- Builds a deterministic, disposable SQLite graph and search index with integrity and generation + evidence. +- Exposes project-bound CLI and MCP read, proposal, application, rendering, and visualization + surfaces according to the startup policy. +- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, bounded + output, and deterministic continuation. +- Creates isolated documentation changesets, validates complete projected graphs, and applies only + one explicitly approved changeset hash through a separately bound canonical applier. +- Supports complete-projection adapters and opt-in incremental adapters with reverse-dependency + invalidation, bounded extraction caches, and a clean full-build equivalence oracle. +- Publishes function-scoped Logic separately from the primary graph for Python, JavaScript, + TypeScript, and C++ integrations that provide it. +- Compiles manual and portable graph plans into immutable packages for fixed detached renderers, + then records bounded receipts. +- Runs a managed loopback graph viewer with Nodes, Flow, Web, lazy Logic, source inspection, and + branch-aware hiding. +- Generates deterministic Codex, Claude, and OpenClaw client fragments without copying ambient + environment values or secrets. -The post-1.0 incremental compiler is a backward-compatible, optional enhancement. Existing Release -1 adapters that implement only `load_projection()` continue to use the original complete-projection -path without modification. Adapters gain incremental performance only when they additionally -implement the source manifest and extraction methods. Incremental adapters must retain -`load_projection()` as their clean-rebuild fallback and equivalence oracle. +## Adapter platform -MCP bindings can additionally select `--no-ast` when an owner wants to preserve an existing -non-AST adapter. The binding advertises that policy to clients, forbids adapter rewrites that add -AST, Tree-sitter, compiler-AST, or function-Logic extraction, blocks the Logic tool, and rejects -nonempty Logic publication. Complete-projection adapters continue unchanged, and non-AST -incremental fingerprinting and caching remain allowed. +Adapter authors use the public `docforge.adapter_sdk` surface for graph types, complete and +incremental contracts, validation helpers, and `verify_adapter_conformance()`. Existing adapters +that implement only `load_projection()` remain supported. Incremental adapters add a manifest and +source extraction while retaining `load_projection()` as the complete graph oracle. An adapter +that publishes Logic incrementally also supplies `load_complete_assembly()` so the complete oracle +covers both graph and Logic. -DocForge2 bindings may also declare -`--capability-mode read|proposal|application|operator`. Bootstrap returns one versioned effective -policy and the actual startup-gated capabilities. Existing tool surfaces and the legacy no-AST -payload remain compatible. +The repository includes bounded reference integrations for: -`docforge_get_task_context` is an additive read tool for `change`, `implementation`, `failure`, -`ownership`, `test`, `operation`, and `release` work. It derives a closed version-1 retrieval plan, -executes it against one immutable index generation, and returns a hash-bound context capsule. -Project relation names remain authoritative. DocForge classifies only its versioned alias set and -preserves every unknown relation as `unclassified` instead of guessing semantics. +- Python, using the standard-library AST and publishing only project-local imports; +- JavaScript and TypeScript, using their distinct optional Tree-sitter grammars and publishing only + project-local static relative imports and re-exports; +- C++, using `compile_commands.json` as translation-unit inventory and fingerprint evidence, without + executing its commands or a compiler, and publishing only directly resolvable project-local + quoted includes. -`docforge_get_generation_diff` reports the latest verified primary-graph transition through one -bounded disposable receipt. It includes exact node and edge change counts, hash-bound retained -details, and explicit truncation. Paged results use one top-level cursor and a versioned -`receipt_header`; `stored_receipt_hash` identifies the complete persisted receipt. The read never -exposes Logic details, loads canonical source, repairs derived state, or invents history. +These integrations demonstrate the adapter contract; they do not claim resolved calls, +inheritance, types, runtime behavior, macro expansion, compiler include semantics, or semantic +ownership. Python works from the base wheel. Install the `javascript`, `typescript`, or `cpp` +extra for the corresponding grammar, or `languages` for all three: -## Independent projections +```bash +uv pip install "/absolute/path/DocForge[javascript]" +uv pip install "/absolute/path/DocForge[typescript]" +uv pip install "/absolute/path/DocForge[cpp]" +uv pip install "/absolute/path/DocForge[languages]" +``` -Manual compilation, portable graph rendering, and the live viewer consume the same validated graph -generation through separate boundaries: +Reference projects use the fixed `.docforge/reference-adapter.toml` descriptor and the installed +`docforge.reference_mcp` module. That server is read-only and exposes no proposal or application +surface. + +Project-owned adapters use `AdapterLauncherV1` and +`generate_adapter_client_configuration()` to produce a client fragment from an explicitly +constructed adapter project. The launcher is immutable and contains no command, caller arguments, +working directory, environment, discovery rule, or callable selector. It resolves through isolated +Python to one installed, project-owned top-level module; the only trusted dotted exception is the +fixed `docforge.reference_mcp` binding. Generic `docforge configure` intentionally refuses custom +adapters because it cannot safely reconstruct project-owned composition. + +See the [Adapter authoring guide](docs/ADAPTER_AUTHORING_GUIDE.md), [Reference +adapters](docs/REFERENCE_ADAPTERS.md), and [Agent integration](docs/AGENT_INTEGRATION.md) for the +supported routes. + +## Authority and projections + +Canonical Markdown, TOML, adapter-declared sources, and descriptor files own project facts. The +SQLite index, extraction cache, render packages, previews, portable artifacts, receipts, client +fragments, and viewer processes are derived and replaceable. + +The primary graph contains project nodes and relationships. Logic is a separate, lazy, +function-scoped control-flow projection. Manual output, portable graph output, and the live viewer +are independent consumers of one validated generation: ```text validated generation @@ -91,190 +110,142 @@ validated generation └── pinned index → managed read-only live viewer ``` -Plans, packages, and receipts are canonical, versioned, hash-identified, bounded, and contain no -project object, SQLite handle, absolute project path, command, or caller-selected renderer module. -Renderers cannot select graph facts, crawl canonical sources, choose publication paths, or mutate -project state. +The version-2 projection policy selects each consumer independently: -Declare portable graph output separately from manual views: - -```toml -[graph_render] -output_root = ".docforge/portable-graph" - -[[graph_render.views]] -id = "architecture" -renderer = "portable_graph_html" -output = "architecture.html" -title = "Architecture" -root = "architecture.overview" -initial_mode = "web" -depth = 3 -max_nodes = 250 -max_edges = 1000 -max_work = 100000 -include_logic = false +```text +manual: auto | explicit | disabled +portable_graph: explicit | disabled +live_viewer: on-demand | disabled ``` -Plan, publish, and inspect it explicitly: +The process capability policy is separate. Capability mode controls the registered read, +proposal, application, or reserved operator surface; descriptor writers and the startup-bound +canonical applier determine whether mutations are actually authorized. `--no-ast` is a restrictive +binding policy over adapter evolution, Logic publication, and Logic retrieval. It is not a parser +inspection mechanism or a filesystem sandbox, and the AST/Tree-sitter reference integrations +should not be presented as no-AST adapters. + +Read [Policy precedence](docs/POLICY_PRECEDENCE.md), [Rendering and +visualization](docs/RENDERING_AND_VISUALIZATION.md), and [Legacy and no-AST +operation](docs/LEGACY_AND_NO_AST.md) before changing those boundaries. + +## Five-minute generic project + +Requirements are Python 3.12 or newer and an installed DocForge environment. From a development +checkout, `uv sync --group dev` creates `.venv`: ```bash -.venv/bin/docforge --project-root "$PROJECT" graph-plan architecture -.venv/bin/docforge --project-root "$PROJECT" graph-render architecture -.venv/bin/docforge --project-root "$PROJECT" graph-render-status architecture -``` - -The version-2 projection policy independently selects manual -`auto|explicit|disabled`, portable graph `explicit|disabled`, and live viewer -`on-demand|disabled`. Use `--manual-render-policy`, `--portable-graph-policy`, and -`--live-viewer-policy` on CLI/MCP startup or generated client configuration. Status remains -available when the corresponding active operation is disabled. - -Manual fragment reuse is disposable. Cold record publication is guarded by byte-exact comparison -with a full detached render; warm records are independently recomputed and validated inside the -worker. Full rendering remains the recovery and equivalence oracle. Portable artifacts commit -content-addressed output and renderer evidence before one bounded generation/view manifest; status -does not plan or render. - -## Graph views - -The browser presents the primary architecture graph through three complementary views and loads a -fourth function-scoped view only when requested: - -- **Nodes** shows a bounded, relation-neutral neighborhood around the focus. It is the broad - inspection view for seeing stored incoming and outgoing relationships without changing their - direction. Semantic cards distinguish structure, behavior, dependencies, execution, data, - evidence, context, and other relationships. -- **Flow** shows semantic origin-to-destination paths that terminate at the focus. DocForge - reverses prerequisite-style relationships for presentation, so imports, dependencies, reads, - inheritance, definitions, and tests flow toward the thing they help create or exercise. -- **Web** shows the larger convergence picture: Flow contributors plus contextual relationships, - callers, containers, and direct members or execution dependencies owned by the focus. -- **Logic** shows the possible static control paths inside a focused Python, JavaScript, or C++ - function or method. Entry, decisions, actions, loops, convergence points, returns, and - exceptions connect through explicit - `TRUE`, `FALSE`, `NEXT`, `CASE`, `LOOP`, `RETURN`, and `RAISE` paths. Logic is stored separately - and does not add statement-level noise to Nodes, Flow, Web, or search. - -Graph cards show the node's readable leaf name and kind without clipping either value. The full -qualified identity remains available in the tooltip, compact descriptor, and full inspector. -The left browser panel can combine text, family, node-kind, language, and capability filters. -Quick presets expose Logic-ready nodes, Python callables, tests, routes, and documentation without -requiring users to know stable IDs. Selecting a canvas node emphasizes its directly connected -neighbors and edges while muting unrelated paths. - -**Hide node** removes noise without changing the index. In Flow and Web, hiding a contributor also -removes upstream ancestors that no longer have a path to the focus. Nodes between the hidden -contributor and the focus stay visible, and alternate ancestor paths remain intact. In Logic, -hiding a step inserts an explicit omitted-path bridge so downstream control flow remains readable. -**Restore hidden** restores the presentation. - -## Five-minute start - -Requirements are Python 3.12+, `uv`, and Node.js/npm. - -```bash -git clone forgejo@repo.andraxion.net:administrator/DocForge2.git /absolute/path/DocForge2 -cd /absolute/path/DocForge2 +git clone /absolute/path/DocForge +cd /absolute/path/DocForge uv sync --group dev -npm ci PROJECT=/absolute/path/MyProject +.venv/bin/docforge --project-root "$PROJECT" onboard +.venv/bin/docforge --project-root "$PROJECT" onboard \ + --scaffold \ + --project-id my-project \ + --title "My Project" +``` + +The first command is read-only. Scaffolding is explicit and create-only: it writes a generic +descriptor, one canonical overview node, and a built-in manual template, then indexes and renders +them. Detected source languages remain `adapter_required` until a validated frontend is selected. + +Operate the configured project: + +```bash .venv/bin/docforge --project-root "$PROJECT" validate .venv/bin/docforge --project-root "$PROJECT" reindex +.venv/bin/docforge --project-root "$PROJECT" search architecture .venv/bin/docforge --project-root "$PROJECT" visualize ``` -Install the persistent per-user graph viewer once: +Install the persistent per-user graph viewer manager once when using the live viewer: ```bash .venv/bin/docforge-viewer-manager install-user-service ``` -Start an MCP server for one project: +Start one generic, read-only MCP server: ```bash .venv/bin/docforge-mcp \ --project-root "$PROJECT" \ - --proposal-writer project-editor + --capability-mode read ``` -Add `--canonical-applier project-editor` only when that MCP integration should expose the -hash-bound `docforge_apply_changeset` tool. - -Preview a read-only Codex fragment and diagnose an installed binding: - -```bash -.venv/bin/docforge configure codex --project "$PROJECT" -.venv/bin/docforge doctor --client codex --project "$PROJECT" -``` - -Pass `--output /absolute/path/docforge.toml` only when creating a standalone fragment. DocForge -never replaces or merges an existing different client file. - -For an unconfigured codebase, begin with a read-only language and documentation assessment: - -```bash -.venv/bin/docforge --project-root /absolute/path/MyProject onboard -``` - -Add `--scaffold`, a stable project ID, and a title to create, index, and render a generic starter -manual. Source files are reported separately and require a validated language frontend before -DocForge describes them as a source graph. +Add a descriptor-authorized proposal writer only when the client should create proposals. Add a +matching `--canonical-applier` only when that integration should expose exact-hash application. +See the [quickstart](docs/NEW_PROJECT_QUICKSTART.md) for the reference-adapter and generated-client +paths. ## Documentation -- [User manual](docs/USER_MANUAL.md) — features, setup, visualization, CLI, MCP, apply, adapters, - and troubleshooting. -- [Core contract](docs/CONTRACT.md) — invariants and security boundary. -- [Milestone 0 compatibility](docs/COMPATIBILITY.md) — preserved package, CLI, MCP, adapter, - schema, changeset, rendering, and no-AST guarantees. -- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) — validation evidence, cold and warm - performance, memory, rendering and response sizes, bottlenecks, and missing coverage. -- [Milestone 0 closeout](docs/MILESTONE_0_CLOSEOUT.md) — lineage, migration, security scan, - repository state, and fresh-clone proof. -- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) — warm operation latency, structured work, - status, retrieval, and memory measurements. -- [Milestone 1 closeout](docs/MILESTONE_1_CLOSEOUT.md) — fast-core contracts, adversarial - validation, compatibility boundaries, and exact candidate evidence. -- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) — task context, generation diff, client - configuration, doctor, response-size, counter, and memory measurements. -- [Milestone 2 closeout](docs/MILESTONE_2_CLOSEOUT.md) — implemented contracts, adversarial - validation, exclusions, and exact candidate evidence. -- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) — manual, fragment, worker, portable graph, - status, equivalence, response-size, and memory measurements. -- [Milestone 3 closeout](docs/MILESTONE_3_CLOSEOUT.md) — independent projection contracts, - adversarial validation, compatibility boundaries, and exact candidate evidence. -- [MCP contract](docs/MCP_CONTRACT.md) — exact tool and process boundary. -- [Viewer manager](docs/VIEWER_MANAGER.md) — native service setup and lifecycle. -- [Adapter decision](docs/APPLICATION_DECISION.md) — why custom adapters own canonical - serialization. -- [Incremental adapter indexing](docs/INCREMENTAL_INDEXING.md) — source-scoped extraction, - invalidation, equivalence, relationship changes, and the lazy Logic boundary. -- [Project onboarding](docs/PROJECT_ONBOARDING.md) — repository assessment, safe manual - scaffolding, language frontends, source/manual integration, proof, and MCP activation. -- [Language adapter authoring](docs/ADAPTER_AUTHORING_GUIDE.md) — implementation sequence, - stable identities, overlap ownership, normalization, incremental equivalence, troubleshooting, - and the complete adapter proof matrix. +### Learn and operate + +- [New-project quickstart](docs/NEW_PROJECT_QUICKSTART.md) +- [Core concepts and authority](docs/CORE_CONCEPTS_AND_AUTHORITY.md) +- [Project descriptor](docs/PROJECT_DESCRIPTOR.md) +- [Policy precedence](docs/POLICY_PRECEDENCE.md) +- [Reference adapters](docs/REFERENCE_ADAPTERS.md) +- [Agent integration](docs/AGENT_INTEGRATION.md) +- [User manual](docs/USER_MANUAL.md) +- [Generated command reference](docs/COMMAND_REFERENCE.md) +- [Project onboarding](docs/PROJECT_ONBOARDING.md) +- [Rendering and visualization](docs/RENDERING_AND_VISUALIZATION.md) +- [Recovery and performance](docs/RECOVERY_AND_PERFORMANCE.md) +- [Security](docs/SECURITY.md) + +### Contracts and compatibility + +- [Core contract](docs/CONTRACT.md) +- [MCP contract](docs/MCP_CONTRACT.md) +- [Compatibility contract](docs/COMPATIBILITY.md) +- [Adapter authoring guide](docs/ADAPTER_AUTHORING_GUIDE.md) +- [Incremental indexing](docs/INCREMENTAL_INDEXING.md) +- [Adapter application decision](docs/APPLICATION_DECISION.md) +- [Legacy and no-AST operation](docs/LEGACY_AND_NO_AST.md) +- [Migrating from version 1](docs/MIGRATING_FROM_V1.md) +- [Viewer manager](docs/VIEWER_MANAGER.md) + +### Milestone evidence + +- [Milestone 4 baseline](docs/MILESTONE_4_BASELINE.md) +- [Milestone 4 closeout](docs/MILESTONE_4_CLOSEOUT.md) +- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) and [closeout](docs/MILESTONE_3_CLOSEOUT.md) +- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) and [closeout](docs/MILESTONE_2_CLOSEOUT.md) +- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) and [closeout](docs/MILESTONE_1_CLOSEOUT.md) +- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) and [closeout](docs/MILESTONE_0_CLOSEOUT.md) + +Historical milestone records preserve the facts and dependency observations of their frozen +candidates. Use the current guides and contracts for present behavior. ## Development -Run the complete repository-native gate: +Run the repository-native gate: ```bash make gate ``` -Focused entry points are available as `make contract`, `make test`, `make type`, -`make benchmark-smoke`, `make benchmark`, `make benchmark-m1-smoke`, and -`make benchmark-m1`. Milestone 2 adds `make benchmark-m2-smoke` and `make benchmark-m2`. -Milestone 3 adds `make accessibility`, `make benchmark-m3-smoke`, `make benchmark-m3`, and -`make benchmark-m3-full`. +Milestone 4 maintenance entry points include: -The committed 1,000-node baseline and its measurement method are under `benchmarks/`. +```bash +make adoption-m4 +make benchmark-m4-smoke +make benchmark-m4 +make benchmark-m4-full +make command-reference-check +make docs-check +``` -Pass `--diagnostics` to `docforge` or `docforge-mcp` to attach bounded request-local stage timings -and compiler-work counters. Diagnostics are disabled by default and are dropped before primary MCP -results when the configured output budget is tight. +`adoption-m4` builds and exercises a fresh base wheel without Tree-sitter packages and proves the +Python reference plus a real isolated read-only MCP retrieval. The optional-language integrations +have their own focused tests and extras. `benchmark-m4` runs the maintained full adapter workload; +the smoke target is for routine gate coverage, not final performance evidence. -See [AGENTS.md](AGENTS.md) before changing core boundaries. +Pass `--diagnostics` to `docforge` or `docforge-mcp` for bounded request-local timings and compiler +work counters. Diagnostics are disabled by default and do not displace a primary result when the +configured output budget is tight. + +Read [AGENTS.md](AGENTS.md) before changing core boundaries. diff --git a/SLICE_HISTORY.md b/SLICE_HISTORY.md index 5b4cd75..0caab02 100644 --- a/SLICE_HISTORY.md +++ b/SLICE_HISTORY.md @@ -1,5 +1,50 @@ # Completed slices +## DocForge2 Milestone 4 adapter SDK and product documentation + +### Changed + +- Added stable `docforge.adapter_sdk` authoring imports and a conformance helper for deterministic + complete and exact complete/incremental primary graph plus Logic equivalence. +- Added bounded Python, JavaScript, TypeScript, and C++ reference integrations with explicit + unsupported-fact reports and optional heavy frontends. +- Added a closed `.docforge/reference-adapter.toml` contract and fixed read-only reference MCP + server. +- Added immutable project-bound launcher declarations and deterministic Codex, Claude, and OpenClaw + configuration generation for custom adapters. +- Generated CLI and MCP references from live implementation metadata and added race-safe + publication plus repository drift checks. +- Added strict documentation-graph and fresh-wheel adoption gates and dedicated onboarding, + authority, descriptor, policy, adapter, legacy/no-AST, rendering, agent-integration, security, + recovery/performance, and v1 migration guides. + +### Verification + +- The executable candidate passed 347 tests and 402 subtests, 142 contract tests and 268 subtests, + three accessibility flows, lint, strict types, compilation, dependencies, builds, adoption, and + every maintained smoke benchmark. +- The offline base wheel contained no Tree-sitter distribution and ran a real Python reference + build/check and isolated 21-tool MCP retrieval flow. Missing C++ support failed with the exact + `docforge[cpp]` remediation. +- The clean 334-source benchmark produced 1,002 nodes and exact full/incremental graph and Logic + evidence, warm zero Python parsing/extraction, and exact corrupt-cache and corrupt-index + recovery below every latency, memory, and response gate. +- Gitleaks found no findings in the Milestone 4 commit range or candidate tree. + +### Limits + +- The reference integrations publish syntax and local static relationships. They do not claim + compiler-resolved calls, types, inheritance, macros, runtime behavior, or semantic ownership. +- C++ compilation-database commands are inert and never executed. +- C++ manifest include discovery uses Tree-sitter and makes no zero-warm-parser claim. +- No WorldForge, ScrapeStation, legacy repository, production binding, language distribution, + self-hosting, tag, or release was changed. + +### Next gate + +Milestone 5 remains directional. Create a new active-slice contract before stabilization, +versioning, tagging, or publication. + ## DocForge2 Milestone 3 independent projections ### Changed diff --git a/benchmarks/README.md b/benchmarks/README.md index 2141f1a..acfedd2 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -42,6 +42,14 @@ make benchmark-m3 make benchmark-m3-full ``` +Run the Milestone 4 adapter SDK smoke, maintained full, and fresh-wheel adoption gates: + +```bash +make benchmark-m4-smoke +make benchmark-m4-full +make adoption-m4 +``` + The benchmark creates canonical sources, derived state, changesets, rendered output, and caches only in a disposable temporary directory. It does not read another project, self-host DocForge, or mutate repository content. @@ -71,6 +79,13 @@ sweeps, add/change/delete/reorder equivalence, portable publication, receipt-onl memory, detached worker peak memory, and response size. Its interpretation is in [`docs/MILESTONE_3_BASELINE.md`](../docs/MILESTONE_3_BASELINE.md). +`milestone4-2026-07-29.json` is the clean-tree adapter SDK and reference-Python baseline captured +from commit `95271dcf2e48045b9d3aed9b9ea09c7fc155692c`. It measures cold and warm +incremental builds, complete/incremental graph-plus-Logic equivalence, corrupt extraction-cache +recovery, corrupt-index recovery, parser and extraction counters, traced and process memory, and +bounded response size. Its interpretation is in +[`docs/MILESTONE_4_BASELINE.md`](../docs/MILESTONE_4_BASELINE.md). + The generic fixtures expose whole-source and projection scaling. They do not replace incremental adapter equivalence tests. Milestone 3's portable fixture contains 1,000 Nodes/Flow/Web nodes and 999 edges; portable version 1 deliberately excludes Logic. @@ -89,4 +104,5 @@ backlinks, both traversal directions, paged context, render receipt states, and freshness. The reported p95 uses the nearest-rank method; with ten samples it is the maximum. `process_peak_rss_kib` is the cumulative main-process `RUSAGE_SELF` high-water mark and excludes the detached viewer worker. It is diagnostic and not operation-local. Milestone 3 memory gates use -per-operation `tracemalloc` peaks and detached worker receipt peaks instead. +per-operation `tracemalloc` peaks and detached worker receipt peaks instead. Milestone 4 uses +per-operation `tracemalloc` peaks and a cumulative benchmark-process high-water gate. diff --git a/benchmarks/milestone4-2026-07-29.json b/benchmarks/milestone4-2026-07-29.json new file mode 100644 index 0000000..2f993d3 --- /dev/null +++ b/benchmarks/milestone4-2026-07-29.json @@ -0,0 +1,205 @@ +{ + "benchmark": "docforge2_milestone4", + "environment": { + "implementation": "CPython", + "machine": "x86_64", + "platform": "Linux-7.1.3-200.nobara.fc44.x86_64-x86_64-with-glibc2.43", + "python": "3.14.6" + }, + "evidence": { + "cache_recovery_exact": true, + "complete_assembly_hash": "2888182ab765fbffe3ba873c1613345640e8e6d89be74ddfca7452c0a5056345", + "edge_count": 1001, + "edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35", + "full_incremental_graph_and_logic_exact": true, + "index_recovery_exact": true, + "logic_edge_count": 2338, + "logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c", + "logic_node_count": 2338, + "logic_projection_count": 334, + "node_count": 1002, + "node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f", + "source_count": 334, + "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", + "warm_zero_ast_parse": true, + "warm_zero_source_extraction": true + }, + "evidence_sha256": "4a8db461a311b5df4abd5aa00063e9a347d8b9dba19b30e35684f561d5271549", + "fixture": { + "expected_node_count": 1002, + "kind": "synthetic_python_reference_adapter", + "nodes_per_source": 3, + "source_count": 334 + }, + "memory": { + "process_peak_bytes": 78798848, + "process_peak_limit_bytes": 536870912 + }, + "method": { + "ast_instrumentation": "temporary counter around stdlib ast.parse", + "clock": "time.perf_counter_ns", + "extraction_instrumentation": "reference adapter extract_source entry counter", + "full_mode_source_requirement": 334, + "in_process_peak_memory": "tracemalloc per measured invocation", + "process_peak_memory": "resource.getrusage(RUSAGE_SELF).ru_maxrss", + "response_size": "UTF-8 bytes of compact sorted JSON", + "samples": 3, + "threshold_basis": "Regression tripwires leave several times the Milestone 3 1,000-node allowances for AST, Logic, extraction-cache, and SQLite work." + }, + "mode": "full", + "operations": { + "cold_incremental_build": { + "max_ms": 1599.76, + "maximum_response_bytes": 14580, + "maximum_traced_peak_bytes": 68573540, + "median_ms": 1599.76, + "min_ms": 1599.76, + "p95_limit_ms": 30000.0, + "p95_ms": 1599.76, + "response_limit_bytes": 262144, + "samples": 1, + "stable_result": { + "ast_parse_calls": 668, + "cache_hits": 0, + "deleted_sources": 0, + "edge_count": 1001, + "edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35", + "extraction_calls": 334, + "invalidated_sources": 334, + "logic_edge_count": 2338, + "logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c", + "logic_node_count": 2338, + "logic_projection_count": 334, + "node_count": 1002, + "node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f", + "reparsed_sources": 334, + "revision": "30ef23aa4106", + "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", + "status": "ok", + "total_sources": 334 + }, + "traced_peak_limit_bytes": 268435456 + }, + "corrupt_extraction_cache_recovery": { + "max_ms": 1695.406, + "maximum_response_bytes": 14580, + "maximum_traced_peak_bytes": 69776923, + "median_ms": 1695.406, + "min_ms": 1695.406, + "p95_limit_ms": 30000.0, + "p95_ms": 1695.406, + "response_limit_bytes": 262144, + "samples": 1, + "stable_result": { + "ast_parse_calls": 668, + "cache_hits": 0, + "deleted_sources": 0, + "edge_count": 1001, + "edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35", + "extraction_calls": 334, + "invalidated_sources": 334, + "logic_edge_count": 2338, + "logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c", + "logic_node_count": 2338, + "logic_projection_count": 334, + "node_count": 1002, + "node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f", + "reparsed_sources": 334, + "revision": "30ef23aa4106", + "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", + "status": "ok", + "total_sources": 334 + }, + "traced_peak_limit_bytes": 268435456 + }, + "corrupt_index_recovery": { + "max_ms": 1707.879, + "maximum_response_bytes": 14779, + "maximum_traced_peak_bytes": 68561924, + "median_ms": 1707.879, + "min_ms": 1707.879, + "p95_limit_ms": 30000.0, + "p95_ms": 1707.879, + "response_limit_bytes": 262144, + "samples": 1, + "stable_result": { + "action": "rebuilt", + "ast_parse_calls": 0, + "cache_hits": 334, + "edge_count": 1001, + "edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35", + "extraction_calls": 0, + "initial_error_code": "invalid_index", + "logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c", + "logic_projection_count": 334, + "node_count": 1002, + "node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f", + "reparsed_sources": 0, + "revision": "30ef23aa4106", + "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", + "status": "ok" + }, + "traced_peak_limit_bytes": 268435456 + }, + "full_incremental_equivalence": { + "max_ms": 1897.919, + "maximum_response_bytes": 230, + "maximum_traced_peak_bytes": 64305919, + "median_ms": 1897.919, + "min_ms": 1897.919, + "p95_limit_ms": 30000.0, + "p95_ms": 1897.919, + "response_limit_bytes": 262144, + "samples": 1, + "stable_result": { + "ast_parse_calls": 1336, + "edge_count": 1001, + "extraction_calls": 668, + "logic_projection_count": 334, + "node_count": 1002, + "project_id": "milestone4-python-benchmark", + "revision": "30ef23aa4106", + "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", + "status": "ok" + }, + "traced_peak_limit_bytes": 268435456 + }, + "warm_incremental_build": { + "max_ms": 1265.387, + "maximum_response_bytes": 14578, + "maximum_traced_peak_bytes": 69997166, + "median_ms": 1244.427, + "min_ms": 1239.941, + "p95_limit_ms": 20000.0, + "p95_ms": 1265.387, + "response_limit_bytes": 262144, + "samples": 3, + "stable_result": { + "ast_parse_calls": 0, + "cache_hits": 334, + "deleted_sources": 0, + "edge_count": 1001, + "edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35", + "extraction_calls": 0, + "invalidated_sources": 0, + "logic_edge_count": 2338, + "logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c", + "logic_node_count": 2338, + "logic_projection_count": 334, + "node_count": 1002, + "node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f", + "reparsed_sources": 0, + "revision": "30ef23aa4106", + "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", + "status": "ok", + "total_sources": 334 + }, + "traced_peak_limit_bytes": 268435456 + } + }, + "schema_version": 1, + "source": { + "dirty": false, + "revision": "95271dcf2e48045b9d3aed9b9ea09c7fc155692c" + } +} diff --git a/docs/ADAPTER_AUTHORING_GUIDE.md b/docs/ADAPTER_AUTHORING_GUIDE.md index f77c3fe..df37df7 100644 --- a/docs/ADAPTER_AUTHORING_GUIDE.md +++ b/docs/ADAPTER_AUTHORING_GUIDE.md @@ -8,6 +8,30 @@ Use this guide after the repository assessment in [Project Onboarding](PROJECT_ONBOARDING.md). The onboarding checklist decides whether an adapter is needed. This guide defines how to build and prove one. +## Public authoring surface + +Adapter authors should import typed contracts, validation helpers, graph models, and the +conformance helper from `docforge.adapter_sdk`: + +```python +from docforge.adapter_sdk import ( + AdapterAssembly, + AdapterImplementation, + AdapterManifest, + AdapterProject, + AdapterProjectSettings, + AdapterProjection, + AdapterSource, + AdapterSourceProjection, + LogicProjection, + verify_adapter_conformance, +) +``` + +Language-specific modules under `docforge.adapters` are repository reference implementations, not +the general authoring namespace. Their exact, deliberately narrow behavior is documented in +[Reference Adapters](REFERENCE_ADAPTERS.md). + ## Required outcome A production adapter must provide one reproducible public graph from authoritative project @@ -51,11 +75,39 @@ class MyAdapter: manifest: AdapterManifest, contributions: tuple[AdapterSourceProjection, ...], ) -> AdapterAssembly: ... + def load_complete_assembly(self) -> AdapterAssembly: ... def load_projection(self) -> AdapterProjection: ... ``` `assemble_projection()` is optional only when extraction units already have disjoint ownership. -`load_projection()` is always required. It is the clean rebuild and equivalence oracle. +`load_projection()` is always required. It is the clean primary-graph compatibility oracle. +`load_complete_assembly()` supplies the cache-independent complete graph-plus-Logic oracle. It is +required when incremental contributions publish Logic; without it, DocForge cannot prove Logic +parity. + +## What the conformance helper proves + +Run the public helper against a fresh confined cache root: + +```python +report = verify_adapter_conformance( + MyAdapter(project_root), + cache_root=project_root / ".docforge" / "cache" / "adapter-conformance", +) +``` + +`verify_adapter_conformance()` proves: + +- two repeated complete assemblies are exactly deterministic, including Logic; +- when `load_complete_assembly()` exists, its primary graph exactly matches + `load_projection()`; and +- when the incremental methods exist, the assembled incremental graph and Logic exactly match the + independent complete oracle. + +The report records stable identity, counts, and an assembly hash. This helper does not by itself +prove process confinement, implementation restart behavior, corrupt-cache recovery, warm zero +parsing, retrieval, or every case in the proof matrix below. Keep those as separate focused and +integration tests. ## Step 1: define authority before parsing @@ -324,6 +376,25 @@ The manifest remains a current source snapshot, not a Git-index snapshot. A Git- omit a deleted source whether its deletion is unstaged or staged. Staging is never a required DocForge synchronization step. +### Current aggregate bounds + +DocForge limits one extraction-cache generation to 10,000 source contributions and 64,000,000 +encoded bytes. Malformed, incompatible, missing, oversized, or unsafe cache data is treated as a +cache miss. + +The assembled graph is bounded by the effective project `Limits.max_nodes`. When adapter settings +do not override limits, DocForge selects at least 10,000 nodes and raises that bound to the +manifest's `estimated_nodes` when larger. Aggregate assembly ceilings are: + +- nodes: `max_nodes`; +- relationships: `max_nodes * 32`; +- Logic nodes across all functions: `max_nodes * 32`; and +- Logic edges across all functions: `max_nodes * 64`. + +Adapters should set realistic limits and fail before retaining unbounded frontend evidence. The +cache and assembly bounds do not replace each adapter's own bounded source, command, parser, or +per-function limits. + ## Step 9: keep the complete path independent The full rebuild must not read the incremental extraction cache. Otherwise equivalence compares @@ -470,7 +541,8 @@ For every such change: - [ ] Two independent complete builds match exactly. - [ ] Incremental extraction uses authoritative dependencies. - [ ] The complete oracle is independent of the cache. -- [ ] Complete and incremental public projections match exactly. +- [ ] Complete and incremental public graph and Logic projections match exactly. +- [ ] `verify_adapter_conformance()` passes, and the separate proof-matrix cases also pass. - [ ] Corrupt, missing, and interrupted cache cases fail safely. - [ ] Session composition and family isolation are proven. - [ ] Viewer, query, context, and Logic retrieval are proven. diff --git a/docs/AGENT_INTEGRATION.md b/docs/AGENT_INTEGRATION.md new file mode 100644 index 0000000..0f5ae3b --- /dev/null +++ b/docs/AGENT_INTEGRATION.md @@ -0,0 +1,105 @@ +# Agent integration + +DocForge generates deterministic, project-bound MCP client fragments for Codex, Claude, and +OpenClaw. Generic projects and custom adapters use different configuration APIs, but both produce +fixed standard-input/output bindings with empty generated environments and explicit capability +policy. + +## Fixed reference binding + +Configure one of the in-repository adapters with +`.docforge/reference-adapter.toml` as described in +[Reference Adapters](REFERENCE_ADAPTERS.md), then start: + +```bash +python -I -m docforge.reference_mcp \ + --project-root /absolute/path/to/project \ + --capability-mode read +``` + +`docforge.reference_mcp` is a fixed trusted module in the installed DocForge distribution. It +selects a provider only from the validated reference configuration and reports binding metadata +containing: + +- `server_module = "docforge.reference_mcp"`; +- `adapter_mode = "reference"`; +- the selected `reference_language`; and +- the exact `reference_config_hash`. + +The reference binding registers exactly the 21 read tools listed in +[MCP Boundary](MCP_CONTRACT.md#read-tools). It has no proposal or canonical-application surface. + +## Generate a reference client fragment + +The public launcher and generator APIs are: + +```python +from pathlib import Path + +from docforge.adapter_launcher import AdapterLauncherV1 +from docforge.client_config import generate_adapter_client_configuration +from docforge.reference_mcp import REFERENCE_MCP_MODULE, create_reference_project + +root = Path("/absolute/path/to/project") +project = create_reference_project(root) +launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE) + +plan = generate_adapter_client_configuration( + project, + launcher, + "codex", # "codex", "claude", or "openclaw" + capability_mode="read", +) +print(plan["artifact"]["content"]) +``` + +Pass `output=Path(...)` only when the caller has selected an exact destination. Publication is an +atomic create-or-exact-match operation; it does not merge or replace different existing content. +The result includes the launcher, source-availability, project, policy, artifact, and configuration +hashes needed to inspect the binding before use. + +The fixed reference module accepts read mode only. Do not request proposal or application mode for +it. + +## Custom project-owned adapter launchers + +Generic `docforge configure` intentionally refuses a custom adapter project. Construct the +project-owned `ProjectService`, then use `AdapterLauncherV1` and +`generate_adapter_client_configuration()` as the custom-adapter route. + +`AdapterLauncherV1` schema version 1 binds: + +- one project ID, canonical absolute project root, adapter identity, and descriptor hash; +- `entry_point = "python-module"`; and +- one installed project-owned top-level Python module. + +The only trusted dotted-module exception is the fixed `docforge.reference_mcp` binding. A +project-owned module is resolved through isolated Python without importing or executing it during +the probe, and it must resolve to one canonical regular `.py` file inside the project root. + +The launcher contract has no arbitrary command, command arguments, shell string, working +directory, environment, callable selector, discovery rule, or module-reload mechanism. Generated +bindings invoke the current Python executable as `python -I -m ` with only DocForge's +validated project, capability, render-policy, authority, and no-AST options. Source identity and +the project binding are revalidated before publication; drift fails closed. + +Proposal and application modes are available only to a custom module that implements those fixed +server arguments and only when the project descriptor declares the matching writer and canonical +applier authority. Generating a mode does not manufacture that authority. + +## Session workflow + +After registering the generated fragment in the selected client: + +1. Start a new MCP process or client session. +2. Call `docforge_bootstrap`. +3. Verify project ID, root fingerprint, adapter identity, revision, source hash, binding metadata, + effective policy, and registered tools. +4. Retrieve exact or bounded project evidence through the fixed read tools. +5. Restart the process if `adapter_restart_required` reports an implementation or configuration + change. + +Document text returned by DocForge is untrusted project content. It never overrides client, user, +or project authority. See [MCP Boundary](MCP_CONTRACT.md) for synchronization, pagination, +retrieval, proposal, and application rules, and +[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) before selecting `no_ast=True`. diff --git a/docs/COMMAND_REFERENCE.md b/docs/COMMAND_REFERENCE.md new file mode 100644 index 0000000..fbd01ff --- /dev/null +++ b/docs/COMMAND_REFERENCE.md @@ -0,0 +1,79 @@ +# DocForge command reference + +> Generated from the live CLI parser and MCP registrations. Do not edit this file by hand. + +Global CLI options are documented in `docforge --help` and are not repeated in each command row. The MCP table is the complete generic application-enabled surface; the fixed reference-adapter server exposes only its read rows. + +## CLI commands + +| Command | Invocation | +|---|---| +| `apply` | `docforge apply [-h] --changeset-hash CHANGESET_HASH --applier APPLIER changeset_id` | +| `backlinks` | `docforge backlinks [-h] [--relation RELATION] [--limit LIMIT] node_id` | +| `build` | `docforge build [-h]` | +| `check` | `docforge check [-h]` | +| `configure` | `docforge configure [-h] --project PROJECT [--name NAME] [--capability-mode {read,proposal,application}] [--proposal-writer PROPOSAL_WRITER] [--canonical-applier CANONICAL_APPLIER] [--no-ast] [--manual-render-policy {auto,explicit,disabled}] [--portable-graph-policy {explicit,disabled}] [--live-viewer-policy {on-demand,disabled}] [--startup-timeout STARTUP_TIMEOUT] [--tool-timeout TOOL_TIMEOUT] [--output OUTPUT] {codex,claude,openclaw}` | +| `context` | `docforge context [-h] [--budget BUDGET] [--limit LIMIT] [--cursor CURSOR] profile` | +| `dependencies` | `docforge dependencies [-h] [--depth DEPTH] [--limit LIMIT] node_id` | +| `doctor` | `docforge doctor [-h] --client {codex,claude,openclaw} [--project PROJECT] [--config CONFIG] [--server-name SERVER_NAME]` | +| `filter` | `docforge filter [-h] [--family FAMILY] [--authority AUTHORITY] [--status STATUS] [--tag TAG] [--limit LIMIT]` | +| `generation-diff` | `docforge generation-diff [-h] [--limit LIMIT] [--cursor CURSOR]` | +| `graph-plan` | `docforge graph-plan [-h] view_id` | +| `graph-render` | `docforge graph-render [-h] view_id` | +| `graph-render-status` | `docforge graph-render-status [-h] [view_id]` | +| `impact` | `docforge impact [-h] [--depth DEPTH] [--limit LIMIT] node_id` | +| `info` | `docforge info [-h]` | +| `onboard` | `docforge onboard [-h] [--language LANGUAGE] [--scaffold] [--project-id PROJECT_ID] [--title TITLE] [--content-root CONTENT_ROOT]` | +| `preview` | `docforge preview [-h] changeset_id view_id` | +| `reindex` | `docforge reindex [-h]` | +| `render` | `docforge render [-h] view_id` | +| `render-status` | `docforge render-status [-h] [--deep] [view_id]` | +| `search` | `docforge search [-h] [--limit LIMIT] query` | +| `show` | `docforge show [-h] node_id` | +| `sync` | `docforge sync [-h]` | +| `validate` | `docforge validate [-h]` | +| `validate-index` | `docforge validate-index [-h]` | +| `visualization-status` | `docforge visualization-status [-h]` | +| `visualization-stop` | `docforge visualization-stop [-h]` | +| `visualize` | `docforge visualize [-h] [--node NODE \| --query QUERY] [--depth DEPTH] [--no-open]` | + +## MCP tools + +| Surface | Tool | Required arguments | Optional arguments | Input schema SHA-256 | Description | +|---|---|---|---|---|---| +| read | `docforge_backlinks` | `node_id` | `limit`, `relation` | `bf9a5936e2d1ee70897dc0795ec9d716be63c9447e6fdcd3890d5dd92f45f690` | Return bounded incoming relationships for one exact stable node. | +| read | `docforge_bootstrap` | — | — | `af6c17a59a7713cfb4497f0f66b05efc091abbfe109573bb86d673e39b9b7413` | Synchronize and report the complete fixed project binding and workflow. | +| read | `docforge_dependencies` | `node_id` | `depth`, `limit` | `1283a8a030cf9369493a13e2aa87317d371470904a397910bc0b52997b3ec3b8` | Traverse declared depends_on relationships within the configured depth limit. | +| read | `docforge_filter_nodes` | — | `authority`, `family`, `limit`, `status`, `tag` | `2a51c4d8f4d76d89a2e7f90b3b06d14c7e16303b9c81e62af6747286b58cf68b` | Filter current nodes deterministically by validated metadata. | +| read | `docforge_get_context` | `profile` | `budget`, `cursor`, `limit` | `2f164828c2891a88d0ea7173790a72cc6f3da1279f14da2ca495e8989880b298` | Compile bounded cited context from one configured profile with explicit omissions. | +| read | `docforge_get_contract` | — | — | `19eb6785523ed9db645d17f61df2b8b584ac10ad308199e9a96e6e176a87169e` | Report canonical and derived boundaries plus allowed and excluded operations. | +| read | `docforge_get_generation_diff` | — | `cursor`, `limit` | `0a2fccfea4bf5d78481d40b49e7c812ded9fbe9c93cb79296ede78f22b259fcf` | Return the latest bounded primary-graph generation transition. | +| read | `docforge_get_logic` | `owner_node_id` | — | `56909ee4d72143b470528ea2b3544325e1e1920dbcc758f3caee4a4924f904a1` | Return the lazy control-flow projection owned by one function or method. | +| read | `docforge_get_node` | `node_id` | — | `52408b0c3a297e7b5cf96b293fd6d4625f05c4b090c02bfe806d25ce6c2d1c68` | Return one exact stable node from the current validated project index. | +| read | `docforge_get_task_context` | `task`, `task_kind` | `budget`, `cursor`, `focus_node_id`, `limit` | `0ffbc030bbe5cca2ae24920ec812ac976be9090da4fa2887d87526aba64a21f7` | Return one bounded task-shaped context capsule with explicit evidence gaps. | +| read | `docforge_graph_plan` | `view_id` | — | `7a014ba692f57badca9ca2098b88f82f4749ada6d71f5a3b8bc141abc86645f9` | Plan one declared portable graph without publishing derived output. | +| read | `docforge_graph_render_status` | — | `view_id` | `1d2f3604b0151ba2e228cc6ce1829d7f4244193b53d85df46eb6a3e608c59564` | Report portable-graph publication state without rendering. | +| read | `docforge_impact` | `node_id` | `depth`, `limit` | `1c71db5bb4e1afe180d98dd5000a757b7e3fd1a048b5f68af13d9b7ac56e2ce2` | Traverse bounded incoming relationships and report exact paths. | +| read | `docforge_project_info` | — | — | `a42973543d5e6f4147953d175f6aec43d228cd6bd1a63f3ca4e84a51ad7e8a64` | Report the fixed project identity, revision, source hash, and index health. | +| read | `docforge_render_status` | — | `deep`, `view_id` | `5ea5683bd74e685d4cabb1c960a66eff23a001bd2c68e1af350c6e46e936b9fd` | Report receipt state, or explicitly recompute the side-effect-free render oracle. | +| read | `docforge_search` | `query` | `limit` | `735fff8bd82b20730b0907db8bf61eb0de9fff240ef4fa239ce555779ae49047` | Run bounded lexical search over the current validated project index. | +| read | `docforge_stop_visualization` | — | — | `066a6e34019f4abcd3523f24d9e447775e1479adbea1134341f434a6bb5f8d35` | Explicitly stop this project's persistent read-only graph browser. | +| read | `docforge_sync` | — | — | `a2ac50c03b9b0e08b271216f76920279563ed264bef06d6791695c68dcb3b2a2` | Ensure the disposable project index matches current canonical sources. | +| read | `docforge_validate_project` | — | — | `3192ba97a6d7fff13008c5ed3bfa549d1eeea95332579592d96b8fd326ad06f7` | Validate current canonical sources and graph without writing any project file. | +| read | `docforge_visualization_status` | — | — | `03751f891e0a9318355a6b28be551eec6009c0f96f05ff99536e2f401b76f87f` | Report this project's managed graph browser lifecycle state. | +| read | `docforge_visualize` | — | `depth`, `node_id`, `query` | `cdc9669ba934e591379ea684934d99e2e80ba3f181d6c58867774a8064c32f4b` | Start the fixed read-only graph browser for this configured project. | +| proposal | `docforge_abandon_changeset` | `changeset_id`, `expected_changeset_hash`, `reason` | — | `8813e40ffc12b1ae3d9abe73db8de214085a4a8042323146283fffa8f7476a62` | Mark one proposal abandoned while preserving its audit record. | +| proposal | `docforge_create_changeset` | `changeset_id` | — | `0de93025b9caaa9e33b4b2b3bcfabdde087c944daba48dd77ae3c8f757c52ab8` | Create an empty hash-bound proposal under the configured isolated changeset root. | +| proposal | `docforge_get_changeset` | `changeset_id` | `cursor`, `limit` | `919737a8e732cf95305a2964cddf98f1fdce9ce2cfb52ff659436c244a38495d` | Inspect a stored proposal even when its canonical base has become stale. | +| proposal | `docforge_get_changeset_diff` | `changeset_id` | `cursor`, `limit` | `5cc1d2b035f37e3866766cfa0c3541b144762b764bf455a641ab459df03f8a59` | Return a deterministic structured and textual diff without applying the proposal. | +| proposal | `docforge_list_changesets` | — | `cursor`, `include_history`, `limit`, `status` | `b96519477781cb20acbfc9502a4b7d64a9119e0bd8d71e27290e90677aaf9ced` | List active proposals by default, with optional lifecycle history. | +| proposal | `docforge_preview_changeset` | `changeset_id`, `view_id` | — | `d84e1710a5cac52607037a7e581fcb221c42d3776f510eefe42fb6779fee7fb3` | Render one validated changeset through a declared view into its isolated preview path. | +| proposal | `docforge_propose_node_create` | `changeset_id`, `content`, `expected_changeset_hash`, `metadata`, `node_id`, `rationale`, `relationship_changes`, `target_source` | — | `8953b502b9d8d9c097935bae51fce4e830286eace9df3cb6c37b7e0542be5a4e` | Append one validated node creation without writing its canonical target. | +| proposal | `docforge_propose_node_delete` | `changeset_id`, `expected_changeset_hash`, `expected_content_hash`, `node_id`, `rationale`, `relationship_changes` | — | `8fb6a22e5c1ead3793384652e4b61e75cc23c2725906542533a121401f7faaee` | Append one validated deletion with explicit incident relationship removals. | +| proposal | `docforge_propose_node_move` | `changeset_id`, `expected_changeset_hash`, `expected_content_hash`, `node_id`, `rationale`, `target_source` | — | `32ef6fc4b9795a6a8e30f4f81f63d84d7350e983773f796524382d71cb870a67` | Append one validated same-format node move without moving a canonical file. | +| proposal | `docforge_propose_node_update` | `changeset_id`, `content`, `expected_changeset_hash`, `expected_content_hash`, `metadata`, `node_id`, `rationale`, `relationship_changes` | — | `6a29a73b9e50c9fe22cb6d056bbcdacef30c93c0822c8a9276ef33f9e51e7728` | Append one validated node update without changing canonical content. | +| proposal | `docforge_propose_relationship_update` | `changeset_id`, `expected_changeset_hash`, `expected_content_hash`, `node_id`, `rationale`, `relationship_changes` | — | `8e21fc7ba35289940dd7f12294b14e8553163ab901002dcd4a3998cdb33556c3` | Queue hash-bound relationship changes without rewriting node content. | +| proposal | `docforge_rebase_changeset` | `changeset_id`, `expected_changeset_hash` | — | `505434b2f9f611649221b95d0251db347c44752d1c73e5d69e1abfd916966788` | Safely rebase a proposal when every touched fact remains unchanged. | +| proposal | `docforge_register_changes` | `changeset_id`, `operations` | — | `c5a7d6ac07d3d3b95c13b0c57ad9b69360f2f063b1c8e22dd5f64f6d3a83dc5b` | Atomically register and validate a complete hash-bound proposal. | +| proposal | `docforge_validate_changeset` | `changeset_id` | `cursor`, `limit` | `cb9df5104dbefc3382900ec812ea581a43612d1577619071f1f03674041652c7` | Validate a proposal against its exact canonical base and other active proposals. | +| application | `docforge_apply_changeset` | `changeset_id`, `expected_changeset_hash` | — | `80259f5001f36b14f960c10bd94751d1431077b72e081fb5a5c806b79073b085` | Apply one exact validated changeset and refresh declared derived state. | diff --git a/docs/COMPATIBILITY.md b/docs/COMPATIBILITY.md index 3925f59..8eca1c9 100644 --- a/docs/COMPATIBILITY.md +++ b/docs/COMPATIBILITY.md @@ -31,16 +31,31 @@ model, index, rendering, application, and MCP factory names imported from these remain supported: - `docforge.adapter_contract` +- `docforge.adapter_sdk` +- `docforge.adapter_launcher` - `docforge.application` +- `docforge.client_config` - `docforge.index` - `docforge.mcp_server` - `docforge.models` - `docforge.policy` - `docforge.render_contract` +- `docforge.reference_config` +- `docforge.reference_mcp` Names beginning with an underscore are implementation details. New public names may be added without breaking this contract. +The repository Python, JavaScript/TypeScript, and C++ adapters are supported reference +implementations. Their documented configuration, evidence limits, and unsupported-fact reports are +compatibility surfaces; their internal parser helpers are not adapter-authoring imports. + +Milestone 4 adds three version-1 schemas without changing existing descriptor or result schemas: + +- `schemas/adapter-launcher.schema.json` +- `schemas/adapter-client-configuration.schema.json` +- `schemas/reference-adapter.schema.json` + ## CLI and MCP surfaces Existing `docforge` command names and arguments remain supported. Existing `docforge-mcp` tool @@ -233,6 +248,11 @@ Milestone 3 adds `ManualRenderPlanV1`, `GraphViewPlanV1`, projection package and and projection policy version 2. These are additive submodule and schema contracts. They do not change the legacy task-context, adapter, changeset, or effective-policy contracts described above. +Milestone 4 adds `docforge.adapter_sdk`, the fixed reference configuration and read-only MCP +binding, immutable adapter launchers, and generated adapter client fragments. A legacy adapter with +only `load_projection()` remains first-class and need not adopt incremental extraction, Logic, a +reference configuration, or launcher metadata. + ## Safety boundary DocForge remains bound to one explicit project root. It rejects absolute paths, root escapes, and @@ -245,8 +265,9 @@ publication, or project switching. Milestone 0 records rather than redesigns these areas: - Generic warm reads still repeat whole-project discovery, parsing, and validation. -- Tree-sitter and the JavaScript and C++ grammars remain mandatory installation dependencies even - when their runtime modules are unused. +- The base wheel intentionally omits Tree-sitter. JavaScript, TypeScript, and C++ syntax evidence + requires the matching `docforge[javascript]`, `docforge[typescript]`, or `docforge[cpp]` extra. + Python reference evidence uses the standard library and remains available in the base wheel. - Several version strings and defaults remain duplicated. - One individually oversized context entry is represented as explicit bounded omission evidence; callers use targeted retrieval for that node. diff --git a/docs/CONTRACT.md b/docs/CONTRACT.md index cfbaf30..177a62d 100644 --- a/docs/CONTRACT.md +++ b/docs/CONTRACT.md @@ -29,6 +29,9 @@ commit when Git is available; it cannot change repository state. - Projection package: `schemas/projection-package.schema.json`, version 1. - Projection receipt: `schemas/projection-receipt.schema.json`, version 1. - Independent projection policy: `schemas/projection-policy.schema.json`, version 2. +- Adapter launcher: `schemas/adapter-launcher.schema.json`, version 1. +- Adapter client configuration: `schemas/adapter-client-configuration.schema.json`, version 1. +- Reference adapter configuration: `schemas/reference-adapter.schema.json`, version 1. - Index schema: version 3, disposable and reproducible. - Index attestation: schema version 1, disposable and reproducible. - Core, CLI, and MCP server: version 1.3.0.dev0. @@ -103,6 +106,42 @@ complete projection, opens SQLite, starts MCP, executes the configured command, builds, renders, starts a viewer, or writes configuration. Unprovable client behavior is a warning, not an invented success. +## Public adapter SDK and reference binding + +`docforge.adapter_sdk` is the stable adapter-authoring import boundary. It exposes the typed +projection, manifest, source contribution, complete assembly, project wrapper, graph model, and +conformance contracts needed by an adapter without requiring authors to import core implementation +modules. + +Complete evidence includes the primary graph and function Logic. An incremental adapter that +publishes Logic must implement `load_complete_assembly()` as an independent complete oracle. +`verify_adapter_conformance()` proves repeated complete determinism, equality between the complete +assembly and `load_projection()`, and exact complete/incremental graph-plus-Logic parity. Separate +tests remain responsible for confinement, restart behavior, no-AST behavior, cache recovery, and +retrieval. + +Adapter assemblies are bounded before publication. Primary nodes use the descriptor `max_nodes` +limit. Primary edges, Logic nodes, and Logic edges use fixed deterministic multipliers over that +limit. Version-1 extraction caches are regular-file-only, bounded to 10,000 sources and +64,000,000 bytes, and are treated as misses when corrupt, oversized, foreign, or incompatible. + +`.docforge/reference-adapter.toml` is a closed version-1 selection among `python`, `javascript`, +`typescript`, and `cpp`. It declares one project identity and explicit non-overlapping source +roots. C++ additionally requires a confined `compile_commands.json`. It cannot declare a command, +module, environment, writer, applier, or remote endpoint. + +`python -m docforge.reference_mcp --project-root ROOT` constructs only the selected fixed +repository reference adapter and exposes the read surface. It never registers proposal or +application tools. + +`AdapterLauncherV1` is an immutable project-bound Python-module declaration. It accepts no +arbitrary command, arguments, working directory, environment, discovery, callable selector, or +module reload. Custom launchers resolve one installed top-level module through isolated Python and +require its origin inside the project root. The fixed `docforge.reference_mcp` module is the only +trusted dotted exception. `generate_adapter_client_configuration()` binds generated Codex, +Claude, and OpenClaw fragments to that launcher, current source availability, effective policy, +descriptor, interpreter, and exact artifact bytes. + ## Isolated proposal model Create, update, move, and delete are ordered node operations inside an isolated changeset. Every diff --git a/docs/CORE_CONCEPTS_AND_AUTHORITY.md b/docs/CORE_CONCEPTS_AND_AUTHORITY.md new file mode 100644 index 0000000..4fc1c9d --- /dev/null +++ b/docs/CORE_CONCEPTS_AND_AUTHORITY.md @@ -0,0 +1,165 @@ +# Core concepts and authority + +DocForge is a project-bound knowledge compiler. It turns explicit canonical project facts into +validated graphs and bounded derived views without transferring authority to the index, an agent, +or a renderer. + +## One project, one explicit root + +Every operation is bound to one canonical real project directory. Paths in descriptors and adapter +configuration are project-relative and confined beneath that root. A CLI or MCP process does not +discover or switch projects after startup. + +The project root determines: + +- which descriptor and canonical sources may be read; +- where derived cache and changeset roots may exist; +- which project identity, revision, source hash, and generation appear in results; +- which writer, applier, rendering, and viewer policies can be selected. + +Generated client fragments preserve that binding. They are machine-local configuration +projections, not portable project authority. + +## Canonical facts and derived evidence + +Canonical inputs own the facts: + +- generic Markdown or TOML node sources; +- authority files named by a generic descriptor; +- an adapter's declared canonical sources and implementation boundary; +- `.docforge/project.toml` for a generic project; +- `.docforge/reference-adapter.toml` for a fixed reference integration. + +Everything DocForge builds from those inputs is derived: + +- SQLite indexes, attestations, generation receipts, and generation diffs; +- incremental extraction caches; +- task-context capsules and query responses; +- changeset previews; +- render plans, immutable packages, fragments, artifacts, and receipts; +- portable graph publications and live-viewer processes; +- generated Codex, Claude, and OpenClaw fragments. + +Derived state may be discarded and rebuilt. A derived artifact can prove what it was bound to, but +it cannot override current canonical content. + +## Nodes, relationships, and Logic + +The primary graph contains nodes and directed relationships. + +A node has a stable project-wide ID, title, family, authority, status, tags, summary, content, +source identity, and content hash. A generic Markdown file contains one node beneath a TOML +metadata block; a generic TOML source may contain multiple nodes. Adapter nodes use the same public +graph contract. + +A relationship is an exact `(source, relation, target)` triple. The project descriptor defines the +allowed relation names. DocForge gives `depends_on` special acyclic validation, but it does not +invent domain meaning for a project's other relation names. Retrieval recognizes only a versioned +alias set for task planning and reports unknown allowed relations as `unclassified`. + +Logic is deliberately separate. It is a lazy function-scoped control-flow projection owned by one +primary node. Decisions, actions, loops, convergence points, returns, and exceptions connect +through explicit branch edges. Logic does not add statement-level nodes to ordinary Nodes, Flow, +Web, search, or generation-diff results. Python, JavaScript, TypeScript, and C++ integrations may +publish Logic when their adapter contract supports it. + +## Authority, status, family, and tags + +These fields answer different questions: + +- `authority` describes the role of the content. The generic vocabulary is `authoritative`, + `approved_plan`, `derived`, `proposal`, and `historical`. +- `status` is project-defined lifecycle state such as `current`, `active`, or `verified`. +- `family` is a project-defined content grouping used for filtering, profiles, rendering, and + writer permissions. +- `tags` are exact project labels for retrieval and presentation. + +An `authoritative` node can still become stale; authority is not a freshness claim. A `derived` +node is still canonical if it is stored in a declared canonical source; the label describes its +role, not whether DocForge may silently regenerate it. Status and authority never grant an MCP +writer permission. + +## Validation, synchronization, and generations + +Validation loads the complete canonical graph and rejects unsafe paths, invalid source formats, +duplicate IDs, unresolved relationships, prohibited cycles, violated project limits, and +adapter-specific contract failures. + +The disposable index is published atomically only after the complete graph and SQLite integrity +checks pass. Its attestation binds the whole index file. A successful replacement is the derived +publication commit point; later receipt-writing trouble is reported as degraded evidence rather +than as a false claim that the replacement failed. + +MCP operations automatically synchronize derived state under a project lock before normal work. +A graph generation identifies one validated indexed snapshot. Generation-pinned retrieval and +rendering do not reopen mutable sources behind an older snapshot. + +The latest generation diff is one bounded primary-graph transition, not a history database. It +contains no Logic details or source text. + +## Complete and incremental adapters + +`load_projection()` is the compatibility baseline and clean graph oracle. An incremental adapter +adds: + +- `load_manifest()` for cheap project identity, source inventory, fingerprints, and dependencies; +- `extract_source()` for one cacheable source contribution; +- optionally `assemble_projection()` to normalize overlapping contributions. + +When incremental contributions contain Logic, `load_complete_assembly()` supplies a +cache-independent complete graph-plus-Logic oracle. Warm cache behavior is an optimization, never a +different authority path. Corrupt or incompatible extraction caches are treated as misses, and a +clean complete build remains the equivalence and recovery boundary. + +The public types, validators, and conformance helper are exported from `docforge.adapter_sdk`. See +the [Adapter authoring guide](ADAPTER_AUTHORING_GUIDE.md) and [Incremental +indexing](INCREMENTAL_INDEXING.md). + +## Proposals are not canonical changes + +A changeset is an isolated, ordered proposal over an exact canonical base. Each operation names +preconditions, and the final changeset has a content-derived hash. Validation projects the complete +resulting graph before application. + +Canonical application requires: + +1. a writer declared in the descriptor; +2. a process started with the matching proposal and application authority; +3. one explicit final changeset hash; +4. unchanged source, relationship, permission, and graph preconditions. + +Application does not perform Git mutation, build, deployment, or publication. Until exact-hash +application succeeds, canonical project files remain unchanged. + +## Three independent output projections + +Manual rendering, portable graph publication, and the live viewer are separate: + +- a manual is a declared derived HTML view over selected nodes; +- a portable graph is a content-addressed static Nodes, Flow, or Web artifact; +- the live viewer is a managed loopback process pinned to one validated index generation and can + request lazy Logic. + +Their policies compose independently. Disabling one does not transfer its authority to another. +None is canonical documentation or a retrieval authority. See [Rendering and +visualization](RENDERING_AND_VISUALIZATION.md). + +## Binding policy is not project truth + +Capability mode, no-AST preservation, diagnostics, and projection modes describe one running +process or generated client binding. They do not rewrite the descriptor or canonical graph. + +`--no-ast` forbids AST-family adapter evolution and Logic publication/retrieval for that binding. +It does not inspect parser implementation, sandbox the filesystem, or convert an existing +AST/Tree-sitter adapter into a no-AST adapter. Read [Policy precedence](POLICY_PRECEDENCE.md) and +[Legacy and no-AST operation](LEGACY_AND_NO_AST.md). + +## Trust the narrowest evidence + +DocForge reports stable IDs, hashes, generations, omissions, truncation, and provenance limits so a +consumer can distinguish proof from inference. A source inventory is not a semantic graph; a +syntax-level relationship is not compiler resolution; a configuration doctor is not a connection +test; a viewer snapshot is not continuous monitoring; and a benchmark is not a release. + +Continue with the [Project descriptor](PROJECT_DESCRIPTOR.md), [New-project +quickstart](NEW_PROJECT_QUICKSTART.md), or [Core contract](CONTRACT.md). diff --git a/docs/INCREMENTAL_INDEXING.md b/docs/INCREMENTAL_INDEXING.md index 4abb941..8af16da 100644 --- a/docs/INCREMENTAL_INDEXING.md +++ b/docs/INCREMENTAL_INDEXING.md @@ -4,6 +4,10 @@ DocForge Release 1 adapters return one complete immutable projection. That contr supported. The incremental compiler adds an opt-in source-scoped contract that avoids reparsing unchanged files while preserving the same validated, atomically published graph. +Import these contracts from the public `docforge.adapter_sdk` facade. See +[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for the preserved one-method contract and +[Reference Adapters](REFERENCE_ADAPTERS.md) for the four maintained implementations. + ## Release 1 compatibility The incremental interface is additive: @@ -34,18 +38,26 @@ Incremental indexing is an extraction optimization. It does not weaken publicati An interrupted extraction never replaces the last validated SQLite index. A malformed, incompatible, or missing cache is a cache miss, not a partial graph. +One cache generation is bounded to 10,000 source contributions and 64,000,000 encoded bytes. +Aggregate graph and Logic assembly limits are described in the +[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md#current-aggregate-bounds). + ## Adapter contract -An incremental loader implements all three methods: +An incremental loader implements the first, second, and fourth methods. It implements +`load_complete_assembly()` as well when it publishes Logic: ```python class MyAdapter: def load_manifest(self) -> AdapterManifest: ... def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ... + def load_complete_assembly(self) -> AdapterAssembly: ... def load_projection(self) -> AdapterProjection: ... ``` -`load_projection()` remains the deterministic full-rebuild fallback and equivalence oracle. +`load_projection()` remains the deterministic full-rebuild primary-graph oracle. An incremental +adapter that publishes Logic must additionally implement `load_complete_assembly()` as the +cache-independent complete graph-plus-Logic oracle. Each `AdapterSource` declares: @@ -137,7 +149,8 @@ fresh process must import and validate the current adapter. ``` Adapters can call `AdapterProject.verify_incremental_equivalence()` in release and contract tests. -The check compares project identity, revision, source hash, nodes, and relationships against +The check compares project identity, revision, source hash, nodes, relationships, and Logic against +the independent complete assembly. It also requires the complete assembly's primary graph to match `load_projection()`. ## Manual changes and relationships @@ -170,12 +183,29 @@ logic empty until they implement a language analyzer. This boundary prevents thousands of boolean expressions and basic blocks from polluting Nodes, Flow, Web, ordinary search, or architectural traversal. The Logic tab and `docforge_get_logic` request one function-scoped projection on demand. The built-in analyzers cover Python, -JavaScript, and C++. Python uses the standard-library AST. JavaScript and C++ share pinned -Tree-sitter infrastructure with thin language-aware control-flow profiles. Parsers run only while -extracting a changed source contribution; ordinary graph reads do not load or execute them. A -grammar alone supplies syntax, not control-flow meaning, so each new language still needs a small -semantic profile for its branch, loop, case, exception, and termination constructs. All analyzers -report possible static paths; they do not claim runtime branch outcomes. +JavaScript, TypeScript, and C++. Python uses the standard-library AST. JavaScript, TypeScript, and +C++ use distinct optional Tree-sitter grammars with thin language-aware control-flow profiles. +Ordinary graph reads use stored projections and do not load or execute these parsers. A grammar +alone supplies syntax, not control-flow meaning, so each new language still needs a profile for +its branch, loop, case, exception, and termination constructs. All analyzers report possible +static paths; they do not claim runtime branch outcomes. + +## Manifest and warm-parser scope + +Parser work is language-specific and must be measured at the correct boundary: + +- Python manifest construction fingerprints source and tokenizes local imports without calling + `ast.parse`. +- JavaScript and TypeScript manifest construction lexes static relative module specifiers without + invoking their distinct Tree-sitter extraction parsers. Focused tests prove this behavior on an + unchanged warm build. +- The C++ reference manifest parses inventoried sources with `tree-sitter-cpp` to discover quoted + include dependencies. A warm C++ extraction-cache hit is not a zero-parser claim. + +The maintained Python benchmark instruments the unchanged warm path and requires zero +`ast.parse` calls and zero `extract_source` calls. That exact zero-parser benchmark claim is Python +only. JavaScript and TypeScript retain focused parser-free-manifest tests; C++ deliberately does +not. ## Full rebuilds diff --git a/docs/LEGACY_AND_NO_AST.md b/docs/LEGACY_AND_NO_AST.md new file mode 100644 index 0000000..c1125c5 --- /dev/null +++ b/docs/LEGACY_AND_NO_AST.md @@ -0,0 +1,80 @@ +# Legacy adapters and no-AST policy + +DocForge preserves the original one-method adapter contract while offering incremental extraction, +complete graph-plus-Logic assemblies, and an independently selectable no-AST MCP policy. These are +separate compatibility boundaries. + +## One-method adapters remain valid + +An existing adapter that implements only: + +```python +def load_projection(self) -> AdapterProjection: ... +``` + +continues to work. It receives the same projection validation, indexing, querying, visualization, +and MCP behavior as before. It does not receive extraction-cache speedups and publishes no Logic +through the complete assembly contract. + +Incremental adoption is additive. Implement `load_manifest()` and `extract_source()` while keeping +`load_projection()` as the independent complete graph oracle. If incremental contributions publish +Logic, also implement `load_complete_assembly()` so complete and incremental graph-plus-Logic +output can be compared exactly. The public authoring surface is `docforge.adapter_sdk`; see the +[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md). + +## What no-AST means + +`--no-ast` is an immutable policy on one MCP process binding. It preserves the configured adapter +strategy but forbids AST, Tree-sitter, compiler-AST, and function-Logic evolution for that binding. +It also rejects nonempty Logic before index publication and rejects a pre-existing index that +contains Logic before reads or live visualization. + +Start the generic server with: + +```bash +docforge-mcp --project-root /absolute/path/to/project --no-ast +``` + +Project-owned integrations select the same policy with +`create_project_server(..., no_ast=True)` or `create_read_only_server(..., no_ast=True)`. +`docforge_bootstrap` and `docforge_get_contract` report the effective policy, and +`docforge_get_logic` returns `adapter_policy_forbids_logic`. + +Changing this policy requires a new process. It does not rewrite or reload an adapter in place. + +## What no-AST does not mean + +No-AST is not: + +- an inspection mechanism that proves which parser an arbitrary adapter uses; +- a filesystem, Python-import, or process sandbox; +- a promise that the adapter performs no non-AST fingerprinting, dependency discovery, caching, or + complete-projection loading; or +- permission to change project files, adapter code, or the configured capability surface. + +Repository permissions and project instructions remain responsible for processes that have direct +filesystem access. The MCP boundary still excludes shell execution, arbitrary file operations, +Git mutation, builds, deployment, publication, project switching, and cross-project retrieval. + +## Reference adapters are not no-AST adapters + +The built-in Python reference extracts syntax and Logic with the standard-library AST. The +JavaScript, TypeScript, and C++ references extract syntax and Logic with their optional +Tree-sitter grammars. Their normal projections contain Logic, so they must not be presented or +configured as no-AST adapters. + +The narrower performance statements in [Reference Adapters](REFERENCE_ADAPTERS.md) concern +specific manifest and warm-cache paths. A parser-free manifest does not turn an adapter that +publishes AST-derived Logic into a no-AST adapter. + +## Migration choices + +Keep a one-method adapter when its complete rebuild is acceptable and no function Logic is needed. +Adopt incremental extraction when measured source parsing dominates synchronization. Adopt the +complete assembly contract when Logic is published or overlapping raw evidence needs deterministic +ownership. + +Choose no-AST only when the project binding must forbid AST-derived publication and future adapter +upgrades of that kind. It is a binding policy, not a substitute for adapter confinement, +conformance tests, restart detection, cache recovery, or the full proof matrix in +[Incremental Adapter Indexing](INCREMENTAL_INDEXING.md). diff --git a/docs/MCP_CONTRACT.md b/docs/MCP_CONTRACT.md index b710bb8..d1c345d 100644 --- a/docs/MCP_CONTRACT.md +++ b/docs/MCP_CONTRACT.md @@ -28,6 +28,30 @@ Bootstrap and contract results include `effective_policy` schema version 1 plus registered surface and startup-bound proposal/application access. The project descriptor remains schema version 1 and does not silently acquire machine-specific process policy. +## Fixed reference binding + +`docforge.reference_mcp` is the fixed runnable binding for the in-repository Python, JavaScript, +TypeScript, and C++ reference adapters: + +```bash +python -I -m docforge.reference_mcp \ + --project-root /absolute/project \ + --capability-mode read +``` + +It loads only `.docforge/reference-adapter.toml`, selects only the fixed provider for the declared +language, and constructs the same project-bound read surface through `create_read_only_server()`. +Bootstrap binding metadata includes `server_module = "docforge.reference_mcp"`, +`adapter_mode = "reference"`, the selected `reference_language`, and the exact +`reference_config_hash`. + +The reference command accepts read capability mode only. It registers all 21 read tools below and +no isolated proposal or canonical-application tools. The configuration cannot select a provider, +module, command, arguments, working directory, environment, or arbitrary discovery behavior. See +[Reference Adapters](REFERENCE_ADAPTERS.md) for language scope and +[Agent Integration](AGENT_INTEGRATION.md) for the immutable launcher and generated client +fragments. + ## Read tools - `docforge_bootstrap` @@ -45,6 +69,8 @@ schema version 1 and does not silently acquire machine-specific process policy. - `docforge_get_task_context` - `docforge_validate_project` - `docforge_render_status` +- `docforge_graph_plan` +- `docforge_graph_render_status` - `docforge_visualize` - `docforge_stop_visualization` - `docforge_visualization_status` @@ -349,3 +375,6 @@ permissions and project instructions remain responsible for that broader boundar The legacy `adapter_policy` object remains byte-compatible. It is now a projection of the versioned `effective_policy`; `--no-ast` restrictively overrides adapter evolution, AST analysis, and Logic indexing without widening any other capability. + +See [Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for the compatibility boundary and +why the AST and Tree-sitter reference adapters are not no-AST adapters. diff --git a/docs/MIGRATING_FROM_V1.md b/docs/MIGRATING_FROM_V1.md new file mode 100644 index 0000000..dab9669 --- /dev/null +++ b/docs/MIGRATING_FROM_V1.md @@ -0,0 +1,79 @@ +# Migrating from DocForge v1 + +DocForge2 preserves the `docforge` distribution, Python package, CLI command, MCP tool prefix, +generic project descriptor version, and legacy adapter entry point. Migration is an additive +validation exercise, not a canonical-content rewrite. + +Read [compatibility](COMPATIBILITY.md), [legacy and no-AST operation](LEGACY_AND_NO_AST.md), and +[recovery](RECOVERY_AND_PERFORMANCE.md) before changing a production binding. + +## What remains compatible + +- Generic Markdown and TOML projects keep `.docforge/project.toml` schema version 1. +- A custom adapter implementing only `load_projection()` remains valid. +- Existing canonical nodes, stable IDs, source paths, relationship vocabulary, changesets, and + reviewed hashes are not silently rewritten. +- Existing CLI and MCP names remain available. New fields, commands, tools, plans, policies, and + reference integrations are additive. +- `docforge-mcp --no-ast` remains the shorthand for the preserved no-AST binding. + +Disposable index and cache schemas may change. Rebuild them rather than copying them as authority. + +## Recommended migration + +1. Record the v1 package version, adapter identity, project descriptor, canonical source hash, + active changesets, generated client configuration, and current rendered outputs. +2. Back up canonical sources and active proposal files. Derived `.docforge/cache`, preview, + portable-graph, and viewer state need not be authoritative backups. +3. Install the DocForge2 candidate in a separate environment. Do not repoint the production MCP + binding yet. +4. Run the existing adapter through complete loading and `ProjectIndex.check()`. For incremental + adapters, verify complete/incremental equivalence. Logic-producing incremental adapters must + implement the independent complete assembly oracle. +5. Compare project ID, adapter, revision, source hash, primary node and edge hashes, Logic policy, + and retrieval results with the v1 evidence. +6. Recompose effective capability and projection policy. Do not assume that a new default grants + proposal, application, rendering, or viewer access. +7. Rebuild disposable indexes, extraction caches, fragments, receipts, previews, and portable + artifacts from the validated candidate. +8. Generate a new client fragment. Generic projects use `docforge configure`; custom adapters use + `AdapterLauncherV1` with `generate_adapter_client_configuration()`. +9. Run doctor, real MCP bootstrap, representative retrieval, rendering status, recovery, and + exact-hash proposal/application checks in a disposable or shadow environment. +10. Repoint one production binding only after the candidate and rollback procedure pass. + +## Adopting reference adapters + +The repository reference adapters are narrow syntax evidence, not automatic semantic replacements +for mature v1 integrations. Python publishes modules, classes, functions, and local imports. +JavaScript and TypeScript publish syntax plus static project-relative imports and re-exports. +C++ uses a confined `compile_commands.json` as inert translation-unit inventory and publishes +directly resolvable project-local quoted includes. + +Do not replace a compiler-, language-server-, or project-owned semantic adapter if the task depends +on resolved calls, types, inheritance, macros, compiler include semantics, runtime facts, or +ownership. Use [reference adapters](REFERENCE_ADAPTERS.md) as a measured starting point. + +## Changesets + +Existing changesets remain bound to their original project, root, base revision, source hash, +writer, operations, and exact hash. Do not edit a changeset to make it look current. Retrieve and +validate it through the candidate. Rebase only when every precondition remains true; otherwise +create and review a new proposal. + +Before enabling canonical application, prove the exact serializer round trip and recovery behavior +for the selected adapter. The fixed reference MCP server is intentionally read-only. + +## Rollback + +Keep the v1 environment, configuration fragment, and service definition until the DocForge2 +binding has passed live validation. To roll back: + +1. Stop the DocForge2 MCP/viewer processes. +2. Restore the previous client or service binding. +3. Restore canonical sources only if a verified application receipt says they changed and the + adapter-specific rollback requires it. +4. Discard DocForge2-derived indexes, receipts, fragments, previews, and portable artifacts. +5. Restart v1 and verify its recorded project/source identity. + +Never move a published tag or reuse a version identity for a corrected release. diff --git a/docs/MILESTONE_4_BASELINE.md b/docs/MILESTONE_4_BASELINE.md new file mode 100644 index 0000000..8aa1abb --- /dev/null +++ b/docs/MILESTONE_4_BASELINE.md @@ -0,0 +1,95 @@ +# Milestone 4 baseline + +## Scope and method + +This baseline records the adapter SDK, Python reference adapter, incremental equivalence, and +recovery behavior completed in Milestone 4. It was captured on 2026-07-29 from clean executable +candidate `95271dcf2e48045b9d3aed9b9ea09c7fc155692c`. + +The maintained command was: + +```bash +.venv/bin/python tools/milestone4_benchmark.py \ + --mode full \ + --output benchmarks/milestone4-2026-07-29.json +``` + +The synthetic project contains 334 Python files. Each file contributes one module, one function, +and one argument node, for 1,002 primary nodes. Imports form a deterministic chain. Durations use +`time.perf_counter_ns()` and nearest-rank p95. Each ordinary result is serialized as compact sorted +JSON for its response-size gate. Per-operation memory uses `tracemalloc`; cumulative process +high-water uses `RUSAGE_SELF`. + +Environment: + +- Linux 7.1.3-200.nobara.fc44.x86_64 with glibc 2.43. +- CPython 3.14.6. +- x86_64. +- Three warm samples; cold, equivalence, and recovery operations run once. +- Per-operation traced-memory ceiling: 268,435,456 bytes. +- Process high-water ceiling: 536,870,912 bytes. +- Response ceiling: 524,288 bytes. + +The complete machine-readable result is +[`benchmarks/milestone4-2026-07-29.json`](../benchmarks/milestone4-2026-07-29.json). +Its SHA-256 is `b6a871dde730a119fc0a138c47bd25f2c533ed3171a9a3d098075aef08173600`. +The stable evidence payload SHA-256 is +`4a8db461a311b5df4abd5aa00063e9a347d8b9dba19b30e35684f561d5271549`. + +## Results + +| Operation | Median | p95 | Limit | Traced peak | Response | +|---|---:|---:|---:|---:|---:| +| Cold incremental build | 1,599.760 ms | 1,599.760 ms | 20,000 ms | 68,573,540 B | 14,580 B | +| Warm incremental build | 1,244.427 ms | 1,265.387 ms | 20,000 ms | 69,997,166 B | 14,578 B | +| Complete/incremental equivalence | 1,897.919 ms | 1,897.919 ms | 30,000 ms | 64,305,919 B | 230 B | +| Corrupt extraction-cache recovery | 1,695.406 ms | 1,695.406 ms | 30,000 ms | 69,776,923 B | 14,580 B | +| Corrupt index recovery | 1,707.879 ms | 1,707.879 ms | 30,000 ms | 68,561,924 B | 14,779 B | + +Process high-water was 78,798,848 bytes. The regression limits intentionally leave multiple times +the measured headroom; they are tripwires, not performance promises. + +## Deterministic graph and Logic evidence + +The candidate produced: + +- 1,002 primary nodes with hash + `f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f`. +- 1,001 primary edges with hash + `00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35`. +- 334 Logic projections containing 2,338 Logic nodes and 2,338 Logic edges, with hash + `c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c`. +- Complete assembly hash + `2888182ab765fbffe3ba873c1613345640e8e6d89be74ddfca7452c0a5056345`. +- Source hash + `30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2`. + +The independently loaded complete assembly and the incremental assembly matched exactly across +project identity, revision, source, nodes, edges, and Logic. + +## Work and recovery gates + +The warm build recorded all 334 sources as cache hits, zero reparsed sources, zero `ast.parse` +calls, and zero `extract_source` calls. Corrupting the extraction cache forced all sources through +extraction and reproduced the same graph and Logic hashes. Corrupting SQLite rebuilt the index +entirely from cache hits with zero parsing or extraction and reproduced those hashes. + +This zero-parser evidence applies to the Python benchmark. Focused JavaScript and TypeScript tests +prove parser-free manifests. The C++ reference manifest uses Tree-sitter for bounded quoted-include +discovery and does not make a zero-warm-parser claim. + +## Fresh-wheel and reference evidence + +The offline adoption proof built and installed the base wheel without Tree-sitter packages, +built and checked a real Python reference project, started the isolated reference MCP server, and +performed bootstrap, search, and exact retrieval over its 21 read tools. Selecting C++ without its +extra failed with `optional_dependency_missing` and `docforge[cpp]` remediation. + +Focused fixtures produced: + +- Python: 14 nodes, 13 edges, 5 Logic projections. +- JavaScript: 14 nodes, 14 edges, 5 Logic projections. +- TypeScript: 13 nodes, 14 edges, 4 Logic projections. +- C++: 17 nodes, 16 edges, 5 Logic projections. + +These fixture counts verify implementations; they are not language-wide completeness claims. diff --git a/docs/MILESTONE_4_CLOSEOUT.md b/docs/MILESTONE_4_CLOSEOUT.md new file mode 100644 index 0000000..b5eda38 --- /dev/null +++ b/docs/MILESTONE_4_CLOSEOUT.md @@ -0,0 +1,83 @@ +# Milestone 4 closeout + +## Outcome + +Milestone 4 is complete. New projects can adopt a public adapter SDK or one of four narrow +repository reference integrations, attach a fixed read-only MCP server, and follow maintained +product documentation without reading core implementation. + +Implemented contracts: + +- Stable `docforge.adapter_sdk` authoring imports. +- Independent complete primary-graph-plus-Logic oracle and exact incremental equivalence. +- Bounded adapter assemblies and version-1 extraction caches. +- Base Python, optional JavaScript, optional TypeScript, and optional C++ reference integrations. +- Closed `.docforge/reference-adapter.toml` and fixed `docforge.reference_mcp` read-only binding. +- Immutable, project-bound, launchable `AdapterLauncherV1` declarations and generated Codex, + Claude, and OpenClaw fragments for custom adapters. +- Live implementation-derived CLI and MCP reference tables with race-safe publication and drift + checking. +- Strict documentation graph, link, anchor, H1, reachability, required-page, generated-notice, and + documented-reference-config checks. +- Offline fresh-wheel adoption and maintained 1,002-node scale/recovery gates. + +## Candidate evidence + +The frozen executable candidate is +`95271dcf2e48045b9d3aed9b9ea09c7fc155692c`. + +Its complete executable gate passed: + +- Ruff formatting and Python lint. +- HTML, rendered-manual HTML, portable-graph HTML, CSS, and JavaScript checks. +- Pyright with zero diagnostics. +- Warning-strict compilation. +- 142 contract tests and 268 subtests. +- 347 complete tests and 402 subtests. +- Three Playwright and axe accessibility flows for the manual, portable graph, and live viewer. +- Lock and npm dependency-tree checks. +- Wheel and source-distribution builds. +- Offline fresh-wheel adoption. +- Milestone 0, 1, 2, 3, and 4 smoke benchmarks. + +The clean full benchmark passed exact graph-plus-Logic equivalence, warm zero Python parser and +extraction work, corrupt-cache recovery, corrupt-index recovery, response, memory, and latency +gates. Exact results are in [the Milestone 4 baseline](MILESTONE_4_BASELINE.md) and +[`benchmarks/milestone4-2026-07-29.json`](../benchmarks/milestone4-2026-07-29.json). + +Gitleaks 8.30.1 scanned the Milestone 4 commit range and candidate tree with no findings. The SSH +remote syntax prevents Gitleaks from constructing finding hyperlinks; it does not affect scanning. + +## Reference scope + +- Python uses the standard-library AST and publishes syntax plus local imports. +- JavaScript and TypeScript use distinct optional Tree-sitter grammars and publish syntax plus + project-local static relative imports and re-exports. +- C++ uses a confined `compile_commands.json` as inert translation-unit inventory and publishes + syntax plus directly resolvable project-local quoted includes. +- The references do not claim resolved calls, inheritance, types, symbol references, compiler + include semantics, macro semantics, runtime behavior, or semantic ownership. + +The C++ reference never executes a compiler or compilation-database command. It is not a Clang +semantic adapter. + +## Preserved boundaries + +- The `docforge` distribution, Python package, CLI, MCP executable, and tool names remain. +- Generic projects and one-method `load_projection()` adapters remain supported. +- Descriptor schema version 1, index schema version 3, effective policy version 1, and no-AST + behavior remain. +- Heavy language frontends are optional. Base generic and Python operation installs no + Tree-sitter distribution. +- Reference MCP is read-only. Proposal and application remain explicit project-owned gates. +- No language adapter was separately published. +- No WorldForge, ScrapeStation, legacy-repository, production-binding, storage, or self-hosting + change was made. +- No tag or Forgejo release was created for Milestone 4. + +## Later work + +Milestone 5 owns stabilization and the first DocForge2 release. Release identity, compatibility +matrix, migration and recovery proofs, comparative real-task evidence, versioning, tagging, and +publication must be validated there. Remote adapters, render farms, third-party renderers, +cross-project graphs, storage replacement, and self-hosting remain deferred without measured need. diff --git a/docs/NEW_PROJECT_QUICKSTART.md b/docs/NEW_PROJECT_QUICKSTART.md index a3460ba..808fb3e 100644 --- a/docs/NEW_PROJECT_QUICKSTART.md +++ b/docs/NEW_PROJECT_QUICKSTART.md @@ -1,6 +1,276 @@ -# DocForge setup moved to the user manual +# New-project quickstart -The complete installation, project setup, visualization, CLI, MCP, application, adapter, and -troubleshooting reference now lives in the [DocForge user manual](USER_MANUAL.md). +This guide takes a new installation from an empty project binding to a validated generic manual or +one of DocForge's fixed reference source graphs. For the complete operating reference, see the +[user manual](USER_MANUAL.md). -This file remains only so existing bookmarks and links continue to resolve. +## 1. Install DocForge + +DocForge requires Python 3.12 or newer. A base installation includes the generic project service, +the public adapter SDK, the Python reference integration, CLI, MCP server, renderers, and viewer +assets. It does not install Tree-sitter. + +From a source checkout: + +```bash +git clone /absolute/path/DocForge +cd /absolute/path/DocForge +uv sync --group dev + +DOCFORGE=/absolute/path/DocForge/.venv/bin/docforge +DOCFORGE_MCP=/absolute/path/DocForge/.venv/bin/docforge-mcp +DOCFORGE_PYTHON=/absolute/path/DocForge/.venv/bin/python +``` + +For an isolated consumer environment, install the checkout or a built wheel with `uv pip install`. +Add only the language extras that project needs: + +```bash +uv venv /absolute/path/docforge-env --python 3.12 +uv pip install --python /absolute/path/docforge-env/bin/python /absolute/path/DocForge +uv pip install --python /absolute/path/docforge-env/bin/python \ + "/absolute/path/DocForge[javascript]" +uv pip install --python /absolute/path/docforge-env/bin/python \ + "/absolute/path/DocForge[typescript]" +uv pip install --python /absolute/path/docforge-env/bin/python \ + "/absolute/path/DocForge[cpp]" +``` + +The `languages` extra installs all three optional grammar families. JavaScript and TypeScript are +separate extras because they use distinct grammars. The C++ extra supplies a syntax grammar, not a +compiler or Clang semantic frontend. + +Confirm the installation: + +```bash +"$DOCFORGE" --help +"$DOCFORGE_MCP" --help +``` + +## 2. Choose a project route + +Use a generic project when Markdown or TOML documentation is canonical. Use a fixed reference +adapter when you want a bounded source inventory and syntax-level graph for Python, JavaScript, +TypeScript, or C++. Use a project-owned adapter when the production contract must supply richer +semantics. + +- Generic manual: continue with [Create a generic project](#create-a-generic-project). +- Fixed source example: continue with [Use a reference adapter](#use-a-reference-adapter). +- Production language frontend: follow the [Adapter authoring guide](ADAPTER_AUTHORING_GUIDE.md). + +## Create a generic project + +Set the absolute project root and assess it without writing: + +```bash +PROJECT=/absolute/path/MyProject +"$DOCFORGE" --project-root "$PROJECT" onboard +``` + +The assessment reports detected languages, build evidence, documentation candidates, current +configuration, and available capabilities. Detection never invents a source graph. + +Create a generic starter explicitly: + +```bash +"$DOCFORGE" --project-root "$PROJECT" onboard \ + --scaffold \ + --project-id my-project \ + --title "My Project" +``` + +Scaffolding is create-only and refuses existing target files. It creates: + +- `.docforge/project.toml`, the generic project descriptor; +- `docs/docforge/content/architecture-overview.md`, one canonical node; +- `.docforge/templates/manual.html`, one built-in-renderer template; +- derived index, receipt, and rendered output below `.docforge`. + +If a language was detected, source graph status remains `adapter_required`. The generic starter +does not claim source semantics. + +Validate and inspect it: + +```bash +"$DOCFORGE" --project-root "$PROJECT" validate +"$DOCFORGE" --project-root "$PROJECT" reindex +"$DOCFORGE" --project-root "$PROJECT" search architecture +"$DOCFORGE" --project-root "$PROJECT" show architecture.overview +"$DOCFORGE" --project-root "$PROJECT" render-status +``` + +The descriptor is explained field by field in [Project descriptor](PROJECT_DESCRIPTOR.md). Add +canonical nodes only after choosing their authority, stable IDs, families, statuses, and allowed +relationships; [Core concepts and authority](CORE_CONCEPTS_AND_AUTHORITY.md) defines those terms. + +### Start a generic MCP binding + +Start read-only first: + +```bash +"$DOCFORGE_MCP" \ + --project-root "$PROJECT" \ + --capability-mode read +``` + +Call `docforge_bootstrap` before other tools. It reports the exact project identity, generation, +effective policy, projection policy, available capabilities, and recommended first read. + +To enable proposals, the descriptor must declare the writer and the process must select it: + +```bash +"$DOCFORGE_MCP" \ + --project-root "$PROJECT" \ + --capability-mode proposal \ + --proposal-writer project-editor +``` + +Canonical application is a separate startup gate. Do not add it to a read-only client: + +```bash +"$DOCFORGE_MCP" \ + --project-root "$PROJECT" \ + --capability-mode application \ + --proposal-writer project-editor \ + --canonical-applier project-editor +``` + +Application accepts one exact reviewed changeset hash. It does not commit, push, build, deploy, or +publish the project. + +### Generate a generic client fragment + +Preview is side-effect free: + +```bash +"$DOCFORGE" configure codex --project "$PROJECT" +"$DOCFORGE" configure claude --project "$PROJECT" +"$DOCFORGE" configure openclaw --project "$PROJECT" +``` + +Add `--output /absolute/path/new-fragment` to create one new private standalone file. Generation +does not merge with or replace a different existing file. Diagnose an installed binding with: + +```bash +"$DOCFORGE" doctor --client codex --project "$PROJECT" +``` + +Doctor is a bounded configuration inspector, not a connection test. See [Agent +integration](AGENT_INTEGRATION.md) for client-specific layouts and limitations. + +## Use a reference adapter + +Reference adapters read one fixed descriptor: +`.docforge/reference-adapter.toml`. The path is not selectable. + +Create a Python project configuration: + +```toml reference-adapter +schema_version = 1 +project_id = "my-python-project" +title = "My Python Project" +language = "python" +source_roots = ["src"] +``` + +JavaScript uses `language = "javascript"` and the `javascript` extra. TypeScript uses +`language = "typescript"` and the `typescript` extra. + +C++ additionally requires a confined compilation database: + +```toml reference-adapter +schema_version = 1 +project_id = "my-cpp-project" +title = "My C++ Project" +language = "cpp" +source_roots = ["src", "include"] +compilation_database = "compile_commands.json" +``` + +The C++ integration reads `compile_commands.json` only as bounded translation-unit inventory and +fingerprint evidence. It never executes a recorded command or compiler. + +Run the fixed read-only server through the same installed Python interpreter: + +```bash +"$DOCFORGE_PYTHON" -I -m docforge.reference_mcp \ + --project-root "$PROJECT" \ + --capability-mode read +``` + +The binding chooses one in-package provider from the descriptor language. It accepts no provider +module, command, argument list, working directory, environment, discovery rule, proposal writer, +or canonical applier. Its MCP surface is the read subset documented in the [generated command +reference](COMMAND_REFERENCE.md). + +### Generate a reference-adapter client fragment + +Generic `docforge configure` intentionally refuses custom adapters. Construct the adapter project +and immutable launcher, then call the custom-adapter generator: + +```python +from pathlib import Path + +from docforge.adapter_launcher import AdapterLauncherV1 +from docforge.client_config import generate_adapter_client_configuration +from docforge.reference_mcp import REFERENCE_MCP_MODULE, create_reference_project + +root = Path("/absolute/path/MyProject").resolve(strict=True) +project = create_reference_project(root) +launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE) + +preview = generate_adapter_client_configuration( + project, + launcher, + "codex", + capability_mode="read", +) +print(preview["artifact"]["content"]) +``` + +Use `client="claude"` or `client="openclaw"` for those formats. Pass an absolute `output` path only +when creating a new standalone private fragment. The generated launch is bound to the selected +project, descriptor hash, adapter identity, installed module, isolated interpreter, policy, and +source availability evidence. + +Read [Reference adapters](REFERENCE_ADAPTERS.md) before relying on the graph. The Python example +publishes local imports, the JavaScript and TypeScript examples publish project-local static +relative imports and re-exports, and the C++ example publishes directly resolvable project-local +quoted includes. None is a complete semantic compiler frontend. + +## 3. Add visualization only when needed + +Install the per-user viewer manager once: + +```bash +/absolute/path/DocForge/.venv/bin/docforge-viewer-manager install-user-service +``` + +Then start a project-bound snapshot: + +```bash +"$DOCFORGE" --project-root "$PROJECT" visualize +"$DOCFORGE" --project-root "$PROJECT" visualization-status +``` + +The listener is loopback-only and tokenized. The viewer is a derived snapshot, not canonical +authority and not a continuously monitored filesystem view. Read [Rendering and +visualization](RENDERING_AND_VISUALIZATION.md) for manual, portable, and live-viewer differences. + +## 4. Verify the maintained checkout + +Contributors can run: + +```bash +make command-reference-check +make docs-check +make adoption-m4 +make benchmark-m4-smoke +``` + +Run `make gate` before a release candidate. `benchmark-m4-smoke` is routine coverage; +`benchmark-m4` is the maintained full adapter workload. + +Continue with [Project onboarding](PROJECT_ONBOARDING.md) for production integration, +[Policy precedence](POLICY_PRECEDENCE.md) before widening a binding, and [Security](SECURITY.md) +before exposing any MCP process. diff --git a/docs/POLICY_PRECEDENCE.md b/docs/POLICY_PRECEDENCE.md new file mode 100644 index 0000000..2b275b1 --- /dev/null +++ b/docs/POLICY_PRECEDENCE.md @@ -0,0 +1,202 @@ +# Policy precedence + +DocForge composes an immutable effective policy for each project-bound process. It resolves +restrictions in a fixed order: + +```text +core safety + > explicit binding + > no-AST shorthand + > resource availability +``` + +A lower layer can make a requested operation unavailable; it cannot override a higher-layer +prohibition. Bootstrap and generated client evidence report the composed result and hashes so a +client does not need to infer policy from command-line arguments. + +## 1. Core safety + +Core safety is unconditional. No capability mode exposes: + +- arbitrary file access or project switching; +- arbitrary renderer, module, command, shell, argument, working-directory, or environment + selection; +- Git mutation; +- project builds or compiler execution; +- deployment or publication. + +Canonical application is limited to DocForge's validated serializer boundary. Adapter launch is +limited to the immutable launcher contract. Rendering is limited to declared views and fixed +built-in workers. An indexed instruction cannot alter any of these rules. + +## 2. Explicit binding + +A process is bound at startup to one project root, descriptor, adapter, capability mode, writer and +applier identities when present, no-AST selection, diagnostics selection, and projection modes. +The binding does not change during the process lifetime. + +Capability modes are: + +- `read`: register only the read surface; +- `proposal`: add proposal tools when a valid selected writer is available; +- `application`: require a startup-bound canonical applier and expose exact-hash application; +- `operator`: reserved; it currently adds no tools. + +Mode describes the maximum registered surface. Actual authority can be narrower. A descriptor must +declare the selected writer, including allowed families and operation types. Application requires +the matching configured writer, changeset creator, and canonical-applier identity. A mode name +cannot create a missing descriptor grant. + +Generic generated client fragments default to read mode. Other construction paths preserve their +documented compatible factory defaults. Treat `docforge_bootstrap.session_contract` and its actual +capabilities as authoritative for a running server. + +## 3. No-AST shorthand + +`--no-ast` is a restrictive compatibility shorthand. It composes: + +- adapter evolution `preserve`; +- AST analysis `forbidden`; +- Logic indexing `off`; +- `docforge_get_logic` blocked; +- prohibitions on AST, Tree-sitter, compiler-AST, and function-Logic upgrades. + +Non-AST source fingerprinting and incremental caching remain allowed. Existing one-method adapters +continue to use `load_projection()`. + +The shorthand does not inspect how an existing adapter was implemented, sandbox its filesystem +reads, or transform an AST-based adapter into a no-AST adapter. Do not run the Python, +JavaScript/TypeScript, or C++ syntax reference integrations and then describe the binding as a +proved no-AST integration. See [Legacy and no-AST operation](LEGACY_AND_NO_AST.md). + +## 4. Resource availability + +Even an allowed policy cannot create a missing resource: + +- application mode requires a startup-bound canonical applier; +- manual `explicit` or `auto` requires declared manual render configuration; +- manual `auto` also requires canonical application in the current operation or server; +- portable graph `explicit` requires declared portable graph configuration; +- live viewer `on-demand` requires the viewer runtime; +- an optional reference language requires its installed extra; +- a project-owned launcher module must be installed, isolated, project-owned, and unchanged. + +Unavailable requested modes fail before hidden work. Missing optional language grammars return an +actionable install target such as `docforge[typescript]` or `docforge[cpp]`; DocForge does not +silently downgrade to a different frontend. + +## Effective process policy + +The version-1 effective policy reports: + +- capability mode and whether it came from a factory default or explicit selection; +- adapter evolution, AST analysis, and Logic indexing; +- automatic synchronization and validated integrity; +- the compatible manual-render projection; +- profiling state; +- blocked tools and prohibitions; +- the exact precedence list. + +This version-1 projection preserves compatibility. It is not the complete version-2 rendering +policy; portable graph and live-viewer choices are reported separately. + +## Independent projection policy + +Manual rendering, portable graph publication, and live visualization use a separate immutable +version-2 policy: + +```text +manual: auto | explicit | disabled +portable_graph: explicit | disabled +live_viewer: on-demand | disabled +``` + +When a selector is omitted, composition uses availability-aware defaults: + +- manual is `auto` only when manual configuration and canonical application are both available; + otherwise it is `explicit` when configured, or `disabled`; +- portable graph is `explicit` when configured, otherwise `disabled`; +- live viewer is `on-demand` when the runtime is available, otherwise `disabled`. + +An explicit non-disabled selection for a missing resource returns +`projection_policy_unavailable`. An active operation prohibited by the selected policy returns +`projection_policy_forbids_operation` before planning, rendering, or viewer startup. + +Status remains intentionally narrower than active work. Manual and portable receipt-only status +are available when their active operations are disabled. Viewer status and explicit stop remain +available when viewer start is disabled. + +Ordinary standalone CLI rendering uses manual `explicit`. Manual `auto` belongs to a canonical +application operation that owns automatic regeneration. + +## Diagnostics do not grant authority + +`--diagnostics` enables bounded request-local stage timings and compiler-work counters. It does not +enable tools, broaden paths, retain project content, or displace a primary MCP result that already +needs the response budget. + +## Descriptor policy and process policy + +The project descriptor is canonical project configuration. The process policy is a runtime +restriction. They compose by intersection: + +```text +operation is available + only if core permits it + and the startup binding registers it + and no-AST permits it + and required resources exist + and the descriptor grants the requested project authority + and current graph/hash preconditions validate +``` + +Changing a descriptor does not retarget a running project-owned process. Descriptor, adapter +implementation, or launcher drift requires a fresh process. + +## Common decisions + +For an agent that only reads documentation: + +```bash +docforge-mcp \ + --project-root /absolute/path/MyProject \ + --capability-mode read +``` + +For an agent that may prepare reviewable proposals: + +```bash +docforge-mcp \ + --project-root /absolute/path/MyProject \ + --capability-mode proposal \ + --proposal-writer project-editor +``` + +For a tightly controlled application process: + +```bash +docforge-mcp \ + --project-root /absolute/path/MyProject \ + --capability-mode application \ + --proposal-writer project-editor \ + --canonical-applier project-editor +``` + +For a read binding with every active output projection disabled: + +```bash +docforge-mcp \ + --project-root /absolute/path/MyProject \ + --capability-mode read \ + --manual-render-policy disabled \ + --portable-graph-policy disabled \ + --live-viewer-policy disabled +``` + +Prefer the narrowest binding that completes the workflow. Call `docforge_bootstrap` first and use +the returned effective policy, projection policy, actual capabilities, and prohibitions rather +than assumptions based on client configuration. + +See the [MCP contract](MCP_CONTRACT.md), [Security](SECURITY.md), [Project +descriptor](PROJECT_DESCRIPTOR.md), and [Rendering and +visualization](RENDERING_AND_VISUALIZATION.md). diff --git a/docs/PROJECT_DESCRIPTOR.md b/docs/PROJECT_DESCRIPTOR.md new file mode 100644 index 0000000..6e1be88 --- /dev/null +++ b/docs/PROJECT_DESCRIPTOR.md @@ -0,0 +1,279 @@ +# Project descriptor + +A generic DocForge project is selected by one fixed file: +`.docforge/project.toml` beneath an explicit project root. The descriptor is schema version 1. +DocForge validates both the JSON-schema shape in `schemas/project.schema.json` and runtime +invariants that schema alone cannot prove. + +Reference integrations use a different fixed descriptor, +`.docforge/reference-adapter.toml`; see [Reference adapters](REFERENCE_ADAPTERS.md). +Project-owned adapters construct the same runtime `ProjectDescriptor` contract through the public +adapter SDK. + +## Complete generic example + +```toml +schema_version = 1 +project_id = "my-project" +title = "My Project" +adapter = "generic" + +[sources] +content_roots = ["docs/docforge/content"] +authority_files = [] + +[derived] +cache_root = ".docforge/cache" +index = ".docforge/cache/index.sqlite3" + +[changesets] +root = ".docforge/changesets" + +[[changesets.writers]] +id = "project-editor" +families = ["architecture", "operations", "system"] +operations = ["create", "update", "move", "delete"] + +[render] +template_root = ".docforge/templates" +preview_root = ".docforge/previews" + +[[render.views]] +id = "manual" +renderer = "generic_html" +template = "manual.html" +output = ".docforge/rendered/manual.html" +title = "My Project Manual" +families = ["architecture", "operations", "system"] + +[graph_render] +output_root = ".docforge/portable-graph" + +[[graph_render.views]] +id = "architecture" +renderer = "portable_graph_html" +output = "architecture.html" +title = "Architecture" +root = "architecture.overview" +initial_mode = "web" +depth = 3 +max_nodes = 250 +max_edges = 1000 +max_work = 100000 +families = ["architecture", "system"] +relations = ["depends_on", "owns", "calls", "reads", "writes", "tested_by", "relates_to"] +authorities = [] +statuses = ["current", "active", "verified"] +tags = [] +include_logic = false + +[graph] +allowed_relations = [ + "calls", + "depends_on", + "owns", + "reads", + "relates_to", + "tested_by", + "writes", +] + +[limits] +max_source_bytes = 500000 +max_nodes = 10000 +max_query_chars = 500 +max_results = 100 +max_traversal_depth = 6 +max_context_tokens = 12000 +max_tool_output_chars = 200000 +max_changesets = 100 +max_changeset_operations = 100 +max_changeset_bytes = 1000000 +max_render_views = 20 +max_template_bytes = 1000000 +max_render_bytes = 1000000 + +[[profiles]] +id = "development" +families = ["architecture", "operations", "system"] +statuses = ["current", "active", "verified"] +required_nodes = ["architecture.overview"] +token_budget = 8000 +dependency_depth = 3 +``` + +Rendering sections are optional. Profiles and limits may also be omitted; runtime defaults then +apply. The required top-level fields are `schema_version`, `project_id`, `title`, `adapter`, +`sources`, `derived`, `changesets`, and `graph`. + +## Identity fields + +`schema_version` must be `1`. + +`project_id` is the stable machine identity. It is lowercase and may contain digits, dots, +underscores, and hyphens after its first character. Do not derive it from a mutable display title. + +`title` is the human-readable project name. + +`adapter` is `generic` for this file format. Project-owned adapters publish a validated +`adapter_id@adapter_version` identity through their runtime descriptor; changing adapter identity +invalidates incompatible derived state. + +The descriptor's byte content contributes to a descriptor hash. Client fragments, launchers, index +evidence, and policy results use that hash to detect drift. + +## Canonical sources + +`sources.content_roots` lists the project-relative directories containing generic Markdown and TOML +nodes. Each path must resolve beneath the project root. Canonical content roots may not overlap the +derived cache. + +`sources.authority_files` lists additional project-relative regular files whose content belongs to +the canonical project identity. They are not automatically parsed as nodes. + +A Markdown node contains one TOML metadata block followed by Markdown content: + +```markdown ++++ +schema_version = 1 +id = "architecture.overview" +title = "Architecture overview" +family = "architecture" +authority = "authoritative" +status = "current" +tags = ["architecture"] +summary = "Defines the top-level architecture and ownership." ++++ + +# Architecture overview + +Describe systems, ownership, runtime flow, failure behavior, and proof. +``` + +Every node ID is project-wide and stable. Every relationship target must resolve. A TOML source may +contain multiple `[[nodes]]` records; proposal-enabled multi-node files need stable +`source_anchor` values where creation or movement requires an exact record boundary. + +## Derived state + +`derived.cache_root` owns disposable indexes, attestations, extraction caches, render receipts, +projection artifacts, and viewer registry state. + +`derived.index` must be inside `derived.cache_root`. Canonical content and cache paths must not +overlap. + +Derived state is not a backup. If it is deleted or rejected as corrupt, DocForge rebuilds it from +validated canonical sources. + +## Changesets and writers + +`changesets.root` is the confined proposal store. It must not overlap canonical content or the +derived cache. + +Each `changesets.writers` entry declares: + +- a stable writer `id`; +- the node `families` that writer may change; +- allowed `operations`: `create`, `update`, `move`, and/or `delete`. + +The descriptor grant is necessary but not sufficient. A process must also select that writer at +startup, and canonical application requires a separately bound matching applier. Capability mode +does not broaden the descriptor grant. See [Policy precedence](POLICY_PRECEDENCE.md). + +## Allowed relationships + +`graph.allowed_relations` is the exact project vocabulary accepted on edges. It must be nonempty. +Relationship names are stable IDs. DocForge rejects relationships that are not declared and gives +`depends_on` additional cycle validation. + +The core does not reinterpret a custom relationship just because its spelling resembles a known +term. Task-context retrieval classifies only the documented versioned aliases and preserves +unknown allowed relationships as `unclassified`. + +## Context profiles + +Each `profiles` entry defines one bounded context compilation: + +- `id` selects the profile; +- `families` and `statuses` filter eligible nodes; +- `required_nodes` names stable nodes that must be present; +- `token_budget` limits compiled content; +- `dependency_depth` bounds relationship expansion. + +Profiles choose derived retrieval scope. They do not change node authority or writer permissions. + +## Limits + +Positive limits bound input, graph, retrieval, proposal, and rendering work. Current defaults are: + +- `max_source_bytes = 1000000` +- `max_nodes = 10000` +- `max_query_chars = 500` +- `max_results = 100` +- `max_traversal_depth = 8` +- `max_context_tokens = 32000` +- `max_tool_output_chars = 200000` +- `max_changesets = 1000` +- `max_changeset_operations = 100` +- `max_changeset_bytes = 1000000` +- `max_render_views = 100` +- `max_template_bytes = 1000000` +- `max_render_bytes = 1000000` + +Smaller project limits are useful policy. They cannot widen fixed internal worker, package, +response, or cache ceilings. In particular, detached renderer transfer has its own fixed boundary +even if a compatibility descriptor retains a larger `max_render_bytes`. + +## Manual rendering + +The optional `render` section declares: + +- one confined `template_root`; +- one isolated `preview_root`; +- one or more stable views. + +Each view uses the built-in `generic_html` renderer, a template beneath `template_root`, one +declared output path, a title, and a family filter. Paths may not overlap canonical content, +authority files, changesets, cache, templates, or previews in unsafe ways. + +Templates are inert UTF-8 files with a fixed token vocabulary. They cannot select executable +renderers or commands. See [Rendering and visualization](RENDERING_AND_VISUALIZATION.md). + +## Portable graph rendering + +The optional `graph_render` section declares an output root and one or more +`portable_graph_html` views. Each view selects exactly one: + +- `root`, an exact stable node ID; or +- `query`, a bounded metadata-only lexical seed. + +It may then restrict families, relations, authorities, statuses, and tags, plus depth, node, edge, +and work limits. `initial_mode` is `nodes`, `flow`, or `web`. Portable graph contract version 1 +requires `include_logic = false`. + +The declared `output` is relative to `graph_render.output_root`. + +## Paths and confinement + +Descriptor paths are project-relative. Absolute paths and parent traversal are rejected. Runtime +validation also rejects symlink escapes, unexpected file types, unsafe overlap, changing path +identity during sensitive reads or publication, and derived outputs outside their declared roots. + +The explicit CLI `--project-root` is the only project selector. The MCP process binds it at startup +and exposes no project-switching tool. + +## Validate changes safely + +After editing the descriptor: + +```bash +docforge --project-root /absolute/path/MyProject validate +docforge --project-root /absolute/path/MyProject reindex +docforge --project-root /absolute/path/MyProject check +``` + +Descriptor or adapter implementation drift makes a project-owned running process fail closed; start +a fresh process after changing those boundaries. + +For first-time creation, prefer the create-only [New-project +quickstart](NEW_PROJECT_QUICKSTART.md). For full invariants, read the [Core contract](CONTRACT.md). diff --git a/docs/PROJECT_ONBOARDING.md b/docs/PROJECT_ONBOARDING.md index 90a75ef..a16efea 100644 --- a/docs/PROJECT_ONBOARDING.md +++ b/docs/PROJECT_ONBOARDING.md @@ -5,6 +5,11 @@ DocForge onboarding has two separate outcomes: 1. A generic manual can be configured, indexed, rendered, visualized, and exposed through the MCP. 2. A source graph additionally requires one validated language frontend per source language. +DocForge ships narrow fixed references for Python, JavaScript, TypeScript, and C++. They are useful +for syntax-scoped projects and adoption proof, but production semantic requirements may still +require a project-owned compiler or language-service adapter. Review +[Reference Adapters](REFERENCE_ADAPTERS.md) before selecting a frontend. + The onboarding command never claims that source semantics exist merely because it found source files. It reports each detected language as `adapter_required` until a project integration supplies and proves that frontend. @@ -64,6 +69,32 @@ immediately usable through the generic CLI, viewer, and MCP. The starter overview records detected languages and states that the source graph is unavailable until a language frontend passes the adapter proof. That limitation is deliberate. +## Configure a fixed reference adapter + +The generic onboarding scaffold and fixed reference configuration are separate project routes. +For a syntax-scoped reference project, create `.docforge/reference-adapter.toml`: + +```toml +schema_version = 1 +project_id = "my-python-project" +title = "My Python project" +language = "python" +source_roots = ["src"] +``` + +Then start the fixed read-only binding: + +```bash +python -I -m docforge.reference_mcp \ + --project-root /absolute/path/MyProject \ + --capability-mode read +``` + +The configuration selects only a fixed in-repository provider and cannot name a command or custom +module. JavaScript, TypeScript, and C++ require their respective optional extras; C++ also requires +`compilation_database = "compile_commands.json"`. The complete configuration and supported-fact +contract are in [Reference Adapters](REFERENCE_ADAPTERS.md). + ## Complete onboarding checklist ### 1. Repository assessment @@ -118,6 +149,13 @@ For every source language: - [ ] Define dependency discovery. - [ ] State unsupported semantic facts explicitly. +Decide whether the project needs production semantic evidence or the narrower syntax-only +reference scope. The Python reference publishes only project-local imports. JavaScript and +TypeScript publish only project-local static relative imports and re-exports. The C++ reference +publishes only directly resolvable project-local quoted includes and does not run a compiler. None +of those references resolves calls, inheritance, types, symbols, runtime behavior, or semantic +ownership. + All frontends emit the same DocForge contracts: - `AdapterManifest` inventories fingerprinted extraction units and dependencies. @@ -151,6 +189,11 @@ this checklist summarizes. - [ ] Assign shared symbols to one deterministic source contribution. - [ ] Record compiler-derived project include dependencies. +These are production semantic-adapter expectations. The built-in C++ reference uses +`compile_commands.json` only as bounded translation-unit inventory and fingerprint evidence. It +parses commands and arguments as inert data, executes no command or compiler, and does not claim +compiler include semantics, symbol ownership, or a Clang-derived graph. + #### Rust - [ ] Read the Cargo workspace and package graph. @@ -240,6 +283,10 @@ from filenames. Done when a new session can identify and retrieve the correct project without being told its file layout. +The fixed reference server is read-only. Generic and custom-adapter client generation, including +the immutable custom launcher boundary, is documented in +[Agent Integration](AGENT_INTEGRATION.md). + ### 10. Operating guide and maintenance - [ ] Record the authority and progressive-reading order. @@ -286,14 +333,18 @@ to one configured project root. DocForge does not let an MCP call install dependencies, run project builds, modify Git, deploy, or publish. A project integration may use its own normal development workflow for those actions. -## Frontend packaging direction +## Reference and production frontend boundaries -Reusable language frontends should be separate packages or project-owned adapters over the public -DocForge contracts. They must not put language-specific rules into the graph, index, viewer, or MCP -core. +The public authoring namespace is `docforge.adapter_sdk`. Project-owned and separately distributed +production frontends should build on that contract without putting language-specific rules into +the graph, index, viewer, or MCP core. -Worldforge is the first complete C++ reference integration. A reusable C++ package should be -extracted only after that integration proves stable ownership, compiler dependency invalidation, -and complete/incremental equivalence. Rust and Java frontends should then implement the same -contract using their authoritative build and language tooling rather than copying C++ extraction -rules. +The in-repository Python, JavaScript, TypeScript, and C++ adapters are reference implementations. +Their syntax-scoped behavior is useful without becoming a claim that every project in those +languages has complete semantic coverage. A compiler-backed production C++ adapter may resolve +build flags, calls, types, inheritance, include semantics, and ownership when it can prove those +facts. The reference C++ adapter does none of that and is not a Clang semantic adapter. + +New Rust, Java, or other frontends should use their authoritative build and language tooling and +must pass the graph-plus-Logic complete/incremental contract in the +[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md). diff --git a/docs/RECOVERY_AND_PERFORMANCE.md b/docs/RECOVERY_AND_PERFORMANCE.md new file mode 100644 index 0000000..62f6499 --- /dev/null +++ b/docs/RECOVERY_AND_PERFORMANCE.md @@ -0,0 +1,86 @@ +# Recovery and performance + +DocForge keeps canonical project evidence separate from disposable indexes, caches, receipts, and +rendered projections. Recovery rebuilds derived state from current authority; it does not rewrite +canonical content to make a cache look valid. + +For the underlying boundaries, see [core authority](CORE_CONCEPTS_AND_AUTHORITY.md) and the +[security model](SECURITY.md). + +## Recovery order + +Use the narrowest verified recovery: + +1. Run `docforge check` or the corresponding MCP status operation. +2. Run `docforge sync` to repair a missing or stale disposable index when safe. +3. Run `docforge reindex` for an explicit complete rebuild. +4. Restart a long-running MCP or viewer binding after descriptor or adapter implementation drift. +5. Regenerate a derived client fragment, command reference, preview, manual, or portable graph + from current project evidence. + +Never delete or rewrite canonical sources, active changesets, or reviewed hashes as cache cleanup. + +## Extraction-cache recovery + +Incremental adapter contributions are stored in a version-1 disposable extraction cache. Reads are +no-follow, regular-file-only, identity-checked, bounded to 64,000,000 bytes and 10,000 source +records, and fail to a cache miss on malformed or incompatible data. Publication is atomic. + +After a miss, current manifest sources are extracted again and the complete assembly is validated. +The active SQLite generation remains authoritative for reads until a new verified index is +published. An extraction cache may therefore be safely ahead of the last index; the two files do +not pretend to be one transaction. + +## Index and receipt recovery + +A validated SQLite index is a generation-pinned derived snapshot. Missing, corrupt, unattested, or +stale indexes rebuild from the complete project or adapter oracle. Generation receipts and render +receipts are post-publication evidence. Failure to write a receipt after a committed artifact is +reported as degraded committed success, not as permission to repeat a mutation. + +The live viewer pins one validated index identity. Index replacement makes the running snapshot +stale and causes a later visualize request to start a fresh worker. + +## Proposal and application recovery + +Hash or base conflicts are not cache failures. Retrieve the current changeset and diff, then +review the new exact hash. Rebase is allowed only when every touched node, relationship, source, +permission, and graph invariant still matches. A content conflict requires a new proposal. + +If a canonical serializer fails its round-trip check, use its reported rollback state. Do not +reapply a changeset whose application may already have committed. See +[migrating from v1](MIGRATING_FROM_V1.md) for rollback planning. + +## Milestone 4 scale evidence + +The maintained Python reference benchmark creates 334 Python source files and produces: + +- 1,002 primary nodes and 1,001 primary edges. +- 334 Logic projections with 2,338 Logic nodes and 2,338 Logic edges. +- Exact complete/incremental primary graph and Logic equality. +- Exact output after corrupt extraction-cache recovery and corrupt-index recovery. +- Zero `ast.parse` calls and zero `extract_source` calls during a warm build. + +On the frozen Milestone 4 candidate, operation p95 values were 1.266 to 1.898 seconds. Per-operation +traced peaks were about 61 to 67 MiB, and process high-water was 78,798,848 bytes. These are +regression measurements from one machine, not universal latency promises. The machine-readable +record is `../benchmarks/milestone4-2026-07-29.json`; detailed method and hashes are in +[the Milestone 4 baseline](MILESTONE_4_BASELINE.md). + +The zero-parser claim is deliberately Python-only. Focused JavaScript and TypeScript tests prove +their manifests avoid Tree-sitter. The C++ reference manifest currently uses Tree-sitter while +discovering bounded quoted includes, so a warm C++ cache hit is not evidence of zero parser work. + +## Maintained gates + +Run: + +```bash +make gate +make adoption-m4 +make benchmark-m4-full +make docs-check +``` + +The main gate includes smoke benchmarks. Full milestone evidence is recorded separately from a +clean candidate so smoke or dirty-tree results cannot become release claims. diff --git a/docs/REFERENCE_ADAPTERS.md b/docs/REFERENCE_ADAPTERS.md new file mode 100644 index 0000000..4287651 --- /dev/null +++ b/docs/REFERENCE_ADAPTERS.md @@ -0,0 +1,149 @@ +# Reference adapters + +DocForge includes fixed, syntax-scoped reference adapters for Python, JavaScript, TypeScript, and +C++. They demonstrate the public adapter contract, deterministic complete and incremental +publication, function Logic, cache recovery, and a runnable read-only MCP binding. They are not +compiler or language-service replacements. + +Use a reference adapter when its deliberately narrow graph is sufficient or when proving a fresh +DocForge integration. Use the [Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md) for a +production adapter that needs resolved symbols, calls, inheritance, types, build semantics, or +semantic ownership. + +## Installation + +The Python reference adapter uses the standard library and is available in the base wheel. The +other frontends are separate optional extras: + +```bash +python -m pip install docforge +python -m pip install "docforge[javascript]" +python -m pip install "docforge[typescript]" +python -m pip install "docforge[cpp]" +``` + +Install `docforge[languages]` only when one environment intentionally needs all three Tree-sitter +frontends. JavaScript and TypeScript use distinct grammar packages and distinct extras. + +## Fixed project configuration + +The runnable reference binding reads exactly +`.docforge/reference-adapter.toml` below the selected project root. The configuration is data only: +it cannot name a provider, Python module, command, arguments, working directory, environment, or +discovery rule. + +Reference adapter configuration `.docforge/reference-adapter.toml`: + +```toml +schema_version = 1 +project_id = "example-python" +title = "Example Python project" +language = "python" +source_roots = ["src"] +``` + +`language` is one of `python`, `javascript`, `typescript`, or `cpp`. Source roots must be sorted, +unique, non-overlapping project-relative directories outside `.docforge`. The configuration, +roots, and inventoried source files must be regular confined paths without symlink traversal. The +configuration is limited to 65,536 bytes, at most 64 source roots, and 65,536 examined inventory +entries. + +C++ additionally requires one explicit project-confined compilation database: + +```toml +schema_version = 1 +project_id = "example-cpp" +title = "Example C++ project" +language = "cpp" +source_roots = ["include", "src"] +compilation_database = "compile_commands.json" +``` + +`compilation_database` is forbidden for the other three languages. + +## Published scope + +All four adapters publish deterministic file, module or translation-unit, class or struct, and +function nodes where their syntax supports those categories. They publish lexical `contains` +relationships, a narrow set of project-local `depends_on` relationships, and function-scoped +Logic. Calls that appear inside Logic are syntax steps, not resolved symbol relationships. + +### Python + +Python uses `ast` from the standard library for source extraction and Logic. Manifest construction +does not build a Python AST; it tokenizes imports only far enough to publish dependencies that +resolve to another module in the declared source inventory. + +It does not import or execute project code. It does not resolve dynamic imports, calls, +inheritance, imported symbols, types, overloads, re-exports, decorators, metaclasses, descriptors, +or runtime-generated behavior. + +### JavaScript and TypeScript + +JavaScript uses the optional `tree-sitter-javascript` grammar. TypeScript uses the distinct +optional `tree-sitter-typescript` grammar. Their focused incremental tests prove that manifest +construction and an unchanged warm build do not invoke the Tree-sitter extraction parser. + +Only static relative imports and re-exports that resolve to another inventoried source file become +dependencies. Dynamic `import()`, `require()`, bare package specifiers, aliases, `tsconfig` paths, +loader hooks, types, interfaces, overloads, calls, inheritance, symbols, and runtime behavior are +not resolved. The adapters never import, compile, transpile, or execute project code. + +### C++ + +C++ treats `compile_commands.json` as the authoritative bounded translation-unit inventory and as +fingerprint evidence. Commands, arguments, directories, and output fields are parsed as inert +data. The adapter executes no compiler, build tool, command, project binary, or project code. + +The reference C++ manifest uses the optional `tree-sitter-cpp` grammar to parse inventoried sources +and discover quoted includes. It publishes a dependency only when that quoted include resolves +directly to a real project-local header in the declared roots. Angle-bracket includes, compiler +include paths, frameworks, generated headers, compiler-provided headers, conditional compilation, +and macro expansion are omitted. + +This is not a Clang semantic adapter. It does not claim resolved calls, types, templates, aliases, +concepts, references, inheritance, overload ownership, out-of-line semantic ownership, macro +semantics, or compiler include semantics. A warm C++ extraction-cache hit is therefore not evidence +that manifest construction performed zero parser work. + +## Complete and incremental proof + +Each reference adapter implements: + +- `load_projection()` for the complete primary graph compatibility oracle; +- `load_complete_assembly()` for the complete graph-plus-Logic oracle; +- `load_manifest()` and `extract_source()` for incremental extraction; and +- deterministic assembly of the complete current contribution set. + +The maintained fixtures prove complete determinism, exact complete/incremental graph and Logic +parity, reverse-dependency invalidation, additions and deletions, corrupt extraction-cache +recovery, confinement, and exact unsupported-fact inventories. See +[Incremental Adapter Indexing](INCREMENTAL_INDEXING.md) for the cache contract and +[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for compatibility and policy limits. + +Current fixture evidence is: + +| Language | Primary nodes | Relationships | Logic projections | +|---|---:|---:|---:| +| Python | 14 | 13 | 5 | +| JavaScript | 14 | 14 | 5 | +| TypeScript | 13 | 14 | 4 | +| C++ | 17 | 16 | 5 | + +These are regression-fixture shapes, not promises for arbitrary repositories. + +## Read-only reference server + +Start the fixed server with: + +```bash +python -I -m docforge.reference_mcp \ + --project-root /absolute/path/to/project \ + --capability-mode read +``` + +The server selects one of the four in-repository providers solely from the validated fixed +configuration. Its cache stays below `.docforge/cache/reference-adapter/`. It registers +the 21-tool read surface and no proposal or application tools. See +[Agent Integration](AGENT_INTEGRATION.md) for generated client fragments and +[MCP Boundary](MCP_CONTRACT.md) for the exact tool contract. diff --git a/docs/RENDERING_AND_VISUALIZATION.md b/docs/RENDERING_AND_VISUALIZATION.md new file mode 100644 index 0000000..90bbeb4 --- /dev/null +++ b/docs/RENDERING_AND_VISUALIZATION.md @@ -0,0 +1,233 @@ +# Rendering and visualization + +DocForge has three independent ways to present one validated graph generation: + +```text +validated generation + ├── manual plan → immutable package → detached HTML renderer → declared manual output + ├── graph plan → immutable package → detached graph renderer → portable static artifact + └── pinned index → managed loopback viewer → interactive Nodes, Flow, Web, and lazy Logic +``` + +They share validated facts but not authority, publication, or lifecycle. A manual render does not +publish a portable graph. A portable graph does not start the live viewer. None is canonical +project content. + +## Manual rendering + +A generic descriptor may declare a manual template root, isolated preview root, and stable views: + +```toml +[render] +template_root = ".docforge/templates" +preview_root = ".docforge/previews" + +[[render.views]] +id = "manual" +renderer = "generic_html" +template = "manual.html" +output = ".docforge/rendered/manual.html" +title = "My Project Manual" +families = ["architecture", "operations", "system"] +``` + +The renderer name is fixed to `generic_html`. Templates are confined UTF-8 assets with a fixed +token vocabulary. They cannot name commands, Python modules, executable renderers, or arbitrary +publication paths. Raw HTML in canonical Markdown is disabled by the pinned CommonMark path. + +Plan and render identity covers the canonical source hash, optional changeset hash, selected nodes +and relationships, view configuration, template hash, renderer contract, and parser version. + +Use: + +```bash +docforge --project-root "$PROJECT" render-status +docforge --project-root "$PROJECT" render-status manual --deep +docforge --project-root "$PROJECT" render manual +docforge --project-root "$PROJECT" preview CHANGESET_ID manual +``` + +Normal status verifies bounded source, configuration, template, output, renderer, and receipt +identities without reconstructing output. `--deep` explicitly runs the side-effect-free full-render +oracle. A changeset preview writes only to the isolated preview root. + +Manual `auto` means that a successful canonical application owns regeneration of every declared +manual view. It does not mean background rendering, and ordinary standalone CLI render cannot +select `auto`; use `explicit`. + +## Portable graph rendering + +Portable graph configuration is separate: + +```toml +[graph_render] +output_root = ".docforge/portable-graph" + +[[graph_render.views]] +id = "architecture" +renderer = "portable_graph_html" +output = "architecture.html" +title = "Architecture" +root = "architecture.overview" +initial_mode = "web" +depth = 3 +max_nodes = 250 +max_edges = 1000 +max_work = 100000 +families = ["architecture", "system"] +relations = ["depends_on", "owns", "calls", "reads", "writes", "tested_by", "relates_to"] +authorities = [] +statuses = ["current", "active", "verified"] +tags = [] +include_logic = false +``` + +A view selects exactly one exact `root` node or bounded metadata-only lexical `query`. Exact +filters and node, edge, depth, and work limits close the selection. The initial mode is `nodes`, +`flow`, or `web`. Portable graph contract version 1 excludes function-scoped Logic. + +Use: + +```bash +docforge --project-root "$PROJECT" graph-plan architecture +docforge --project-root "$PROJECT" graph-render architecture +docforge --project-root "$PROJECT" graph-render-status architecture +``` + +`graph-plan` validates and returns the generation-pinned plan without publication. `graph-render` +is an explicit local CLI publication action. It commits a content-addressed artifact, renderer +receipt, and bounded generation/view manifest in that order; the manifest is the publication +commit. `graph-render-status` checks only bounded committed evidence and never plans or renders. + +MCP may plan and inspect portable graph evidence, but it does not publish the portable artifact. +Use the explicit project-local CLI command for publication. + +Portable output is a complete static artifact. JavaScript is progressive enhancement, not a +requirement for the graph facts to be present. + +## Live viewer + +The live viewer is a managed, read-only browser over one generation-pinned validated SQLite file. +Install the per-user manager once: + +```bash +docforge-viewer-manager install-user-service +``` + +Then operate a project viewer: + +```bash +docforge --project-root "$PROJECT" visualize +docforge --project-root "$PROJECT" visualize --node architecture.overview +docforge --project-root "$PROJECT" visualize --query persistence +docforge --project-root "$PROJECT" visualization-status +docforge --project-root "$PROJECT" visualization-stop +``` + +`visualize` accepts only an optional stable node ID or lexical query, bounded traversal depth, and +the local `--no-open` presentation choice. It does not accept a project root override, database +path, SQL, template, command, renderer, bind address, or module. + +The HTTP listener binds to `127.0.0.1` on an operating-system-selected port. A random token is part +of every accepted path. The server supports only `GET` and `HEAD`, sets no-store and restrictive +browser security headers, and has no write, project-selection, arbitrary-query, or static +filesystem endpoint. + +The viewer is a snapshot. Index replacement or alteration makes that snapshot fail closed; start a +new visualization to use a new validated generation. Source inspection reads only project-confined +source evidence bound to the pinned generation. + +## Nodes, Flow, Web, and Logic + +The interactive viewer offers four complementary projections: + +- **Nodes** shows a bounded relation-neutral incoming and outgoing neighborhood. +- **Flow** presents semantic contributors toward the focus. Prerequisite-style stored + relationships may be reversed for presentation without changing stored direction. +- **Web** expands the convergence picture with contributors, callers, containers, members, and + contextual relationships. +- **Logic** loads a function-scoped control-flow projection only when requested. + +Logic supports the explicit control paths published by Python, JavaScript, TypeScript, and C++ +integrations. It stays outside the primary graph, portable graph version 1, search, and +generation-diff receipts. + +Hiding a node is presentation-only. In Flow and Web, ancestors without another path to the focus +are removed. In Logic, an omitted-path bridge preserves downstream readability. Restore reverses +the presentation change; neither action mutates the index. + +## Immutable plan and worker boundary + +Manual and graph plans are versioned, canonical JSON with deterministic ordering and fixed +structural and byte limits. They contain selected graph facts and bounded content, but no live +project object, SQLite handle, absolute project or index path, arbitrary query, command, +executable path, or caller-selected renderer module. + +A projection package binds one plan to inert assets, fixed component versions, a closed built-in +renderer identity, and an exact artifact inventory. The detached worker: + +- runs one fixed private module through isolated Python; +- uses a trusted working directory and sanitized environment; +- accepts one canonical newline-terminated JSON request; +- returns one bounded canonical JSON response; +- has a fixed renderer allowlist and timeout; +- cannot select graph facts, read project state, choose output paths, or mutate canonical files. + +The package contract is bounded, and actual artifact transfer has a fixed 20,000,000-byte ceiling. +A larger descriptor `max_render_bytes` compatibility value does not widen that worker boundary. + +## Manual fragment reuse + +Manual fragments are disposable semantic cache records, not publication authority. A cold record +is accepted only after byte-exact comparison against a full detached render. On a warm hit, the +worker independently recomputes the expected fragment before reuse. + +Corrupt, forged, stale, incompatible, individually oversized, or aggregate-oversized records fall +back to the complete full-render oracle. The current cache is bounded to 10,000 records and +64,000,000 bytes. + +Fragment reuse is a correctness and recovery boundary. Do not promise a speedup without current +measurements. + +## Projection policy + +The version-2 projection policy is independent of capability mode: + +```text +manual: auto | explicit | disabled +portable_graph: explicit | disabled +live_viewer: on-demand | disabled +``` + +Set global CLI flags before the subcommand: + +```bash +docforge --project-root "$PROJECT" --manual-render-policy disabled render manual +docforge --project-root "$PROJECT" --portable-graph-policy disabled graph-plan architecture +docforge --project-root "$PROJECT" --live-viewer-policy disabled visualize +``` + +An active operation blocked by policy fails before hidden work. Receipt-only manual and portable +status remain available. Viewer status and explicit stop remain available when viewer startup is +disabled. + +A non-disabled selection also requires its resource: declared manual configuration, declared +portable graph configuration, or viewer runtime. Manual `auto` additionally requires canonical +application. Read [Policy precedence](POLICY_PRECEDENCE.md) for exact composition. + +## Failure and recovery + +Render input drift detected before replacement fails without publishing a current receipt for stale +output. Portable graph publication uses content-addressed evidence and a final manifest commit. +Status never repairs implicitly. + +Recovery is explicit: + +- rerun a manual render from validated canonical state; +- use deep manual status for the full, side-effect-free equivalence oracle; +- rerun portable graph publication, or repair only from validated content-addressed evidence; +- stop and restart a stale live viewer against a current validated index. + +Do not recover a derived output by editing its receipt or treating it as canonical. See [Recovery +and performance](RECOVERY_AND_PERFORMANCE.md), [Security](SECURITY.md), and the [Viewer +manager](VIEWER_MANAGER.md). diff --git a/docs/SECURITY.md b/docs/SECURITY.md new file mode 100644 index 0000000..4c8569e --- /dev/null +++ b/docs/SECURITY.md @@ -0,0 +1,101 @@ +# Security model + +DocForge is a project-bound knowledge compiler. Its security boundary is an explicit project root, +closed configuration, bounded data, and exact identities. It is not a general process sandbox. + +Start with [core authority](CORE_CONCEPTS_AND_AUTHORITY.md), then use +[policy precedence](POLICY_PRECEDENCE.md) to decide which capabilities a server should expose. + +## Project and path confinement + +Descriptors, reference-adapter configurations, canonical sources, authority files, templates, +changesets, caches, indexes, previews, and declared outputs are resolved against one project root. +DocForge rejects absolute paths where only project-relative paths are allowed, parent traversal, +symbolic-link escapes, unsafe file types, and protected-root overlap. Important reads use +no-follow file descriptors and compare file identity before and after reading. + +Confinement protects DocForge operations. It does not stop another process with repository access +from changing files. Long-running bindings revalidate descriptor and adapter implementation +identity and require a restart after drift. + +## Untrusted project content + +Documentation, source text, templates, adapter metadata, compiler-database entries, and changeset +content are data. They cannot redefine policy or instruct DocForge to execute a command. Generic +manual rendering disables raw HTML, accepts a fixed template vocabulary, and rejects script-like +content. Portable graph and manual workers accept validated inert packages and fixed built-in +renderer identities. + +The C++ reference adapter reads `compile_commands.json` only as bounded translation-unit inventory +and fingerprint evidence. It never executes the recorded command, compiler, response file, or +project program. + +## Adapter launcher boundary + +`AdapterLauncherV1` contains one Python module name and project identity. It contains no command, +shell string, arbitrary argument list, working directory, environment, discovery rule, or callable +selector. + +Custom modules must be installed top-level modules. An isolated `python -I` probe resolves the +module without importing it and requires its regular-file origin to remain inside the bound +project. The sole trusted dotted exception is the packaged `docforge.reference_mcp` module. Client +fragments use the exact validated interpreter and canonical fixed arguments with an empty +environment. + +This proves that the declared module is resolvable and project-bound. It does not make arbitrary +module code safe. Project owners remain responsible for the implementation they install. + +## Mutation boundary + +Normal MCP and the fixed reference server are read-only. Proposal tools exist only when a +startup-bound writer is authorized by the descriptor. Canonical application exists only when a +matching applier is explicitly configured. + +Every proposal append, rebase, abandonment, and application is hash-bound. Application requires +the exact changeset hash that was reviewed. Source identity, content hashes, permissions, +conflicts, graph validity, and serializer round trips are checked before success. DocForge never +turns prose approval into a fuzzy merge. + +## Derived state and publication + +SQLite indexes, source-generation receipts, extraction caches, render fragments, previews, and +portable artifacts are disposable. Corrupt, stale, foreign, oversized, or mismatched derived +state is rejected or rebuilt from current project evidence. + +Generated command-reference publication serializes cooperating writers and uses no-clobber or +compare-and-swap publication. A raced target is restored or retained for recovery instead of being +silently discarded. Projection publication records when an artifact was committed but later +receipt verification degraded, so a completed mutation is never reported as an ordinary failure. + +## Limits and denial-of-service resistance + +Inputs, results, traversal, context, changesets, renders, worker protocols, manifests, adapter +assemblies, and extraction caches have explicit count and byte limits. Incremental extraction +caches are capped at 10,000 sources and 64,000,000 bytes. Adapter primary nodes use the project +`max_nodes` limit; edges and Logic have deterministic multipliers over that limit. + +Limits reduce accidental and adversarial amplification. An in-process adapter can still allocate +memory before returning data, so only trusted project-owned adapter code should run in the server +process. + +## Secrets and network behavior + +DocForge does not copy the parent environment into generated client fragments or detached +projection workers. Doctor checks never return environment values. The live viewer binds to +loopback, uses an unguessable URL token, supports read-only methods, and serves no arbitrary +filesystem tree. + +Project secrets must not be placed in canonical documentation, adapter configuration, compiler +databases, templates, or changesets. Repository release gates include secret scanning, but that +scan is not a substitute for credential hygiene. + +## No-AST boundary + +`--no-ast` is a binding policy that preserves the selected adapter and prohibits Logic publication +and retrieval. It is not a parser detector, filesystem sandbox, or promise that unrelated +processes cannot parse source. See [legacy and no-AST operation](LEGACY_AND_NO_AST.md). + +## Reporting and recovery + +Do not bypass a confinement, identity, policy, hash, or limit error. Preserve the failing evidence, +stop the affected binding, and follow [recovery and performance](RECOVERY_AND_PERFORMANCE.md). diff --git a/docs/USER_MANUAL.md b/docs/USER_MANUAL.md index 640f9ae..21f3007 100644 --- a/docs/USER_MANUAL.md +++ b/docs/USER_MANUAL.md @@ -35,6 +35,8 @@ incremental methods while retaining the full loader as a fallback. relationship keys, source inspection, branch-aware node hiding, panel resizing, zooming, and managed idle shutdown. - A generic Markdown/TOML adapter plus contracts for deterministic project-owned adapters. +- A public adapter SDK, optional Python/JavaScript/TypeScript/C++ reference integrations, and one + fixed read-only reference MCP binding. DocForge does not run shell commands from documentation, mutate Git, build an application, deploy, publish, choose a project globally, or cross project boundaries. @@ -100,6 +102,49 @@ uv run pytest -q Use the executables under `/absolute/path/DocForge/.venv/bin/` when DocForge is not installed into the active shell environment. +For an installed distribution, choose only the language extras the project needs: + +```bash +python -m pip install docforge +python -m pip install 'docforge[javascript]' +python -m pip install 'docforge[typescript]' +python -m pip install 'docforge[cpp]' +``` + +The base wheel contains the Python reference adapter and no Tree-sitter distribution. JavaScript, +TypeScript, and C++ require their matching optional extras. `docforge[languages]` installs all +three optional frontend groups. + +### Configure a reference source project + +Reference adapters are a narrow alternative to the generic documentation descriptor. Create +`.docforge/reference-adapter.toml`: + +```toml reference-adapter +schema_version = 1 +project_id = "my-python-project" +title = "My Python Project" +language = "python" +source_roots = ["src"] +``` + +Start the fixed read-only server: + +```bash +python -I -m docforge.reference_mcp \ + --project-root /absolute/path/MyProject \ + --capability-mode read +``` + +For C++, set `language = "cpp"` and add a project-relative +`compilation_database = "compile_commands.json"`. The database is inert bounded inventory; the +reference adapter does not execute its commands or compiler. + +The reference integrations publish syntax and local static relationships only. They do not claim +resolved calls, types, inheritance, macro behavior, compiler include semantics, runtime behavior, +or semantic ownership. See [reference adapters](REFERENCE_ADAPTERS.md) for exact evidence and +limitations. + ### Assess and onboard an unconfigured project Run a read-only assessment before writing configuration: @@ -478,6 +523,10 @@ Every command emits deterministic JSON: docforge --project-root /absolute/path/MyProject ``` +The sections below group common workflows. The implementation-derived list of all 28 current +commands, exact invocations, 36 generic MCP tools, arguments, and input-schema hashes is the +[generated command reference](COMMAND_REFERENCE.md). + ### Project and index commands ```text @@ -590,6 +639,31 @@ declares the named writer, and application requires the same writer/applier iden `--no-ast` to preserve the no-AST binding. Generic CLI generation refuses project-owned adapters because it cannot safely reconstruct their composition. +A custom adapter owner supplies the already constructed project and immutable launcher through the +Python API: + +```python +from docforge.adapter_launcher import AdapterLauncherV1 +from docforge.client_config import generate_adapter_client_configuration + +launcher = AdapterLauncherV1.for_project( + project, + module="my_project_docforge", +) +fragment = generate_adapter_client_configuration( + project, + launcher, + "codex", + capability_mode="read", +) +``` + +The top-level module must be installed for the exact isolated Python environment and resolve to a +regular file inside the project root. The fixed `docforge.reference_mcp` module is the only trusted +dotted exception. Generation probes resolution without importing the custom module, binds current +source availability and policy, and emits no arbitrary command, arguments, working directory, or +environment. + Select projection behavior independently: ```bash @@ -684,6 +758,10 @@ docforge-mcp \ Omit `--proposal-writer` when the MCP client should not create or append proposals. +For `.docforge/reference-adapter.toml`, use the fixed `docforge.reference_mcp` command shown in +[setup](#configure-a-reference-source-project). It exposes exactly the 21 read tools and never +registers proposal or application tools. + Select the session's declared surface explicitly when useful: ```bash @@ -986,6 +1064,28 @@ changes outside this process boundary. ## Troubleshooting +### `optional_dependency_missing` + +Install the exact extra named in the error into the same Python environment that starts DocForge: + +```bash +python -m pip install 'docforge[javascript]' +python -m pip install 'docforge[typescript]' +python -m pip install 'docforge[cpp]' +``` + +Do not install every frontend merely to suppress the check. A missing optional parser is a closed, +actionable capability error and does not affect base generic or Python reference operation. + +### `adapter_launcher_unavailable` or `invalid_adapter_launcher` + +Use one installed top-level Python module whose resolved regular-file origin is inside the project +root, or use the fixed `docforge.reference_mcp` binding. Arbitrary dotted modules, packages, +stdlib modules, missing modules, commands, argument strings, working directories, and environment +injection are rejected. Test the exact generated fragment rather than editing its command by hand. + +See [agent integration](AGENT_INTEGRATION.md) and the [security model](SECURITY.md). + ### `adapter_restart_required` The project-local adapter code, its declared descriptor, or another implementation file changed @@ -1128,8 +1228,11 @@ make gate Use `make benchmark` for the historical Milestone 0 baseline, `make benchmark-m1` for the counter-gated warm-operation benchmark, `make benchmark-m2` for agent workflow gates, and `make benchmark-m3-full` for the ten-sample 1,000-node projection, worker, fragment, status, -equivalence, response-size, and memory gates. `make accessibility` runs the generated manual, -portable graph, and live viewer axe and keyboard flows. +equivalence, response-size, and memory gates. `make benchmark-m4-full` runs the 1,002-node adapter +and recovery benchmark. `make adoption-m4` performs the offline fresh-wheel proof. +`make command-reference-check` rejects command-reference drift, and `make docs-check` validates +the maintained documentation graph. `make accessibility` runs the generated manual, portable +graph, and live viewer axe and keyboard flows. Project-specific vocabulary, extraction rules, and serialization belong in the project adapter. Generic core behavior must remain deterministic, project-bound, and recoverable. diff --git a/pyproject.toml b/pyproject.toml index 6024c3e..e3c16be 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -13,9 +13,26 @@ authors = [{ name = "Worldforge contributors" }] dependencies = [ "markdown-it-py>=4.2,<5", "mcp>=1.28,<2", +] + +[project.optional-dependencies] +javascript = [ + "tree-sitter>=0.25,<0.26", + "tree-sitter-javascript>=0.25,<0.26", +] +typescript = [ + "tree-sitter>=0.25,<0.26", + "tree-sitter-typescript>=0.23,<0.24", +] +cpp = [ + "tree-sitter>=0.25,<0.26", + "tree-sitter-cpp>=0.23,<0.24", +] +languages = [ "tree-sitter>=0.25,<0.26", "tree-sitter-cpp>=0.23,<0.24", "tree-sitter-javascript>=0.25,<0.26", + "tree-sitter-typescript>=0.23,<0.24", ] [dependency-groups] @@ -23,6 +40,10 @@ dev = [ "jsonschema>=4.25,<5", "pytest>=9.1,<10", "ruff>=0.15,<1", + "tree-sitter>=0.25,<0.26", + "tree-sitter-cpp>=0.23,<0.24", + "tree-sitter-javascript>=0.25,<0.26", + "tree-sitter-typescript>=0.23,<0.24", ] [project.scripts] diff --git a/schemas/adapter-client-configuration.schema.json b/schemas/adapter-client-configuration.schema.json new file mode 100644 index 0000000..e316c93 --- /dev/null +++ b/schemas/adapter-client-configuration.schema.json @@ -0,0 +1,358 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://docforge.local/schema/adapter-client-configuration-v1.json", + "title": "DocForge project-owned adapter client configuration", + "$defs": { + "sha256": { + "type": "string", + "pattern": "^[0-9a-f]{64}$" + }, + "project": { + "type": "object", + "required": [ + "project_id", + "project_root", + "project_root_fingerprint", + "adapter", + "descriptor_hash" + ], + "properties": { + "project_id": { + "type": "string", + "pattern": "^[a-z0-9][a-z0-9._-]{1,127}$" + }, + "project_root": { + "type": "string", + "minLength": 1, + "maxLength": 4096 + }, + "project_root_fingerprint": { + "type": "string", + "pattern": "^[0-9a-f]{16}$" + }, + "adapter": { + "type": "string", + "pattern": "^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}@[A-Za-z0-9][A-Za-z0-9_.+-]{0,63}$" + }, + "descriptor_hash": { "$ref": "#/$defs/sha256" } + }, + "additionalProperties": false + }, + "source_availability": { + "type": "object", + "required": [ + "schema_version", + "status", + "method", + "revision", + "source_hash" + ], + "properties": { + "schema_version": { "const": 1 }, + "status": { "const": "available" }, + "method": { + "enum": ["incremental-state", "complete-projection"] + }, + "revision": { + "type": "string", + "minLength": 1, + "maxLength": 4096 + }, + "source_hash": { "$ref": "#/$defs/sha256" } + }, + "additionalProperties": false + }, + "binding": { + "type": "object", + "required": [ + "transport", + "capability_mode", + "adapter_policy", + "render_policy", + "command", + "args", + "environment", + "timeouts", + "launcher_hash" + ], + "properties": { + "transport": { "const": "stdio" }, + "capability_mode": { + "enum": ["read", "proposal", "application"] + }, + "adapter_policy": { + "$ref": "https://docforge.local/schema/client-configuration-v1.json#/$defs/adapter_policy" + }, + "render_policy": { + "type": "object", + "required": ["manual", "graph", "live_viewer"], + "properties": { + "manual": { "enum": ["auto", "explicit", "disabled"] }, + "graph": { "const": "disabled" }, + "live_viewer": { "const": "on-demand" } + }, + "additionalProperties": false + }, + "command": { + "type": "string", + "minLength": 1, + "maxLength": 4096 + }, + "args": { + "type": "array", + "minItems": 7, + "maxItems": 64, + "prefixItems": [ + { "const": "-I" }, + { "const": "-m" }, + { + "type": "string", + "minLength": 1, + "maxLength": 255, + "pattern": "^[A-Za-z_][A-Za-z0-9_]*(\\.[A-Za-z_][A-Za-z0-9_]*){0,31}$", + "not": { "const": "docforge.mcp_server" } + }, + { "const": "--project-root" }, + { + "type": "string", + "minLength": 1, + "maxLength": 4096 + }, + { "const": "--capability-mode" }, + { "enum": ["read", "proposal", "application"] } + ], + "items": { + "type": "string", + "minLength": 1, + "maxLength": 4096 + } + }, + "environment": { + "type": "object", + "maxProperties": 0 + }, + "timeouts": { + "type": "object", + "required": ["startup_seconds", "tool_seconds"], + "properties": { + "startup_seconds": { + "type": "integer", + "minimum": 1, + "maximum": 3600 + }, + "tool_seconds": { + "type": "integer", + "minimum": 1, + "maximum": 86400 + } + }, + "additionalProperties": false + }, + "launcher_hash": { "$ref": "#/$defs/sha256" } + }, + "additionalProperties": false + }, + "projection_availability": { + "type": "object", + "required": [ + "manual_configured", + "portable_graph_configured", + "application_enabled", + "live_viewer_available" + ], + "properties": { + "manual_configured": { "type": "boolean" }, + "portable_graph_configured": { "type": "boolean" }, + "application_enabled": { "type": "boolean" }, + "live_viewer_available": { "const": true } + }, + "additionalProperties": false + } + }, + "type": "object", + "required": [ + "status", + "schema_version", + "operation", + "action", + "client", + "server_name", + "project", + "launcher", + "launcher_hash", + "source_availability", + "source_availability_hash", + "binding", + "effective_policy", + "projection_policy", + "projection_policy_hash", + "projection_availability", + "artifact", + "configuration_hash", + "warnings" + ], + "properties": { + "status": { "const": "ok" }, + "schema_version": { "const": 1 }, + "operation": { "const": "adapter_client.configure" }, + "action": { "enum": ["preview", "write"] }, + "client": { "enum": ["codex", "claude", "openclaw"] }, + "server_name": { + "type": "string", + "pattern": "^[a-z0-9][a-z0-9_-]{0,63}$" + }, + "project": { "$ref": "#/$defs/project" }, + "launcher": { + "$ref": "https://docforge.local/schema/adapter-launcher-v1.json" + }, + "launcher_hash": { "$ref": "#/$defs/sha256" }, + "source_availability": { + "$ref": "#/$defs/source_availability" + }, + "source_availability_hash": { "$ref": "#/$defs/sha256" }, + "binding": { "$ref": "#/$defs/binding" }, + "effective_policy": { + "$ref": "https://docforge.local/schema/client-configuration-v1.json#/$defs/effective_policy" + }, + "projection_policy": { + "$ref": "https://docforge.local/schema/client-configuration-v1.json#/$defs/projection_policy" + }, + "projection_policy_hash": { "$ref": "#/$defs/sha256" }, + "projection_availability": { + "$ref": "#/$defs/projection_availability" + }, + "artifact": { + "$ref": "https://docforge.local/schema/client-configuration-v1.json#/properties/artifact" + }, + "configuration_hash": { "$ref": "#/$defs/sha256" }, + "warnings": { + "$ref": "https://docforge.local/schema/client-configuration-v1.json#/properties/warnings" + } + }, + "allOf": [ + { + "if": { + "properties": { "client": { "const": "codex" } }, + "required": ["client"] + }, + "then": { + "properties": { + "artifact": { + "properties": { + "format": { "const": "codex-toml-fragment-v1" } + } + } + } + } + }, + { + "if": { + "properties": { "client": { "const": "claude" } }, + "required": ["client"] + }, + "then": { + "properties": { + "artifact": { + "properties": { + "format": { "const": "claude-json-fragment-v1" } + } + }, + "warnings": { + "contains": { + "properties": { + "code": { "const": "timeout_format_unverified" } + }, + "required": ["code"] + } + } + } + } + }, + { + "if": { + "properties": { "client": { "const": "openclaw" } }, + "required": ["client"] + }, + "then": { + "properties": { + "artifact": { + "properties": { + "format": { "const": "openclaw-json-fragment-v1" } + } + } + } + } + }, + { + "if": { + "properties": { "action": { "const": "preview" } }, + "required": ["action"] + }, + "then": { + "properties": { + "artifact": { + "properties": { + "output_path": { "type": "null" }, + "write_state": { "const": "not_requested" }, + "durability": { "const": "not_applicable" } + } + } + } + }, + "else": { + "properties": { + "artifact": { + "properties": { + "write_state": { "enum": ["created", "unchanged"] } + } + } + } + } + }, + { + "if": { + "properties": { + "binding": { + "properties": { + "adapter_policy": { + "properties": { + "mode": { "const": "preserve-no-ast" } + }, + "required": ["mode"] + } + }, + "required": ["adapter_policy"] + } + }, + "required": ["binding"] + }, + "then": { + "properties": { + "binding": { + "properties": { + "args": { + "contains": { "const": "--no-ast" }, + "minContains": 1, + "maxContains": 1 + } + } + } + } + }, + "else": { + "properties": { + "binding": { + "properties": { + "args": { + "not": { + "contains": { "const": "--no-ast" } + } + } + } + } + } + } + } + ], + "additionalProperties": false +} diff --git a/schemas/adapter-launcher.schema.json b/schemas/adapter-launcher.schema.json new file mode 100644 index 0000000..4428828 --- /dev/null +++ b/schemas/adapter-launcher.schema.json @@ -0,0 +1,46 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://docforge.local/schema/adapter-launcher-v1.json", + "title": "DocForge project-owned adapter launcher", + "type": "object", + "required": [ + "schema_version", + "entry_point", + "project_id", + "project_root", + "adapter", + "descriptor_hash", + "module" + ], + "properties": { + "schema_version": { "const": 1 }, + "entry_point": { "const": "python-module" }, + "project_id": { + "type": "string", + "pattern": "^[a-z0-9][a-z0-9._-]{1,127}$" + }, + "project_root": { + "type": "string", + "minLength": 1, + "maxLength": 4096 + }, + "adapter": { + "type": "string", + "pattern": "^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}@[A-Za-z0-9][A-Za-z0-9_.+-]{0,63}$" + }, + "descriptor_hash": { + "type": "string", + "pattern": "^[0-9a-f]{64}$" + }, + "module": { + "type": "string", + "minLength": 1, + "maxLength": 255, + "anyOf": [ + { "pattern": "^[A-Za-z_][A-Za-z0-9_]*$" }, + { "const": "docforge.reference_mcp" } + ] + } + }, + "additionalProperties": false +} diff --git a/schemas/reference-adapter.schema.json b/schemas/reference-adapter.schema.json new file mode 100644 index 0000000..bc61277 --- /dev/null +++ b/schemas/reference-adapter.schema.json @@ -0,0 +1,78 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://andraxion.net/docforge/schemas/reference-adapter.schema.json", + "title": "DocForge Reference Adapter Configuration", + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version", + "project_id", + "title", + "language", + "source_roots" + ], + "properties": { + "schema_version": { + "const": 1 + }, + "project_id": { + "type": "string", + "pattern": "^[a-z0-9][a-z0-9._-]{1,127}$" + }, + "title": { + "type": "string", + "minLength": 1, + "maxLength": 256 + }, + "language": { + "enum": [ + "python", + "javascript", + "typescript", + "cpp" + ] + }, + "source_roots": { + "type": "array", + "minItems": 1, + "maxItems": 64, + "uniqueItems": true, + "items": { + "type": "string", + "minLength": 1, + "pattern": "^(?!/)(?!.*(?:^|/)\\.\\.(?:/|$)).+$" + } + }, + "compilation_database": { + "type": "string", + "minLength": 1, + "pattern": "^(?!/)(?!.*(?:^|/)\\.\\.(?:/|$)).+$" + } + }, + "allOf": [ + { + "if": { + "properties": { + "language": { + "const": "cpp" + } + }, + "required": [ + "language" + ] + }, + "then": { + "required": [ + "compilation_database" + ] + }, + "else": { + "not": { + "required": [ + "compilation_database" + ] + } + } + } + ] +} diff --git a/src/docforge/adapter_contract.py b/src/docforge/adapter_contract.py index 7dd1445..c0829bb 100644 --- a/src/docforge/adapter_contract.py +++ b/src/docforge/adapter_contract.py @@ -21,6 +21,8 @@ from .adapter_validation import ( from .config_validation import ID_PATTERN from .errors import DocForgeError from .incremental import ( + MAX_EXTRACTION_CACHE_BYTES, + MAX_EXTRACTION_CACHE_SOURCES, CachedSource, ExtractionCache, affected_sources, @@ -40,6 +42,10 @@ from .models import ( ) from .telemetry import increment, stage +MAX_ADAPTER_EDGE_MULTIPLIER = 32 +MAX_ADAPTER_LOGIC_NODE_MULTIPLIER = 32 +MAX_ADAPTER_LOGIC_EDGE_MULTIPLIER = 64 + @dataclass(frozen=True) class AdapterNode: @@ -147,6 +153,13 @@ class AdapterAssembly: logic: tuple[LogicProjection, ...] = () +@runtime_checkable +class CompleteAdapterAssemblyLoader(Protocol): + """Load one cache-independent complete graph and Logic equivalence oracle.""" + + def load_complete_assembly(self) -> AdapterAssembly: ... + + class AdapterLoader(Protocol): """Load one current, deterministic, project-confined adapter projection.""" @@ -201,6 +214,14 @@ def _extract_adapter_source( return loader.extract_source(source) +def _load_complete_adapter_assembly( + loader: CompleteAdapterAssemblyLoader, +) -> AdapterAssembly: + increment("adapter_projection_loads") + with stage("adapter.projection"): + return loader.load_complete_assembly() + + @dataclass(frozen=True) class AdapterImplementation: """One confined implementation boundary that must remain stable for a process.""" @@ -232,6 +253,38 @@ class _ImplementationSnapshot: files: tuple[tuple[str, str], ...] +@dataclass(frozen=True) +class AdapterConformanceReport: + """Stable evidence that one adapter passed its complete and incremental contracts.""" + + schema_version: int + project_id: str + adapter_id: str + adapter_version: str + revision: str + source_hash: str + assembly_hash: str + node_count: int + edge_count: int + logic_projection_count: int + incremental: bool + + def as_dict(self) -> dict[str, object]: + return { + "schema_version": self.schema_version, + "project_id": self.project_id, + "adapter_id": self.adapter_id, + "adapter_version": self.adapter_version, + "revision": self.revision, + "source_hash": self.source_hash, + "assembly_hash": self.assembly_hash, + "node_count": self.node_count, + "edge_count": self.edge_count, + "logic_projection_count": self.logic_projection_count, + "incremental": self.incremental, + } + + class AdapterProject: """Expose a validated adapter projection through the standard index boundary.""" @@ -417,6 +470,9 @@ class AdapterProject: canonical_sources, captured, ) + self._enforce_assembly_limits( + AdapterAssembly(projection=projection, logic=self._last_logic) + ) return ProjectSnapshot( descriptor=self.descriptor, nodes=projection.core_nodes(), @@ -523,43 +579,57 @@ class AdapterProject: return self._last_logic def verify_incremental_equivalence(self) -> dict[str, object]: - """Prove the incremental and full loader contracts produce the same graph.""" + """Prove incremental graph and Logic output matches an independent complete oracle.""" self.validate_runtime() - if self._incremental_loader is None: + loader = self._incremental_loader + if loader is None: raise DocForgeError( "incremental_disabled", "Adapter does not implement incremental extraction" ) - incremental = self._load_incremental() + incremental = self._load_incremental_assembly() full = _load_adapter_projection(self.loader) validate_projection(full) - fields = { - "project_id": incremental.project_id == full.project_id, - "adapter_id": incremental.adapter_id == full.adapter_id, - "adapter_version": incremental.adapter_version == full.adapter_version, - "revision": incremental.revision == full.revision, - "source_hash": incremental.source_hash == full.source_hash, - "nodes": incremental.nodes == full.nodes, - "edges": incremental.edges == full.edges, - } + complete = AdapterAssembly(full) + if isinstance(loader, CompleteAdapterAssemblyLoader): + complete = _load_complete_adapter_assembly(loader) + _validate_adapter_assembly(complete) + oracle_fields = _projection_equivalence_fields(complete.projection, full) + oracle_mismatches = [field for field, matches in oracle_fields.items() if not matches] + if oracle_mismatches: + raise DocForgeError( + "complete_oracle_mismatch", + "Complete adapter assembly does not match load_projection()", + fields=oracle_mismatches, + ) + elif incremental.logic: + raise DocForgeError( + "complete_logic_oracle_required", + "Incremental Logic publication requires load_complete_assembly()", + ) + fields = _assembly_equivalence_fields(incremental, complete) mismatches = [field for field, matches in fields.items() if not matches] if mismatches: raise DocForgeError( "incremental_mismatch", - "Incremental extraction does not match a full adapter projection", + "Incremental extraction does not match a complete adapter assembly", fields=mismatches, ) self.validate_runtime() return { "status": "ok", - "project_id": incremental.project_id, - "revision": incremental.revision, - "source_hash": incremental.source_hash, - "node_count": len(incremental.nodes), - "edge_count": len(incremental.edges), + "project_id": incremental.projection.project_id, + "revision": incremental.projection.revision, + "source_hash": incremental.projection.source_hash, + "node_count": len(incremental.projection.nodes), + "edge_count": len(incremental.projection.edges), + "logic_projection_count": len(incremental.logic), } def _load_incremental(self) -> AdapterProjection: + return self._load_incremental_assembly().projection + + def _load_incremental_assembly(self) -> AdapterAssembly: loader = self._incremental_loader if loader is None: raise DocForgeError("incremental_disabled", "Incremental adapter is not configured") @@ -567,11 +637,14 @@ class AdapterProject: validate_manifest(manifest) if manifest.identity() != self._identity: raise DocForgeError("adapter_changed", "Adapter identity changed during the operation") + self._enforce_manifest_limits(manifest) cache = load_extraction_cache( self._cache_path, project_id=manifest.project_id, adapter_id=manifest.adapter_id, adapter_version=manifest.adapter_version, + max_bytes=MAX_EXTRACTION_CACHE_BYTES, + max_sources=MAX_EXTRACTION_CACHE_SOURCES, ) cached = {source.source_id: source for source in cache.sources} if cache else {} current = {source.source_id: source for source in manifest.sources} @@ -658,6 +731,7 @@ class AdapterProject: key=lambda projection: projection.owner_node_id, ) ) + assembly = AdapterAssembly(projection=projection, logic=logic_projections) identity = ( projection.project_id, projection.adapter_id, @@ -674,21 +748,8 @@ class AdapterProject: "invalid_adapter", "Incremental assembly changed the manifest-bound project identity", ) - validate_projection(projection) - owners = [projection.owner_node_id for projection in logic_projections] - if len(owners) != len(set(owners)): - raise DocForgeError( - "invalid_adapter", "A primary graph node may own only one logic projection" - ) - node_ids = {item.node.node_id for item in projection.nodes} - for logic_projection in logic_projections: - validate_logic_projection(logic_projection) - if logic_projection.owner_node_id not in node_ids: - raise DocForgeError( - "invalid_adapter", - "A Logic projection owner must exist in the assembled primary graph", - owner_node_id=logic_projection.owner_node_id, - ) + _validate_adapter_assembly(assembly) + self._enforce_assembly_limits(assembly) stable = loader.load_manifest() validate_manifest(stable) if stable != manifest: @@ -704,6 +765,8 @@ class AdapterProject: adapter_version=manifest.adapter_version, sources=tuple(cache_records), ), + max_bytes=MAX_EXTRACTION_CACHE_BYTES, + max_sources=MAX_EXTRACTION_CACHE_SOURCES, ) self._last_logic = logic_projections self._last_build_report = { @@ -716,7 +779,41 @@ class AdapterProject: "cache_hit_ids": hits, "reparsed_source_ids": reparsed, } - return projection + return assembly + + def _enforce_manifest_limits(self, manifest: AdapterManifest) -> None: + if len(manifest.sources) > MAX_EXTRACTION_CACHE_SOURCES: + raise DocForgeError( + "adapter_limit", + "Adapter manifest exceeds the bounded source limit", + maximum=MAX_EXTRACTION_CACHE_SOURCES, + actual=len(manifest.sources), + ) + + def _enforce_assembly_limits(self, assembly: AdapterAssembly) -> None: + maximum_nodes = self.descriptor.limits.max_nodes + maximum_edges = maximum_nodes * MAX_ADAPTER_EDGE_MULTIPLIER + maximum_logic_nodes = maximum_nodes * MAX_ADAPTER_LOGIC_NODE_MULTIPLIER + maximum_logic_edges = maximum_nodes * MAX_ADAPTER_LOGIC_EDGE_MULTIPLIER + actual_nodes = len(assembly.projection.nodes) + actual_edges = len(assembly.projection.edges) + actual_logic_nodes = sum(len(projection.nodes) for projection in assembly.logic) + actual_logic_edges = sum(len(projection.edges) for projection in assembly.logic) + limits = ( + ("nodes", actual_nodes, maximum_nodes), + ("edges", actual_edges, maximum_edges), + ("logic_nodes", actual_logic_nodes, maximum_logic_nodes), + ("logic_edges", actual_logic_edges, maximum_logic_edges), + ) + for label, actual, maximum in limits: + if actual > maximum: + raise DocForgeError( + "adapter_limit", + f"Adapter assembly exceeds the configured {label.replace('_', ' ')} limit", + kind=label, + maximum=maximum, + actual=actual, + ) def canonical_source_paths(self) -> tuple[Path, ...]: """Adapters validate their own source sets before producing a projection.""" @@ -1032,6 +1129,141 @@ class AdapterProject: outputs.add(output) +def _validate_adapter_assembly(assembly: AdapterAssembly) -> None: + validate_projection(assembly.projection) + ordered_logic = tuple(sorted(assembly.logic, key=lambda projection: projection.owner_node_id)) + if assembly.logic != ordered_logic: + raise DocForgeError( + "invalid_adapter", + "Complete Logic projections must be deterministically ordered", + ) + owners = [projection.owner_node_id for projection in assembly.logic] + if len(owners) != len(set(owners)): + raise DocForgeError( + "invalid_adapter", "A primary graph node may own only one logic projection" + ) + node_ids = {item.node.node_id for item in assembly.projection.nodes} + for logic_projection in assembly.logic: + validate_logic_projection(logic_projection) + if logic_projection.owner_node_id not in node_ids: + raise DocForgeError( + "invalid_adapter", + "A Logic projection owner must exist in the assembled primary graph", + owner_node_id=logic_projection.owner_node_id, + ) + + +def _projection_equivalence_fields( + candidate: AdapterProjection, + reference: AdapterProjection, +) -> dict[str, bool]: + return { + "project_id": candidate.project_id == reference.project_id, + "title": candidate.title == reference.title, + "adapter_id": candidate.adapter_id == reference.adapter_id, + "adapter_version": candidate.adapter_version == reference.adapter_version, + "root": candidate.root == reference.root, + "revision": candidate.revision == reference.revision, + "source_hash": candidate.source_hash == reference.source_hash, + "nodes": candidate.nodes == reference.nodes, + "edges": candidate.edges == reference.edges, + } + + +def _assembly_equivalence_fields( + candidate: AdapterAssembly, + reference: AdapterAssembly, +) -> dict[str, bool]: + return { + **_projection_equivalence_fields(candidate.projection, reference.projection), + "logic": candidate.logic == reference.logic, + } + + +def _assembly_hash(assembly: AdapterAssembly) -> str: + projection = assembly.projection + payload = { + "projection": { + "project_id": projection.project_id, + "title": projection.title, + "adapter_id": projection.adapter_id, + "adapter_version": projection.adapter_version, + "root": str(projection.root), + "revision": projection.revision, + "source_hash": projection.source_hash, + "nodes": [item.as_dict() for item in projection.nodes], + "edges": [item.as_dict() for item in projection.edges], + }, + "logic": [logic_projection.as_dict() for logic_projection in assembly.logic], + } + return hashlib.sha256( + json.dumps(payload, sort_keys=True, separators=(",", ":")).encode() + ).hexdigest() + + +def _complete_reference_assembly(loader: AdapterLoader) -> AdapterAssembly: + if isinstance(loader, CompleteAdapterAssemblyLoader): + assembly = _load_complete_adapter_assembly(loader) + else: + assembly = AdapterAssembly(_load_adapter_projection(loader)) + _validate_adapter_assembly(assembly) + return assembly + + +def verify_adapter_conformance( + loader: AdapterLoader, + *, + cache_root: Path, + settings: AdapterProjectSettings | None = None, +) -> AdapterConformanceReport: + """Prove deterministic complete output and, when supported, incremental parity.""" + + project = AdapterProject(loader, cache_root=cache_root, settings=settings) + project.validate_runtime() + first = _complete_reference_assembly(loader) + second = _complete_reference_assembly(loader) + deterministic_fields = _assembly_equivalence_fields(first, second) + nondeterministic = [field for field, matches in deterministic_fields.items() if not matches] + if nondeterministic: + raise DocForgeError( + "nondeterministic_adapter", + "Repeated complete adapter assemblies do not match", + fields=nondeterministic, + ) + + if isinstance(loader, CompleteAdapterAssemblyLoader): + graph_oracle = _load_adapter_projection(loader) + validate_projection(graph_oracle) + oracle_fields = _projection_equivalence_fields(first.projection, graph_oracle) + oracle_mismatches = [field for field, matches in oracle_fields.items() if not matches] + if oracle_mismatches: + raise DocForgeError( + "complete_oracle_mismatch", + "Complete adapter assembly does not match load_projection()", + fields=oracle_mismatches, + ) + + incremental = isinstance(loader, IncrementalAdapterLoader) + if incremental: + project.verify_incremental_equivalence() + + project.validate_runtime() + projection = first.projection + return AdapterConformanceReport( + schema_version=1, + project_id=projection.project_id, + adapter_id=projection.adapter_id, + adapter_version=projection.adapter_version, + revision=projection.revision, + source_hash=projection.source_hash, + assembly_hash=_assembly_hash(first), + node_count=len(projection.nodes), + edge_count=len(projection.edges), + logic_projection_count=len(first.logic), + incremental=incremental, + ) + + @dataclass(frozen=True) class ShadowArtifact: """One named deterministic byte artifact used by a shadow comparison.""" diff --git a/src/docforge/adapter_launcher.py b/src/docforge/adapter_launcher.py new file mode 100644 index 0000000..f86be68 --- /dev/null +++ b/src/docforge/adapter_launcher.py @@ -0,0 +1,392 @@ +"""Strict, data-only launch contracts for project-owned adapter MCP modules.""" + +from __future__ import annotations + +import json +import re +import stat +import subprocess +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Literal, cast + +from .changeset_contract import document_hash +from .config_validation import ID_PATTERN +from .errors import DocForgeError +from .models import IncrementalStateProject, ProjectService, RuntimeValidatedProject + +ADAPTER_LAUNCHER_SCHEMA_VERSION = 1 +_PYTHON_MODULE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*){0,31}") +_TOP_LEVEL_PYTHON_MODULE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*") +_ADAPTER_IDENTITY = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}@[A-Za-z0-9][A-Za-z0-9_.+-]{0,63}") +_SHA256 = re.compile(r"[0-9a-f]{64}") +_TRUSTED_DOTTED_MODULE = "docforge.reference_mcp" +_MODULE_PROBE_TIMEOUT_SECONDS = 5 +_MAX_MODULE_PROBE_BYTES = 8_192 +_MODULE_PROBE = ( + "import importlib.util,json,sys;" + "spec=importlib.util.find_spec(sys.argv[1]);" + "result=None if spec is None else {" + "'has_loader':spec.loader is not None," + "'is_package':spec.submodule_search_locations is not None," + "'origin':spec.origin};" + "print(json.dumps(result,sort_keys=True,separators=(',',':')))" +) +_run_module_probe = subprocess.run +__all__ = [ + "ADAPTER_LAUNCHER_SCHEMA_VERSION", + "AdapterLauncherV1", + "AdapterSourceAvailabilityV1", + "adapter_source_availability", + "validate_adapter_launcher", +] + + +@dataclass(frozen=True) +class AdapterLauncherV1: + """One immutable, project-bound Python module launch declaration. + + The contract intentionally has no command, shell string, working directory, + environment, arbitrary arguments, discovery rule, or callable selector. + """ + + schema_version: int + entry_point: Literal["python-module"] + project_id: str + project_root: Path + adapter: str + descriptor_hash: str + module: str + + def __post_init__(self) -> None: + _validate_launcher_fields(self) + + @classmethod + def for_project( + cls, + project: ProjectService, + *, + module: str, + ) -> AdapterLauncherV1: + """Bind a structural module entry point to one already constructed adapter project.""" + + descriptor = project.descriptor + if descriptor.adapter == "generic": + raise DocForgeError( + "adapter_launcher_unavailable", + "Project-owned launcher configuration requires a custom adapter", + ) + launcher = cls( + schema_version=ADAPTER_LAUNCHER_SCHEMA_VERSION, + entry_point="python-module", + project_id=descriptor.project_id, + project_root=descriptor.root, + adapter=descriptor.adapter, + descriptor_hash=descriptor.descriptor_hash, + module=module, + ) + validate_adapter_launcher(project, launcher) + return launcher + + def as_dict(self) -> dict[str, object]: + return { + "schema_version": self.schema_version, + "entry_point": self.entry_point, + "project_id": self.project_id, + "project_root": str(self.project_root), + "adapter": self.adapter, + "descriptor_hash": self.descriptor_hash, + "module": self.module, + } + + @property + def launcher_hash(self) -> str: + return document_hash(self.as_dict()) + + +@dataclass(frozen=True) +class AdapterSourceAvailabilityV1: + """One bounded source identity observed without executing the launcher module.""" + + schema_version: int + status: Literal["available"] + method: Literal["incremental-state", "complete-projection"] + revision: str + source_hash: str + + def __post_init__(self) -> None: + revision = _runtime_value(self.revision) + if ( + self.schema_version != 1 + or self.status != "available" + or self.method not in {"incremental-state", "complete-projection"} + or not isinstance(revision, str) + or not revision + or _SHA256.fullmatch(self.source_hash) is None + ): + raise DocForgeError( + "invalid_adapter_launcher", + "Adapter source availability evidence is invalid", + ) + + def as_dict(self) -> dict[str, object]: + return { + "schema_version": self.schema_version, + "status": self.status, + "method": self.method, + "revision": self.revision, + "source_hash": self.source_hash, + } + + @property + def availability_hash(self) -> str: + return document_hash(self.as_dict()) + + +def validate_adapter_launcher( + project: ProjectService, + launcher: AdapterLauncherV1, +) -> None: + """Require one launcher to match the exact current adapter project binding.""" + + _validate_launcher_fields(launcher) + descriptor = project.descriptor + _validate_project_root(descriptor.root) + if descriptor.adapter == "generic": + raise DocForgeError( + "adapter_launcher_unavailable", + "Project-owned launcher configuration requires a custom adapter", + ) + if ( + launcher.project_id != descriptor.project_id + or launcher.project_root != descriptor.root + or launcher.adapter != descriptor.adapter + or launcher.descriptor_hash != descriptor.descriptor_hash + ): + raise DocForgeError( + "adapter_launcher_mismatch", + "Adapter launcher does not match the selected project binding", + project_id=descriptor.project_id, + adapter=descriptor.adapter, + ) + if isinstance(project, RuntimeValidatedProject): + project.validate_runtime() + + +def adapter_source_availability( + project: ProjectService, + launcher: AdapterLauncherV1, +) -> AdapterSourceAvailabilityV1: + """Capture current source identity without importing or executing the launcher module.""" + + validate_adapter_launcher(project, launcher) + state = project.incremental_state() if isinstance(project, IncrementalStateProject) else None + if state is not None: + method: Literal["incremental-state", "complete-projection"] = "incremental-state" + revision = state.revision + source_hash = state.source_hash + else: + snapshot = project.load() + descriptor = snapshot.descriptor + if ( + descriptor.project_id != launcher.project_id + or descriptor.root != launcher.project_root + or descriptor.adapter != launcher.adapter + or descriptor.descriptor_hash != launcher.descriptor_hash + ): + raise DocForgeError( + "adapter_launcher_mismatch", + "Loaded adapter projection drifted from its launcher binding", + ) + method = "complete-projection" + revision = snapshot.revision + source_hash = snapshot.source_hash + validate_adapter_launcher(project, launcher) + return AdapterSourceAvailabilityV1( + schema_version=1, + status="available", + method=method, + revision=revision, + source_hash=source_hash, + ) + + +def _validate_launcher_fields(launcher: AdapterLauncherV1) -> None: + if type(launcher.schema_version) is not int or launcher.schema_version != 1: + raise DocForgeError( + "invalid_adapter_launcher", + "Adapter launcher schema version is unsupported", + supported=1, + ) + if launcher.entry_point != "python-module": + raise DocForgeError( + "invalid_adapter_launcher", + "Adapter launcher entry point must be one isolated Python module", + ) + project_id = _runtime_value(launcher.project_id) + project_root = _runtime_value(launcher.project_root) + adapter = _runtime_value(launcher.adapter) + descriptor_hash = _runtime_value(launcher.descriptor_hash) + module = _runtime_value(launcher.module) + if not isinstance(project_id, str) or ID_PATTERN.fullmatch(project_id) is None: + raise DocForgeError("invalid_adapter_launcher", "Adapter launcher project ID is invalid") + if not isinstance(project_root, Path): + raise DocForgeError("invalid_adapter_launcher", "Adapter launcher project root is invalid") + _validate_project_root(project_root) + if not isinstance(adapter, str) or _ADAPTER_IDENTITY.fullmatch(adapter) is None: + raise DocForgeError("invalid_adapter_launcher", "Adapter launcher identity is invalid") + if not isinstance(descriptor_hash, str) or _SHA256.fullmatch(descriptor_hash) is None: + raise DocForgeError("invalid_adapter_launcher", "Adapter descriptor hash is invalid") + if ( + not isinstance(module, str) + or len(module) > 255 + or _PYTHON_MODULE.fullmatch(module) is None + or module == "docforge.mcp_server" + or (module != _TRUSTED_DOTTED_MODULE and _TOP_LEVEL_PYTHON_MODULE.fullmatch(module) is None) + ): + raise DocForgeError( + "invalid_adapter_launcher", + ( + "Adapter launcher module must be one installed project-owned top-level " + "Python module or the fixed DocForge reference binding" + ), + ) + _validate_isolated_module(project_root, module) + + +def _validate_isolated_module(project_root: Path, module: str) -> None: + """Prove ``python -I -m`` can resolve one module without importing project code.""" + + executable = Path(sys.executable) + try: + executable_status = executable.stat() + except OSError as error: + raise DocForgeError( + "adapter_launcher_unavailable", + "The isolated Python executable is unavailable", + ) from error + if not stat.S_ISREG(executable_status.st_mode): + raise DocForgeError( + "adapter_launcher_unavailable", + "The isolated Python executable is not a regular file", + ) + try: + completed = _run_module_probe( + [str(executable), "-I", "-c", _MODULE_PROBE, module], + capture_output=True, + text=True, + timeout=_MODULE_PROBE_TIMEOUT_SECONDS, + check=False, + ) + except (OSError, subprocess.TimeoutExpired) as error: + raise DocForgeError( + "adapter_launcher_unavailable", + "The isolated adapter launcher probe could not complete", + module=module, + ) from error + if ( + completed.returncode != 0 + or len(completed.stdout.encode("utf-8")) > _MAX_MODULE_PROBE_BYTES + or len(completed.stderr.encode("utf-8")) > _MAX_MODULE_PROBE_BYTES + ): + raise DocForgeError( + "adapter_launcher_unavailable", + "The isolated adapter launcher module could not be resolved", + module=module, + ) + try: + result: object = json.loads(completed.stdout) + except (UnicodeError, json.JSONDecodeError) as error: + raise DocForgeError( + "adapter_launcher_unavailable", + "The isolated adapter launcher probe returned invalid evidence", + module=module, + ) from error + if result is None: + raise DocForgeError( + "adapter_launcher_unavailable", + "The isolated adapter launcher module is not installed", + module=module, + ) + if not isinstance(result, dict): + raise DocForgeError( + "adapter_launcher_unavailable", + "The isolated adapter launcher probe returned invalid evidence", + module=module, + ) + evidence = cast(dict[object, object], result) + if ( + set(evidence) != {"has_loader", "is_package", "origin"} + or evidence["has_loader"] is not True + or evidence["is_package"] is not False + or not isinstance(evidence["origin"], str) + ): + raise DocForgeError( + "invalid_adapter_launcher", + "The isolated adapter launcher must resolve to one executable module file", + module=module, + ) + origin = Path(evidence["origin"]) + try: + origin_status = origin.lstat() + resolved_origin = origin.resolve(strict=True) + except OSError as error: + raise DocForgeError( + "adapter_launcher_unavailable", + "The isolated adapter launcher module origin is unavailable", + module=module, + ) from error + if ( + not origin.is_absolute() + or stat.S_ISLNK(origin_status.st_mode) + or not stat.S_ISREG(origin_status.st_mode) + or resolved_origin != origin + or origin.suffix != ".py" + ): + raise DocForgeError( + "invalid_adapter_launcher", + "The isolated adapter launcher must be one canonical Python module file", + module=module, + ) + if module == _TRUSTED_DOTTED_MODULE: + expected = Path(__file__).with_name("reference_mcp.py").resolve(strict=True) + if origin != expected: + raise DocForgeError( + "adapter_launcher_mismatch", + "The fixed reference launcher resolved outside this DocForge installation", + module=module, + ) + elif not origin.is_relative_to(project_root): + raise DocForgeError( + "invalid_adapter_launcher", + "The installed adapter launcher module is not owned by the selected project", + module=module, + ) + + +def _validate_project_root(root: Path) -> None: + try: + status = root.lstat() + resolved = root.resolve(strict=True) + except OSError as error: + raise DocForgeError( + "invalid_adapter_launcher", + "Adapter launcher project root is unavailable", + ) from error + if ( + not root.is_absolute() + or stat.S_ISLNK(status.st_mode) + or not stat.S_ISDIR(status.st_mode) + or resolved != root + ): + raise DocForgeError( + "invalid_adapter_launcher", + "Adapter launcher project root must be one canonical real directory", + ) + + +def _runtime_value(value: object) -> object: + """Keep runtime validation explicit even when static callers are typed.""" + + return value diff --git a/src/docforge/adapter_sdk.py b/src/docforge/adapter_sdk.py new file mode 100644 index 0000000..888f7f2 --- /dev/null +++ b/src/docforge/adapter_sdk.py @@ -0,0 +1,69 @@ +"""Public typed surface for project adapter implementations and conformance checks.""" + +from .adapter_contract import ( + AdapterAssembly, + AdapterConformanceReport, + AdapterEdge, + AdapterImplementation, + AdapterLoader, + AdapterManifest, + AdapterNode, + AdapterProject, + AdapterProjection, + AdapterProjectSettings, + AdapterSource, + AdapterSourceProjection, + CompleteAdapterAssemblyLoader, + IncrementalAdapterAssembler, + IncrementalAdapterLoader, + verify_adapter_conformance, +) +from .adapter_validation import ( + validate_logic_projection, + validate_manifest, + validate_projection, + validate_source_projection, +) +from .models import ( + Edge, + Limits, + LogicEdge, + LogicNode, + LogicProjection, + Node, + ProposalWriter, + RenderConfig, + RenderView, +) + +__all__ = [ + "AdapterAssembly", + "AdapterConformanceReport", + "AdapterEdge", + "AdapterImplementation", + "AdapterLoader", + "AdapterManifest", + "AdapterNode", + "AdapterProject", + "AdapterProjection", + "AdapterProjectSettings", + "AdapterSource", + "AdapterSourceProjection", + "CompleteAdapterAssemblyLoader", + "Edge", + "IncrementalAdapterAssembler", + "IncrementalAdapterLoader", + "Limits", + "LogicEdge", + "LogicNode", + "LogicProjection", + "Node", + "ProposalWriter", + "RenderConfig", + "RenderView", + "validate_logic_projection", + "validate_manifest", + "validate_projection", + "validate_source_projection", + "verify_adapter_conformance", +] diff --git a/src/docforge/adapters/__init__.py b/src/docforge/adapters/__init__.py new file mode 100644 index 0000000..0ae5694 --- /dev/null +++ b/src/docforge/adapters/__init__.py @@ -0,0 +1,25 @@ +"""Repository-owned reference adapters built on the public adapter SDK.""" + +from .javascript import ( + JAVASCRIPT_ADAPTER_VERSION, + JAVASCRIPT_EXTRACTOR_VERSION, + JavaScriptReferenceAdapter, + JavaScriptUnsupportedFact, +) +from .python import ( + PYTHON_ADAPTER_VERSION, + PYTHON_EXTRACTOR_VERSION, + PythonReferenceAdapter, + PythonUnsupportedFact, +) + +__all__ = [ + "JAVASCRIPT_ADAPTER_VERSION", + "JAVASCRIPT_EXTRACTOR_VERSION", + "PYTHON_ADAPTER_VERSION", + "PYTHON_EXTRACTOR_VERSION", + "JavaScriptReferenceAdapter", + "JavaScriptUnsupportedFact", + "PythonReferenceAdapter", + "PythonUnsupportedFact", +] diff --git a/src/docforge/adapters/cpp.py b/src/docforge/adapters/cpp.py new file mode 100644 index 0000000..11e6425 --- /dev/null +++ b/src/docforge/adapters/cpp.py @@ -0,0 +1,1349 @@ +"""Deterministic, syntax-only C++ reference adapter. + +The adapter accepts one explicit, project-confined ``compile_commands.json`` as +the authoritative translation-unit inventory. The database, compiler command, +arguments, and C++ sources are parsed only as inert data. No compiler, build +tool, project binary, or project code is ever executed. + +Only quoted includes that resolve directly to a real, project-local header are +published as dependencies. Tree-sitter proves lexical declarations and +function control flow; compiler-dependent facts are deliberately omitted. +""" + +from __future__ import annotations + +import hashlib +import importlib +import json +import os +import stat +from collections.abc import Iterable, Sequence +from dataclasses import dataclass, replace +from functools import lru_cache +from pathlib import Path, PurePosixPath +from typing import TYPE_CHECKING, Protocol, cast + +from ..adapter_sdk import ( + AdapterAssembly, + AdapterEdge, + AdapterManifest, + AdapterNode, + AdapterProjection, + AdapterSource, + AdapterSourceProjection, + Edge, +) +from ..adapter_sdk import Node as GraphNode +from ..errors import DocForgeError +from ..treesitter_logic import ( + TreeSitterLogicOwner, + analyze_cpp_source, + discover_cpp_functions, +) + +if TYPE_CHECKING: + from tree_sitter import Language + +CPP_ADAPTER_ID = "docforge.reference.cpp" +CPP_ADAPTER_VERSION = "1" +CPP_EXTRACTOR_VERSION = "tree-sitter-cpp@1" +CPP_IDENTITY_VERSION = "cpp-reference-id@1" +CPP_SUPPORT_SCHEMA_VERSION = 1 + +_TRANSLATION_UNIT_SUFFIXES = frozenset({".c++", ".cc", ".cpp", ".cxx"}) +_HEADER_SUFFIXES = frozenset({".h", ".hh", ".hpp", ".hxx", ".inc", ".ipp", ".tpp"}) +_SCOPE_TYPES = { + "namespace_definition": "namespace", + "class_specifier": "class", + "struct_specifier": "struct", +} +_COMPILE_ENTRY_KEYS = frozenset({"arguments", "command", "directory", "file", "output"}) +_SUPPORTED_FACTS = ( + "cpp_file", + "cpp_translation_unit", + "cpp_namespace", + "cpp_class", + "cpp_struct", + "cpp_function", + "lexical_containment", + "project_local_quoted_include_dependency", + "function_logic", +) + + +@dataclass(frozen=True) +class CppUnsupportedFact: + """One compiler-dependent fact this syntax-only adapter does not claim.""" + + code: str + description: str + + def as_dict(self) -> dict[str, str]: + return {"code": self.code, "description": self.description} + + +_UNSUPPORTED_FACTS = ( + CppUnsupportedFact( + "call_resolution", + "Calls appear only inside function Logic and are not resolved to symbols.", + ), + CppUnsupportedFact( + "type_resolution", + "Types, templates, aliases, concepts, variables, and references are not resolved.", + ), + CppUnsupportedFact( + "inheritance_resolution", + "Base-specifier syntax is not resolved to local or external types.", + ), + CppUnsupportedFact( + "overload_ownership", + "Overload sets and out-of-line semantic ownership are not compiler-resolved.", + ), + CppUnsupportedFact( + "macro_semantics", + "Conditional compilation and macro expansion are not evaluated.", + ), + CppUnsupportedFact( + "compiler_command_execution", + "Compilation database commands and arguments are retained as fingerprints only.", + ), + CppUnsupportedFact( + "nonlocal_include_resolution", + "Angle-bracket, include-path, framework, generated, and compiler-provided " + "headers are omitted.", + ), +) + + +class _Point(Protocol): + row: int + + +class _TreeNode(Protocol): + type: str + has_error: bool + is_error: bool + is_missing: bool + start_byte: int + end_byte: int + start_point: _Point + named_children: list[_TreeNode] + + def child_by_field_name(self, name: str, /) -> _TreeNode | None: ... + + +@dataclass(frozen=True) +class _CompilationEntry: + source_path: str + directory: str + command: str | None + arguments: tuple[str, ...] | None + output: str | None + + def canonical_bytes(self) -> bytes: + value: dict[str, object] = { + "directory": self.directory, + "file": self.source_path, + } + if self.command is not None: + value["command"] = self.command + if self.arguments is not None: + value["arguments"] = list(self.arguments) + if self.output is not None: + value["output"] = self.output + return json.dumps( + value, + ensure_ascii=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + + +@dataclass(frozen=True) +class _SourceRecord: + source_id: str + source_path: str + translation_unit: bool + fingerprint: str + text: str + raw: bytes + root: _TreeNode + compilation_entry: _CompilationEntry | None = None + dependencies: tuple[str, ...] = () + + +@dataclass(frozen=True) +class _Definition: + kind: str + qualified_name: str + logic_qualified_name: str | None + line: int + parent_qualified_name: str | None + node: _TreeNode + node_id: str + + +@lru_cache(maxsize=1) +def _cpp_language() -> Language: + try: + tree_sitter = importlib.import_module("tree_sitter") + grammar = importlib.import_module("tree_sitter_cpp") + except ModuleNotFoundError as error: + raise DocForgeError( + "optional_dependency_missing", + "The requested language frontend is not installed", + extra="cpp", + install="docforge[cpp]", + missing_module=error.name, + ) from error + return cast("Language", tree_sitter.Language(grammar.language())) + + +class CppReferenceAdapter: + """Reference C++ adapter bound to one explicit compilation database.""" + + def __init__( + self, + root: Path, + *, + source_roots: Iterable[str | Path], + compilation_database: str | Path, + project_id: str = "cpp-reference", + title: str = "C++ reference project", + max_translation_units: int = 4_096, + max_headers: int = 8_192, + max_database_bytes: int = 8_000_000, + max_source_bytes: int = 2_000_000, + max_command_chars: int = 64_000, + max_arguments: int = 8_192, + max_logic_nodes_per_function: int = 2_000, + ) -> None: + resolved_root = root.resolve(strict=True) + if not resolved_root.is_dir(): + raise DocForgeError("invalid_adapter", "C++ adapter root must be a directory") + if ( + max_translation_units < 1 + or max_headers < 1 + or max_database_bytes < 1 + or max_source_bytes < 1 + or max_command_chars < 1 + or max_arguments < 1 + or max_logic_nodes_per_function < 2 + ): + raise ValueError("C++ adapter limits must be positive") + self.root = resolved_root + self.project_id = project_id + self.title = title + self.max_translation_units = max_translation_units + self.max_headers = max_headers + self.max_database_bytes = max_database_bytes + self.max_source_bytes = max_source_bytes + self.max_command_chars = max_command_chars + self.max_arguments = max_arguments + self.max_logic_nodes_per_function = max_logic_nodes_per_function + self.source_roots = self._normalize_source_roots(source_roots) + self.compilation_database = self._normalize_database_path(compilation_database) + + def support_report(self) -> dict[str, object]: + """Return deterministic scope, frontend, and limitation evidence.""" + + return { + "schema_version": CPP_SUPPORT_SCHEMA_VERSION, + "adapter_id": CPP_ADAPTER_ID, + "adapter_version": CPP_ADAPTER_VERSION, + "extractor_version": CPP_EXTRACTOR_VERSION, + "identity_version": CPP_IDENTITY_VERSION, + "frontend": "tree-sitter-cpp", + "compilation_database": self.compilation_database, + "compilation_database_is_authoritative": True, + "source_roots": list(self.source_roots), + "executes_compiler": False, + "executes_project_code": False, + "translation_unit_suffixes": sorted(_TRANSLATION_UNIT_SUFFIXES), + "header_suffixes": sorted(_HEADER_SUFFIXES), + "supported_facts": list(_SUPPORTED_FACTS), + "unsupported_facts": [fact.as_dict() for fact in _UNSUPPORTED_FACTS], + } + + def unsupported_facts(self) -> tuple[CppUnsupportedFact, ...]: + return _UNSUPPORTED_FACTS + + def load_manifest(self) -> AdapterManifest: + records = self._inventory() + path_sources = {record.source_path: record.source_id for record in records} + sources = tuple( + sorted( + ( + AdapterSource( + source_id=record.source_id, + source_path=record.source_path, + fingerprint=record.fingerprint, + extractor_version=CPP_EXTRACTOR_VERSION, + dependencies=tuple( + sorted( + path_sources[path] + for path in record.dependencies + if path_sources[path] != record.source_id + ) + ), + ) + for record in records + ), + key=lambda source: source.source_id, + ) + ) + source_hash = self._source_hash(sources) + return AdapterManifest( + project_id=self.project_id, + title=self.title, + adapter_id=CPP_ADAPTER_ID, + adapter_version=CPP_ADAPTER_VERSION, + root=self.root, + revision=source_hash[:12], + source_hash=source_hash, + families=("code",), + allowed_relations=("contains", "depends_on"), + sources=sources, + estimated_nodes=max(1, len(sources) * 10), + ) + + def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: + entry: _CompilationEntry | None = None + translation_unit = source.source_id.startswith("cpp.tu.") + if translation_unit: + entries = {item.source_path: item for item in self._compilation_entries()} + entry = entries.get(source.source_path) + if entry is None: + raise DocForgeError( + "stale_adapter_source", + "C++ translation unit is no longer in the compilation database", + source=source.source_path, + ) + record = self._read_source(source.source_path, compilation_entry=entry) + if ( + record.source_id != source.source_id + or record.fingerprint != source.fingerprint + or record.translation_unit != translation_unit + ): + raise DocForgeError( + "stale_adapter_source", + "C++ source or compilation entry changed after its manifest was captured", + source=source.source_path, + ) + return self._extract_record(record, source) + + def assemble_projection( + self, + manifest: AdapterManifest, + contributions: tuple[AdapterSourceProjection, ...], + ) -> AdapterAssembly: + expected = tuple(source.source_id for source in manifest.sources) + actual = tuple(sorted(contribution.source_id for contribution in contributions)) + if actual != expected: + raise DocForgeError( + "invalid_adapter", + "C++ assembly contributions do not match the current manifest", + ) + nodes = tuple( + sorted( + (node for contribution in contributions for node in contribution.nodes), + key=lambda item: item.node.node_id, + ) + ) + edges = tuple( + sorted( + (edge for contribution in contributions for edge in contribution.edges), + key=lambda item: ( + item.edge.source_id, + item.edge.relation, + item.edge.target_id, + ), + ) + ) + logic = tuple( + sorted( + (projection for contribution in contributions for projection in contribution.logic), + key=lambda projection: projection.owner_node_id, + ) + ) + return AdapterAssembly( + projection=AdapterProjection( + project_id=manifest.project_id, + title=manifest.title, + adapter_id=manifest.adapter_id, + adapter_version=manifest.adapter_version, + root=manifest.root, + revision=manifest.revision, + source_hash=manifest.source_hash, + nodes=nodes, + edges=edges, + ), + logic=logic, + ) + + def load_assembly(self) -> AdapterAssembly: + """Load one cache-independent complete graph and Logic assembly.""" + + manifest = self.load_manifest() + contributions = tuple(self.extract_source(source) for source in manifest.sources) + if self.load_manifest() != manifest: + raise DocForgeError( + "source_changed", + "C++ inputs changed during complete adapter extraction", + ) + return self.assemble_projection(manifest, contributions) + + def load_complete_assembly(self) -> AdapterAssembly: + """Implement the SDK's independent complete-assembly oracle.""" + + return self.load_assembly() + + def load_projection(self) -> AdapterProjection: + """Preserve the complete-projection compatibility contract.""" + + return self.load_assembly().projection + + def _normalize_database_path(self, value: str | Path) -> str: + supplied = Path(value) + candidate = supplied if supplied.is_absolute() else self.root / supplied + normalized = Path(os.path.abspath(candidate)) + if normalized.name != "compile_commands.json" or not normalized.is_relative_to(self.root): + raise DocForgeError( + "path_escape", + "C++ compilation database must be an explicit project-confined " + "compile_commands.json", + ) + relative = normalized.relative_to(self.root).as_posix() + self._validate_real_file( + normalized, + relative, + unavailable_code="invalid_compilation_database", + label="C++ compilation database", + ) + return relative + + def _normalize_source_roots( + self, + source_roots: Iterable[str | Path], + ) -> tuple[str, ...]: + normalized: list[str] = [] + for value in source_roots: + candidate = PurePosixPath(Path(value).as_posix()) + if ( + candidate.is_absolute() + or not candidate.parts + or ".." in candidate.parts + or str(candidate) in {"", "."} + ): + raise DocForgeError( + "path_escape", + "C++ source roots must be explicit project-relative directories", + ) + relative = candidate.as_posix() + absolute = self.root.joinpath(*candidate.parts) + if absolute.is_symlink(): + raise DocForgeError("path_escape", "C++ source root cannot be a symbolic link") + try: + resolved = absolute.resolve(strict=True) + except OSError as error: + raise DocForgeError( + "invalid_adapter", + "C++ source root does not exist", + source_root=relative, + ) from error + if ( + resolved != absolute + or not resolved.is_dir() + or not resolved.is_relative_to(self.root) + ): + raise DocForgeError( + "path_escape", + "C++ source root must be a confined real directory", + source_root=relative, + ) + normalized.append(relative) + result = tuple(sorted(set(normalized))) + if not result: + raise DocForgeError("invalid_adapter", "At least one C++ source root is required") + if len(result) != len(normalized): + raise DocForgeError("invalid_adapter", "C++ source roots must be unique") + paths = [PurePosixPath(value) for value in result] + for index, left in enumerate(paths): + for right in paths[index + 1 :]: + if left in right.parents or right in left.parents: + raise DocForgeError( + "invalid_adapter", + "C++ source roots must not overlap", + ) + return result + + def _source_root_for(self, path: PurePosixPath) -> str: + matches = [ + source_root + for source_root in self.source_roots + if path == PurePosixPath(source_root) or PurePosixPath(source_root) in path.parents + ] + if len(matches) != 1: + raise DocForgeError( + "path_escape", + "C++ source is outside the declared source roots", + source=path.as_posix(), + ) + return matches[0] + + def _compilation_entries(self) -> tuple[_CompilationEntry, ...]: + database = self.root.joinpath(*PurePosixPath(self.compilation_database).parts) + raw = self._read_real_file( + database, + self.compilation_database, + maximum=self.max_database_bytes, + unavailable_code="invalid_compilation_database", + label="C++ compilation database", + ) + try: + text = raw.decode("utf-8") + except UnicodeDecodeError as error: + raise DocForgeError( + "invalid_compilation_database", + "C++ compilation database must use UTF-8", + ) from error + try: + parsed = cast( + object, + json.loads(text, object_pairs_hook=self._unique_json_object), + ) + except (TypeError, ValueError, json.JSONDecodeError) as error: + raise DocForgeError( + "invalid_compilation_database", + "C++ compilation database is not valid duplicate-free JSON", + ) from error + if not isinstance(parsed, list) or not parsed: + raise DocForgeError( + "invalid_compilation_database", + "C++ compilation database must be a non-empty JSON array", + ) + parsed_items = cast("list[object]", parsed) + if len(parsed_items) > self.max_translation_units: + raise DocForgeError( + "adapter_too_large", + "C++ translation-unit inventory exceeds the configured limit", + maximum=self.max_translation_units, + ) + entries = tuple( + self._compilation_entry(value, index) for index, value in enumerate(parsed_items) + ) + paths = [entry.source_path for entry in entries] + if len(paths) != len(set(paths)): + duplicates = sorted(path for path in set(paths) if paths.count(path) > 1) + raise DocForgeError( + "duplicate_translation_unit", + "C++ compilation database repeats a normalized translation unit", + sources=duplicates, + ) + return tuple(sorted(entries, key=lambda entry: entry.source_path)) + + def _compilation_entry(self, value: object, index: int) -> _CompilationEntry: + if not isinstance(value, dict): + raise DocForgeError( + "invalid_compilation_database", + "Each C++ compilation entry must be a JSON object", + entry=index, + ) + raw_document = cast("dict[object, object]", value) + if any(not isinstance(key, str) for key in raw_document): + raise DocForgeError( + "invalid_compilation_database", + "Each C++ compilation entry must have string keys", + entry=index, + ) + document = cast("dict[str, object]", raw_document) + keys = set(document) + if ( + not keys <= _COMPILE_ENTRY_KEYS + or not {"directory", "file"} <= keys + or (("command" in keys) == ("arguments" in keys)) + ): + raise DocForgeError( + "invalid_compilation_database", + "Each C++ compilation entry requires directory, file, and exactly " + "one of command or arguments", + entry=index, + ) + directory_value = self._bounded_string( + document["directory"], + "directory", + index, + maximum=self.max_command_chars, + ) + file_value = self._bounded_string( + document["file"], + "file", + index, + maximum=self.max_command_chars, + ) + directory_path = Path(directory_value) + directory_candidate = ( + directory_path if directory_path.is_absolute() else self.root / directory_path + ) + directory = Path(os.path.abspath(directory_candidate)) + if not directory.is_relative_to(self.root): + raise DocForgeError( + "path_escape", + "C++ compilation entry directory escapes the project root", + entry=index, + ) + self._validate_real_directory(directory, index) + file_path = Path(file_value) + file_candidate = file_path if file_path.is_absolute() else directory / file_path + source = Path(os.path.abspath(file_candidate)) + if ( + not source.is_relative_to(self.root) + or source.suffix.casefold() not in _TRANSLATION_UNIT_SUFFIXES + ): + raise DocForgeError( + "path_escape", + "C++ compilation entry file must be a confined C++ translation unit", + entry=index, + ) + relative = source.relative_to(self.root).as_posix() + self._source_root_for(PurePosixPath(relative)) + self._validate_real_file( + source, + relative, + unavailable_code="invalid_compilation_database", + label="C++ translation unit", + ) + command: str | None = None + arguments: tuple[str, ...] | None = None + if "command" in value: + command = self._bounded_string( + document["command"], + "command", + index, + maximum=self.max_command_chars, + ) + else: + raw_arguments = document["arguments"] + if not isinstance(raw_arguments, list): + raise DocForgeError( + "invalid_compilation_database", + "C++ compilation arguments must be a non-empty bounded string array", + entry=index, + ) + argument_items = cast("list[object]", raw_arguments) + if not argument_items or len(argument_items) > self.max_arguments: + raise DocForgeError( + "invalid_compilation_database", + "C++ compilation arguments must be a non-empty bounded string array", + entry=index, + ) + arguments = tuple( + self._bounded_string( + argument, + "arguments", + index, + maximum=self.max_command_chars, + ) + for argument in argument_items + ) + output = ( + self._bounded_string( + document["output"], + "output", + index, + maximum=self.max_command_chars, + ) + if "output" in value + else None + ) + directory_relative = directory.relative_to(self.root).as_posix() or "." + return _CompilationEntry( + source_path=relative, + directory=directory_relative, + command=command, + arguments=arguments, + output=output, + ) + + @staticmethod + def _unique_json_object(pairs: list[tuple[str, object]]) -> dict[str, object]: + result: dict[str, object] = {} + for key, value in pairs: + if key in result: + raise ValueError(f"duplicate JSON key: {key}") + result[key] = value + return result + + @staticmethod + def _bounded_string( + value: object, + field: str, + index: int, + *, + maximum: int, + ) -> str: + if ( + not isinstance(value, str) + or not value + or len(value) > maximum + or any(ord(character) < 32 or ord(character) == 127 for character in value) + ): + raise DocForgeError( + "invalid_compilation_database", + f"C++ compilation entry {field} must be a non-empty bounded control-free string", + entry=index, + ) + return value + + def _validate_real_directory(self, path: Path, index: int) -> None: + try: + before = path.lstat() + resolved = path.resolve(strict=True) + except OSError as error: + raise DocForgeError( + "invalid_compilation_database", + "C++ compilation entry directory does not exist", + entry=index, + ) from error + if ( + stat.S_ISLNK(before.st_mode) + or not stat.S_ISDIR(before.st_mode) + or resolved != path + or not resolved.is_relative_to(self.root) + ): + raise DocForgeError( + "path_escape", + "C++ compilation entry directory must be a confined real directory", + entry=index, + ) + + def _inventory(self) -> tuple[_SourceRecord, ...]: + entries = self._compilation_entries() + records: dict[str, _SourceRecord] = {} + queue: list[tuple[str, _CompilationEntry | None]] = [ + (entry.source_path, entry) for entry in entries + ] + header_count = 0 + while queue: + source_path, compilation_entry = queue.pop(0) + existing = records.get(source_path) + if existing is not None: + if ( + compilation_entry is not None + and existing.compilation_entry != compilation_entry + ): + raise DocForgeError( + "duplicate_translation_unit", + "C++ source has conflicting compilation entries", + source=source_path, + ) + continue + record = self._read_source(source_path, compilation_entry=compilation_entry) + dependencies: list[str] = [] + for include in self._quoted_includes(record): + resolved = self._resolve_quoted_header(source_path, include) + if resolved is None or resolved == source_path: + continue + dependencies.append(resolved) + if resolved not in records and all(item[0] != resolved for item in queue): + header_count += 1 + if header_count > self.max_headers: + raise DocForgeError( + "adapter_too_large", + "C++ project-local header inventory exceeds the configured limit", + maximum=self.max_headers, + ) + queue.append((resolved, None)) + records[source_path] = replace( + record, + dependencies=tuple(sorted(set(dependencies))), + ) + queue.sort(key=lambda item: item[0]) + return tuple(sorted(records.values(), key=lambda record: record.source_path)) + + def _read_source( + self, + relative: str, + *, + compilation_entry: _CompilationEntry | None, + ) -> _SourceRecord: + path = PurePosixPath(relative) + translation_unit = compilation_entry is not None + allowed_suffixes = _TRANSLATION_UNIT_SUFFIXES if translation_unit else _HEADER_SUFFIXES + if ( + path.is_absolute() + or ".." in path.parts + or path.suffix.casefold() not in allowed_suffixes + ): + raise DocForgeError("path_escape", "C++ source path is unsafe", source=relative) + self._source_root_for(path) + candidate = self.root.joinpath(*path.parts) + raw = self._read_real_file( + candidate, + relative, + maximum=self.max_source_bytes, + unavailable_code="stale_adapter_source", + label="C++ source", + ) + try: + text = raw.decode("utf-8-sig") + except UnicodeDecodeError as error: + raise DocForgeError( + "invalid_cpp_source", + "C++ source must use UTF-8", + source=relative, + ) from error + parse_raw, root = self._parse(text) + if root.has_error: + raise DocForgeError( + "invalid_cpp_source", + "C++ source cannot be parsed by the reference adapter", + source=relative, + line=self._first_error_line(root) or 1, + ) + source_id = self._source_id(relative, translation_unit=translation_unit) + digest = hashlib.sha256() + digest.update(CPP_EXTRACTOR_VERSION.encode()) + digest.update(b"\0source\0") + digest.update(raw) + if compilation_entry is not None: + digest.update(b"\0compilation-entry\0") + digest.update(compilation_entry.canonical_bytes()) + return _SourceRecord( + source_id=source_id, + source_path=relative, + translation_unit=translation_unit, + fingerprint=digest.hexdigest(), + text=text, + raw=parse_raw, + root=root, + compilation_entry=compilation_entry, + ) + + @staticmethod + def _parse(source: str) -> tuple[bytes, _TreeNode]: + try: + tree_sitter = importlib.import_module("tree_sitter") + except ModuleNotFoundError as error: + raise DocForgeError( + "optional_dependency_missing", + "The requested language frontend is not installed", + extra="cpp", + install="docforge[cpp]", + missing_module=error.name, + ) from error + raw = source.encode("utf-8") + tree = tree_sitter.Parser(_cpp_language()).parse(raw) + return raw, cast("_TreeNode", tree.root_node) + + @classmethod + def _first_error_line(cls, node: _TreeNode) -> int | None: + if node.is_error or node.is_missing: + return node.start_point.row + 1 + for child in node.named_children: + line = cls._first_error_line(child) + if line is not None: + return line + return None + + def _resolve_quoted_header(self, source_path: str, include: str) -> str | None: + if ( + not include + or "\\" in include + or "\x00" in include + or PurePosixPath(include).is_absolute() + ): + return None + source_parent = self.root.joinpath(*PurePosixPath(source_path).parent.parts) + candidate = Path(os.path.abspath(source_parent / include)) + if not candidate.is_relative_to(self.root): + raise DocForgeError( + "path_escape", + "Quoted C++ include escapes the project root", + source=source_path, + include=include, + ) + if candidate.is_symlink(): + raise DocForgeError( + "path_escape", + "Quoted C++ include resolves through a symbolic link", + source=source_path, + include=include, + ) + if not candidate.exists(): + return None + if candidate.suffix.casefold() not in _HEADER_SUFFIXES: + return None + relative = candidate.relative_to(self.root).as_posix() + self._source_root_for(PurePosixPath(relative)) + self._validate_real_file( + candidate, + relative, + unavailable_code="invalid_cpp_source", + label="Quoted C++ header", + ) + return relative + + @classmethod + def _quoted_includes(cls, record: _SourceRecord) -> tuple[str, ...]: + result: set[str] = set() + stack = [record.root] + while stack: + node = stack.pop() + if node.type == "preproc_include": + path = node.child_by_field_name("path") + if path is not None: + value = cls._text(path, record.raw) + if ( + len(value) >= 2 + and value.startswith('"') + and value.endswith('"') + and "\\" not in value + and "\x00" not in value + ): + result.add(value[1:-1]) + stack.extend(node.named_children) + return tuple(sorted(result)) + + def _extract_record( + self, + record: _SourceRecord, + source: AdapterSource, + ) -> AdapterSourceProjection: + file_id = self._file_node_id(source.source_id) + root_id = file_id + nodes: list[AdapterNode] = [ + self._node( + node_id=file_id, + title=record.source_path, + kind="file", + qualified_name=record.source_path, + content=record.text, + source_path=record.source_path, + anchor="L1", + ) + ] + edges: list[AdapterEdge] = [] + if record.translation_unit: + root_id = self._translation_unit_node_id(source.source_id) + nodes.append( + self._node( + node_id=root_id, + title=record.source_path, + kind="translation_unit", + qualified_name=record.source_path, + content=f"C++ translation unit {record.source_path}.", + source_path=record.source_path, + anchor="L1", + ) + ) + edges.append(self._edge(root_id, "contains", file_id, "compilation_database")) + definitions = self._definitions(record) + definition_ids = { + definition.qualified_name: definition.node_id for definition in definitions + } + for definition in definitions: + parent_id = ( + definition_ids.get(definition.parent_qualified_name, file_id) + if definition.parent_qualified_name is not None + else file_id + ) + anchor = f"L{definition.line}" + nodes.append( + self._node( + node_id=definition.node_id, + title=definition.qualified_name, + kind=definition.kind, + qualified_name=definition.qualified_name, + content=self._text(definition.node, record.raw), + source_path=record.source_path, + anchor=anchor, + ) + ) + edges.append( + self._edge( + parent_id, + "contains", + definition.node_id, + "tree_sitter_syntax", + anchor=anchor, + ) + ) + for dependency in source.dependencies: + edges.append( + self._edge( + file_id, + "depends_on", + self._file_node_id(dependency), + "quoted_project_header", + ) + ) + owners = tuple( + TreeSitterLogicOwner( + owner_node_id=definition.node_id, + qualified_name=cast(str, definition.logic_qualified_name), + line=definition.line, + ) + for definition in definitions + if definition.kind == "function" + ) + logic = analyze_cpp_source( + record.text, + source_id=source.source_id, + owners=owners, + filename=record.source_path, + max_nodes_per_function=self.max_logic_nodes_per_function, + ) + return AdapterSourceProjection( + source_id=source.source_id, + fingerprint=source.fingerprint, + nodes=tuple(sorted(nodes, key=lambda item: item.node.node_id)), + edges=tuple( + sorted( + edges, + key=lambda item: ( + item.edge.source_id, + item.edge.relation, + item.edge.target_id, + ), + ) + ), + logic=logic, + ) + + def _definitions(self, record: _SourceRecord) -> tuple[_Definition, ...]: + definitions: list[_Definition] = [] + identities: set[tuple[str, int, str]] = set() + + def visit(node: _TreeNode, scopes: tuple[str, ...]) -> None: + definition = self._definition(record, node, scopes) + next_scopes = scopes + if definition is not None: + identity = ( + definition.qualified_name, + definition.line, + definition.kind, + ) + if identity in identities: + raise DocForgeError( + "ambiguous_cpp_symbol", + "C++ source repeats a syntax-local symbol identity", + source=record.source_path, + qualified_name=definition.qualified_name, + line=definition.line, + ) + identities.add(identity) + definitions.append(definition) + if definition.kind != "function": + next_scopes = tuple(definition.qualified_name.split(".")) + for child in node.named_children: + visit(child, next_scopes) + + visit(record.root, ()) + ordered = tuple(sorted(definitions, key=lambda item: item.node_id)) + public_functions = { + (function.qualified_name, function.line) + for function in discover_cpp_functions(record.text) + } + extracted_functions = { + (cast(str, definition.logic_qualified_name), definition.line) + for definition in ordered + if definition.kind == "function" + } + if extracted_functions != public_functions: + raise DocForgeError( + "invalid_adapter", + "C++ function discovery drifted from the Logic frontend", + source=record.source_path, + ) + if len(public_functions) != len( + [definition for definition in ordered if definition.kind == "function"] + ): + raise DocForgeError( + "ambiguous_cpp_symbol", + "C++ source has functions that cannot receive unique syntax-local Logic owners", + source=record.source_path, + ) + return ordered + + def _definition( + self, + record: _SourceRecord, + node: _TreeNode, + scopes: tuple[str, ...], + ) -> _Definition | None: + kind = _SCOPE_TYPES.get(node.type) + name: str | None = None + logic_name: str | None = None + if kind is not None: + name = self._field_text(node, "name", record.raw) + elif node.type == "function_definition": + declarator = node.child_by_field_name("declarator") + logic_name = self._declarator_name(declarator, record.raw) + name = logic_name + kind = "function" if name is not None else None + if kind is None or name is None: + return None + normalized_name = name.replace("::", ".") + if "::" in name: + qualified_name = normalized_name + else: + qualified_name = ".".join((*scopes, normalized_name)) + if kind == "function": + logic_name = ".".join((*scopes, name)) + parent = ".".join(scopes) or None + line = node.start_point.row + 1 + return _Definition( + kind=kind, + qualified_name=qualified_name, + logic_qualified_name=logic_name if kind == "function" else None, + line=line, + parent_qualified_name=parent, + node=node, + node_id=self._symbol_node_id( + kind, + record.source_id, + qualified_name, + line=line, + ), + ) + + @classmethod + def _declarator_name(cls, node: _TreeNode | None, raw: bytes) -> str | None: + if node is None: + return None + if node.type in { + "identifier", + "field_identifier", + "operator_name", + "destructor_name", + "qualified_identifier", + }: + return cls._text(node, raw) + for field in ("declarator", "name"): + child = node.child_by_field_name(field) + result = cls._declarator_name(child, raw) + if result: + return result + for child in node.named_children: + result = cls._declarator_name(child, raw) + if result: + return result + return None + + @staticmethod + def _source_id(relative: str, *, translation_unit: bool) -> str: + source_kind = "tu" if translation_unit else "header" + digest = hashlib.sha256( + f"{CPP_IDENTITY_VERSION}\0{source_kind}\0{relative}".encode() + ).hexdigest()[:24] + return f"cpp.{source_kind}.{digest}" + + @staticmethod + def _file_node_id(source_id: str) -> str: + if source_id.startswith("cpp.tu."): + return source_id.replace("cpp.tu.", "cpp.file.", 1) + return source_id.replace("cpp.header.", "cpp.file.", 1) + + @staticmethod + def _translation_unit_node_id(source_id: str) -> str: + return source_id.replace("cpp.tu.", "cpp.translation-unit.", 1) + + @staticmethod + def _symbol_node_id( + kind: str, + source_id: str, + qualified_name: str, + *, + line: int, + ) -> str: + digest = hashlib.sha256( + (f"{CPP_IDENTITY_VERSION}\0{kind}\0{source_id}\0{qualified_name}\0{line}").encode() + ).hexdigest()[:24] + return f"cpp.{kind}.{digest}" + + @staticmethod + def _source_hash(sources: Sequence[AdapterSource]) -> str: + digest = hashlib.sha256() + digest.update(CPP_ADAPTER_VERSION.encode()) + digest.update(CPP_EXTRACTOR_VERSION.encode()) + digest.update(CPP_IDENTITY_VERSION.encode()) + for source in sources: + digest.update(b"\0source\0") + digest.update(source.source_id.encode()) + digest.update(b"\0path\0") + digest.update(source.source_path.encode()) + digest.update(b"\0fingerprint\0") + digest.update(source.fingerprint.encode()) + for dependency in source.dependencies: + digest.update(b"\0dependency\0") + digest.update(dependency.encode()) + return digest.hexdigest() + + @staticmethod + def _text(node: _TreeNode, raw: bytes) -> str: + return raw[node.start_byte : node.end_byte].decode("utf-8") + + @classmethod + def _field_text(cls, node: _TreeNode, field: str, raw: bytes) -> str | None: + child = node.child_by_field_name(field) + return cls._text(child, raw) if child is not None else None + + @staticmethod + def _node( + *, + node_id: str, + title: str, + kind: str, + qualified_name: str, + content: str, + source_path: str, + anchor: str, + ) -> AdapterNode: + normalized = content.strip() or f"C++ {kind} {qualified_name}." + return AdapterNode( + node=GraphNode( + node_id=node_id, + title=title, + family="code", + authority="derived", + status="active", + tags=tuple(sorted({"cpp", kind})), + summary=f"C++ {kind} syntax fact for {qualified_name}.", + content=normalized, + source_path=source_path, + source_anchor=anchor, + content_hash=hashlib.sha256(normalized.encode()).hexdigest(), + ), + metadata=( + ("evidence", "tree-sitter-syntax"), + ("extractor", CPP_EXTRACTOR_VERSION), + ("identity", CPP_IDENTITY_VERSION), + ("kind", kind), + ("language", "cpp"), + ("qualified_name", qualified_name), + ), + ) + + @staticmethod + def _edge( + source_id: str, + relation: str, + target_id: str, + evidence: str, + *, + anchor: str | None = None, + ) -> AdapterEdge: + metadata = [("evidence", evidence)] + if anchor is not None: + metadata.append(("source_anchor", anchor)) + return AdapterEdge( + edge=Edge(source_id, relation, target_id), + metadata=tuple(metadata), + ) + + def _validate_real_file( + self, + path: Path, + relative: str, + *, + unavailable_code: str, + label: str, + ) -> os.stat_result: + try: + before = path.lstat() + resolved = path.resolve(strict=True) + except OSError as error: + raise DocForgeError( + unavailable_code, + f"{label} is unavailable", + source=relative, + ) from error + if ( + stat.S_ISLNK(before.st_mode) + or not stat.S_ISREG(before.st_mode) + or resolved != path + or not resolved.is_relative_to(self.root) + ): + raise DocForgeError( + "path_escape", + f"{label} must be a confined real regular file", + source=relative, + ) + return before + + def _read_real_file( + self, + path: Path, + relative: str, + *, + maximum: int, + unavailable_code: str, + label: str, + ) -> bytes: + before = self._validate_real_file( + path, + relative, + unavailable_code=unavailable_code, + label=label, + ) + if before.st_size > maximum: + raise DocForgeError( + "source_too_large", + f"{label} exceeds the configured adapter limit", + source=relative, + maximum=maximum, + ) + descriptor = -1 + try: + flags = os.O_RDONLY | getattr(os, "O_BINARY", 0) + flags |= getattr(os, "O_NOFOLLOW", 0) + descriptor = os.open(path, flags) + opened = os.fstat(descriptor) + if not stat.S_ISREG(opened.st_mode) or (opened.st_dev, opened.st_ino) != ( + before.st_dev, + before.st_ino, + ): + raise DocForgeError( + "path_escape", + f"{label} identity changed before it was opened", + source=relative, + ) + with os.fdopen(descriptor, "rb") as handle: + descriptor = -1 + raw = handle.read(maximum + 1) + after_open = os.fstat(handle.fileno()) + after = path.lstat() + except DocForgeError: + raise + except OSError as error: + raise DocForgeError( + unavailable_code, + f"{label} changed while being read", + source=relative, + ) from error + finally: + if descriptor >= 0: + os.close(descriptor) + identities = ( + (before.st_dev, before.st_ino, before.st_size, before.st_mtime_ns), + ( + after_open.st_dev, + after_open.st_ino, + after_open.st_size, + after_open.st_mtime_ns, + ), + (after.st_dev, after.st_ino, after.st_size, after.st_mtime_ns), + ) + if ( + len(raw) > maximum + or identities[0] != identities[1] + or identities[0] != identities[2] + or len(raw) != after.st_size + ): + raise DocForgeError( + unavailable_code, + f"{label} changed while being read", + source=relative, + ) + return raw diff --git a/src/docforge/adapters/javascript.py b/src/docforge/adapters/javascript.py new file mode 100644 index 0000000..37f9238 --- /dev/null +++ b/src/docforge/adapters/javascript.py @@ -0,0 +1,1157 @@ +"""Deterministic JavaScript and TypeScript reference adapter. + +The adapter parses explicitly declared source roots as untrusted data with the +optional pinned Tree-sitter frontends. It never imports, compiles, transpiles, +or executes project code. Only relative static imports and re-exports that +resolve to another inventoried source file become dependency evidence. +""" + +from __future__ import annotations + +import hashlib +import importlib +import os +import posixpath +import stat +from collections.abc import Iterable, Sequence +from dataclasses import dataclass +from functools import lru_cache +from pathlib import Path, PurePosixPath +from typing import TYPE_CHECKING, Protocol, cast + +from ..adapter_sdk import ( + AdapterAssembly, + AdapterEdge, + AdapterManifest, + AdapterNode, + AdapterProjection, + AdapterSource, + AdapterSourceProjection, + Edge, +) +from ..adapter_sdk import ( + Node as GraphNode, +) +from ..errors import DocForgeError +from ..treesitter_logic import ( + DiscoveredFunction, + TreeSitterLogicOwner, + analyze_javascript_source, + analyze_typescript_source, + discover_javascript_functions, + discover_typescript_functions, +) + +if TYPE_CHECKING: + from tree_sitter import Language + +JAVASCRIPT_ADAPTER_ID = "docforge.reference.javascript" +JAVASCRIPT_ADAPTER_VERSION = "1" +JAVASCRIPT_EXTRACTOR_VERSION = "tree-sitter-script@1" +JAVASCRIPT_IDENTITY_VERSION = "javascript-reference-id@1" +JAVASCRIPT_SUPPORT_SCHEMA_VERSION = 1 + +_JAVASCRIPT_SUFFIXES = frozenset({".js", ".mjs", ".cjs"}) +_TYPESCRIPT_SUFFIXES = frozenset({".ts", ".mts", ".cts"}) +_SOURCE_SUFFIXES = (*sorted(_JAVASCRIPT_SUFFIXES), *sorted(_TYPESCRIPT_SUFFIXES)) +_FUNCTION_TYPES = frozenset( + {"function_declaration", "generator_function_declaration", "method_definition"} +) + +_SUPPORTED_FACTS = ( + "script_file", + "script_module", + "script_class", + "script_function", + "lexical_containment", + "project_local_static_import_dependency", + "function_logic", +) + + +@dataclass(frozen=True) +class JavaScriptUnsupportedFact: + """One semantic fact this syntax-only reference adapter does not claim.""" + + code: str + description: str + + def as_dict(self) -> dict[str, str]: + return {"code": self.code, "description": self.description} + + +_UNSUPPORTED_FACTS = ( + JavaScriptUnsupportedFact( + "call_resolution", + "Calls are represented only inside function Logic and are not resolved to symbols.", + ), + JavaScriptUnsupportedFact( + "dynamic_module_resolution", + "Dynamic import(), require(), package exports, and loader hooks are not dependencies.", + ), + JavaScriptUnsupportedFact( + "inheritance_resolution", + "Class extends and implements clauses are not resolved.", + ), + JavaScriptUnsupportedFact( + "module_configuration_resolution", + "Bare specifiers, aliases, tsconfig paths, and JavaScript-to-TypeScript " + "remapping are omitted.", + ), + JavaScriptUnsupportedFact( + "runtime_generated_facts", + "Decorators, prototypes, proxies, eval, and executed module code are never evaluated.", + ), + JavaScriptUnsupportedFact( + "type_and_symbol_resolution", + "Types, interfaces, overloads, variables, references, and re-exported " + "symbols are not resolved.", + ), +) + + +class _Point(Protocol): + row: int + + +class _TreeNode(Protocol): + type: str + has_error: bool + is_error: bool + is_missing: bool + start_byte: int + end_byte: int + start_point: _Point + named_children: list[_TreeNode] + + def child_by_field_name(self, name: str, /) -> _TreeNode | None: ... + + +@dataclass(frozen=True) +class _SourceRecord: + source_id: str + source_path: str + module_name: str + language: str + fingerprint: str + text: str + raw: bytes + root: _TreeNode | None + + +@dataclass(frozen=True) +class _Definition: + kind: str + qualified_name: str + line: int + parent_qualified_name: str | None + node: _TreeNode + node_id: str + + +@lru_cache(maxsize=2) +def _language(language: str) -> Language: + if language == "javascript": + grammar_module = "tree_sitter_javascript" + grammar_function = "language" + extra = "javascript" + elif language == "typescript": + grammar_module = "tree_sitter_typescript" + grammar_function = "language_typescript" + extra = "typescript" + else: + raise ValueError(f"Unsupported script language: {language}") + try: + tree_sitter = importlib.import_module("tree_sitter") + grammar = importlib.import_module(grammar_module) + except ModuleNotFoundError as error: + raise DocForgeError( + "optional_dependency_missing", + "The requested language frontend is not installed", + extra=extra, + install=f"docforge[{extra}]", + missing_module=error.name, + ) from error + language_type = tree_sitter.Language + language_factory = getattr(grammar, grammar_function) + return cast("Language", language_type(language_factory())) + + +class JavaScriptReferenceAdapter: + """Reference JavaScript/TypeScript adapter over explicit source roots.""" + + def __init__( + self, + root: Path, + *, + source_roots: Iterable[str | Path], + project_id: str = "javascript-reference", + title: str = "JavaScript and TypeScript reference project", + max_sources: int = 4_096, + max_source_bytes: int = 1_000_000, + max_logic_nodes_per_function: int = 2_000, + ) -> None: + resolved_root = root.resolve(strict=True) + if not resolved_root.is_dir(): + raise DocForgeError("invalid_adapter", "Script adapter root must be a directory") + if max_sources < 1 or max_source_bytes < 1 or max_logic_nodes_per_function < 2: + raise ValueError("Script adapter limits must be positive") + self.root = resolved_root + self.project_id = project_id + self.title = title + self.max_sources = max_sources + self.max_source_bytes = max_source_bytes + self.max_logic_nodes_per_function = max_logic_nodes_per_function + self.source_roots = self._normalize_source_roots(source_roots) + + def support_report(self) -> dict[str, object]: + """Return deterministic scope, frontend, and limitation evidence.""" + + return { + "schema_version": JAVASCRIPT_SUPPORT_SCHEMA_VERSION, + "adapter_id": JAVASCRIPT_ADAPTER_ID, + "adapter_version": JAVASCRIPT_ADAPTER_VERSION, + "extractor_version": JAVASCRIPT_EXTRACTOR_VERSION, + "identity_version": JAVASCRIPT_IDENTITY_VERSION, + "frontends": { + "javascript": "tree-sitter-javascript", + "typescript": "tree-sitter-typescript", + }, + "imports_project_code": False, + "executes_project_code": False, + "source_roots": list(self.source_roots), + "source_suffixes": list(_SOURCE_SUFFIXES), + "supported_facts": list(_SUPPORTED_FACTS), + "unsupported_facts": [fact.as_dict() for fact in _UNSUPPORTED_FACTS], + } + + def unsupported_facts(self) -> tuple[JavaScriptUnsupportedFact, ...]: + return _UNSUPPORTED_FACTS + + def load_manifest(self) -> AdapterManifest: + records = self._inventory(parse=False) + path_sources = {record.source_path: record.source_id for record in records} + sources = tuple( + sorted( + ( + AdapterSource( + source_id=record.source_id, + source_path=record.source_path, + fingerprint=record.fingerprint, + extractor_version=(f"{JAVASCRIPT_EXTRACTOR_VERSION}:{record.language}"), + dependencies=self._local_dependencies(record, path_sources), + ) + for record in records + ), + key=lambda source: source.source_id, + ) + ) + source_hash = self._source_hash(sources) + return AdapterManifest( + project_id=self.project_id, + title=self.title, + adapter_id=JAVASCRIPT_ADAPTER_ID, + adapter_version=JAVASCRIPT_ADAPTER_VERSION, + root=self.root, + revision=source_hash[:12], + source_hash=source_hash, + families=("code",), + allowed_relations=("contains", "depends_on"), + sources=sources, + estimated_nodes=max(1, len(sources) * 8), + ) + + def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: + record = self._read_source(source.source_path) + if record.source_id != source.source_id or record.fingerprint != source.fingerprint: + raise DocForgeError( + "stale_adapter_source", + "Script source changed after its manifest was captured", + source=source.source_path, + ) + return self._extract_record(record, source) + + def assemble_projection( + self, + manifest: AdapterManifest, + contributions: tuple[AdapterSourceProjection, ...], + ) -> AdapterAssembly: + expected = tuple(source.source_id for source in manifest.sources) + actual = tuple(sorted(contribution.source_id for contribution in contributions)) + if actual != expected: + raise DocForgeError( + "invalid_adapter", + "Script assembly contributions do not match the current manifest", + ) + nodes = tuple( + sorted( + (node for contribution in contributions for node in contribution.nodes), + key=lambda item: item.node.node_id, + ) + ) + edges = tuple( + sorted( + (edge for contribution in contributions for edge in contribution.edges), + key=lambda item: ( + item.edge.source_id, + item.edge.relation, + item.edge.target_id, + ), + ) + ) + logic = tuple( + sorted( + (projection for contribution in contributions for projection in contribution.logic), + key=lambda projection: projection.owner_node_id, + ) + ) + return AdapterAssembly( + projection=AdapterProjection( + project_id=manifest.project_id, + title=manifest.title, + adapter_id=manifest.adapter_id, + adapter_version=manifest.adapter_version, + root=manifest.root, + revision=manifest.revision, + source_hash=manifest.source_hash, + nodes=nodes, + edges=edges, + ), + logic=logic, + ) + + def load_assembly(self) -> AdapterAssembly: + """Load one cache-independent complete graph and Logic assembly.""" + + manifest = self.load_manifest() + contributions = tuple(self.extract_source(source) for source in manifest.sources) + if self.load_manifest() != manifest: + raise DocForgeError( + "source_changed", + "Script sources changed during complete adapter extraction", + ) + return self.assemble_projection(manifest, contributions) + + def load_complete_assembly(self) -> AdapterAssembly: + return self.load_assembly() + + def load_projection(self) -> AdapterProjection: + return self.load_assembly().projection + + def _normalize_source_roots(self, source_roots: Iterable[str | Path]) -> tuple[str, ...]: + normalized: list[str] = [] + for value in source_roots: + candidate = PurePosixPath(Path(value).as_posix()) + if ( + candidate.is_absolute() + or not candidate.parts + or ".." in candidate.parts + or str(candidate) in {"", "."} + ): + raise DocForgeError( + "path_escape", + "Script source roots must be explicit project-relative directories", + ) + relative = candidate.as_posix() + absolute = self.root.joinpath(*candidate.parts) + if absolute.is_symlink(): + raise DocForgeError("path_escape", "Script source root cannot be a symbolic link") + try: + resolved = absolute.resolve(strict=True) + except OSError as error: + raise DocForgeError( + "invalid_adapter", + "Script source root does not exist", + source_root=relative, + ) from error + if ( + resolved != absolute + or not resolved.is_dir() + or not resolved.is_relative_to(self.root) + ): + raise DocForgeError( + "path_escape", + "Script source root must be a confined real directory", + source_root=relative, + ) + normalized.append(relative) + result = tuple(sorted(set(normalized))) + if not result: + raise DocForgeError("invalid_adapter", "At least one script source root is required") + if len(result) != len(normalized): + raise DocForgeError("invalid_adapter", "Script source roots must be unique") + paths = [PurePosixPath(value) for value in result] + for index, left in enumerate(paths): + for right in paths[index + 1 :]: + if left in right.parents or right in left.parents: + raise DocForgeError( + "invalid_adapter", + "Script source roots must not overlap", + ) + return result + + def _inventory(self, *, parse: bool = True) -> tuple[_SourceRecord, ...]: + paths: list[str] = [] + for source_root in self.source_roots: + absolute_root = self.root.joinpath(*PurePosixPath(source_root).parts) + if ( + absolute_root.is_symlink() + or not absolute_root.is_dir() + or absolute_root.resolve(strict=True) != absolute_root + ): + raise DocForgeError( + "path_escape", + "Script source root changed after adapter binding", + source_root=source_root, + ) + for directory, directory_names, file_names in os.walk( + absolute_root, + topdown=True, + followlinks=False, + ): + current = Path(directory) + for name in tuple(directory_names): + child = current / name + if child.is_symlink(): + raise DocForgeError( + "path_escape", + "Script source inventory contains a symbolic-link directory", + source=child.relative_to(self.root).as_posix(), + ) + directory_names[:] = sorted( + name for name in directory_names if name != "node_modules" + ) + for name in sorted(file_names): + if Path(name).suffix not in _SOURCE_SUFFIXES: + continue + child = current / name + relative = child.relative_to(self.root).as_posix() + if child.is_symlink(): + raise DocForgeError( + "path_escape", + "Script source inventory contains a symbolic-link file", + source=relative, + ) + paths.append(relative) + if len(paths) > self.max_sources: + raise DocForgeError( + "adapter_too_large", + "Script source inventory exceeds the configured limit", + maximum=self.max_sources, + ) + records = tuple(self._read_source(path, parse=parse) for path in sorted(paths)) + modules = [record.module_name for record in records] + if len(modules) != len(set(modules)): + duplicates = sorted(module for module in set(modules) if modules.count(module) > 1) + raise DocForgeError( + "ambiguous_script_module", + "Declared script source roots produce duplicate module names", + modules=duplicates, + ) + return records + + def _read_source(self, relative: str, *, parse: bool = True) -> _SourceRecord: + path = PurePosixPath(relative) + if path.is_absolute() or ".." in path.parts or path.suffix not in _SOURCE_SUFFIXES: + raise DocForgeError("path_escape", "Script source path is unsafe") + source_root = self._source_root_for(path) + candidate = self.root.joinpath(*path.parts) + try: + before = candidate.lstat() + except OSError as error: + raise DocForgeError( + "stale_adapter_source", + "Script source is unavailable", + source=relative, + ) from error + if ( + stat.S_ISLNK(before.st_mode) + or not stat.S_ISREG(before.st_mode) + or candidate.resolve(strict=True) != candidate + or not candidate.is_relative_to(self.root) + ): + raise DocForgeError( + "path_escape", + "Script source must be a confined regular file", + source=relative, + ) + if before.st_size > self.max_source_bytes: + raise DocForgeError( + "source_too_large", + "Script source exceeds the configured adapter limit", + source=relative, + maximum=self.max_source_bytes, + ) + descriptor = -1 + try: + flags = os.O_RDONLY | getattr(os, "O_BINARY", 0) + flags |= getattr(os, "O_NOFOLLOW", 0) + descriptor = os.open(candidate, flags) + opened = os.fstat(descriptor) + if not stat.S_ISREG(opened.st_mode) or (opened.st_dev, opened.st_ino) != ( + before.st_dev, + before.st_ino, + ): + raise DocForgeError( + "path_escape", + "Script source identity changed before it was opened", + source=relative, + ) + with os.fdopen(descriptor, "rb") as handle: + descriptor = -1 + raw = handle.read(self.max_source_bytes + 1) + after_open = os.fstat(handle.fileno()) + after = candidate.lstat() + except DocForgeError: + raise + except OSError as error: + raise DocForgeError( + "stale_adapter_source", + "Script source changed while being read", + source=relative, + ) from error + finally: + if descriptor >= 0: + os.close(descriptor) + identities = ( + (before.st_dev, before.st_ino, before.st_size, before.st_mtime_ns), + ( + after_open.st_dev, + after_open.st_ino, + after_open.st_size, + after_open.st_mtime_ns, + ), + (after.st_dev, after.st_ino, after.st_size, after.st_mtime_ns), + ) + if ( + len(raw) > self.max_source_bytes + or identities[0] != identities[1] + or identities[0] != identities[2] + or len(raw) != after.st_size + ): + raise DocForgeError( + "stale_adapter_source", + "Script source changed while being read", + source=relative, + ) + try: + text = raw.decode("utf-8-sig") + except UnicodeDecodeError as error: + raise DocForgeError( + "invalid_script_source", + "Script source must use UTF-8", + source=relative, + ) from error + language = self._language_for_suffix(path.suffix) + parse_raw = text.encode("utf-8") + root: _TreeNode | None = None + if parse: + parse_raw, root = self._parse(text, language) + if root.has_error: + raise DocForgeError( + "invalid_script_source", + f"{language.title()} source cannot be parsed by the reference adapter", + source=relative, + line=self._first_error_line(root) or 1, + ) + return _SourceRecord( + source_id=self._source_id(relative), + source_path=relative, + module_name=self._module_name(path, PurePosixPath(source_root)), + language=language, + fingerprint=hashlib.sha256(raw).hexdigest(), + text=text, + raw=parse_raw, + root=root, + ) + + @staticmethod + def _parse(source: str, language: str) -> tuple[bytes, _TreeNode]: + try: + tree_sitter = importlib.import_module("tree_sitter") + except ModuleNotFoundError as error: + raise DocForgeError( + "optional_dependency_missing", + "The requested language frontend is not installed", + extra=language, + install=f"docforge[{language}]", + missing_module=error.name, + ) from error + raw = source.encode("utf-8") + parser = tree_sitter.Parser(_language(language)) + tree = parser.parse(raw) + return raw, cast("_TreeNode", tree.root_node) + + @classmethod + def _first_error_line(cls, node: _TreeNode) -> int | None: + if node.is_error or node.is_missing: + return node.start_point.row + 1 + for child in node.named_children: + line = cls._first_error_line(child) + if line is not None: + return line + return None + + def _source_root_for(self, path: PurePosixPath) -> str: + matches = [ + source_root + for source_root in self.source_roots + if path == PurePosixPath(source_root) or PurePosixPath(source_root) in path.parents + ] + if len(matches) != 1: + raise DocForgeError( + "path_escape", + "Script source is outside the declared source roots", + source=path.as_posix(), + ) + return matches[0] + + @staticmethod + def _language_for_suffix(suffix: str) -> str: + if suffix in _JAVASCRIPT_SUFFIXES: + return "javascript" + if suffix in _TYPESCRIPT_SUFFIXES: + return "typescript" + raise DocForgeError("invalid_adapter", "Unsupported script source suffix") + + @staticmethod + def _module_name(path: PurePosixPath, source_root: PurePosixPath) -> str: + local = path.relative_to(source_root) + parts = list(local.with_suffix("").parts) + if parts and parts[-1] == "index": + parts.pop() + if not parts: + parts = list(source_root.parts) + return ".".join(parts) + + @staticmethod + def _source_id(relative: str) -> str: + digest = hashlib.sha256( + f"{JAVASCRIPT_IDENTITY_VERSION}\0source\0{relative}".encode() + ).hexdigest()[:24] + return f"javascript.source.{digest}" + + @staticmethod + def _file_node_id(source_id: str) -> str: + return source_id.replace("javascript.source.", "javascript.file.", 1) + + @staticmethod + def _module_node_id(source_id: str) -> str: + return source_id.replace("javascript.source.", "javascript.module.", 1) + + @staticmethod + def _symbol_node_id(kind: str, source_id: str, qualified_name: str) -> str: + digest = hashlib.sha256( + (f"{JAVASCRIPT_IDENTITY_VERSION}\0{kind}\0{source_id}\0{qualified_name}").encode() + ).hexdigest()[:24] + return f"javascript.{kind}.{digest}" + + @staticmethod + def _source_hash(sources: Sequence[AdapterSource]) -> str: + digest = hashlib.sha256() + digest.update(JAVASCRIPT_ADAPTER_VERSION.encode()) + digest.update(JAVASCRIPT_EXTRACTOR_VERSION.encode()) + digest.update(JAVASCRIPT_IDENTITY_VERSION.encode()) + for source in sources: + digest.update(b"\0source\0") + digest.update(source.source_id.encode()) + digest.update(b"\0path\0") + digest.update(source.source_path.encode()) + digest.update(b"\0fingerprint\0") + digest.update(source.fingerprint.encode()) + digest.update(b"\0extractor\0") + digest.update(source.extractor_version.encode()) + for dependency in source.dependencies: + digest.update(b"\0dependency\0") + digest.update(dependency.encode()) + return digest.hexdigest() + + def _local_dependencies( + self, + record: _SourceRecord, + path_sources: dict[str, str], + ) -> tuple[str, ...]: + dependencies: set[str] = set() + for specifier in self._static_specifiers(record): + target_path = self._resolve_relative_specifier( + record.source_path, specifier, path_sources + ) + if target_path is None: + continue + target = path_sources[target_path] + if target != record.source_id: + dependencies.add(target) + return tuple(sorted(dependencies)) + + @classmethod + def _static_specifiers(cls, record: _SourceRecord) -> tuple[str, ...]: + result = { + value + for value in cls._lexical_static_specifiers(record.text) + if value.startswith(("./", "../")) + } + return tuple(sorted(result)) + + @staticmethod + def _script_tokens(source: str) -> tuple[tuple[str, str], ...]: + """Tokenize only enough JavaScript syntax to inventory static module specifiers.""" + + tokens: list[tuple[str, str]] = [] + position = 0 + length = len(source) + while position < length: + character = source[position] + if character in " \t\f\v": + position += 1 + continue + if character in "\r\n": + if character == "\r" and position + 1 < length and source[position + 1] == "\n": + position += 1 + tokens.append(("newline", "\n")) + position += 1 + continue + if source.startswith("//", position): + position += 2 + while position < length and source[position] not in "\r\n": + position += 1 + continue + if source.startswith("/*", position): + position += 2 + while position < length and not source.startswith("*/", position): + if source[position] in "\r\n": + tokens.append(("newline", "\n")) + if ( + source[position] == "\r" + and position + 1 < length + and source[position + 1] == "\n" + ): + position += 1 + position += 1 + position = min(length, position + 2) + continue + if character in {'"', "'"}: + quote = character + position += 1 + value: list[str] = [] + escaped = False + closed = False + while position < length: + current = source[position] + if current == "\\": + escaped = True + position += 2 + continue + if current == quote: + position += 1 + closed = True + break + if current in "\r\n": + break + value.append(current) + position += 1 + tokens.append(("string" if closed and not escaped else "opaque", "".join(value))) + continue + if character == "`": + position += 1 + while position < length: + current = source[position] + if current == "\\": + position += 2 + continue + position += 1 + if current == "`": + break + tokens.append(("opaque", "template")) + continue + if character.isalpha() or character in {"_", "$"}: + start = position + position += 1 + while position < length and ( + source[position].isalnum() or source[position] in {"_", "$"} + ): + position += 1 + tokens.append(("name", source[start:position])) + continue + tokens.append(("punct", character)) + position += 1 + return tuple(tokens) + + @classmethod + def _lexical_static_specifiers(cls, source: str) -> tuple[str, ...]: + tokens = cls._script_tokens(source) + result: set[str] = set() + for position, (kind, value) in enumerate(tokens): + if kind != "name" or value not in {"import", "export"}: + continue + if position and tokens[position - 1] == ("punct", "."): + continue + keyword = value + index = position + 1 + depth = 0 + clause_seen = False + line_break = False + while index < len(tokens): + token_kind, token_value = tokens[index] + if token_kind == "newline" and depth == 0: + line_break = True + if keyword == "export" and not clause_seen: + break + index += 1 + continue + if token_kind == "punct": + if token_value in "([{": + depth += 1 + clause_seen = True + elif token_value in ")]}": + depth = max(0, depth - 1) + clause_seen = True + elif token_value == ";" and depth == 0: + break + elif token_value in {"*", ","}: + clause_seen = True + elif token_kind == "string": + if keyword == "import" and not clause_seen: + result.add(token_value) + break + elif token_kind == "name": + if line_break and token_value in { + "class", + "const", + "export", + "function", + "import", + "let", + "return", + "throw", + "var", + }: + break + if keyword == "export" and token_value in { + "async", + "class", + "const", + "default", + "function", + "let", + "var", + }: + break + if token_value == "from" and clause_seen: + source_index = index + 1 + while source_index < len(tokens) and tokens[source_index][0] == "newline": + source_index += 1 + if source_index < len(tokens) and tokens[source_index][0] == "string": + result.add(tokens[source_index][1]) + break + if keyword == "import" and token_value not in {"type"}: + clause_seen = True + if token_kind != "newline": + line_break = False + index += 1 + return tuple(sorted(result)) + + @staticmethod + def _resolve_relative_specifier( + source_path: str, + specifier: str, + path_sources: dict[str, str], + ) -> str | None: + parent = PurePosixPath(source_path).parent.as_posix() + normalized = posixpath.normpath(posixpath.join(parent, specifier)) + if normalized == ".." or normalized.startswith("../") or normalized.startswith("/"): + return None + candidate = PurePosixPath(normalized) + if candidate.suffix in _SOURCE_SUFFIXES: + return normalized if normalized in path_sources else None + candidates = [ + *(f"{normalized}{suffix}" for suffix in _SOURCE_SUFFIXES), + *(f"{normalized}/index{suffix}" for suffix in _SOURCE_SUFFIXES), + ] + matches = [value for value in candidates if value in path_sources] + return matches[0] if len(matches) == 1 else None + + def _extract_record( + self, + record: _SourceRecord, + source: AdapterSource, + ) -> AdapterSourceProjection: + if record.root is None: + raise DocForgeError( + "invalid_adapter", + "Script extraction requires a parsed source record", + source=record.source_path, + ) + file_id = self._file_node_id(source.source_id) + module_id = self._module_node_id(source.source_id) + nodes: list[AdapterNode] = [ + self._node( + node_id=file_id, + title=record.source_path, + kind="file", + qualified_name=record.source_path, + language=record.language, + content=record.text, + source_path=record.source_path, + anchor="L1", + ), + self._node( + node_id=module_id, + title=record.module_name, + kind="module", + qualified_name=record.module_name, + language=record.language, + content=f"{record.language.title()} module {record.module_name}.", + source_path=record.source_path, + anchor="L1", + ), + ] + edges: list[AdapterEdge] = [self._edge(file_id, "contains", module_id, "syntax")] + definitions = self._definitions(record) + definition_ids = { + definition.qualified_name: definition.node_id for definition in definitions + } + for definition in definitions: + parent_id = ( + definition_ids.get(definition.parent_qualified_name, module_id) + if definition.parent_qualified_name is not None + else module_id + ) + anchor = f"L{definition.line}" + content = self._text(definition.node, record.raw).strip() + nodes.append( + self._node( + node_id=definition.node_id, + title=definition.qualified_name, + kind=definition.kind, + qualified_name=f"{record.module_name}.{definition.qualified_name}", + language=record.language, + content=content, + source_path=record.source_path, + anchor=anchor, + ) + ) + edges.append( + self._edge( + parent_id, + "contains", + definition.node_id, + "tree_sitter_syntax", + anchor=anchor, + ) + ) + for dependency in source.dependencies: + edges.append( + self._edge( + module_id, + "depends_on", + self._module_node_id(dependency), + "static_relative_module", + ) + ) + functions = self._discovered_functions(record) + owners = tuple( + TreeSitterLogicOwner( + owner_node_id=self._function_definition(definitions, function).node_id, + qualified_name=function.qualified_name, + line=function.line, + ) + for function in functions + ) + analyzer = ( + analyze_javascript_source + if record.language == "javascript" + else analyze_typescript_source + ) + logic = analyzer( + record.text, + source_id=source.source_id, + owners=owners, + filename=record.source_path, + max_nodes_per_function=self.max_logic_nodes_per_function, + ) + return AdapterSourceProjection( + source_id=source.source_id, + fingerprint=source.fingerprint, + nodes=tuple(sorted(nodes, key=lambda item: item.node.node_id)), + edges=tuple( + sorted( + edges, + key=lambda item: ( + item.edge.source_id, + item.edge.relation, + item.edge.target_id, + ), + ) + ), + logic=logic, + ) + + def _definitions(self, record: _SourceRecord) -> tuple[_Definition, ...]: + root = record.root + if root is None: + raise DocForgeError( + "invalid_adapter", + "Script definition extraction requires a parsed source record", + source=record.source_path, + ) + definitions: list[_Definition] = [] + seen: set[str] = set() + + def visit(node: _TreeNode, scopes: tuple[str, ...]) -> None: + definition = self._definition(record, node, scopes) + next_scopes = scopes + if definition is not None: + if definition.qualified_name in seen: + raise DocForgeError( + "ambiguous_script_symbol", + "Script source repeats a class or function identity", + source=record.source_path, + qualified_name=definition.qualified_name, + ) + seen.add(definition.qualified_name) + definitions.append(definition) + next_scopes = (*scopes, definition.qualified_name.rsplit(".", 1)[-1]) + for child in node.named_children: + visit(child, next_scopes) + + visit(root, ()) + ordered = tuple(sorted(definitions, key=lambda item: item.node_id)) + public_functions = { + (function.qualified_name, function.line) + for function in self._discovered_functions(record) + } + extracted_functions = { + (definition.qualified_name, definition.line) + for definition in ordered + if definition.kind == "function" + } + if extracted_functions != public_functions: + raise DocForgeError( + "invalid_adapter", + "Reference function discovery drifted from the Logic frontend", + source=record.source_path, + ) + return ordered + + def _definition( + self, + record: _SourceRecord, + node: _TreeNode, + scopes: tuple[str, ...], + ) -> _Definition | None: + kind: str | None = None + name: str | None = None + definition_node = node + if node.type in {"class_declaration", "class"}: + name = self._field_text(node, "name", record.raw) + kind = "class" if name is not None else None + elif node.type in _FUNCTION_TYPES: + name = self._field_text(node, "name", record.raw) + kind = "function" if name is not None else None + elif node.type == "variable_declarator": + value = node.child_by_field_name("value") + if value is not None and value.type in {"arrow_function", "function_expression"}: + name = self._field_text(node, "name", record.raw) + definition_node = value + kind = "function" if name is not None else None + if kind is None or name is None: + return None + qualified_name = ".".join((*scopes, name)) + parent = ".".join(scopes) or None + return _Definition( + kind=kind, + qualified_name=qualified_name, + line=node.start_point.row + 1, + parent_qualified_name=parent, + node=definition_node, + node_id=self._symbol_node_id(kind, record.source_id, qualified_name), + ) + + @staticmethod + def _discovered_functions(record: _SourceRecord) -> tuple[DiscoveredFunction, ...]: + if record.language == "javascript": + return discover_javascript_functions(record.text) + return discover_typescript_functions(record.text) + + @staticmethod + def _function_definition( + definitions: tuple[_Definition, ...], + function: DiscoveredFunction, + ) -> _Definition: + matches = [ + definition + for definition in definitions + if definition.kind == "function" + and definition.qualified_name == function.qualified_name + and definition.line == function.line + ] + if len(matches) != 1: + raise DocForgeError( + "invalid_adapter", + "Logic owner does not match one extracted script function", + qualified_name=function.qualified_name, + line=function.line, + ) + return matches[0] + + @staticmethod + def _text(node: _TreeNode, raw: bytes) -> str: + return raw[node.start_byte : node.end_byte].decode("utf-8") + + @classmethod + def _field_text(cls, node: _TreeNode, field: str, raw: bytes) -> str | None: + child = node.child_by_field_name(field) + return cls._text(child, raw) if child is not None else None + + @staticmethod + def _node( + *, + node_id: str, + title: str, + kind: str, + qualified_name: str, + language: str, + content: str, + source_path: str, + anchor: str, + ) -> AdapterNode: + normalized = content.strip() or f"{language.title()} {kind} {qualified_name}." + return AdapterNode( + node=GraphNode( + node_id=node_id, + title=title, + family="code", + authority="derived", + status="active", + tags=tuple(sorted({language, kind})), + summary=f"{language.title()} {kind} fact for {qualified_name}.", + content=normalized, + source_path=source_path, + source_anchor=anchor, + content_hash=hashlib.sha256(normalized.encode()).hexdigest(), + ), + metadata=( + ("extractor", JAVASCRIPT_EXTRACTOR_VERSION), + ("identity", JAVASCRIPT_IDENTITY_VERSION), + ("kind", kind), + ("language", language), + ("qualified_name", qualified_name), + ), + ) + + @staticmethod + def _edge( + source_id: str, + relation: str, + target_id: str, + evidence: str, + *, + anchor: str | None = None, + ) -> AdapterEdge: + metadata = [("evidence", evidence)] + if anchor is not None: + metadata.append(("source_anchor", anchor)) + return AdapterEdge( + edge=Edge(source_id, relation, target_id), + metadata=tuple(metadata), + ) diff --git a/src/docforge/adapters/python.py b/src/docforge/adapters/python.py new file mode 100644 index 0000000..6a754d6 --- /dev/null +++ b/src/docforge/adapters/python.py @@ -0,0 +1,948 @@ +"""Deterministic stdlib-AST reference adapter for explicitly confined Python roots. + +The adapter parses Python source as untrusted data. It never imports or executes +project code. Its intentionally narrow dependency model publishes only imports +that resolve to another module in the same declared source inventory. +""" + +from __future__ import annotations + +import ast +import hashlib +import io +import os +import stat +import tokenize +from collections.abc import Iterable, Sequence +from dataclasses import dataclass +from pathlib import Path, PurePosixPath + +from ..adapter_sdk import ( + AdapterAssembly, + AdapterEdge, + AdapterManifest, + AdapterNode, + AdapterProjection, + AdapterSource, + AdapterSourceProjection, + Edge, + Node, +) +from ..errors import DocForgeError +from ..python_logic import PythonLogicOwner, analyze_python_source + +PYTHON_ADAPTER_ID = "docforge.reference.python" +PYTHON_ADAPTER_VERSION = "1" +PYTHON_EXTRACTOR_VERSION = "stdlib-ast@1" +PYTHON_IDENTITY_VERSION = "python-reference-id@1" +PYTHON_SUPPORT_SCHEMA_VERSION = 1 + +_SUPPORTED_FACTS = ( + "python_file", + "python_module", + "python_class", + "python_function", + "lexical_containment", + "local_import_dependency", + "function_logic", +) + + +@dataclass(frozen=True) +class PythonUnsupportedFact: + """One semantic fact this syntax-only reference adapter does not claim.""" + + code: str + description: str + + def as_dict(self) -> dict[str, str]: + return {"code": self.code, "description": self.description} + + +_UNSUPPORTED_FACTS = ( + PythonUnsupportedFact( + "call_resolution", + "Calls are represented only inside function Logic and are not resolved to symbols.", + ), + PythonUnsupportedFact( + "dynamic_import_resolution", + "Imports performed through runtime calls are not dependency evidence.", + ), + PythonUnsupportedFact( + "inheritance_resolution", + "Class bases are syntax only and are not resolved to local or external types.", + ), + PythonUnsupportedFact( + "runtime_generated_facts", + "Decorators, metaclasses, descriptors, and executed module code are never evaluated.", + ), + PythonUnsupportedFact( + "symbol_reference_resolution", + "Imported names, variable references, types, overloads, and re-exports are not resolved.", + ), +) + + +@dataclass(frozen=True) +class _SourceRecord: + source_id: str + source_path: str + module_name: str + fingerprint: str + raw: bytes + text: str + tree: ast.Module | None + dependencies: tuple[str, ...] = () + + +@dataclass(frozen=True) +class _Definition: + kind: str + qualified_name: str + parent_node_id: str + node: ast.ClassDef | ast.FunctionDef | ast.AsyncFunctionDef + node_id: str + + +class PythonReferenceAdapter: + """Reference Python adapter over explicit, non-overlapping source roots.""" + + def __init__( + self, + root: Path, + *, + source_roots: Iterable[str | Path], + project_id: str = "python-reference", + title: str = "Python reference project", + max_sources: int = 4_096, + max_source_bytes: int = 1_000_000, + max_logic_nodes_per_function: int = 2_000, + ) -> None: + resolved_root = root.resolve(strict=True) + if not resolved_root.is_dir(): + raise DocForgeError("invalid_adapter", "Python adapter root must be a directory") + if max_sources < 1 or max_source_bytes < 1 or max_logic_nodes_per_function < 2: + raise ValueError("Python adapter limits must be positive") + self.root = resolved_root + self.project_id = project_id + self.title = title + self.max_sources = max_sources + self.max_source_bytes = max_source_bytes + self.max_logic_nodes_per_function = max_logic_nodes_per_function + self.source_roots = self._normalize_source_roots(source_roots) + + def support_report(self) -> dict[str, object]: + """Return deterministic, machine-readable scope and limitation evidence.""" + + return { + "schema_version": PYTHON_SUPPORT_SCHEMA_VERSION, + "adapter_id": PYTHON_ADAPTER_ID, + "adapter_version": PYTHON_ADAPTER_VERSION, + "extractor_version": PYTHON_EXTRACTOR_VERSION, + "identity_version": PYTHON_IDENTITY_VERSION, + "frontend": "python-stdlib-ast", + "imports_project_code": False, + "executes_project_code": False, + "source_roots": list(self.source_roots), + "supported_facts": list(_SUPPORTED_FACTS), + "unsupported_facts": [fact.as_dict() for fact in _UNSUPPORTED_FACTS], + } + + def unsupported_facts(self) -> tuple[PythonUnsupportedFact, ...]: + """Return the adapter's fixed unsupported semantic fact inventory.""" + + return _UNSUPPORTED_FACTS + + def load_manifest(self) -> AdapterManifest: + records = self._inventory(parse=False) + module_sources = {record.module_name: record.source_id for record in records} + if len(module_sources) != len(records): + raise DocForgeError( + "ambiguous_python_module", + "Declared Python source roots produce duplicate module names", + ) + sources = tuple( + sorted( + ( + AdapterSource( + source_id=record.source_id, + source_path=record.source_path, + fingerprint=record.fingerprint, + extractor_version=PYTHON_EXTRACTOR_VERSION, + dependencies=self._local_dependencies(record, module_sources), + ) + for record in records + ), + key=lambda source: source.source_id, + ) + ) + source_hash = self._source_hash(sources) + return AdapterManifest( + project_id=self.project_id, + title=self.title, + adapter_id=PYTHON_ADAPTER_ID, + adapter_version=PYTHON_ADAPTER_VERSION, + root=self.root, + revision=source_hash[:12], + source_hash=source_hash, + families=("code",), + allowed_relations=("contains", "depends_on"), + sources=sources, + estimated_nodes=max(1, len(sources) * 8), + ) + + def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: + record = self._read_source(source.source_path) + if record.source_id != source.source_id or record.fingerprint != source.fingerprint: + raise DocForgeError( + "stale_adapter_source", + "Python source changed after its manifest was captured", + source=source.source_path, + ) + return self._extract_record(record, source) + + def assemble_projection( + self, + manifest: AdapterManifest, + contributions: tuple[AdapterSourceProjection, ...], + ) -> AdapterAssembly: + expected = tuple(source.source_id for source in manifest.sources) + actual = tuple(sorted(contribution.source_id for contribution in contributions)) + if actual != expected: + raise DocForgeError( + "invalid_adapter", + "Python assembly contributions do not match the current manifest", + ) + nodes = tuple( + sorted( + (node for contribution in contributions for node in contribution.nodes), + key=lambda item: item.node.node_id, + ) + ) + edges = tuple( + sorted( + (edge for contribution in contributions for edge in contribution.edges), + key=lambda item: ( + item.edge.source_id, + item.edge.relation, + item.edge.target_id, + ), + ) + ) + logic = tuple( + sorted( + (projection for contribution in contributions for projection in contribution.logic), + key=lambda projection: projection.owner_node_id, + ) + ) + return AdapterAssembly( + projection=AdapterProjection( + project_id=manifest.project_id, + title=manifest.title, + adapter_id=manifest.adapter_id, + adapter_version=manifest.adapter_version, + root=manifest.root, + revision=manifest.revision, + source_hash=manifest.source_hash, + nodes=nodes, + edges=edges, + ), + logic=logic, + ) + + def load_assembly(self) -> AdapterAssembly: + """Load one cache-independent complete graph and Logic assembly.""" + + manifest = self.load_manifest() + contributions = tuple(self.extract_source(source) for source in manifest.sources) + stable = self.load_manifest() + if stable != manifest: + raise DocForgeError( + "source_changed", + "Python sources changed during complete adapter extraction", + ) + return self.assemble_projection(manifest, contributions) + + def load_complete_assembly(self) -> AdapterAssembly: + """Implement the SDK's independent complete-assembly oracle.""" + + return self.load_assembly() + + def load_projection(self) -> AdapterProjection: + """Preserve the first-class legacy complete-projection contract.""" + + return self.load_assembly().projection + + def _normalize_source_roots(self, source_roots: Iterable[str | Path]) -> tuple[str, ...]: + normalized: list[str] = [] + for value in source_roots: + candidate = PurePosixPath(Path(value).as_posix()) + if ( + candidate.is_absolute() + or not candidate.parts + or ".." in candidate.parts + or str(candidate) in {"", "."} + ): + raise DocForgeError( + "path_escape", + "Python source roots must be explicit project-relative directories", + ) + relative = candidate.as_posix() + absolute = self.root.joinpath(*candidate.parts) + if absolute.is_symlink(): + raise DocForgeError("path_escape", "Python source root cannot be a symbolic link") + try: + resolved = absolute.resolve(strict=True) + except OSError as error: + raise DocForgeError( + "invalid_adapter", + "Python source root does not exist", + source_root=relative, + ) from error + if ( + resolved != absolute + or not resolved.is_dir() + or not resolved.is_relative_to(self.root) + ): + raise DocForgeError( + "path_escape", + "Python source root must be a confined real directory", + source_root=relative, + ) + normalized.append(relative) + result = tuple(sorted(set(normalized))) + if not result: + raise DocForgeError("invalid_adapter", "At least one Python source root is required") + if len(result) != len(normalized): + raise DocForgeError("invalid_adapter", "Python source roots must be unique") + paths = [PurePosixPath(value) for value in result] + for index, left in enumerate(paths): + for right in paths[index + 1 :]: + if left in right.parents or right in left.parents: + raise DocForgeError( + "invalid_adapter", + "Python source roots must not overlap", + ) + return result + + def _inventory(self, *, parse: bool = True) -> tuple[_SourceRecord, ...]: + paths: list[str] = [] + for source_root in self.source_roots: + absolute_root = self.root.joinpath(*PurePosixPath(source_root).parts) + if ( + absolute_root.is_symlink() + or not absolute_root.is_dir() + or absolute_root.resolve(strict=True) != absolute_root + ): + raise DocForgeError( + "path_escape", + "Python source root changed after adapter binding", + source_root=source_root, + ) + for directory, directory_names, file_names in os.walk( + absolute_root, + topdown=True, + followlinks=False, + ): + current = Path(directory) + for name in tuple(directory_names): + child = current / name + if child.is_symlink(): + raise DocForgeError( + "path_escape", + "Python source inventory contains a symbolic-link directory", + source=child.relative_to(self.root).as_posix(), + ) + directory_names[:] = sorted( + name for name in directory_names if name != "__pycache__" + ) + for name in sorted(file_names): + if not name.endswith(".py"): + continue + child = current / name + relative = child.relative_to(self.root).as_posix() + if child.is_symlink(): + raise DocForgeError( + "path_escape", + "Python source inventory contains a symbolic-link file", + source=relative, + ) + paths.append(relative) + if len(paths) > self.max_sources: + raise DocForgeError( + "adapter_too_large", + "Python source inventory exceeds the configured limit", + maximum=self.max_sources, + ) + records = tuple(self._read_source(path, parse=parse) for path in sorted(paths)) + modules = [record.module_name for record in records] + if len(modules) != len(set(modules)): + duplicates = sorted(module for module in set(modules) if modules.count(module) > 1) + raise DocForgeError( + "ambiguous_python_module", + "Declared Python source roots produce duplicate module names", + modules=duplicates, + ) + return records + + def _read_source(self, relative: str, *, parse: bool = True) -> _SourceRecord: + path = PurePosixPath(relative) + if path.is_absolute() or ".." in path.parts or path.suffix != ".py": + raise DocForgeError("path_escape", "Python source path is unsafe") + source_root = self._source_root_for(path) + candidate = self.root.joinpath(*path.parts) + try: + before = candidate.lstat() + except OSError as error: + raise DocForgeError( + "stale_adapter_source", + "Python source is unavailable", + source=relative, + ) from error + if ( + stat.S_ISLNK(before.st_mode) + or not stat.S_ISREG(before.st_mode) + or candidate.resolve(strict=True) != candidate + or not candidate.is_relative_to(self.root) + ): + raise DocForgeError( + "path_escape", + "Python source must be a confined regular file", + source=relative, + ) + if before.st_size > self.max_source_bytes: + raise DocForgeError( + "source_too_large", + "Python source exceeds the configured adapter limit", + source=relative, + maximum=self.max_source_bytes, + ) + descriptor = -1 + try: + flags = os.O_RDONLY | getattr(os, "O_BINARY", 0) + flags |= getattr(os, "O_NOFOLLOW", 0) + descriptor = os.open(candidate, flags) + opened = os.fstat(descriptor) + if not stat.S_ISREG(opened.st_mode) or (opened.st_dev, opened.st_ino) != ( + before.st_dev, + before.st_ino, + ): + raise DocForgeError( + "path_escape", + "Python source identity changed before it was opened", + source=relative, + ) + with os.fdopen(descriptor, "rb") as handle: + descriptor = -1 + raw = handle.read(self.max_source_bytes + 1) + after_open = os.fstat(handle.fileno()) + after = candidate.lstat() + except DocForgeError: + raise + except OSError as error: + raise DocForgeError( + "stale_adapter_source", + "Python source changed while being read", + source=relative, + ) from error + finally: + if descriptor >= 0: + os.close(descriptor) + identity_before = ( + before.st_dev, + before.st_ino, + before.st_size, + before.st_mtime_ns, + ) + identity_after = ( + after.st_dev, + after.st_ino, + after.st_size, + after.st_mtime_ns, + ) + identity_opened = ( + after_open.st_dev, + after_open.st_ino, + after_open.st_size, + after_open.st_mtime_ns, + ) + if ( + len(raw) > self.max_source_bytes + or identity_before != identity_opened + or identity_before != identity_after + or len(raw) != after.st_size + ): + raise DocForgeError( + "stale_adapter_source", + "Python source changed while being read", + source=relative, + ) + text = self._decode_source(raw, relative) + tree: ast.Module | None = None + if parse: + try: + tree = ast.parse(text, filename=relative, type_comments=True) + except SyntaxError as error: + raise DocForgeError( + "invalid_python_source", + "Python source cannot be parsed by the reference adapter", + source=relative, + line=error.lineno, + ) from error + module_name = self._module_name(path, PurePosixPath(source_root)) + return _SourceRecord( + source_id=self._source_id(relative), + source_path=relative, + module_name=module_name, + fingerprint=hashlib.sha256(raw).hexdigest(), + raw=raw, + text=text, + tree=tree, + ) + + def _source_root_for(self, path: PurePosixPath) -> str: + matches = [ + source_root + for source_root in self.source_roots + if path == PurePosixPath(source_root) or PurePosixPath(source_root) in path.parents + ] + if len(matches) != 1: + raise DocForgeError( + "path_escape", + "Python source is outside the declared source roots", + source=path.as_posix(), + ) + return matches[0] + + @staticmethod + def _decode_source(raw: bytes, source_path: str) -> str: + try: + encoding, _ = tokenize.detect_encoding(io.BytesIO(raw).readline) + return raw.decode(encoding) + except (LookupError, SyntaxError, UnicodeDecodeError) as error: + raise DocForgeError( + "invalid_python_source", + "Python source encoding is invalid", + source=source_path, + ) from error + + @staticmethod + def _module_name(path: PurePosixPath, source_root: PurePosixPath) -> str: + local = path.relative_to(source_root) + parts = list(local.with_suffix("").parts) + if parts and parts[-1] == "__init__": + parts.pop() + if not parts: + parts = list(source_root.parts) + return ".".join(parts) + + @staticmethod + def _source_id(relative: str) -> str: + digest = hashlib.sha256( + f"{PYTHON_IDENTITY_VERSION}\0source\0{relative}".encode() + ).hexdigest()[:24] + return f"python.source.{digest}" + + @staticmethod + def _file_node_id(source_id: str) -> str: + return source_id.replace("python.source.", "python.file.", 1) + + @staticmethod + def _module_node_id(source_id: str) -> str: + return source_id.replace("python.source.", "python.module.", 1) + + @staticmethod + def _symbol_node_id(kind: str, source_id: str, qualified_name: str) -> str: + digest = hashlib.sha256( + (f"{PYTHON_IDENTITY_VERSION}\0{kind}\0{source_id}\0{qualified_name}").encode() + ).hexdigest()[:24] + return f"python.{kind}.{digest}" + + @staticmethod + def _source_hash(sources: Sequence[AdapterSource]) -> str: + digest = hashlib.sha256() + digest.update(PYTHON_ADAPTER_VERSION.encode()) + digest.update(PYTHON_EXTRACTOR_VERSION.encode()) + digest.update(PYTHON_IDENTITY_VERSION.encode()) + for source in sources: + digest.update(b"\0source\0") + digest.update(source.source_id.encode()) + digest.update(b"\0path\0") + digest.update(source.source_path.encode()) + digest.update(b"\0fingerprint\0") + digest.update(source.fingerprint.encode()) + for dependency in source.dependencies: + digest.update(b"\0dependency\0") + digest.update(dependency.encode()) + return digest.hexdigest() + + def _local_dependencies( + self, + record: _SourceRecord, + module_sources: dict[str, str], + ) -> tuple[str, ...]: + dependencies: set[str] = set() + package = ( + record.module_name + if record.source_path.endswith("/__init__.py") + else record.module_name.rpartition(".")[0] + ) + for level, module, imported_names in self._lexical_imports(record): + candidates: list[str] = [] + if not imported_names and level == 0 and module is not None: + candidates.append(module) + else: + base = self._import_from_base(package, level, module) + if base: + candidates.append(base) + candidates.extend(f"{base}.{name}" for name in imported_names if name != "*") + for module_name in candidates: + target = module_sources.get(module_name) + if target is not None and target != record.source_id: + dependencies.add(target) + return tuple(sorted(dependencies)) + + @staticmethod + def _lexical_imports( + record: _SourceRecord, + ) -> tuple[tuple[int, str | None, tuple[str, ...]], ...]: + """Inventory import dependencies without constructing a Python AST.""" + + try: + tokens = tuple(tokenize.generate_tokens(io.StringIO(record.text).readline)) + except (IndentationError, tokenize.TokenError) as error: + line = error.args[1][0] if len(error.args) > 1 else None + raise DocForgeError( + "invalid_python_source", + "Python source cannot be tokenized by the reference adapter", + source=record.source_path, + line=line, + ) from error + + statements: list[list[tokenize.TokenInfo]] = [] + current: list[tokenize.TokenInfo] = [] + nesting = 0 + ignored = { + tokenize.ENCODING, + tokenize.INDENT, + tokenize.DEDENT, + tokenize.NL, + tokenize.COMMENT, + } + for token in tokens: + if token.type in ignored: + continue + if token.type == tokenize.OP: + if token.string in "([{": + nesting += 1 + elif token.string in ")]}": + nesting = max(0, nesting - 1) + elif token.string == ";" and nesting == 0: + if current: + statements.append(current) + current = [] + continue + if token.type in {tokenize.NEWLINE, tokenize.ENDMARKER} and nesting == 0: + if current: + statements.append(current) + current = [] + continue + current.append(token) + + imports: list[tuple[int, str | None, tuple[str, ...]]] = [] + for statement in statements: + for position, token in enumerate(statement): + if token.type != tokenize.NAME or token.string not in {"import", "from"}: + continue + if token.string == "import": + names = PythonReferenceAdapter._imported_names(statement[position + 1 :]) + imports.extend((0, name, ()) for name in names) + break + import_position = next( + ( + index + for index in range(position + 1, len(statement)) + if statement[index].type == tokenize.NAME + and statement[index].string == "import" + ), + None, + ) + if import_position is None: + break + prefix = statement[position + 1 : import_position] + level = 0 + for item in prefix: + if item.type == tokenize.NAME: + break + if item.type == tokenize.OP and set(item.string) == {"."}: + level += len(item.string) + module_parts = [item.string for item in prefix if item.type == tokenize.NAME] + module = ".".join(module_parts) or None + names = PythonReferenceAdapter._imported_names(statement[import_position + 1 :]) + imports.append((level, module, names)) + break + return tuple(imports) + + @staticmethod + def _imported_names(tokens: Sequence[tokenize.TokenInfo]) -> tuple[str, ...]: + names: list[str] = [] + current: list[str] = [] + skip_alias = False + for token in tokens: + if token.type == tokenize.NAME and token.string == "as": + skip_alias = True + continue + if token.type == tokenize.OP and token.string == ",": + if current: + names.append(".".join(current)) + current = [] + skip_alias = False + continue + if token.type == tokenize.OP and token.string == "*": + if not skip_alias: + current.append("*") + continue + if token.type == tokenize.NAME and not skip_alias: + current.append(token.string) + if current: + names.append(".".join(current)) + return tuple(name for name in names if name) + + @staticmethod + def _import_from_base(package: str, level: int, module: str | None) -> str: + if level == 0: + return module or "" + package_parts = package.split(".") if package else [] + keep = len(package_parts) - (level - 1) + if keep < 0: + return "" + prefix = package_parts[:keep] + if module: + prefix.extend(module.split(".")) + return ".".join(prefix) + + def _extract_record( + self, + record: _SourceRecord, + source: AdapterSource, + ) -> AdapterSourceProjection: + tree = record.tree + if tree is None: + raise DocForgeError( + "invalid_adapter", + "Python extraction requires a parsed source record", + source=record.source_path, + ) + file_id = self._file_node_id(source.source_id) + module_id = self._module_node_id(source.source_id) + nodes: list[AdapterNode] = [ + self._node( + node_id=file_id, + title=record.source_path, + kind="file", + qualified_name=record.source_path, + content=record.text, + source_path=record.source_path, + anchor="L1", + ), + self._node( + node_id=module_id, + title=record.module_name, + kind="module", + qualified_name=record.module_name, + content=( + ast.get_docstring(tree, clean=False) or f"Python module {record.module_name}." + ), + source_path=record.source_path, + anchor="L1", + ), + ] + edges: list[AdapterEdge] = [self._edge(file_id, "contains", module_id, "syntax")] + definitions = self._definitions(record, module_id) + owners: list[PythonLogicOwner] = [] + for definition in definitions: + content = ast.get_source_segment(record.text, definition.node) + if content is None or not content.strip(): + content = f"Python {definition.kind} {definition.qualified_name}." + anchor = f"L{definition.node.lineno}" + nodes.append( + self._node( + node_id=definition.node_id, + title=definition.qualified_name, + kind=definition.kind, + qualified_name=f"{record.module_name}.{definition.qualified_name}", + content=content, + source_path=record.source_path, + anchor=anchor, + asynchronous=isinstance(definition.node, ast.AsyncFunctionDef), + ) + ) + edges.append( + self._edge( + definition.parent_node_id, + "contains", + definition.node_id, + "syntax", + anchor=anchor, + ) + ) + if definition.kind == "function": + owners.append( + PythonLogicOwner( + owner_node_id=definition.node_id, + qualified_name=definition.qualified_name, + line=definition.node.lineno, + ) + ) + for dependency in source.dependencies: + edges.append( + self._edge( + module_id, + "depends_on", + self._module_node_id(dependency), + "local_import", + ) + ) + logic = analyze_python_source( + record.text, + source_id=source.source_id, + owners=owners, + filename=record.source_path, + max_nodes_per_function=self.max_logic_nodes_per_function, + ) + return AdapterSourceProjection( + source_id=source.source_id, + fingerprint=source.fingerprint, + nodes=tuple(sorted(nodes, key=lambda item: item.node.node_id)), + edges=tuple( + sorted( + edges, + key=lambda item: ( + item.edge.source_id, + item.edge.relation, + item.edge.target_id, + ), + ) + ), + logic=logic, + ) + + def _definitions( + self, + record: _SourceRecord, + module_node_id: str, + ) -> tuple[_Definition, ...]: + tree = record.tree + if tree is None: + raise DocForgeError( + "invalid_adapter", + "Python definition extraction requires a parsed source record", + source=record.source_path, + ) + definitions: list[_Definition] = [] + qualified_names: set[str] = set() + adapter = self + + class Collector(ast.NodeVisitor): + def __init__(self) -> None: + self.scope_names: list[str] = [] + self.scope_node_ids: list[str] = [module_node_id] + + def _definition( + self, + kind: str, + node: ast.ClassDef | ast.FunctionDef | ast.AsyncFunctionDef, + ) -> None: + qualified_name = ".".join((*self.scope_names, node.name)) + if qualified_name in qualified_names: + raise DocForgeError( + "ambiguous_python_symbol", + "Python source repeats a class or function identity", + source=record.source_path, + qualified_name=qualified_name, + ) + qualified_names.add(qualified_name) + node_id = adapter._symbol_node_id( + kind, + record.source_id, + qualified_name, + ) + definitions.append( + _Definition( + kind=kind, + qualified_name=qualified_name, + parent_node_id=self.scope_node_ids[-1], + node=node, + node_id=node_id, + ) + ) + self.scope_names.append(node.name) + self.scope_node_ids.append(node_id) + self.generic_visit(node) + self.scope_node_ids.pop() + self.scope_names.pop() + + def visit_ClassDef(self, node: ast.ClassDef) -> None: + self._definition("class", node) + + def visit_FunctionDef(self, node: ast.FunctionDef) -> None: + self._definition("function", node) + + def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None: + self._definition("function", node) + + Collector().visit(tree) + return tuple(sorted(definitions, key=lambda item: item.node_id)) + + @staticmethod + def _node( + *, + node_id: str, + title: str, + kind: str, + qualified_name: str, + content: str, + source_path: str, + anchor: str, + asynchronous: bool = False, + ) -> AdapterNode: + normalized = content.strip() or f"Python {kind} {qualified_name}." + tags = tuple(sorted({"python", kind, *(("async",) if asynchronous else ())})) + return AdapterNode( + node=Node( + node_id=node_id, + title=title, + family="code", + authority="derived", + status="active", + tags=tags, + summary=f"Python {kind} fact for {qualified_name}.", + content=normalized, + source_path=source_path, + source_anchor=anchor, + content_hash=hashlib.sha256(normalized.encode()).hexdigest(), + ), + metadata=( + ("extractor", PYTHON_EXTRACTOR_VERSION), + ("identity", PYTHON_IDENTITY_VERSION), + ("kind", kind), + ("qualified_name", qualified_name), + ), + ) + + @staticmethod + def _edge( + source_id: str, + relation: str, + target_id: str, + evidence: str, + *, + anchor: str | None = None, + ) -> AdapterEdge: + metadata = [("evidence", evidence)] + if anchor is not None: + metadata.append(("source_anchor", anchor)) + return AdapterEdge( + edge=Edge(source_id, relation, target_id), + metadata=tuple(metadata), + ) diff --git a/src/docforge/client_config.py b/src/docforge/client_config.py index 18c9de6..d02e493 100644 --- a/src/docforge/client_config.py +++ b/src/docforge/client_config.py @@ -16,6 +16,12 @@ from dataclasses import dataclass from pathlib import Path from typing import Literal, cast +from .adapter_launcher import ( + AdapterLauncherV1, + AdapterSourceAvailabilityV1, + adapter_source_availability, + validate_adapter_launcher, +) from .changeset_contract import document_hash from .errors import DocForgeError from .models import ProjectDescriptor, ProjectService @@ -1079,3 +1085,450 @@ def generate_client_configuration( } _validate_configuration_result(result, trusted_descriptor=descriptor) return result + + +def generate_adapter_client_configuration( + project: ProjectService, + launcher: AdapterLauncherV1, + client: str, + *, + server_name: str | None = None, + capability_mode: str = "read", + proposal_writer: str | None = None, + canonical_applier: str | None = None, + no_ast: bool = False, + manual_render_policy: str | None = None, + portable_graph_policy: str | None = None, + live_viewer_policy: str | None = None, + startup_timeout: int = 30, + tool_timeout: int = 300, + output: Path | None = None, +) -> dict[str, object]: + """Generate one client fragment for an explicitly constructed project-owned adapter.""" + + validate_adapter_launcher(project, launcher) + selected_client = _client_name(client) + selected_mode = _capability_mode(capability_mode) + startup_seconds = _bounded_seconds( + startup_timeout, + field="startup_timeout", + maximum=3_600, + ) + tool_seconds = _bounded_seconds( + tool_timeout, + field="tool_timeout", + maximum=86_400, + ) + descriptor = project.descriptor + _validate_adapter_authority( + descriptor, + selected_mode=selected_mode, + proposal_writer=proposal_writer, + canonical_applier=canonical_applier, + ) + source_availability = adapter_source_availability(project, launcher) + fingerprint = project_root_fingerprint(descriptor.root) + selected_name = _validated_server_name( + server_name, + project_id=descriptor.project_id, + fingerprint=fingerprint, + ) + executable = Path(os.path.abspath(sys.executable)) + try: + executable_status = executable.stat() + except OSError as error: + raise DocForgeError( + "client_configuration_unavailable", + "Current Python executable cannot be inspected", + ) from error + if not stat.S_ISREG(executable_status.st_mode) or not os.access(executable, os.X_OK): + raise DocForgeError( + "client_configuration_unavailable", + "Current Python executable is not a runnable regular file", + ) + + policy = compose_effective_policy( + selected_mode=selected_mode, + capability_source="explicit", + no_ast=no_ast, + diagnostics=False, + render_configured=descriptor.render is not None, + application_enabled=canonical_applier is not None, + ) + projection_policy = compose_projection_policy( + manual=manual_render_policy, + portable_graph=portable_graph_policy, + live_viewer=live_viewer_policy, + manual_configured=descriptor.render is not None, + portable_graph_configured=descriptor.graph_render is not None, + application_enabled=canonical_applier is not None, + ) + default_projection_policy = compose_projection_policy( + manual_configured=descriptor.render is not None, + portable_graph_configured=descriptor.graph_render is not None, + application_enabled=canonical_applier is not None, + ) + arguments = [ + "-I", + "-m", + launcher.module, + "--project-root", + str(descriptor.root), + "--capability-mode", + selected_mode, + ] + if proposal_writer is not None: + arguments.extend(("--proposal-writer", proposal_writer)) + if canonical_applier is not None: + arguments.extend(("--canonical-applier", canonical_applier)) + for option, selected, default in ( + ( + "--manual-render-policy", + projection_policy.manual, + default_projection_policy.manual, + ), + ( + "--portable-graph-policy", + projection_policy.portable_graph, + default_projection_policy.portable_graph, + ), + ( + "--live-viewer-policy", + projection_policy.live_viewer, + default_projection_policy.live_viewer, + ), + ): + if selected != default: + arguments.extend((option, selected)) + if no_ast: + arguments.append("--no-ast") + + artifact_format, content, warning = _artifact( + selected_client, + server_name=selected_name, + command=str(executable), + arguments=arguments, + startup_timeout=startup_seconds, + tool_timeout=tool_seconds, + ) + + def validate_current_binding() -> None: + validate_adapter_launcher(project, launcher) + current = adapter_source_availability(project, launcher) + if current != source_availability: + raise DocForgeError( + "source_changed", + "Adapter sources changed during client configuration generation", + ) + + if output is None: + validate_current_binding() + write_state = "not_requested" + durability = "not_applicable" + publication_warning = None + output_path = None + else: + validate_current_binding() + write_state, durability, publication_warning, published_path = _atomic_write( + output, + content, + validate_binding=validate_current_binding, + ) + output_path = str(published_path) if published_path is not None else None + artifact = { + "format": artifact_format, + "content": content, + "content_sha256": hashlib.sha256(content.encode("utf-8")).hexdigest(), + "output_path": output_path, + "write_state": write_state, + "durability": durability, + } + binding = { + "transport": "stdio", + "capability_mode": selected_mode, + "adapter_policy": policy.adapter_policy(), + "render_policy": { + "manual": policy.manual_render, + "graph": policy.graph_render, + "live_viewer": policy.live_viewer, + }, + "command": str(executable), + "args": arguments, + "environment": {}, + "timeouts": { + "startup_seconds": startup_seconds, + "tool_seconds": tool_seconds, + }, + "launcher_hash": launcher.launcher_hash, + } + project_binding = { + "project_id": descriptor.project_id, + "project_root": str(descriptor.root), + "project_root_fingerprint": fingerprint, + "adapter": descriptor.adapter, + "descriptor_hash": descriptor.descriptor_hash, + } + projection_availability = { + "manual_configured": descriptor.render is not None, + "portable_graph_configured": descriptor.graph_render is not None, + "application_enabled": canonical_applier is not None, + "live_viewer_available": True, + } + policy_payload = policy.as_dict() + result: dict[str, object] = { + "status": "ok", + "schema_version": 1, + "operation": "adapter_client.configure", + "action": "write" if output is not None else "preview", + "client": selected_client, + "server_name": selected_name, + "project": project_binding, + "launcher": launcher.as_dict(), + "launcher_hash": launcher.launcher_hash, + "source_availability": source_availability.as_dict(), + "source_availability_hash": source_availability.availability_hash, + "binding": binding, + "effective_policy": policy_payload, + "projection_policy": projection_policy.as_dict(), + "projection_policy_hash": projection_policy.policy_hash, + "projection_availability": projection_availability, + "artifact": artifact, + "warnings": [ + *([] if warning is None else [{"code": "timeout_format_unverified"}]), + *([] if publication_warning is None else [{"code": publication_warning}]), + ], + } + result["configuration_hash"] = document_hash(_adapter_configuration_hash_payload(result)) + _validate_adapter_configuration_result( + result, + project=project, + launcher=launcher, + source_availability=source_availability, + ) + return result + + +def _validate_adapter_authority( + descriptor: ProjectDescriptor, + *, + selected_mode: CapabilityMode, + proposal_writer: str | None, + canonical_applier: str | None, +) -> None: + writer_ids = {writer.writer_id for writer in descriptor.proposal_writers} + if selected_mode == "read": + if proposal_writer is not None or canonical_applier is not None: + raise DocForgeError( + "invalid_capability_binding", + "Read configuration cannot bind proposal or application authority", + ) + elif selected_mode == "proposal": + if proposal_writer is None or proposal_writer not in writer_ids: + raise DocForgeError( + "capability_unavailable", + "Proposal configuration requires a descriptor-declared writer", + required="proposal_writer", + ) + if canonical_applier is not None: + raise DocForgeError( + "invalid_capability_binding", + "Proposal configuration cannot bind a canonical applier", + ) + elif ( + proposal_writer is None + or canonical_applier is None + or proposal_writer != canonical_applier + or proposal_writer not in writer_ids + ): + raise DocForgeError( + "capability_unavailable", + "Application configuration requires one declared writer/applier identity", + required="matching_declared_writer_and_applier", + ) + + +def _adapter_configuration_hash_payload(result: dict[str, object]) -> dict[str, object]: + artifact = cast(dict[str, object], result["artifact"]) + return { + "schema_version": 1, + "client": result["client"], + "server_name": result["server_name"], + "project": result["project"], + "launcher": result["launcher"], + "launcher_hash": result["launcher_hash"], + "source_availability": result["source_availability"], + "source_availability_hash": result["source_availability_hash"], + "binding": result["binding"], + "effective_policy": result["effective_policy"], + "projection_policy": result["projection_policy"], + "projection_policy_hash": result["projection_policy_hash"], + "projection_availability": result["projection_availability"], + "artifact_format": artifact["format"], + "artifact_content_sha256": artifact["content_sha256"], + } + + +def _validate_adapter_configuration_result( + result: dict[str, object], + *, + project: ProjectService, + launcher: AdapterLauncherV1, + source_availability: AdapterSourceAvailabilityV1, +) -> None: + validate_adapter_launcher(project, launcher) + descriptor = project.descriptor + project_binding = cast(dict[str, object], result["project"]) + if ( + result["launcher"] != launcher.as_dict() + or result["launcher_hash"] != launcher.launcher_hash + or result["source_availability"] != source_availability.as_dict() + or result["source_availability_hash"] != source_availability.availability_hash + or project_binding + != { + "project_id": descriptor.project_id, + "project_root": str(descriptor.root), + "project_root_fingerprint": project_root_fingerprint(descriptor.root), + "adapter": descriptor.adapter, + "descriptor_hash": descriptor.descriptor_hash, + } + ): + raise AssertionError("Generated adapter client identity drifted") + + artifact = cast(dict[str, object], result["artifact"]) + binding = cast(dict[str, object], result["binding"]) + content = cast(str, artifact["content"]) + if artifact["content_sha256"] != hashlib.sha256(content.encode("utf-8")).hexdigest(): + raise AssertionError("Generated adapter client content hash drifted") + timeouts = cast(dict[str, object], binding["timeouts"]) + expected_format, expected_content, _ = _artifact( + cast(ClientName, result["client"]), + server_name=cast(str, result["server_name"]), + command=cast(str, binding["command"]), + arguments=cast(list[str], binding["args"]), + startup_timeout=cast(int, timeouts["startup_seconds"]), + tool_timeout=cast(int, timeouts["tool_seconds"]), + ) + if artifact["format"] != expected_format or content != expected_content: + raise AssertionError("Generated adapter client artifact drifted from its binding") + if binding["environment"] != {} or binding["launcher_hash"] != launcher.launcher_hash: + raise AssertionError("Generated adapter launch boundary drifted") + + adapter_policy = cast(dict[str, object], binding["adapter_policy"]) + render_policy = cast(dict[str, object], binding["render_policy"]) + projection_policy = cast(dict[str, object], result["projection_policy"]) + projection_availability = cast(dict[str, object], result["projection_availability"]) + arguments = cast(list[str], binding["args"]) + prefix = [ + "-I", + "-m", + launcher.module, + "--project-root", + str(descriptor.root), + "--capability-mode", + cast(str, binding["capability_mode"]), + ] + if arguments[:7] != prefix: + raise AssertionError("Generated adapter client arguments drifted from their binding") + remaining = arguments[7:] + no_ast_argument = "--no-ast" in arguments + if no_ast_argument: + if remaining[-1:] != ["--no-ast"] or arguments.count("--no-ast") != 1: + raise AssertionError("Generated adapter no-AST argument layout drifted") + remaining = remaining[:-1] + projection_arguments: dict[str, str] = {} + authority_arguments: list[str] = [] + projection_options = { + "--manual-render-policy": "manual", + "--portable-graph-policy": "portable_graph", + "--live-viewer-policy": "live_viewer", + } + position = 0 + while position < len(remaining): + option = remaining[position] + field = projection_options.get(option) + if field is None: + authority_arguments.append(option) + position += 1 + continue + if position + 1 >= len(remaining) or option in projection_arguments: + raise AssertionError("Generated adapter projection argument layout drifted") + value = remaining[position + 1] + projection_arguments[option] = value + if projection_policy[field] != value: + raise AssertionError("Generated adapter projection policy argument drifted") + position += 2 + mode = binding["capability_mode"] + if ( + (mode == "read" and authority_arguments) + or ( + mode == "proposal" + and ( + len(authority_arguments) != 2 + or authority_arguments[0] != "--proposal-writer" + or not authority_arguments[1] + ) + ) + or ( + mode == "application" + and ( + len(authority_arguments) != 4 + or authority_arguments[0] != "--proposal-writer" + or authority_arguments[2] != "--canonical-applier" + or not authority_arguments[1] + or authority_arguments[1] != authority_arguments[3] + ) + ) + ): + raise AssertionError("Generated adapter authority argument layout drifted") + + expected_projection_policy = compose_projection_policy( + manual=projection_arguments.get("--manual-render-policy"), + portable_graph=projection_arguments.get("--portable-graph-policy"), + live_viewer=projection_arguments.get("--live-viewer-policy"), + manual_configured=cast(bool, projection_availability["manual_configured"]), + portable_graph_configured=cast( + bool, + projection_availability["portable_graph_configured"], + ), + application_enabled=cast( + bool, + projection_availability["application_enabled"], + ), + live_viewer_available=cast( + bool, + projection_availability["live_viewer_available"], + ), + ) + policy = cast(dict[str, object], result["effective_policy"]) + composed_policy = compose_effective_policy( + selected_mode=cast(CapabilityMode, mode), + capability_source="explicit", + no_ast=adapter_policy["mode"] == "preserve-no-ast", + diagnostics=False, + render_configured=render_policy["manual"] != "disabled", + application_enabled=mode == "application", + ) + if ( + projection_policy != expected_projection_policy.as_dict() + or projection_availability["manual_configured"] != (descriptor.render is not None) + or projection_availability["portable_graph_configured"] + != (descriptor.graph_render is not None) + or projection_availability["application_enabled"] != (mode == "application") + or projection_availability["live_viewer_available"] is not True + or policy != composed_policy.as_dict() + or adapter_policy != composed_policy.adapter_policy() + or no_ast_argument != (adapter_policy["mode"] == "preserve-no-ast") + or binding["launcher_hash"] != result["launcher_hash"] + ): + raise AssertionError("Generated adapter client policy drifted") + if result["projection_policy_hash"] != hashlib.sha256( + json.dumps( + projection_policy, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + ).encode("utf-8") + ).hexdigest() or result["configuration_hash"] != document_hash( + _adapter_configuration_hash_payload(result) + ): + raise AssertionError("Generated adapter client hash drifted") diff --git a/src/docforge/command_reference.py b/src/docforge/command_reference.py new file mode 100644 index 0000000..909422e --- /dev/null +++ b/src/docforge/command_reference.py @@ -0,0 +1,242 @@ +"""Deterministic command references derived from live CLI and MCP registrations.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +from collections.abc import Collection, Iterable, Mapping +from dataclasses import dataclass +from typing import Any, cast + +from mcp.types import Tool + + +@dataclass(frozen=True) +class CliCommandReference: + """One CLI command and the exact normalized usage emitted by argparse.""" + + name: str + invocation: str + + +@dataclass(frozen=True) +class McpToolReference: + """One registered MCP tool with arguments derived from its input schema.""" + + surface: str + name: str + required_arguments: tuple[str, ...] + optional_arguments: tuple[str, ...] + description: str + input_schema_hash: str + + +def cli_command_references( + parser: argparse.ArgumentParser | None = None, +) -> tuple[CliCommandReference, ...]: + """Read CLI reference rows from the parser used by ``docforge``.""" + + effective_parser = parser or _docforge_parser() + commands = _command_parsers(effective_parser) + return tuple( + CliCommandReference( + name=name, + invocation=_normalize_usage(command_parser.format_usage()), + ) + for name, command_parser in sorted(commands.items()) + ) + + +def mcp_tool_references( + tools: Iterable[Tool], + *, + expected_names: Collection[str] | None = None, +) -> tuple[McpToolReference, ...]: + """Read MCP reference rows from registered tools returned by ``list_tools``. + + ``expected_names`` makes documentation generation fail closed when the selected + server surface is incomplete or has drifted. + """ + + surface_by_name = _mcp_surface_by_name() + references: list[McpToolReference] = [] + observed: set[str] = set() + for tool in tools: + if tool.name in observed: + raise ValueError(f"MCP tool registration repeats {tool.name!r}") + observed.add(tool.name) + try: + surface = surface_by_name[tool.name] + except KeyError as error: + raise ValueError( + f"MCP tool {tool.name!r} has no declared capability surface" + ) from error + schema = tool.inputSchema + properties = _schema_properties(schema) + required = _schema_required(schema, properties) + references.append( + McpToolReference( + surface=surface, + name=tool.name, + required_arguments=tuple(sorted(required)), + optional_arguments=tuple(sorted(set(properties) - required)), + description=_normalize_text(tool.description or ""), + input_schema_hash=_canonical_hash(schema), + ) + ) + if expected_names is not None: + expected = set(expected_names) + if observed != expected: + missing = sorted(expected - observed) + unexpected = sorted(observed - expected) + raise ValueError( + "Registered MCP tools do not match the requested reference surface: " + f"missing={missing!r}, unexpected={unexpected!r}" + ) + surface_rank = {"read": 0, "proposal": 1, "application": 2} + return tuple(sorted(references, key=lambda item: (surface_rank[item.surface], item.name))) + + +def render_cli_reference_markdown(references: Iterable[CliCommandReference]) -> str: + """Render a deterministic Markdown table for CLI commands.""" + + rows = sorted(references, key=lambda item: item.name) + lines = [ + "| Command | Invocation |", + "|---|---|", + ] + lines.extend( + f"| `{_escape_markdown(item.name)}` | `{_escape_markdown(item.invocation)}` |" + for item in rows + ) + return "\n".join(lines) + "\n" + + +def render_mcp_reference_markdown(references: Iterable[McpToolReference]) -> str: + """Render a deterministic Markdown table for registered MCP tools.""" + + surface_rank = {"read": 0, "proposal": 1, "application": 2} + rows = sorted(references, key=lambda item: (surface_rank[item.surface], item.name)) + lines = [ + "| Surface | Tool | Required arguments | Optional arguments | " + "Input schema SHA-256 | Description |", + "|---|---|---|---|---|---|", + ] + lines.extend( + "| " + f"{_escape_markdown(item.surface)} | " + f"`{_escape_markdown(item.name)}` | " + f"{_argument_list(item.required_arguments)} | " + f"{_argument_list(item.optional_arguments)} | " + f"`{item.input_schema_hash}` | " + f"{_escape_markdown(item.description) or '—'} |" + for item in rows + ) + return "\n".join(lines) + "\n" + + +def render_command_reference_markdown( + cli_references: Iterable[CliCommandReference], + mcp_references: Iterable[McpToolReference], +) -> str: + """Render the complete deterministic CLI and MCP command reference.""" + + return ( + "# DocForge command reference\n\n" + "> Generated from the live CLI parser and MCP registrations. Do not edit this file " + "by hand.\n\n" + "Global CLI options are documented in `docforge --help` and are not repeated in each " + "command row. The MCP table is the complete generic application-enabled surface; " + "the fixed reference-adapter server exposes only its read rows.\n\n" + "## CLI commands\n\n" + f"{render_cli_reference_markdown(cli_references)}" + "\n## MCP tools\n\n" + f"{render_mcp_reference_markdown(mcp_references)}" + ) + + +def _docforge_parser() -> argparse.ArgumentParser: + from .cli import _parser # pyright: ignore[reportPrivateUsage] + + return _parser() + + +def _command_parsers( + parser: argparse.ArgumentParser, +) -> Mapping[str, argparse.ArgumentParser]: + actions = parser._actions # pyright: ignore[reportPrivateUsage] + subparsers = [ + action + for action in actions + if action.dest == "command" and isinstance(action.choices, dict) + ] + if len(subparsers) != 1: + raise ValueError("CLI parser must define exactly one command subparser") + return cast(Mapping[str, argparse.ArgumentParser], subparsers[0].choices) + + +def _mcp_surface_by_name() -> dict[str, str]: + from .mcp_server import APPLICATION_TOOLS, PROPOSAL_TOOLS, READ_TOOLS + + groups = { + "read": READ_TOOLS, + "proposal": PROPOSAL_TOOLS, + "application": APPLICATION_TOOLS, + } + surface_by_name: dict[str, str] = {} + for surface, names in groups.items(): + for name in names: + if name in surface_by_name: + raise ValueError(f"MCP tool surface declaration repeats {name!r}") + surface_by_name[name] = surface + return surface_by_name + + +def _normalize_usage(usage: str) -> str: + return _normalize_text(usage.removeprefix("usage: ")) + + +def _normalize_text(value: str) -> str: + return " ".join(value.split()) + + +def _schema_properties(schema: Mapping[str, Any]) -> dict[str, Any]: + raw_properties: object = schema.get("properties", {}) + if not isinstance(raw_properties, dict): + raise ValueError("MCP tool input schema properties must be a string-keyed object") + properties: dict[str, Any] = {} + for name, value in cast(dict[object, object], raw_properties).items(): + if not isinstance(name, str): + raise ValueError("MCP tool input schema properties must be a string-keyed object") + properties[name] = value + return properties + + +def _schema_required(schema: Mapping[str, Any], properties: Mapping[str, Any]) -> set[str]: + raw_required: object = schema.get("required", []) + if not isinstance(raw_required, list): + raise ValueError("MCP tool input schema required arguments must be strings") + required_names: set[str] = set() + for name in cast(list[object], raw_required): + if not isinstance(name, str): + raise ValueError("MCP tool input schema required arguments must be strings") + required_names.add(name) + if not required_names <= set(properties): + raise ValueError("MCP tool input schema requires an undeclared argument") + return required_names + + +def _canonical_hash(payload: Mapping[str, Any]) -> str: + encoded = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode() + return hashlib.sha256(encoded).hexdigest() + + +def _argument_list(arguments: tuple[str, ...]) -> str: + if not arguments: + return "—" + return ", ".join(f"`{_escape_markdown(argument)}`" for argument in arguments) + + +def _escape_markdown(value: str) -> str: + return value.replace("\\", "\\\\").replace("|", "\\|").replace("`", "\\`") diff --git a/src/docforge/incremental.py b/src/docforge/incremental.py index 776111d..7f5f511 100644 --- a/src/docforge/incremental.py +++ b/src/docforge/incremental.py @@ -4,6 +4,7 @@ from __future__ import annotations import json import os +import stat import tempfile from dataclasses import dataclass from pathlib import Path @@ -12,6 +13,8 @@ from typing import Any, cast from .errors import DocForgeError EXTRACTION_CACHE_SCHEMA_VERSION = 1 +MAX_EXTRACTION_CACHE_BYTES = 64_000_000 +MAX_EXTRACTION_CACHE_SOURCES = 10_000 @dataclass(frozen=True) @@ -42,13 +45,39 @@ def load_extraction_cache( project_id: str, adapter_id: str, adapter_version: str, + max_bytes: int = MAX_EXTRACTION_CACHE_BYTES, + max_sources: int = MAX_EXTRACTION_CACHE_SOURCES, ) -> ExtractionCache | None: """Read a cache generation, treating malformed or incompatible data as a miss.""" - if not path.is_file() or path.is_symlink(): + if max_bytes < 1 or max_sources < 1: + raise ValueError("Extraction cache limits must be positive") + try: + descriptor = os.open( + path, + os.O_RDONLY + | getattr(os, "O_CLOEXEC", 0) + | getattr(os, "O_NOFOLLOW", 0) + | getattr(os, "O_NONBLOCK", 0), + ) + except OSError: return None try: - raw = json.loads(path.read_text(encoding="utf-8")) + with os.fdopen(descriptor, "rb") as handle: + before = os.fstat(handle.fileno()) + if not stat.S_ISREG(before.st_mode) or before.st_size > max_bytes: + return None + encoded = handle.read(max_bytes + 1) + after = os.fstat(handle.fileno()) + if ( + len(encoded) > max_bytes + or before.st_dev != after.st_dev + or before.st_ino != after.st_ino + or before.st_size != after.st_size + or before.st_mtime_ns != after.st_mtime_ns + ): + return None + raw = json.loads(encoded.decode("utf-8")) if not isinstance(raw, dict): return None document = cast(dict[str, Any], raw) @@ -62,8 +91,11 @@ def load_extraction_cache( raw_sources = document.get("sources") if not isinstance(raw_sources, list): return None + source_items = cast(list[object], raw_sources) + if len(source_items) > max_sources: + return None sources: list[CachedSource] = [] - for raw_source in cast(list[object], raw_sources): + for raw_source in source_items: if not isinstance(raw_source, dict): return None item = cast(dict[str, object], raw_source) @@ -110,9 +142,24 @@ def load_extraction_cache( return None -def write_extraction_cache(path: Path, cache: ExtractionCache) -> None: +def write_extraction_cache( + path: Path, + cache: ExtractionCache, + *, + max_bytes: int = MAX_EXTRACTION_CACHE_BYTES, + max_sources: int = MAX_EXTRACTION_CACHE_SOURCES, +) -> None: """Atomically publish one validated extraction-cache generation.""" + if max_bytes < 1 or max_sources < 1: + raise ValueError("Extraction cache limits must be positive") + if len(cache.sources) > max_sources: + raise DocForgeError( + "cache_limit", + "Incremental extraction cache exceeds its source limit", + maximum=max_sources, + actual=len(cache.sources), + ) path.parent.mkdir(parents=True, exist_ok=True) document = { "schema_version": EXTRACTION_CACHE_SCHEMA_VERSION, @@ -131,17 +178,23 @@ def write_extraction_cache(path: Path, cache: ExtractionCache) -> None: for source in cache.sources ], } + encoded = json.dumps(document, sort_keys=True, separators=(",", ":")).encode("utf-8") + b"\n" + if len(encoded) > max_bytes: + raise DocForgeError( + "cache_limit", + "Incremental extraction cache exceeds its byte limit", + maximum=max_bytes, + actual=len(encoded), + ) with tempfile.NamedTemporaryFile( - mode="w", - encoding="utf-8", + mode="wb", prefix="extractions-", suffix=".json", dir=path.parent, delete=False, ) as descriptor: temporary = Path(descriptor.name) - json.dump(document, descriptor, sort_keys=True, separators=(",", ":")) - descriptor.write("\n") + descriptor.write(encoded) descriptor.flush() os.fsync(descriptor.fileno()) try: diff --git a/src/docforge/reference_config.py b/src/docforge/reference_config.py new file mode 100644 index 0000000..c6557a2 --- /dev/null +++ b/src/docforge/reference_config.py @@ -0,0 +1,428 @@ +"""Strict, data-only configuration for the runnable reference adapters.""" + +from __future__ import annotations + +import hashlib +import os +import stat +import tomllib +from dataclasses import dataclass +from pathlib import Path, PurePosixPath +from typing import Literal, cast + +from .config_validation import ID_PATTERN +from .errors import DocForgeError + +REFERENCE_ADAPTER_SCHEMA_VERSION = 1 +REFERENCE_ADAPTER_CONFIG = PurePosixPath(".docforge/reference-adapter.toml") +MAX_REFERENCE_CONFIG_BYTES = 65_536 +MAX_REFERENCE_INVENTORY_ENTRIES = 65_536 + +ReferenceLanguage = Literal["python", "javascript", "typescript", "cpp"] +REFERENCE_LANGUAGES: tuple[ReferenceLanguage, ...] = ( + "python", + "javascript", + "typescript", + "cpp", +) + +_CONFIG_FIELDS = frozenset( + { + "schema_version", + "project_id", + "title", + "language", + "source_roots", + "compilation_database", + } +) +_LANGUAGE_SUFFIXES: dict[ReferenceLanguage, frozenset[str]] = { + "python": frozenset({".py"}), + "javascript": frozenset({".cjs", ".js", ".mjs"}), + "typescript": frozenset({".cts", ".mts", ".ts"}), + "cpp": frozenset({".c", ".cc", ".cpp", ".cxx", ".h", ".hh", ".hpp", ".hxx"}), +} +_KNOWN_SOURCE_SUFFIXES = frozenset( + suffix for suffixes in _LANGUAGE_SUFFIXES.values() for suffix in suffixes +) + +__all__ = [ + "MAX_REFERENCE_CONFIG_BYTES", + "MAX_REFERENCE_INVENTORY_ENTRIES", + "REFERENCE_ADAPTER_CONFIG", + "REFERENCE_ADAPTER_SCHEMA_VERSION", + "REFERENCE_LANGUAGES", + "ReferenceAdapterConfigV1", + "ReferenceLanguage", + "load_reference_adapter_config", +] + + +@dataclass(frozen=True) +class ReferenceAdapterConfigV1: + """One immutable, project-confined reference adapter selection.""" + + schema_version: Literal[1] + project_id: str + title: str + language: ReferenceLanguage + source_roots: tuple[Path, ...] + compilation_database: Path | None + project_root: Path + config_path: Path + config_hash: str + + def as_dict(self) -> dict[str, object]: + document: dict[str, object] = { + "schema_version": self.schema_version, + "project_id": self.project_id, + "title": self.title, + "language": self.language, + "source_roots": [ + path.relative_to(self.project_root).as_posix() for path in self.source_roots + ], + } + if self.compilation_database is not None: + document["compilation_database"] = self.compilation_database.relative_to( + self.project_root + ).as_posix() + return document + + +def load_reference_adapter_config(project_root: str | Path) -> ReferenceAdapterConfigV1: + """Load the fixed reference-adapter descriptor without executing project content.""" + + root = _project_root(project_root) + config_path = root.joinpath(*REFERENCE_ADAPTER_CONFIG.parts) + try: + resolved_config = config_path.resolve(strict=True) + except (OSError, ValueError) as error: + raise DocForgeError( + "missing_path", + "Reference adapter configuration does not exist", + path=str(config_path), + ) from error + if resolved_config != config_path or not resolved_config.is_relative_to(root): + raise DocForgeError( + "path_escape", + "Reference adapter configuration must not traverse symbolic links", + ) + raw = _read_regular_file( + config_path, + field="reference adapter configuration", + maximum_bytes=MAX_REFERENCE_CONFIG_BYTES, + ) + try: + document = cast(dict[str, object], tomllib.loads(raw.decode("utf-8"))) + except UnicodeDecodeError as error: + raise DocForgeError( + "invalid_reference_config", + "Reference adapter configuration must be UTF-8", + ) from error + except tomllib.TOMLDecodeError as error: + raise DocForgeError( + "invalid_reference_config", + f"Invalid reference adapter configuration: {error}", + ) from error + + unknown = sorted(set(document) - _CONFIG_FIELDS) + if unknown: + raise DocForgeError( + "invalid_reference_config", + "Reference adapter configuration has unknown fields", + fields=unknown, + ) + if document.get("schema_version") != REFERENCE_ADAPTER_SCHEMA_VERSION: + raise DocForgeError( + "invalid_reference_config", + "Reference adapter schema_version must be 1", + ) + project_id = _string(document, "project_id") + if ID_PATTERN.fullmatch(project_id) is None: + raise DocForgeError( + "invalid_reference_config", + "Reference adapter project_id is not a stable ID", + project_id=project_id, + ) + title = _string(document, "title") + if len(title) > 256: + raise DocForgeError( + "invalid_reference_config", + "Reference adapter title exceeds its size limit", + maximum_characters=256, + ) + raw_language = _string(document, "language") + if raw_language not in REFERENCE_LANGUAGES: + raise DocForgeError( + "unsupported_reference_language", + "Reference adapter language is not supported", + language=raw_language, + supported=list(REFERENCE_LANGUAGES), + ) + language = raw_language + source_roots = _source_roots(root, document.get("source_roots")) + + raw_compilation_database = document.get("compilation_database") + compilation_database: Path | None = None + if language == "cpp": + if raw_compilation_database is None: + raise DocForgeError( + "invalid_reference_config", + "C++ reference adapters require compilation_database", + ) + compilation_database = _confined_path( + root, + raw_compilation_database, + field="compilation_database", + expected="file", + ) + elif raw_compilation_database is not None: + raise DocForgeError( + "invalid_reference_config", + "compilation_database is allowed only for the C++ reference adapter", + language=language, + ) + + _validate_language_inventory(source_roots, language) + return ReferenceAdapterConfigV1( + schema_version=1, + project_id=project_id, + title=title, + language=language, + source_roots=source_roots, + compilation_database=compilation_database, + project_root=root, + config_path=config_path, + config_hash=hashlib.sha256(raw).hexdigest(), + ) + + +def _project_root(raw: str | Path) -> Path: + path = Path(raw) + if path.is_symlink(): + raise DocForgeError("path_escape", "Reference project root must not be a symlink") + try: + root = path.resolve(strict=True) + except (OSError, ValueError) as error: + raise DocForgeError( + "missing_path", + "Reference project root does not exist", + path=str(path), + ) from error + if not root.is_dir(): + raise DocForgeError("invalid_path", "Reference project root must be a directory") + return root + + +def _read_regular_file(path: Path, *, field: str, maximum_bytes: int) -> bytes: + try: + before = path.lstat() + except OSError as error: + raise DocForgeError("missing_path", f"{field} does not exist", path=str(path)) from error + if stat.S_ISLNK(before.st_mode) or not stat.S_ISREG(before.st_mode): + raise DocForgeError("invalid_path", f"{field} must be a regular non-symlink file") + if before.st_size > maximum_bytes: + raise DocForgeError( + "invalid_reference_config", + f"{field} exceeds its size limit", + maximum_bytes=maximum_bytes, + ) + + def identity(value: os.stat_result) -> tuple[int, int, int, int, int, int]: + return ( + value.st_dev, + value.st_ino, + value.st_mode, + value.st_size, + value.st_mtime_ns, + value.st_ctime_ns, + ) + + flags = os.O_RDONLY | getattr(os, "O_CLOEXEC", 0) | getattr(os, "O_NOFOLLOW", 0) + try: + descriptor = os.open(path, flags) + except OSError as error: + raise DocForgeError( + "reference_config_changed", + f"{field} changed while it was opened", + ) from error + try: + opened = os.fstat(descriptor) + chunks: list[bytes] = [] + remaining = maximum_bytes + 1 + while remaining: + chunk = os.read(descriptor, min(remaining, 65_536)) + if not chunk: + break + chunks.append(chunk) + remaining -= len(chunk) + raw = b"".join(chunks) + opened_after = os.fstat(descriptor) + except OSError as error: + raise DocForgeError( + "reference_config_changed", + f"{field} changed while it was read", + ) from error + finally: + os.close(descriptor) + try: + after = path.lstat() + except OSError as error: + raise DocForgeError( + "reference_config_changed", + f"{field} changed while it was read", + ) from error + + if ( + identity(before) != identity(opened) + or identity(opened) != identity(opened_after) + or identity(opened_after) != identity(after) + or len(raw) != opened_after.st_size + or len(raw) > maximum_bytes + ): + raise DocForgeError( + "reference_config_changed", + f"{field} changed while it was read", + ) + return raw + + +def _string(document: dict[str, object], field: str) -> str: + value = document.get(field) + if not isinstance(value, str) or not value or value != value.strip(): + raise DocForgeError( + "invalid_reference_config", + f"Reference adapter {field} must be a non-empty normalized string", + ) + return value + + +def _source_roots(root: Path, raw: object) -> tuple[Path, ...]: + if not isinstance(raw, list) or not raw: + raise DocForgeError( + "invalid_reference_config", + "Reference adapter source_roots must be a non-empty string list", + ) + values = cast(list[object], raw) + if len(values) > 64: + raise DocForgeError( + "invalid_reference_config", + "Reference adapter source_roots exceeds its item limit", + maximum_items=64, + ) + paths = tuple( + _confined_path(root, value, field="source_roots", expected="directory") for value in values + ) + if len(paths) != len(set(paths)): + raise DocForgeError( + "invalid_reference_config", + "Reference adapter source_roots contains duplicates", + ) + ordered = tuple(sorted(paths)) + if paths != ordered: + raise DocForgeError( + "invalid_reference_config", + "Reference adapter source_roots must be sorted", + ) + private_root = root / ".docforge" + if any( + path == private_root + or path.is_relative_to(private_root) + or any( + path.is_relative_to(other) or other.is_relative_to(path) + for other in ordered + if other != path + ) + for path in ordered + ): + raise DocForgeError( + "invalid_reference_config", + "Reference adapter source_roots must be non-overlapping and outside .docforge", + ) + return ordered + + +def _confined_path(root: Path, raw: object, *, field: str, expected: str) -> Path: + if not isinstance(raw, str) or not raw or raw != raw.strip(): + raise DocForgeError( + "invalid_reference_config", + f"{field} must be a non-empty normalized relative path", + ) + relative = Path(raw) + if relative.is_absolute() or ".." in relative.parts or relative == Path("."): + raise DocForgeError("path_escape", f"{field} must stay inside the project root", path=raw) + candidate = root / relative + if candidate.is_symlink(): + raise DocForgeError("path_escape", f"{field} must not be a symlink", path=raw) + try: + resolved = candidate.resolve(strict=True) + except (OSError, ValueError) as error: + raise DocForgeError("missing_path", f"{field} does not exist", path=raw) from error + if resolved != candidate or not resolved.is_relative_to(root): + raise DocForgeError( + "path_escape", + f"{field} must not traverse symlinks or leave the project root", + path=raw, + ) + if expected == "directory" and not resolved.is_dir(): + raise DocForgeError("invalid_path", f"{field} must identify a directory", path=raw) + if expected == "file" and (not resolved.is_file() or resolved.is_symlink()): + raise DocForgeError( + "invalid_path", + f"{field} must identify a regular non-symlink file", + path=raw, + ) + return resolved + + +def _validate_language_inventory( + source_roots: tuple[Path, ...], + language: ReferenceLanguage, +) -> None: + expected = _LANGUAGE_SUFFIXES[language] + selected = 0 + examined = 0 + other_languages: set[str] = set() + for source_root in source_roots: + for directory, names, files in os.walk( + source_root, + followlinks=False, + onerror=_inventory_error, + ): + base = Path(directory) + for name in (*names, *files): + examined += 1 + if examined > MAX_REFERENCE_INVENTORY_ENTRIES: + raise DocForgeError( + "reference_inventory_too_large", + "Reference source inventory exceeds its entry limit", + maximum_entries=MAX_REFERENCE_INVENTORY_ENTRIES, + ) + candidate = base / name + if candidate.is_symlink(): + raise DocForgeError( + "path_escape", + "Reference source roots must not contain symlinks", + path=candidate.as_posix(), + ) + for name in files: + suffix = Path(name).suffix + if suffix in expected: + selected += 1 + elif suffix in _KNOWN_SOURCE_SUFFIXES: + other_languages.add(suffix) + if selected == 0: + raise DocForgeError( + "reference_language_mismatch", + "Declared source roots contain no sources for the selected language", + language=language, + observed_suffixes=sorted(other_languages), + ) + + +def _inventory_error(error: OSError) -> None: + raise DocForgeError( + "invalid_path", + "Reference source inventory cannot be read safely", + path=error.filename, + ) from error diff --git a/src/docforge/reference_mcp.py b/src/docforge/reference_mcp.py new file mode 100644 index 0000000..bf4a76e --- /dev/null +++ b/src/docforge/reference_mcp.py @@ -0,0 +1,196 @@ +"""Runnable, read-only MCP binding for fixed in-repository reference adapters.""" + +from __future__ import annotations + +import argparse +import importlib +from pathlib import Path +from typing import cast + +from mcp.server.fastmcp import FastMCP + +from .adapter_sdk import ( + AdapterImplementation, + AdapterLoader, + AdapterProject, + AdapterProjectSettings, +) +from .errors import DocForgeError +from .mcp_server import create_read_only_server +from .reference_config import ReferenceAdapterConfigV1, load_reference_adapter_config + +REFERENCE_MCP_MODULE = "docforge.reference_mcp" +_FIXED_PROVIDERS = { + "python": ("docforge.adapters.python", "PythonReferenceAdapter"), + "javascript": ("docforge.adapters.javascript", "JavaScriptReferenceAdapter"), + "typescript": ("docforge.adapters.javascript", "JavaScriptReferenceAdapter"), + "cpp": ("docforge.adapters.cpp", "CppReferenceAdapter"), +} + +__all__ = [ + "REFERENCE_MCP_MODULE", + "create_reference_project", + "create_reference_server", + "main", +] + + +def create_reference_project(project_root: str | Path) -> AdapterProject: + """Construct one configured adapter project from fixed internal providers.""" + + config = load_reference_adapter_config(project_root) + return _project_from_config(config) + + +def create_reference_server( + project_root: str | Path, + *, + no_ast: bool = False, + diagnostics: bool = False, + capability_mode: str | None = None, + manual_projection_policy: str | None = None, + portable_graph_policy: str | None = None, + live_viewer_policy: str | None = None, +) -> FastMCP: + """Create the fixed read-only MCP surface for one reference project.""" + + config = load_reference_adapter_config(project_root) + project = _project_from_config(config) + return create_read_only_server( + project, + binding_metadata={ + "server_module": REFERENCE_MCP_MODULE, + "adapter_mode": "reference", + "reference_language": config.language, + "reference_config_hash": config.config_hash, + }, + no_ast=no_ast, + diagnostics=diagnostics, + capability_mode=capability_mode, + manual_projection_policy=manual_projection_policy, + portable_graph_policy=portable_graph_policy, + live_viewer_policy=live_viewer_policy, + ) + + +def _project_from_config(config: ReferenceAdapterConfigV1) -> AdapterProject: + loader = _adapter_loader(config) + cache_root = config.project_root / ".docforge" / "cache" / "reference-adapter" / config.language + return AdapterProject( + loader, + cache_root=cache_root, + settings=AdapterProjectSettings( + descriptor_path=config.config_path, + implementation=AdapterImplementation(files=(config.config_path,)), + ), + ) + + +def _adapter_loader(config: ReferenceAdapterConfigV1) -> AdapterLoader: + module_name, class_name = _FIXED_PROVIDERS[config.language] + try: + module = importlib.import_module(module_name) + except ModuleNotFoundError as error: + if error.name != module_name: + raise DocForgeError( + "optional_dependency_missing", + "The selected reference adapter dependency is not installed", + language=config.language, + missing_module=error.name, + install=_install_target(config.language), + ) from error + raise DocForgeError( + "reference_adapter_unavailable", + "The selected fixed reference adapter is not available", + language=config.language, + ) from error + constructor = getattr(module, class_name, None) + if not callable(constructor): + raise DocForgeError( + "reference_adapter_unavailable", + "The selected fixed reference adapter is not available", + language=config.language, + ) + arguments: dict[str, object] = { + "source_roots": tuple( + path.relative_to(config.project_root).as_posix() for path in config.source_roots + ), + "project_id": config.project_id, + "title": config.title, + } + if config.language == "cpp": + if config.compilation_database is None: + raise DocForgeError( + "invalid_reference_config", + "C++ reference adapters require compilation_database", + ) + arguments["compilation_database"] = config.compilation_database.relative_to( + config.project_root + ).as_posix() + loader = constructor(config.project_root, **arguments) + if not callable(getattr(loader, "load_projection", None)): + raise DocForgeError( + "invalid_adapter", + "The selected fixed provider does not implement the adapter contract", + language=config.language, + ) + return cast(AdapterLoader, loader) + + +def _install_target(language: str) -> str: + if language in {"javascript", "typescript"}: + return f"docforge[{language}]" + if language == "cpp": + return "docforge[cpp]" + return "docforge" + + +def main() -> None: + """Run the fixed reference binding over stdio.""" + + parser = argparse.ArgumentParser(prog="python -m docforge.reference_mcp") + parser.add_argument("--project-root", type=Path, required=True) + parser.add_argument( + "--no-ast", + action="store_true", + help=( + "Preserve the existing adapter and forbid AST, Tree-sitter, compiler-AST, " + "and function-Logic extraction changes" + ), + ) + parser.add_argument( + "--diagnostics", + action="store_true", + help="Attach bounded request-local stage timings and counters", + ) + parser.add_argument( + "--capability-mode", + choices=("read",), + help="Expose the fixed read-only project-bound capability surface", + ) + parser.add_argument( + "--manual-render-policy", + choices=("auto", "explicit", "disabled"), + ) + parser.add_argument( + "--portable-graph-policy", + choices=("explicit", "disabled"), + ) + parser.add_argument( + "--live-viewer-policy", + choices=("on-demand", "disabled"), + ) + arguments = parser.parse_args() + create_reference_server( + arguments.project_root, + no_ast=arguments.no_ast, + diagnostics=arguments.diagnostics, + capability_mode=arguments.capability_mode, + manual_projection_policy=arguments.manual_render_policy, + portable_graph_policy=arguments.portable_graph_policy, + live_viewer_policy=arguments.live_viewer_policy, + ).run(transport="stdio") + + +if __name__ == "__main__": + main() diff --git a/src/docforge/treesitter_logic.py b/src/docforge/treesitter_logic.py index 41f981a..c6892bb 100644 --- a/src/docforge/treesitter_logic.py +++ b/src/docforge/treesitter_logic.py @@ -9,18 +9,20 @@ data; it is never imported, compiled, or executed. from __future__ import annotations import hashlib +import importlib from collections.abc import Iterable from dataclasses import dataclass from functools import lru_cache - -import tree_sitter_cpp -import tree_sitter_javascript -from tree_sitter import Language, Node, Parser +from typing import TYPE_CHECKING from .errors import DocForgeError from .models import LogicEdge, LogicNode, LogicProjection +if TYPE_CHECKING: + from tree_sitter import Language, Node + _COMMENT_NODE_TYPES = frozenset({"comment"}) +_SCRIPT_LANGUAGES = frozenset({"javascript", "typescript"}) @dataclass(frozen=True) @@ -79,7 +81,34 @@ class _LanguageProfile: def _javascript_profile() -> _LanguageProfile: return _LanguageProfile( name="javascript", - language=Language(tree_sitter_javascript.language()), + language=_optional_language( + grammar_module="tree_sitter_javascript", + grammar_function="language", + extra="javascript", + ), + root_type="program", + block_types=frozenset({"program", "statement_block"}), + function_types=frozenset( + {"function_declaration", "generator_function_declaration", "method_definition"} + ), + loop_types=frozenset( + {"while_statement", "do_statement", "for_statement", "for_in_statement"} + ), + return_types=frozenset({"return_statement"}), + raise_types=frozenset({"throw_statement"}), + switch_case_types=frozenset({"switch_case", "switch_default"}), + ) + + +@lru_cache(maxsize=1) +def _typescript_profile() -> _LanguageProfile: + return _LanguageProfile( + name="typescript", + language=_optional_language( + grammar_module="tree_sitter_typescript", + grammar_function="language_typescript", + extra="typescript", + ), root_type="program", block_types=frozenset({"program", "statement_block"}), function_types=frozenset( @@ -98,7 +127,11 @@ def _javascript_profile() -> _LanguageProfile: def _cpp_profile() -> _LanguageProfile: return _LanguageProfile( name="cpp", - language=Language(tree_sitter_cpp.language()), + language=_optional_language( + grammar_module="tree_sitter_cpp", + grammar_function="language", + extra="cpp", + ), root_type="translation_unit", block_types=frozenset({"translation_unit", "compound_statement"}), function_types=frozenset({"function_definition"}), @@ -116,6 +149,28 @@ def _cpp_profile() -> _LanguageProfile: ) +def _optional_language( + *, + grammar_module: str, + grammar_function: str, + extra: str, +) -> Language: + try: + tree_sitter = importlib.import_module("tree_sitter") + grammar = importlib.import_module(grammar_module) + except ModuleNotFoundError as error: + raise DocForgeError( + "optional_dependency_missing", + "The requested language frontend is not installed", + extra=extra, + install=f"docforge[{extra}]", + missing_module=error.name, + ) from error + language_type = tree_sitter.Language + language_factory = getattr(grammar, grammar_function) + return language_type(language_factory()) + + def analyze_javascript_source( source: str, *, @@ -136,12 +191,38 @@ def analyze_javascript_source( ) +def analyze_typescript_source( + source: str, + *, + source_id: str, + owners: Iterable[TreeSitterLogicOwner], + filename: str = "", + max_nodes_per_function: int = 2_000, +) -> tuple[LogicProjection, ...]: + """Build control-flow projections for named TypeScript functions and methods.""" + + return _analyze_tree_sitter_source( + source, + source_id=source_id, + owners=owners, + filename=filename, + profile=_typescript_profile(), + max_nodes_per_function=max_nodes_per_function, + ) + + def discover_javascript_functions(source: str) -> tuple[DiscoveredFunction, ...]: """Return named JavaScript functions, methods, and assigned arrow functions.""" return _discover_functions(source, _javascript_profile()) +def discover_typescript_functions(source: str) -> tuple[DiscoveredFunction, ...]: + """Return named TypeScript functions, methods, and assigned arrow functions.""" + + return _discover_functions(source, _typescript_profile()) + + def discover_cpp_functions(source: str) -> tuple[DiscoveredFunction, ...]: """Return named C++ functions and methods.""" @@ -172,6 +253,8 @@ def _discover_functions( source: str, profile: _LanguageProfile, ) -> tuple[DiscoveredFunction, ...]: + from tree_sitter import Parser + raw = source.encode("utf-8") parser = Parser(profile.language) tree = parser.parse(raw) @@ -212,6 +295,8 @@ def _analyze_tree_sitter_source( profile: _LanguageProfile, max_nodes_per_function: int, ) -> tuple[LogicProjection, ...]: + from tree_sitter import Parser + if max_nodes_per_function < 2: raise ValueError("max_nodes_per_function must allow entry and exit nodes") raw = source.encode("utf-8") @@ -293,7 +378,7 @@ def _function_definitions( def _scope_name(node: Node, raw: bytes, profile: _LanguageProfile) -> str | None: - if profile.name == "javascript" and node.type in {"class_declaration", "class"}: + if profile.name in _SCRIPT_LANGUAGES and node.type in {"class_declaration", "class"}: return _field_text(node, "name", raw) if profile.name == "cpp" and node.type in { "namespace_definition", @@ -307,11 +392,11 @@ def _scope_name(node: Node, raw: bytes, profile: _LanguageProfile) -> str | None def _function_name(node: Node, raw: bytes, profile: _LanguageProfile) -> str | None: if node.type in profile.function_types: - if profile.name == "javascript": + if profile.name in _SCRIPT_LANGUAGES: return _field_text(node, "name", raw) declarator = node.child_by_field_name("declarator") return _declarator_name(declarator, raw) if declarator is not None else None - if profile.name != "javascript" or node.type != "variable_declarator": + if profile.name not in _SCRIPT_LANGUAGES or node.type != "variable_declarator": return None value = node.child_by_field_name("value") if value is None or value.type not in {"arrow_function", "function_expression"}: diff --git a/tests/fixtures/reference-cpp/compile_commands.json b/tests/fixtures/reference-cpp/compile_commands.json new file mode 100644 index 0000000..a34b931 --- /dev/null +++ b/tests/fixtures/reference-cpp/compile_commands.json @@ -0,0 +1,19 @@ +[ + { + "arguments": [ + "c++", + "-std=c++20", + "-Iinclude", + "-c", + "src/main.cpp" + ], + "directory": ".", + "file": "src/main.cpp" + }, + { + "command": "c++ -std=c++20 -Iinclude -c src/worker.cpp", + "directory": ".", + "file": "src/worker.cpp", + "output": "worker.o" + } +] diff --git a/tests/fixtures/reference-cpp/include/config.hpp b/tests/fixtures/reference-cpp/include/config.hpp new file mode 100644 index 0000000..7e57679 --- /dev/null +++ b/tests/fixtures/reference-cpp/include/config.hpp @@ -0,0 +1,15 @@ +#pragma once + +#include "detail.hpp" + +namespace demo { + +struct Config { + int limit; +}; + +inline int effective_limit(const Config& config) { + return normalize(config.limit); +} + +} // namespace demo diff --git a/tests/fixtures/reference-cpp/include/detail.hpp b/tests/fixtures/reference-cpp/include/detail.hpp new file mode 100644 index 0000000..63a76f6 --- /dev/null +++ b/tests/fixtures/reference-cpp/include/detail.hpp @@ -0,0 +1,12 @@ +#pragma once + +namespace demo { + +inline int normalize(int value) { + if (value < 0) { + return 0; + } + return value; +} + +} // namespace demo diff --git a/tests/fixtures/reference-cpp/src/main.cpp b/tests/fixtures/reference-cpp/src/main.cpp new file mode 100644 index 0000000..6f298ae --- /dev/null +++ b/tests/fixtures/reference-cpp/src/main.cpp @@ -0,0 +1,17 @@ +#include "../include/config.hpp" + +namespace demo { + +int execute(int value) { + Config config{value}; + if (value == 0) { + return effective_limit(config); + } + return normalize(value); +} + +} // namespace demo + +int main() { + return demo::execute(3); +} diff --git a/tests/fixtures/reference-cpp/src/worker.cpp b/tests/fixtures/reference-cpp/src/worker.cpp new file mode 100644 index 0000000..1d1cbcb --- /dev/null +++ b/tests/fixtures/reference-cpp/src/worker.cpp @@ -0,0 +1,17 @@ +#include "../include/config.hpp" + +namespace demo { + +class Worker { +public: + int run(int value) { + for (int attempt = 0; attempt < 2; ++attempt) { + if (value > attempt) { + return effective_limit(Config{value}); + } + } + return 0; + } +}; + +} // namespace demo diff --git a/tests/fixtures/reference-javascript/src/app/index.js b/tests/fixtures/reference-javascript/src/app/index.js new file mode 100644 index 0000000..4721b77 --- /dev/null +++ b/tests/fixtures/reference-javascript/src/app/index.js @@ -0,0 +1,8 @@ +import { Service } from "./service.mjs"; +export { clamp } from "./shared.js"; + +throw new Error("The reference adapter must never execute project code"); + +export function applicationName() { + return Service.name; +} diff --git a/tests/fixtures/reference-javascript/src/app/service.mjs b/tests/fixtures/reference-javascript/src/app/service.mjs new file mode 100644 index 0000000..80a7882 --- /dev/null +++ b/tests/fixtures/reference-javascript/src/app/service.mjs @@ -0,0 +1,14 @@ +import { clamp } from "./shared.js"; + +export class Service { + run(value) { + return clamp(value); + } +} + +export const buildService = (enabled) => { + if (enabled) { + return new Service(); + } + return null; +}; diff --git a/tests/fixtures/reference-javascript/src/app/shared.js b/tests/fixtures/reference-javascript/src/app/shared.js new file mode 100644 index 0000000..e502fbe --- /dev/null +++ b/tests/fixtures/reference-javascript/src/app/shared.js @@ -0,0 +1,11 @@ +export const DEFAULT_LIMIT = 3; + +export function clamp(value, limit = DEFAULT_LIMIT) { + if (value < 0) { + return 0; + } + if (value > limit) { + return limit; + } + return value; +} diff --git a/tests/fixtures/reference-javascript/src/app/worker.cjs b/tests/fixtures/reference-javascript/src/app/worker.cjs new file mode 100644 index 0000000..30592c9 --- /dev/null +++ b/tests/fixtures/reference-javascript/src/app/worker.cjs @@ -0,0 +1,5 @@ +export { Service } from "./service.mjs"; + +export function execute(worker, value) { + return worker.run(value); +} diff --git a/tests/fixtures/reference-python/src/sample/__init__.py b/tests/fixtures/reference-python/src/sample/__init__.py new file mode 100644 index 0000000..cc3b58a --- /dev/null +++ b/tests/fixtures/reference-python/src/sample/__init__.py @@ -0,0 +1,7 @@ +"""Small multi-file package for the Python reference adapter.""" + +from .service import Service + + +def package_name() -> str: + return Service.__name__ diff --git a/tests/fixtures/reference-python/src/sample/service.py b/tests/fixtures/reference-python/src/sample/service.py new file mode 100644 index 0000000..b559183 --- /dev/null +++ b/tests/fixtures/reference-python/src/sample/service.py @@ -0,0 +1,18 @@ +"""Service behavior with class, method, function, and local import evidence.""" + +import json + +from .shared import clamp + +raise RuntimeError("The reference adapter must never execute project code") + + +class Service: + def run(self, value: int) -> int: + return clamp(value) if json.__name__ else value + + +async def build_service(enabled: bool) -> Service | None: + if enabled: + return Service() + return None diff --git a/tests/fixtures/reference-python/src/sample/shared.py b/tests/fixtures/reference-python/src/sample/shared.py new file mode 100644 index 0000000..42a1fe9 --- /dev/null +++ b/tests/fixtures/reference-python/src/sample/shared.py @@ -0,0 +1,11 @@ +"""Shared values used by dependent modules.""" + +DEFAULT_LIMIT = 3 + + +def clamp(value: int, limit: int = DEFAULT_LIMIT) -> int: + if value < 0: + return 0 + if value > limit: + return limit + return value diff --git a/tests/fixtures/reference-python/src/sample/worker.py b/tests/fixtures/reference-python/src/sample/worker.py new file mode 100644 index 0000000..7d1962d --- /dev/null +++ b/tests/fixtures/reference-python/src/sample/worker.py @@ -0,0 +1,8 @@ +"""A transitive dependent used to prove reverse invalidation.""" + +from sample.service import Service + + +def execute(value: int) -> int: + worker = Service() + return worker.run(value) diff --git a/tests/fixtures/reference-typescript/src/app/index.ts b/tests/fixtures/reference-typescript/src/app/index.ts new file mode 100644 index 0000000..30e3fb4 --- /dev/null +++ b/tests/fixtures/reference-typescript/src/app/index.ts @@ -0,0 +1,8 @@ +import { Service } from "./service"; +export type { Choice } from "./types"; + +throw new Error("The reference adapter must never execute project code"); + +export function applicationName(): string { + return Service.name; +} diff --git a/tests/fixtures/reference-typescript/src/app/service.ts b/tests/fixtures/reference-typescript/src/app/service.ts new file mode 100644 index 0000000..aec4304 --- /dev/null +++ b/tests/fixtures/reference-typescript/src/app/service.ts @@ -0,0 +1,17 @@ +import type { Choice } from "./types"; + +export class Service { + run(choice: Choice): number { + if (choice.enabled) { + return choice.value; + } + return 0; + } +} + +export const buildService = (choice: Choice): Service | null => { + if (choice.enabled) { + return new Service(); + } + return null; +}; diff --git a/tests/fixtures/reference-typescript/src/app/types.mts b/tests/fixtures/reference-typescript/src/app/types.mts new file mode 100644 index 0000000..9d09da8 --- /dev/null +++ b/tests/fixtures/reference-typescript/src/app/types.mts @@ -0,0 +1,6 @@ +export interface Choice { + enabled: boolean; + value: number; +} + +export const DEFAULT_LIMIT: number = 3; diff --git a/tests/fixtures/reference-typescript/src/app/worker.cts b/tests/fixtures/reference-typescript/src/app/worker.cts new file mode 100644 index 0000000..4fc33e1 --- /dev/null +++ b/tests/fixtures/reference-typescript/src/app/worker.cts @@ -0,0 +1,6 @@ +import { Service } from "./service"; +import type { Choice } from "./types"; + +export function execute(worker: Service, choice: Choice): number { + return worker.run(choice); +} diff --git a/tests/test_adapter_contract.py b/tests/test_adapter_contract.py index e4bc3f5..ee1b183 100644 --- a/tests/test_adapter_contract.py +++ b/tests/test_adapter_contract.py @@ -39,6 +39,7 @@ from docforge.mcp_server import ( ) from docforge.models import ( Edge, + Limits, LogicEdge, LogicNode, LogicProjection, @@ -248,6 +249,36 @@ class NonLogicIncrementalLoader(IncrementalLoader): return replace(super().extract_source(source), logic=()) +class CompleteLogicIncrementalLoader(IncrementalLoader): + """Supply an independent complete Logic oracle alongside the legacy graph oracle.""" + + def __init__(self, root: Path, *, mismatched_logic: bool = False) -> None: + super().__init__(root) + self.mismatched_logic = mismatched_logic + + def load_complete_assembly(self) -> AdapterAssembly: + projection = super().load_projection() + if "guide.workflow" not in self.sources: + return AdapterAssembly(projection) + manifest = self.load_manifest() + workflow = next( + source for source in manifest.sources if source.source_id == "guide.workflow" + ) + logic = self.extract_source(workflow).logic + if self.mismatched_logic: + current = logic[0] + logic = ( + replace( + current, + nodes=( + current.nodes[0], + replace(current.nodes[1], label="Different complete return"), + ), + ), + ) + return AdapterAssembly(projection, logic) + + class AdapterContractTests(unittest.TestCase): def projection(self, root: Path) -> AdapterProjection: foundation = Node( @@ -311,6 +342,23 @@ class AdapterContractTests(unittest.TestCase): self.assertEqual("Workflow", index.get_node("guide.workflow")["node"]["title"]) self.assertEqual(projection.identity(), projection.identity()) + def test_adapter_project_enforces_graph_bounds_before_publication(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory).resolve() + project = AdapterProject( + Loader(self.projection(root)), + cache_root=root / ".cache" / "bounded", + settings=AdapterProjectSettings(limits=Limits(max_nodes=1)), + ) + + with self.assertRaises(DocForgeError) as captured: + project.load() + + self.assertEqual("adapter_limit", captured.exception.code) + self.assertEqual("nodes", captured.exception.details["kind"]) + self.assertEqual(1, captured.exception.details["maximum"]) + self.assertEqual(2, captured.exception.details["actual"]) + def test_projection_rejects_unsorted_metadata_graph_and_identity_changes(self) -> None: with tempfile.TemporaryDirectory() as directory: root = Path(directory).resolve() @@ -782,6 +830,46 @@ class AdapterContractTests(unittest.TestCase): self.assertEqual("ok", equivalent["status"]) self.assertEqual(1, equivalent["node_count"]) + def test_logic_equivalence_requires_an_independent_complete_oracle(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory).resolve() + project = AdapterProject( + IncrementalLoader(root), + cache_root=root / ".cache" / "logic-without-oracle", + ) + + with self.assertRaises(DocForgeError) as captured: + project.verify_incremental_equivalence() + + self.assertEqual("complete_logic_oracle_required", captured.exception.code) + + def test_logic_equivalence_uses_the_complete_assembly_oracle(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory).resolve() + project = AdapterProject( + CompleteLogicIncrementalLoader(root), + cache_root=root / ".cache" / "logic-oracle", + ) + + result = project.verify_incremental_equivalence() + + self.assertEqual("ok", result["status"]) + self.assertEqual(1, result["logic_projection_count"]) + + def test_logic_equivalence_reports_exact_mismatch_field(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory).resolve() + project = AdapterProject( + CompleteLogicIncrementalLoader(root, mismatched_logic=True), + cache_root=root / ".cache" / "logic-mismatch", + ) + + with self.assertRaises(DocForgeError) as captured: + project.verify_incremental_equivalence() + + self.assertEqual("incremental_mismatch", captured.exception.code) + self.assertEqual(["logic"], captured.exception.details["fields"]) + def test_incremental_adapter_can_assemble_overlapping_source_evidence(self) -> None: with tempfile.TemporaryDirectory() as directory: root = Path(directory).resolve() diff --git a/tests/test_adapter_launcher.py b/tests/test_adapter_launcher.py new file mode 100644 index 0000000..e03df33 --- /dev/null +++ b/tests/test_adapter_launcher.py @@ -0,0 +1,475 @@ +from __future__ import annotations + +import asyncio +import json +import os +import shutil +import subprocess +import sys +import tempfile +import tomllib +import unittest +import venv +from dataclasses import replace +from pathlib import Path +from unittest import mock + +from jsonschema import Draft202012Validator +from mcp import ClientSession, StdioServerParameters +from mcp.client.stdio import stdio_client +from referencing import Registry, Resource + +from docforge.adapter_contract import AdapterNode, AdapterProject, AdapterProjection +from docforge.adapter_launcher import AdapterLauncherV1 +from docforge.client_config import ( + _validate_adapter_configuration_result, + generate_adapter_client_configuration, + generate_client_configuration, +) +from docforge.errors import DocForgeError +from docforge.models import Node +from docforge.project import Project +from docforge.reference_mcp import ( + REFERENCE_MCP_MODULE, + create_reference_project, +) + +ROOT = Path(__file__).resolve().parents[1] +FIXTURES = ROOT / "tests" / "fixtures" +SCHEMAS = ROOT / "schemas" +LAUNCHER_SCHEMA = json.loads((SCHEMAS / "adapter-launcher.schema.json").read_text(encoding="utf-8")) +ADAPTER_CONFIGURATION_SCHEMA = json.loads( + (SCHEMAS / "adapter-client-configuration.schema.json").read_text(encoding="utf-8") +) +GENERIC_CONFIGURATION_SCHEMA = json.loads( + (SCHEMAS / "client-configuration.schema.json").read_text(encoding="utf-8") +) +SCHEMA_REGISTRY = Registry().with_resources( + ( + (LAUNCHER_SCHEMA["$id"], Resource.from_contents(LAUNCHER_SCHEMA)), + ( + GENERIC_CONFIGURATION_SCHEMA["$id"], + Resource.from_contents(GENERIC_CONFIGURATION_SCHEMA), + ), + ( + ADAPTER_CONFIGURATION_SCHEMA["$id"], + Resource.from_contents(ADAPTER_CONFIGURATION_SCHEMA), + ), + ) +) +ADAPTER_CONFIGURATION_VALIDATOR = Draft202012Validator( + ADAPTER_CONFIGURATION_SCHEMA, + registry=SCHEMA_REGISTRY, +) + + +class Loader: + def __init__(self, projection: AdapterProjection) -> None: + self.projection = projection + self.load_calls = 0 + + def load_projection(self) -> AdapterProjection: + self.load_calls += 1 + return self.projection + + +class DriftingLoader(Loader): + def __init__( + self, + projection: AdapterProjection, + changed: AdapterProjection, + ) -> None: + super().__init__(projection) + self.changed = changed + + def load_projection(self) -> AdapterProjection: + self.load_calls += 1 + return self.projection if self.load_calls < 3 else self.changed + + +class AdapterLauncherTests(unittest.TestCase): + def copy_fixture(self, destination: Path) -> Path: + root = destination / "alpha" + shutil.copytree(FIXTURES / "alpha", root) + return root + + def copy_reference_fixture(self, destination: Path) -> Path: + root = destination / "reference-python" + shutil.copytree(FIXTURES / "reference-python", root) + config = root / ".docforge" / "reference-adapter.toml" + config.parent.mkdir(parents=True) + config.write_text( + "\n".join( + ( + "schema_version = 1", + 'project_id = "reference-python"', + 'title = "Runnable Python reference"', + 'language = "python"', + 'source_roots = ["src"]', + "", + ) + ), + encoding="utf-8", + ) + return root.resolve() + + def projection( + self, + root: Path, + *, + source_hash: str = "3" * 64, + ) -> AdapterProjection: + node = Node( + node_id="source.entry", + title="Entry", + family="source", + authority="derived", + status="active", + tags=("python",), + summary="Reference adapter entry.", + content="Reference adapter content.", + source_path="src/entry.py", + source_anchor="L1", + content_hash="2" * 64, + ) + return AdapterProjection( + project_id="adapter-client-fixture", + title="Adapter client fixture", + adapter_id="fixture-client", + adapter_version="1", + root=root, + revision="fixture-revision", + source_hash=source_hash, + nodes=(AdapterNode(node),), + edges=(), + ) + + def project( + self, + root: Path, + *, + loader: Loader | None = None, + ) -> tuple[AdapterProject, Loader]: + effective_loader = loader or Loader(self.projection(root)) + return ( + AdapterProject( + effective_loader, + cache_root=root / ".docforge-cache" / "adapter-client", + ), + effective_loader, + ) + + def test_launcher_and_all_client_fragments_are_deterministic_and_schema_valid(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory).resolve() + project, _ = self.project(root) + launcher = AdapterLauncherV1.for_project( + project, + module=REFERENCE_MCP_MODULE, + ) + Draft202012Validator(LAUNCHER_SCHEMA).validate(launcher.as_dict()) + previous = os.environ.get("DOCFORGE_ADAPTER_LAUNCHER_SECRET") + os.environ["DOCFORGE_ADAPTER_LAUNCHER_SECRET"] = "must-not-appear" + try: + with mock.patch("docforge.client_config.subprocess.run") as executed: + results: dict[str, dict[str, object]] = {} + for client in ("codex", "claude", "openclaw"): + first = generate_adapter_client_configuration( + project, + launcher, + client, + no_ast=True, + ) + second = generate_adapter_client_configuration( + project, + launcher, + client, + no_ast=True, + ) + self.assertEqual(first, second) + ADAPTER_CONFIGURATION_VALIDATOR.validate(first) + self.assertEqual(launcher.launcher_hash, first["launcher_hash"]) + self.assertEqual( + launcher.launcher_hash, + first["binding"]["launcher_hash"], + ) + self.assertEqual( + [ + "-I", + "-m", + REFERENCE_MCP_MODULE, + "--project-root", + str(root), + "--capability-mode", + "read", + "--no-ast", + ], + first["binding"]["args"], + ) + self.assertEqual({}, first["binding"]["environment"]) + self.assertNotIn("cwd", first["binding"]) + self.assertNotIn( + "must-not-appear", + json.dumps(first, sort_keys=True), + ) + results[client] = first + executed.assert_not_called() + finally: + if previous is None: + os.environ.pop("DOCFORGE_ADAPTER_LAUNCHER_SECRET", None) + else: + os.environ["DOCFORGE_ADAPTER_LAUNCHER_SECRET"] = previous + + self.assertIn( + "mcp_servers", + tomllib.loads(results["codex"]["artifact"]["content"]), + ) + self.assertIn( + "mcpServers", + json.loads(results["claude"]["artifact"]["content"]), + ) + self.assertIn( + "mcp", + json.loads(results["openclaw"]["artifact"]["content"]), + ) + + def test_launcher_rejects_module_path_entry_point_and_argument_injection(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory).resolve() + project, _ = self.project(root) + launcher = AdapterLauncherV1.for_project( + project, + module=REFERENCE_MCP_MODULE, + ) + for module in ( + "-m", + "fixture_adapter.mcp_server", + "fixture_adapter.mcp_server --debug", + "fixture_adapter:mcp_server", + "fixture_adapter/mcp_server", + ".fixture_adapter", + "docforge.mcp_server", + "fixture_adapter.$server", + ): + with self.subTest(module=module), self.assertRaises(DocForgeError) as captured: + replace(launcher, module=module) + self.assertEqual("invalid_adapter_launcher", captured.exception.code) + + with self.assertRaises(DocForgeError): + replace(launcher, project_root=Path("relative/project")) + with self.assertRaises(DocForgeError): + replace(launcher, entry_point="console-script") # type: ignore[arg-type] + payload = launcher.as_dict() + payload["arguments"] = ["--shell", "command"] + with self.assertRaises(TypeError): + AdapterLauncherV1(**payload) # type: ignore[arg-type] + + result = generate_adapter_client_configuration(project, launcher, "codex") + result["binding"]["args"].append("--arbitrary") + with self.assertRaisesRegex(AssertionError, "Generated adapter client"): + _validate_adapter_configuration_result( + result, + project=project, + launcher=launcher, + source_availability=project_source_availability(result), + ) + + def test_wrong_project_descriptor_and_source_drift_fail_closed(self) -> None: + with tempfile.TemporaryDirectory() as directory: + parent = Path(directory) + first_root = (parent / "first").resolve() + second_root = (parent / "second").resolve() + first_root.mkdir() + second_root.mkdir() + first, _ = self.project(first_root) + second, _ = self.project(second_root) + launcher = AdapterLauncherV1.for_project( + first, + module=REFERENCE_MCP_MODULE, + ) + + with self.assertRaises(DocForgeError) as wrong_project: + generate_adapter_client_configuration(second, launcher, "codex") + self.assertEqual("adapter_launcher_mismatch", wrong_project.exception.code) + with self.assertRaises(DocForgeError) as descriptor_drift: + generate_adapter_client_configuration( + first, + replace(launcher, descriptor_hash="0" * 64), + "codex", + ) + self.assertEqual("adapter_launcher_mismatch", descriptor_drift.exception.code) + + base = self.projection(first_root) + changed = replace(base, source_hash="4" * 64) + drifting, _ = self.project( + first_root, + loader=DriftingLoader(base, changed), + ) + drifting_launcher = AdapterLauncherV1.for_project( + drifting, + module=REFERENCE_MCP_MODULE, + ) + with self.assertRaises(DocForgeError) as source_drift: + generate_adapter_client_configuration( + drifting, + drifting_launcher, + "openclaw", + ) + self.assertEqual("source_changed", source_drift.exception.code) + + def test_generic_configuration_remains_separate_and_compatible(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + generic = Project.open(root) + result = generate_client_configuration(generic, "codex", no_ast=True) + Draft202012Validator(GENERIC_CONFIGURATION_SCHEMA).validate(result) + self.assertEqual("client.configure", result["operation"]) + self.assertNotIn("launcher", result) + with self.assertRaises(DocForgeError) as generic_launcher: + AdapterLauncherV1.for_project( + generic, + module=REFERENCE_MCP_MODULE, + ) + self.assertEqual("adapter_launcher_unavailable", generic_launcher.exception.code) + + custom_root = (Path(directory) / "custom").resolve() + custom_root.mkdir() + custom, _ = self.project(custom_root) + with self.assertRaises(DocForgeError) as generic_api: + generate_client_configuration(custom, "codex") + self.assertEqual("missing_config", generic_api.exception.code) + + def test_installed_project_owned_top_level_module_launches_without_probe_execution( + self, + ) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory).resolve() + environment = root / ".launcher-venv" + venv.EnvBuilder(with_pip=False).create(environment) + executable = environment / "bin" / "python" + site_packages = Path( + subprocess.run( + [ + str(executable), + "-I", + "-c", + "import site; print(site.getsitepackages()[0])", + ], + check=True, + capture_output=True, + text=True, + ).stdout.strip() + ) + (site_packages / "fixture-adapter.pth").write_text( + f"{root}\n", + encoding="utf-8", + ) + sentinel = root / "fixture-adapter-executed" + (root / "fixture_adapter.py").write_text( + "\n".join( + ( + "import argparse", + "import json", + "from pathlib import Path", + "parser = argparse.ArgumentParser()", + 'parser.add_argument("--project-root", required=True)', + ( + 'parser.add_argument("--capability-mode", ' + 'choices=("read",), required=True)' + ), + "arguments = parser.parse_args()", + f"Path({str(sentinel)!r}).write_text('executed', encoding='utf-8')", + ( + "print(json.dumps({'project_root': arguments.project_root}, " + "sort_keys=True))" + ), + "", + ) + ), + encoding="utf-8", + ) + + project, _ = self.project(root) + with mock.patch.object(sys, "executable", str(executable)): + launcher = AdapterLauncherV1.for_project( + project, + module="fixture_adapter", + ) + result = generate_adapter_client_configuration( + project, + launcher, + "codex", + ) + self.assertFalse(sentinel.exists()) + Draft202012Validator(LAUNCHER_SCHEMA).validate(launcher.as_dict()) + binding = result["binding"] + completed = subprocess.run( + [binding["command"], *binding["args"]], + check=True, + capture_output=True, + text=True, + timeout=10, + ) + self.assertEqual({"project_root": str(root)}, json.loads(completed.stdout)) + self.assertEqual("executed", sentinel.read_text(encoding="utf-8")) + + def test_uninstalled_packages_and_non_project_modules_fail_closed(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory).resolve() + project, _ = self.project(root) + for module, code in ( + ("fixture_adapter", "adapter_launcher_unavailable"), + ("os", "adapter_launcher_unavailable"), + ("json", "invalid_adapter_launcher"), + ): + with self.subTest(module=module), self.assertRaises(DocForgeError) as captured: + AdapterLauncherV1.for_project(project, module=module) + self.assertEqual(code, captured.exception.code) + + def test_generated_reference_binding_starts_a_real_isolated_mcp_process(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.copy_reference_fixture(Path(directory)) + project = create_reference_project(root) + launcher = AdapterLauncherV1.for_project( + project, + module=REFERENCE_MCP_MODULE, + ) + result = generate_adapter_client_configuration( + project, + launcher, + "codex", + ) + binding = result["binding"] + + async def inspect() -> tuple[str, ...]: + parameters = StdioServerParameters( + command=binding["command"], + args=binding["args"], + env=binding["environment"], + ) + async with ( + stdio_client(parameters) as streams, + ClientSession(*streams) as session, + ): + await session.initialize() + return tuple(tool.name for tool in (await session.list_tools()).tools) + + tools = asyncio.run(inspect()) + self.assertIn("docforge_bootstrap", tools) + + +def project_source_availability(result: dict[str, object]): + from docforge.adapter_launcher import AdapterSourceAvailabilityV1 + + payload = result["source_availability"] + return AdapterSourceAvailabilityV1( + schema_version=payload["schema_version"], + status=payload["status"], + method=payload["method"], + revision=payload["revision"], + source_hash=payload["source_hash"], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_adapter_sdk.py b/tests/test_adapter_sdk.py new file mode 100644 index 0000000..059d095 --- /dev/null +++ b/tests/test_adapter_sdk.py @@ -0,0 +1,261 @@ +from __future__ import annotations + +import hashlib +import tempfile +import unittest +from dataclasses import replace +from pathlib import Path + +import docforge.adapter_sdk as sdk +from docforge.errors import DocForgeError + + +class ReferenceIncrementalAdapter: + """Small public-SDK-only adapter used to prove the conformance entry point.""" + + def __init__(self, root: Path) -> None: + self.root = root + + @staticmethod + def _hash(value: str) -> str: + return hashlib.sha256(value.encode()).hexdigest() + + def _node(self) -> sdk.AdapterNode: + content = "A deterministic reference function." + return sdk.AdapterNode( + sdk.Node( + node_id="reference.run", + title="Reference run", + family="source", + authority="derived", + status="current", + tags=("python",), + summary=content, + content=content, + source_path="src/reference.py", + source_anchor="L1", + content_hash=self._hash(content), + ) + ) + + @staticmethod + def _logic() -> sdk.LogicProjection: + return sdk.LogicProjection( + owner_node_id="reference.run", + source_id="source.reference", + nodes=( + sdk.LogicNode("entry", "entry", "Enter run", "L1"), + sdk.LogicNode("return", "return", "Return", "L2"), + ), + edges=(sdk.LogicEdge("entry", "return", "return", "RETURN", 0),), + ) + + def load_manifest(self) -> sdk.AdapterManifest: + fingerprint = self._hash("def run(): return True\n") + return sdk.AdapterManifest( + project_id="sdk-reference", + title="SDK reference", + adapter_id="python-reference", + adapter_version="1", + root=self.root, + revision=fingerprint[:12], + source_hash=fingerprint, + families=("source",), + allowed_relations=(), + sources=( + sdk.AdapterSource( + source_id="source.reference", + source_path="src/reference.py", + fingerprint=fingerprint, + extractor_version="python-reference@1", + ), + ), + estimated_nodes=1, + ) + + def extract_source(self, source: sdk.AdapterSource) -> sdk.AdapterSourceProjection: + return sdk.AdapterSourceProjection( + source_id=source.source_id, + fingerprint=source.fingerprint, + nodes=(self._node(),), + edges=(), + logic=(self._logic(),), + ) + + def load_projection(self) -> sdk.AdapterProjection: + manifest = self.load_manifest() + return sdk.AdapterProjection( + project_id=manifest.project_id, + title=manifest.title, + adapter_id=manifest.adapter_id, + adapter_version=manifest.adapter_version, + root=manifest.root, + revision=manifest.revision, + source_hash=manifest.source_hash, + nodes=(self._node(),), + edges=(), + ) + + def load_complete_assembly(self) -> sdk.AdapterAssembly: + return sdk.AdapterAssembly( + projection=self.load_projection(), + logic=(self._logic(),), + ) + + +class LegacyAdapter: + def __init__(self, projection: sdk.AdapterProjection) -> None: + self.projection = projection + + def load_projection(self) -> sdk.AdapterProjection: + return self.projection + + +class LogicWithoutCompleteAdapter: + def __init__(self, root: Path) -> None: + self.reference = ReferenceIncrementalAdapter(root) + + def load_manifest(self) -> sdk.AdapterManifest: + return self.reference.load_manifest() + + def extract_source(self, source: sdk.AdapterSource) -> sdk.AdapterSourceProjection: + return self.reference.extract_source(source) + + def load_projection(self) -> sdk.AdapterProjection: + return self.reference.load_projection() + + +class NondeterministicCompleteAdapter(ReferenceIncrementalAdapter): + def __init__(self, root: Path) -> None: + super().__init__(root) + self.complete_calls = 0 + + def load_complete_assembly(self) -> sdk.AdapterAssembly: + self.complete_calls += 1 + assembly = super().load_complete_assembly() + if self.complete_calls % 2 == 0: + logic = assembly.logic[0] + return replace( + assembly, + logic=( + replace( + logic, + nodes=( + logic.nodes[0], + replace(logic.nodes[1], label="Unstable return"), + ), + ), + ), + ) + return assembly + + +class AdapterSdkTests(unittest.TestCase): + def test_public_facade_contains_the_complete_authoring_surface(self) -> None: + expected = { + "AdapterAssembly", + "AdapterConformanceReport", + "AdapterManifest", + "AdapterNode", + "AdapterProjection", + "AdapterSource", + "AdapterSourceProjection", + "CompleteAdapterAssemblyLoader", + "Edge", + "LogicEdge", + "LogicNode", + "LogicProjection", + "Node", + "verify_adapter_conformance", + } + + self.assertLessEqual(expected, set(sdk.__all__)) + for name in expected: + with self.subTest(name=name): + self.assertTrue(hasattr(sdk, name)) + + def test_conformance_proves_deterministic_complete_graph_and_logic_parity(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory).resolve() + report = sdk.verify_adapter_conformance( + ReferenceIncrementalAdapter(root), + cache_root=root / ".cache" / "conformance", + ) + + self.assertEqual(1, report.schema_version) + self.assertTrue(report.incremental) + self.assertEqual(1, report.node_count) + self.assertEqual(0, report.edge_count) + self.assertEqual(1, report.logic_projection_count) + self.assertEqual(64, len(report.assembly_hash)) + self.assertEqual(report.assembly_hash, report.as_dict()["assembly_hash"]) + + def test_conformance_keeps_load_projection_only_adapters_first_class(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory).resolve() + content = "Legacy adapter content." + projection = sdk.AdapterProjection( + project_id="legacy-sdk", + title="Legacy SDK", + adapter_id="legacy", + adapter_version="1", + root=root, + revision="legacy", + source_hash=hashlib.sha256(content.encode()).hexdigest(), + nodes=( + sdk.AdapterNode( + sdk.Node( + node_id="legacy.node", + title="Legacy node", + family="guide", + authority="authoritative", + status="current", + tags=("legacy",), + summary=content, + content=content, + source_path="docs/legacy.md", + source_anchor=None, + content_hash=hashlib.sha256(content.encode()).hexdigest(), + ) + ), + ), + edges=(), + ) + + report = sdk.verify_adapter_conformance( + LegacyAdapter(projection), + cache_root=root / ".cache" / "legacy-conformance", + ) + + self.assertFalse(report.incremental) + self.assertEqual(0, report.logic_projection_count) + self.assertEqual(projection.identity(), projection.identity()) + + def test_conformance_requires_a_complete_oracle_for_incremental_logic(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory).resolve() + + with self.assertRaises(DocForgeError) as captured: + sdk.verify_adapter_conformance( + LogicWithoutCompleteAdapter(root), + cache_root=root / ".cache" / "missing-logic-oracle", + ) + + self.assertEqual("complete_logic_oracle_required", captured.exception.code) + + def test_conformance_rejects_nondeterministic_complete_logic(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory).resolve() + + with self.assertRaises(DocForgeError) as captured: + sdk.verify_adapter_conformance( + NondeterministicCompleteAdapter(root), + cache_root=root / ".cache" / "nondeterministic", + ) + + self.assertEqual("nondeterministic_adapter", captured.exception.code) + self.assertEqual(["logic"], captured.exception.details["fields"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_command_reference.py b/tests/test_command_reference.py new file mode 100644 index 0000000..e0f1741 --- /dev/null +++ b/tests/test_command_reference.py @@ -0,0 +1,141 @@ +from __future__ import annotations + +import argparse +import hashlib +import json +import shutil +import tempfile +import unittest +from pathlib import Path + +from mcp.shared.memory import create_connected_server_and_client_session +from mcp.types import Tool + +from docforge.cli import _parser +from docforge.command_reference import ( + cli_command_references, + mcp_tool_references, + render_cli_reference_markdown, + render_command_reference_markdown, + render_mcp_reference_markdown, +) +from docforge.index import ProjectIndex +from docforge.mcp_server import ALL_TOOLS, APPLICATION_TOOLS, create_server +from docforge.project import Project + +ROOT = Path(__file__).resolve().parents[1] +FIXTURES = ROOT / "tests" / "fixtures" + + +class CommandReferenceTests(unittest.IsolatedAsyncioTestCase): + def copy_fixture(self, destination: Path) -> Path: + root = destination / "alpha" + shutil.copytree(FIXTURES / "alpha", root) + return root + + def test_cli_reference_is_derived_from_the_real_parser(self) -> None: + parser = _parser() + references = cli_command_references(parser) + subparsers = next( + action + for action in parser._actions # pyright: ignore[reportPrivateUsage] + if isinstance( + action, + argparse._SubParsersAction, # pyright: ignore[reportPrivateUsage] + ) + ) + expected = { + name: " ".join(command.format_usage().removeprefix("usage: ").split()) + for name, command in subparsers.choices.items() + } + + self.assertEqual(tuple(sorted(expected)), tuple(item.name for item in references)) + self.assertEqual(expected, {item.name: item.invocation for item in references}) + rendered = render_cli_reference_markdown(references) + self.assertEqual(rendered, render_cli_reference_markdown(reversed(references))) + self.assertIn("| `search` | `docforge search [-h] [--limit LIMIT] query` |", rendered) + self.assertIn( + "| `graph-render-status` | `docforge graph-render-status [-h] [view_id]` |", + rendered, + ) + + async def test_mcp_reference_is_derived_from_registered_tool_metadata(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + ProjectIndex(Project.open(root)).build() + async with create_connected_server_and_client_session( + create_server( + root, + "alpha-editor", + canonical_applier_id="alpha-editor", + ), + raise_exceptions=True, + ) as session: + tools = (await session.list_tools()).tools + + expected_names = (*ALL_TOOLS, *APPLICATION_TOOLS) + references = mcp_tool_references(tools, expected_names=expected_names) + by_name = {item.name: item for item in references} + tools_by_name = {tool.name: tool for tool in tools} + + self.assertEqual(set(expected_names), set(by_name)) + for name, tool in tools_by_name.items(): + with self.subTest(tool=name): + schema_hash = hashlib.sha256( + json.dumps( + tool.inputSchema, + sort_keys=True, + separators=(",", ":"), + ).encode() + ).hexdigest() + properties = set(tool.inputSchema.get("properties", {})) + required = set(tool.inputSchema.get("required", [])) + reference = by_name[name] + self.assertEqual(" ".join((tool.description or "").split()), reference.description) + self.assertEqual(tuple(sorted(required)), reference.required_arguments) + self.assertEqual( + tuple(sorted(properties - required)), + reference.optional_arguments, + ) + self.assertEqual(schema_hash, reference.input_schema_hash) + + rendered = render_mcp_reference_markdown(references) + self.assertEqual(rendered, render_mcp_reference_markdown(reversed(references))) + self.assertIn("| read | `docforge_bootstrap` | — | — |", rendered) + self.assertIn( + "| application | `docforge_apply_changeset` | " + "`changeset_id`, `expected_changeset_hash` | — |", + rendered, + ) + complete = render_command_reference_markdown( + cli_command_references(), + references, + ) + self.assertEqual(1, complete.count("## CLI commands")) + self.assertEqual(1, complete.count("## MCP tools")) + self.assertTrue(complete.startswith("# DocForge command reference\n")) + self.assertIn("Do not edit this file by hand.", complete) + self.assertIn("fixed reference-adapter server exposes only its read rows", complete) + + def test_mcp_reference_fails_closed_on_registration_drift(self) -> None: + known = Tool( + name="docforge_bootstrap", + description="Bootstrap.", + inputSchema={"type": "object", "properties": {}}, + ) + unknown = Tool( + name="docforge_unregistered", + description="Unknown.", + inputSchema={"type": "object", "properties": {}}, + ) + + with self.assertRaisesRegex(ValueError, "no declared capability surface"): + mcp_tool_references((unknown,)) + with self.assertRaisesRegex(ValueError, "missing="): + mcp_tool_references((known,), expected_names=ALL_TOOLS) + with self.assertRaisesRegex(ValueError, "repeats"): + mcp_tool_references((known, known)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_command_reference_tool.py b/tests/test_command_reference_tool.py new file mode 100644 index 0000000..6498087 --- /dev/null +++ b/tests/test_command_reference_tool.py @@ -0,0 +1,404 @@ +from __future__ import annotations + +import asyncio +import contextlib +import hashlib +import io +import os +import tempfile +import threading +import unittest +from pathlib import Path +from unittest import mock + +from docforge.mcp_server import ALL_TOOLS, APPLICATION_TOOLS +from tools import generate_command_reference as command_reference_tool +from tools.generate_command_reference import ( + EXPECTED_CLI_ROWS, + EXPECTED_MCP_ROWS, + CommandReferenceDrift, + CommandReferenceToolError, + collect_command_reference_rows, + generate_command_reference_bytes, + main, + publish_or_check_command_reference, +) + +ROOT = Path(__file__).resolve().parents[1] +ALPHA = ROOT / "tests" / "fixtures" / "alpha" + + +def _tree_snapshot(root: Path) -> tuple[tuple[str, str, str], ...]: + records: list[tuple[str, str, str]] = [] + for path in sorted(root.rglob("*")): + relative = path.relative_to(root).as_posix() + if path.is_symlink(): + records.append((relative, "symlink", os.readlink(path))) + elif path.is_dir(): + records.append((relative, "directory", "")) + else: + records.append( + ( + relative, + "file", + hashlib.sha256(path.read_bytes()).hexdigest(), + ) + ) + return tuple(records) + + +class CommandReferenceToolTests(unittest.TestCase): + def test_generated_bytes_are_stable_and_never_mutate_the_fixture(self) -> None: + before = _tree_snapshot(ALPHA) + + first = generate_command_reference_bytes(ALPHA) + middle = _tree_snapshot(ALPHA) + second = generate_command_reference_bytes(ALPHA) + + self.assertEqual(first, second) + self.assertEqual(before, middle) + self.assertEqual(before, _tree_snapshot(ALPHA)) + self.assertTrue(first.endswith(b"\n")) + self.assertTrue(first.startswith(b"# DocForge command reference\n")) + self.assertIn(b"Do not edit this file by hand.", first) + self.assertEqual(1, first.count(b"## CLI commands")) + self.assertEqual(1, first.count(b"## MCP tools")) + + def test_registration_collection_is_complete(self) -> None: + cli_rows, mcp_rows = asyncio.run( + collect_command_reference_rows( + ALPHA, + proposal_writer="alpha-editor", + canonical_applier="alpha-editor", + ) + ) + + self.assertEqual(EXPECTED_CLI_ROWS, len(cli_rows)) + self.assertEqual(EXPECTED_MCP_ROWS, len(mcp_rows)) + self.assertEqual(len(cli_rows), len({item.name for item in cli_rows})) + self.assertEqual( + set((*ALL_TOOLS, *APPLICATION_TOOLS)), + {item.name for item in mcp_rows}, + ) + self.assertEqual( + {"read", "proposal", "application"}, + {item.surface for item in mcp_rows}, + ) + + def test_atomic_write_check_and_drift_detection(self) -> None: + content = generate_command_reference_bytes(ALPHA) + with tempfile.TemporaryDirectory() as directory: + repository = Path(directory).resolve() + (repository / "generated").mkdir() + output = Path("generated/command-reference.md") + target = repository / output + + self.assertEqual( + "written", + publish_or_check_command_reference( + content, + repository_root=repository, + project_root=ALPHA, + output=output, + check=False, + ), + ) + self.assertEqual(content, target.read_bytes()) + self.assertEqual( + "current", + publish_or_check_command_reference( + content, + repository_root=repository, + project_root=ALPHA, + output=output, + check=True, + ), + ) + target.write_text("stale\n", encoding="utf-8") + with self.assertRaises(CommandReferenceDrift): + publish_or_check_command_reference( + content, + repository_root=repository, + project_root=ALPHA, + output=output, + check=True, + ) + self.assertEqual(b"stale\n", target.read_bytes()) + self.assertEqual( + "written", + publish_or_check_command_reference( + content, + repository_root=repository, + project_root=ALPHA, + output=output, + check=False, + ), + ) + self.assertEqual(content, target.read_bytes()) + self.assertEqual( + "unchanged", + publish_or_check_command_reference( + content, + repository_root=repository, + project_root=ALPHA, + output=output, + check=False, + ), + ) + + def test_output_confinement_rejects_escapes_and_symlinks(self) -> None: + content = b"reference\n" + with tempfile.TemporaryDirectory() as directory: + repository = Path(directory).resolve() + generated = repository / "generated" + generated.mkdir() + outside = repository.parent / f"{repository.name}-outside.md" + symlink = generated / "reference.md" + symlink.symlink_to(outside) + + for output in (Path("../escape.md"), Path("generated/reference.md")): + with ( + self.subTest(output=output), + self.assertRaises(CommandReferenceToolError), + ): + publish_or_check_command_reference( + content, + repository_root=repository, + project_root=ALPHA, + output=output, + check=False, + ) + symlink.unlink() + (repository / "linked").symlink_to(generated, target_is_directory=True) + with self.assertRaises(CommandReferenceToolError): + publish_or_check_command_reference( + content, + repository_root=repository, + project_root=ALPHA, + output=Path("linked/reference.md"), + check=False, + ) + + with self.assertRaises(CommandReferenceToolError): + publish_or_check_command_reference( + content, + repository_root=ROOT, + project_root=ALPHA, + output=Path("tests/fixtures/alpha/generated-reference.md"), + check=False, + ) + + def test_existing_target_race_is_restored_without_overwrite(self) -> None: + with tempfile.TemporaryDirectory() as directory: + repository = Path(directory).resolve() + generated = repository / "generated" + generated.mkdir() + output = Path("generated/reference.md") + target = repository / output + target.write_bytes(b"initial\n") + original_exchange = command_reference_tool._rename_exchange + injected = False + + def exchange_after_race(directory_fd: int, first: str, second: str) -> None: + nonlocal injected + if not injected: + injected = True + target.write_bytes(b"concurrent\n") + original_exchange(directory_fd, first, second) + + with ( + mock.patch.object( + command_reference_tool, + "_rename_exchange", + side_effect=exchange_after_race, + ), + self.assertRaisesRegex( + CommandReferenceToolError, + "changed during atomic publication", + ), + ): + publish_or_check_command_reference( + b"generated\n", + repository_root=repository, + project_root=ALPHA, + output=output, + check=False, + ) + + self.assertEqual(b"concurrent\n", target.read_bytes()) + self.assertEqual( + [], + list(generated.glob(".reference.md.docforge-command-reference-*")), + ) + + def test_missing_target_race_is_never_clobbered(self) -> None: + with tempfile.TemporaryDirectory() as directory: + repository = Path(directory).resolve() + generated = repository / "generated" + generated.mkdir() + output = Path("generated/reference.md") + target = repository / output + original_link = command_reference_tool._link_no_replace + injected = False + + def link_after_race(directory_fd: int, source: str, destination: str) -> None: + nonlocal injected + if not injected: + injected = True + target.write_bytes(b"concurrent\n") + original_link(directory_fd, source, destination) + + with ( + mock.patch.object( + command_reference_tool, + "_link_no_replace", + side_effect=link_after_race, + ), + self.assertRaisesRegex(CommandReferenceToolError, "appeared"), + ): + publish_or_check_command_reference( + b"generated\n", + repository_root=repository, + project_root=ALPHA, + output=output, + check=False, + ) + + self.assertEqual(b"concurrent\n", target.read_bytes()) + self.assertEqual( + [], + list(generated.glob(".reference.md.docforge-command-reference-*")), + ) + + def test_concurrent_generators_are_serialized_around_inspection_and_write(self) -> None: + with tempfile.TemporaryDirectory() as directory: + repository = Path(directory).resolve() + (repository / "generated").mkdir() + output = Path("generated/reference.md") + target = repository / output + target.write_bytes(b"initial\n") + first_entered = threading.Event() + release_first = threading.Event() + second_started = threading.Event() + second_entered = threading.Event() + results: list[str] = [] + errors: list[BaseException] = [] + original_replace = command_reference_tool._atomic_replace + invocation_count = 0 + invocation_lock = threading.Lock() + + def blocking_replace( + directory_fd: int, + name: str, + content: bytes, + *, + expected: object, + expected_content: bytes | None, + ) -> None: + nonlocal invocation_count + with invocation_lock: + invocation_count += 1 + invocation = invocation_count + if invocation == 1: + first_entered.set() + if not release_first.wait(timeout=5): + raise AssertionError("Timed out waiting to release first generator") + else: + second_entered.set() + original_replace( + directory_fd, + name, + content, + expected=expected, # type: ignore[arg-type] + expected_content=expected_content, + ) + + def publish(content: bytes, *, started: threading.Event | None = None) -> None: + if started is not None: + started.set() + try: + results.append( + publish_or_check_command_reference( + content, + repository_root=repository, + project_root=ALPHA, + output=output, + check=False, + ) + ) + except BaseException as error: + errors.append(error) + + with mock.patch.object( + command_reference_tool, + "_atomic_replace", + side_effect=blocking_replace, + ): + first = threading.Thread(target=publish, args=(b"first\n",)) + second = threading.Thread( + target=publish, + args=(b"second\n",), + kwargs={"started": second_started}, + ) + first.start() + self.assertTrue(first_entered.wait(timeout=5)) + second.start() + self.assertTrue(second_started.wait(timeout=5)) + self.assertFalse(second_entered.wait(timeout=0.1)) + release_first.set() + first.join(timeout=5) + second.join(timeout=5) + + self.assertFalse(first.is_alive()) + self.assertFalse(second.is_alive()) + self.assertEqual([], errors) + self.assertEqual(["written", "written"], results) + self.assertEqual(b"second\n", target.read_bytes()) + + def test_command_runs_with_explicit_repository_and_project_roots(self) -> None: + with tempfile.TemporaryDirectory() as directory: + repository = Path(directory).resolve() + (repository / "generated").mkdir() + output = Path("generated/reference.md") + stdout = io.StringIO() + + with contextlib.redirect_stdout(stdout): + status = main( + ( + "--repository-root", + str(repository), + "--project-root", + str(ALPHA), + "--output", + str(output), + ) + ) + + self.assertEqual(0, status) + self.assertTrue((repository / output).is_file()) + self.assertIn('"cli_rows":28', stdout.getvalue()) + self.assertIn('"mcp_rows":36', stdout.getvalue()) + + target = repository / output + target.write_text("drift\n", encoding="utf-8") + stderr = io.StringIO() + with contextlib.redirect_stderr(stderr): + check_status = main( + ( + "--repository-root", + str(repository), + "--project-root", + str(ALPHA), + "--output", + str(output), + "--check", + ) + ) + + self.assertEqual(1, check_status) + self.assertEqual("drift\n", target.read_text(encoding="utf-8")) + self.assertIn("missing or stale", stderr.getvalue()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_cpp_reference_adapter.py b/tests/test_cpp_reference_adapter.py new file mode 100644 index 0000000..70c3bdf --- /dev/null +++ b/tests/test_cpp_reference_adapter.py @@ -0,0 +1,486 @@ +from __future__ import annotations + +import json +import shutil +import tempfile +import unittest +from pathlib import Path +from typing import cast +from unittest.mock import patch + +from docforge.adapter_sdk import ( + AdapterProject, + AdapterSource, + AdapterSourceProjection, + verify_adapter_conformance, +) +from docforge.adapters.cpp import ( + CPP_ADAPTER_ID, + CPP_ADAPTER_VERSION, + CPP_EXTRACTOR_VERSION, + CppReferenceAdapter, +) +from docforge.errors import DocForgeError +from docforge.index import ProjectIndex + +ROOT = Path(__file__).resolve().parents[1] +FIXTURE = ROOT / "tests" / "fixtures" / "reference-cpp" + + +class RecordingCppAdapter(CppReferenceAdapter): + def __init__(self, root: Path) -> None: + super().__init__( + root, + source_roots=("include", "src"), + compilation_database="compile_commands.json", + project_id="reference-cpp", + title="Reference C++ fixture", + ) + self.extracted_paths: list[str] = [] + + def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: + self.extracted_paths.append(source.source_path) + return super().extract_source(source) + + +class CppReferenceAdapterTests(unittest.TestCase): + def copy_fixture(self, parent: Path) -> Path: + root = parent / "reference-cpp" + shutil.copytree(FIXTURE, root) + return root.resolve() + + def adapter(self, root: Path) -> RecordingCppAdapter: + return RecordingCppAdapter(root) + + @staticmethod + def build_metrics(result: dict[str, object]) -> dict[str, object]: + return cast(dict[str, object], result["build"]) + + @staticmethod + def read_database(root: Path) -> list[dict[str, object]]: + return cast( + list[dict[str, object]], + json.loads((root / "compile_commands.json").read_text(encoding="utf-8")), + ) + + @staticmethod + def write_database(root: Path, entries: list[dict[str, object]]) -> None: + (root / "compile_commands.json").write_text( + json.dumps(entries, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + def test_complete_assembly_is_deterministic_conformant_and_syntax_scoped(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + adapter = self.adapter(root) + + first = adapter.load_assembly() + second = adapter.load_complete_assembly() + projection = adapter.load_projection() + + self.assertEqual(first, second) + self.assertEqual(first.projection, projection) + self.assertEqual(CPP_ADAPTER_ID, projection.adapter_id) + self.assertEqual(CPP_ADAPTER_VERSION, projection.adapter_version) + self.assertEqual(17, len(projection.nodes)) + self.assertEqual(16, len(projection.edges)) + self.assertEqual(5, len(first.logic)) + self.assertEqual( + list(projection.nodes), + sorted(projection.nodes, key=lambda item: item.node.node_id), + ) + self.assertEqual( + {"contains", "depends_on"}, + {item.edge.relation for item in projection.edges}, + ) + self.assertTrue( + any( + item.node.title == "demo.Worker" and ("kind", "class") in item.metadata + for item in projection.nodes + ) + ) + self.assertTrue( + any( + item.node.title == "demo.Config" and ("kind", "struct") in item.metadata + for item in projection.nodes + ) + ) + self.assertTrue( + any( + item.node.title == "demo.Worker.run" and ("kind", "function") in item.metadata + for item in projection.nodes + ) + ) + self.assertEqual( + {"include/config.hpp", "include/detail.hpp", "src/main.cpp", "src/worker.cpp"}, + {item.node.source_path for item in projection.nodes}, + ) + self.assertFalse( + {"calls", "inherits", "type_of"} & {item.edge.relation for item in projection.edges} + ) + + report = adapter.support_report() + self.assertEqual(1, report["schema_version"]) + self.assertEqual(CPP_EXTRACTOR_VERSION, report["extractor_version"]) + self.assertEqual(["include", "src"], report["source_roots"]) + self.assertTrue(report["compilation_database_is_authoritative"]) + self.assertFalse(report["executes_compiler"]) + self.assertFalse(report["executes_project_code"]) + self.assertEqual( + [ + "call_resolution", + "type_resolution", + "inheritance_resolution", + "overload_ownership", + "macro_semantics", + "compiler_command_execution", + "nonlocal_include_resolution", + ], + [item.code for item in adapter.unsupported_facts()], + ) + self.assertFalse((root / ".cache").exists()) + + conformance = verify_adapter_conformance( + adapter, + cache_root=root / ".cache" / "conformance", + ) + self.assertEqual("reference-cpp", conformance.project_id) + self.assertEqual(17, conformance.node_count) + self.assertEqual(16, conformance.edge_count) + self.assertEqual(5, conformance.logic_projection_count) + self.assertTrue(conformance.incremental) + + def test_incremental_cold_warm_source_header_and_flag_invalidation(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + adapter = self.adapter(root) + project = AdapterProject(adapter, cache_root=root / ".cache" / "incremental") + index = ProjectIndex(project) + + cold = index.build() + self.assertEqual(4, self.build_metrics(cold)["reparsed_sources"]) + self.assertEqual( + [ + "include/config.hpp", + "include/detail.hpp", + "src/main.cpp", + "src/worker.cpp", + ], + sorted(adapter.extracted_paths), + ) + + adapter.extracted_paths.clear() + warm = index.build() + self.assertEqual(4, self.build_metrics(warm)["cache_hits"]) + self.assertEqual(0, self.build_metrics(warm)["reparsed_sources"]) + self.assertEqual([], adapter.extracted_paths) + + main = root / "src" / "main.cpp" + main.write_text( + main.read_text(encoding="utf-8").replace( + "return demo::execute(3);", + "return demo::execute(4);", + ), + encoding="utf-8", + ) + adapter.extracted_paths.clear() + source_changed = index.build() + self.assertEqual(1, self.build_metrics(source_changed)["invalidated_sources"]) + self.assertEqual(["src/main.cpp"], adapter.extracted_paths) + + detail = root / "include" / "detail.hpp" + detail.write_text( + detail.read_text(encoding="utf-8").replace( + "return value;", + "return value + 0;", + ), + encoding="utf-8", + ) + adapter.extracted_paths.clear() + header_changed = index.build() + self.assertEqual(4, self.build_metrics(header_changed)["invalidated_sources"]) + self.assertEqual( + [ + "include/config.hpp", + "include/detail.hpp", + "src/main.cpp", + "src/worker.cpp", + ], + sorted(adapter.extracted_paths), + ) + + entries = self.read_database(root) + arguments = cast(list[str], entries[0]["arguments"]) + arguments.append("-DREFERENCE_FLAG=1") + self.write_database(root, entries) + adapter.extracted_paths.clear() + flag_changed = index.build() + self.assertEqual(1, self.build_metrics(flag_changed)["invalidated_sources"]) + self.assertEqual(["src/main.cpp"], adapter.extracted_paths) + + equivalent = project.verify_incremental_equivalence() + self.assertEqual("ok", equivalent["status"]) + self.assertEqual(17, equivalent["node_count"]) + self.assertEqual(16, equivalent["edge_count"]) + self.assertEqual(5, equivalent["logic_projection_count"]) + + def test_add_delete_and_corrupt_cache_recover_without_stale_facts(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + adapter = self.adapter(root) + cache_root = root / ".cache" / "incremental" + project = AdapterProject(adapter, cache_root=cache_root) + index = ProjectIndex(project) + index.build() + + extra = root / "src" / "extra.cpp" + extra.write_text( + '#include "../include/config.hpp"\n\n' + "int extra(int value) {\n" + " return demo::normalize(value);\n" + "}\n", + encoding="utf-8", + ) + entries = self.read_database(root) + entries.append( + { + "arguments": ["c++", "-std=c++20", "-c", "src/extra.cpp"], + "directory": ".", + "file": "src/extra.cpp", + } + ) + self.write_database(root, entries) + adapter.extracted_paths.clear() + added = index.build() + self.assertEqual(1, self.build_metrics(added)["reparsed_sources"]) + self.assertEqual(["src/extra.cpp"], adapter.extracted_paths) + self.assertIn( + "src/extra.cpp", + {node.source_path for node in project.load().nodes}, + ) + + worker = root / "src" / "worker.cpp" + entries = [ + entry for entry in self.read_database(root) if entry["file"] != "src/worker.cpp" + ] + self.write_database(root, entries) + worker.unlink() + adapter.extracted_paths.clear() + deleted = index.build() + self.assertEqual(1, self.build_metrics(deleted)["deleted_sources"]) + self.assertNotIn( + "src/worker.cpp", + {node.source_path for node in project.load().nodes}, + ) + + (cache_root / "extractions.json").write_text("{broken", encoding="utf-8") + adapter.extracted_paths.clear() + recovered = index.build() + self.assertEqual(4, self.build_metrics(recovered)["reparsed_sources"]) + self.assertEqual( + [ + "include/config.hpp", + "include/detail.hpp", + "src/extra.cpp", + "src/main.cpp", + ], + sorted(adapter.extracted_paths), + ) + self.assertEqual("ok", project.verify_incremental_equivalence()["status"]) + + def test_database_shape_bounds_duplicates_and_paths_fail_closed(self) -> None: + invalid_documents: list[tuple[str, list[dict[str, object]], str]] = [] + baseline = cast( + list[dict[str, object]], + json.loads((FIXTURE / "compile_commands.json").read_text(encoding="utf-8")), + ) + + both = json.loads(json.dumps(baseline)) + both[0]["command"] = "c++ -c src/main.cpp" + invalid_documents.append(("both command forms", both, "invalid_compilation_database")) + + neither = json.loads(json.dumps(baseline)) + del neither[1]["command"] + invalid_documents.append(("neither command form", neither, "invalid_compilation_database")) + + bad_argument = json.loads(json.dumps(baseline)) + bad_argument[0]["arguments"] = ["c++", 7] + invalid_documents.append( + ("non-string argument", bad_argument, "invalid_compilation_database") + ) + + control_command = json.loads(json.dumps(baseline)) + control_command[1]["command"] = "c++ -c src/worker.cpp\nmalicious" + invalid_documents.append( + ("control command", control_command, "invalid_compilation_database") + ) + + unknown_key = json.loads(json.dumps(baseline)) + unknown_key[0]["environment"] = {"PATH": "/tmp"} + invalid_documents.append(("unknown key", unknown_key, "invalid_compilation_database")) + + duplicate = json.loads(json.dumps(baseline)) + duplicate.append(json.loads(json.dumps(duplicate[0]))) + invalid_documents.append(("duplicate TU", duplicate, "duplicate_translation_unit")) + + escaped_directory = json.loads(json.dumps(baseline)) + escaped_directory[0]["directory"] = ".." + invalid_documents.append(("escaped directory", escaped_directory, "path_escape")) + + escaped_file = json.loads(json.dumps(baseline)) + escaped_file[0]["file"] = "../outside.cpp" + invalid_documents.append(("escaped file", escaped_file, "path_escape")) + + for name, document, code in invalid_documents: + with self.subTest(name=name), tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + self.write_database(root, document) + adapter = self.adapter(root) + with self.assertRaises(DocForgeError) as captured: + adapter.load_manifest() + self.assertEqual(code, captured.exception.code) + + with tempfile.TemporaryDirectory() as directory: + parent = Path(directory) + root = self.copy_fixture(parent) + outside_database = parent / "compile_commands.json" + outside_database.write_text("[]\n", encoding="utf-8") + database = root / "compile_commands.json" + database.unlink() + database.symlink_to(outside_database) + with self.assertRaises(DocForgeError) as database_link: + self.adapter(root) + self.assertEqual("path_escape", database_link.exception.code) + + with tempfile.TemporaryDirectory() as directory: + parent = Path(directory) + root = self.copy_fixture(parent) + outside_source = parent / "worker.cpp" + outside_source.write_text("int outside() { return 1; }\n", encoding="utf-8") + worker = root / "src" / "worker.cpp" + worker.unlink() + worker.symlink_to(outside_source) + adapter = self.adapter(root) + with self.assertRaises(DocForgeError) as source_link: + adapter.load_manifest() + self.assertEqual("path_escape", source_link.exception.code) + + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + bounded_adapters = ( + ( + CppReferenceAdapter( + root, + source_roots=("include", "src"), + compilation_database="compile_commands.json", + max_database_bytes=1, + ), + "source_too_large", + ), + ( + CppReferenceAdapter( + root, + source_roots=("include", "src"), + compilation_database="compile_commands.json", + max_source_bytes=1, + ), + "source_too_large", + ), + ( + CppReferenceAdapter( + root, + source_roots=("include", "src"), + compilation_database="compile_commands.json", + max_command_chars=32, + ), + "invalid_compilation_database", + ), + ( + CppReferenceAdapter( + root, + source_roots=("include", "src"), + compilation_database="compile_commands.json", + max_arguments=2, + ), + "invalid_compilation_database", + ), + ( + CppReferenceAdapter( + root, + source_roots=("include", "src"), + compilation_database="compile_commands.json", + max_translation_units=1, + ), + "adapter_too_large", + ), + ( + CppReferenceAdapter( + root, + source_roots=("include", "src"), + compilation_database="compile_commands.json", + max_headers=1, + ), + "adapter_too_large", + ), + ) + for adapter, code in bounded_adapters: + with self.subTest(limit=code), self.assertRaises(DocForgeError) as bounded: + adapter.load_manifest() + self.assertEqual(code, bounded.exception.code) + + def test_source_roots_no_ast_and_optional_extra_fail_closed(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + with self.assertRaises(DocForgeError) as translation_unit_outside: + CppReferenceAdapter( + root, + source_roots=("include",), + compilation_database="compile_commands.json", + ).load_manifest() + self.assertEqual("path_escape", translation_unit_outside.exception.code) + + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + other = root / "other" + other.mkdir() + (other / "outside.hpp").write_text( + "#pragma once\nstruct Outside {};\n", + encoding="utf-8", + ) + main = root / "src" / "main.cpp" + main.write_text( + '#include "../other/outside.hpp"\n' + main.read_text(encoding="utf-8"), + encoding="utf-8", + ) + with self.assertRaises(DocForgeError) as header_outside: + self.adapter(root).load_manifest() + self.assertEqual("path_escape", header_outside.exception.code) + + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + adapter = self.adapter(root) + project = AdapterProject(adapter, cache_root=root / ".cache" / "no-ast") + with self.assertRaises(DocForgeError) as no_ast: + ProjectIndex(project, allow_logic=False).build() + self.assertEqual("adapter_policy_forbids_logic", no_ast.exception.code) + self.assertFalse(project.descriptor.index_path.exists()) + + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + adapter = self.adapter(root) + missing = ModuleNotFoundError( + "No module named 'tree_sitter'", + name="tree_sitter", + ) + with ( + patch( + "docforge.adapters.cpp.importlib.import_module", + side_effect=missing, + ), + self.assertRaises(DocForgeError) as optional, + ): + adapter.load_manifest() + self.assertEqual("optional_dependency_missing", optional.exception.code) + self.assertEqual("cpp", optional.exception.details["extra"]) + self.assertEqual("docforge[cpp]", optional.exception.details["install"]) diff --git a/tests/test_docs_check.py b/tests/test_docs_check.py new file mode 100644 index 0000000..d4cb2e4 --- /dev/null +++ b/tests/test_docs_check.py @@ -0,0 +1,198 @@ +from __future__ import annotations + +import shutil +import tempfile +import unittest +from pathlib import Path + +from tools.check_documentation import ( + COMMAND_REFERENCE_NOTICE, + MAX_DIAGNOSTICS, + DocumentationPolicy, + check_documentation, +) + +ROOT = Path(__file__).resolve().parents[1] +SCHEMA = ROOT / "schemas/reference-adapter.schema.json" + + +class DocumentationCheckTests(unittest.TestCase): + def repository(self) -> Path: + root = Path(self.enterContext(tempfile.TemporaryDirectory())) + (root / "docs").mkdir() + (root / "schemas").mkdir() + shutil.copyfile(SCHEMA, root / "schemas/reference-adapter.schema.json") + (root / "pending.txt").write_text("", encoding="utf-8") + return root + + @staticmethod + def policy(*required: str) -> DocumentationPolicy: + return DocumentationPolicy( + required_pages=tuple(Path(item) for item in required), + pending_inventory=Path("pending.txt"), + ) + + @staticmethod + def command_reference() -> str: + return f"# DocForge command reference\n\n{COMMAND_REFERENCE_NOTICE}\n\n## CLI commands\n" + + def test_valid_graph_h1_anchors_and_reference_adapter_toml_pass(self) -> None: + root = self.repository() + (root / "README.md").write_text( + "# Documentation\n\n" + "[Guide](docs/GUIDE.md#adapter-setup) and " + "[commands](docs/COMMAND_REFERENCE.md). [Top](#documentation)\n", + encoding="utf-8", + ) + (root / "docs/GUIDE.md").write_text( + "# Guide\n\n" + "## Adapter *setup*\n\n" + "Reference adapter configuration `.docforge/reference-adapter.toml`:\n\n" + "```toml\n" + "schema_version = 1\n" + 'project_id = "example-project"\n' + 'title = "Example"\n' + 'language = "python"\n' + 'source_roots = ["src"]\n' + "```\n\n" + "[Back](../README.md#documentation)\n", + encoding="utf-8", + ) + (root / "docs/COMMAND_REFERENCE.md").write_text( + self.command_reference(), + encoding="utf-8", + ) + + report = check_documentation( + root, + policy=self.policy( + "README.md", + "docs/GUIDE.md", + "docs/COMMAND_REFERENCE.md", + ), + ) + + self.assertTrue(report.ok, [item.render() for item in report.diagnostics]) + self.assertEqual(3, report.markdown_pages) + self.assertEqual(1, report.reference_adapter_examples) + + def test_links_anchors_h1_and_generated_notice_fail_closed(self) -> None: + root = self.repository() + (root / "README.md").write_text( + "# Documentation\n\n" + "[bad anchor](docs/GUIDE.md#missing) " + "[missing file](docs/ABSENT.md) " + "[commands](docs/COMMAND_REFERENCE.md).\n", + encoding="utf-8", + ) + (root / "docs/GUIDE.md").write_text( + "# Guide\n\n# Duplicate\n", + encoding="utf-8", + ) + (root / "docs/COMMAND_REFERENCE.md").write_text( + "# DocForge command reference\n\n## CLI commands\n", + encoding="utf-8", + ) + + report = check_documentation( + root, + policy=self.policy( + "README.md", + "docs/GUIDE.md", + "docs/COMMAND_REFERENCE.md", + ), + ) + + self.assertFalse(report.ok) + self.assertEqual( + {"DOC016", "DOC017", "DOC020", "DOC028"}, + {item.code for item in report.diagnostics}, + ) + + def test_only_prior_milestone_records_are_reachability_exempt(self) -> None: + root = self.repository() + (root / "README.md").write_text("# Documentation\n", encoding="utf-8") + (root / "docs/MILESTONE_2_BASELINE.md").write_text( + "# Historical evidence\n", + encoding="utf-8", + ) + (root / "docs/ORPHAN.md").write_text("# Orphan\n", encoding="utf-8") + + report = check_documentation( + root, + policy=self.policy("README.md"), + ) + + orphans = [item.path for item in report.diagnostics if item.code == "DOC023"] + self.assertEqual(["docs/ORPHAN.md"], orphans) + + def test_pending_page_becomes_a_hard_failure_as_soon_as_it_exists(self) -> None: + root = self.repository() + (root / "README.md").write_text("# Documentation\n", encoding="utf-8") + (root / "pending.txt").write_text("docs/FUTURE.md\n", encoding="utf-8") + policy = self.policy("README.md", "docs/FUTURE.md") + + pending = check_documentation(root, policy=policy) + + self.assertTrue(pending.ok) + self.assertEqual(1, pending.pending_pages) + + (root / "docs/FUTURE.md").write_text("# Future\n", encoding="utf-8") + stale = check_documentation(root, policy=policy) + + self.assertIn("DOC014", {item.code for item in stale.diagnostics}) + self.assertIn("DOC023", {item.code for item in stale.diagnostics}) + + def test_unknown_pending_page_and_invalid_toml_are_reported(self) -> None: + root = self.repository() + (root / "README.md").write_text( + "# Documentation\n\n[Adapter](docs/ADAPTER.md)\n", + encoding="utf-8", + ) + (root / "docs/ADAPTER.md").write_text( + "# Adapter\n\n" + "```toml reference-adapter\n" + "schema_version = 1\n" + 'project_id = "cpp-example"\n' + 'title = "C++"\n' + 'language = "cpp"\n' + 'source_roots = ["src"]\n' + "```\n", + encoding="utf-8", + ) + (root / "pending.txt").write_text("docs/UNKNOWN.md\n", encoding="utf-8") + + report = check_documentation( + root, + policy=self.policy("README.md", "docs/ADAPTER.md"), + ) + + self.assertIn("DOC012", {item.code for item in report.diagnostics}) + schema_failures = [item for item in report.diagnostics if item.code == "DOC026"] + self.assertEqual(1, len(schema_failures)) + self.assertIn("compilation_database", schema_failures[0].message) + + def test_diagnostics_are_deterministically_bounded(self) -> None: + root = self.repository() + (root / "README.md").write_text("# Documentation\n", encoding="utf-8") + for index in range(MAX_DIAGNOSTICS + 7): + (root / "docs" / f"ORPHAN_{index:03d}.md").write_text( + f"# Orphan {index}\n", + encoding="utf-8", + ) + + report = check_documentation( + root, + policy=self.policy("README.md"), + ) + + self.assertEqual(MAX_DIAGNOSTICS, len(report.diagnostics)) + self.assertEqual(7, report.omitted_diagnostics) + self.assertEqual( + sorted(report.diagnostics), + list(report.diagnostics), + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_documentation_check.py b/tests/test_documentation_check.py new file mode 100644 index 0000000..de2d851 --- /dev/null +++ b/tests/test_documentation_check.py @@ -0,0 +1,176 @@ +from __future__ import annotations + +import json +import tempfile +import unittest +from pathlib import Path + +from tools.check_documentation import ( + DocumentationPolicy, + check_documentation, +) + + +class DocumentationCheckTests(unittest.TestCase): + def repository(self, parent: Path) -> Path: + root = parent / "repository" + (root / "docs").mkdir(parents=True) + (root / "schemas").mkdir() + (root / "schemas" / "reference-adapter.schema.json").write_text( + json.dumps( + { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "additionalProperties": False, + "required": [ + "schema_version", + "project_id", + "title", + "language", + "source_roots", + ], + "properties": { + "schema_version": {"const": 1}, + "project_id": {"type": "string"}, + "title": {"type": "string"}, + "language": {"enum": ["python", "javascript", "typescript", "cpp"]}, + "source_roots": { + "type": "array", + "minItems": 1, + "items": {"type": "string"}, + }, + "compilation_database": {"type": "string"}, + }, + } + ), + encoding="utf-8", + ) + return root + + def test_linked_pages_h1_anchors_and_reference_example_pass(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.repository(Path(directory)) + (root / "README.md").write_text( + "# Project\n\n[Guide](docs/GUIDE.md#reference-adapter-configuration)\n", + encoding="utf-8", + ) + (root / "docs" / "GUIDE.md").write_text( + "\n".join( + ( + "# Guide", + "", + "## Reference adapter configuration", + "", + "`.docforge/reference-adapter.toml`:", + "", + "```toml", + "schema_version = 1", + 'project_id = "fixture"', + 'title = "Fixture"', + 'language = "python"', + 'source_roots = ["src"]', + "```", + "", + ) + ), + encoding="utf-8", + ) + + report = check_documentation( + root, + policy=DocumentationPolicy( + required_pages=(Path("README.md"), Path("docs/GUIDE.md")), + pending_inventory=None, + ), + ) + + self.assertTrue(report.ok, [item.render() for item in report.diagnostics]) + self.assertEqual(2, report.markdown_pages) + self.assertEqual(1, report.reference_adapter_examples) + + def test_missing_anchor_orphan_and_duplicate_h1_fail_deterministically(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.repository(Path(directory)) + (root / "README.md").write_text( + "# Project\n\n[Broken](docs/GUIDE.md#missing)\n", + encoding="utf-8", + ) + (root / "docs" / "GUIDE.md").write_text( + "# Guide\n\n# Duplicate\n", + encoding="utf-8", + ) + (root / "docs" / "ORPHAN.md").write_text("# Orphan\n", encoding="utf-8") + + report = check_documentation( + root, + policy=DocumentationPolicy( + required_pages=(Path("README.md"), Path("docs/GUIDE.md")), + pending_inventory=None, + ), + ) + + self.assertFalse(report.ok) + codes = [item.code for item in report.diagnostics] + self.assertEqual(["DOC016", "DOC017", "DOC023"], sorted(codes)) + + def test_pending_inventory_allows_absence_but_rejects_existing_page(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.repository(Path(directory)) + (root / "README.md").write_text("# Project\n", encoding="utf-8") + pending_path = root / "pending.txt" + pending_path.write_text("docs/FUTURE.md\n", encoding="utf-8") + policy = DocumentationPolicy( + required_pages=(Path("README.md"), Path("docs/FUTURE.md")), + pending_inventory=Path("pending.txt"), + ) + + missing = check_documentation(root, policy=policy) + self.assertTrue(missing.ok) + self.assertEqual(1, missing.pending_pages) + + (root / "docs" / "FUTURE.md").write_text("# Future\n", encoding="utf-8") + stale = check_documentation(root, policy=policy) + self.assertEqual(["DOC014", "DOC023"], [item.code for item in stale.diagnostics]) + + def test_invalid_reference_example_reports_schema_path(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.repository(Path(directory)) + (root / "README.md").write_text( + "# Project\n\n[Guide](docs/GUIDE.md)\n", + encoding="utf-8", + ) + (root / "docs" / "GUIDE.md").write_text( + "\n".join( + ( + "# Guide", + "", + "Reference adapter configuration:", + "", + "```toml", + 'language = "python"', + 'source_roots = ["src"]', + "```", + "", + ) + ), + encoding="utf-8", + ) + + report = check_documentation( + root, + policy=DocumentationPolicy( + required_pages=(Path("README.md"), Path("docs/GUIDE.md")), + pending_inventory=None, + ), + ) + + schema_errors = [item for item in report.diagnostics if item.code == "DOC026"] + self.assertEqual(3, len(schema_errors)) + messages = " ".join(item.message for item in schema_errors) + self.assertIn("schema_version", messages) + self.assertIn("project_id", messages) + self.assertIn("title", messages) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_incremental_cache.py b/tests/test_incremental_cache.py new file mode 100644 index 0000000..cfd2a9d --- /dev/null +++ b/tests/test_incremental_cache.py @@ -0,0 +1,102 @@ +from __future__ import annotations + +import hashlib +import os +import tempfile +import unittest +from pathlib import Path + +from docforge.errors import DocForgeError +from docforge.incremental import ( + CachedSource, + ExtractionCache, + load_extraction_cache, + write_extraction_cache, +) + + +class ExtractionCacheBoundsTests(unittest.TestCase): + @staticmethod + def cache(*, payload: str = "ok", count: int = 1) -> ExtractionCache: + sources = tuple( + CachedSource( + source_id=f"source-{index}", + source_path=f"src/{index}.txt", + fingerprint=hashlib.sha256(str(index).encode()).hexdigest(), + extractor_version="fixture@1", + dependencies=(), + payload={"content": payload}, + ) + for index in range(count) + ) + return ExtractionCache("fixture", "fixture-adapter", "1", sources) + + def test_bounded_round_trip_and_oversized_read_is_a_cache_miss(self) -> None: + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "extractions.json" + write_extraction_cache(path, self.cache(), max_bytes=1_000, max_sources=2) + loaded = load_extraction_cache( + path, + project_id="fixture", + adapter_id="fixture-adapter", + adapter_version="1", + max_bytes=1_000, + max_sources=2, + ) + self.assertEqual(self.cache(), loaded) + + with path.open("ab") as handle: + handle.write(b" " * 1_000) + self.assertIsNone( + load_extraction_cache( + path, + project_id="fixture", + adapter_id="fixture-adapter", + adapter_version="1", + max_bytes=1_000, + max_sources=2, + ) + ) + + def test_write_rejects_byte_and_source_limits_without_replacing_target(self) -> None: + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "extractions.json" + path.write_bytes(b"preserved\n") + for cache, maximum_bytes, maximum_sources in ( + (self.cache(payload="x" * 500), 100, 2), + (self.cache(count=2), 1_000, 1), + ): + with self.assertRaises(DocForgeError) as captured: + write_extraction_cache( + path, + cache, + max_bytes=maximum_bytes, + max_sources=maximum_sources, + ) + self.assertEqual("cache_limit", captured.exception.code) + self.assertEqual(b"preserved\n", path.read_bytes()) + + def test_symlink_and_non_regular_targets_are_cache_misses(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + real = root / "real.json" + write_extraction_cache(real, self.cache(), max_bytes=1_000, max_sources=2) + linked = root / "linked.json" + linked.symlink_to(real) + fifo = root / "fifo" + os.mkfifo(fifo) + for path in (linked, fifo): + self.assertIsNone( + load_extraction_cache( + path, + project_id="fixture", + adapter_id="fixture-adapter", + adapter_version="1", + max_bytes=1_000, + max_sources=2, + ) + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_javascript_reference_adapter.py b/tests/test_javascript_reference_adapter.py new file mode 100644 index 0000000..14c5b96 --- /dev/null +++ b/tests/test_javascript_reference_adapter.py @@ -0,0 +1,267 @@ +from __future__ import annotations + +import shutil +import tempfile +import unittest +from pathlib import Path +from typing import cast +from unittest import mock + +import docforge.adapters.javascript as javascript_adapter +from docforge.adapter_sdk import ( + AdapterProject, + AdapterSource, + AdapterSourceProjection, + verify_adapter_conformance, +) +from docforge.adapters.javascript import ( + JAVASCRIPT_ADAPTER_ID, + JAVASCRIPT_ADAPTER_VERSION, + JAVASCRIPT_EXTRACTOR_VERSION, + JavaScriptReferenceAdapter, +) +from docforge.errors import DocForgeError +from docforge.index import ProjectIndex + +ROOT = Path(__file__).resolve().parents[1] +FIXTURES = ROOT / "tests" / "fixtures" + + +class RecordingScriptAdapter(JavaScriptReferenceAdapter): + def __init__(self, root: Path, project_id: str) -> None: + super().__init__( + root, + source_roots=("src",), + project_id=project_id, + title=f"{project_id} fixture", + ) + self.extracted_paths: list[str] = [] + + def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: + self.extracted_paths.append(source.source_path) + return super().extract_source(source) + + +class JavaScriptReferenceAdapterTests(unittest.TestCase): + CASES = ( + ("reference-javascript", "javascript", 14, 14, 5, "src/app/shared.js"), + ("reference-typescript", "typescript", 13, 14, 4, "src/app/types.mts"), + ) + + def copy_fixture(self, parent: Path, fixture: str) -> Path: + root = parent / fixture + shutil.copytree(FIXTURES / fixture, root) + return root.resolve() + + @staticmethod + def build_metrics(result: dict[str, object]) -> dict[str, object]: + return cast(dict[str, object], result["build"]) + + def test_both_grammars_produce_deterministic_complete_logic_assemblies(self) -> None: + for fixture, language, nodes, edges, logic, _shared in self.CASES: + with self.subTest(language=language), tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory), fixture) + adapter = RecordingScriptAdapter(root, fixture) + + first = adapter.load_assembly() + second = adapter.load_complete_assembly() + projection = adapter.load_projection() + + self.assertEqual(first, second) + self.assertEqual(first.projection, projection) + self.assertEqual(JAVASCRIPT_ADAPTER_ID, projection.adapter_id) + self.assertEqual(JAVASCRIPT_ADAPTER_VERSION, projection.adapter_version) + self.assertEqual(nodes, len(projection.nodes)) + self.assertEqual(edges, len(projection.edges)) + self.assertEqual(logic, len(first.logic)) + self.assertEqual( + {language}, + {dict(item.metadata)["language"] for item in projection.nodes}, + ) + self.assertEqual( + {"contains", "depends_on"}, + {item.edge.relation for item in projection.edges}, + ) + self.assertTrue( + any( + item.node.title == "Service.run" + and dict(item.metadata)["kind"] == "function" + for item in projection.nodes + ) + ) + self.assertFalse((root / ".cache").exists()) + + report = adapter.support_report() + self.assertEqual(1, report["schema_version"]) + self.assertEqual(JAVASCRIPT_EXTRACTOR_VERSION, report["extractor_version"]) + self.assertFalse(report["imports_project_code"]) + self.assertFalse(report["executes_project_code"]) + self.assertEqual( + [ + "call_resolution", + "dynamic_module_resolution", + "inheritance_resolution", + "module_configuration_resolution", + "runtime_generated_facts", + "type_and_symbol_resolution", + ], + [item.code for item in adapter.unsupported_facts()], + ) + + conformance = verify_adapter_conformance( + adapter, + cache_root=root / ".cache" / "conformance", + ) + self.assertEqual(nodes, conformance.node_count) + self.assertEqual(edges, conformance.edge_count) + self.assertEqual(logic, conformance.logic_projection_count) + self.assertTrue(conformance.incremental) + + def test_cold_warm_reverse_dependency_and_exact_equivalence_for_both_grammars( + self, + ) -> None: + for fixture, language, nodes, edges, logic, shared_path in self.CASES: + with self.subTest(language=language), tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory), fixture) + adapter = RecordingScriptAdapter(root, fixture) + project = AdapterProject(adapter, cache_root=root / ".cache" / "incremental") + index = ProjectIndex(project) + + cold = index.build() + self.assertEqual(4, self.build_metrics(cold)["reparsed_sources"]) + self.assertEqual(4, len(adapter.extracted_paths)) + + adapter.extracted_paths.clear() + with mock.patch.object( + adapter, + "_parse", + wraps=adapter._parse, # pyright: ignore[reportPrivateUsage] + ) as parsed: + warm = index.build() + self.assertEqual(4, self.build_metrics(warm)["cache_hits"]) + self.assertEqual(0, self.build_metrics(warm)["reparsed_sources"]) + self.assertEqual([], adapter.extracted_paths) + parsed.assert_not_called() + + shared = root / shared_path + shared.write_text( + shared.read_text(encoding="utf-8") + "\n// changed dependency evidence\n", + encoding="utf-8", + ) + adapter.extracted_paths.clear() + with mock.patch.object( + adapter, + "_parse", + wraps=adapter._parse, # pyright: ignore[reportPrivateUsage] + ) as parsed: + changed = index.build() + self.assertEqual(4, self.build_metrics(changed)["invalidated_sources"]) + self.assertEqual(4, len(adapter.extracted_paths)) + self.assertEqual(4, parsed.call_count) + + equivalent = project.verify_incremental_equivalence() + self.assertEqual("ok", equivalent["status"]) + self.assertEqual(nodes, equivalent["node_count"]) + self.assertEqual(edges, equivalent["edge_count"]) + self.assertEqual(logic, equivalent["logic_projection_count"]) + + def test_add_delete_and_corrupt_cache_recovery_for_both_grammars(self) -> None: + additions = { + "javascript": ( + "src/app/extra.js", + "export function extra() { return 'extra'; }\n", + ), + "typescript": ( + "src/app/extra.ts", + "export function extra(): string { return 'extra'; }\n", + ), + } + for fixture, language, _nodes, _edges, _logic, _shared_path in self.CASES: + with self.subTest(language=language), tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory), fixture) + adapter = RecordingScriptAdapter(root, fixture) + cache_root = root / ".cache" / "incremental" + project = AdapterProject(adapter, cache_root=cache_root) + index = ProjectIndex(project) + index.build() + + relative, content = additions[language] + added_path = root / relative + added_path.write_text(content, encoding="utf-8") + adapter.extracted_paths.clear() + added = index.build() + self.assertEqual(1, self.build_metrics(added)["reparsed_sources"]) + self.assertEqual([relative], adapter.extracted_paths) + self.assertIn(relative, {node.source_path for node in project.load().nodes}) + + added_path.unlink() + adapter.extracted_paths.clear() + deleted = index.build() + self.assertEqual(1, self.build_metrics(deleted)["deleted_sources"]) + self.assertNotIn(relative, {node.source_path for node in project.load().nodes}) + + (cache_root / "extractions.json").write_text("{broken", encoding="utf-8") + adapter.extracted_paths.clear() + recovered = index.build() + self.assertEqual(4, self.build_metrics(recovered)["reparsed_sources"]) + self.assertEqual(4, len(adapter.extracted_paths)) + self.assertEqual("ok", project.verify_incremental_equivalence()["status"]) + + def test_confinement_and_no_ast_reject_logic_for_both_grammars(self) -> None: + for fixture, language, _nodes, _edges, _logic, _shared_path in self.CASES: + with self.subTest(language=language), tempfile.TemporaryDirectory() as directory: + parent = Path(directory) + root = self.copy_fixture(parent, fixture) + + for source_root in (root / "src", Path("../outside")): + with self.assertRaises(DocForgeError) as captured: + JavaScriptReferenceAdapter(root, source_roots=(source_root,)) + self.assertEqual("path_escape", captured.exception.code) + + outside = parent / "outside" + outside.mkdir() + linked_root = root / "linked" + linked_root.symlink_to(outside, target_is_directory=True) + with self.assertRaises(DocForgeError) as linked: + JavaScriptReferenceAdapter(root, source_roots=("linked",)) + self.assertEqual("path_escape", linked.exception.code) + + adapter = RecordingScriptAdapter(root, fixture) + project = AdapterProject(adapter, cache_root=root / ".cache" / "no-ast") + with self.assertRaises(DocForgeError) as no_ast: + ProjectIndex(project, allow_logic=False).build() + self.assertEqual("adapter_policy_forbids_logic", no_ast.exception.code) + self.assertFalse(project.descriptor.index_path.exists()) + + def test_missing_optional_frontends_report_exact_install_remediation(self) -> None: + for fixture, language, _nodes, _edges, _logic, _shared_path in self.CASES: + grammar_module = ( + "tree_sitter_javascript" if language == "javascript" else "tree_sitter_typescript" + ) + with self.subTest(language=language), tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory), fixture) + adapter = RecordingScriptAdapter(root, fixture) + missing = ModuleNotFoundError( + f"No module named {grammar_module!r}", + name=grammar_module, + ) + with ( + mock.patch.object( + javascript_adapter.importlib, + "import_module", + side_effect=missing, + ), + self.assertRaises(DocForgeError) as captured, + ): + source = adapter.load_manifest().sources[0] + adapter.extract_source(source) + self.assertEqual("optional_dependency_missing", captured.exception.code) + self.assertEqual(language, captured.exception.details["extra"]) + self.assertEqual( + f"docforge[{language}]", + captured.exception.details["install"], + ) + self.assertEqual( + grammar_module, + captured.exception.details["missing_module"], + ) diff --git a/tests/test_milestone4_adoption.py b/tests/test_milestone4_adoption.py new file mode 100644 index 0000000..af13c3e --- /dev/null +++ b/tests/test_milestone4_adoption.py @@ -0,0 +1,137 @@ +from __future__ import annotations + +import json +import subprocess +import tempfile +import unittest +from pathlib import Path +from unittest import mock + +from tools.milestone4_adoption import ( + AdoptionProofError, + find_wheel, + parse_evidence, + run_adoption_proof, +) + + +def _evidence() -> dict[str, object]: + return { + "schema_version": 1, + "base_frontend_modules": [], + "base_frontend_distributions": [], + "python": {"build_status": "ok", "check_status": "ok", "node_count": 3}, + "mcp": { + "bootstrap_status": "ok", + "node_id": "python.class.service", + "read_tool_count": 20, + }, + "cpp_without_extra": { + "code": "optional_dependency_missing", + "install": "docforge[cpp]", + "missing_module": "tree_sitter", + }, + } + + +class Milestone4AdoptionTests(unittest.TestCase): + def test_evidence_requires_base_isolation_python_mcp_and_actionable_cpp_failure( + self, + ) -> None: + self.assertEqual(_evidence(), parse_evidence(json.dumps(_evidence()))) + + for mutation in ( + {"base_frontend_modules": ["tree_sitter"]}, + {"python": {"build_status": "error", "check_status": "ok"}}, + {"mcp": {"bootstrap_status": "error"}}, + { + "cpp_without_extra": { + "code": "optional_dependency_missing", + "install": "docforge[languages]", + } + }, + ): + evidence = {**_evidence(), **mutation} + with self.subTest(mutation=mutation), self.assertRaises(AdoptionProofError): + parse_evidence(json.dumps(evidence)) + + def test_wheel_discovery_rejects_missing_or_ambiguous_artifacts(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + with self.assertRaises(AdoptionProofError): + find_wheel(root) + first = root / "docforge-1-py3-none-any.whl" + first.touch() + self.assertEqual(first, find_wheel(root)) + (root / "docforge-2-py3-none-any.whl").touch() + with self.assertRaises(AdoptionProofError): + find_wheel(root) + + def test_orchestration_is_offline_locked_and_installs_the_wheel_without_extras( + self, + ) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory).resolve() + (root / "pyproject.toml").write_text("[project]\n", encoding="utf-8") + (root / "uv.lock").write_text("version = 1\n", encoding="utf-8") + commands: list[list[str]] = [] + + def execute( + arguments: list[str], + *, + cwd: Path, + timeout: int, + environment: dict[str, str] | None = None, + ) -> subprocess.CompletedProcess[str]: + del cwd, timeout + commands.append(arguments) + self.assertIsNotNone(environment) + assert environment is not None + self.assertEqual("1", environment["UV_OFFLINE"]) + self.assertNotIn("PYTHONPATH", environment) + if arguments[1] == "build": + output = Path(arguments[arguments.index("--out-dir") + 1]) + output.mkdir() + (output / "docforge-1-py3-none-any.whl").write_bytes(b"wheel") + elif arguments[1] == "export": + output = Path(arguments[arguments.index("--output-file") + 1]) + output.write_text("mcp==1\n", encoding="utf-8") + elif arguments[1] == "venv": + virtual_environment = Path(arguments[-1]) + (virtual_environment / "bin").mkdir(parents=True) + (virtual_environment / "bin" / "python").touch() + elif arguments[0].endswith("/bin/python"): + return subprocess.CompletedProcess( + arguments, + 0, + stdout=json.dumps(_evidence()), + stderr="", + ) + return subprocess.CompletedProcess(arguments, 0, stdout="", stderr="") + + with mock.patch( + "tools.milestone4_adoption._run_checked", + side_effect=execute, + ): + result = run_adoption_proof(root, uv="uv") + + self.assertEqual("offline", result["network"]) + wheel = result["wheel"] + self.assertIsInstance(wheel, dict) + assert isinstance(wheel, dict) + self.assertEqual("docforge-1-py3-none-any.whl", wheel["filename"]) + network_commands = ( + command + for command in commands + if command[0] == "uv" and command[1] in {"build", "export", "pip"} + ) + self.assertTrue(all("--offline" in command for command in network_commands)) + export = next(command for command in commands if command[1] == "export") + self.assertIn("--frozen", export) + self.assertIn("--no-dev", export) + wheel_install = next(command for command in commands if command[1:3] == ["pip", "install"]) + self.assertIn("--no-deps", wheel_install) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_public_contract.py b/tests/test_public_contract.py index e32af11..6aba6e8 100644 --- a/tests/test_public_contract.py +++ b/tests/test_public_contract.py @@ -43,6 +43,7 @@ PUBLIC_IMPORTS = { ), "docforge.adapter_contract": ( "AdapterAssembly", + "AdapterConformanceReport", "AdapterEdge", "AdapterImplementation", "AdapterLoader", @@ -53,15 +54,41 @@ PUBLIC_IMPORTS = { "AdapterProjectSettings", "AdapterSource", "AdapterSourceProjection", + "CompleteAdapterAssemblyLoader", "IncrementalAdapterAssembler", "IncrementalAdapterLoader", + "verify_adapter_conformance", + ), + "docforge.adapter_sdk": ( + "AdapterAssembly", + "AdapterConformanceReport", + "AdapterLoader", + "AdapterManifest", + "AdapterProject", + "AdapterProjection", + "AdapterSourceProjection", + "CompleteAdapterAssemblyLoader", + "IncrementalAdapterLoader", + "LogicProjection", + "Node", + "verify_adapter_conformance", ), "docforge.application": ( "CanonicalApplier", "CanonicalApplicationService", "GenericCanonicalApplier", ), - "docforge.client_config": ("generate_client_configuration",), + "docforge.adapter_launcher": ( + "ADAPTER_LAUNCHER_SCHEMA_VERSION", + "AdapterLauncherV1", + "AdapterSourceAvailabilityV1", + "adapter_source_availability", + "validate_adapter_launcher", + ), + "docforge.client_config": ( + "generate_adapter_client_configuration", + "generate_client_configuration", + ), "docforge.doctor": ("run_doctor",), "docforge.index": ("ProjectIndex",), "docforge.graph_projection": ( @@ -118,6 +145,18 @@ PUBLIC_IMPORTS = { "projection_hash", ), "docforge.projection_worker": ("render_projection_in_worker",), + "docforge.reference_config": ( + "REFERENCE_ADAPTER_CONFIG", + "REFERENCE_ADAPTER_SCHEMA_VERSION", + "REFERENCE_LANGUAGES", + "ReferenceAdapterConfigV1", + "load_reference_adapter_config", + ), + "docforge.reference_mcp": ( + "REFERENCE_MCP_MODULE", + "create_reference_project", + "create_reference_server", + ), "docforge.retrieval": ( "ContextCapsuleV1", "RetrievalPlanV1", diff --git a/tests/test_python_reference_adapter.py b/tests/test_python_reference_adapter.py new file mode 100644 index 0000000..0755321 --- /dev/null +++ b/tests/test_python_reference_adapter.py @@ -0,0 +1,282 @@ +from __future__ import annotations + +import shutil +import tempfile +import unittest +from pathlib import Path +from typing import cast +from unittest import mock + +import docforge.adapters.python as python_adapter +from docforge.adapter_sdk import ( + AdapterProject, + AdapterSource, + AdapterSourceProjection, + verify_adapter_conformance, +) +from docforge.adapters.python import ( + PYTHON_ADAPTER_ID, + PYTHON_ADAPTER_VERSION, + PYTHON_EXTRACTOR_VERSION, + PythonReferenceAdapter, +) +from docforge.errors import DocForgeError +from docforge.index import ProjectIndex + +ROOT = Path(__file__).resolve().parents[1] +FIXTURE = ROOT / "tests" / "fixtures" / "reference-python" + + +class RecordingPythonAdapter(PythonReferenceAdapter): + def __init__(self, root: Path) -> None: + super().__init__( + root, + source_roots=("src",), + project_id="reference-python", + title="Reference Python fixture", + ) + self.extracted_paths: list[str] = [] + + def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: + self.extracted_paths.append(source.source_path) + return super().extract_source(source) + + +class PythonReferenceAdapterTests(unittest.TestCase): + def copy_fixture(self, parent: Path) -> Path: + root = parent / "reference-python" + shutil.copytree(FIXTURE, root) + return root.resolve() + + def adapter(self, root: Path) -> RecordingPythonAdapter: + return RecordingPythonAdapter(root) + + @staticmethod + def build_metrics(result: dict[str, object]) -> dict[str, object]: + return cast(dict[str, object], result["build"]) + + def test_complete_assembly_is_exact_ordered_cache_independent_and_scoped(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + adapter = self.adapter(root) + + first = adapter.load_assembly() + second = adapter.load_complete_assembly() + projection = adapter.load_projection() + + self.assertEqual(first, second) + self.assertEqual(first.projection, projection) + self.assertEqual(PYTHON_ADAPTER_ID, projection.adapter_id) + self.assertEqual(PYTHON_ADAPTER_VERSION, projection.adapter_version) + self.assertEqual(14, len(projection.nodes)) + self.assertEqual(13, len(projection.edges)) + self.assertEqual(5, len(first.logic)) + self.assertEqual( + list(projection.nodes), + sorted(projection.nodes, key=lambda item: item.node.node_id), + ) + self.assertEqual( + {"contains", "depends_on"}, {item.edge.relation for item in projection.edges} + ) + self.assertFalse(any(item.node.title == "json" for item in projection.nodes)) + self.assertTrue( + any( + item.node.title == "Service" and ("kind", "class") in item.metadata + for item in projection.nodes + ) + ) + self.assertTrue( + any( + item.node.title == "Service.run" and ("kind", "function") in item.metadata + for item in projection.nodes + ) + ) + + report = adapter.support_report() + self.assertEqual(1, report["schema_version"]) + self.assertEqual(PYTHON_EXTRACTOR_VERSION, report["extractor_version"]) + self.assertFalse(report["imports_project_code"]) + self.assertFalse(report["executes_project_code"]) + self.assertEqual( + [ + "call_resolution", + "dynamic_import_resolution", + "inheritance_resolution", + "runtime_generated_facts", + "symbol_reference_resolution", + ], + [item.code for item in adapter.unsupported_facts()], + ) + self.assertFalse((root / ".cache").exists()) + + conformance = verify_adapter_conformance( + adapter, + cache_root=root / ".cache" / "conformance", + ) + self.assertEqual("reference-python", conformance.project_id) + self.assertEqual(14, conformance.node_count) + self.assertEqual(13, conformance.edge_count) + self.assertEqual(5, conformance.logic_projection_count) + self.assertTrue(conformance.incremental) + + def test_incremental_cold_warm_reverse_dependencies_and_logic_equivalence(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + adapter = self.adapter(root) + project = AdapterProject(adapter, cache_root=root / ".cache" / "incremental") + index = ProjectIndex(project) + + cold = index.build() + self.assertEqual(4, self.build_metrics(cold)["reparsed_sources"]) + self.assertEqual( + [ + "src/sample/__init__.py", + "src/sample/service.py", + "src/sample/shared.py", + "src/sample/worker.py", + ], + sorted(adapter.extracted_paths), + ) + + adapter.extracted_paths.clear() + original_parse = python_adapter.ast.parse + with mock.patch.object( + python_adapter.ast, + "parse", + wraps=original_parse, + ) as parsed: + warm = index.build() + self.assertEqual(4, self.build_metrics(warm)["cache_hits"]) + self.assertEqual(0, self.build_metrics(warm)["reparsed_sources"]) + self.assertEqual([], adapter.extracted_paths) + parsed.assert_not_called() + + shared = root / "src" / "sample" / "shared.py" + shared.write_text( + shared.read_text(encoding="utf-8").replace( + "DEFAULT_LIMIT = 3", "DEFAULT_LIMIT = 5" + ), + encoding="utf-8", + ) + adapter.extracted_paths.clear() + with mock.patch.object( + python_adapter.ast, + "parse", + wraps=original_parse, + ) as parsed: + changed = index.build() + self.assertEqual(4, self.build_metrics(changed)["invalidated_sources"]) + self.assertGreater(parsed.call_count, 0) + self.assertEqual( + [ + "src/sample/__init__.py", + "src/sample/service.py", + "src/sample/shared.py", + "src/sample/worker.py", + ], + sorted(adapter.extracted_paths), + ) + + equivalent = project.verify_incremental_equivalence() + self.assertEqual("ok", equivalent["status"]) + self.assertEqual(14, equivalent["node_count"]) + self.assertEqual(13, equivalent["edge_count"]) + self.assertEqual(5, equivalent["logic_projection_count"]) + + def test_add_rename_delete_and_corrupt_cache_recover_without_stale_facts(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + adapter = self.adapter(root) + cache_root = root / ".cache" / "incremental" + project = AdapterProject(adapter, cache_root=cache_root) + index = ProjectIndex(project) + index.build() + + extra = root / "src" / "sample" / "extra.py" + extra.write_text( + '"""Added source."""\n\n\ndef extra() -> str:\n return "extra"\n', + encoding="utf-8", + ) + adapter.extracted_paths.clear() + added = index.build() + self.assertEqual(1, self.build_metrics(added)["reparsed_sources"]) + self.assertEqual(["src/sample/extra.py"], adapter.extracted_paths) + self.assertIn( + "src/sample/extra.py", + {node.source_path for node in project.load().nodes}, + ) + + worker = root / "src" / "sample" / "worker.py" + runner = root / "src" / "sample" / "runner.py" + worker.rename(runner) + adapter.extracted_paths.clear() + renamed = index.build() + self.assertEqual(1, self.build_metrics(renamed)["deleted_sources"]) + self.assertEqual(1, self.build_metrics(renamed)["reparsed_sources"]) + renamed_paths = {node.source_path for node in project.load().nodes} + self.assertNotIn("src/sample/worker.py", renamed_paths) + self.assertIn("src/sample/runner.py", renamed_paths) + + extra.unlink() + adapter.extracted_paths.clear() + deleted = index.build() + self.assertEqual(1, self.build_metrics(deleted)["deleted_sources"]) + self.assertNotIn( + "src/sample/extra.py", + {node.source_path for node in project.load().nodes}, + ) + + (cache_root / "extractions.json").write_text("{broken", encoding="utf-8") + adapter.extracted_paths.clear() + recovered = index.build() + self.assertEqual(4, self.build_metrics(recovered)["reparsed_sources"]) + self.assertEqual( + [ + "src/sample/__init__.py", + "src/sample/runner.py", + "src/sample/service.py", + "src/sample/shared.py", + ], + sorted(adapter.extracted_paths), + ) + self.assertEqual("ok", project.verify_incremental_equivalence()["status"]) + + def test_confinement_and_no_ast_policy_reject_logic_without_importing_code(self) -> None: + with tempfile.TemporaryDirectory() as directory: + parent = Path(directory) + root = self.copy_fixture(parent) + + for source_root in (root / "src", Path("../outside")): + with self.subTest(source_root=source_root): + with self.assertRaises(DocForgeError) as captured: + PythonReferenceAdapter(root, source_roots=(source_root,)) + self.assertEqual("path_escape", captured.exception.code) + + outside = parent / "outside" + outside.mkdir() + linked_root = root / "linked" + linked_root.symlink_to(outside, target_is_directory=True) + with self.assertRaises(DocForgeError) as linked: + PythonReferenceAdapter(root, source_roots=("linked",)) + self.assertEqual("path_escape", linked.exception.code) + + external_source = outside / "external.py" + external_source.write_text("def outside():\n return True\n", encoding="utf-8") + worker = root / "src" / "sample" / "worker.py" + worker.unlink() + worker.symlink_to(external_source) + adapter = self.adapter(root) + with self.assertRaises(DocForgeError) as source_link: + adapter.load_manifest() + self.assertEqual("path_escape", source_link.exception.code) + + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + adapter = self.adapter(root) + project = AdapterProject(adapter, cache_root=root / ".cache" / "no-ast") + + with self.assertRaises(DocForgeError) as no_ast: + ProjectIndex(project, allow_logic=False).build() + + self.assertEqual("adapter_policy_forbids_logic", no_ast.exception.code) + self.assertFalse(project.descriptor.index_path.exists()) diff --git a/tests/test_reference_mcp.py b/tests/test_reference_mcp.py new file mode 100644 index 0000000..5b8ad06 --- /dev/null +++ b/tests/test_reference_mcp.py @@ -0,0 +1,296 @@ +from __future__ import annotations + +import io +import json +import shutil +import sys +import tempfile +import unittest +from contextlib import redirect_stderr +from pathlib import Path +from unittest import mock + +from jsonschema import Draft202012Validator +from mcp.shared.memory import create_connected_server_and_client_session + +from docforge.adapter_launcher import AdapterLauncherV1 +from docforge.client_config import generate_adapter_client_configuration +from docforge.errors import DocForgeError +from docforge.index import ProjectIndex +from docforge.mcp_server import READ_TOOLS +from docforge.reference_config import ( + REFERENCE_ADAPTER_CONFIG, + load_reference_adapter_config, +) +from docforge.reference_mcp import ( + REFERENCE_MCP_MODULE, + create_reference_project, + create_reference_server, + main, +) + +ROOT = Path(__file__).resolve().parents[1] +FIXTURE = ROOT / "tests" / "fixtures" / "reference-python" +CPP_FIXTURE = ROOT / "tests" / "fixtures" / "reference-cpp" +SCHEMA = json.loads( + (ROOT / "schemas" / "reference-adapter.schema.json").read_text(encoding="utf-8") +) + + +class ReferenceMcpTests(unittest.IsolatedAsyncioTestCase): + def copy_fixture(self, parent: Path) -> Path: + root = parent / "reference-python" + shutil.copytree(FIXTURE, root) + return root.resolve() + + def copy_cpp_fixture(self, parent: Path) -> Path: + root = parent / "reference-cpp" + shutil.copytree(CPP_FIXTURE, root) + return root.resolve() + + @staticmethod + def write_config( + root: Path, + *, + language: str = "python", + source_roots: tuple[str, ...] = ("src",), + extra: str = "", + ) -> Path: + config = root.joinpath(*REFERENCE_ADAPTER_CONFIG.parts) + config.parent.mkdir(parents=True, exist_ok=True) + quoted_roots = ", ".join(json.dumps(item) for item in source_roots) + config.write_text( + "\n".join( + ( + "schema_version = 1", + 'project_id = "reference-python"', + 'title = "Runnable Python reference"', + f"language = {json.dumps(language)}", + f"source_roots = [{quoted_roots}]", + extra, + "", + ) + ), + encoding="utf-8", + ) + return config + + async def test_python_config_build_check_and_real_mcp_retrieval(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + self.write_config(root) + config = load_reference_adapter_config(root) + + Draft202012Validator(SCHEMA).validate(config.as_dict()) + self.assertEqual("python", config.language) + self.assertEqual((root / "src",), config.source_roots) + self.assertIsNone(config.compilation_database) + self.assertEqual(64, len(config.config_hash)) + + project = create_reference_project(root) + index = ProjectIndex(project) + built = index.build() + checked = index.check() + self.assertEqual("ok", built["status"]) + self.assertEqual("ok", checked["status"]) + self.assertTrue(project.descriptor.index_path.is_relative_to(root / ".docforge")) + + async with create_connected_server_and_client_session( + create_reference_server(root, capability_mode="read"), + raise_exceptions=True, + ) as session: + tools = tuple(tool.name for tool in (await session.list_tools()).tools) + bootstrap = await session.call_tool("docforge_bootstrap", {}) + search = await session.call_tool( + "docforge_search", + {"query": "Service", "limit": 5}, + ) + node_id = search.structuredContent["results"][0]["node_id"] + node = await session.call_tool("docforge_get_node", {"node_id": node_id}) + + self.assertEqual(READ_TOOLS, tools) + self.assertEqual("ok", bootstrap.structuredContent["status"]) + self.assertEqual( + REFERENCE_MCP_MODULE, + bootstrap.structuredContent["binding"]["server_module"], + ) + self.assertEqual( + "python", + bootstrap.structuredContent["binding"]["reference_language"], + ) + self.assertGreater(search.structuredContent["count"], 0) + self.assertEqual(node_id, node.structuredContent["node"]["node_id"]) + + async def test_cpp_config_build_check_and_real_mcp_retrieval(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.copy_cpp_fixture(Path(directory)) + self.write_config( + root, + language="cpp", + source_roots=("include", "src"), + extra='compilation_database = "compile_commands.json"', + ) + config = load_reference_adapter_config(root) + Draft202012Validator(SCHEMA).validate(config.as_dict()) + + project = create_reference_project(root) + index = ProjectIndex(project) + self.assertEqual("ok", index.build()["status"]) + self.assertEqual("ok", index.check()["status"]) + + async with create_connected_server_and_client_session( + create_reference_server(root, capability_mode="read"), + raise_exceptions=True, + ) as session: + bootstrap = await session.call_tool("docforge_bootstrap", {}) + search = await session.call_tool( + "docforge_search", + {"query": "worker", "limit": 5}, + ) + node_id = search.structuredContent["results"][0]["node_id"] + node = await session.call_tool("docforge_get_node", {"node_id": node_id}) + + self.assertEqual("cpp", config.language) + self.assertEqual(root / "compile_commands.json", config.compilation_database) + self.assertEqual( + "cpp", + bootstrap.structuredContent["binding"]["reference_language"], + ) + self.assertEqual(node_id, node.structuredContent["node"]["node_id"]) + + def test_launcher_generates_exact_isolated_module_invocation(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + self.write_config(root) + project = create_reference_project(root) + launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE) + + with mock.patch("docforge.client_config.subprocess.run") as executed: + result = generate_adapter_client_configuration( + project, + launcher, + "codex", + capability_mode="read", + ) + + self.assertEqual( + [ + "-I", + "-m", + REFERENCE_MCP_MODULE, + "--project-root", + str(root), + "--capability-mode", + "read", + ], + result["binding"]["args"], + ) + self.assertEqual({}, result["binding"]["environment"]) + self.assertNotIn("cwd", result["binding"]) + executed.assert_not_called() + + def test_config_drift_requires_runtime_restart(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + config_path = self.write_config(root) + project = create_reference_project(root) + project.validate_runtime() + + config_path.write_text( + config_path.read_text(encoding="utf-8").replace( + "Runnable Python reference", + "Changed Python reference", + ), + encoding="utf-8", + ) + with self.assertRaises(DocForgeError) as captured: + project.validate_runtime() + self.assertEqual("adapter_restart_required", captured.exception.code) + self.assertEqual( + [REFERENCE_ADAPTER_CONFIG.as_posix()], + captured.exception.details["changed"], + ) + + def test_config_rejects_unknown_fields_languages_and_language_mismatch(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = self.copy_fixture(Path(directory)) + for language, extra, code in ( + ("ruby", "", "unsupported_reference_language"), + ("javascript", "", "reference_language_mismatch"), + ("python", 'command = "python project.py"', "invalid_reference_config"), + ( + "python", + 'compilation_database = "compile_commands.json"', + "invalid_reference_config", + ), + ("cpp", "", "invalid_reference_config"), + ): + with self.subTest(language=language, extra=extra): + self.write_config(root, language=language, extra=extra) + with self.assertRaises(DocForgeError) as captured: + load_reference_adapter_config(root) + self.assertEqual(code, captured.exception.code) + + def test_config_rejects_path_escape_and_source_symlinks(self) -> None: + with tempfile.TemporaryDirectory() as directory: + parent = Path(directory) + root = self.copy_fixture(parent) + outside = parent / "outside" + outside.mkdir() + + self.write_config(root, source_roots=("../outside",)) + with self.assertRaises(DocForgeError) as escaped: + load_reference_adapter_config(root) + self.assertEqual("path_escape", escaped.exception.code) + + linked = root / "linked" + linked.symlink_to(outside, target_is_directory=True) + self.write_config(root, source_roots=("linked",)) + with self.assertRaises(DocForgeError) as linked_root: + load_reference_adapter_config(root) + self.assertEqual("path_escape", linked_root.exception.code) + + self.write_config(root) + external_source = outside / "external.py" + external_source.write_text("def external(): pass\n", encoding="utf-8") + source_link = root / "src" / "external.py" + source_link.symlink_to(external_source) + with self.assertRaises(DocForgeError) as linked_source: + load_reference_adapter_config(root) + self.assertEqual("path_escape", linked_source.exception.code) + + def test_config_file_must_be_a_regular_non_symlink(self) -> None: + with tempfile.TemporaryDirectory() as directory: + parent = Path(directory) + root = self.copy_fixture(parent) + config_path = self.write_config(root) + outside = parent / "reference-adapter.toml" + shutil.copyfile(config_path, outside) + config_path.unlink() + config_path.symlink_to(outside) + + with self.assertRaises(DocForgeError) as captured: + load_reference_adapter_config(root) + self.assertEqual("path_escape", captured.exception.code) + + def test_reference_module_cli_advertises_read_mode_only(self) -> None: + with ( + redirect_stderr(io.StringIO()), + mock.patch.object( + sys, + "argv", + [ + REFERENCE_MCP_MODULE, + "--project-root", + "/irrelevant", + "--capability-mode", + "proposal", + ], + ), + mock.patch("docforge.reference_mcp.create_reference_server") as create_server, + self.assertRaises(SystemExit) as captured, + ): + main() + + self.assertEqual(2, captured.exception.code) + create_server.assert_not_called() diff --git a/tests/test_treesitter_logic.py b/tests/test_treesitter_logic.py index 51913b3..dad4da3 100644 --- a/tests/test_treesitter_logic.py +++ b/tests/test_treesitter_logic.py @@ -7,6 +7,8 @@ from docforge.treesitter_logic import ( TreeSitterLogicOwner, analyze_cpp_source, analyze_javascript_source, + analyze_typescript_source, + discover_typescript_functions, ) @@ -200,5 +202,44 @@ public: self.assertEqual(missing.exception.code, "missing_logic_owner") +class TypeScriptLogicTests(unittest.TestCase): + def test_typed_function_and_method_use_the_typescript_grammar(self) -> None: + source = """ +interface Choice { + enabled: boolean; +} + +function choose(choice: Choice): number { + if (choice.enabled) { + return 1; + } + return 0; +} + +class Worker { + run(choice: Choice): number { + return choose(choice); + } +} +""".strip() + + discovered = discover_typescript_functions(source) + self.assertEqual( + [("Worker.run", 13), ("choose", 5)], + [(item.qualified_name, item.line) for item in discovered], + ) + projection = analyze_typescript_source( + source, + source_id="source.typescript", + owners=(TreeSitterLogicOwner("ts.symbol.choose", "choose", 5),), + )[0] + + self.assertIn("choice.enabled", {node.label for node in projection.nodes}) + self.assertEqual( + {"return"}, + {edge.relation for edge in projection.edges if edge.relation == "return"}, + ) + + if __name__ == "__main__": unittest.main() diff --git a/tools/check_documentation.py b/tools/check_documentation.py new file mode 100644 index 0000000..df32b56 --- /dev/null +++ b/tools/check_documentation.py @@ -0,0 +1,698 @@ +"""Validate the maintained DocForge Markdown documentation graph.""" + +from __future__ import annotations + +import argparse +import html +import json +import re +import stat +import sys +import tomllib +import unicodedata +from collections import deque +from collections.abc import Iterable, Sequence +from dataclasses import dataclass +from pathlib import Path +from typing import Protocol, cast +from urllib.parse import unquote, urlsplit + +from jsonschema import Draft202012Validator +from jsonschema.exceptions import ValidationError +from markdown_it import MarkdownIt +from markdown_it.token import Token + +REPOSITORY_ROOT = Path(__file__).resolve().parents[1] +DEFAULT_PENDING_INVENTORY = Path("tools/docs_pending_m4_pages.txt") +COMMAND_REFERENCE = Path("docs/COMMAND_REFERENCE.md") +COMMAND_REFERENCE_H1 = "DocForge command reference" +COMMAND_REFERENCE_NOTICE = ( + "> Generated from the live CLI parser and MCP registrations. Do not edit this file by hand." +) +REFERENCE_ADAPTER_SCHEMA = Path("schemas/reference-adapter.schema.json") +MAX_MARKDOWN_PAGES = 256 +MAX_PAGE_BYTES = 2_000_000 +MAX_TOTAL_BYTES = 20_000_000 +MAX_DIAGNOSTICS = 100 + +REQUIRED_MILESTONE_4_PAGES = ( + Path("README.md"), + Path("docs/ADAPTER_AUTHORING_GUIDE.md"), + Path("docs/AGENT_INTEGRATION.md"), + Path("docs/APPLICATION_DECISION.md"), + Path("docs/COMMAND_REFERENCE.md"), + Path("docs/COMPATIBILITY.md"), + Path("docs/CONTRACT.md"), + Path("docs/CORE_CONCEPTS_AND_AUTHORITY.md"), + Path("docs/INCREMENTAL_INDEXING.md"), + Path("docs/LEGACY_AND_NO_AST.md"), + Path("docs/MCP_CONTRACT.md"), + Path("docs/MIGRATING_FROM_V1.md"), + Path("docs/MILESTONE_4_BASELINE.md"), + Path("docs/MILESTONE_4_CLOSEOUT.md"), + Path("docs/NEW_PROJECT_QUICKSTART.md"), + Path("docs/POLICY_PRECEDENCE.md"), + Path("docs/PROJECT_DESCRIPTOR.md"), + Path("docs/PROJECT_ONBOARDING.md"), + Path("docs/RECOVERY_AND_PERFORMANCE.md"), + Path("docs/REFERENCE_ADAPTERS.md"), + Path("docs/RENDERING_AND_VISUALIZATION.md"), + Path("docs/SECURITY.md"), + Path("docs/USER_MANUAL.md"), + Path("docs/VIEWER_MANAGER.md"), +) + +_HISTORICAL_RECORD = re.compile(r"^docs/MILESTONE_[0-3]_(?:BASELINE|CLOSEOUT)\.md$") +_HTML_ID = re.compile( + r"""\bid\s*=\s*(?:"([^"]+)"|'([^']+)')""", + re.IGNORECASE, +) +_REFERENCE_ADAPTER_KEY = re.compile(r"(?m)^(?:language|source_roots|compilation_database)\s*=") +_URI_SCHEME = re.compile(r"^[A-Za-z][A-Za-z0-9+.-]*:") + + +@dataclass(frozen=True, order=True) +class Diagnostic: + """One deterministic documentation validation failure.""" + + path: str + line: int + code: str + message: str + + def render(self) -> str: + """Render one bounded compiler-style diagnostic.""" + + return f"{self.path}:{self.line}: {self.code}: {self.message}" + + +@dataclass(frozen=True) +class DocumentationPolicy: + """Repository-relative pages and temporary inventory accepted by one run.""" + + required_pages: tuple[Path, ...] = REQUIRED_MILESTONE_4_PAGES + pending_inventory: Path | None = DEFAULT_PENDING_INVENTORY + + +@dataclass(frozen=True) +class DocumentationReport: + """Bounded deterministic result from one complete documentation check.""" + + diagnostics: tuple[Diagnostic, ...] + omitted_diagnostics: int + markdown_pages: int + pending_pages: int + reference_adapter_examples: int + + @property + def ok(self) -> bool: + """Return whether every documentation invariant passed.""" + + return not self.diagnostics and self.omitted_diagnostics == 0 + + +@dataclass(frozen=True) +class _Page: + relative: Path + source: str + tokens: tuple[Token, ...] + anchors: frozenset[str] + + +class _Diagnostics: + def __init__(self) -> None: + self._items: list[Diagnostic] = [] + self._omitted = 0 + + def add(self, path: Path | str, line: int, code: str, message: str) -> None: + relative = path.as_posix() if isinstance(path, Path) else path + item = Diagnostic(relative, max(1, line), code, message) + if len(self._items) < MAX_DIAGNOSTICS: + self._items.append(item) + else: + self._omitted += 1 + + def result(self) -> tuple[tuple[Diagnostic, ...], int]: + return tuple(sorted(self._items)), self._omitted + + +class _Validator(Protocol): + def iter_errors(self, instance: object) -> Iterable[ValidationError]: ... + + +def check_documentation( + repository_root: Path, + *, + policy: DocumentationPolicy | None = None, +) -> DocumentationReport: + """Check one repository's maintained Markdown pages without modifying it.""" + + root = repository_root.resolve(strict=True) + if not root.is_dir(): + raise ValueError(f"repository root is not a directory: {root}") + effective_policy = policy or DocumentationPolicy() + + diagnostics = _Diagnostics() + parser = MarkdownIt("commonmark", {"html": True, "typographer": False}) + page_paths = _discover_pages(root, diagnostics) + pages = _load_pages(root, page_paths, parser, diagnostics) + pending = _load_pending_inventory(root, effective_policy, diagnostics) + _check_required_inventory( + root, + effective_policy.required_pages, + pending, + diagnostics, + ) + _check_h1s(pages, diagnostics) + link_graph = _check_links(root, pages, parser, diagnostics) + _check_reachability(pages, link_graph, diagnostics) + reference_examples = _check_reference_adapter_examples(root, pages, diagnostics) + _check_command_reference(pages, diagnostics) + items, omitted = diagnostics.result() + return DocumentationReport( + diagnostics=items, + omitted_diagnostics=omitted, + markdown_pages=len(pages), + pending_pages=len(pending), + reference_adapter_examples=reference_examples, + ) + + +def _discover_pages(root: Path, diagnostics: _Diagnostics) -> tuple[Path, ...]: + candidates = [Path("README.md")] + docs = root / "docs" + if not docs.is_dir(): + diagnostics.add("docs", 1, "DOC001", "maintained documentation directory is missing") + return tuple(candidates) + + candidates.extend( + path.relative_to(root) + for path in docs.rglob("*.md") + if not path.is_symlink() and path.is_file() + ) + unique = tuple(sorted(set(candidates), key=Path.as_posix)) + if len(unique) > MAX_MARKDOWN_PAGES: + diagnostics.add( + "docs", + 1, + "DOC002", + f"found {len(unique)} Markdown pages; limit is {MAX_MARKDOWN_PAGES}", + ) + return unique[:MAX_MARKDOWN_PAGES] + return unique + + +def _load_pages( + root: Path, + paths: tuple[Path, ...], + parser: MarkdownIt, + diagnostics: _Diagnostics, +) -> dict[Path, _Page]: + pages: dict[Path, _Page] = {} + total_bytes = 0 + for relative in paths: + absolute = root / relative + try: + metadata = absolute.lstat() + except FileNotFoundError: + diagnostics.add(relative, 1, "DOC003", "maintained Markdown page is missing") + continue + if not stat.S_ISREG(metadata.st_mode): + diagnostics.add(relative, 1, "DOC004", "maintained Markdown page is not a regular file") + continue + if metadata.st_size > MAX_PAGE_BYTES: + diagnostics.add( + relative, + 1, + "DOC005", + f"page is {metadata.st_size} bytes; limit is {MAX_PAGE_BYTES}", + ) + continue + total_bytes += metadata.st_size + if total_bytes > MAX_TOTAL_BYTES: + diagnostics.add( + relative, + 1, + "DOC006", + f"total Markdown input exceeds {MAX_TOTAL_BYTES} bytes", + ) + break + try: + source = absolute.read_text(encoding="utf-8") + except UnicodeDecodeError: + diagnostics.add(relative, 1, "DOC007", "page is not valid UTF-8") + continue + tokens = tuple(parser.parse(source)) + pages[relative] = _Page( + relative=relative, + source=source, + tokens=tokens, + anchors=_anchors(tokens), + ) + return pages + + +def _load_pending_inventory( + root: Path, + policy: DocumentationPolicy, + diagnostics: _Diagnostics, +) -> frozenset[Path]: + inventory = policy.pending_inventory + if inventory is None: + return frozenset() + if inventory.is_absolute(): + diagnostics.add( + inventory.as_posix(), + 1, + "DOC008", + "pending inventory path must be repository-relative", + ) + return frozenset() + absolute = root / inventory + try: + lines = absolute.read_text(encoding="utf-8").splitlines() + except FileNotFoundError: + diagnostics.add(inventory, 1, "DOC009", "pending inventory file is missing") + return frozenset() + except UnicodeDecodeError: + diagnostics.add(inventory, 1, "DOC010", "pending inventory is not valid UTF-8") + return frozenset() + + required = set(policy.required_pages) + pending: set[Path] = set() + for line_number, raw in enumerate(lines, start=1): + value = raw.strip() + if not value or value.startswith("#"): + continue + path = Path(value) + if path.is_absolute() or ".." in path.parts or path.as_posix() != value: + diagnostics.add( + inventory, + line_number, + "DOC011", + f"unsafe pending inventory path: {value!r}", + ) + continue + if path not in required: + diagnostics.add( + inventory, + line_number, + "DOC012", + f"pending page is not in the required inventory: {value}", + ) + continue + if path in pending: + diagnostics.add( + inventory, + line_number, + "DOC013", + f"duplicate pending page: {value}", + ) + continue + pending.add(path) + return frozenset(pending) + + +def _check_required_inventory( + root: Path, + required_pages: tuple[Path, ...], + pending: frozenset[Path], + diagnostics: _Diagnostics, +) -> None: + for relative in sorted(set(required_pages), key=Path.as_posix): + exists = (root / relative).is_file() + if relative in pending: + if exists: + diagnostics.add( + relative, + 1, + "DOC014", + "page exists but remains in the pending inventory", + ) + continue + if not exists: + diagnostics.add(relative, 1, "DOC015", "required Milestone 4 page is missing") + + +def _check_h1s(pages: dict[Path, _Page], diagnostics: _Diagnostics) -> None: + for relative, page in pages.items(): + h1_lines = [ + _token_line(token) + for token in page.tokens + if token.type == "heading_open" and token.tag == "h1" + ] + if len(h1_lines) != 1: + rendered = ", ".join(str(line) for line in h1_lines) or "none" + diagnostics.add( + relative, + h1_lines[1] if len(h1_lines) > 1 else 1, + "DOC016", + f"expected exactly one H1; found {len(h1_lines)} (lines: {rendered})", + ) + + +def _check_links( + root: Path, + pages: dict[Path, _Page], + parser: MarkdownIt, + diagnostics: _Diagnostics, +) -> dict[Path, frozenset[Path]]: + graph: dict[Path, frozenset[Path]] = {} + anchor_cache = {relative: page.anchors for relative, page in pages.items()} + for relative, page in pages.items(): + destinations: set[Path] = set() + for href, line in _page_links(page.tokens): + target = _resolve_local_link(root, relative, href, line, diagnostics) + if target is None: + continue + target_relative, fragment = target + if target_relative in pages: + destinations.add(target_relative) + if not fragment: + continue + anchors = anchor_cache.get(target_relative) + if anchors is None: + anchors = _load_link_target_anchors( + root, + target_relative, + parser, + diagnostics, + ) + if anchors is None: + continue + anchor_cache[target_relative] = anchors + if fragment not in anchors: + diagnostics.add( + relative, + line, + "DOC017", + f"missing anchor #{fragment} in {target_relative.as_posix()}", + ) + graph[relative] = frozenset(destinations) + return graph + + +def _page_links(tokens: tuple[Token, ...]) -> Iterable[tuple[str, int]]: + for token in tokens: + children = token.children or [] + for child in children: + attribute = "href" if child.type == "link_open" else "src" + if child.type not in {"link_open", "image"}: + continue + raw = child.attrs.get(attribute) + if isinstance(raw, str): + yield raw, _token_line(token) + + +def _resolve_local_link( + root: Path, + source: Path, + href: str, + line: int, + diagnostics: _Diagnostics, +) -> tuple[Path, str] | None: + if not href or href.startswith("//") or _URI_SCHEME.match(href): + return None + try: + parsed = urlsplit(href) + except ValueError as error: + diagnostics.add(source, line, "DOC018", f"invalid link {href!r}: {error}") + return None + if parsed.netloc: + return None + decoded_path = unquote(parsed.path) + fragment = unquote(parsed.fragment) + if "\x00" in decoded_path or decoded_path.startswith("/"): + diagnostics.add(source, line, "DOC019", f"local link escapes the repository: {href!r}") + return None + absolute = (root / source if not decoded_path else root / source.parent / decoded_path).resolve( + strict=False + ) + try: + target = absolute.relative_to(root) + except ValueError: + diagnostics.add(source, line, "DOC019", f"local link escapes the repository: {href!r}") + return None + if not absolute.is_file(): + diagnostics.add(source, line, "DOC020", f"missing local link target: {target.as_posix()}") + return None + return target, fragment + + +def _load_link_target_anchors( + root: Path, + relative: Path, + parser: MarkdownIt, + diagnostics: _Diagnostics, +) -> frozenset[str] | None: + if relative.suffix.lower() not in {".md", ".markdown"}: + diagnostics.add( + relative, + 1, + "DOC021", + "link uses an anchor on a non-Markdown target", + ) + return None + absolute = root / relative + try: + metadata = absolute.lstat() + if not stat.S_ISREG(metadata.st_mode) or metadata.st_size > MAX_PAGE_BYTES: + raise ValueError("target is not a bounded regular Markdown file") + source = absolute.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError, ValueError) as error: + diagnostics.add(relative, 1, "DOC022", f"cannot inspect linked Markdown anchors: {error}") + return None + return _anchors(tuple(parser.parse(source))) + + +def _anchors(tokens: tuple[Token, ...]) -> frozenset[str]: + anchors: set[str] = set() + slug_counts: dict[str, int] = {} + for index, token in enumerate(tokens): + if token.type in {"html_block", "html_inline"}: + anchors.update(_html_ids(token.content)) + if token.children: + for child in token.children: + if child.type == "html_inline": + anchors.update(_html_ids(child.content)) + if token.type != "heading_open" or index + 1 >= len(tokens): + continue + inline = tokens[index + 1] + base = _heading_slug(_inline_text(inline)) + duplicate = slug_counts.get(base, 0) + slug_counts[base] = duplicate + 1 + anchors.add(base if duplicate == 0 else f"{base}-{duplicate}") + return frozenset(anchors) + + +def _inline_text(token: Token) -> str: + if not token.children: + return token.content + pieces: list[str] = [] + for child in token.children: + if child.type in {"text", "code_inline"}: + pieces.append(child.content) + elif child.type in {"softbreak", "hardbreak"}: + pieces.append(" ") + elif child.type == "image": + pieces.append(child.content) + elif child.type == "html_inline": + pieces.append(re.sub(r"<[^>]*>", "", child.content)) + return "".join(pieces) + + +def _heading_slug(value: str) -> str: + characters: list[str] = [] + for character in html.unescape(value).casefold().strip(): + category = unicodedata.category(character) + if category[0] in {"L", "M", "N"} or character in {"-", "_"}: + characters.append(character) + elif character.isspace(): + characters.append("-") + return "".join(characters) + + +def _html_ids(value: str) -> set[str]: + return {first or second for first, second in _HTML_ID.findall(value)} + + +def _check_reachability( + pages: dict[Path, _Page], + graph: dict[Path, frozenset[Path]], + diagnostics: _Diagnostics, +) -> None: + root_page = Path("README.md") + if root_page not in pages: + return + reached = {root_page} + queue = deque([root_page]) + while queue: + current = queue.popleft() + for target in sorted(graph.get(current, frozenset()), key=Path.as_posix): + if target not in reached: + reached.add(target) + queue.append(target) + for relative in sorted(set(pages) - reached, key=Path.as_posix): + if _HISTORICAL_RECORD.fullmatch(relative.as_posix()): + continue + diagnostics.add( + relative, + 1, + "DOC023", + "maintained page is not reachable from README.md", + ) + + +def _check_reference_adapter_examples( + root: Path, + pages: dict[Path, _Page], + diagnostics: _Diagnostics, +) -> int: + schema_path = root / REFERENCE_ADAPTER_SCHEMA + try: + schema_document = cast( + dict[str, object], + json.loads(schema_path.read_text(encoding="utf-8")), + ) + Draft202012Validator.check_schema(schema_document) + except (OSError, UnicodeDecodeError, json.JSONDecodeError, TypeError) as error: + diagnostics.add( + REFERENCE_ADAPTER_SCHEMA, + 1, + "DOC024", + f"cannot load reference-adapter schema: {error}", + ) + return 0 + validator = cast(_Validator, Draft202012Validator(schema_document)) + count = 0 + for relative, page in pages.items(): + previous_inline: Token | None = None + for token in page.tokens: + if token.type == "inline": + previous_inline = token + continue + if token.type != "fence" or token.info.split(maxsplit=1)[:1] != ["toml"]: + continue + if not _is_reference_adapter_example(token, previous_inline): + continue + count += 1 + line = _token_line(token) + try: + document = tomllib.loads(token.content) + except tomllib.TOMLDecodeError as error: + diagnostics.add( + relative, + line, + "DOC025", + f"invalid reference-adapter TOML example: {error}", + ) + continue + errors = sorted( + validator.iter_errors(document), + key=lambda error: tuple(str(part) for part in error.absolute_path), + ) + for error in errors: + location = ".".join(str(part) for part in error.absolute_path) or "" + diagnostics.add( + relative, + line, + "DOC026", + f"reference-adapter example {location}: {error.message}", + ) + return count + + +def _is_reference_adapter_example(token: Token, previous_inline: Token | None) -> bool: + info = token.info.casefold() + if "reference-adapter" in info or "reference_adapter" in info: + return True + if previous_inline is not None and previous_inline.map is not None and token.map is not None: + adjacent = token.map[0] - previous_inline.map[1] <= 1 + context = previous_inline.content.casefold() + if adjacent and ( + "reference-adapter.toml" in context or "reference adapter configuration" in context + ): + return True + keys = set(_REFERENCE_ADAPTER_KEY.findall(token.content)) + return {"language", "source_roots"}.issubset(keys) + + +def _check_command_reference( + pages: dict[Path, _Page], + diagnostics: _Diagnostics, +) -> None: + page = pages.get(COMMAND_REFERENCE) + if page is None: + return + lines = page.source.splitlines() + if not lines or lines[0] != f"# {COMMAND_REFERENCE_H1}": + diagnostics.add( + COMMAND_REFERENCE, + 1, + "DOC027", + f"generated reference must begin with '# {COMMAND_REFERENCE_H1}'", + ) + if COMMAND_REFERENCE_NOTICE not in lines[:5]: + diagnostics.add( + COMMAND_REFERENCE, + 2, + "DOC028", + "generated reference notice is missing or changed", + ) + + +def _token_line(token: Token) -> int: + return token.map[0] + 1 if token.map else 1 + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + description="Check deterministic DocForge documentation invariants." + ) + parser.add_argument("--repository-root", type=Path, default=REPOSITORY_ROOT) + parser.add_argument( + "--pending-inventory", + type=Path, + default=DEFAULT_PENDING_INVENTORY, + help="repository-relative list of required pages not authored yet", + ) + return parser + + +def main(arguments: Sequence[str] | None = None) -> int: + parsed = _parser().parse_args(arguments) + try: + report = check_documentation( + parsed.repository_root, + policy=DocumentationPolicy(pending_inventory=parsed.pending_inventory), + ) + except Exception as error: + print(f"documentation check failed: {error}", file=sys.stderr) + return 2 + if not report.ok: + for diagnostic in report.diagnostics: + print(diagnostic.render(), file=sys.stderr) + if report.omitted_diagnostics: + print( + f"... {report.omitted_diagnostics} additional diagnostics omitted " + f"(limit {MAX_DIAGNOSTICS})", + file=sys.stderr, + ) + return 1 + print( + json.dumps( + { + "markdown_pages": report.markdown_pages, + "pending_pages": report.pending_pages, + "reference_adapter_examples": report.reference_adapter_examples, + "status": "ok", + }, + sort_keys=True, + separators=(",", ":"), + ) + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/docs_pending_m4_pages.txt b/tools/docs_pending_m4_pages.txt new file mode 100644 index 0000000..6b179f8 --- /dev/null +++ b/tools/docs_pending_m4_pages.txt @@ -0,0 +1 @@ +# Milestone 4 required-page inventory is complete. diff --git a/tools/generate_command_reference.py b/tools/generate_command_reference.py new file mode 100644 index 0000000..71d0c9f --- /dev/null +++ b/tools/generate_command_reference.py @@ -0,0 +1,533 @@ +"""Generate or verify the deterministic DocForge CLI and MCP command reference.""" + +from __future__ import annotations + +import argparse +import asyncio +import contextlib +import ctypes +import errno +import fcntl +import json +import os +import secrets +import shutil +import stat +import sys +import tempfile +from collections.abc import Sequence +from dataclasses import dataclass +from pathlib import Path +from typing import Literal, Protocol, cast + +from mcp.shared.memory import create_connected_server_and_client_session + +from docforge.command_reference import ( + CliCommandReference, + McpToolReference, + cli_command_references, + mcp_tool_references, + render_command_reference_markdown, +) +from docforge.index import ProjectIndex +from docforge.mcp_server import ALL_TOOLS, APPLICATION_TOOLS, create_server +from docforge.project import Project + +REPOSITORY_ROOT = Path(__file__).resolve().parents[1] +DEFAULT_PROJECT_ROOT = REPOSITORY_ROOT / "tests" / "fixtures" / "alpha" +EXPECTED_CLI_ROWS = 28 +EXPECTED_MCP_ROWS = 36 +MAX_REFERENCE_BYTES = 5_000_000 +RENAME_EXCHANGE = 2 + + +class CommandReferenceToolError(RuntimeError): + """One safe repository-tool validation or publication failure.""" + + +class CommandReferenceDrift(CommandReferenceToolError): + """The checked output does not match current registered command metadata.""" + + +@dataclass(frozen=True) +class _FileIdentity: + device: int + inode: int + mode: int + owner: int + group: int + size: int + modified_ns: int + changed_ns: int + + +class _RenameAt2(Protocol): + argtypes: list[object] + restype: object + + def __call__( + self, + old_directory_fd: int, + old_name: bytes, + new_directory_fd: int, + new_name: bytes, + flags: int, + /, + ) -> int: ... + + +async def collect_command_reference_rows( + project_root: Path, + *, + proposal_writer: str, + canonical_applier: str, +) -> tuple[tuple[CliCommandReference, ...], tuple[McpToolReference, ...]]: + """Collect the real CLI parser and full project-bound MCP registration surface.""" + + source = _safe_project_root(project_root) + with tempfile.TemporaryDirectory(prefix="docforge-command-reference-") as directory: + copied = Path(directory) / "project" + shutil.copytree(source, copied, symlinks=True) + ProjectIndex(Project.open(copied)).build() + server = create_server( + copied, + proposal_writer, + canonical_applier_id=canonical_applier, + ) + async with create_connected_server_and_client_session( + server, + raise_exceptions=True, + ) as session: + tools = tuple((await session.list_tools()).tools) + + cli_rows = cli_command_references() + expected_mcp_names = (*ALL_TOOLS, *APPLICATION_TOOLS) + mcp_rows = mcp_tool_references(tools, expected_names=expected_mcp_names) + if len(cli_rows) != EXPECTED_CLI_ROWS: + raise CommandReferenceToolError( + f"Expected {EXPECTED_CLI_ROWS} CLI rows, found {len(cli_rows)}" + ) + if len(mcp_rows) != EXPECTED_MCP_ROWS: + raise CommandReferenceToolError( + f"Expected {EXPECTED_MCP_ROWS} MCP rows, found {len(mcp_rows)}" + ) + return cli_rows, mcp_rows + + +def generate_command_reference_bytes( + project_root: Path, + *, + proposal_writer: str = "alpha-editor", + canonical_applier: str = "alpha-editor", +) -> bytes: + """Render one stable Markdown document from an isolated project copy.""" + + cli_rows, mcp_rows = asyncio.run( + collect_command_reference_rows( + project_root, + proposal_writer=proposal_writer, + canonical_applier=canonical_applier, + ) + ) + rendered = render_command_reference_markdown(cli_rows, mcp_rows).encode("utf-8") + if len(rendered) > MAX_REFERENCE_BYTES: + raise CommandReferenceToolError("Generated command reference exceeds its byte limit") + return rendered + + +def publish_or_check_command_reference( + content: bytes, + *, + repository_root: Path, + project_root: Path, + output: Path, + check: bool, +) -> Literal["current", "unchanged", "written"]: + """Safely check or atomically publish one repository-confined Markdown file.""" + + if len(content) > MAX_REFERENCE_BYTES: + raise CommandReferenceToolError("Command reference exceeds its byte limit") + root = _safe_repository_root(repository_root) + source = _safe_project_root(project_root) + relative = _safe_output_relative(root, source, output) + parent_parts = relative.parts[:-1] + target_name = relative.name + parent_fd = _open_relative_directory(root, parent_parts) + try: + fcntl.flock(parent_fd, fcntl.LOCK_EX) + try: + initial = _file_identity(parent_fd, target_name) + existing = ( + _read_regular_file(parent_fd, target_name, limit=MAX_REFERENCE_BYTES) + if initial is not None + else None + ) + if _file_identity(parent_fd, target_name) != initial: + raise CommandReferenceToolError("Output target changed during inspection") + if check: + if existing != content: + raise CommandReferenceDrift( + f"Command reference is missing or stale: {relative.as_posix()}" + ) + return "current" + if existing == content: + return "unchanged" + _atomic_replace( + parent_fd, + target_name, + content, + expected=initial, + expected_content=existing, + ) + return "written" + finally: + fcntl.flock(parent_fd, fcntl.LOCK_UN) + finally: + os.close(parent_fd) + + +def _safe_repository_root(path: Path) -> Path: + if path.is_symlink(): + raise CommandReferenceToolError("Repository root must not be a symbolic link") + try: + resolved = path.resolve(strict=True) + except OSError as error: + raise CommandReferenceToolError("Repository root does not exist") from error + if not resolved.is_dir(): + raise CommandReferenceToolError("Repository root must be a directory") + return resolved + + +def _safe_project_root(path: Path) -> Path: + if path.is_symlink(): + raise CommandReferenceToolError("Project fixture root must not be a symbolic link") + try: + resolved = path.resolve(strict=True) + except OSError as error: + raise CommandReferenceToolError("Project fixture root does not exist") from error + if not resolved.is_dir(): + raise CommandReferenceToolError("Project fixture root must be a directory") + return resolved + + +def _safe_output_relative(root: Path, project_root: Path, output: Path) -> Path: + candidate = output if output.is_absolute() else root / output + absolute = Path(os.path.abspath(candidate)) + if absolute == root or not absolute.is_relative_to(root): + raise CommandReferenceToolError("Output must remain inside the repository root") + if absolute == project_root or absolute.is_relative_to(project_root): + raise CommandReferenceToolError("Output must not modify the source project fixture") + relative = absolute.relative_to(root) + if relative.suffix != ".md": + raise CommandReferenceToolError("Output must be one Markdown file") + if not relative.name or len(relative.parts) < 2: + raise CommandReferenceToolError("Output must be below an existing repository directory") + return relative + + +def _open_relative_directory(root: Path, parts: tuple[str, ...]) -> int: + flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW + descriptor = os.open(root, flags) + try: + for part in parts: + if part in {"", ".", ".."}: + raise CommandReferenceToolError("Output directory is unsafe") + try: + next_descriptor = os.open(part, flags, dir_fd=descriptor) + except OSError as error: + raise CommandReferenceToolError( + "Output parent must be an existing non-symlink directory" + ) from error + os.close(descriptor) + descriptor = next_descriptor + return descriptor + except Exception: + os.close(descriptor) + raise + + +def _file_identity(directory_fd: int, name: str) -> _FileIdentity | None: + try: + status = os.stat(name, dir_fd=directory_fd, follow_symlinks=False) + except FileNotFoundError: + return None + if not stat.S_ISREG(status.st_mode): + raise CommandReferenceToolError("Output target must be a regular file") + return _FileIdentity( + device=status.st_dev, + inode=status.st_ino, + mode=status.st_mode, + owner=status.st_uid, + group=status.st_gid, + size=status.st_size, + modified_ns=status.st_mtime_ns, + changed_ns=status.st_ctime_ns, + ) + + +def _read_regular_file(directory_fd: int, name: str, *, limit: int) -> bytes: + try: + descriptor = os.open(name, os.O_RDONLY | os.O_NOFOLLOW, dir_fd=directory_fd) + except OSError as error: + raise CommandReferenceToolError("Output target could not be opened safely") from error + try: + before = os.fstat(descriptor) + if not stat.S_ISREG(before.st_mode) or before.st_size > limit: + raise CommandReferenceToolError("Output target is not a bounded regular file") + chunks: list[bytes] = [] + remaining = limit + 1 + while remaining: + chunk = os.read(descriptor, min(remaining, 64 * 1024)) + if not chunk: + break + chunks.append(chunk) + remaining -= len(chunk) + content = b"".join(chunks) + after = os.fstat(descriptor) + if len(content) > limit or _identity_from_stat(before) != _identity_from_stat(after): + raise CommandReferenceToolError("Output target changed during inspection") + return content + finally: + os.close(descriptor) + + +def _identity_from_stat(status: os.stat_result) -> _FileIdentity: + return _FileIdentity( + device=status.st_dev, + inode=status.st_ino, + mode=status.st_mode, + owner=status.st_uid, + group=status.st_gid, + size=status.st_size, + modified_ns=status.st_mtime_ns, + changed_ns=status.st_ctime_ns, + ) + + +def _atomic_replace( + directory_fd: int, + name: str, + content: bytes, + *, + expected: _FileIdentity | None, + expected_content: bytes | None, +) -> None: + temporary_name = f".{name}.docforge-command-reference-{secrets.token_hex(12)}" + flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW + try: + descriptor = os.open(temporary_name, flags, 0o644, dir_fd=directory_fd) + except OSError as error: + raise CommandReferenceToolError("Could not create atomic output temporary") from error + try: + view = memoryview(content) + while view: + written = os.write(descriptor, view) + if written <= 0: + raise CommandReferenceToolError("Could not write command reference output") + view = view[written:] + os.fsync(descriptor) + except Exception: + os.close(descriptor) + _unlink_at(directory_fd, temporary_name) + raise + else: + os.close(descriptor) + temporary_contains_only_staged_content = True + try: + staged = _file_identity(directory_fd, temporary_name) + if staged is None: + raise CommandReferenceToolError("Atomic output temporary disappeared") + if _file_identity(directory_fd, name) != expected: + raise CommandReferenceToolError("Output target changed before atomic publication") + if expected is None: + _link_no_replace(directory_fd, temporary_name, name) + _unlink_at(directory_fd, temporary_name) + os.fsync(directory_fd) + return + if expected_content is None: + raise CommandReferenceToolError("Expected output content was not captured") + + _rename_exchange(directory_fd, temporary_name, name) + temporary_contains_only_staged_content = False + displaced = _file_identity(directory_fd, temporary_name) + published = _file_identity(directory_fd, name) + displaced_content = _read_regular_file( + directory_fd, + temporary_name, + limit=MAX_REFERENCE_BYTES, + ) + if ( + displaced is None + or published is None + or not _same_identity_after_rename(displaced, expected) + or displaced_content != expected_content + or not _same_identity_after_rename(published, staged) + ): + try: + _rename_exchange(directory_fd, temporary_name, name) + except Exception as error: + raise CommandReferenceToolError( + "Output target raced publication; displaced data was retained " + f"in {temporary_name}" + ) from error + temporary_contains_only_staged_content = True + if not _same_identity_after_rename( + _file_identity(directory_fd, name), + displaced, + ) or not _same_identity_after_rename( + _file_identity(directory_fd, temporary_name), + staged, + ): + raise CommandReferenceToolError( + "Output target raced publication and could not be safely restored" + ) + raise CommandReferenceToolError("Output target changed during atomic publication") + + _unlink_at(directory_fd, temporary_name) + os.fsync(directory_fd) + except Exception: + if temporary_contains_only_staged_content: + _unlink_at(directory_fd, temporary_name) + raise + + +def _same_identity_after_rename( + actual: _FileIdentity | None, + expected: _FileIdentity | None, +) -> bool: + if actual is None or expected is None: + return False + return ( + actual.device, + actual.inode, + actual.mode, + actual.owner, + actual.group, + actual.size, + actual.modified_ns, + ) == ( + expected.device, + expected.inode, + expected.mode, + expected.owner, + expected.group, + expected.size, + expected.modified_ns, + ) + + +def _link_no_replace(directory_fd: int, source: str, target: str) -> None: + try: + os.link( + source, + target, + src_dir_fd=directory_fd, + dst_dir_fd=directory_fd, + follow_symlinks=False, + ) + except FileExistsError as error: + raise CommandReferenceToolError( + "Output target appeared during atomic publication" + ) from error + except OSError as error: + raise CommandReferenceToolError("Could not publish atomic output") from error + + +def _rename_exchange(directory_fd: int, first: str, second: str) -> None: + rename_at2 = _load_rename_at2() + ctypes.set_errno(0) + result = rename_at2( + directory_fd, + os.fsencode(first), + directory_fd, + os.fsencode(second), + RENAME_EXCHANGE, + ) + if result == 0: + return + error_number = ctypes.get_errno() + if error_number in {errno.ENOSYS, errno.EINVAL, errno.EOPNOTSUPP}: + raise CommandReferenceToolError( + "Atomic exchange publication is unavailable on this filesystem" + ) + raise CommandReferenceToolError("Could not exchange atomic output") from OSError( + error_number, + os.strerror(error_number), + ) + + +def _load_rename_at2() -> _RenameAt2: + library = ctypes.CDLL(None, use_errno=True) + try: + rename_at2 = cast(_RenameAt2, library.renameat2) + except AttributeError as error: + raise CommandReferenceToolError( + "Atomic exchange publication is unavailable on this platform" + ) from error + rename_at2.argtypes = [ + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_uint, + ] + rename_at2.restype = ctypes.c_int + return rename_at2 + + +def _unlink_at(directory_fd: int, name: str) -> None: + with contextlib.suppress(FileNotFoundError): + os.unlink(name, dir_fd=directory_fd) + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + description="Generate or check deterministic DocForge command-reference Markdown." + ) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--check", action="store_true") + parser.add_argument("--repository-root", type=Path, default=REPOSITORY_ROOT) + parser.add_argument("--project-root", type=Path, default=DEFAULT_PROJECT_ROOT) + parser.add_argument("--proposal-writer", default="alpha-editor") + parser.add_argument("--canonical-applier", default="alpha-editor") + return parser + + +def main(arguments: Sequence[str] | None = None) -> int: + parsed = _parser().parse_args(arguments) + try: + content = generate_command_reference_bytes( + parsed.project_root, + proposal_writer=parsed.proposal_writer, + canonical_applier=parsed.canonical_applier, + ) + state = publish_or_check_command_reference( + content, + repository_root=parsed.repository_root, + project_root=parsed.project_root, + output=parsed.output, + check=parsed.check, + ) + except CommandReferenceDrift as error: + print(str(error), file=sys.stderr) + return 1 + except Exception as error: + print(f"command-reference generation failed: {error}", file=sys.stderr) + return 2 + print( + json.dumps( + { + "status": state, + "output": str(parsed.output), + "cli_rows": EXPECTED_CLI_ROWS, + "mcp_rows": EXPECTED_MCP_ROWS, + }, + sort_keys=True, + separators=(",", ":"), + ) + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/milestone4_adoption.py b/tools/milestone4_adoption.py new file mode 100644 index 0000000..3e4d47e --- /dev/null +++ b/tools/milestone4_adoption.py @@ -0,0 +1,430 @@ +#!/usr/bin/env python3 +"""Run the offline Milestone 4 fresh-wheel adoption proof.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path +from typing import cast + +PROOF_SCHEMA_VERSION = 1 +COMMAND_TIMEOUT_SECONDS = 120 +VALIDATION_TIMEOUT_SECONDS = 45 +OPTIONAL_FRONTEND_MODULES = ( + "tree_sitter", + "tree_sitter_cpp", + "tree_sitter_javascript", + "tree_sitter_typescript", +) + +_VALIDATION_PROGRAM = r""" +from __future__ import annotations + +import asyncio +import importlib.metadata +import importlib.util +import json +import sys +from pathlib import Path + +from docforge.errors import DocForgeError +from docforge.index import ProjectIndex +from docforge.reference_mcp import create_reference_project +from mcp import ClientSession, StdioServerParameters +from mcp.client.stdio import stdio_client + + +def write_config( + root: Path, + *, + language: str, + source_roots: tuple[str, ...], + compilation_database: str | None = None, +) -> None: + config = root / ".docforge" / "reference-adapter.toml" + config.parent.mkdir(parents=True) + roots = ", ".join(json.dumps(item) for item in source_roots) + lines = [ + "schema_version = 1", + 'project_id = "fresh-wheel-adoption"', + 'title = "Fresh wheel adoption proof"', + f"language = {json.dumps(language)}", + f"source_roots = [{roots}]", + ] + if compilation_database is not None: + lines.append(f"compilation_database = {json.dumps(compilation_database)}") + config.write_text("\n".join((*lines, "")), encoding="utf-8") + + +async def retrieve(python: Path, root: Path) -> dict[str, object]: + parameters = StdioServerParameters( + command=str(python), + args=[ + "-I", + "-m", + "docforge.reference_mcp", + "--project-root", + str(root), + "--capability-mode", + "read", + ], + env={}, + ) + async with ( + stdio_client(parameters) as streams, + ClientSession(*streams) as session, + ): + await session.initialize() + tools = tuple(tool.name for tool in (await session.list_tools()).tools) + bootstrap = await session.call_tool("docforge_bootstrap", {}) + search = await session.call_tool( + "docforge_search", + {"query": "Service", "limit": 5}, + ) + if search.isError or not search.structuredContent: + raise RuntimeError("fresh-wheel reference MCP search failed") + results = search.structuredContent.get("results") + if not isinstance(results, list) or not results: + raise RuntimeError("fresh-wheel reference MCP search returned no evidence") + first = results[0] + if not isinstance(first, dict) or not isinstance(first.get("node_id"), str): + raise RuntimeError("fresh-wheel reference MCP search returned invalid evidence") + node_id = first["node_id"] + node = await session.call_tool("docforge_get_node", {"node_id": node_id}) + if bootstrap.isError or not bootstrap.structuredContent: + raise RuntimeError("fresh-wheel reference MCP bootstrap failed") + if node.isError or not node.structuredContent: + raise RuntimeError("fresh-wheel reference MCP node retrieval failed") + retrieved = node.structuredContent.get("node") + if not isinstance(retrieved, dict) or retrieved.get("node_id") != node_id: + raise RuntimeError("fresh-wheel reference MCP returned the wrong node") + return { + "bootstrap_status": bootstrap.structuredContent.get("status"), + "node_id": node_id, + "read_tool_count": len(tools), + } + + +workspace = Path(sys.argv[1]).resolve(strict=True) +python = Path(sys.executable).absolute() + +present_modules = [ + name + for name in ( + "tree_sitter", + "tree_sitter_cpp", + "tree_sitter_javascript", + "tree_sitter_typescript", + ) + if importlib.util.find_spec(name) is not None +] +installed_frontends = sorted( + distribution.metadata["Name"] + for distribution in importlib.metadata.distributions() + if distribution.metadata["Name"].casefold().startswith("tree-sitter") +) +if present_modules or installed_frontends: + raise RuntimeError( + f"base wheel unexpectedly installed language frontends: " + f"{present_modules or installed_frontends}" + ) + +python_root = workspace / "python-project" +(python_root / "src" / "sample").mkdir(parents=True) +(python_root / "src" / "sample" / "__init__.py").write_text("", encoding="utf-8") +(python_root / "src" / "sample" / "service.py").write_text( + "class Service:\n" + " def execute(self, ready: bool) -> str:\n" + " if ready:\n" + ' return "ready"\n' + ' return "waiting"\n', + encoding="utf-8", +) +write_config(python_root, language="python", source_roots=("src",)) + +project = create_reference_project(python_root) +index = ProjectIndex(project) +built = index.build() +checked = index.check() +if built.get("status") != "ok" or checked.get("status") != "ok": + raise RuntimeError("fresh-wheel Python reference build/check failed") +mcp = asyncio.run(retrieve(python, python_root)) +if mcp["bootstrap_status"] != "ok": + raise RuntimeError("fresh-wheel reference MCP bootstrap did not report ok") + +cpp_root = workspace / "cpp-project" +(cpp_root / "src").mkdir(parents=True) +(cpp_root / "src" / "main.cpp").write_text("int main() { return 0; }\n", encoding="utf-8") +(cpp_root / "compile_commands.json").write_text( + json.dumps( + [ + { + "directory": str(cpp_root), + "file": "src/main.cpp", + "arguments": ["c++", "-c", "src/main.cpp"], + } + ], + sort_keys=True, + ) + + "\n", + encoding="utf-8", +) +write_config( + cpp_root, + language="cpp", + source_roots=("src",), + compilation_database="compile_commands.json", +) + +try: + ProjectIndex(create_reference_project(cpp_root)).build() +except DocForgeError as error: + if error.code != "optional_dependency_missing": + raise + if error.details.get("install") != "docforge[cpp]": + raise RuntimeError("C++ missing-extra failure was not actionable") from error + cpp_failure = { + "code": error.code, + "install": error.details["install"], + "missing_module": error.details.get("missing_module"), + } +else: + raise RuntimeError("C++ reference build unexpectedly worked without docforge[cpp]") + +print( + json.dumps( + { + "schema_version": 1, + "base_frontend_modules": present_modules, + "base_frontend_distributions": installed_frontends, + "python": { + "build_status": built["status"], + "check_status": checked["status"], + "node_count": checked["node_count"], + }, + "mcp": mcp, + "cpp_without_extra": cpp_failure, + }, + sort_keys=True, + ) +) +""" + + +class AdoptionProofError(RuntimeError): + """The disposable adoption proof could not be completed.""" + + +def _run_checked( + arguments: list[str], + *, + cwd: Path, + timeout: int, + environment: dict[str, str] | None = None, +) -> subprocess.CompletedProcess[str]: + result = subprocess.run( + arguments, + cwd=cwd, + env=environment, + text=True, + capture_output=True, + check=False, + timeout=timeout, + ) + if result.returncode != 0: + detail = result.stderr.strip() or result.stdout.strip() or "no command output" + raise AdoptionProofError( + f"command failed with exit {result.returncode}: {arguments[0]}\n{detail}" + ) + return result + + +def find_wheel(directory: Path) -> Path: + wheels = tuple(sorted(directory.glob("docforge-*.whl"))) + if len(wheels) != 1: + raise AdoptionProofError(f"expected exactly one DocForge wheel, found {len(wheels)}") + return wheels[0] + + +def parse_evidence(raw: str) -> dict[str, object]: + try: + loaded: object = json.loads(raw) + except json.JSONDecodeError as error: + raise AdoptionProofError("fresh-wheel validation returned invalid JSON") from error + if not isinstance(loaded, dict): + raise AdoptionProofError("fresh-wheel validation returned the wrong schema") + document = cast(dict[str, object], loaded) + if document.get("schema_version") != PROOF_SCHEMA_VERSION: + raise AdoptionProofError("fresh-wheel validation returned the wrong schema") + if document.get("base_frontend_modules") != []: + raise AdoptionProofError("fresh-wheel validation found optional frontend modules") + if document.get("base_frontend_distributions") != []: + raise AdoptionProofError("fresh-wheel validation found optional frontend distributions") + raw_python = document.get("python") + raw_mcp = document.get("mcp") + raw_cpp = document.get("cpp_without_extra") + python = cast(dict[str, object], raw_python) if isinstance(raw_python, dict) else None + mcp = cast(dict[str, object], raw_mcp) if isinstance(raw_mcp, dict) else None + cpp = cast(dict[str, object], raw_cpp) if isinstance(raw_cpp, dict) else None + if ( + python is None + or python.get("build_status") != "ok" + or python.get("check_status") != "ok" + or mcp is None + or mcp.get("bootstrap_status") != "ok" + or cpp is None + or cpp.get("code") != "optional_dependency_missing" + or cpp.get("install") != "docforge[cpp]" + ): + raise AdoptionProofError("fresh-wheel validation evidence is incomplete") + return document + + +def run_adoption_proof(repository_root: Path, *, uv: str = "uv") -> dict[str, object]: + """Build and exercise a base wheel without using the network.""" + + root = repository_root.resolve(strict=True) + if not root.joinpath("pyproject.toml").is_file() or not root.joinpath("uv.lock").is_file(): + raise AdoptionProofError("repository root lacks pyproject.toml or uv.lock") + + offline_environment = { + **os.environ, + "UV_OFFLINE": "1", + "UV_PYTHON_DOWNLOADS": "never", + } + offline_environment.pop("PYTHONPATH", None) + with tempfile.TemporaryDirectory(prefix="docforge-m4-adoption-") as directory: + workspace = Path(directory).resolve() + distributions = workspace / "dist" + requirements = workspace / "base-requirements.txt" + virtual_environment = workspace / "venv" + python = virtual_environment / "bin" / "python" + + _run_checked( + [ + uv, + "build", + "--wheel", + "--offline", + "--no-python-downloads", + "--out-dir", + str(distributions), + str(root), + ], + cwd=root, + timeout=COMMAND_TIMEOUT_SECONDS, + environment=offline_environment, + ) + wheel = find_wheel(distributions) + _run_checked( + [ + uv, + "export", + "--frozen", + "--offline", + "--no-dev", + "--no-emit-project", + "--output-file", + str(requirements), + ], + cwd=root, + timeout=COMMAND_TIMEOUT_SECONDS, + environment=offline_environment, + ) + exported = requirements.read_text(encoding="utf-8").casefold() + if any(module.replace("_", "-") in exported for module in OPTIONAL_FRONTEND_MODULES): + raise AdoptionProofError("locked base dependency export contains a language frontend") + _run_checked( + [ + uv, + "venv", + "--python", + sys.executable, + "--no-project", + "--no-python-downloads", + str(virtual_environment), + ], + cwd=workspace, + timeout=COMMAND_TIMEOUT_SECONDS, + environment=offline_environment, + ) + _run_checked( + [ + uv, + "pip", + "sync", + "--python", + str(python), + "--offline", + "--strict", + str(requirements), + ], + cwd=workspace, + timeout=COMMAND_TIMEOUT_SECONDS, + environment=offline_environment, + ) + _run_checked( + [ + uv, + "pip", + "install", + "--python", + str(python), + "--offline", + "--strict", + "--no-deps", + str(wheel), + ], + cwd=workspace, + timeout=COMMAND_TIMEOUT_SECONDS, + environment=offline_environment, + ) + validation = _run_checked( + [str(python), "-I", "-c", _VALIDATION_PROGRAM, str(workspace)], + cwd=workspace, + timeout=VALIDATION_TIMEOUT_SECONDS, + environment=offline_environment, + ) + evidence = parse_evidence(validation.stdout.strip()) + evidence["wheel"] = { + "filename": wheel.name, + "sha256": hashlib.sha256(wheel.read_bytes()).hexdigest(), + } + evidence["network"] = "offline" + return evidence + + +def main() -> None: + parser = argparse.ArgumentParser( + description="Run the offline Milestone 4 fresh-wheel adoption proof" + ) + parser.add_argument( + "--repository-root", + type=Path, + default=Path(__file__).resolve().parents[1], + ) + parser.add_argument("--uv", default=shutil.which("uv") or "uv") + parser.add_argument("--output", type=Path) + arguments = parser.parse_args() + + try: + evidence = run_adoption_proof(arguments.repository_root, uv=arguments.uv) + except (AdoptionProofError, OSError, subprocess.SubprocessError) as error: + print(json.dumps({"status": "error", "error": str(error)}, sort_keys=True)) + raise SystemExit(1) from error + + payload = json.dumps({"status": "ok", **evidence}, indent=2, sort_keys=True) + "\n" + if arguments.output is not None: + arguments.output.write_text(payload, encoding="utf-8") + print(payload, end="") + + +if __name__ == "__main__": + main() diff --git a/tools/milestone4_benchmark.py b/tools/milestone4_benchmark.py new file mode 100644 index 0000000..c6a41c1 --- /dev/null +++ b/tools/milestone4_benchmark.py @@ -0,0 +1,498 @@ +"""Milestone 4 Python adapter SDK, incremental, and recovery benchmark gates.""" + +from __future__ import annotations + +import argparse +import ast +import gc +import hashlib +import json +import math +import platform +import resource +import statistics +import subprocess +import sys +import tempfile +import time +import tracemalloc +from collections.abc import Callable, Generator, Mapping +from contextlib import contextmanager +from dataclasses import dataclass +from pathlib import Path +from typing import cast +from unittest import mock + +from docforge.adapter_sdk import ( + AdapterAssembly, + AdapterProject, + AdapterSource, + AdapterSourceProjection, +) +from docforge.adapters.python import PythonReferenceAdapter +from docforge.index import ProjectIndex + +ROOT = Path(__file__).resolve().parents[1] +FULL_SOURCE_COUNT = 334 +SMOKE_SOURCE_COUNT = 12 +NODES_PER_SOURCE = 3 +DEFAULT_FULL_SAMPLES = 3 +DEFAULT_SMOKE_SAMPLES = 1 + +# These are regression tripwires, not performance targets. They deliberately +# leave several times the Milestone 3 1,000-node in-process allowances for the +# additional AST, Logic, extraction-cache, and SQLite work in this gate. +MAX_TRACED_PEAK_BYTES = 256 * 1024 * 1024 +MAX_PROCESS_PEAK_BYTES = 512 * 1024 * 1024 +MAX_RESPONSE_BYTES = 256 * 1024 +COLD_LIMIT_MS = 30_000.0 +WARM_LIMIT_MS = 20_000.0 +EQUIVALENCE_LIMIT_MS = 30_000.0 +RECOVERY_LIMIT_MS = 30_000.0 + + +@dataclass(frozen=True) +class _InstrumentedResult: + value: Mapping[str, object] + ast_parse_calls: int + extraction_calls: int + + +class _RecordingPythonAdapter(PythonReferenceAdapter): + """Count extraction entry points without weakening the production adapter.""" + + def __init__(self, root: Path) -> None: + super().__init__( + root, + source_roots=("src",), + project_id="milestone4-python-benchmark", + title="Milestone 4 Python adapter benchmark", + ) + self.extraction_calls = 0 + + def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: + self.extraction_calls += 1 + return super().extract_source(source) + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + description=("Gate DocForge2 Milestone 4 adapter behavior on a disposable Python project.") + ) + parser.add_argument("--mode", choices=("smoke", "full"), default="full") + parser.add_argument("--sources", type=int) + parser.add_argument("--samples", type=int) + parser.add_argument("--output", type=Path) + return parser + + +def encode_report(value: object) -> str: + """Serialize benchmark evidence in one deterministic JSON representation.""" + + return json.dumps(value, sort_keys=True, indent=2, ensure_ascii=False) + "\n" + + +def _compact_json(value: object) -> bytes: + return json.dumps( + value, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + ).encode("utf-8") + + +def _sha256(value: object) -> str: + return hashlib.sha256(_compact_json(value)).hexdigest() + + +def _git(arguments: list[str]) -> str: + return subprocess.run( + ["git", *arguments], + cwd=ROOT, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + +def write_python_fixture(root: Path, source_count: int) -> None: + """Write a deterministic project with three primary nodes per source.""" + + if source_count < 2: + raise ValueError("The Milestone 4 fixture requires at least two sources") + package = root / "src" / "benchmark" + package.mkdir(parents=True) + for index in range(source_count): + dependency = "" + call = f"value = {index}" + if index: + dependency = f"from benchmark.module_{index - 1:04d} import compute_{index - 1:04d}\n\n" + call = f"value = compute_{index - 1:04d}(value)" + source = ( + f'"""Synthetic adapter benchmark module {index:04d}."""\n\n' + f"{dependency}" + f"def compute_{index:04d}(value: int) -> int:\n" + f' """Return one deterministic branch result."""\n' + f" {call}\n" + " if value % 2:\n" + f" return value + {index + 1}\n" + " return value\n" + ) + (package / f"module_{index:04d}.py").write_text(source, encoding="utf-8") + + +@contextmanager +def _count_ast_parses() -> Generator[mock.MagicMock]: + with mock.patch.object(ast, "parse", wraps=ast.parse) as parsed: + yield parsed + + +def _instrument( + adapter: _RecordingPythonAdapter, + operation: Callable[[], Mapping[str, object]], +) -> _InstrumentedResult: + adapter.extraction_calls = 0 + with _count_ast_parses() as counter: + value = operation() + return _InstrumentedResult( + value=value, + ast_parse_calls=counter.call_count, + extraction_calls=adapter.extraction_calls, + ) + + +def _build_summary(sample: _InstrumentedResult) -> dict[str, object]: + value = sample.value + build = cast(Mapping[str, object], value["build"]) + return { + "status": value["status"], + "revision": value["revision"], + "source_hash": value["source_hash"], + "node_hash": value["node_hash"], + "node_count": value["node_count"], + "edge_hash": value["edge_hash"], + "edge_count": value["edge_count"], + "logic_hash": value["logic_hash"], + "logic_projection_count": value["logic_projection_count"], + "logic_node_count": value["logic_node_count"], + "logic_edge_count": value["logic_edge_count"], + "cache_hits": build["cache_hits"], + "reparsed_sources": build["reparsed_sources"], + "invalidated_sources": build["invalidated_sources"], + "deleted_sources": build["deleted_sources"], + "total_sources": build["total_sources"], + "ast_parse_calls": sample.ast_parse_calls, + "extraction_calls": sample.extraction_calls, + } + + +def _equivalence_summary(sample: _InstrumentedResult) -> dict[str, object]: + return { + **sample.value, + "ast_parse_calls": sample.ast_parse_calls, + "extraction_calls": sample.extraction_calls, + } + + +def _synchronization_summary(sample: _InstrumentedResult) -> dict[str, object]: + synchronization = cast(Mapping[str, object], sample.value["synchronization"]) + build = cast(Mapping[str, object], synchronization["build"]) + return { + "status": sample.value["status"], + "revision": sample.value["revision"], + "source_hash": sample.value["source_hash"], + "node_hash": sample.value["node_hash"], + "node_count": sample.value["node_count"], + "edge_hash": sample.value["edge_hash"], + "edge_count": sample.value["edge_count"], + "logic_hash": sample.value["logic_hash"], + "logic_projection_count": sample.value["logic_projection_count"], + "action": synchronization["action"], + "initial_error_code": cast(Mapping[str, object], synchronization["initial_error"])["code"], + "cache_hits": build["cache_hits"], + "reparsed_sources": build["reparsed_sources"], + "ast_parse_calls": sample.ast_parse_calls, + "extraction_calls": sample.extraction_calls, + } + + +def _measure( + operation: Callable[[], _InstrumentedResult], + *, + samples: int, + p95_limit_ms: float, + summary: Callable[[_InstrumentedResult], Mapping[str, object]], +) -> tuple[dict[str, object], _InstrumentedResult]: + durations: list[float] = [] + traced_peaks: list[int] = [] + response_sizes: list[int] = [] + stable_summary: dict[str, object] | None = None + last: _InstrumentedResult | None = None + for _ in range(samples): + gc.collect() + tracemalloc.start() + started = time.perf_counter_ns() + try: + current = operation() + elapsed_ms = (time.perf_counter_ns() - started) / 1_000_000 + _, traced_peak = tracemalloc.get_traced_memory() + finally: + tracemalloc.stop() + current_summary = dict(summary(current)) + if stable_summary is None: + stable_summary = current_summary + elif current_summary != stable_summary: + raise RuntimeError("Milestone 4 operation changed its deterministic result") + response_bytes = len(_compact_json(current.value)) + if response_bytes > MAX_RESPONSE_BYTES: + raise RuntimeError( + "Milestone 4 response exceeded its fixed byte boundary: " + f"{response_bytes} > {MAX_RESPONSE_BYTES}" + ) + if traced_peak > MAX_TRACED_PEAK_BYTES: + raise RuntimeError( + "Milestone 4 operation exceeded its traced-memory boundary: " + f"{traced_peak} > {MAX_TRACED_PEAK_BYTES}" + ) + durations.append(elapsed_ms) + traced_peaks.append(traced_peak) + response_sizes.append(response_bytes) + last = current + ordered = sorted(durations) + p95_index = max(0, math.ceil(len(ordered) * 0.95) - 1) + p95 = ordered[p95_index] + if p95 > p95_limit_ms: + raise RuntimeError(f"Milestone 4 operation p95 {p95:.3f} ms exceeds {p95_limit_ms:.3f} ms") + assert stable_summary is not None + assert last is not None + return ( + { + "samples": samples, + "median_ms": round(statistics.median(ordered), 3), + "p95_ms": round(p95, 3), + "min_ms": round(ordered[0], 3), + "max_ms": round(ordered[-1], 3), + "p95_limit_ms": p95_limit_ms, + "maximum_response_bytes": max(response_sizes), + "response_limit_bytes": MAX_RESPONSE_BYTES, + "maximum_traced_peak_bytes": max(traced_peaks), + "traced_peak_limit_bytes": MAX_TRACED_PEAK_BYTES, + "stable_result": stable_summary, + }, + last, + ) + + +def _semantic_assembly_hash(assembly: AdapterAssembly) -> str: + projection = assembly.projection + return _sha256( + { + "project_id": projection.project_id, + "title": projection.title, + "adapter_id": projection.adapter_id, + "adapter_version": projection.adapter_version, + "revision": projection.revision, + "source_hash": projection.source_hash, + "nodes": [item.as_dict() for item in projection.nodes], + "edges": [item.as_dict() for item in projection.edges], + "logic": [item.as_dict() for item in assembly.logic], + } + ) + + +def _benchmark(root: Path, source_count: int, samples: int) -> dict[str, object]: + write_python_fixture(root, source_count) + adapter = _RecordingPythonAdapter(root) + cache_root = root / ".docforge" / "benchmark" + project = AdapterProject(adapter, cache_root=cache_root) + index = ProjectIndex(project) + + cold, cold_sample = _measure( + lambda: _instrument(adapter, index.build), + samples=1, + p95_limit_ms=COLD_LIMIT_MS, + summary=_build_summary, + ) + cold_summary = _build_summary(cold_sample) + expected_nodes = source_count * NODES_PER_SOURCE + if ( + cold_summary["node_count"] != expected_nodes + or cold_summary["logic_projection_count"] != source_count + or cold_summary["reparsed_sources"] != source_count + or cast(int, cold_summary["ast_parse_calls"]) < source_count + or cold_summary["extraction_calls"] != source_count + ): + raise RuntimeError("Cold adapter build did not cover the complete synthetic fixture") + + warm, warm_sample = _measure( + lambda: _instrument(adapter, index.build), + samples=samples, + p95_limit_ms=WARM_LIMIT_MS, + summary=_build_summary, + ) + warm_summary = _build_summary(warm_sample) + if ( + warm_summary["cache_hits"] != source_count + or warm_summary["reparsed_sources"] != 0 + or warm_summary["ast_parse_calls"] != 0 + or warm_summary["extraction_calls"] != 0 + ): + raise RuntimeError("Warm adapter build performed forbidden parsing or extraction") + + equivalence, equivalence_sample = _measure( + lambda: _instrument(adapter, project.verify_incremental_equivalence), + samples=1, + p95_limit_ms=EQUIVALENCE_LIMIT_MS, + summary=_equivalence_summary, + ) + equivalence_summary = _equivalence_summary(equivalence_sample) + if ( + equivalence_summary["status"] != "ok" + or equivalence_summary["node_count"] != expected_nodes + or equivalence_summary["logic_projection_count"] != source_count + ): + raise RuntimeError("Full and incremental graph plus Logic output was not equivalent") + complete_assembly = adapter.load_complete_assembly() + assembly_hash = _semantic_assembly_hash(complete_assembly) + + extraction_cache = cache_root / "extractions.json" + extraction_cache.write_bytes(b"{broken") + cache_recovery, cache_recovery_sample = _measure( + lambda: _instrument(adapter, index.build), + samples=1, + p95_limit_ms=RECOVERY_LIMIT_MS, + summary=_build_summary, + ) + cache_recovery_summary = _build_summary(cache_recovery_sample) + if ( + cache_recovery_summary["reparsed_sources"] != source_count + or cast(int, cache_recovery_summary["ast_parse_calls"]) < source_count + or cache_recovery_summary["node_hash"] != cold_summary["node_hash"] + or cache_recovery_summary["logic_hash"] != cold_summary["logic_hash"] + ): + raise RuntimeError("Corrupt extraction-cache recovery changed adapter output") + + index.path.write_bytes(b"not-a-sqlite-index") + index_recovery, index_recovery_sample = _measure( + lambda: _instrument(adapter, index.synchronize), + samples=1, + p95_limit_ms=RECOVERY_LIMIT_MS, + summary=_synchronization_summary, + ) + index_recovery_summary = _synchronization_summary(index_recovery_sample) + if ( + index_recovery_summary["action"] != "rebuilt" + or index_recovery_summary["cache_hits"] != source_count + or index_recovery_summary["reparsed_sources"] != 0 + or index_recovery_summary["ast_parse_calls"] != 0 + or index_recovery_summary["extraction_calls"] != 0 + or index_recovery_summary["node_hash"] != cold_summary["node_hash"] + or index_recovery_summary["logic_hash"] != cold_summary["logic_hash"] + ): + raise RuntimeError("Corrupt derived-index recovery changed adapter output") + + evidence = { + "source_count": source_count, + "node_count": expected_nodes, + "edge_count": cold_summary["edge_count"], + "logic_projection_count": source_count, + "logic_node_count": cold_summary["logic_node_count"], + "logic_edge_count": cold_summary["logic_edge_count"], + "source_hash": cold_summary["source_hash"], + "node_hash": cold_summary["node_hash"], + "edge_hash": cold_summary["edge_hash"], + "logic_hash": cold_summary["logic_hash"], + "complete_assembly_hash": assembly_hash, + "full_incremental_graph_and_logic_exact": True, + "warm_zero_ast_parse": True, + "warm_zero_source_extraction": True, + "cache_recovery_exact": True, + "index_recovery_exact": True, + } + return { + "fixture": { + "kind": "synthetic_python_reference_adapter", + "source_count": source_count, + "nodes_per_source": NODES_PER_SOURCE, + "expected_node_count": expected_nodes, + }, + "operations": { + "cold_incremental_build": cold, + "warm_incremental_build": warm, + "full_incremental_equivalence": equivalence, + "corrupt_extraction_cache_recovery": cache_recovery, + "corrupt_index_recovery": index_recovery, + }, + "evidence": evidence, + "evidence_sha256": _sha256(evidence), + } + + +def main() -> int: + arguments = _parser().parse_args() + default_sources = FULL_SOURCE_COUNT if arguments.mode == "full" else SMOKE_SOURCE_COUNT + default_samples = DEFAULT_FULL_SAMPLES if arguments.mode == "full" else DEFAULT_SMOKE_SAMPLES + source_count = default_sources if arguments.sources is None else arguments.sources + samples = default_samples if arguments.samples is None else arguments.samples + if not 2 <= source_count <= FULL_SOURCE_COUNT: + raise SystemExit(f"--sources must be between 2 and {FULL_SOURCE_COUNT}") + if arguments.mode == "full" and source_count != FULL_SOURCE_COUNT: + raise SystemExit(f"--mode full requires exactly {FULL_SOURCE_COUNT} sources") + if samples < 1: + raise SystemExit("--samples must be positive") + + with tempfile.TemporaryDirectory(prefix="docforge-milestone4-") as directory: + measurement = _benchmark(Path(directory).resolve(), source_count, samples) + + process_peak_bytes = int(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss) * 1024 + if process_peak_bytes > MAX_PROCESS_PEAK_BYTES: + raise RuntimeError( + "Milestone 4 process peak exceeded its memory boundary: " + f"{process_peak_bytes} > {MAX_PROCESS_PEAK_BYTES}" + ) + status = _git(["status", "--porcelain"]) + report: dict[str, object] = { + "schema_version": 1, + "benchmark": "docforge2_milestone4", + "mode": arguments.mode, + "source": { + "revision": _git(["rev-parse", "HEAD"]), + "dirty": bool(status), + }, + "environment": { + "platform": platform.platform(), + "machine": platform.machine(), + "python": platform.python_version(), + "implementation": platform.python_implementation(), + }, + "method": { + "clock": "time.perf_counter_ns", + "in_process_peak_memory": "tracemalloc per measured invocation", + "process_peak_memory": "resource.getrusage(RUSAGE_SELF).ru_maxrss", + "response_size": "UTF-8 bytes of compact sorted JSON", + "ast_instrumentation": "temporary counter around stdlib ast.parse", + "extraction_instrumentation": "reference adapter extract_source entry counter", + "samples": samples, + "full_mode_source_requirement": FULL_SOURCE_COUNT, + "threshold_basis": ( + "Regression tripwires leave several times the Milestone 3 1,000-node " + "allowances for AST, Logic, extraction-cache, and SQLite work." + ), + }, + **measurement, + "memory": { + "process_peak_bytes": process_peak_bytes, + "process_peak_limit_bytes": MAX_PROCESS_PEAK_BYTES, + }, + } + encoded = encode_report(report) + if arguments.output is not None: + output = arguments.output.resolve() + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text(encoded, encoding="utf-8") + sys.stdout.write(encoded) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/uv.lock b/uv.lock index 21b0bfd..de8ab13 100644 --- a/uv.lock +++ b/uv.lock @@ -211,9 +211,26 @@ source = { editable = "." } dependencies = [ { name = "markdown-it-py" }, { name = "mcp" }, +] + +[package.optional-dependencies] +cpp = [ + { name = "tree-sitter" }, + { name = "tree-sitter-cpp" }, +] +javascript = [ + { name = "tree-sitter" }, + { name = "tree-sitter-javascript" }, +] +languages = [ { name = "tree-sitter" }, { name = "tree-sitter-cpp" }, { name = "tree-sitter-javascript" }, + { name = "tree-sitter-typescript" }, +] +typescript = [ + { name = "tree-sitter" }, + { name = "tree-sitter-typescript" }, ] [package.dev-dependencies] @@ -221,22 +238,38 @@ dev = [ { name = "jsonschema" }, { name = "pytest" }, { name = "ruff" }, + { name = "tree-sitter" }, + { name = "tree-sitter-cpp" }, + { name = "tree-sitter-javascript" }, + { name = "tree-sitter-typescript" }, ] [package.metadata] requires-dist = [ { name = "markdown-it-py", specifier = ">=4.2,<5" }, { name = "mcp", specifier = ">=1.28,<2" }, - { name = "tree-sitter", specifier = ">=0.25,<0.26" }, - { name = "tree-sitter-cpp", specifier = ">=0.23,<0.24" }, - { name = "tree-sitter-javascript", specifier = ">=0.25,<0.26" }, + { name = "tree-sitter", marker = "extra == 'cpp'", specifier = ">=0.25,<0.26" }, + { name = "tree-sitter", marker = "extra == 'javascript'", specifier = ">=0.25,<0.26" }, + { name = "tree-sitter", marker = "extra == 'languages'", specifier = ">=0.25,<0.26" }, + { name = "tree-sitter", marker = "extra == 'typescript'", specifier = ">=0.25,<0.26" }, + { name = "tree-sitter-cpp", marker = "extra == 'cpp'", specifier = ">=0.23,<0.24" }, + { name = "tree-sitter-cpp", marker = "extra == 'languages'", specifier = ">=0.23,<0.24" }, + { name = "tree-sitter-javascript", marker = "extra == 'javascript'", specifier = ">=0.25,<0.26" }, + { name = "tree-sitter-javascript", marker = "extra == 'languages'", specifier = ">=0.25,<0.26" }, + { name = "tree-sitter-typescript", marker = "extra == 'languages'", specifier = ">=0.23,<0.24" }, + { name = "tree-sitter-typescript", marker = "extra == 'typescript'", specifier = ">=0.23,<0.24" }, ] +provides-extras = ["javascript", "typescript", "cpp", "languages"] [package.metadata.requires-dev] dev = [ { name = "jsonschema", specifier = ">=4.25,<5" }, { name = "pytest", specifier = ">=9.1,<10" }, { name = "ruff", specifier = ">=0.15,<1" }, + { name = "tree-sitter", specifier = ">=0.25,<0.26" }, + { name = "tree-sitter-cpp", specifier = ">=0.23,<0.24" }, + { name = "tree-sitter-javascript", specifier = ">=0.25,<0.26" }, + { name = "tree-sitter-typescript", specifier = ">=0.23,<0.24" }, ] [[package]] @@ -804,6 +837,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/2e/1f/f9eba1038b7d4394410f3c0a6ec2122b590cd7acb03f196e52fa57ebbe72/tree_sitter_javascript-0.25.0-cp310-abi3-win_arm64.whl", hash = "sha256:622a69d677aa7f6ee2931d8c77c981a33f0ebb6d275aa9d43d3397c879a9bb0b", size = 61668, upload-time = "2025-09-01T07:13:43.803Z" }, ] +[[package]] +name = "tree-sitter-typescript" +version = "0.23.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1e/fc/bb52958f7e399250aee093751e9373a6311cadbe76b6e0d109b853757f35/tree_sitter_typescript-0.23.2.tar.gz", hash = "sha256:7b167b5827c882261cb7a50dfa0fb567975f9b315e87ed87ad0a0a3aedb3834d", size = 773053, upload-time = "2024-11-11T02:36:11.396Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/28/95/4c00680866280e008e81dd621fd4d3f54aa3dad1b76b857a19da1b2cc426/tree_sitter_typescript-0.23.2-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:3cd752d70d8e5371fdac6a9a4df9d8924b63b6998d268586f7d374c9fba2a478", size = 286677, upload-time = "2024-11-11T02:35:58.839Z" }, + { url = "https://files.pythonhosted.org/packages/8f/2f/1f36fda564518d84593f2740d5905ac127d590baf5c5753cef2a88a89c15/tree_sitter_typescript-0.23.2-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:c7cc1b0ff5d91bac863b0e38b1578d5505e718156c9db577c8baea2557f66de8", size = 302008, upload-time = "2024-11-11T02:36:00.733Z" }, + { url = "https://files.pythonhosted.org/packages/96/2d/975c2dad292aa9994f982eb0b69cc6fda0223e4b6c4ea714550477d8ec3a/tree_sitter_typescript-0.23.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4b1eed5b0b3a8134e86126b00b743d667ec27c63fc9de1b7bb23168803879e31", size = 351987, upload-time = "2024-11-11T02:36:02.669Z" }, + { url = "https://files.pythonhosted.org/packages/49/d1/a71c36da6e2b8a4ed5e2970819b86ef13ba77ac40d9e333cb17df6a2c5db/tree_sitter_typescript-0.23.2-cp39-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e96d36b85bcacdeb8ff5c2618d75593ef12ebaf1b4eace3477e2bdb2abb1752c", size = 344960, upload-time = "2024-11-11T02:36:04.443Z" }, + { url = "https://files.pythonhosted.org/packages/7f/cb/f57b149d7beed1a85b8266d0c60ebe4c46e79c9ba56bc17b898e17daf88e/tree_sitter_typescript-0.23.2-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:8d4f0f9bcb61ad7b7509d49a1565ff2cc363863644a234e1e0fe10960e55aea0", size = 340245, upload-time = "2024-11-11T02:36:06.473Z" }, + { url = "https://files.pythonhosted.org/packages/8b/ab/dd84f0e2337296a5f09749f7b5483215d75c8fa9e33738522e5ed81f7254/tree_sitter_typescript-0.23.2-cp39-abi3-win_amd64.whl", hash = "sha256:3f730b66396bc3e11811e4465c41ee45d9e9edd6de355a58bbbc49fa770da8f9", size = 278015, upload-time = "2024-11-11T02:36:07.631Z" }, + { url = "https://files.pythonhosted.org/packages/9f/e4/81f9a935789233cf412a0ed5fe04c883841d2c8fb0b7e075958a35c65032/tree_sitter_typescript-0.23.2-cp39-abi3-win_arm64.whl", hash = "sha256:05db58f70b95ef0ea126db5560f3775692f609589ed6f8dd0af84b7f19f1cbb7", size = 274052, upload-time = "2024-11-11T02:36:09.514Z" }, +] + [[package]] name = "typing-extensions" version = "4.16.0"