diff --git a/ACTIVE_SLICE.md b/ACTIVE_SLICE.md index 22b7caa..4d6b119 100644 --- a/ACTIVE_SLICE.md +++ b/ACTIVE_SLICE.md @@ -1,15 +1,13 @@ # Milestone state ```text -Last completed milestone: 4 — adapter SDK and product documentation. -Evidence: Frozen executable candidate 95271dcf2e48045b9d3aed9b9ea09c7fc155692c; 347 tests and 402 subtests; 3 accessibility flows; offline fresh-wheel adoption; clean 1,002-node graph-plus-Logic benchmark; no secret-scan findings. +Last completed milestone: 3 — independent projections +Outcome: Manual output, portable graph artifacts, and the live viewer are independent generation-pinned consumers of the validated graph. +Evidence: Clean candidate f5dccb5e1c312121f1af63780162f593d9363b98; 281 tests and 272 subtests; 3 accessibility flows; clean 1,000-node ten-sample benchmark; isolated wheel proof; no secret-scan findings. Active milestone: None. -Outcome: New projects can adopt DocForge incrementally, and users can understand the CLI, MCP server, policies, adapters, renderers, and agent integration without reading implementation code. -Status: Milestone 4 is closed. Milestone 5 has not started. -Next directional milestone: 5 — stabilization and first DocForge2 release. +Next directional milestone: 4 — adapter SDK and product documentation. +Status: Milestone 3 is closed. Milestone 4 has not started. ``` -Milestone 5 remains directional context. Do not begin release stabilization, versioning, tagging, -or publication without a new active-slice contract based on the Milestone 4 evidence. - -The completed Milestone 4 contract and its exclusions are preserved in `SLICE_HISTORY.md`. +Milestones 4–5 remain directional context. Do not begin Milestone 4 without a new active-slice +contract. diff --git a/DEVELOPMENT_NOTES.md b/DEVELOPMENT_NOTES.md index 78f6be8..2548012 100644 --- a/DEVELOPMENT_NOTES.md +++ b/DEVELOPMENT_NOTES.md @@ -784,108 +784,3 @@ proof. Gitleaks 8.30.1 found no findings across the six Milestone 3 commits or c Milestone 3 is complete. No tag, release, production integration repointing, WorldForge change, ScrapeStation change, storage rewrite, or self-hosting dependency was introduced. Milestone 4 remains directional and has not started. - -## Milestone 4 — complete: adapter SDK and product documentation - -Milestone 4 activated from the verified Milestone 3 closeout. Its scope was deliberately additive: -freeze an adapter-authoring boundary, prove narrow language references, make heavy frontends -optional, generate command and client integration evidence, and document adoption. WorldForge, -ScrapeStation, the legacy repository, production bindings, self-hosting, storage replacement, and -release publication remained excluded. - -### Adapter SDK and conformance - -`docforge.adapter_sdk` is now the public authoring facade for typed projections, manifests, source -contributions, complete assemblies, incremental loaders, project bindings, core graph models, and -conformance reports. Complete evidence includes primary graph and function Logic. An incremental -adapter that publishes Logic must provide an independent `load_complete_assembly()` oracle. - -The conformance helper repeats complete loading for determinism, compares the complete assembly -with `load_projection()`, and proves exact complete/incremental graph-plus-Logic parity. It does not -pretend to replace separate confinement, restart, no-AST, corrupt-cache, or retrieval tests. - -An adversarial review found that extraction caches and adapter assemblies lacked aggregate -boundaries. Version-1 cache reads and writes are now regular-file-only, identity-checked, capped at -64,000,000 bytes and 10,000 sources, and fail safely to a miss. Adapter projects enforce primary -node limits and deterministic edge and Logic multipliers before publication. - -### Reference language evidence - -The Python reference uses the standard-library AST and publishes files, modules, classes, -functions, arguments, local static imports, and function Logic. Its manifest uses tokenization -rather than AST parsing, so unchanged warm builds perform no syntax parse. - -JavaScript and TypeScript use separate pinned optional Tree-sitter grammars. They publish files, -modules, classes, functions/methods, static project-relative imports/re-exports, and function -Logic. Their manifests use a closed comment/string-aware ESM scan and do not load Tree-sitter. - -C++ uses explicit non-overlapping source roots and one confined `compile_commands.json` as -translation-unit inventory and fingerprint evidence. It validates entries but never executes a -command, compiler, response file, or project program. It publishes syntax and directly resolvable -project-local quoted includes. Its manifest currently parses for include discovery, so no -zero-warm-parser claim is made. - -None of the references claim compiler-resolved calls, inheritance, types, symbol references, -compiler include semantics, macro semantics, runtime behavior, or semantic ownership. - -The first packaging proof exposed that language parsers were inherited as mandatory dependencies. -The base wheel now depends only on Markdown and MCP packages. Python works in the base wheel; -JavaScript, TypeScript, C++, and all-language extras install their respective Tree-sitter -frontends. Missing extras return a closed error with the exact install target. - -### Fixed reference MCP and launchers - -A strict `.docforge/reference-adapter.toml` selects one fixed reference language, project -identity, and explicit source roots. C++ additionally requires the compilation database. The fixed -`python -m docforge.reference_mcp` binding constructs only repository-owned providers and exposes -the 21 read tools. - -The initial launcher design generated `python -I -m` for arbitrary project module names without -proving that isolated Python could resolve them. The corrected `AdapterLauncherV1` accepts only an -installed project-confined top-level module or the exact trusted `docforge.reference_mcp` -exception. A fixed bounded `find_spec` probe runs under isolated Python without importing project -code. Tests then launch both a real project module and the real reference MCP process. - -Generated Codex, Claude, and OpenClaw fragments bind the exact launcher, descriptor, source -availability, effective and projection policy, interpreter, canonical arguments, empty -environment, artifact bytes, and hashes. There is no arbitrary command, argument list, working -directory, environment, discovery, or callable selector. - -### Generated reference and documentation gates - -CLI tables are derived from the real argparse subcommands. MCP tables are derived from a real -application-enabled server's `list_tools()` registrations, including capability surface, -arguments, descriptions, and input-schema hashes. The generated artifact contains 28 CLI rows and -36 MCP rows. - -An adversarial review found a target-change race between the generator's identity check and -replacement. Publication now locks cooperating generators, uses no-clobber publication for a -missing target, and uses Linux atomic exchange plus displaced-byte and file-identity verification -for an existing target. Raced data is restored or retained for recovery rather than discarded. - -`docs-check` validates generated drift, local files and anchors, one H1 per page, README-rooted -reachability, required Milestone 4 inventory, the generated notice, and every documented -reference-adapter TOML example against the packaged schema. - -### Candidate, adoption, and scale evidence - -Frozen executable candidate `95271dcf2e48045b9d3aed9b9ea09c7fc155692c` passed formatting, -Python and web lint, strict Pyright, compilation, 142 contract tests and 268 subtests, 347 total -tests and 402 subtests, three accessibility flows, lock and dependency checks, package builds, -offline wheel adoption, and every maintained smoke benchmark. - -The offline adoption proof installed the base wheel from the lock without network access. It -proved no Tree-sitter package or module was installed, built and checked a real Python reference -project, performed isolated MCP bootstrap/search/get-node over the exact 21 read tools, and proved -that base-only C++ fails with `optional_dependency_missing` and `docforge[cpp]` remediation. - -The clean 334-source benchmark produced 1,002 primary nodes, 1,001 primary edges, 334 Logic -projections, 2,338 Logic nodes, and 2,338 Logic edges. Complete and incremental output matched -exactly. Warm build p95 was 1,265.387 ms with zero `ast.parse` and zero `extract_source` calls. -Complete equivalence, corrupt cache recovery, and corrupt index recovery were all below 1.9 -seconds. The largest per-operation traced peak was 69,997,166 bytes and process high-water was -78,798,848 bytes. Exact hashes and method are recorded in -`benchmarks/milestone4-2026-07-29.json` and `docs/MILESTONE_4_BASELINE.md`. - -Milestone 4 is complete. No tag or release was created. Milestone 5 remains unstarted until its own -active-slice contract is frozen. diff --git a/Makefile b/Makefile index 203ae12..64d52c8 100644 --- a/Makefile +++ b/Makefile @@ -5,7 +5,7 @@ NPM := npm PYTHONPYCACHEPREFIX := /tmp/docforge-quality-pycache PYTEST_BASETEMP := /tmp/docforge-quality-pytest -.PHONY: accessibility adoption-m4 benchmark benchmark-m1 benchmark-m1-smoke benchmark-m2 benchmark-m2-smoke benchmark-m3 benchmark-m3-full benchmark-m3-smoke benchmark-m4 benchmark-m4-full benchmark-m4-smoke benchmark-smoke build command-reference-check compile contract dependencies docs-check format-check gate lint lock test type +.PHONY: accessibility benchmark benchmark-m1 benchmark-m1-smoke benchmark-m2 benchmark-m2-smoke benchmark-m3 benchmark-m3-full benchmark-m3-smoke benchmark-smoke build compile contract dependencies format-check gate lint lock test type accessibility: $(NPM) run test:accessibility @@ -59,17 +59,6 @@ dependencies: build: $(UV) build -adoption-m4: - $(PYTHON) tools/milestone4_adoption.py - -command-reference-check: - $(PYTHON) tools/generate_command_reference.py \ - --output docs/COMMAND_REFERENCE.md --check > /dev/null - -docs-check: command-reference-check - $(PYTHON) tools/check_documentation.py \ - --pending-inventory tools/docs_pending_m4_pages.txt - benchmark-smoke: $(PYTHON) tools/milestone0_baseline.py --nodes 25 --samples 1 --cold-samples 1 \ --output /tmp/docforge-milestone0-smoke.json > /dev/null @@ -100,13 +89,4 @@ benchmark-m3: benchmark-m3-full: benchmark-m3 -benchmark-m4-smoke: - $(PYTHON) tools/milestone4_benchmark.py --mode smoke \ - --output /tmp/docforge-milestone4-smoke.json > /dev/null - -benchmark-m4: - $(PYTHON) tools/milestone4_benchmark.py --mode full - -benchmark-m4-full: benchmark-m4 - -gate: format-check lint type compile contract test accessibility lock dependencies build docs-check benchmark-smoke benchmark-m1-smoke benchmark-m2-smoke benchmark-m3-smoke benchmark-m4-smoke +gate: format-check lint type compile contract test accessibility lock dependencies build benchmark-smoke benchmark-m1-smoke benchmark-m2-smoke benchmark-m3-smoke diff --git a/README.md b/README.md index 6e14013..20ba862 100644 --- a/README.md +++ b/README.md @@ -1,107 +1,88 @@ # DocForge -DocForge is a project-scoped documentation and source graph for people and AI agents. Canonical -project files remain authoritative; DocForge validates them, builds disposable search and -relationship indexes, compiles bounded context, manages reviewable changesets, renders declared -manuals and portable graph views, and serves one project-bound MCP surface. +DocForge is a project-scoped documentation graph for people and AI agents. It validates canonical +documentation, builds a disposable search and relationship index, compiles bounded context, renders +declared manuals, visualizes project structure, and manages reviewable documentation changesets. -DocForge never treats indexed text as instructions. It does not run project build commands, -compilers, Git operations, deployments, or arbitrary renderers, and it does not select projects -globally. +## What it does -DocForge 1.0.0 remains the first stable release in this product line. This checkout is the -post-1.0 development line (`1.3.0.dev0`) and contains the Milestone 4 adapter-platform work. -Milestone 4 is not a new product release, and Milestone 5 release and publication work remains -separate. +- Validates stable Markdown/TOML nodes and typed relationships. +- Builds a deterministic SQLite search and graph index. +- Exposes project-bound CLI and MCP query surfaces. +- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, and bounded + continuation. +- Records one bounded, versioned latest-generation graph transition without creating a history + database. +- Generates deterministic project-bound Codex, Claude, and OpenClaw client fragments without + copying ambient secrets. +- Diagnoses one client binding through bounded read-only checks without starting MCP or rebuilding + project state. +- Automatically synchronizes disposable indexes before MCP work. +- Creates, validates, diffs, and previews isolated changesets. +- Registers complete proposals atomically without caller-managed hash chaining. +- Applies one explicitly approved changeset hash through CLI or gated MCP. +- Supports opt-in incremental adapters with reverse-dependency invalidation and full-build + equivalence checks. +- Detects project-local adapter implementation and configuration changes and requires a fresh + project-bound process before any further MCP work. +- Keeps function-scoped control-flow projections separate from the primary architecture graph. +- Compiles manuals and portable graph artifacts from separate versioned, generation-pinned plans + and immutable packages. +- Runs built-in manual and graph renderers in fixed detached workers with validated receipts, + bounded transfer, and no project-path authority. +- Enforces independent manual, portable-graph, and live-viewer policy while keeping status + receipt-only. +- Runs a managed loopback graph browser with neighborhood, semantic Flow, convergence Web, + function-scoped Logic, source inspection, and branch-aware node hiding. +- Supports generic documentation projects and project-owned source adapters. -## Start here +DocForge never treats indexed text as instructions. It does not run shell commands, mutate Git, +build applications, deploy, publish, or select projects globally. -- New installation or first project: [New-project quickstart](docs/NEW_PROJECT_QUICKSTART.md) -- Mental model and authority: [Core concepts and authority](docs/CORE_CONCEPTS_AND_AUTHORITY.md) -- Complete configuration shape: [Project descriptor](docs/PROJECT_DESCRIPTOR.md) -- Fixed runnable examples: [Reference adapters](docs/REFERENCE_ADAPTERS.md) -- Agent and client setup: [Agent integration](docs/AGENT_INTEGRATION.md) -- Exact live command and tool inventory: [Generated command reference](docs/COMMAND_REFERENCE.md) -- Task-oriented operating guide: [User manual](docs/USER_MANUAL.md) +For implementation projects, the recommended cadence is to read canonical documentation during +intake, keep it read-only through implementation and focused testing, freeze and validate a release +candidate, then perform one atomic documentation closeout before the final commit and tag. This +keeps the manual authoritative without using it as an implementation notebook. -## Current capabilities +## Release 1 -- Validates Markdown and TOML nodes, stable IDs, typed relationships, project limits, and confined - paths. -- Builds a deterministic, disposable SQLite graph and search index with integrity and generation - evidence. -- Exposes project-bound CLI and MCP read, proposal, application, rendering, and visualization - surfaces according to the startup policy. -- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, bounded - output, and deterministic continuation. -- Creates isolated documentation changesets, validates complete projected graphs, and applies only - one explicitly approved changeset hash through a separately bound canonical applier. -- Supports complete-projection adapters and opt-in incremental adapters with reverse-dependency - invalidation, bounded extraction caches, and a clean full-build equivalence oracle. -- Publishes function-scoped Logic separately from the primary graph for Python, JavaScript, - TypeScript, and C++ integrations that provide it. -- Compiles manual and portable graph plans into immutable packages for fixed detached renderers, - then records bounded receipts. -- Runs a managed loopback graph viewer with Nodes, Flow, Web, lazy Logic, source inspection, and - branch-aware hiding. -- Generates deterministic Codex, Claude, and OpenClaw client fragments without copying ambient - environment values or secrets. +DocForge 1.0.0 is the first stable product release. It combines the project-scoped graph, CLI and +MCP query surfaces, reviewable hash-approved changesets, generic and project-owned adapters, +declared rendering, and the complete Nodes/Flow/Web visualization model in one supported release. -## Adapter platform +The post-1.0 incremental compiler is a backward-compatible, optional enhancement. Existing Release +1 adapters that implement only `load_projection()` continue to use the original complete-projection +path without modification. Adapters gain incremental performance only when they additionally +implement the source manifest and extraction methods. Incremental adapters must retain +`load_projection()` as their clean-rebuild fallback and equivalence oracle. -Adapter authors use the public `docforge.adapter_sdk` surface for graph types, complete and -incremental contracts, validation helpers, and `verify_adapter_conformance()`. Existing adapters -that implement only `load_projection()` remain supported. Incremental adapters add a manifest and -source extraction while retaining `load_projection()` as the complete graph oracle. An adapter -that publishes Logic incrementally also supplies `load_complete_assembly()` so the complete oracle -covers both graph and Logic. +MCP bindings can additionally select `--no-ast` when an owner wants to preserve an existing +non-AST adapter. The binding advertises that policy to clients, forbids adapter rewrites that add +AST, Tree-sitter, compiler-AST, or function-Logic extraction, blocks the Logic tool, and rejects +nonempty Logic publication. Complete-projection adapters continue unchanged, and non-AST +incremental fingerprinting and caching remain allowed. -The repository includes bounded reference integrations for: +DocForge2 bindings may also declare +`--capability-mode read|proposal|application|operator`. Bootstrap returns one versioned effective +policy and the actual startup-gated capabilities. Existing tool surfaces and the legacy no-AST +payload remain compatible. -- Python, using the standard-library AST and publishing only project-local imports; -- JavaScript and TypeScript, using their distinct optional Tree-sitter grammars and publishing only - project-local static relative imports and re-exports; -- C++, using `compile_commands.json` as translation-unit inventory and fingerprint evidence, without - executing its commands or a compiler, and publishing only directly resolvable project-local - quoted includes. +`docforge_get_task_context` is an additive read tool for `change`, `implementation`, `failure`, +`ownership`, `test`, `operation`, and `release` work. It derives a closed version-1 retrieval plan, +executes it against one immutable index generation, and returns a hash-bound context capsule. +Project relation names remain authoritative. DocForge classifies only its versioned alias set and +preserves every unknown relation as `unclassified` instead of guessing semantics. -These integrations demonstrate the adapter contract; they do not claim resolved calls, -inheritance, types, runtime behavior, macro expansion, compiler include semantics, or semantic -ownership. Python works from the base wheel. Install the `javascript`, `typescript`, or `cpp` -extra for the corresponding grammar, or `languages` for all three: +`docforge_get_generation_diff` reports the latest verified primary-graph transition through one +bounded disposable receipt. It includes exact node and edge change counts, hash-bound retained +details, and explicit truncation. Paged results use one top-level cursor and a versioned +`receipt_header`; `stored_receipt_hash` identifies the complete persisted receipt. The read never +exposes Logic details, loads canonical source, repairs derived state, or invents history. -```bash -uv pip install "/absolute/path/DocForge[javascript]" -uv pip install "/absolute/path/DocForge[typescript]" -uv pip install "/absolute/path/DocForge[cpp]" -uv pip install "/absolute/path/DocForge[languages]" -``` +## Independent projections -Reference projects use the fixed `.docforge/reference-adapter.toml` descriptor and the installed -`docforge.reference_mcp` module. That server is read-only and exposes no proposal or application -surface. - -Project-owned adapters use `AdapterLauncherV1` and -`generate_adapter_client_configuration()` to produce a client fragment from an explicitly -constructed adapter project. The launcher is immutable and contains no command, caller arguments, -working directory, environment, discovery rule, or callable selector. It resolves through isolated -Python to one installed, project-owned top-level module; the only trusted dotted exception is the -fixed `docforge.reference_mcp` binding. Generic `docforge configure` intentionally refuses custom -adapters because it cannot safely reconstruct project-owned composition. - -See the [Adapter authoring guide](docs/ADAPTER_AUTHORING_GUIDE.md), [Reference -adapters](docs/REFERENCE_ADAPTERS.md), and [Agent integration](docs/AGENT_INTEGRATION.md) for the -supported routes. - -## Authority and projections - -Canonical Markdown, TOML, adapter-declared sources, and descriptor files own project facts. The -SQLite index, extraction cache, render packages, previews, portable artifacts, receipts, client -fragments, and viewer processes are derived and replaceable. - -The primary graph contains project nodes and relationships. Logic is a separate, lazy, -function-scoped control-flow projection. Manual output, portable graph output, and the live viewer -are independent consumers of one validated generation: +Manual compilation, portable graph rendering, and the live viewer consume the same validated graph +generation through separate boundaries: ```text validated generation @@ -110,142 +91,190 @@ validated generation └── pinned index → managed read-only live viewer ``` -The version-2 projection policy selects each consumer independently: +Plans, packages, and receipts are canonical, versioned, hash-identified, bounded, and contain no +project object, SQLite handle, absolute project path, command, or caller-selected renderer module. +Renderers cannot select graph facts, crawl canonical sources, choose publication paths, or mutate +project state. -```text -manual: auto | explicit | disabled -portable_graph: explicit | disabled -live_viewer: on-demand | disabled +Declare portable graph output separately from manual views: + +```toml +[graph_render] +output_root = ".docforge/portable-graph" + +[[graph_render.views]] +id = "architecture" +renderer = "portable_graph_html" +output = "architecture.html" +title = "Architecture" +root = "architecture.overview" +initial_mode = "web" +depth = 3 +max_nodes = 250 +max_edges = 1000 +max_work = 100000 +include_logic = false ``` -The process capability policy is separate. Capability mode controls the registered read, -proposal, application, or reserved operator surface; descriptor writers and the startup-bound -canonical applier determine whether mutations are actually authorized. `--no-ast` is a restrictive -binding policy over adapter evolution, Logic publication, and Logic retrieval. It is not a parser -inspection mechanism or a filesystem sandbox, and the AST/Tree-sitter reference integrations -should not be presented as no-AST adapters. - -Read [Policy precedence](docs/POLICY_PRECEDENCE.md), [Rendering and -visualization](docs/RENDERING_AND_VISUALIZATION.md), and [Legacy and no-AST -operation](docs/LEGACY_AND_NO_AST.md) before changing those boundaries. - -## Five-minute generic project - -Requirements are Python 3.12 or newer and an installed DocForge environment. From a development -checkout, `uv sync --group dev` creates `.venv`: +Plan, publish, and inspect it explicitly: ```bash -git clone /absolute/path/DocForge -cd /absolute/path/DocForge +.venv/bin/docforge --project-root "$PROJECT" graph-plan architecture +.venv/bin/docforge --project-root "$PROJECT" graph-render architecture +.venv/bin/docforge --project-root "$PROJECT" graph-render-status architecture +``` + +The version-2 projection policy independently selects manual +`auto|explicit|disabled`, portable graph `explicit|disabled`, and live viewer +`on-demand|disabled`. Use `--manual-render-policy`, `--portable-graph-policy`, and +`--live-viewer-policy` on CLI/MCP startup or generated client configuration. Status remains +available when the corresponding active operation is disabled. + +Manual fragment reuse is disposable. Cold record publication is guarded by byte-exact comparison +with a full detached render; warm records are independently recomputed and validated inside the +worker. Full rendering remains the recovery and equivalence oracle. Portable artifacts commit +content-addressed output and renderer evidence before one bounded generation/view manifest; status +does not plan or render. + +## Graph views + +The browser presents the primary architecture graph through three complementary views and loads a +fourth function-scoped view only when requested: + +- **Nodes** shows a bounded, relation-neutral neighborhood around the focus. It is the broad + inspection view for seeing stored incoming and outgoing relationships without changing their + direction. Semantic cards distinguish structure, behavior, dependencies, execution, data, + evidence, context, and other relationships. +- **Flow** shows semantic origin-to-destination paths that terminate at the focus. DocForge + reverses prerequisite-style relationships for presentation, so imports, dependencies, reads, + inheritance, definitions, and tests flow toward the thing they help create or exercise. +- **Web** shows the larger convergence picture: Flow contributors plus contextual relationships, + callers, containers, and direct members or execution dependencies owned by the focus. +- **Logic** shows the possible static control paths inside a focused Python, JavaScript, or C++ + function or method. Entry, decisions, actions, loops, convergence points, returns, and + exceptions connect through explicit + `TRUE`, `FALSE`, `NEXT`, `CASE`, `LOOP`, `RETURN`, and `RAISE` paths. Logic is stored separately + and does not add statement-level noise to Nodes, Flow, Web, or search. + +Graph cards show the node's readable leaf name and kind without clipping either value. The full +qualified identity remains available in the tooltip, compact descriptor, and full inspector. +The left browser panel can combine text, family, node-kind, language, and capability filters. +Quick presets expose Logic-ready nodes, Python callables, tests, routes, and documentation without +requiring users to know stable IDs. Selecting a canvas node emphasizes its directly connected +neighbors and edges while muting unrelated paths. + +**Hide node** removes noise without changing the index. In Flow and Web, hiding a contributor also +removes upstream ancestors that no longer have a path to the focus. Nodes between the hidden +contributor and the focus stay visible, and alternate ancestor paths remain intact. In Logic, +hiding a step inserts an explicit omitted-path bridge so downstream control flow remains readable. +**Restore hidden** restores the presentation. + +## Five-minute start + +Requirements are Python 3.12+, `uv`, and Node.js/npm. + +```bash +git clone forgejo@repo.andraxion.net:administrator/DocForge2.git /absolute/path/DocForge2 +cd /absolute/path/DocForge2 uv sync --group dev +npm ci PROJECT=/absolute/path/MyProject -.venv/bin/docforge --project-root "$PROJECT" onboard -.venv/bin/docforge --project-root "$PROJECT" onboard \ - --scaffold \ - --project-id my-project \ - --title "My Project" -``` - -The first command is read-only. Scaffolding is explicit and create-only: it writes a generic -descriptor, one canonical overview node, and a built-in manual template, then indexes and renders -them. Detected source languages remain `adapter_required` until a validated frontend is selected. - -Operate the configured project: - -```bash .venv/bin/docforge --project-root "$PROJECT" validate .venv/bin/docforge --project-root "$PROJECT" reindex -.venv/bin/docforge --project-root "$PROJECT" search architecture .venv/bin/docforge --project-root "$PROJECT" visualize ``` -Install the persistent per-user graph viewer manager once when using the live viewer: +Install the persistent per-user graph viewer once: ```bash .venv/bin/docforge-viewer-manager install-user-service ``` -Start one generic, read-only MCP server: +Start an MCP server for one project: ```bash .venv/bin/docforge-mcp \ --project-root "$PROJECT" \ - --capability-mode read + --proposal-writer project-editor ``` -Add a descriptor-authorized proposal writer only when the client should create proposals. Add a -matching `--canonical-applier` only when that integration should expose exact-hash application. -See the [quickstart](docs/NEW_PROJECT_QUICKSTART.md) for the reference-adapter and generated-client -paths. +Add `--canonical-applier project-editor` only when that MCP integration should expose the +hash-bound `docforge_apply_changeset` tool. + +Preview a read-only Codex fragment and diagnose an installed binding: + +```bash +.venv/bin/docforge configure codex --project "$PROJECT" +.venv/bin/docforge doctor --client codex --project "$PROJECT" +``` + +Pass `--output /absolute/path/docforge.toml` only when creating a standalone fragment. DocForge +never replaces or merges an existing different client file. + +For an unconfigured codebase, begin with a read-only language and documentation assessment: + +```bash +.venv/bin/docforge --project-root /absolute/path/MyProject onboard +``` + +Add `--scaffold`, a stable project ID, and a title to create, index, and render a generic starter +manual. Source files are reported separately and require a validated language frontend before +DocForge describes them as a source graph. ## Documentation -### Learn and operate - -- [New-project quickstart](docs/NEW_PROJECT_QUICKSTART.md) -- [Core concepts and authority](docs/CORE_CONCEPTS_AND_AUTHORITY.md) -- [Project descriptor](docs/PROJECT_DESCRIPTOR.md) -- [Policy precedence](docs/POLICY_PRECEDENCE.md) -- [Reference adapters](docs/REFERENCE_ADAPTERS.md) -- [Agent integration](docs/AGENT_INTEGRATION.md) -- [User manual](docs/USER_MANUAL.md) -- [Generated command reference](docs/COMMAND_REFERENCE.md) -- [Project onboarding](docs/PROJECT_ONBOARDING.md) -- [Rendering and visualization](docs/RENDERING_AND_VISUALIZATION.md) -- [Recovery and performance](docs/RECOVERY_AND_PERFORMANCE.md) -- [Security](docs/SECURITY.md) - -### Contracts and compatibility - -- [Core contract](docs/CONTRACT.md) -- [MCP contract](docs/MCP_CONTRACT.md) -- [Compatibility contract](docs/COMPATIBILITY.md) -- [Adapter authoring guide](docs/ADAPTER_AUTHORING_GUIDE.md) -- [Incremental indexing](docs/INCREMENTAL_INDEXING.md) -- [Adapter application decision](docs/APPLICATION_DECISION.md) -- [Legacy and no-AST operation](docs/LEGACY_AND_NO_AST.md) -- [Migrating from version 1](docs/MIGRATING_FROM_V1.md) -- [Viewer manager](docs/VIEWER_MANAGER.md) - -### Milestone evidence - -- [Milestone 4 baseline](docs/MILESTONE_4_BASELINE.md) -- [Milestone 4 closeout](docs/MILESTONE_4_CLOSEOUT.md) -- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) and [closeout](docs/MILESTONE_3_CLOSEOUT.md) -- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) and [closeout](docs/MILESTONE_2_CLOSEOUT.md) -- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) and [closeout](docs/MILESTONE_1_CLOSEOUT.md) -- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) and [closeout](docs/MILESTONE_0_CLOSEOUT.md) - -Historical milestone records preserve the facts and dependency observations of their frozen -candidates. Use the current guides and contracts for present behavior. +- [User manual](docs/USER_MANUAL.md) — features, setup, visualization, CLI, MCP, apply, adapters, + and troubleshooting. +- [Core contract](docs/CONTRACT.md) — invariants and security boundary. +- [Milestone 0 compatibility](docs/COMPATIBILITY.md) — preserved package, CLI, MCP, adapter, + schema, changeset, rendering, and no-AST guarantees. +- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) — validation evidence, cold and warm + performance, memory, rendering and response sizes, bottlenecks, and missing coverage. +- [Milestone 0 closeout](docs/MILESTONE_0_CLOSEOUT.md) — lineage, migration, security scan, + repository state, and fresh-clone proof. +- [Milestone 1 baseline](docs/MILESTONE_1_BASELINE.md) — warm operation latency, structured work, + status, retrieval, and memory measurements. +- [Milestone 1 closeout](docs/MILESTONE_1_CLOSEOUT.md) — fast-core contracts, adversarial + validation, compatibility boundaries, and exact candidate evidence. +- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) — task context, generation diff, client + configuration, doctor, response-size, counter, and memory measurements. +- [Milestone 2 closeout](docs/MILESTONE_2_CLOSEOUT.md) — implemented contracts, adversarial + validation, exclusions, and exact candidate evidence. +- [Milestone 3 baseline](docs/MILESTONE_3_BASELINE.md) — manual, fragment, worker, portable graph, + status, equivalence, response-size, and memory measurements. +- [Milestone 3 closeout](docs/MILESTONE_3_CLOSEOUT.md) — independent projection contracts, + adversarial validation, compatibility boundaries, and exact candidate evidence. +- [MCP contract](docs/MCP_CONTRACT.md) — exact tool and process boundary. +- [Viewer manager](docs/VIEWER_MANAGER.md) — native service setup and lifecycle. +- [Adapter decision](docs/APPLICATION_DECISION.md) — why custom adapters own canonical + serialization. +- [Incremental adapter indexing](docs/INCREMENTAL_INDEXING.md) — source-scoped extraction, + invalidation, equivalence, relationship changes, and the lazy Logic boundary. +- [Project onboarding](docs/PROJECT_ONBOARDING.md) — repository assessment, safe manual + scaffolding, language frontends, source/manual integration, proof, and MCP activation. +- [Language adapter authoring](docs/ADAPTER_AUTHORING_GUIDE.md) — implementation sequence, + stable identities, overlap ownership, normalization, incremental equivalence, troubleshooting, + and the complete adapter proof matrix. ## Development -Run the repository-native gate: +Run the complete repository-native gate: ```bash make gate ``` -Milestone 4 maintenance entry points include: +Focused entry points are available as `make contract`, `make test`, `make type`, +`make benchmark-smoke`, `make benchmark`, `make benchmark-m1-smoke`, and +`make benchmark-m1`. Milestone 2 adds `make benchmark-m2-smoke` and `make benchmark-m2`. +Milestone 3 adds `make accessibility`, `make benchmark-m3-smoke`, `make benchmark-m3`, and +`make benchmark-m3-full`. -```bash -make adoption-m4 -make benchmark-m4-smoke -make benchmark-m4 -make benchmark-m4-full -make command-reference-check -make docs-check -``` +The committed 1,000-node baseline and its measurement method are under `benchmarks/`. -`adoption-m4` builds and exercises a fresh base wheel without Tree-sitter packages and proves the -Python reference plus a real isolated read-only MCP retrieval. The optional-language integrations -have their own focused tests and extras. `benchmark-m4` runs the maintained full adapter workload; -the smoke target is for routine gate coverage, not final performance evidence. +Pass `--diagnostics` to `docforge` or `docforge-mcp` to attach bounded request-local stage timings +and compiler-work counters. Diagnostics are disabled by default and are dropped before primary MCP +results when the configured output budget is tight. -Pass `--diagnostics` to `docforge` or `docforge-mcp` for bounded request-local timings and compiler -work counters. Diagnostics are disabled by default and do not displace a primary result when the -configured output budget is tight. - -Read [AGENTS.md](AGENTS.md) before changing core boundaries. +See [AGENTS.md](AGENTS.md) before changing core boundaries. diff --git a/SLICE_HISTORY.md b/SLICE_HISTORY.md index 0caab02..5b4cd75 100644 --- a/SLICE_HISTORY.md +++ b/SLICE_HISTORY.md @@ -1,50 +1,5 @@ # Completed slices -## DocForge2 Milestone 4 adapter SDK and product documentation - -### Changed - -- Added stable `docforge.adapter_sdk` authoring imports and a conformance helper for deterministic - complete and exact complete/incremental primary graph plus Logic equivalence. -- Added bounded Python, JavaScript, TypeScript, and C++ reference integrations with explicit - unsupported-fact reports and optional heavy frontends. -- Added a closed `.docforge/reference-adapter.toml` contract and fixed read-only reference MCP - server. -- Added immutable project-bound launcher declarations and deterministic Codex, Claude, and OpenClaw - configuration generation for custom adapters. -- Generated CLI and MCP references from live implementation metadata and added race-safe - publication plus repository drift checks. -- Added strict documentation-graph and fresh-wheel adoption gates and dedicated onboarding, - authority, descriptor, policy, adapter, legacy/no-AST, rendering, agent-integration, security, - recovery/performance, and v1 migration guides. - -### Verification - -- The executable candidate passed 347 tests and 402 subtests, 142 contract tests and 268 subtests, - three accessibility flows, lint, strict types, compilation, dependencies, builds, adoption, and - every maintained smoke benchmark. -- The offline base wheel contained no Tree-sitter distribution and ran a real Python reference - build/check and isolated 21-tool MCP retrieval flow. Missing C++ support failed with the exact - `docforge[cpp]` remediation. -- The clean 334-source benchmark produced 1,002 nodes and exact full/incremental graph and Logic - evidence, warm zero Python parsing/extraction, and exact corrupt-cache and corrupt-index - recovery below every latency, memory, and response gate. -- Gitleaks found no findings in the Milestone 4 commit range or candidate tree. - -### Limits - -- The reference integrations publish syntax and local static relationships. They do not claim - compiler-resolved calls, types, inheritance, macros, runtime behavior, or semantic ownership. -- C++ compilation-database commands are inert and never executed. -- C++ manifest include discovery uses Tree-sitter and makes no zero-warm-parser claim. -- No WorldForge, ScrapeStation, legacy repository, production binding, language distribution, - self-hosting, tag, or release was changed. - -### Next gate - -Milestone 5 remains directional. Create a new active-slice contract before stabilization, -versioning, tagging, or publication. - ## DocForge2 Milestone 3 independent projections ### Changed diff --git a/benchmarks/README.md b/benchmarks/README.md index acfedd2..2141f1a 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -42,14 +42,6 @@ make benchmark-m3 make benchmark-m3-full ``` -Run the Milestone 4 adapter SDK smoke, maintained full, and fresh-wheel adoption gates: - -```bash -make benchmark-m4-smoke -make benchmark-m4-full -make adoption-m4 -``` - The benchmark creates canonical sources, derived state, changesets, rendered output, and caches only in a disposable temporary directory. It does not read another project, self-host DocForge, or mutate repository content. @@ -79,13 +71,6 @@ sweeps, add/change/delete/reorder equivalence, portable publication, receipt-onl memory, detached worker peak memory, and response size. Its interpretation is in [`docs/MILESTONE_3_BASELINE.md`](../docs/MILESTONE_3_BASELINE.md). -`milestone4-2026-07-29.json` is the clean-tree adapter SDK and reference-Python baseline captured -from commit `95271dcf2e48045b9d3aed9b9ea09c7fc155692c`. It measures cold and warm -incremental builds, complete/incremental graph-plus-Logic equivalence, corrupt extraction-cache -recovery, corrupt-index recovery, parser and extraction counters, traced and process memory, and -bounded response size. Its interpretation is in -[`docs/MILESTONE_4_BASELINE.md`](../docs/MILESTONE_4_BASELINE.md). - The generic fixtures expose whole-source and projection scaling. They do not replace incremental adapter equivalence tests. Milestone 3's portable fixture contains 1,000 Nodes/Flow/Web nodes and 999 edges; portable version 1 deliberately excludes Logic. @@ -104,5 +89,4 @@ backlinks, both traversal directions, paged context, render receipt states, and freshness. The reported p95 uses the nearest-rank method; with ten samples it is the maximum. `process_peak_rss_kib` is the cumulative main-process `RUSAGE_SELF` high-water mark and excludes the detached viewer worker. It is diagnostic and not operation-local. Milestone 3 memory gates use -per-operation `tracemalloc` peaks and detached worker receipt peaks instead. Milestone 4 uses -per-operation `tracemalloc` peaks and a cumulative benchmark-process high-water gate. +per-operation `tracemalloc` peaks and detached worker receipt peaks instead. diff --git a/benchmarks/milestone4-2026-07-29.json b/benchmarks/milestone4-2026-07-29.json deleted file mode 100644 index 2f993d3..0000000 --- a/benchmarks/milestone4-2026-07-29.json +++ /dev/null @@ -1,205 +0,0 @@ -{ - "benchmark": "docforge2_milestone4", - "environment": { - "implementation": "CPython", - "machine": "x86_64", - "platform": "Linux-7.1.3-200.nobara.fc44.x86_64-x86_64-with-glibc2.43", - "python": "3.14.6" - }, - "evidence": { - "cache_recovery_exact": true, - "complete_assembly_hash": "2888182ab765fbffe3ba873c1613345640e8e6d89be74ddfca7452c0a5056345", - "edge_count": 1001, - "edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35", - "full_incremental_graph_and_logic_exact": true, - "index_recovery_exact": true, - "logic_edge_count": 2338, - "logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c", - "logic_node_count": 2338, - "logic_projection_count": 334, - "node_count": 1002, - "node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f", - "source_count": 334, - "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", - "warm_zero_ast_parse": true, - "warm_zero_source_extraction": true - }, - "evidence_sha256": "4a8db461a311b5df4abd5aa00063e9a347d8b9dba19b30e35684f561d5271549", - "fixture": { - "expected_node_count": 1002, - "kind": "synthetic_python_reference_adapter", - "nodes_per_source": 3, - "source_count": 334 - }, - "memory": { - "process_peak_bytes": 78798848, - "process_peak_limit_bytes": 536870912 - }, - "method": { - "ast_instrumentation": "temporary counter around stdlib ast.parse", - "clock": "time.perf_counter_ns", - "extraction_instrumentation": "reference adapter extract_source entry counter", - "full_mode_source_requirement": 334, - "in_process_peak_memory": "tracemalloc per measured invocation", - "process_peak_memory": "resource.getrusage(RUSAGE_SELF).ru_maxrss", - "response_size": "UTF-8 bytes of compact sorted JSON", - "samples": 3, - "threshold_basis": "Regression tripwires leave several times the Milestone 3 1,000-node allowances for AST, Logic, extraction-cache, and SQLite work." - }, - "mode": "full", - "operations": { - "cold_incremental_build": { - "max_ms": 1599.76, - "maximum_response_bytes": 14580, - "maximum_traced_peak_bytes": 68573540, - "median_ms": 1599.76, - "min_ms": 1599.76, - "p95_limit_ms": 30000.0, - "p95_ms": 1599.76, - "response_limit_bytes": 262144, - "samples": 1, - "stable_result": { - "ast_parse_calls": 668, - "cache_hits": 0, - "deleted_sources": 0, - "edge_count": 1001, - "edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35", - "extraction_calls": 334, - "invalidated_sources": 334, - "logic_edge_count": 2338, - "logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c", - "logic_node_count": 2338, - "logic_projection_count": 334, - "node_count": 1002, - "node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f", - "reparsed_sources": 334, - "revision": "30ef23aa4106", - "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", - "status": "ok", - "total_sources": 334 - }, - "traced_peak_limit_bytes": 268435456 - }, - "corrupt_extraction_cache_recovery": { - "max_ms": 1695.406, - "maximum_response_bytes": 14580, - "maximum_traced_peak_bytes": 69776923, - "median_ms": 1695.406, - "min_ms": 1695.406, - "p95_limit_ms": 30000.0, - "p95_ms": 1695.406, - "response_limit_bytes": 262144, - "samples": 1, - "stable_result": { - "ast_parse_calls": 668, - "cache_hits": 0, - "deleted_sources": 0, - "edge_count": 1001, - "edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35", - "extraction_calls": 334, - "invalidated_sources": 334, - "logic_edge_count": 2338, - "logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c", - "logic_node_count": 2338, - "logic_projection_count": 334, - "node_count": 1002, - "node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f", - "reparsed_sources": 334, - "revision": "30ef23aa4106", - "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", - "status": "ok", - "total_sources": 334 - }, - "traced_peak_limit_bytes": 268435456 - }, - "corrupt_index_recovery": { - "max_ms": 1707.879, - "maximum_response_bytes": 14779, - "maximum_traced_peak_bytes": 68561924, - "median_ms": 1707.879, - "min_ms": 1707.879, - "p95_limit_ms": 30000.0, - "p95_ms": 1707.879, - "response_limit_bytes": 262144, - "samples": 1, - "stable_result": { - "action": "rebuilt", - "ast_parse_calls": 0, - "cache_hits": 334, - "edge_count": 1001, - "edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35", - "extraction_calls": 0, - "initial_error_code": "invalid_index", - "logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c", - "logic_projection_count": 334, - "node_count": 1002, - "node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f", - "reparsed_sources": 0, - "revision": "30ef23aa4106", - "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", - "status": "ok" - }, - "traced_peak_limit_bytes": 268435456 - }, - "full_incremental_equivalence": { - "max_ms": 1897.919, - "maximum_response_bytes": 230, - "maximum_traced_peak_bytes": 64305919, - "median_ms": 1897.919, - "min_ms": 1897.919, - "p95_limit_ms": 30000.0, - "p95_ms": 1897.919, - "response_limit_bytes": 262144, - "samples": 1, - "stable_result": { - "ast_parse_calls": 1336, - "edge_count": 1001, - "extraction_calls": 668, - "logic_projection_count": 334, - "node_count": 1002, - "project_id": "milestone4-python-benchmark", - "revision": "30ef23aa4106", - "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", - "status": "ok" - }, - "traced_peak_limit_bytes": 268435456 - }, - "warm_incremental_build": { - "max_ms": 1265.387, - "maximum_response_bytes": 14578, - "maximum_traced_peak_bytes": 69997166, - "median_ms": 1244.427, - "min_ms": 1239.941, - "p95_limit_ms": 20000.0, - "p95_ms": 1265.387, - "response_limit_bytes": 262144, - "samples": 3, - "stable_result": { - "ast_parse_calls": 0, - "cache_hits": 334, - "deleted_sources": 0, - "edge_count": 1001, - "edge_hash": "00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35", - "extraction_calls": 0, - "invalidated_sources": 0, - "logic_edge_count": 2338, - "logic_hash": "c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c", - "logic_node_count": 2338, - "logic_projection_count": 334, - "node_count": 1002, - "node_hash": "f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f", - "reparsed_sources": 0, - "revision": "30ef23aa4106", - "source_hash": "30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2", - "status": "ok", - "total_sources": 334 - }, - "traced_peak_limit_bytes": 268435456 - } - }, - "schema_version": 1, - "source": { - "dirty": false, - "revision": "95271dcf2e48045b9d3aed9b9ea09c7fc155692c" - } -} diff --git a/docs/ADAPTER_AUTHORING_GUIDE.md b/docs/ADAPTER_AUTHORING_GUIDE.md index df37df7..f77c3fe 100644 --- a/docs/ADAPTER_AUTHORING_GUIDE.md +++ b/docs/ADAPTER_AUTHORING_GUIDE.md @@ -8,30 +8,6 @@ Use this guide after the repository assessment in [Project Onboarding](PROJECT_ONBOARDING.md). The onboarding checklist decides whether an adapter is needed. This guide defines how to build and prove one. -## Public authoring surface - -Adapter authors should import typed contracts, validation helpers, graph models, and the -conformance helper from `docforge.adapter_sdk`: - -```python -from docforge.adapter_sdk import ( - AdapterAssembly, - AdapterImplementation, - AdapterManifest, - AdapterProject, - AdapterProjectSettings, - AdapterProjection, - AdapterSource, - AdapterSourceProjection, - LogicProjection, - verify_adapter_conformance, -) -``` - -Language-specific modules under `docforge.adapters` are repository reference implementations, not -the general authoring namespace. Their exact, deliberately narrow behavior is documented in -[Reference Adapters](REFERENCE_ADAPTERS.md). - ## Required outcome A production adapter must provide one reproducible public graph from authoritative project @@ -75,39 +51,11 @@ class MyAdapter: manifest: AdapterManifest, contributions: tuple[AdapterSourceProjection, ...], ) -> AdapterAssembly: ... - def load_complete_assembly(self) -> AdapterAssembly: ... def load_projection(self) -> AdapterProjection: ... ``` `assemble_projection()` is optional only when extraction units already have disjoint ownership. -`load_projection()` is always required. It is the clean primary-graph compatibility oracle. -`load_complete_assembly()` supplies the cache-independent complete graph-plus-Logic oracle. It is -required when incremental contributions publish Logic; without it, DocForge cannot prove Logic -parity. - -## What the conformance helper proves - -Run the public helper against a fresh confined cache root: - -```python -report = verify_adapter_conformance( - MyAdapter(project_root), - cache_root=project_root / ".docforge" / "cache" / "adapter-conformance", -) -``` - -`verify_adapter_conformance()` proves: - -- two repeated complete assemblies are exactly deterministic, including Logic; -- when `load_complete_assembly()` exists, its primary graph exactly matches - `load_projection()`; and -- when the incremental methods exist, the assembled incremental graph and Logic exactly match the - independent complete oracle. - -The report records stable identity, counts, and an assembly hash. This helper does not by itself -prove process confinement, implementation restart behavior, corrupt-cache recovery, warm zero -parsing, retrieval, or every case in the proof matrix below. Keep those as separate focused and -integration tests. +`load_projection()` is always required. It is the clean rebuild and equivalence oracle. ## Step 1: define authority before parsing @@ -376,25 +324,6 @@ The manifest remains a current source snapshot, not a Git-index snapshot. A Git- omit a deleted source whether its deletion is unstaged or staged. Staging is never a required DocForge synchronization step. -### Current aggregate bounds - -DocForge limits one extraction-cache generation to 10,000 source contributions and 64,000,000 -encoded bytes. Malformed, incompatible, missing, oversized, or unsafe cache data is treated as a -cache miss. - -The assembled graph is bounded by the effective project `Limits.max_nodes`. When adapter settings -do not override limits, DocForge selects at least 10,000 nodes and raises that bound to the -manifest's `estimated_nodes` when larger. Aggregate assembly ceilings are: - -- nodes: `max_nodes`; -- relationships: `max_nodes * 32`; -- Logic nodes across all functions: `max_nodes * 32`; and -- Logic edges across all functions: `max_nodes * 64`. - -Adapters should set realistic limits and fail before retaining unbounded frontend evidence. The -cache and assembly bounds do not replace each adapter's own bounded source, command, parser, or -per-function limits. - ## Step 9: keep the complete path independent The full rebuild must not read the incremental extraction cache. Otherwise equivalence compares @@ -541,8 +470,7 @@ For every such change: - [ ] Two independent complete builds match exactly. - [ ] Incremental extraction uses authoritative dependencies. - [ ] The complete oracle is independent of the cache. -- [ ] Complete and incremental public graph and Logic projections match exactly. -- [ ] `verify_adapter_conformance()` passes, and the separate proof-matrix cases also pass. +- [ ] Complete and incremental public projections match exactly. - [ ] Corrupt, missing, and interrupted cache cases fail safely. - [ ] Session composition and family isolation are proven. - [ ] Viewer, query, context, and Logic retrieval are proven. diff --git a/docs/AGENT_INTEGRATION.md b/docs/AGENT_INTEGRATION.md deleted file mode 100644 index 0f5ae3b..0000000 --- a/docs/AGENT_INTEGRATION.md +++ /dev/null @@ -1,105 +0,0 @@ -# Agent integration - -DocForge generates deterministic, project-bound MCP client fragments for Codex, Claude, and -OpenClaw. Generic projects and custom adapters use different configuration APIs, but both produce -fixed standard-input/output bindings with empty generated environments and explicit capability -policy. - -## Fixed reference binding - -Configure one of the in-repository adapters with -`.docforge/reference-adapter.toml` as described in -[Reference Adapters](REFERENCE_ADAPTERS.md), then start: - -```bash -python -I -m docforge.reference_mcp \ - --project-root /absolute/path/to/project \ - --capability-mode read -``` - -`docforge.reference_mcp` is a fixed trusted module in the installed DocForge distribution. It -selects a provider only from the validated reference configuration and reports binding metadata -containing: - -- `server_module = "docforge.reference_mcp"`; -- `adapter_mode = "reference"`; -- the selected `reference_language`; and -- the exact `reference_config_hash`. - -The reference binding registers exactly the 21 read tools listed in -[MCP Boundary](MCP_CONTRACT.md#read-tools). It has no proposal or canonical-application surface. - -## Generate a reference client fragment - -The public launcher and generator APIs are: - -```python -from pathlib import Path - -from docforge.adapter_launcher import AdapterLauncherV1 -from docforge.client_config import generate_adapter_client_configuration -from docforge.reference_mcp import REFERENCE_MCP_MODULE, create_reference_project - -root = Path("/absolute/path/to/project") -project = create_reference_project(root) -launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE) - -plan = generate_adapter_client_configuration( - project, - launcher, - "codex", # "codex", "claude", or "openclaw" - capability_mode="read", -) -print(plan["artifact"]["content"]) -``` - -Pass `output=Path(...)` only when the caller has selected an exact destination. Publication is an -atomic create-or-exact-match operation; it does not merge or replace different existing content. -The result includes the launcher, source-availability, project, policy, artifact, and configuration -hashes needed to inspect the binding before use. - -The fixed reference module accepts read mode only. Do not request proposal or application mode for -it. - -## Custom project-owned adapter launchers - -Generic `docforge configure` intentionally refuses a custom adapter project. Construct the -project-owned `ProjectService`, then use `AdapterLauncherV1` and -`generate_adapter_client_configuration()` as the custom-adapter route. - -`AdapterLauncherV1` schema version 1 binds: - -- one project ID, canonical absolute project root, adapter identity, and descriptor hash; -- `entry_point = "python-module"`; and -- one installed project-owned top-level Python module. - -The only trusted dotted-module exception is the fixed `docforge.reference_mcp` binding. A -project-owned module is resolved through isolated Python without importing or executing it during -the probe, and it must resolve to one canonical regular `.py` file inside the project root. - -The launcher contract has no arbitrary command, command arguments, shell string, working -directory, environment, callable selector, discovery rule, or module-reload mechanism. Generated -bindings invoke the current Python executable as `python -I -m ` with only DocForge's -validated project, capability, render-policy, authority, and no-AST options. Source identity and -the project binding are revalidated before publication; drift fails closed. - -Proposal and application modes are available only to a custom module that implements those fixed -server arguments and only when the project descriptor declares the matching writer and canonical -applier authority. Generating a mode does not manufacture that authority. - -## Session workflow - -After registering the generated fragment in the selected client: - -1. Start a new MCP process or client session. -2. Call `docforge_bootstrap`. -3. Verify project ID, root fingerprint, adapter identity, revision, source hash, binding metadata, - effective policy, and registered tools. -4. Retrieve exact or bounded project evidence through the fixed read tools. -5. Restart the process if `adapter_restart_required` reports an implementation or configuration - change. - -Document text returned by DocForge is untrusted project content. It never overrides client, user, -or project authority. See [MCP Boundary](MCP_CONTRACT.md) for synchronization, pagination, -retrieval, proposal, and application rules, and -[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) before selecting `no_ast=True`. diff --git a/docs/COMMAND_REFERENCE.md b/docs/COMMAND_REFERENCE.md deleted file mode 100644 index fbd01ff..0000000 --- a/docs/COMMAND_REFERENCE.md +++ /dev/null @@ -1,79 +0,0 @@ -# DocForge command reference - -> Generated from the live CLI parser and MCP registrations. Do not edit this file by hand. - -Global CLI options are documented in `docforge --help` and are not repeated in each command row. The MCP table is the complete generic application-enabled surface; the fixed reference-adapter server exposes only its read rows. - -## CLI commands - -| Command | Invocation | -|---|---| -| `apply` | `docforge apply [-h] --changeset-hash CHANGESET_HASH --applier APPLIER changeset_id` | -| `backlinks` | `docforge backlinks [-h] [--relation RELATION] [--limit LIMIT] node_id` | -| `build` | `docforge build [-h]` | -| `check` | `docforge check [-h]` | -| `configure` | `docforge configure [-h] --project PROJECT [--name NAME] [--capability-mode {read,proposal,application}] [--proposal-writer PROPOSAL_WRITER] [--canonical-applier CANONICAL_APPLIER] [--no-ast] [--manual-render-policy {auto,explicit,disabled}] [--portable-graph-policy {explicit,disabled}] [--live-viewer-policy {on-demand,disabled}] [--startup-timeout STARTUP_TIMEOUT] [--tool-timeout TOOL_TIMEOUT] [--output OUTPUT] {codex,claude,openclaw}` | -| `context` | `docforge context [-h] [--budget BUDGET] [--limit LIMIT] [--cursor CURSOR] profile` | -| `dependencies` | `docforge dependencies [-h] [--depth DEPTH] [--limit LIMIT] node_id` | -| `doctor` | `docforge doctor [-h] --client {codex,claude,openclaw} [--project PROJECT] [--config CONFIG] [--server-name SERVER_NAME]` | -| `filter` | `docforge filter [-h] [--family FAMILY] [--authority AUTHORITY] [--status STATUS] [--tag TAG] [--limit LIMIT]` | -| `generation-diff` | `docforge generation-diff [-h] [--limit LIMIT] [--cursor CURSOR]` | -| `graph-plan` | `docforge graph-plan [-h] view_id` | -| `graph-render` | `docforge graph-render [-h] view_id` | -| `graph-render-status` | `docforge graph-render-status [-h] [view_id]` | -| `impact` | `docforge impact [-h] [--depth DEPTH] [--limit LIMIT] node_id` | -| `info` | `docforge info [-h]` | -| `onboard` | `docforge onboard [-h] [--language LANGUAGE] [--scaffold] [--project-id PROJECT_ID] [--title TITLE] [--content-root CONTENT_ROOT]` | -| `preview` | `docforge preview [-h] changeset_id view_id` | -| `reindex` | `docforge reindex [-h]` | -| `render` | `docforge render [-h] view_id` | -| `render-status` | `docforge render-status [-h] [--deep] [view_id]` | -| `search` | `docforge search [-h] [--limit LIMIT] query` | -| `show` | `docforge show [-h] node_id` | -| `sync` | `docforge sync [-h]` | -| `validate` | `docforge validate [-h]` | -| `validate-index` | `docforge validate-index [-h]` | -| `visualization-status` | `docforge visualization-status [-h]` | -| `visualization-stop` | `docforge visualization-stop [-h]` | -| `visualize` | `docforge visualize [-h] [--node NODE \| --query QUERY] [--depth DEPTH] [--no-open]` | - -## MCP tools - -| Surface | Tool | Required arguments | Optional arguments | Input schema SHA-256 | Description | -|---|---|---|---|---|---| -| read | `docforge_backlinks` | `node_id` | `limit`, `relation` | `bf9a5936e2d1ee70897dc0795ec9d716be63c9447e6fdcd3890d5dd92f45f690` | Return bounded incoming relationships for one exact stable node. | -| read | `docforge_bootstrap` | — | — | `af6c17a59a7713cfb4497f0f66b05efc091abbfe109573bb86d673e39b9b7413` | Synchronize and report the complete fixed project binding and workflow. | -| read | `docforge_dependencies` | `node_id` | `depth`, `limit` | `1283a8a030cf9369493a13e2aa87317d371470904a397910bc0b52997b3ec3b8` | Traverse declared depends_on relationships within the configured depth limit. | -| read | `docforge_filter_nodes` | — | `authority`, `family`, `limit`, `status`, `tag` | `2a51c4d8f4d76d89a2e7f90b3b06d14c7e16303b9c81e62af6747286b58cf68b` | Filter current nodes deterministically by validated metadata. | -| read | `docforge_get_context` | `profile` | `budget`, `cursor`, `limit` | `2f164828c2891a88d0ea7173790a72cc6f3da1279f14da2ca495e8989880b298` | Compile bounded cited context from one configured profile with explicit omissions. | -| read | `docforge_get_contract` | — | — | `19eb6785523ed9db645d17f61df2b8b584ac10ad308199e9a96e6e176a87169e` | Report canonical and derived boundaries plus allowed and excluded operations. | -| read | `docforge_get_generation_diff` | — | `cursor`, `limit` | `0a2fccfea4bf5d78481d40b49e7c812ded9fbe9c93cb79296ede78f22b259fcf` | Return the latest bounded primary-graph generation transition. | -| read | `docforge_get_logic` | `owner_node_id` | — | `56909ee4d72143b470528ea2b3544325e1e1920dbcc758f3caee4a4924f904a1` | Return the lazy control-flow projection owned by one function or method. | -| read | `docforge_get_node` | `node_id` | — | `52408b0c3a297e7b5cf96b293fd6d4625f05c4b090c02bfe806d25ce6c2d1c68` | Return one exact stable node from the current validated project index. | -| read | `docforge_get_task_context` | `task`, `task_kind` | `budget`, `cursor`, `focus_node_id`, `limit` | `0ffbc030bbe5cca2ae24920ec812ac976be9090da4fa2887d87526aba64a21f7` | Return one bounded task-shaped context capsule with explicit evidence gaps. | -| read | `docforge_graph_plan` | `view_id` | — | `7a014ba692f57badca9ca2098b88f82f4749ada6d71f5a3b8bc141abc86645f9` | Plan one declared portable graph without publishing derived output. | -| read | `docforge_graph_render_status` | — | `view_id` | `1d2f3604b0151ba2e228cc6ce1829d7f4244193b53d85df46eb6a3e608c59564` | Report portable-graph publication state without rendering. | -| read | `docforge_impact` | `node_id` | `depth`, `limit` | `1c71db5bb4e1afe180d98dd5000a757b7e3fd1a048b5f68af13d9b7ac56e2ce2` | Traverse bounded incoming relationships and report exact paths. | -| read | `docforge_project_info` | — | — | `a42973543d5e6f4147953d175f6aec43d228cd6bd1a63f3ca4e84a51ad7e8a64` | Report the fixed project identity, revision, source hash, and index health. | -| read | `docforge_render_status` | — | `deep`, `view_id` | `5ea5683bd74e685d4cabb1c960a66eff23a001bd2c68e1af350c6e46e936b9fd` | Report receipt state, or explicitly recompute the side-effect-free render oracle. | -| read | `docforge_search` | `query` | `limit` | `735fff8bd82b20730b0907db8bf61eb0de9fff240ef4fa239ce555779ae49047` | Run bounded lexical search over the current validated project index. | -| read | `docforge_stop_visualization` | — | — | `066a6e34019f4abcd3523f24d9e447775e1479adbea1134341f434a6bb5f8d35` | Explicitly stop this project's persistent read-only graph browser. | -| read | `docforge_sync` | — | — | `a2ac50c03b9b0e08b271216f76920279563ed264bef06d6791695c68dcb3b2a2` | Ensure the disposable project index matches current canonical sources. | -| read | `docforge_validate_project` | — | — | `3192ba97a6d7fff13008c5ed3bfa549d1eeea95332579592d96b8fd326ad06f7` | Validate current canonical sources and graph without writing any project file. | -| read | `docforge_visualization_status` | — | — | `03751f891e0a9318355a6b28be551eec6009c0f96f05ff99536e2f401b76f87f` | Report this project's managed graph browser lifecycle state. | -| read | `docforge_visualize` | — | `depth`, `node_id`, `query` | `cdc9669ba934e591379ea684934d99e2e80ba3f181d6c58867774a8064c32f4b` | Start the fixed read-only graph browser for this configured project. | -| proposal | `docforge_abandon_changeset` | `changeset_id`, `expected_changeset_hash`, `reason` | — | `8813e40ffc12b1ae3d9abe73db8de214085a4a8042323146283fffa8f7476a62` | Mark one proposal abandoned while preserving its audit record. | -| proposal | `docforge_create_changeset` | `changeset_id` | — | `0de93025b9caaa9e33b4b2b3bcfabdde087c944daba48dd77ae3c8f757c52ab8` | Create an empty hash-bound proposal under the configured isolated changeset root. | -| proposal | `docforge_get_changeset` | `changeset_id` | `cursor`, `limit` | `919737a8e732cf95305a2964cddf98f1fdce9ce2cfb52ff659436c244a38495d` | Inspect a stored proposal even when its canonical base has become stale. | -| proposal | `docforge_get_changeset_diff` | `changeset_id` | `cursor`, `limit` | `5cc1d2b035f37e3866766cfa0c3541b144762b764bf455a641ab459df03f8a59` | Return a deterministic structured and textual diff without applying the proposal. | -| proposal | `docforge_list_changesets` | — | `cursor`, `include_history`, `limit`, `status` | `b96519477781cb20acbfc9502a4b7d64a9119e0bd8d71e27290e90677aaf9ced` | List active proposals by default, with optional lifecycle history. | -| proposal | `docforge_preview_changeset` | `changeset_id`, `view_id` | — | `d84e1710a5cac52607037a7e581fcb221c42d3776f510eefe42fb6779fee7fb3` | Render one validated changeset through a declared view into its isolated preview path. | -| proposal | `docforge_propose_node_create` | `changeset_id`, `content`, `expected_changeset_hash`, `metadata`, `node_id`, `rationale`, `relationship_changes`, `target_source` | — | `8953b502b9d8d9c097935bae51fce4e830286eace9df3cb6c37b7e0542be5a4e` | Append one validated node creation without writing its canonical target. | -| proposal | `docforge_propose_node_delete` | `changeset_id`, `expected_changeset_hash`, `expected_content_hash`, `node_id`, `rationale`, `relationship_changes` | — | `8fb6a22e5c1ead3793384652e4b61e75cc23c2725906542533a121401f7faaee` | Append one validated deletion with explicit incident relationship removals. | -| proposal | `docforge_propose_node_move` | `changeset_id`, `expected_changeset_hash`, `expected_content_hash`, `node_id`, `rationale`, `target_source` | — | `32ef6fc4b9795a6a8e30f4f81f63d84d7350e983773f796524382d71cb870a67` | Append one validated same-format node move without moving a canonical file. | -| proposal | `docforge_propose_node_update` | `changeset_id`, `content`, `expected_changeset_hash`, `expected_content_hash`, `metadata`, `node_id`, `rationale`, `relationship_changes` | — | `6a29a73b9e50c9fe22cb6d056bbcdacef30c93c0822c8a9276ef33f9e51e7728` | Append one validated node update without changing canonical content. | -| proposal | `docforge_propose_relationship_update` | `changeset_id`, `expected_changeset_hash`, `expected_content_hash`, `node_id`, `rationale`, `relationship_changes` | — | `8e21fc7ba35289940dd7f12294b14e8553163ab901002dcd4a3998cdb33556c3` | Queue hash-bound relationship changes without rewriting node content. | -| proposal | `docforge_rebase_changeset` | `changeset_id`, `expected_changeset_hash` | — | `505434b2f9f611649221b95d0251db347c44752d1c73e5d69e1abfd916966788` | Safely rebase a proposal when every touched fact remains unchanged. | -| proposal | `docforge_register_changes` | `changeset_id`, `operations` | — | `c5a7d6ac07d3d3b95c13b0c57ad9b69360f2f063b1c8e22dd5f64f6d3a83dc5b` | Atomically register and validate a complete hash-bound proposal. | -| proposal | `docforge_validate_changeset` | `changeset_id` | `cursor`, `limit` | `cb9df5104dbefc3382900ec812ea581a43612d1577619071f1f03674041652c7` | Validate a proposal against its exact canonical base and other active proposals. | -| application | `docforge_apply_changeset` | `changeset_id`, `expected_changeset_hash` | — | `80259f5001f36b14f960c10bd94751d1431077b72e081fb5a5c806b79073b085` | Apply one exact validated changeset and refresh declared derived state. | diff --git a/docs/COMPATIBILITY.md b/docs/COMPATIBILITY.md index 8eca1c9..3925f59 100644 --- a/docs/COMPATIBILITY.md +++ b/docs/COMPATIBILITY.md @@ -31,31 +31,16 @@ model, index, rendering, application, and MCP factory names imported from these remain supported: - `docforge.adapter_contract` -- `docforge.adapter_sdk` -- `docforge.adapter_launcher` - `docforge.application` -- `docforge.client_config` - `docforge.index` - `docforge.mcp_server` - `docforge.models` - `docforge.policy` - `docforge.render_contract` -- `docforge.reference_config` -- `docforge.reference_mcp` Names beginning with an underscore are implementation details. New public names may be added without breaking this contract. -The repository Python, JavaScript/TypeScript, and C++ adapters are supported reference -implementations. Their documented configuration, evidence limits, and unsupported-fact reports are -compatibility surfaces; their internal parser helpers are not adapter-authoring imports. - -Milestone 4 adds three version-1 schemas without changing existing descriptor or result schemas: - -- `schemas/adapter-launcher.schema.json` -- `schemas/adapter-client-configuration.schema.json` -- `schemas/reference-adapter.schema.json` - ## CLI and MCP surfaces Existing `docforge` command names and arguments remain supported. Existing `docforge-mcp` tool @@ -248,11 +233,6 @@ Milestone 3 adds `ManualRenderPlanV1`, `GraphViewPlanV1`, projection package and and projection policy version 2. These are additive submodule and schema contracts. They do not change the legacy task-context, adapter, changeset, or effective-policy contracts described above. -Milestone 4 adds `docforge.adapter_sdk`, the fixed reference configuration and read-only MCP -binding, immutable adapter launchers, and generated adapter client fragments. A legacy adapter with -only `load_projection()` remains first-class and need not adopt incremental extraction, Logic, a -reference configuration, or launcher metadata. - ## Safety boundary DocForge remains bound to one explicit project root. It rejects absolute paths, root escapes, and @@ -265,9 +245,8 @@ publication, or project switching. Milestone 0 records rather than redesigns these areas: - Generic warm reads still repeat whole-project discovery, parsing, and validation. -- The base wheel intentionally omits Tree-sitter. JavaScript, TypeScript, and C++ syntax evidence - requires the matching `docforge[javascript]`, `docforge[typescript]`, or `docforge[cpp]` extra. - Python reference evidence uses the standard library and remains available in the base wheel. +- Tree-sitter and the JavaScript and C++ grammars remain mandatory installation dependencies even + when their runtime modules are unused. - Several version strings and defaults remain duplicated. - One individually oversized context entry is represented as explicit bounded omission evidence; callers use targeted retrieval for that node. diff --git a/docs/CONTRACT.md b/docs/CONTRACT.md index 177a62d..cfbaf30 100644 --- a/docs/CONTRACT.md +++ b/docs/CONTRACT.md @@ -29,9 +29,6 @@ commit when Git is available; it cannot change repository state. - Projection package: `schemas/projection-package.schema.json`, version 1. - Projection receipt: `schemas/projection-receipt.schema.json`, version 1. - Independent projection policy: `schemas/projection-policy.schema.json`, version 2. -- Adapter launcher: `schemas/adapter-launcher.schema.json`, version 1. -- Adapter client configuration: `schemas/adapter-client-configuration.schema.json`, version 1. -- Reference adapter configuration: `schemas/reference-adapter.schema.json`, version 1. - Index schema: version 3, disposable and reproducible. - Index attestation: schema version 1, disposable and reproducible. - Core, CLI, and MCP server: version 1.3.0.dev0. @@ -106,42 +103,6 @@ complete projection, opens SQLite, starts MCP, executes the configured command, builds, renders, starts a viewer, or writes configuration. Unprovable client behavior is a warning, not an invented success. -## Public adapter SDK and reference binding - -`docforge.adapter_sdk` is the stable adapter-authoring import boundary. It exposes the typed -projection, manifest, source contribution, complete assembly, project wrapper, graph model, and -conformance contracts needed by an adapter without requiring authors to import core implementation -modules. - -Complete evidence includes the primary graph and function Logic. An incremental adapter that -publishes Logic must implement `load_complete_assembly()` as an independent complete oracle. -`verify_adapter_conformance()` proves repeated complete determinism, equality between the complete -assembly and `load_projection()`, and exact complete/incremental graph-plus-Logic parity. Separate -tests remain responsible for confinement, restart behavior, no-AST behavior, cache recovery, and -retrieval. - -Adapter assemblies are bounded before publication. Primary nodes use the descriptor `max_nodes` -limit. Primary edges, Logic nodes, and Logic edges use fixed deterministic multipliers over that -limit. Version-1 extraction caches are regular-file-only, bounded to 10,000 sources and -64,000,000 bytes, and are treated as misses when corrupt, oversized, foreign, or incompatible. - -`.docforge/reference-adapter.toml` is a closed version-1 selection among `python`, `javascript`, -`typescript`, and `cpp`. It declares one project identity and explicit non-overlapping source -roots. C++ additionally requires a confined `compile_commands.json`. It cannot declare a command, -module, environment, writer, applier, or remote endpoint. - -`python -m docforge.reference_mcp --project-root ROOT` constructs only the selected fixed -repository reference adapter and exposes the read surface. It never registers proposal or -application tools. - -`AdapterLauncherV1` is an immutable project-bound Python-module declaration. It accepts no -arbitrary command, arguments, working directory, environment, discovery, callable selector, or -module reload. Custom launchers resolve one installed top-level module through isolated Python and -require its origin inside the project root. The fixed `docforge.reference_mcp` module is the only -trusted dotted exception. `generate_adapter_client_configuration()` binds generated Codex, -Claude, and OpenClaw fragments to that launcher, current source availability, effective policy, -descriptor, interpreter, and exact artifact bytes. - ## Isolated proposal model Create, update, move, and delete are ordered node operations inside an isolated changeset. Every diff --git a/docs/CORE_CONCEPTS_AND_AUTHORITY.md b/docs/CORE_CONCEPTS_AND_AUTHORITY.md deleted file mode 100644 index 4fc1c9d..0000000 --- a/docs/CORE_CONCEPTS_AND_AUTHORITY.md +++ /dev/null @@ -1,165 +0,0 @@ -# Core concepts and authority - -DocForge is a project-bound knowledge compiler. It turns explicit canonical project facts into -validated graphs and bounded derived views without transferring authority to the index, an agent, -or a renderer. - -## One project, one explicit root - -Every operation is bound to one canonical real project directory. Paths in descriptors and adapter -configuration are project-relative and confined beneath that root. A CLI or MCP process does not -discover or switch projects after startup. - -The project root determines: - -- which descriptor and canonical sources may be read; -- where derived cache and changeset roots may exist; -- which project identity, revision, source hash, and generation appear in results; -- which writer, applier, rendering, and viewer policies can be selected. - -Generated client fragments preserve that binding. They are machine-local configuration -projections, not portable project authority. - -## Canonical facts and derived evidence - -Canonical inputs own the facts: - -- generic Markdown or TOML node sources; -- authority files named by a generic descriptor; -- an adapter's declared canonical sources and implementation boundary; -- `.docforge/project.toml` for a generic project; -- `.docforge/reference-adapter.toml` for a fixed reference integration. - -Everything DocForge builds from those inputs is derived: - -- SQLite indexes, attestations, generation receipts, and generation diffs; -- incremental extraction caches; -- task-context capsules and query responses; -- changeset previews; -- render plans, immutable packages, fragments, artifacts, and receipts; -- portable graph publications and live-viewer processes; -- generated Codex, Claude, and OpenClaw fragments. - -Derived state may be discarded and rebuilt. A derived artifact can prove what it was bound to, but -it cannot override current canonical content. - -## Nodes, relationships, and Logic - -The primary graph contains nodes and directed relationships. - -A node has a stable project-wide ID, title, family, authority, status, tags, summary, content, -source identity, and content hash. A generic Markdown file contains one node beneath a TOML -metadata block; a generic TOML source may contain multiple nodes. Adapter nodes use the same public -graph contract. - -A relationship is an exact `(source, relation, target)` triple. The project descriptor defines the -allowed relation names. DocForge gives `depends_on` special acyclic validation, but it does not -invent domain meaning for a project's other relation names. Retrieval recognizes only a versioned -alias set for task planning and reports unknown allowed relations as `unclassified`. - -Logic is deliberately separate. It is a lazy function-scoped control-flow projection owned by one -primary node. Decisions, actions, loops, convergence points, returns, and exceptions connect -through explicit branch edges. Logic does not add statement-level nodes to ordinary Nodes, Flow, -Web, search, or generation-diff results. Python, JavaScript, TypeScript, and C++ integrations may -publish Logic when their adapter contract supports it. - -## Authority, status, family, and tags - -These fields answer different questions: - -- `authority` describes the role of the content. The generic vocabulary is `authoritative`, - `approved_plan`, `derived`, `proposal`, and `historical`. -- `status` is project-defined lifecycle state such as `current`, `active`, or `verified`. -- `family` is a project-defined content grouping used for filtering, profiles, rendering, and - writer permissions. -- `tags` are exact project labels for retrieval and presentation. - -An `authoritative` node can still become stale; authority is not a freshness claim. A `derived` -node is still canonical if it is stored in a declared canonical source; the label describes its -role, not whether DocForge may silently regenerate it. Status and authority never grant an MCP -writer permission. - -## Validation, synchronization, and generations - -Validation loads the complete canonical graph and rejects unsafe paths, invalid source formats, -duplicate IDs, unresolved relationships, prohibited cycles, violated project limits, and -adapter-specific contract failures. - -The disposable index is published atomically only after the complete graph and SQLite integrity -checks pass. Its attestation binds the whole index file. A successful replacement is the derived -publication commit point; later receipt-writing trouble is reported as degraded evidence rather -than as a false claim that the replacement failed. - -MCP operations automatically synchronize derived state under a project lock before normal work. -A graph generation identifies one validated indexed snapshot. Generation-pinned retrieval and -rendering do not reopen mutable sources behind an older snapshot. - -The latest generation diff is one bounded primary-graph transition, not a history database. It -contains no Logic details or source text. - -## Complete and incremental adapters - -`load_projection()` is the compatibility baseline and clean graph oracle. An incremental adapter -adds: - -- `load_manifest()` for cheap project identity, source inventory, fingerprints, and dependencies; -- `extract_source()` for one cacheable source contribution; -- optionally `assemble_projection()` to normalize overlapping contributions. - -When incremental contributions contain Logic, `load_complete_assembly()` supplies a -cache-independent complete graph-plus-Logic oracle. Warm cache behavior is an optimization, never a -different authority path. Corrupt or incompatible extraction caches are treated as misses, and a -clean complete build remains the equivalence and recovery boundary. - -The public types, validators, and conformance helper are exported from `docforge.adapter_sdk`. See -the [Adapter authoring guide](ADAPTER_AUTHORING_GUIDE.md) and [Incremental -indexing](INCREMENTAL_INDEXING.md). - -## Proposals are not canonical changes - -A changeset is an isolated, ordered proposal over an exact canonical base. Each operation names -preconditions, and the final changeset has a content-derived hash. Validation projects the complete -resulting graph before application. - -Canonical application requires: - -1. a writer declared in the descriptor; -2. a process started with the matching proposal and application authority; -3. one explicit final changeset hash; -4. unchanged source, relationship, permission, and graph preconditions. - -Application does not perform Git mutation, build, deployment, or publication. Until exact-hash -application succeeds, canonical project files remain unchanged. - -## Three independent output projections - -Manual rendering, portable graph publication, and the live viewer are separate: - -- a manual is a declared derived HTML view over selected nodes; -- a portable graph is a content-addressed static Nodes, Flow, or Web artifact; -- the live viewer is a managed loopback process pinned to one validated index generation and can - request lazy Logic. - -Their policies compose independently. Disabling one does not transfer its authority to another. -None is canonical documentation or a retrieval authority. See [Rendering and -visualization](RENDERING_AND_VISUALIZATION.md). - -## Binding policy is not project truth - -Capability mode, no-AST preservation, diagnostics, and projection modes describe one running -process or generated client binding. They do not rewrite the descriptor or canonical graph. - -`--no-ast` forbids AST-family adapter evolution and Logic publication/retrieval for that binding. -It does not inspect parser implementation, sandbox the filesystem, or convert an existing -AST/Tree-sitter adapter into a no-AST adapter. Read [Policy precedence](POLICY_PRECEDENCE.md) and -[Legacy and no-AST operation](LEGACY_AND_NO_AST.md). - -## Trust the narrowest evidence - -DocForge reports stable IDs, hashes, generations, omissions, truncation, and provenance limits so a -consumer can distinguish proof from inference. A source inventory is not a semantic graph; a -syntax-level relationship is not compiler resolution; a configuration doctor is not a connection -test; a viewer snapshot is not continuous monitoring; and a benchmark is not a release. - -Continue with the [Project descriptor](PROJECT_DESCRIPTOR.md), [New-project -quickstart](NEW_PROJECT_QUICKSTART.md), or [Core contract](CONTRACT.md). diff --git a/docs/INCREMENTAL_INDEXING.md b/docs/INCREMENTAL_INDEXING.md index 8af16da..4abb941 100644 --- a/docs/INCREMENTAL_INDEXING.md +++ b/docs/INCREMENTAL_INDEXING.md @@ -4,10 +4,6 @@ DocForge Release 1 adapters return one complete immutable projection. That contr supported. The incremental compiler adds an opt-in source-scoped contract that avoids reparsing unchanged files while preserving the same validated, atomically published graph. -Import these contracts from the public `docforge.adapter_sdk` facade. See -[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for the preserved one-method contract and -[Reference Adapters](REFERENCE_ADAPTERS.md) for the four maintained implementations. - ## Release 1 compatibility The incremental interface is additive: @@ -38,26 +34,18 @@ Incremental indexing is an extraction optimization. It does not weaken publicati An interrupted extraction never replaces the last validated SQLite index. A malformed, incompatible, or missing cache is a cache miss, not a partial graph. -One cache generation is bounded to 10,000 source contributions and 64,000,000 encoded bytes. -Aggregate graph and Logic assembly limits are described in the -[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md#current-aggregate-bounds). - ## Adapter contract -An incremental loader implements the first, second, and fourth methods. It implements -`load_complete_assembly()` as well when it publishes Logic: +An incremental loader implements all three methods: ```python class MyAdapter: def load_manifest(self) -> AdapterManifest: ... def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ... - def load_complete_assembly(self) -> AdapterAssembly: ... def load_projection(self) -> AdapterProjection: ... ``` -`load_projection()` remains the deterministic full-rebuild primary-graph oracle. An incremental -adapter that publishes Logic must additionally implement `load_complete_assembly()` as the -cache-independent complete graph-plus-Logic oracle. +`load_projection()` remains the deterministic full-rebuild fallback and equivalence oracle. Each `AdapterSource` declares: @@ -149,8 +137,7 @@ fresh process must import and validate the current adapter. ``` Adapters can call `AdapterProject.verify_incremental_equivalence()` in release and contract tests. -The check compares project identity, revision, source hash, nodes, relationships, and Logic against -the independent complete assembly. It also requires the complete assembly's primary graph to match +The check compares project identity, revision, source hash, nodes, and relationships against `load_projection()`. ## Manual changes and relationships @@ -183,29 +170,12 @@ logic empty until they implement a language analyzer. This boundary prevents thousands of boolean expressions and basic blocks from polluting Nodes, Flow, Web, ordinary search, or architectural traversal. The Logic tab and `docforge_get_logic` request one function-scoped projection on demand. The built-in analyzers cover Python, -JavaScript, TypeScript, and C++. Python uses the standard-library AST. JavaScript, TypeScript, and -C++ use distinct optional Tree-sitter grammars with thin language-aware control-flow profiles. -Ordinary graph reads use stored projections and do not load or execute these parsers. A grammar -alone supplies syntax, not control-flow meaning, so each new language still needs a profile for -its branch, loop, case, exception, and termination constructs. All analyzers report possible -static paths; they do not claim runtime branch outcomes. - -## Manifest and warm-parser scope - -Parser work is language-specific and must be measured at the correct boundary: - -- Python manifest construction fingerprints source and tokenizes local imports without calling - `ast.parse`. -- JavaScript and TypeScript manifest construction lexes static relative module specifiers without - invoking their distinct Tree-sitter extraction parsers. Focused tests prove this behavior on an - unchanged warm build. -- The C++ reference manifest parses inventoried sources with `tree-sitter-cpp` to discover quoted - include dependencies. A warm C++ extraction-cache hit is not a zero-parser claim. - -The maintained Python benchmark instruments the unchanged warm path and requires zero -`ast.parse` calls and zero `extract_source` calls. That exact zero-parser benchmark claim is Python -only. JavaScript and TypeScript retain focused parser-free-manifest tests; C++ deliberately does -not. +JavaScript, and C++. Python uses the standard-library AST. JavaScript and C++ share pinned +Tree-sitter infrastructure with thin language-aware control-flow profiles. Parsers run only while +extracting a changed source contribution; ordinary graph reads do not load or execute them. A +grammar alone supplies syntax, not control-flow meaning, so each new language still needs a small +semantic profile for its branch, loop, case, exception, and termination constructs. All analyzers +report possible static paths; they do not claim runtime branch outcomes. ## Full rebuilds diff --git a/docs/LEGACY_AND_NO_AST.md b/docs/LEGACY_AND_NO_AST.md deleted file mode 100644 index c1125c5..0000000 --- a/docs/LEGACY_AND_NO_AST.md +++ /dev/null @@ -1,80 +0,0 @@ -# Legacy adapters and no-AST policy - -DocForge preserves the original one-method adapter contract while offering incremental extraction, -complete graph-plus-Logic assemblies, and an independently selectable no-AST MCP policy. These are -separate compatibility boundaries. - -## One-method adapters remain valid - -An existing adapter that implements only: - -```python -def load_projection(self) -> AdapterProjection: ... -``` - -continues to work. It receives the same projection validation, indexing, querying, visualization, -and MCP behavior as before. It does not receive extraction-cache speedups and publishes no Logic -through the complete assembly contract. - -Incremental adoption is additive. Implement `load_manifest()` and `extract_source()` while keeping -`load_projection()` as the independent complete graph oracle. If incremental contributions publish -Logic, also implement `load_complete_assembly()` so complete and incremental graph-plus-Logic -output can be compared exactly. The public authoring surface is `docforge.adapter_sdk`; see the -[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md). - -## What no-AST means - -`--no-ast` is an immutable policy on one MCP process binding. It preserves the configured adapter -strategy but forbids AST, Tree-sitter, compiler-AST, and function-Logic evolution for that binding. -It also rejects nonempty Logic before index publication and rejects a pre-existing index that -contains Logic before reads or live visualization. - -Start the generic server with: - -```bash -docforge-mcp --project-root /absolute/path/to/project --no-ast -``` - -Project-owned integrations select the same policy with -`create_project_server(..., no_ast=True)` or `create_read_only_server(..., no_ast=True)`. -`docforge_bootstrap` and `docforge_get_contract` report the effective policy, and -`docforge_get_logic` returns `adapter_policy_forbids_logic`. - -Changing this policy requires a new process. It does not rewrite or reload an adapter in place. - -## What no-AST does not mean - -No-AST is not: - -- an inspection mechanism that proves which parser an arbitrary adapter uses; -- a filesystem, Python-import, or process sandbox; -- a promise that the adapter performs no non-AST fingerprinting, dependency discovery, caching, or - complete-projection loading; or -- permission to change project files, adapter code, or the configured capability surface. - -Repository permissions and project instructions remain responsible for processes that have direct -filesystem access. The MCP boundary still excludes shell execution, arbitrary file operations, -Git mutation, builds, deployment, publication, project switching, and cross-project retrieval. - -## Reference adapters are not no-AST adapters - -The built-in Python reference extracts syntax and Logic with the standard-library AST. The -JavaScript, TypeScript, and C++ references extract syntax and Logic with their optional -Tree-sitter grammars. Their normal projections contain Logic, so they must not be presented or -configured as no-AST adapters. - -The narrower performance statements in [Reference Adapters](REFERENCE_ADAPTERS.md) concern -specific manifest and warm-cache paths. A parser-free manifest does not turn an adapter that -publishes AST-derived Logic into a no-AST adapter. - -## Migration choices - -Keep a one-method adapter when its complete rebuild is acceptable and no function Logic is needed. -Adopt incremental extraction when measured source parsing dominates synchronization. Adopt the -complete assembly contract when Logic is published or overlapping raw evidence needs deterministic -ownership. - -Choose no-AST only when the project binding must forbid AST-derived publication and future adapter -upgrades of that kind. It is a binding policy, not a substitute for adapter confinement, -conformance tests, restart detection, cache recovery, or the full proof matrix in -[Incremental Adapter Indexing](INCREMENTAL_INDEXING.md). diff --git a/docs/MCP_CONTRACT.md b/docs/MCP_CONTRACT.md index d1c345d..b710bb8 100644 --- a/docs/MCP_CONTRACT.md +++ b/docs/MCP_CONTRACT.md @@ -28,30 +28,6 @@ Bootstrap and contract results include `effective_policy` schema version 1 plus registered surface and startup-bound proposal/application access. The project descriptor remains schema version 1 and does not silently acquire machine-specific process policy. -## Fixed reference binding - -`docforge.reference_mcp` is the fixed runnable binding for the in-repository Python, JavaScript, -TypeScript, and C++ reference adapters: - -```bash -python -I -m docforge.reference_mcp \ - --project-root /absolute/project \ - --capability-mode read -``` - -It loads only `.docforge/reference-adapter.toml`, selects only the fixed provider for the declared -language, and constructs the same project-bound read surface through `create_read_only_server()`. -Bootstrap binding metadata includes `server_module = "docforge.reference_mcp"`, -`adapter_mode = "reference"`, the selected `reference_language`, and the exact -`reference_config_hash`. - -The reference command accepts read capability mode only. It registers all 21 read tools below and -no isolated proposal or canonical-application tools. The configuration cannot select a provider, -module, command, arguments, working directory, environment, or arbitrary discovery behavior. See -[Reference Adapters](REFERENCE_ADAPTERS.md) for language scope and -[Agent Integration](AGENT_INTEGRATION.md) for the immutable launcher and generated client -fragments. - ## Read tools - `docforge_bootstrap` @@ -69,8 +45,6 @@ fragments. - `docforge_get_task_context` - `docforge_validate_project` - `docforge_render_status` -- `docforge_graph_plan` -- `docforge_graph_render_status` - `docforge_visualize` - `docforge_stop_visualization` - `docforge_visualization_status` @@ -375,6 +349,3 @@ permissions and project instructions remain responsible for that broader boundar The legacy `adapter_policy` object remains byte-compatible. It is now a projection of the versioned `effective_policy`; `--no-ast` restrictively overrides adapter evolution, AST analysis, and Logic indexing without widening any other capability. - -See [Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for the compatibility boundary and -why the AST and Tree-sitter reference adapters are not no-AST adapters. diff --git a/docs/MIGRATING_FROM_V1.md b/docs/MIGRATING_FROM_V1.md deleted file mode 100644 index dab9669..0000000 --- a/docs/MIGRATING_FROM_V1.md +++ /dev/null @@ -1,79 +0,0 @@ -# Migrating from DocForge v1 - -DocForge2 preserves the `docforge` distribution, Python package, CLI command, MCP tool prefix, -generic project descriptor version, and legacy adapter entry point. Migration is an additive -validation exercise, not a canonical-content rewrite. - -Read [compatibility](COMPATIBILITY.md), [legacy and no-AST operation](LEGACY_AND_NO_AST.md), and -[recovery](RECOVERY_AND_PERFORMANCE.md) before changing a production binding. - -## What remains compatible - -- Generic Markdown and TOML projects keep `.docforge/project.toml` schema version 1. -- A custom adapter implementing only `load_projection()` remains valid. -- Existing canonical nodes, stable IDs, source paths, relationship vocabulary, changesets, and - reviewed hashes are not silently rewritten. -- Existing CLI and MCP names remain available. New fields, commands, tools, plans, policies, and - reference integrations are additive. -- `docforge-mcp --no-ast` remains the shorthand for the preserved no-AST binding. - -Disposable index and cache schemas may change. Rebuild them rather than copying them as authority. - -## Recommended migration - -1. Record the v1 package version, adapter identity, project descriptor, canonical source hash, - active changesets, generated client configuration, and current rendered outputs. -2. Back up canonical sources and active proposal files. Derived `.docforge/cache`, preview, - portable-graph, and viewer state need not be authoritative backups. -3. Install the DocForge2 candidate in a separate environment. Do not repoint the production MCP - binding yet. -4. Run the existing adapter through complete loading and `ProjectIndex.check()`. For incremental - adapters, verify complete/incremental equivalence. Logic-producing incremental adapters must - implement the independent complete assembly oracle. -5. Compare project ID, adapter, revision, source hash, primary node and edge hashes, Logic policy, - and retrieval results with the v1 evidence. -6. Recompose effective capability and projection policy. Do not assume that a new default grants - proposal, application, rendering, or viewer access. -7. Rebuild disposable indexes, extraction caches, fragments, receipts, previews, and portable - artifacts from the validated candidate. -8. Generate a new client fragment. Generic projects use `docforge configure`; custom adapters use - `AdapterLauncherV1` with `generate_adapter_client_configuration()`. -9. Run doctor, real MCP bootstrap, representative retrieval, rendering status, recovery, and - exact-hash proposal/application checks in a disposable or shadow environment. -10. Repoint one production binding only after the candidate and rollback procedure pass. - -## Adopting reference adapters - -The repository reference adapters are narrow syntax evidence, not automatic semantic replacements -for mature v1 integrations. Python publishes modules, classes, functions, and local imports. -JavaScript and TypeScript publish syntax plus static project-relative imports and re-exports. -C++ uses a confined `compile_commands.json` as inert translation-unit inventory and publishes -directly resolvable project-local quoted includes. - -Do not replace a compiler-, language-server-, or project-owned semantic adapter if the task depends -on resolved calls, types, inheritance, macros, compiler include semantics, runtime facts, or -ownership. Use [reference adapters](REFERENCE_ADAPTERS.md) as a measured starting point. - -## Changesets - -Existing changesets remain bound to their original project, root, base revision, source hash, -writer, operations, and exact hash. Do not edit a changeset to make it look current. Retrieve and -validate it through the candidate. Rebase only when every precondition remains true; otherwise -create and review a new proposal. - -Before enabling canonical application, prove the exact serializer round trip and recovery behavior -for the selected adapter. The fixed reference MCP server is intentionally read-only. - -## Rollback - -Keep the v1 environment, configuration fragment, and service definition until the DocForge2 -binding has passed live validation. To roll back: - -1. Stop the DocForge2 MCP/viewer processes. -2. Restore the previous client or service binding. -3. Restore canonical sources only if a verified application receipt says they changed and the - adapter-specific rollback requires it. -4. Discard DocForge2-derived indexes, receipts, fragments, previews, and portable artifacts. -5. Restart v1 and verify its recorded project/source identity. - -Never move a published tag or reuse a version identity for a corrected release. diff --git a/docs/MILESTONE_4_BASELINE.md b/docs/MILESTONE_4_BASELINE.md deleted file mode 100644 index 8aa1abb..0000000 --- a/docs/MILESTONE_4_BASELINE.md +++ /dev/null @@ -1,95 +0,0 @@ -# Milestone 4 baseline - -## Scope and method - -This baseline records the adapter SDK, Python reference adapter, incremental equivalence, and -recovery behavior completed in Milestone 4. It was captured on 2026-07-29 from clean executable -candidate `95271dcf2e48045b9d3aed9b9ea09c7fc155692c`. - -The maintained command was: - -```bash -.venv/bin/python tools/milestone4_benchmark.py \ - --mode full \ - --output benchmarks/milestone4-2026-07-29.json -``` - -The synthetic project contains 334 Python files. Each file contributes one module, one function, -and one argument node, for 1,002 primary nodes. Imports form a deterministic chain. Durations use -`time.perf_counter_ns()` and nearest-rank p95. Each ordinary result is serialized as compact sorted -JSON for its response-size gate. Per-operation memory uses `tracemalloc`; cumulative process -high-water uses `RUSAGE_SELF`. - -Environment: - -- Linux 7.1.3-200.nobara.fc44.x86_64 with glibc 2.43. -- CPython 3.14.6. -- x86_64. -- Three warm samples; cold, equivalence, and recovery operations run once. -- Per-operation traced-memory ceiling: 268,435,456 bytes. -- Process high-water ceiling: 536,870,912 bytes. -- Response ceiling: 524,288 bytes. - -The complete machine-readable result is -[`benchmarks/milestone4-2026-07-29.json`](../benchmarks/milestone4-2026-07-29.json). -Its SHA-256 is `b6a871dde730a119fc0a138c47bd25f2c533ed3171a9a3d098075aef08173600`. -The stable evidence payload SHA-256 is -`4a8db461a311b5df4abd5aa00063e9a347d8b9dba19b30e35684f561d5271549`. - -## Results - -| Operation | Median | p95 | Limit | Traced peak | Response | -|---|---:|---:|---:|---:|---:| -| Cold incremental build | 1,599.760 ms | 1,599.760 ms | 20,000 ms | 68,573,540 B | 14,580 B | -| Warm incremental build | 1,244.427 ms | 1,265.387 ms | 20,000 ms | 69,997,166 B | 14,578 B | -| Complete/incremental equivalence | 1,897.919 ms | 1,897.919 ms | 30,000 ms | 64,305,919 B | 230 B | -| Corrupt extraction-cache recovery | 1,695.406 ms | 1,695.406 ms | 30,000 ms | 69,776,923 B | 14,580 B | -| Corrupt index recovery | 1,707.879 ms | 1,707.879 ms | 30,000 ms | 68,561,924 B | 14,779 B | - -Process high-water was 78,798,848 bytes. The regression limits intentionally leave multiple times -the measured headroom; they are tripwires, not performance promises. - -## Deterministic graph and Logic evidence - -The candidate produced: - -- 1,002 primary nodes with hash - `f7705dedf8dd388857a20d11f459dc74de797bcd12d3ed36e7f3aa75d67c328f`. -- 1,001 primary edges with hash - `00cc90998b6783afc8c9d1fd900409e5e3ba352868c4ba30e9bb2cbd59c52d35`. -- 334 Logic projections containing 2,338 Logic nodes and 2,338 Logic edges, with hash - `c86a3ae74777c2cec3a82c83e6e5bcca0196772ccea63cb13340fa9141593b0c`. -- Complete assembly hash - `2888182ab765fbffe3ba873c1613345640e8e6d89be74ddfca7452c0a5056345`. -- Source hash - `30ef23aa41061de2d4a7c995fe109d7a41518d9ee5493d805dd256581b47dae2`. - -The independently loaded complete assembly and the incremental assembly matched exactly across -project identity, revision, source, nodes, edges, and Logic. - -## Work and recovery gates - -The warm build recorded all 334 sources as cache hits, zero reparsed sources, zero `ast.parse` -calls, and zero `extract_source` calls. Corrupting the extraction cache forced all sources through -extraction and reproduced the same graph and Logic hashes. Corrupting SQLite rebuilt the index -entirely from cache hits with zero parsing or extraction and reproduced those hashes. - -This zero-parser evidence applies to the Python benchmark. Focused JavaScript and TypeScript tests -prove parser-free manifests. The C++ reference manifest uses Tree-sitter for bounded quoted-include -discovery and does not make a zero-warm-parser claim. - -## Fresh-wheel and reference evidence - -The offline adoption proof built and installed the base wheel without Tree-sitter packages, -built and checked a real Python reference project, started the isolated reference MCP server, and -performed bootstrap, search, and exact retrieval over its 21 read tools. Selecting C++ without its -extra failed with `optional_dependency_missing` and `docforge[cpp]` remediation. - -Focused fixtures produced: - -- Python: 14 nodes, 13 edges, 5 Logic projections. -- JavaScript: 14 nodes, 14 edges, 5 Logic projections. -- TypeScript: 13 nodes, 14 edges, 4 Logic projections. -- C++: 17 nodes, 16 edges, 5 Logic projections. - -These fixture counts verify implementations; they are not language-wide completeness claims. diff --git a/docs/MILESTONE_4_CLOSEOUT.md b/docs/MILESTONE_4_CLOSEOUT.md deleted file mode 100644 index b5eda38..0000000 --- a/docs/MILESTONE_4_CLOSEOUT.md +++ /dev/null @@ -1,83 +0,0 @@ -# Milestone 4 closeout - -## Outcome - -Milestone 4 is complete. New projects can adopt a public adapter SDK or one of four narrow -repository reference integrations, attach a fixed read-only MCP server, and follow maintained -product documentation without reading core implementation. - -Implemented contracts: - -- Stable `docforge.adapter_sdk` authoring imports. -- Independent complete primary-graph-plus-Logic oracle and exact incremental equivalence. -- Bounded adapter assemblies and version-1 extraction caches. -- Base Python, optional JavaScript, optional TypeScript, and optional C++ reference integrations. -- Closed `.docforge/reference-adapter.toml` and fixed `docforge.reference_mcp` read-only binding. -- Immutable, project-bound, launchable `AdapterLauncherV1` declarations and generated Codex, - Claude, and OpenClaw fragments for custom adapters. -- Live implementation-derived CLI and MCP reference tables with race-safe publication and drift - checking. -- Strict documentation graph, link, anchor, H1, reachability, required-page, generated-notice, and - documented-reference-config checks. -- Offline fresh-wheel adoption and maintained 1,002-node scale/recovery gates. - -## Candidate evidence - -The frozen executable candidate is -`95271dcf2e48045b9d3aed9b9ea09c7fc155692c`. - -Its complete executable gate passed: - -- Ruff formatting and Python lint. -- HTML, rendered-manual HTML, portable-graph HTML, CSS, and JavaScript checks. -- Pyright with zero diagnostics. -- Warning-strict compilation. -- 142 contract tests and 268 subtests. -- 347 complete tests and 402 subtests. -- Three Playwright and axe accessibility flows for the manual, portable graph, and live viewer. -- Lock and npm dependency-tree checks. -- Wheel and source-distribution builds. -- Offline fresh-wheel adoption. -- Milestone 0, 1, 2, 3, and 4 smoke benchmarks. - -The clean full benchmark passed exact graph-plus-Logic equivalence, warm zero Python parser and -extraction work, corrupt-cache recovery, corrupt-index recovery, response, memory, and latency -gates. Exact results are in [the Milestone 4 baseline](MILESTONE_4_BASELINE.md) and -[`benchmarks/milestone4-2026-07-29.json`](../benchmarks/milestone4-2026-07-29.json). - -Gitleaks 8.30.1 scanned the Milestone 4 commit range and candidate tree with no findings. The SSH -remote syntax prevents Gitleaks from constructing finding hyperlinks; it does not affect scanning. - -## Reference scope - -- Python uses the standard-library AST and publishes syntax plus local imports. -- JavaScript and TypeScript use distinct optional Tree-sitter grammars and publish syntax plus - project-local static relative imports and re-exports. -- C++ uses a confined `compile_commands.json` as inert translation-unit inventory and publishes - syntax plus directly resolvable project-local quoted includes. -- The references do not claim resolved calls, inheritance, types, symbol references, compiler - include semantics, macro semantics, runtime behavior, or semantic ownership. - -The C++ reference never executes a compiler or compilation-database command. It is not a Clang -semantic adapter. - -## Preserved boundaries - -- The `docforge` distribution, Python package, CLI, MCP executable, and tool names remain. -- Generic projects and one-method `load_projection()` adapters remain supported. -- Descriptor schema version 1, index schema version 3, effective policy version 1, and no-AST - behavior remain. -- Heavy language frontends are optional. Base generic and Python operation installs no - Tree-sitter distribution. -- Reference MCP is read-only. Proposal and application remain explicit project-owned gates. -- No language adapter was separately published. -- No WorldForge, ScrapeStation, legacy-repository, production-binding, storage, or self-hosting - change was made. -- No tag or Forgejo release was created for Milestone 4. - -## Later work - -Milestone 5 owns stabilization and the first DocForge2 release. Release identity, compatibility -matrix, migration and recovery proofs, comparative real-task evidence, versioning, tagging, and -publication must be validated there. Remote adapters, render farms, third-party renderers, -cross-project graphs, storage replacement, and self-hosting remain deferred without measured need. diff --git a/docs/NEW_PROJECT_QUICKSTART.md b/docs/NEW_PROJECT_QUICKSTART.md index 808fb3e..a3460ba 100644 --- a/docs/NEW_PROJECT_QUICKSTART.md +++ b/docs/NEW_PROJECT_QUICKSTART.md @@ -1,276 +1,6 @@ -# New-project quickstart +# DocForge setup moved to the user manual -This guide takes a new installation from an empty project binding to a validated generic manual or -one of DocForge's fixed reference source graphs. For the complete operating reference, see the -[user manual](USER_MANUAL.md). +The complete installation, project setup, visualization, CLI, MCP, application, adapter, and +troubleshooting reference now lives in the [DocForge user manual](USER_MANUAL.md). -## 1. Install DocForge - -DocForge requires Python 3.12 or newer. A base installation includes the generic project service, -the public adapter SDK, the Python reference integration, CLI, MCP server, renderers, and viewer -assets. It does not install Tree-sitter. - -From a source checkout: - -```bash -git clone /absolute/path/DocForge -cd /absolute/path/DocForge -uv sync --group dev - -DOCFORGE=/absolute/path/DocForge/.venv/bin/docforge -DOCFORGE_MCP=/absolute/path/DocForge/.venv/bin/docforge-mcp -DOCFORGE_PYTHON=/absolute/path/DocForge/.venv/bin/python -``` - -For an isolated consumer environment, install the checkout or a built wheel with `uv pip install`. -Add only the language extras that project needs: - -```bash -uv venv /absolute/path/docforge-env --python 3.12 -uv pip install --python /absolute/path/docforge-env/bin/python /absolute/path/DocForge -uv pip install --python /absolute/path/docforge-env/bin/python \ - "/absolute/path/DocForge[javascript]" -uv pip install --python /absolute/path/docforge-env/bin/python \ - "/absolute/path/DocForge[typescript]" -uv pip install --python /absolute/path/docforge-env/bin/python \ - "/absolute/path/DocForge[cpp]" -``` - -The `languages` extra installs all three optional grammar families. JavaScript and TypeScript are -separate extras because they use distinct grammars. The C++ extra supplies a syntax grammar, not a -compiler or Clang semantic frontend. - -Confirm the installation: - -```bash -"$DOCFORGE" --help -"$DOCFORGE_MCP" --help -``` - -## 2. Choose a project route - -Use a generic project when Markdown or TOML documentation is canonical. Use a fixed reference -adapter when you want a bounded source inventory and syntax-level graph for Python, JavaScript, -TypeScript, or C++. Use a project-owned adapter when the production contract must supply richer -semantics. - -- Generic manual: continue with [Create a generic project](#create-a-generic-project). -- Fixed source example: continue with [Use a reference adapter](#use-a-reference-adapter). -- Production language frontend: follow the [Adapter authoring guide](ADAPTER_AUTHORING_GUIDE.md). - -## Create a generic project - -Set the absolute project root and assess it without writing: - -```bash -PROJECT=/absolute/path/MyProject -"$DOCFORGE" --project-root "$PROJECT" onboard -``` - -The assessment reports detected languages, build evidence, documentation candidates, current -configuration, and available capabilities. Detection never invents a source graph. - -Create a generic starter explicitly: - -```bash -"$DOCFORGE" --project-root "$PROJECT" onboard \ - --scaffold \ - --project-id my-project \ - --title "My Project" -``` - -Scaffolding is create-only and refuses existing target files. It creates: - -- `.docforge/project.toml`, the generic project descriptor; -- `docs/docforge/content/architecture-overview.md`, one canonical node; -- `.docforge/templates/manual.html`, one built-in-renderer template; -- derived index, receipt, and rendered output below `.docforge`. - -If a language was detected, source graph status remains `adapter_required`. The generic starter -does not claim source semantics. - -Validate and inspect it: - -```bash -"$DOCFORGE" --project-root "$PROJECT" validate -"$DOCFORGE" --project-root "$PROJECT" reindex -"$DOCFORGE" --project-root "$PROJECT" search architecture -"$DOCFORGE" --project-root "$PROJECT" show architecture.overview -"$DOCFORGE" --project-root "$PROJECT" render-status -``` - -The descriptor is explained field by field in [Project descriptor](PROJECT_DESCRIPTOR.md). Add -canonical nodes only after choosing their authority, stable IDs, families, statuses, and allowed -relationships; [Core concepts and authority](CORE_CONCEPTS_AND_AUTHORITY.md) defines those terms. - -### Start a generic MCP binding - -Start read-only first: - -```bash -"$DOCFORGE_MCP" \ - --project-root "$PROJECT" \ - --capability-mode read -``` - -Call `docforge_bootstrap` before other tools. It reports the exact project identity, generation, -effective policy, projection policy, available capabilities, and recommended first read. - -To enable proposals, the descriptor must declare the writer and the process must select it: - -```bash -"$DOCFORGE_MCP" \ - --project-root "$PROJECT" \ - --capability-mode proposal \ - --proposal-writer project-editor -``` - -Canonical application is a separate startup gate. Do not add it to a read-only client: - -```bash -"$DOCFORGE_MCP" \ - --project-root "$PROJECT" \ - --capability-mode application \ - --proposal-writer project-editor \ - --canonical-applier project-editor -``` - -Application accepts one exact reviewed changeset hash. It does not commit, push, build, deploy, or -publish the project. - -### Generate a generic client fragment - -Preview is side-effect free: - -```bash -"$DOCFORGE" configure codex --project "$PROJECT" -"$DOCFORGE" configure claude --project "$PROJECT" -"$DOCFORGE" configure openclaw --project "$PROJECT" -``` - -Add `--output /absolute/path/new-fragment` to create one new private standalone file. Generation -does not merge with or replace a different existing file. Diagnose an installed binding with: - -```bash -"$DOCFORGE" doctor --client codex --project "$PROJECT" -``` - -Doctor is a bounded configuration inspector, not a connection test. See [Agent -integration](AGENT_INTEGRATION.md) for client-specific layouts and limitations. - -## Use a reference adapter - -Reference adapters read one fixed descriptor: -`.docforge/reference-adapter.toml`. The path is not selectable. - -Create a Python project configuration: - -```toml reference-adapter -schema_version = 1 -project_id = "my-python-project" -title = "My Python Project" -language = "python" -source_roots = ["src"] -``` - -JavaScript uses `language = "javascript"` and the `javascript` extra. TypeScript uses -`language = "typescript"` and the `typescript` extra. - -C++ additionally requires a confined compilation database: - -```toml reference-adapter -schema_version = 1 -project_id = "my-cpp-project" -title = "My C++ Project" -language = "cpp" -source_roots = ["src", "include"] -compilation_database = "compile_commands.json" -``` - -The C++ integration reads `compile_commands.json` only as bounded translation-unit inventory and -fingerprint evidence. It never executes a recorded command or compiler. - -Run the fixed read-only server through the same installed Python interpreter: - -```bash -"$DOCFORGE_PYTHON" -I -m docforge.reference_mcp \ - --project-root "$PROJECT" \ - --capability-mode read -``` - -The binding chooses one in-package provider from the descriptor language. It accepts no provider -module, command, argument list, working directory, environment, discovery rule, proposal writer, -or canonical applier. Its MCP surface is the read subset documented in the [generated command -reference](COMMAND_REFERENCE.md). - -### Generate a reference-adapter client fragment - -Generic `docforge configure` intentionally refuses custom adapters. Construct the adapter project -and immutable launcher, then call the custom-adapter generator: - -```python -from pathlib import Path - -from docforge.adapter_launcher import AdapterLauncherV1 -from docforge.client_config import generate_adapter_client_configuration -from docforge.reference_mcp import REFERENCE_MCP_MODULE, create_reference_project - -root = Path("/absolute/path/MyProject").resolve(strict=True) -project = create_reference_project(root) -launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE) - -preview = generate_adapter_client_configuration( - project, - launcher, - "codex", - capability_mode="read", -) -print(preview["artifact"]["content"]) -``` - -Use `client="claude"` or `client="openclaw"` for those formats. Pass an absolute `output` path only -when creating a new standalone private fragment. The generated launch is bound to the selected -project, descriptor hash, adapter identity, installed module, isolated interpreter, policy, and -source availability evidence. - -Read [Reference adapters](REFERENCE_ADAPTERS.md) before relying on the graph. The Python example -publishes local imports, the JavaScript and TypeScript examples publish project-local static -relative imports and re-exports, and the C++ example publishes directly resolvable project-local -quoted includes. None is a complete semantic compiler frontend. - -## 3. Add visualization only when needed - -Install the per-user viewer manager once: - -```bash -/absolute/path/DocForge/.venv/bin/docforge-viewer-manager install-user-service -``` - -Then start a project-bound snapshot: - -```bash -"$DOCFORGE" --project-root "$PROJECT" visualize -"$DOCFORGE" --project-root "$PROJECT" visualization-status -``` - -The listener is loopback-only and tokenized. The viewer is a derived snapshot, not canonical -authority and not a continuously monitored filesystem view. Read [Rendering and -visualization](RENDERING_AND_VISUALIZATION.md) for manual, portable, and live-viewer differences. - -## 4. Verify the maintained checkout - -Contributors can run: - -```bash -make command-reference-check -make docs-check -make adoption-m4 -make benchmark-m4-smoke -``` - -Run `make gate` before a release candidate. `benchmark-m4-smoke` is routine coverage; -`benchmark-m4` is the maintained full adapter workload. - -Continue with [Project onboarding](PROJECT_ONBOARDING.md) for production integration, -[Policy precedence](POLICY_PRECEDENCE.md) before widening a binding, and [Security](SECURITY.md) -before exposing any MCP process. +This file remains only so existing bookmarks and links continue to resolve. diff --git a/docs/POLICY_PRECEDENCE.md b/docs/POLICY_PRECEDENCE.md deleted file mode 100644 index 2b275b1..0000000 --- a/docs/POLICY_PRECEDENCE.md +++ /dev/null @@ -1,202 +0,0 @@ -# Policy precedence - -DocForge composes an immutable effective policy for each project-bound process. It resolves -restrictions in a fixed order: - -```text -core safety - > explicit binding - > no-AST shorthand - > resource availability -``` - -A lower layer can make a requested operation unavailable; it cannot override a higher-layer -prohibition. Bootstrap and generated client evidence report the composed result and hashes so a -client does not need to infer policy from command-line arguments. - -## 1. Core safety - -Core safety is unconditional. No capability mode exposes: - -- arbitrary file access or project switching; -- arbitrary renderer, module, command, shell, argument, working-directory, or environment - selection; -- Git mutation; -- project builds or compiler execution; -- deployment or publication. - -Canonical application is limited to DocForge's validated serializer boundary. Adapter launch is -limited to the immutable launcher contract. Rendering is limited to declared views and fixed -built-in workers. An indexed instruction cannot alter any of these rules. - -## 2. Explicit binding - -A process is bound at startup to one project root, descriptor, adapter, capability mode, writer and -applier identities when present, no-AST selection, diagnostics selection, and projection modes. -The binding does not change during the process lifetime. - -Capability modes are: - -- `read`: register only the read surface; -- `proposal`: add proposal tools when a valid selected writer is available; -- `application`: require a startup-bound canonical applier and expose exact-hash application; -- `operator`: reserved; it currently adds no tools. - -Mode describes the maximum registered surface. Actual authority can be narrower. A descriptor must -declare the selected writer, including allowed families and operation types. Application requires -the matching configured writer, changeset creator, and canonical-applier identity. A mode name -cannot create a missing descriptor grant. - -Generic generated client fragments default to read mode. Other construction paths preserve their -documented compatible factory defaults. Treat `docforge_bootstrap.session_contract` and its actual -capabilities as authoritative for a running server. - -## 3. No-AST shorthand - -`--no-ast` is a restrictive compatibility shorthand. It composes: - -- adapter evolution `preserve`; -- AST analysis `forbidden`; -- Logic indexing `off`; -- `docforge_get_logic` blocked; -- prohibitions on AST, Tree-sitter, compiler-AST, and function-Logic upgrades. - -Non-AST source fingerprinting and incremental caching remain allowed. Existing one-method adapters -continue to use `load_projection()`. - -The shorthand does not inspect how an existing adapter was implemented, sandbox its filesystem -reads, or transform an AST-based adapter into a no-AST adapter. Do not run the Python, -JavaScript/TypeScript, or C++ syntax reference integrations and then describe the binding as a -proved no-AST integration. See [Legacy and no-AST operation](LEGACY_AND_NO_AST.md). - -## 4. Resource availability - -Even an allowed policy cannot create a missing resource: - -- application mode requires a startup-bound canonical applier; -- manual `explicit` or `auto` requires declared manual render configuration; -- manual `auto` also requires canonical application in the current operation or server; -- portable graph `explicit` requires declared portable graph configuration; -- live viewer `on-demand` requires the viewer runtime; -- an optional reference language requires its installed extra; -- a project-owned launcher module must be installed, isolated, project-owned, and unchanged. - -Unavailable requested modes fail before hidden work. Missing optional language grammars return an -actionable install target such as `docforge[typescript]` or `docforge[cpp]`; DocForge does not -silently downgrade to a different frontend. - -## Effective process policy - -The version-1 effective policy reports: - -- capability mode and whether it came from a factory default or explicit selection; -- adapter evolution, AST analysis, and Logic indexing; -- automatic synchronization and validated integrity; -- the compatible manual-render projection; -- profiling state; -- blocked tools and prohibitions; -- the exact precedence list. - -This version-1 projection preserves compatibility. It is not the complete version-2 rendering -policy; portable graph and live-viewer choices are reported separately. - -## Independent projection policy - -Manual rendering, portable graph publication, and live visualization use a separate immutable -version-2 policy: - -```text -manual: auto | explicit | disabled -portable_graph: explicit | disabled -live_viewer: on-demand | disabled -``` - -When a selector is omitted, composition uses availability-aware defaults: - -- manual is `auto` only when manual configuration and canonical application are both available; - otherwise it is `explicit` when configured, or `disabled`; -- portable graph is `explicit` when configured, otherwise `disabled`; -- live viewer is `on-demand` when the runtime is available, otherwise `disabled`. - -An explicit non-disabled selection for a missing resource returns -`projection_policy_unavailable`. An active operation prohibited by the selected policy returns -`projection_policy_forbids_operation` before planning, rendering, or viewer startup. - -Status remains intentionally narrower than active work. Manual and portable receipt-only status -are available when their active operations are disabled. Viewer status and explicit stop remain -available when viewer start is disabled. - -Ordinary standalone CLI rendering uses manual `explicit`. Manual `auto` belongs to a canonical -application operation that owns automatic regeneration. - -## Diagnostics do not grant authority - -`--diagnostics` enables bounded request-local stage timings and compiler-work counters. It does not -enable tools, broaden paths, retain project content, or displace a primary MCP result that already -needs the response budget. - -## Descriptor policy and process policy - -The project descriptor is canonical project configuration. The process policy is a runtime -restriction. They compose by intersection: - -```text -operation is available - only if core permits it - and the startup binding registers it - and no-AST permits it - and required resources exist - and the descriptor grants the requested project authority - and current graph/hash preconditions validate -``` - -Changing a descriptor does not retarget a running project-owned process. Descriptor, adapter -implementation, or launcher drift requires a fresh process. - -## Common decisions - -For an agent that only reads documentation: - -```bash -docforge-mcp \ - --project-root /absolute/path/MyProject \ - --capability-mode read -``` - -For an agent that may prepare reviewable proposals: - -```bash -docforge-mcp \ - --project-root /absolute/path/MyProject \ - --capability-mode proposal \ - --proposal-writer project-editor -``` - -For a tightly controlled application process: - -```bash -docforge-mcp \ - --project-root /absolute/path/MyProject \ - --capability-mode application \ - --proposal-writer project-editor \ - --canonical-applier project-editor -``` - -For a read binding with every active output projection disabled: - -```bash -docforge-mcp \ - --project-root /absolute/path/MyProject \ - --capability-mode read \ - --manual-render-policy disabled \ - --portable-graph-policy disabled \ - --live-viewer-policy disabled -``` - -Prefer the narrowest binding that completes the workflow. Call `docforge_bootstrap` first and use -the returned effective policy, projection policy, actual capabilities, and prohibitions rather -than assumptions based on client configuration. - -See the [MCP contract](MCP_CONTRACT.md), [Security](SECURITY.md), [Project -descriptor](PROJECT_DESCRIPTOR.md), and [Rendering and -visualization](RENDERING_AND_VISUALIZATION.md). diff --git a/docs/PROJECT_DESCRIPTOR.md b/docs/PROJECT_DESCRIPTOR.md deleted file mode 100644 index 6e1be88..0000000 --- a/docs/PROJECT_DESCRIPTOR.md +++ /dev/null @@ -1,279 +0,0 @@ -# Project descriptor - -A generic DocForge project is selected by one fixed file: -`.docforge/project.toml` beneath an explicit project root. The descriptor is schema version 1. -DocForge validates both the JSON-schema shape in `schemas/project.schema.json` and runtime -invariants that schema alone cannot prove. - -Reference integrations use a different fixed descriptor, -`.docforge/reference-adapter.toml`; see [Reference adapters](REFERENCE_ADAPTERS.md). -Project-owned adapters construct the same runtime `ProjectDescriptor` contract through the public -adapter SDK. - -## Complete generic example - -```toml -schema_version = 1 -project_id = "my-project" -title = "My Project" -adapter = "generic" - -[sources] -content_roots = ["docs/docforge/content"] -authority_files = [] - -[derived] -cache_root = ".docforge/cache" -index = ".docforge/cache/index.sqlite3" - -[changesets] -root = ".docforge/changesets" - -[[changesets.writers]] -id = "project-editor" -families = ["architecture", "operations", "system"] -operations = ["create", "update", "move", "delete"] - -[render] -template_root = ".docforge/templates" -preview_root = ".docforge/previews" - -[[render.views]] -id = "manual" -renderer = "generic_html" -template = "manual.html" -output = ".docforge/rendered/manual.html" -title = "My Project Manual" -families = ["architecture", "operations", "system"] - -[graph_render] -output_root = ".docforge/portable-graph" - -[[graph_render.views]] -id = "architecture" -renderer = "portable_graph_html" -output = "architecture.html" -title = "Architecture" -root = "architecture.overview" -initial_mode = "web" -depth = 3 -max_nodes = 250 -max_edges = 1000 -max_work = 100000 -families = ["architecture", "system"] -relations = ["depends_on", "owns", "calls", "reads", "writes", "tested_by", "relates_to"] -authorities = [] -statuses = ["current", "active", "verified"] -tags = [] -include_logic = false - -[graph] -allowed_relations = [ - "calls", - "depends_on", - "owns", - "reads", - "relates_to", - "tested_by", - "writes", -] - -[limits] -max_source_bytes = 500000 -max_nodes = 10000 -max_query_chars = 500 -max_results = 100 -max_traversal_depth = 6 -max_context_tokens = 12000 -max_tool_output_chars = 200000 -max_changesets = 100 -max_changeset_operations = 100 -max_changeset_bytes = 1000000 -max_render_views = 20 -max_template_bytes = 1000000 -max_render_bytes = 1000000 - -[[profiles]] -id = "development" -families = ["architecture", "operations", "system"] -statuses = ["current", "active", "verified"] -required_nodes = ["architecture.overview"] -token_budget = 8000 -dependency_depth = 3 -``` - -Rendering sections are optional. Profiles and limits may also be omitted; runtime defaults then -apply. The required top-level fields are `schema_version`, `project_id`, `title`, `adapter`, -`sources`, `derived`, `changesets`, and `graph`. - -## Identity fields - -`schema_version` must be `1`. - -`project_id` is the stable machine identity. It is lowercase and may contain digits, dots, -underscores, and hyphens after its first character. Do not derive it from a mutable display title. - -`title` is the human-readable project name. - -`adapter` is `generic` for this file format. Project-owned adapters publish a validated -`adapter_id@adapter_version` identity through their runtime descriptor; changing adapter identity -invalidates incompatible derived state. - -The descriptor's byte content contributes to a descriptor hash. Client fragments, launchers, index -evidence, and policy results use that hash to detect drift. - -## Canonical sources - -`sources.content_roots` lists the project-relative directories containing generic Markdown and TOML -nodes. Each path must resolve beneath the project root. Canonical content roots may not overlap the -derived cache. - -`sources.authority_files` lists additional project-relative regular files whose content belongs to -the canonical project identity. They are not automatically parsed as nodes. - -A Markdown node contains one TOML metadata block followed by Markdown content: - -```markdown -+++ -schema_version = 1 -id = "architecture.overview" -title = "Architecture overview" -family = "architecture" -authority = "authoritative" -status = "current" -tags = ["architecture"] -summary = "Defines the top-level architecture and ownership." -+++ - -# Architecture overview - -Describe systems, ownership, runtime flow, failure behavior, and proof. -``` - -Every node ID is project-wide and stable. Every relationship target must resolve. A TOML source may -contain multiple `[[nodes]]` records; proposal-enabled multi-node files need stable -`source_anchor` values where creation or movement requires an exact record boundary. - -## Derived state - -`derived.cache_root` owns disposable indexes, attestations, extraction caches, render receipts, -projection artifacts, and viewer registry state. - -`derived.index` must be inside `derived.cache_root`. Canonical content and cache paths must not -overlap. - -Derived state is not a backup. If it is deleted or rejected as corrupt, DocForge rebuilds it from -validated canonical sources. - -## Changesets and writers - -`changesets.root` is the confined proposal store. It must not overlap canonical content or the -derived cache. - -Each `changesets.writers` entry declares: - -- a stable writer `id`; -- the node `families` that writer may change; -- allowed `operations`: `create`, `update`, `move`, and/or `delete`. - -The descriptor grant is necessary but not sufficient. A process must also select that writer at -startup, and canonical application requires a separately bound matching applier. Capability mode -does not broaden the descriptor grant. See [Policy precedence](POLICY_PRECEDENCE.md). - -## Allowed relationships - -`graph.allowed_relations` is the exact project vocabulary accepted on edges. It must be nonempty. -Relationship names are stable IDs. DocForge rejects relationships that are not declared and gives -`depends_on` additional cycle validation. - -The core does not reinterpret a custom relationship just because its spelling resembles a known -term. Task-context retrieval classifies only the documented versioned aliases and preserves -unknown allowed relationships as `unclassified`. - -## Context profiles - -Each `profiles` entry defines one bounded context compilation: - -- `id` selects the profile; -- `families` and `statuses` filter eligible nodes; -- `required_nodes` names stable nodes that must be present; -- `token_budget` limits compiled content; -- `dependency_depth` bounds relationship expansion. - -Profiles choose derived retrieval scope. They do not change node authority or writer permissions. - -## Limits - -Positive limits bound input, graph, retrieval, proposal, and rendering work. Current defaults are: - -- `max_source_bytes = 1000000` -- `max_nodes = 10000` -- `max_query_chars = 500` -- `max_results = 100` -- `max_traversal_depth = 8` -- `max_context_tokens = 32000` -- `max_tool_output_chars = 200000` -- `max_changesets = 1000` -- `max_changeset_operations = 100` -- `max_changeset_bytes = 1000000` -- `max_render_views = 100` -- `max_template_bytes = 1000000` -- `max_render_bytes = 1000000` - -Smaller project limits are useful policy. They cannot widen fixed internal worker, package, -response, or cache ceilings. In particular, detached renderer transfer has its own fixed boundary -even if a compatibility descriptor retains a larger `max_render_bytes`. - -## Manual rendering - -The optional `render` section declares: - -- one confined `template_root`; -- one isolated `preview_root`; -- one or more stable views. - -Each view uses the built-in `generic_html` renderer, a template beneath `template_root`, one -declared output path, a title, and a family filter. Paths may not overlap canonical content, -authority files, changesets, cache, templates, or previews in unsafe ways. - -Templates are inert UTF-8 files with a fixed token vocabulary. They cannot select executable -renderers or commands. See [Rendering and visualization](RENDERING_AND_VISUALIZATION.md). - -## Portable graph rendering - -The optional `graph_render` section declares an output root and one or more -`portable_graph_html` views. Each view selects exactly one: - -- `root`, an exact stable node ID; or -- `query`, a bounded metadata-only lexical seed. - -It may then restrict families, relations, authorities, statuses, and tags, plus depth, node, edge, -and work limits. `initial_mode` is `nodes`, `flow`, or `web`. Portable graph contract version 1 -requires `include_logic = false`. - -The declared `output` is relative to `graph_render.output_root`. - -## Paths and confinement - -Descriptor paths are project-relative. Absolute paths and parent traversal are rejected. Runtime -validation also rejects symlink escapes, unexpected file types, unsafe overlap, changing path -identity during sensitive reads or publication, and derived outputs outside their declared roots. - -The explicit CLI `--project-root` is the only project selector. The MCP process binds it at startup -and exposes no project-switching tool. - -## Validate changes safely - -After editing the descriptor: - -```bash -docforge --project-root /absolute/path/MyProject validate -docforge --project-root /absolute/path/MyProject reindex -docforge --project-root /absolute/path/MyProject check -``` - -Descriptor or adapter implementation drift makes a project-owned running process fail closed; start -a fresh process after changing those boundaries. - -For first-time creation, prefer the create-only [New-project -quickstart](NEW_PROJECT_QUICKSTART.md). For full invariants, read the [Core contract](CONTRACT.md). diff --git a/docs/PROJECT_ONBOARDING.md b/docs/PROJECT_ONBOARDING.md index a16efea..90a75ef 100644 --- a/docs/PROJECT_ONBOARDING.md +++ b/docs/PROJECT_ONBOARDING.md @@ -5,11 +5,6 @@ DocForge onboarding has two separate outcomes: 1. A generic manual can be configured, indexed, rendered, visualized, and exposed through the MCP. 2. A source graph additionally requires one validated language frontend per source language. -DocForge ships narrow fixed references for Python, JavaScript, TypeScript, and C++. They are useful -for syntax-scoped projects and adoption proof, but production semantic requirements may still -require a project-owned compiler or language-service adapter. Review -[Reference Adapters](REFERENCE_ADAPTERS.md) before selecting a frontend. - The onboarding command never claims that source semantics exist merely because it found source files. It reports each detected language as `adapter_required` until a project integration supplies and proves that frontend. @@ -69,32 +64,6 @@ immediately usable through the generic CLI, viewer, and MCP. The starter overview records detected languages and states that the source graph is unavailable until a language frontend passes the adapter proof. That limitation is deliberate. -## Configure a fixed reference adapter - -The generic onboarding scaffold and fixed reference configuration are separate project routes. -For a syntax-scoped reference project, create `.docforge/reference-adapter.toml`: - -```toml -schema_version = 1 -project_id = "my-python-project" -title = "My Python project" -language = "python" -source_roots = ["src"] -``` - -Then start the fixed read-only binding: - -```bash -python -I -m docforge.reference_mcp \ - --project-root /absolute/path/MyProject \ - --capability-mode read -``` - -The configuration selects only a fixed in-repository provider and cannot name a command or custom -module. JavaScript, TypeScript, and C++ require their respective optional extras; C++ also requires -`compilation_database = "compile_commands.json"`. The complete configuration and supported-fact -contract are in [Reference Adapters](REFERENCE_ADAPTERS.md). - ## Complete onboarding checklist ### 1. Repository assessment @@ -149,13 +118,6 @@ For every source language: - [ ] Define dependency discovery. - [ ] State unsupported semantic facts explicitly. -Decide whether the project needs production semantic evidence or the narrower syntax-only -reference scope. The Python reference publishes only project-local imports. JavaScript and -TypeScript publish only project-local static relative imports and re-exports. The C++ reference -publishes only directly resolvable project-local quoted includes and does not run a compiler. None -of those references resolves calls, inheritance, types, symbols, runtime behavior, or semantic -ownership. - All frontends emit the same DocForge contracts: - `AdapterManifest` inventories fingerprinted extraction units and dependencies. @@ -189,11 +151,6 @@ this checklist summarizes. - [ ] Assign shared symbols to one deterministic source contribution. - [ ] Record compiler-derived project include dependencies. -These are production semantic-adapter expectations. The built-in C++ reference uses -`compile_commands.json` only as bounded translation-unit inventory and fingerprint evidence. It -parses commands and arguments as inert data, executes no command or compiler, and does not claim -compiler include semantics, symbol ownership, or a Clang-derived graph. - #### Rust - [ ] Read the Cargo workspace and package graph. @@ -283,10 +240,6 @@ from filenames. Done when a new session can identify and retrieve the correct project without being told its file layout. -The fixed reference server is read-only. Generic and custom-adapter client generation, including -the immutable custom launcher boundary, is documented in -[Agent Integration](AGENT_INTEGRATION.md). - ### 10. Operating guide and maintenance - [ ] Record the authority and progressive-reading order. @@ -333,18 +286,14 @@ to one configured project root. DocForge does not let an MCP call install dependencies, run project builds, modify Git, deploy, or publish. A project integration may use its own normal development workflow for those actions. -## Reference and production frontend boundaries +## Frontend packaging direction -The public authoring namespace is `docforge.adapter_sdk`. Project-owned and separately distributed -production frontends should build on that contract without putting language-specific rules into -the graph, index, viewer, or MCP core. +Reusable language frontends should be separate packages or project-owned adapters over the public +DocForge contracts. They must not put language-specific rules into the graph, index, viewer, or MCP +core. -The in-repository Python, JavaScript, TypeScript, and C++ adapters are reference implementations. -Their syntax-scoped behavior is useful without becoming a claim that every project in those -languages has complete semantic coverage. A compiler-backed production C++ adapter may resolve -build flags, calls, types, inheritance, include semantics, and ownership when it can prove those -facts. The reference C++ adapter does none of that and is not a Clang semantic adapter. - -New Rust, Java, or other frontends should use their authoritative build and language tooling and -must pass the graph-plus-Logic complete/incremental contract in the -[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md). +Worldforge is the first complete C++ reference integration. A reusable C++ package should be +extracted only after that integration proves stable ownership, compiler dependency invalidation, +and complete/incremental equivalence. Rust and Java frontends should then implement the same +contract using their authoritative build and language tooling rather than copying C++ extraction +rules. diff --git a/docs/RECOVERY_AND_PERFORMANCE.md b/docs/RECOVERY_AND_PERFORMANCE.md deleted file mode 100644 index 62f6499..0000000 --- a/docs/RECOVERY_AND_PERFORMANCE.md +++ /dev/null @@ -1,86 +0,0 @@ -# Recovery and performance - -DocForge keeps canonical project evidence separate from disposable indexes, caches, receipts, and -rendered projections. Recovery rebuilds derived state from current authority; it does not rewrite -canonical content to make a cache look valid. - -For the underlying boundaries, see [core authority](CORE_CONCEPTS_AND_AUTHORITY.md) and the -[security model](SECURITY.md). - -## Recovery order - -Use the narrowest verified recovery: - -1. Run `docforge check` or the corresponding MCP status operation. -2. Run `docforge sync` to repair a missing or stale disposable index when safe. -3. Run `docforge reindex` for an explicit complete rebuild. -4. Restart a long-running MCP or viewer binding after descriptor or adapter implementation drift. -5. Regenerate a derived client fragment, command reference, preview, manual, or portable graph - from current project evidence. - -Never delete or rewrite canonical sources, active changesets, or reviewed hashes as cache cleanup. - -## Extraction-cache recovery - -Incremental adapter contributions are stored in a version-1 disposable extraction cache. Reads are -no-follow, regular-file-only, identity-checked, bounded to 64,000,000 bytes and 10,000 source -records, and fail to a cache miss on malformed or incompatible data. Publication is atomic. - -After a miss, current manifest sources are extracted again and the complete assembly is validated. -The active SQLite generation remains authoritative for reads until a new verified index is -published. An extraction cache may therefore be safely ahead of the last index; the two files do -not pretend to be one transaction. - -## Index and receipt recovery - -A validated SQLite index is a generation-pinned derived snapshot. Missing, corrupt, unattested, or -stale indexes rebuild from the complete project or adapter oracle. Generation receipts and render -receipts are post-publication evidence. Failure to write a receipt after a committed artifact is -reported as degraded committed success, not as permission to repeat a mutation. - -The live viewer pins one validated index identity. Index replacement makes the running snapshot -stale and causes a later visualize request to start a fresh worker. - -## Proposal and application recovery - -Hash or base conflicts are not cache failures. Retrieve the current changeset and diff, then -review the new exact hash. Rebase is allowed only when every touched node, relationship, source, -permission, and graph invariant still matches. A content conflict requires a new proposal. - -If a canonical serializer fails its round-trip check, use its reported rollback state. Do not -reapply a changeset whose application may already have committed. See -[migrating from v1](MIGRATING_FROM_V1.md) for rollback planning. - -## Milestone 4 scale evidence - -The maintained Python reference benchmark creates 334 Python source files and produces: - -- 1,002 primary nodes and 1,001 primary edges. -- 334 Logic projections with 2,338 Logic nodes and 2,338 Logic edges. -- Exact complete/incremental primary graph and Logic equality. -- Exact output after corrupt extraction-cache recovery and corrupt-index recovery. -- Zero `ast.parse` calls and zero `extract_source` calls during a warm build. - -On the frozen Milestone 4 candidate, operation p95 values were 1.266 to 1.898 seconds. Per-operation -traced peaks were about 61 to 67 MiB, and process high-water was 78,798,848 bytes. These are -regression measurements from one machine, not universal latency promises. The machine-readable -record is `../benchmarks/milestone4-2026-07-29.json`; detailed method and hashes are in -[the Milestone 4 baseline](MILESTONE_4_BASELINE.md). - -The zero-parser claim is deliberately Python-only. Focused JavaScript and TypeScript tests prove -their manifests avoid Tree-sitter. The C++ reference manifest currently uses Tree-sitter while -discovering bounded quoted includes, so a warm C++ cache hit is not evidence of zero parser work. - -## Maintained gates - -Run: - -```bash -make gate -make adoption-m4 -make benchmark-m4-full -make docs-check -``` - -The main gate includes smoke benchmarks. Full milestone evidence is recorded separately from a -clean candidate so smoke or dirty-tree results cannot become release claims. diff --git a/docs/REFERENCE_ADAPTERS.md b/docs/REFERENCE_ADAPTERS.md deleted file mode 100644 index 4287651..0000000 --- a/docs/REFERENCE_ADAPTERS.md +++ /dev/null @@ -1,149 +0,0 @@ -# Reference adapters - -DocForge includes fixed, syntax-scoped reference adapters for Python, JavaScript, TypeScript, and -C++. They demonstrate the public adapter contract, deterministic complete and incremental -publication, function Logic, cache recovery, and a runnable read-only MCP binding. They are not -compiler or language-service replacements. - -Use a reference adapter when its deliberately narrow graph is sufficient or when proving a fresh -DocForge integration. Use the [Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md) for a -production adapter that needs resolved symbols, calls, inheritance, types, build semantics, or -semantic ownership. - -## Installation - -The Python reference adapter uses the standard library and is available in the base wheel. The -other frontends are separate optional extras: - -```bash -python -m pip install docforge -python -m pip install "docforge[javascript]" -python -m pip install "docforge[typescript]" -python -m pip install "docforge[cpp]" -``` - -Install `docforge[languages]` only when one environment intentionally needs all three Tree-sitter -frontends. JavaScript and TypeScript use distinct grammar packages and distinct extras. - -## Fixed project configuration - -The runnable reference binding reads exactly -`.docforge/reference-adapter.toml` below the selected project root. The configuration is data only: -it cannot name a provider, Python module, command, arguments, working directory, environment, or -discovery rule. - -Reference adapter configuration `.docforge/reference-adapter.toml`: - -```toml -schema_version = 1 -project_id = "example-python" -title = "Example Python project" -language = "python" -source_roots = ["src"] -``` - -`language` is one of `python`, `javascript`, `typescript`, or `cpp`. Source roots must be sorted, -unique, non-overlapping project-relative directories outside `.docforge`. The configuration, -roots, and inventoried source files must be regular confined paths without symlink traversal. The -configuration is limited to 65,536 bytes, at most 64 source roots, and 65,536 examined inventory -entries. - -C++ additionally requires one explicit project-confined compilation database: - -```toml -schema_version = 1 -project_id = "example-cpp" -title = "Example C++ project" -language = "cpp" -source_roots = ["include", "src"] -compilation_database = "compile_commands.json" -``` - -`compilation_database` is forbidden for the other three languages. - -## Published scope - -All four adapters publish deterministic file, module or translation-unit, class or struct, and -function nodes where their syntax supports those categories. They publish lexical `contains` -relationships, a narrow set of project-local `depends_on` relationships, and function-scoped -Logic. Calls that appear inside Logic are syntax steps, not resolved symbol relationships. - -### Python - -Python uses `ast` from the standard library for source extraction and Logic. Manifest construction -does not build a Python AST; it tokenizes imports only far enough to publish dependencies that -resolve to another module in the declared source inventory. - -It does not import or execute project code. It does not resolve dynamic imports, calls, -inheritance, imported symbols, types, overloads, re-exports, decorators, metaclasses, descriptors, -or runtime-generated behavior. - -### JavaScript and TypeScript - -JavaScript uses the optional `tree-sitter-javascript` grammar. TypeScript uses the distinct -optional `tree-sitter-typescript` grammar. Their focused incremental tests prove that manifest -construction and an unchanged warm build do not invoke the Tree-sitter extraction parser. - -Only static relative imports and re-exports that resolve to another inventoried source file become -dependencies. Dynamic `import()`, `require()`, bare package specifiers, aliases, `tsconfig` paths, -loader hooks, types, interfaces, overloads, calls, inheritance, symbols, and runtime behavior are -not resolved. The adapters never import, compile, transpile, or execute project code. - -### C++ - -C++ treats `compile_commands.json` as the authoritative bounded translation-unit inventory and as -fingerprint evidence. Commands, arguments, directories, and output fields are parsed as inert -data. The adapter executes no compiler, build tool, command, project binary, or project code. - -The reference C++ manifest uses the optional `tree-sitter-cpp` grammar to parse inventoried sources -and discover quoted includes. It publishes a dependency only when that quoted include resolves -directly to a real project-local header in the declared roots. Angle-bracket includes, compiler -include paths, frameworks, generated headers, compiler-provided headers, conditional compilation, -and macro expansion are omitted. - -This is not a Clang semantic adapter. It does not claim resolved calls, types, templates, aliases, -concepts, references, inheritance, overload ownership, out-of-line semantic ownership, macro -semantics, or compiler include semantics. A warm C++ extraction-cache hit is therefore not evidence -that manifest construction performed zero parser work. - -## Complete and incremental proof - -Each reference adapter implements: - -- `load_projection()` for the complete primary graph compatibility oracle; -- `load_complete_assembly()` for the complete graph-plus-Logic oracle; -- `load_manifest()` and `extract_source()` for incremental extraction; and -- deterministic assembly of the complete current contribution set. - -The maintained fixtures prove complete determinism, exact complete/incremental graph and Logic -parity, reverse-dependency invalidation, additions and deletions, corrupt extraction-cache -recovery, confinement, and exact unsupported-fact inventories. See -[Incremental Adapter Indexing](INCREMENTAL_INDEXING.md) for the cache contract and -[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for compatibility and policy limits. - -Current fixture evidence is: - -| Language | Primary nodes | Relationships | Logic projections | -|---|---:|---:|---:| -| Python | 14 | 13 | 5 | -| JavaScript | 14 | 14 | 5 | -| TypeScript | 13 | 14 | 4 | -| C++ | 17 | 16 | 5 | - -These are regression-fixture shapes, not promises for arbitrary repositories. - -## Read-only reference server - -Start the fixed server with: - -```bash -python -I -m docforge.reference_mcp \ - --project-root /absolute/path/to/project \ - --capability-mode read -``` - -The server selects one of the four in-repository providers solely from the validated fixed -configuration. Its cache stays below `.docforge/cache/reference-adapter/`. It registers -the 21-tool read surface and no proposal or application tools. See -[Agent Integration](AGENT_INTEGRATION.md) for generated client fragments and -[MCP Boundary](MCP_CONTRACT.md) for the exact tool contract. diff --git a/docs/RENDERING_AND_VISUALIZATION.md b/docs/RENDERING_AND_VISUALIZATION.md deleted file mode 100644 index 90bbeb4..0000000 --- a/docs/RENDERING_AND_VISUALIZATION.md +++ /dev/null @@ -1,233 +0,0 @@ -# Rendering and visualization - -DocForge has three independent ways to present one validated graph generation: - -```text -validated generation - ├── manual plan → immutable package → detached HTML renderer → declared manual output - ├── graph plan → immutable package → detached graph renderer → portable static artifact - └── pinned index → managed loopback viewer → interactive Nodes, Flow, Web, and lazy Logic -``` - -They share validated facts but not authority, publication, or lifecycle. A manual render does not -publish a portable graph. A portable graph does not start the live viewer. None is canonical -project content. - -## Manual rendering - -A generic descriptor may declare a manual template root, isolated preview root, and stable views: - -```toml -[render] -template_root = ".docforge/templates" -preview_root = ".docforge/previews" - -[[render.views]] -id = "manual" -renderer = "generic_html" -template = "manual.html" -output = ".docforge/rendered/manual.html" -title = "My Project Manual" -families = ["architecture", "operations", "system"] -``` - -The renderer name is fixed to `generic_html`. Templates are confined UTF-8 assets with a fixed -token vocabulary. They cannot name commands, Python modules, executable renderers, or arbitrary -publication paths. Raw HTML in canonical Markdown is disabled by the pinned CommonMark path. - -Plan and render identity covers the canonical source hash, optional changeset hash, selected nodes -and relationships, view configuration, template hash, renderer contract, and parser version. - -Use: - -```bash -docforge --project-root "$PROJECT" render-status -docforge --project-root "$PROJECT" render-status manual --deep -docforge --project-root "$PROJECT" render manual -docforge --project-root "$PROJECT" preview CHANGESET_ID manual -``` - -Normal status verifies bounded source, configuration, template, output, renderer, and receipt -identities without reconstructing output. `--deep` explicitly runs the side-effect-free full-render -oracle. A changeset preview writes only to the isolated preview root. - -Manual `auto` means that a successful canonical application owns regeneration of every declared -manual view. It does not mean background rendering, and ordinary standalone CLI render cannot -select `auto`; use `explicit`. - -## Portable graph rendering - -Portable graph configuration is separate: - -```toml -[graph_render] -output_root = ".docforge/portable-graph" - -[[graph_render.views]] -id = "architecture" -renderer = "portable_graph_html" -output = "architecture.html" -title = "Architecture" -root = "architecture.overview" -initial_mode = "web" -depth = 3 -max_nodes = 250 -max_edges = 1000 -max_work = 100000 -families = ["architecture", "system"] -relations = ["depends_on", "owns", "calls", "reads", "writes", "tested_by", "relates_to"] -authorities = [] -statuses = ["current", "active", "verified"] -tags = [] -include_logic = false -``` - -A view selects exactly one exact `root` node or bounded metadata-only lexical `query`. Exact -filters and node, edge, depth, and work limits close the selection. The initial mode is `nodes`, -`flow`, or `web`. Portable graph contract version 1 excludes function-scoped Logic. - -Use: - -```bash -docforge --project-root "$PROJECT" graph-plan architecture -docforge --project-root "$PROJECT" graph-render architecture -docforge --project-root "$PROJECT" graph-render-status architecture -``` - -`graph-plan` validates and returns the generation-pinned plan without publication. `graph-render` -is an explicit local CLI publication action. It commits a content-addressed artifact, renderer -receipt, and bounded generation/view manifest in that order; the manifest is the publication -commit. `graph-render-status` checks only bounded committed evidence and never plans or renders. - -MCP may plan and inspect portable graph evidence, but it does not publish the portable artifact. -Use the explicit project-local CLI command for publication. - -Portable output is a complete static artifact. JavaScript is progressive enhancement, not a -requirement for the graph facts to be present. - -## Live viewer - -The live viewer is a managed, read-only browser over one generation-pinned validated SQLite file. -Install the per-user manager once: - -```bash -docforge-viewer-manager install-user-service -``` - -Then operate a project viewer: - -```bash -docforge --project-root "$PROJECT" visualize -docforge --project-root "$PROJECT" visualize --node architecture.overview -docforge --project-root "$PROJECT" visualize --query persistence -docforge --project-root "$PROJECT" visualization-status -docforge --project-root "$PROJECT" visualization-stop -``` - -`visualize` accepts only an optional stable node ID or lexical query, bounded traversal depth, and -the local `--no-open` presentation choice. It does not accept a project root override, database -path, SQL, template, command, renderer, bind address, or module. - -The HTTP listener binds to `127.0.0.1` on an operating-system-selected port. A random token is part -of every accepted path. The server supports only `GET` and `HEAD`, sets no-store and restrictive -browser security headers, and has no write, project-selection, arbitrary-query, or static -filesystem endpoint. - -The viewer is a snapshot. Index replacement or alteration makes that snapshot fail closed; start a -new visualization to use a new validated generation. Source inspection reads only project-confined -source evidence bound to the pinned generation. - -## Nodes, Flow, Web, and Logic - -The interactive viewer offers four complementary projections: - -- **Nodes** shows a bounded relation-neutral incoming and outgoing neighborhood. -- **Flow** presents semantic contributors toward the focus. Prerequisite-style stored - relationships may be reversed for presentation without changing stored direction. -- **Web** expands the convergence picture with contributors, callers, containers, members, and - contextual relationships. -- **Logic** loads a function-scoped control-flow projection only when requested. - -Logic supports the explicit control paths published by Python, JavaScript, TypeScript, and C++ -integrations. It stays outside the primary graph, portable graph version 1, search, and -generation-diff receipts. - -Hiding a node is presentation-only. In Flow and Web, ancestors without another path to the focus -are removed. In Logic, an omitted-path bridge preserves downstream readability. Restore reverses -the presentation change; neither action mutates the index. - -## Immutable plan and worker boundary - -Manual and graph plans are versioned, canonical JSON with deterministic ordering and fixed -structural and byte limits. They contain selected graph facts and bounded content, but no live -project object, SQLite handle, absolute project or index path, arbitrary query, command, -executable path, or caller-selected renderer module. - -A projection package binds one plan to inert assets, fixed component versions, a closed built-in -renderer identity, and an exact artifact inventory. The detached worker: - -- runs one fixed private module through isolated Python; -- uses a trusted working directory and sanitized environment; -- accepts one canonical newline-terminated JSON request; -- returns one bounded canonical JSON response; -- has a fixed renderer allowlist and timeout; -- cannot select graph facts, read project state, choose output paths, or mutate canonical files. - -The package contract is bounded, and actual artifact transfer has a fixed 20,000,000-byte ceiling. -A larger descriptor `max_render_bytes` compatibility value does not widen that worker boundary. - -## Manual fragment reuse - -Manual fragments are disposable semantic cache records, not publication authority. A cold record -is accepted only after byte-exact comparison against a full detached render. On a warm hit, the -worker independently recomputes the expected fragment before reuse. - -Corrupt, forged, stale, incompatible, individually oversized, or aggregate-oversized records fall -back to the complete full-render oracle. The current cache is bounded to 10,000 records and -64,000,000 bytes. - -Fragment reuse is a correctness and recovery boundary. Do not promise a speedup without current -measurements. - -## Projection policy - -The version-2 projection policy is independent of capability mode: - -```text -manual: auto | explicit | disabled -portable_graph: explicit | disabled -live_viewer: on-demand | disabled -``` - -Set global CLI flags before the subcommand: - -```bash -docforge --project-root "$PROJECT" --manual-render-policy disabled render manual -docforge --project-root "$PROJECT" --portable-graph-policy disabled graph-plan architecture -docforge --project-root "$PROJECT" --live-viewer-policy disabled visualize -``` - -An active operation blocked by policy fails before hidden work. Receipt-only manual and portable -status remain available. Viewer status and explicit stop remain available when viewer startup is -disabled. - -A non-disabled selection also requires its resource: declared manual configuration, declared -portable graph configuration, or viewer runtime. Manual `auto` additionally requires canonical -application. Read [Policy precedence](POLICY_PRECEDENCE.md) for exact composition. - -## Failure and recovery - -Render input drift detected before replacement fails without publishing a current receipt for stale -output. Portable graph publication uses content-addressed evidence and a final manifest commit. -Status never repairs implicitly. - -Recovery is explicit: - -- rerun a manual render from validated canonical state; -- use deep manual status for the full, side-effect-free equivalence oracle; -- rerun portable graph publication, or repair only from validated content-addressed evidence; -- stop and restart a stale live viewer against a current validated index. - -Do not recover a derived output by editing its receipt or treating it as canonical. See [Recovery -and performance](RECOVERY_AND_PERFORMANCE.md), [Security](SECURITY.md), and the [Viewer -manager](VIEWER_MANAGER.md). diff --git a/docs/SECURITY.md b/docs/SECURITY.md deleted file mode 100644 index 4c8569e..0000000 --- a/docs/SECURITY.md +++ /dev/null @@ -1,101 +0,0 @@ -# Security model - -DocForge is a project-bound knowledge compiler. Its security boundary is an explicit project root, -closed configuration, bounded data, and exact identities. It is not a general process sandbox. - -Start with [core authority](CORE_CONCEPTS_AND_AUTHORITY.md), then use -[policy precedence](POLICY_PRECEDENCE.md) to decide which capabilities a server should expose. - -## Project and path confinement - -Descriptors, reference-adapter configurations, canonical sources, authority files, templates, -changesets, caches, indexes, previews, and declared outputs are resolved against one project root. -DocForge rejects absolute paths where only project-relative paths are allowed, parent traversal, -symbolic-link escapes, unsafe file types, and protected-root overlap. Important reads use -no-follow file descriptors and compare file identity before and after reading. - -Confinement protects DocForge operations. It does not stop another process with repository access -from changing files. Long-running bindings revalidate descriptor and adapter implementation -identity and require a restart after drift. - -## Untrusted project content - -Documentation, source text, templates, adapter metadata, compiler-database entries, and changeset -content are data. They cannot redefine policy or instruct DocForge to execute a command. Generic -manual rendering disables raw HTML, accepts a fixed template vocabulary, and rejects script-like -content. Portable graph and manual workers accept validated inert packages and fixed built-in -renderer identities. - -The C++ reference adapter reads `compile_commands.json` only as bounded translation-unit inventory -and fingerprint evidence. It never executes the recorded command, compiler, response file, or -project program. - -## Adapter launcher boundary - -`AdapterLauncherV1` contains one Python module name and project identity. It contains no command, -shell string, arbitrary argument list, working directory, environment, discovery rule, or callable -selector. - -Custom modules must be installed top-level modules. An isolated `python -I` probe resolves the -module without importing it and requires its regular-file origin to remain inside the bound -project. The sole trusted dotted exception is the packaged `docforge.reference_mcp` module. Client -fragments use the exact validated interpreter and canonical fixed arguments with an empty -environment. - -This proves that the declared module is resolvable and project-bound. It does not make arbitrary -module code safe. Project owners remain responsible for the implementation they install. - -## Mutation boundary - -Normal MCP and the fixed reference server are read-only. Proposal tools exist only when a -startup-bound writer is authorized by the descriptor. Canonical application exists only when a -matching applier is explicitly configured. - -Every proposal append, rebase, abandonment, and application is hash-bound. Application requires -the exact changeset hash that was reviewed. Source identity, content hashes, permissions, -conflicts, graph validity, and serializer round trips are checked before success. DocForge never -turns prose approval into a fuzzy merge. - -## Derived state and publication - -SQLite indexes, source-generation receipts, extraction caches, render fragments, previews, and -portable artifacts are disposable. Corrupt, stale, foreign, oversized, or mismatched derived -state is rejected or rebuilt from current project evidence. - -Generated command-reference publication serializes cooperating writers and uses no-clobber or -compare-and-swap publication. A raced target is restored or retained for recovery instead of being -silently discarded. Projection publication records when an artifact was committed but later -receipt verification degraded, so a completed mutation is never reported as an ordinary failure. - -## Limits and denial-of-service resistance - -Inputs, results, traversal, context, changesets, renders, worker protocols, manifests, adapter -assemblies, and extraction caches have explicit count and byte limits. Incremental extraction -caches are capped at 10,000 sources and 64,000,000 bytes. Adapter primary nodes use the project -`max_nodes` limit; edges and Logic have deterministic multipliers over that limit. - -Limits reduce accidental and adversarial amplification. An in-process adapter can still allocate -memory before returning data, so only trusted project-owned adapter code should run in the server -process. - -## Secrets and network behavior - -DocForge does not copy the parent environment into generated client fragments or detached -projection workers. Doctor checks never return environment values. The live viewer binds to -loopback, uses an unguessable URL token, supports read-only methods, and serves no arbitrary -filesystem tree. - -Project secrets must not be placed in canonical documentation, adapter configuration, compiler -databases, templates, or changesets. Repository release gates include secret scanning, but that -scan is not a substitute for credential hygiene. - -## No-AST boundary - -`--no-ast` is a binding policy that preserves the selected adapter and prohibits Logic publication -and retrieval. It is not a parser detector, filesystem sandbox, or promise that unrelated -processes cannot parse source. See [legacy and no-AST operation](LEGACY_AND_NO_AST.md). - -## Reporting and recovery - -Do not bypass a confinement, identity, policy, hash, or limit error. Preserve the failing evidence, -stop the affected binding, and follow [recovery and performance](RECOVERY_AND_PERFORMANCE.md). diff --git a/docs/USER_MANUAL.md b/docs/USER_MANUAL.md index 21f3007..640f9ae 100644 --- a/docs/USER_MANUAL.md +++ b/docs/USER_MANUAL.md @@ -35,8 +35,6 @@ incremental methods while retaining the full loader as a fallback. relationship keys, source inspection, branch-aware node hiding, panel resizing, zooming, and managed idle shutdown. - A generic Markdown/TOML adapter plus contracts for deterministic project-owned adapters. -- A public adapter SDK, optional Python/JavaScript/TypeScript/C++ reference integrations, and one - fixed read-only reference MCP binding. DocForge does not run shell commands from documentation, mutate Git, build an application, deploy, publish, choose a project globally, or cross project boundaries. @@ -102,49 +100,6 @@ uv run pytest -q Use the executables under `/absolute/path/DocForge/.venv/bin/` when DocForge is not installed into the active shell environment. -For an installed distribution, choose only the language extras the project needs: - -```bash -python -m pip install docforge -python -m pip install 'docforge[javascript]' -python -m pip install 'docforge[typescript]' -python -m pip install 'docforge[cpp]' -``` - -The base wheel contains the Python reference adapter and no Tree-sitter distribution. JavaScript, -TypeScript, and C++ require their matching optional extras. `docforge[languages]` installs all -three optional frontend groups. - -### Configure a reference source project - -Reference adapters are a narrow alternative to the generic documentation descriptor. Create -`.docforge/reference-adapter.toml`: - -```toml reference-adapter -schema_version = 1 -project_id = "my-python-project" -title = "My Python Project" -language = "python" -source_roots = ["src"] -``` - -Start the fixed read-only server: - -```bash -python -I -m docforge.reference_mcp \ - --project-root /absolute/path/MyProject \ - --capability-mode read -``` - -For C++, set `language = "cpp"` and add a project-relative -`compilation_database = "compile_commands.json"`. The database is inert bounded inventory; the -reference adapter does not execute its commands or compiler. - -The reference integrations publish syntax and local static relationships only. They do not claim -resolved calls, types, inheritance, macro behavior, compiler include semantics, runtime behavior, -or semantic ownership. See [reference adapters](REFERENCE_ADAPTERS.md) for exact evidence and -limitations. - ### Assess and onboard an unconfigured project Run a read-only assessment before writing configuration: @@ -523,10 +478,6 @@ Every command emits deterministic JSON: docforge --project-root /absolute/path/MyProject ``` -The sections below group common workflows. The implementation-derived list of all 28 current -commands, exact invocations, 36 generic MCP tools, arguments, and input-schema hashes is the -[generated command reference](COMMAND_REFERENCE.md). - ### Project and index commands ```text @@ -639,31 +590,6 @@ declares the named writer, and application requires the same writer/applier iden `--no-ast` to preserve the no-AST binding. Generic CLI generation refuses project-owned adapters because it cannot safely reconstruct their composition. -A custom adapter owner supplies the already constructed project and immutable launcher through the -Python API: - -```python -from docforge.adapter_launcher import AdapterLauncherV1 -from docforge.client_config import generate_adapter_client_configuration - -launcher = AdapterLauncherV1.for_project( - project, - module="my_project_docforge", -) -fragment = generate_adapter_client_configuration( - project, - launcher, - "codex", - capability_mode="read", -) -``` - -The top-level module must be installed for the exact isolated Python environment and resolve to a -regular file inside the project root. The fixed `docforge.reference_mcp` module is the only trusted -dotted exception. Generation probes resolution without importing the custom module, binds current -source availability and policy, and emits no arbitrary command, arguments, working directory, or -environment. - Select projection behavior independently: ```bash @@ -758,10 +684,6 @@ docforge-mcp \ Omit `--proposal-writer` when the MCP client should not create or append proposals. -For `.docforge/reference-adapter.toml`, use the fixed `docforge.reference_mcp` command shown in -[setup](#configure-a-reference-source-project). It exposes exactly the 21 read tools and never -registers proposal or application tools. - Select the session's declared surface explicitly when useful: ```bash @@ -1064,28 +986,6 @@ changes outside this process boundary. ## Troubleshooting -### `optional_dependency_missing` - -Install the exact extra named in the error into the same Python environment that starts DocForge: - -```bash -python -m pip install 'docforge[javascript]' -python -m pip install 'docforge[typescript]' -python -m pip install 'docforge[cpp]' -``` - -Do not install every frontend merely to suppress the check. A missing optional parser is a closed, -actionable capability error and does not affect base generic or Python reference operation. - -### `adapter_launcher_unavailable` or `invalid_adapter_launcher` - -Use one installed top-level Python module whose resolved regular-file origin is inside the project -root, or use the fixed `docforge.reference_mcp` binding. Arbitrary dotted modules, packages, -stdlib modules, missing modules, commands, argument strings, working directories, and environment -injection are rejected. Test the exact generated fragment rather than editing its command by hand. - -See [agent integration](AGENT_INTEGRATION.md) and the [security model](SECURITY.md). - ### `adapter_restart_required` The project-local adapter code, its declared descriptor, or another implementation file changed @@ -1228,11 +1128,8 @@ make gate Use `make benchmark` for the historical Milestone 0 baseline, `make benchmark-m1` for the counter-gated warm-operation benchmark, `make benchmark-m2` for agent workflow gates, and `make benchmark-m3-full` for the ten-sample 1,000-node projection, worker, fragment, status, -equivalence, response-size, and memory gates. `make benchmark-m4-full` runs the 1,002-node adapter -and recovery benchmark. `make adoption-m4` performs the offline fresh-wheel proof. -`make command-reference-check` rejects command-reference drift, and `make docs-check` validates -the maintained documentation graph. `make accessibility` runs the generated manual, portable -graph, and live viewer axe and keyboard flows. +equivalence, response-size, and memory gates. `make accessibility` runs the generated manual, +portable graph, and live viewer axe and keyboard flows. Project-specific vocabulary, extraction rules, and serialization belong in the project adapter. Generic core behavior must remain deterministic, project-bound, and recoverable. diff --git a/pyproject.toml b/pyproject.toml index e3c16be..6024c3e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -13,26 +13,9 @@ authors = [{ name = "Worldforge contributors" }] dependencies = [ "markdown-it-py>=4.2,<5", "mcp>=1.28,<2", -] - -[project.optional-dependencies] -javascript = [ - "tree-sitter>=0.25,<0.26", - "tree-sitter-javascript>=0.25,<0.26", -] -typescript = [ - "tree-sitter>=0.25,<0.26", - "tree-sitter-typescript>=0.23,<0.24", -] -cpp = [ - "tree-sitter>=0.25,<0.26", - "tree-sitter-cpp>=0.23,<0.24", -] -languages = [ "tree-sitter>=0.25,<0.26", "tree-sitter-cpp>=0.23,<0.24", "tree-sitter-javascript>=0.25,<0.26", - "tree-sitter-typescript>=0.23,<0.24", ] [dependency-groups] @@ -40,10 +23,6 @@ dev = [ "jsonschema>=4.25,<5", "pytest>=9.1,<10", "ruff>=0.15,<1", - "tree-sitter>=0.25,<0.26", - "tree-sitter-cpp>=0.23,<0.24", - "tree-sitter-javascript>=0.25,<0.26", - "tree-sitter-typescript>=0.23,<0.24", ] [project.scripts] diff --git a/schemas/adapter-client-configuration.schema.json b/schemas/adapter-client-configuration.schema.json deleted file mode 100644 index e316c93..0000000 --- a/schemas/adapter-client-configuration.schema.json +++ /dev/null @@ -1,358 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://docforge.local/schema/adapter-client-configuration-v1.json", - "title": "DocForge project-owned adapter client configuration", - "$defs": { - "sha256": { - "type": "string", - "pattern": "^[0-9a-f]{64}$" - }, - "project": { - "type": "object", - "required": [ - "project_id", - "project_root", - "project_root_fingerprint", - "adapter", - "descriptor_hash" - ], - "properties": { - "project_id": { - "type": "string", - "pattern": "^[a-z0-9][a-z0-9._-]{1,127}$" - }, - "project_root": { - "type": "string", - "minLength": 1, - "maxLength": 4096 - }, - "project_root_fingerprint": { - "type": "string", - "pattern": "^[0-9a-f]{16}$" - }, - "adapter": { - "type": "string", - "pattern": "^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}@[A-Za-z0-9][A-Za-z0-9_.+-]{0,63}$" - }, - "descriptor_hash": { "$ref": "#/$defs/sha256" } - }, - "additionalProperties": false - }, - "source_availability": { - "type": "object", - "required": [ - "schema_version", - "status", - "method", - "revision", - "source_hash" - ], - "properties": { - "schema_version": { "const": 1 }, - "status": { "const": "available" }, - "method": { - "enum": ["incremental-state", "complete-projection"] - }, - "revision": { - "type": "string", - "minLength": 1, - "maxLength": 4096 - }, - "source_hash": { "$ref": "#/$defs/sha256" } - }, - "additionalProperties": false - }, - "binding": { - "type": "object", - "required": [ - "transport", - "capability_mode", - "adapter_policy", - "render_policy", - "command", - "args", - "environment", - "timeouts", - "launcher_hash" - ], - "properties": { - "transport": { "const": "stdio" }, - "capability_mode": { - "enum": ["read", "proposal", "application"] - }, - "adapter_policy": { - "$ref": "https://docforge.local/schema/client-configuration-v1.json#/$defs/adapter_policy" - }, - "render_policy": { - "type": "object", - "required": ["manual", "graph", "live_viewer"], - "properties": { - "manual": { "enum": ["auto", "explicit", "disabled"] }, - "graph": { "const": "disabled" }, - "live_viewer": { "const": "on-demand" } - }, - "additionalProperties": false - }, - "command": { - "type": "string", - "minLength": 1, - "maxLength": 4096 - }, - "args": { - "type": "array", - "minItems": 7, - "maxItems": 64, - "prefixItems": [ - { "const": "-I" }, - { "const": "-m" }, - { - "type": "string", - "minLength": 1, - "maxLength": 255, - "pattern": "^[A-Za-z_][A-Za-z0-9_]*(\\.[A-Za-z_][A-Za-z0-9_]*){0,31}$", - "not": { "const": "docforge.mcp_server" } - }, - { "const": "--project-root" }, - { - "type": "string", - "minLength": 1, - "maxLength": 4096 - }, - { "const": "--capability-mode" }, - { "enum": ["read", "proposal", "application"] } - ], - "items": { - "type": "string", - "minLength": 1, - "maxLength": 4096 - } - }, - "environment": { - "type": "object", - "maxProperties": 0 - }, - "timeouts": { - "type": "object", - "required": ["startup_seconds", "tool_seconds"], - "properties": { - "startup_seconds": { - "type": "integer", - "minimum": 1, - "maximum": 3600 - }, - "tool_seconds": { - "type": "integer", - "minimum": 1, - "maximum": 86400 - } - }, - "additionalProperties": false - }, - "launcher_hash": { "$ref": "#/$defs/sha256" } - }, - "additionalProperties": false - }, - "projection_availability": { - "type": "object", - "required": [ - "manual_configured", - "portable_graph_configured", - "application_enabled", - "live_viewer_available" - ], - "properties": { - "manual_configured": { "type": "boolean" }, - "portable_graph_configured": { "type": "boolean" }, - "application_enabled": { "type": "boolean" }, - "live_viewer_available": { "const": true } - }, - "additionalProperties": false - } - }, - "type": "object", - "required": [ - "status", - "schema_version", - "operation", - "action", - "client", - "server_name", - "project", - "launcher", - "launcher_hash", - "source_availability", - "source_availability_hash", - "binding", - "effective_policy", - "projection_policy", - "projection_policy_hash", - "projection_availability", - "artifact", - "configuration_hash", - "warnings" - ], - "properties": { - "status": { "const": "ok" }, - "schema_version": { "const": 1 }, - "operation": { "const": "adapter_client.configure" }, - "action": { "enum": ["preview", "write"] }, - "client": { "enum": ["codex", "claude", "openclaw"] }, - "server_name": { - "type": "string", - "pattern": "^[a-z0-9][a-z0-9_-]{0,63}$" - }, - "project": { "$ref": "#/$defs/project" }, - "launcher": { - "$ref": "https://docforge.local/schema/adapter-launcher-v1.json" - }, - "launcher_hash": { "$ref": "#/$defs/sha256" }, - "source_availability": { - "$ref": "#/$defs/source_availability" - }, - "source_availability_hash": { "$ref": "#/$defs/sha256" }, - "binding": { "$ref": "#/$defs/binding" }, - "effective_policy": { - "$ref": "https://docforge.local/schema/client-configuration-v1.json#/$defs/effective_policy" - }, - "projection_policy": { - "$ref": "https://docforge.local/schema/client-configuration-v1.json#/$defs/projection_policy" - }, - "projection_policy_hash": { "$ref": "#/$defs/sha256" }, - "projection_availability": { - "$ref": "#/$defs/projection_availability" - }, - "artifact": { - "$ref": "https://docforge.local/schema/client-configuration-v1.json#/properties/artifact" - }, - "configuration_hash": { "$ref": "#/$defs/sha256" }, - "warnings": { - "$ref": "https://docforge.local/schema/client-configuration-v1.json#/properties/warnings" - } - }, - "allOf": [ - { - "if": { - "properties": { "client": { "const": "codex" } }, - "required": ["client"] - }, - "then": { - "properties": { - "artifact": { - "properties": { - "format": { "const": "codex-toml-fragment-v1" } - } - } - } - } - }, - { - "if": { - "properties": { "client": { "const": "claude" } }, - "required": ["client"] - }, - "then": { - "properties": { - "artifact": { - "properties": { - "format": { "const": "claude-json-fragment-v1" } - } - }, - "warnings": { - "contains": { - "properties": { - "code": { "const": "timeout_format_unverified" } - }, - "required": ["code"] - } - } - } - } - }, - { - "if": { - "properties": { "client": { "const": "openclaw" } }, - "required": ["client"] - }, - "then": { - "properties": { - "artifact": { - "properties": { - "format": { "const": "openclaw-json-fragment-v1" } - } - } - } - } - }, - { - "if": { - "properties": { "action": { "const": "preview" } }, - "required": ["action"] - }, - "then": { - "properties": { - "artifact": { - "properties": { - "output_path": { "type": "null" }, - "write_state": { "const": "not_requested" }, - "durability": { "const": "not_applicable" } - } - } - } - }, - "else": { - "properties": { - "artifact": { - "properties": { - "write_state": { "enum": ["created", "unchanged"] } - } - } - } - } - }, - { - "if": { - "properties": { - "binding": { - "properties": { - "adapter_policy": { - "properties": { - "mode": { "const": "preserve-no-ast" } - }, - "required": ["mode"] - } - }, - "required": ["adapter_policy"] - } - }, - "required": ["binding"] - }, - "then": { - "properties": { - "binding": { - "properties": { - "args": { - "contains": { "const": "--no-ast" }, - "minContains": 1, - "maxContains": 1 - } - } - } - } - }, - "else": { - "properties": { - "binding": { - "properties": { - "args": { - "not": { - "contains": { "const": "--no-ast" } - } - } - } - } - } - } - } - ], - "additionalProperties": false -} diff --git a/schemas/adapter-launcher.schema.json b/schemas/adapter-launcher.schema.json deleted file mode 100644 index 4428828..0000000 --- a/schemas/adapter-launcher.schema.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://docforge.local/schema/adapter-launcher-v1.json", - "title": "DocForge project-owned adapter launcher", - "type": "object", - "required": [ - "schema_version", - "entry_point", - "project_id", - "project_root", - "adapter", - "descriptor_hash", - "module" - ], - "properties": { - "schema_version": { "const": 1 }, - "entry_point": { "const": "python-module" }, - "project_id": { - "type": "string", - "pattern": "^[a-z0-9][a-z0-9._-]{1,127}$" - }, - "project_root": { - "type": "string", - "minLength": 1, - "maxLength": 4096 - }, - "adapter": { - "type": "string", - "pattern": "^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}@[A-Za-z0-9][A-Za-z0-9_.+-]{0,63}$" - }, - "descriptor_hash": { - "type": "string", - "pattern": "^[0-9a-f]{64}$" - }, - "module": { - "type": "string", - "minLength": 1, - "maxLength": 255, - "anyOf": [ - { "pattern": "^[A-Za-z_][A-Za-z0-9_]*$" }, - { "const": "docforge.reference_mcp" } - ] - } - }, - "additionalProperties": false -} diff --git a/schemas/reference-adapter.schema.json b/schemas/reference-adapter.schema.json deleted file mode 100644 index bc61277..0000000 --- a/schemas/reference-adapter.schema.json +++ /dev/null @@ -1,78 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://andraxion.net/docforge/schemas/reference-adapter.schema.json", - "title": "DocForge Reference Adapter Configuration", - "type": "object", - "additionalProperties": false, - "required": [ - "schema_version", - "project_id", - "title", - "language", - "source_roots" - ], - "properties": { - "schema_version": { - "const": 1 - }, - "project_id": { - "type": "string", - "pattern": "^[a-z0-9][a-z0-9._-]{1,127}$" - }, - "title": { - "type": "string", - "minLength": 1, - "maxLength": 256 - }, - "language": { - "enum": [ - "python", - "javascript", - "typescript", - "cpp" - ] - }, - "source_roots": { - "type": "array", - "minItems": 1, - "maxItems": 64, - "uniqueItems": true, - "items": { - "type": "string", - "minLength": 1, - "pattern": "^(?!/)(?!.*(?:^|/)\\.\\.(?:/|$)).+$" - } - }, - "compilation_database": { - "type": "string", - "minLength": 1, - "pattern": "^(?!/)(?!.*(?:^|/)\\.\\.(?:/|$)).+$" - } - }, - "allOf": [ - { - "if": { - "properties": { - "language": { - "const": "cpp" - } - }, - "required": [ - "language" - ] - }, - "then": { - "required": [ - "compilation_database" - ] - }, - "else": { - "not": { - "required": [ - "compilation_database" - ] - } - } - } - ] -} diff --git a/src/docforge/adapter_contract.py b/src/docforge/adapter_contract.py index c0829bb..7dd1445 100644 --- a/src/docforge/adapter_contract.py +++ b/src/docforge/adapter_contract.py @@ -21,8 +21,6 @@ from .adapter_validation import ( from .config_validation import ID_PATTERN from .errors import DocForgeError from .incremental import ( - MAX_EXTRACTION_CACHE_BYTES, - MAX_EXTRACTION_CACHE_SOURCES, CachedSource, ExtractionCache, affected_sources, @@ -42,10 +40,6 @@ from .models import ( ) from .telemetry import increment, stage -MAX_ADAPTER_EDGE_MULTIPLIER = 32 -MAX_ADAPTER_LOGIC_NODE_MULTIPLIER = 32 -MAX_ADAPTER_LOGIC_EDGE_MULTIPLIER = 64 - @dataclass(frozen=True) class AdapterNode: @@ -153,13 +147,6 @@ class AdapterAssembly: logic: tuple[LogicProjection, ...] = () -@runtime_checkable -class CompleteAdapterAssemblyLoader(Protocol): - """Load one cache-independent complete graph and Logic equivalence oracle.""" - - def load_complete_assembly(self) -> AdapterAssembly: ... - - class AdapterLoader(Protocol): """Load one current, deterministic, project-confined adapter projection.""" @@ -214,14 +201,6 @@ def _extract_adapter_source( return loader.extract_source(source) -def _load_complete_adapter_assembly( - loader: CompleteAdapterAssemblyLoader, -) -> AdapterAssembly: - increment("adapter_projection_loads") - with stage("adapter.projection"): - return loader.load_complete_assembly() - - @dataclass(frozen=True) class AdapterImplementation: """One confined implementation boundary that must remain stable for a process.""" @@ -253,38 +232,6 @@ class _ImplementationSnapshot: files: tuple[tuple[str, str], ...] -@dataclass(frozen=True) -class AdapterConformanceReport: - """Stable evidence that one adapter passed its complete and incremental contracts.""" - - schema_version: int - project_id: str - adapter_id: str - adapter_version: str - revision: str - source_hash: str - assembly_hash: str - node_count: int - edge_count: int - logic_projection_count: int - incremental: bool - - def as_dict(self) -> dict[str, object]: - return { - "schema_version": self.schema_version, - "project_id": self.project_id, - "adapter_id": self.adapter_id, - "adapter_version": self.adapter_version, - "revision": self.revision, - "source_hash": self.source_hash, - "assembly_hash": self.assembly_hash, - "node_count": self.node_count, - "edge_count": self.edge_count, - "logic_projection_count": self.logic_projection_count, - "incremental": self.incremental, - } - - class AdapterProject: """Expose a validated adapter projection through the standard index boundary.""" @@ -470,9 +417,6 @@ class AdapterProject: canonical_sources, captured, ) - self._enforce_assembly_limits( - AdapterAssembly(projection=projection, logic=self._last_logic) - ) return ProjectSnapshot( descriptor=self.descriptor, nodes=projection.core_nodes(), @@ -579,57 +523,43 @@ class AdapterProject: return self._last_logic def verify_incremental_equivalence(self) -> dict[str, object]: - """Prove incremental graph and Logic output matches an independent complete oracle.""" + """Prove the incremental and full loader contracts produce the same graph.""" self.validate_runtime() - loader = self._incremental_loader - if loader is None: + if self._incremental_loader is None: raise DocForgeError( "incremental_disabled", "Adapter does not implement incremental extraction" ) - incremental = self._load_incremental_assembly() + incremental = self._load_incremental() full = _load_adapter_projection(self.loader) validate_projection(full) - complete = AdapterAssembly(full) - if isinstance(loader, CompleteAdapterAssemblyLoader): - complete = _load_complete_adapter_assembly(loader) - _validate_adapter_assembly(complete) - oracle_fields = _projection_equivalence_fields(complete.projection, full) - oracle_mismatches = [field for field, matches in oracle_fields.items() if not matches] - if oracle_mismatches: - raise DocForgeError( - "complete_oracle_mismatch", - "Complete adapter assembly does not match load_projection()", - fields=oracle_mismatches, - ) - elif incremental.logic: - raise DocForgeError( - "complete_logic_oracle_required", - "Incremental Logic publication requires load_complete_assembly()", - ) - fields = _assembly_equivalence_fields(incremental, complete) + fields = { + "project_id": incremental.project_id == full.project_id, + "adapter_id": incremental.adapter_id == full.adapter_id, + "adapter_version": incremental.adapter_version == full.adapter_version, + "revision": incremental.revision == full.revision, + "source_hash": incremental.source_hash == full.source_hash, + "nodes": incremental.nodes == full.nodes, + "edges": incremental.edges == full.edges, + } mismatches = [field for field, matches in fields.items() if not matches] if mismatches: raise DocForgeError( "incremental_mismatch", - "Incremental extraction does not match a complete adapter assembly", + "Incremental extraction does not match a full adapter projection", fields=mismatches, ) self.validate_runtime() return { "status": "ok", - "project_id": incremental.projection.project_id, - "revision": incremental.projection.revision, - "source_hash": incremental.projection.source_hash, - "node_count": len(incremental.projection.nodes), - "edge_count": len(incremental.projection.edges), - "logic_projection_count": len(incremental.logic), + "project_id": incremental.project_id, + "revision": incremental.revision, + "source_hash": incremental.source_hash, + "node_count": len(incremental.nodes), + "edge_count": len(incremental.edges), } def _load_incremental(self) -> AdapterProjection: - return self._load_incremental_assembly().projection - - def _load_incremental_assembly(self) -> AdapterAssembly: loader = self._incremental_loader if loader is None: raise DocForgeError("incremental_disabled", "Incremental adapter is not configured") @@ -637,14 +567,11 @@ class AdapterProject: validate_manifest(manifest) if manifest.identity() != self._identity: raise DocForgeError("adapter_changed", "Adapter identity changed during the operation") - self._enforce_manifest_limits(manifest) cache = load_extraction_cache( self._cache_path, project_id=manifest.project_id, adapter_id=manifest.adapter_id, adapter_version=manifest.adapter_version, - max_bytes=MAX_EXTRACTION_CACHE_BYTES, - max_sources=MAX_EXTRACTION_CACHE_SOURCES, ) cached = {source.source_id: source for source in cache.sources} if cache else {} current = {source.source_id: source for source in manifest.sources} @@ -731,7 +658,6 @@ class AdapterProject: key=lambda projection: projection.owner_node_id, ) ) - assembly = AdapterAssembly(projection=projection, logic=logic_projections) identity = ( projection.project_id, projection.adapter_id, @@ -748,8 +674,21 @@ class AdapterProject: "invalid_adapter", "Incremental assembly changed the manifest-bound project identity", ) - _validate_adapter_assembly(assembly) - self._enforce_assembly_limits(assembly) + validate_projection(projection) + owners = [projection.owner_node_id for projection in logic_projections] + if len(owners) != len(set(owners)): + raise DocForgeError( + "invalid_adapter", "A primary graph node may own only one logic projection" + ) + node_ids = {item.node.node_id for item in projection.nodes} + for logic_projection in logic_projections: + validate_logic_projection(logic_projection) + if logic_projection.owner_node_id not in node_ids: + raise DocForgeError( + "invalid_adapter", + "A Logic projection owner must exist in the assembled primary graph", + owner_node_id=logic_projection.owner_node_id, + ) stable = loader.load_manifest() validate_manifest(stable) if stable != manifest: @@ -765,8 +704,6 @@ class AdapterProject: adapter_version=manifest.adapter_version, sources=tuple(cache_records), ), - max_bytes=MAX_EXTRACTION_CACHE_BYTES, - max_sources=MAX_EXTRACTION_CACHE_SOURCES, ) self._last_logic = logic_projections self._last_build_report = { @@ -779,41 +716,7 @@ class AdapterProject: "cache_hit_ids": hits, "reparsed_source_ids": reparsed, } - return assembly - - def _enforce_manifest_limits(self, manifest: AdapterManifest) -> None: - if len(manifest.sources) > MAX_EXTRACTION_CACHE_SOURCES: - raise DocForgeError( - "adapter_limit", - "Adapter manifest exceeds the bounded source limit", - maximum=MAX_EXTRACTION_CACHE_SOURCES, - actual=len(manifest.sources), - ) - - def _enforce_assembly_limits(self, assembly: AdapterAssembly) -> None: - maximum_nodes = self.descriptor.limits.max_nodes - maximum_edges = maximum_nodes * MAX_ADAPTER_EDGE_MULTIPLIER - maximum_logic_nodes = maximum_nodes * MAX_ADAPTER_LOGIC_NODE_MULTIPLIER - maximum_logic_edges = maximum_nodes * MAX_ADAPTER_LOGIC_EDGE_MULTIPLIER - actual_nodes = len(assembly.projection.nodes) - actual_edges = len(assembly.projection.edges) - actual_logic_nodes = sum(len(projection.nodes) for projection in assembly.logic) - actual_logic_edges = sum(len(projection.edges) for projection in assembly.logic) - limits = ( - ("nodes", actual_nodes, maximum_nodes), - ("edges", actual_edges, maximum_edges), - ("logic_nodes", actual_logic_nodes, maximum_logic_nodes), - ("logic_edges", actual_logic_edges, maximum_logic_edges), - ) - for label, actual, maximum in limits: - if actual > maximum: - raise DocForgeError( - "adapter_limit", - f"Adapter assembly exceeds the configured {label.replace('_', ' ')} limit", - kind=label, - maximum=maximum, - actual=actual, - ) + return projection def canonical_source_paths(self) -> tuple[Path, ...]: """Adapters validate their own source sets before producing a projection.""" @@ -1129,141 +1032,6 @@ class AdapterProject: outputs.add(output) -def _validate_adapter_assembly(assembly: AdapterAssembly) -> None: - validate_projection(assembly.projection) - ordered_logic = tuple(sorted(assembly.logic, key=lambda projection: projection.owner_node_id)) - if assembly.logic != ordered_logic: - raise DocForgeError( - "invalid_adapter", - "Complete Logic projections must be deterministically ordered", - ) - owners = [projection.owner_node_id for projection in assembly.logic] - if len(owners) != len(set(owners)): - raise DocForgeError( - "invalid_adapter", "A primary graph node may own only one logic projection" - ) - node_ids = {item.node.node_id for item in assembly.projection.nodes} - for logic_projection in assembly.logic: - validate_logic_projection(logic_projection) - if logic_projection.owner_node_id not in node_ids: - raise DocForgeError( - "invalid_adapter", - "A Logic projection owner must exist in the assembled primary graph", - owner_node_id=logic_projection.owner_node_id, - ) - - -def _projection_equivalence_fields( - candidate: AdapterProjection, - reference: AdapterProjection, -) -> dict[str, bool]: - return { - "project_id": candidate.project_id == reference.project_id, - "title": candidate.title == reference.title, - "adapter_id": candidate.adapter_id == reference.adapter_id, - "adapter_version": candidate.adapter_version == reference.adapter_version, - "root": candidate.root == reference.root, - "revision": candidate.revision == reference.revision, - "source_hash": candidate.source_hash == reference.source_hash, - "nodes": candidate.nodes == reference.nodes, - "edges": candidate.edges == reference.edges, - } - - -def _assembly_equivalence_fields( - candidate: AdapterAssembly, - reference: AdapterAssembly, -) -> dict[str, bool]: - return { - **_projection_equivalence_fields(candidate.projection, reference.projection), - "logic": candidate.logic == reference.logic, - } - - -def _assembly_hash(assembly: AdapterAssembly) -> str: - projection = assembly.projection - payload = { - "projection": { - "project_id": projection.project_id, - "title": projection.title, - "adapter_id": projection.adapter_id, - "adapter_version": projection.adapter_version, - "root": str(projection.root), - "revision": projection.revision, - "source_hash": projection.source_hash, - "nodes": [item.as_dict() for item in projection.nodes], - "edges": [item.as_dict() for item in projection.edges], - }, - "logic": [logic_projection.as_dict() for logic_projection in assembly.logic], - } - return hashlib.sha256( - json.dumps(payload, sort_keys=True, separators=(",", ":")).encode() - ).hexdigest() - - -def _complete_reference_assembly(loader: AdapterLoader) -> AdapterAssembly: - if isinstance(loader, CompleteAdapterAssemblyLoader): - assembly = _load_complete_adapter_assembly(loader) - else: - assembly = AdapterAssembly(_load_adapter_projection(loader)) - _validate_adapter_assembly(assembly) - return assembly - - -def verify_adapter_conformance( - loader: AdapterLoader, - *, - cache_root: Path, - settings: AdapterProjectSettings | None = None, -) -> AdapterConformanceReport: - """Prove deterministic complete output and, when supported, incremental parity.""" - - project = AdapterProject(loader, cache_root=cache_root, settings=settings) - project.validate_runtime() - first = _complete_reference_assembly(loader) - second = _complete_reference_assembly(loader) - deterministic_fields = _assembly_equivalence_fields(first, second) - nondeterministic = [field for field, matches in deterministic_fields.items() if not matches] - if nondeterministic: - raise DocForgeError( - "nondeterministic_adapter", - "Repeated complete adapter assemblies do not match", - fields=nondeterministic, - ) - - if isinstance(loader, CompleteAdapterAssemblyLoader): - graph_oracle = _load_adapter_projection(loader) - validate_projection(graph_oracle) - oracle_fields = _projection_equivalence_fields(first.projection, graph_oracle) - oracle_mismatches = [field for field, matches in oracle_fields.items() if not matches] - if oracle_mismatches: - raise DocForgeError( - "complete_oracle_mismatch", - "Complete adapter assembly does not match load_projection()", - fields=oracle_mismatches, - ) - - incremental = isinstance(loader, IncrementalAdapterLoader) - if incremental: - project.verify_incremental_equivalence() - - project.validate_runtime() - projection = first.projection - return AdapterConformanceReport( - schema_version=1, - project_id=projection.project_id, - adapter_id=projection.adapter_id, - adapter_version=projection.adapter_version, - revision=projection.revision, - source_hash=projection.source_hash, - assembly_hash=_assembly_hash(first), - node_count=len(projection.nodes), - edge_count=len(projection.edges), - logic_projection_count=len(first.logic), - incremental=incremental, - ) - - @dataclass(frozen=True) class ShadowArtifact: """One named deterministic byte artifact used by a shadow comparison.""" diff --git a/src/docforge/adapter_launcher.py b/src/docforge/adapter_launcher.py deleted file mode 100644 index f86be68..0000000 --- a/src/docforge/adapter_launcher.py +++ /dev/null @@ -1,392 +0,0 @@ -"""Strict, data-only launch contracts for project-owned adapter MCP modules.""" - -from __future__ import annotations - -import json -import re -import stat -import subprocess -import sys -from dataclasses import dataclass -from pathlib import Path -from typing import Literal, cast - -from .changeset_contract import document_hash -from .config_validation import ID_PATTERN -from .errors import DocForgeError -from .models import IncrementalStateProject, ProjectService, RuntimeValidatedProject - -ADAPTER_LAUNCHER_SCHEMA_VERSION = 1 -_PYTHON_MODULE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*){0,31}") -_TOP_LEVEL_PYTHON_MODULE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*") -_ADAPTER_IDENTITY = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}@[A-Za-z0-9][A-Za-z0-9_.+-]{0,63}") -_SHA256 = re.compile(r"[0-9a-f]{64}") -_TRUSTED_DOTTED_MODULE = "docforge.reference_mcp" -_MODULE_PROBE_TIMEOUT_SECONDS = 5 -_MAX_MODULE_PROBE_BYTES = 8_192 -_MODULE_PROBE = ( - "import importlib.util,json,sys;" - "spec=importlib.util.find_spec(sys.argv[1]);" - "result=None if spec is None else {" - "'has_loader':spec.loader is not None," - "'is_package':spec.submodule_search_locations is not None," - "'origin':spec.origin};" - "print(json.dumps(result,sort_keys=True,separators=(',',':')))" -) -_run_module_probe = subprocess.run -__all__ = [ - "ADAPTER_LAUNCHER_SCHEMA_VERSION", - "AdapterLauncherV1", - "AdapterSourceAvailabilityV1", - "adapter_source_availability", - "validate_adapter_launcher", -] - - -@dataclass(frozen=True) -class AdapterLauncherV1: - """One immutable, project-bound Python module launch declaration. - - The contract intentionally has no command, shell string, working directory, - environment, arbitrary arguments, discovery rule, or callable selector. - """ - - schema_version: int - entry_point: Literal["python-module"] - project_id: str - project_root: Path - adapter: str - descriptor_hash: str - module: str - - def __post_init__(self) -> None: - _validate_launcher_fields(self) - - @classmethod - def for_project( - cls, - project: ProjectService, - *, - module: str, - ) -> AdapterLauncherV1: - """Bind a structural module entry point to one already constructed adapter project.""" - - descriptor = project.descriptor - if descriptor.adapter == "generic": - raise DocForgeError( - "adapter_launcher_unavailable", - "Project-owned launcher configuration requires a custom adapter", - ) - launcher = cls( - schema_version=ADAPTER_LAUNCHER_SCHEMA_VERSION, - entry_point="python-module", - project_id=descriptor.project_id, - project_root=descriptor.root, - adapter=descriptor.adapter, - descriptor_hash=descriptor.descriptor_hash, - module=module, - ) - validate_adapter_launcher(project, launcher) - return launcher - - def as_dict(self) -> dict[str, object]: - return { - "schema_version": self.schema_version, - "entry_point": self.entry_point, - "project_id": self.project_id, - "project_root": str(self.project_root), - "adapter": self.adapter, - "descriptor_hash": self.descriptor_hash, - "module": self.module, - } - - @property - def launcher_hash(self) -> str: - return document_hash(self.as_dict()) - - -@dataclass(frozen=True) -class AdapterSourceAvailabilityV1: - """One bounded source identity observed without executing the launcher module.""" - - schema_version: int - status: Literal["available"] - method: Literal["incremental-state", "complete-projection"] - revision: str - source_hash: str - - def __post_init__(self) -> None: - revision = _runtime_value(self.revision) - if ( - self.schema_version != 1 - or self.status != "available" - or self.method not in {"incremental-state", "complete-projection"} - or not isinstance(revision, str) - or not revision - or _SHA256.fullmatch(self.source_hash) is None - ): - raise DocForgeError( - "invalid_adapter_launcher", - "Adapter source availability evidence is invalid", - ) - - def as_dict(self) -> dict[str, object]: - return { - "schema_version": self.schema_version, - "status": self.status, - "method": self.method, - "revision": self.revision, - "source_hash": self.source_hash, - } - - @property - def availability_hash(self) -> str: - return document_hash(self.as_dict()) - - -def validate_adapter_launcher( - project: ProjectService, - launcher: AdapterLauncherV1, -) -> None: - """Require one launcher to match the exact current adapter project binding.""" - - _validate_launcher_fields(launcher) - descriptor = project.descriptor - _validate_project_root(descriptor.root) - if descriptor.adapter == "generic": - raise DocForgeError( - "adapter_launcher_unavailable", - "Project-owned launcher configuration requires a custom adapter", - ) - if ( - launcher.project_id != descriptor.project_id - or launcher.project_root != descriptor.root - or launcher.adapter != descriptor.adapter - or launcher.descriptor_hash != descriptor.descriptor_hash - ): - raise DocForgeError( - "adapter_launcher_mismatch", - "Adapter launcher does not match the selected project binding", - project_id=descriptor.project_id, - adapter=descriptor.adapter, - ) - if isinstance(project, RuntimeValidatedProject): - project.validate_runtime() - - -def adapter_source_availability( - project: ProjectService, - launcher: AdapterLauncherV1, -) -> AdapterSourceAvailabilityV1: - """Capture current source identity without importing or executing the launcher module.""" - - validate_adapter_launcher(project, launcher) - state = project.incremental_state() if isinstance(project, IncrementalStateProject) else None - if state is not None: - method: Literal["incremental-state", "complete-projection"] = "incremental-state" - revision = state.revision - source_hash = state.source_hash - else: - snapshot = project.load() - descriptor = snapshot.descriptor - if ( - descriptor.project_id != launcher.project_id - or descriptor.root != launcher.project_root - or descriptor.adapter != launcher.adapter - or descriptor.descriptor_hash != launcher.descriptor_hash - ): - raise DocForgeError( - "adapter_launcher_mismatch", - "Loaded adapter projection drifted from its launcher binding", - ) - method = "complete-projection" - revision = snapshot.revision - source_hash = snapshot.source_hash - validate_adapter_launcher(project, launcher) - return AdapterSourceAvailabilityV1( - schema_version=1, - status="available", - method=method, - revision=revision, - source_hash=source_hash, - ) - - -def _validate_launcher_fields(launcher: AdapterLauncherV1) -> None: - if type(launcher.schema_version) is not int or launcher.schema_version != 1: - raise DocForgeError( - "invalid_adapter_launcher", - "Adapter launcher schema version is unsupported", - supported=1, - ) - if launcher.entry_point != "python-module": - raise DocForgeError( - "invalid_adapter_launcher", - "Adapter launcher entry point must be one isolated Python module", - ) - project_id = _runtime_value(launcher.project_id) - project_root = _runtime_value(launcher.project_root) - adapter = _runtime_value(launcher.adapter) - descriptor_hash = _runtime_value(launcher.descriptor_hash) - module = _runtime_value(launcher.module) - if not isinstance(project_id, str) or ID_PATTERN.fullmatch(project_id) is None: - raise DocForgeError("invalid_adapter_launcher", "Adapter launcher project ID is invalid") - if not isinstance(project_root, Path): - raise DocForgeError("invalid_adapter_launcher", "Adapter launcher project root is invalid") - _validate_project_root(project_root) - if not isinstance(adapter, str) or _ADAPTER_IDENTITY.fullmatch(adapter) is None: - raise DocForgeError("invalid_adapter_launcher", "Adapter launcher identity is invalid") - if not isinstance(descriptor_hash, str) or _SHA256.fullmatch(descriptor_hash) is None: - raise DocForgeError("invalid_adapter_launcher", "Adapter descriptor hash is invalid") - if ( - not isinstance(module, str) - or len(module) > 255 - or _PYTHON_MODULE.fullmatch(module) is None - or module == "docforge.mcp_server" - or (module != _TRUSTED_DOTTED_MODULE and _TOP_LEVEL_PYTHON_MODULE.fullmatch(module) is None) - ): - raise DocForgeError( - "invalid_adapter_launcher", - ( - "Adapter launcher module must be one installed project-owned top-level " - "Python module or the fixed DocForge reference binding" - ), - ) - _validate_isolated_module(project_root, module) - - -def _validate_isolated_module(project_root: Path, module: str) -> None: - """Prove ``python -I -m`` can resolve one module without importing project code.""" - - executable = Path(sys.executable) - try: - executable_status = executable.stat() - except OSError as error: - raise DocForgeError( - "adapter_launcher_unavailable", - "The isolated Python executable is unavailable", - ) from error - if not stat.S_ISREG(executable_status.st_mode): - raise DocForgeError( - "adapter_launcher_unavailable", - "The isolated Python executable is not a regular file", - ) - try: - completed = _run_module_probe( - [str(executable), "-I", "-c", _MODULE_PROBE, module], - capture_output=True, - text=True, - timeout=_MODULE_PROBE_TIMEOUT_SECONDS, - check=False, - ) - except (OSError, subprocess.TimeoutExpired) as error: - raise DocForgeError( - "adapter_launcher_unavailable", - "The isolated adapter launcher probe could not complete", - module=module, - ) from error - if ( - completed.returncode != 0 - or len(completed.stdout.encode("utf-8")) > _MAX_MODULE_PROBE_BYTES - or len(completed.stderr.encode("utf-8")) > _MAX_MODULE_PROBE_BYTES - ): - raise DocForgeError( - "adapter_launcher_unavailable", - "The isolated adapter launcher module could not be resolved", - module=module, - ) - try: - result: object = json.loads(completed.stdout) - except (UnicodeError, json.JSONDecodeError) as error: - raise DocForgeError( - "adapter_launcher_unavailable", - "The isolated adapter launcher probe returned invalid evidence", - module=module, - ) from error - if result is None: - raise DocForgeError( - "adapter_launcher_unavailable", - "The isolated adapter launcher module is not installed", - module=module, - ) - if not isinstance(result, dict): - raise DocForgeError( - "adapter_launcher_unavailable", - "The isolated adapter launcher probe returned invalid evidence", - module=module, - ) - evidence = cast(dict[object, object], result) - if ( - set(evidence) != {"has_loader", "is_package", "origin"} - or evidence["has_loader"] is not True - or evidence["is_package"] is not False - or not isinstance(evidence["origin"], str) - ): - raise DocForgeError( - "invalid_adapter_launcher", - "The isolated adapter launcher must resolve to one executable module file", - module=module, - ) - origin = Path(evidence["origin"]) - try: - origin_status = origin.lstat() - resolved_origin = origin.resolve(strict=True) - except OSError as error: - raise DocForgeError( - "adapter_launcher_unavailable", - "The isolated adapter launcher module origin is unavailable", - module=module, - ) from error - if ( - not origin.is_absolute() - or stat.S_ISLNK(origin_status.st_mode) - or not stat.S_ISREG(origin_status.st_mode) - or resolved_origin != origin - or origin.suffix != ".py" - ): - raise DocForgeError( - "invalid_adapter_launcher", - "The isolated adapter launcher must be one canonical Python module file", - module=module, - ) - if module == _TRUSTED_DOTTED_MODULE: - expected = Path(__file__).with_name("reference_mcp.py").resolve(strict=True) - if origin != expected: - raise DocForgeError( - "adapter_launcher_mismatch", - "The fixed reference launcher resolved outside this DocForge installation", - module=module, - ) - elif not origin.is_relative_to(project_root): - raise DocForgeError( - "invalid_adapter_launcher", - "The installed adapter launcher module is not owned by the selected project", - module=module, - ) - - -def _validate_project_root(root: Path) -> None: - try: - status = root.lstat() - resolved = root.resolve(strict=True) - except OSError as error: - raise DocForgeError( - "invalid_adapter_launcher", - "Adapter launcher project root is unavailable", - ) from error - if ( - not root.is_absolute() - or stat.S_ISLNK(status.st_mode) - or not stat.S_ISDIR(status.st_mode) - or resolved != root - ): - raise DocForgeError( - "invalid_adapter_launcher", - "Adapter launcher project root must be one canonical real directory", - ) - - -def _runtime_value(value: object) -> object: - """Keep runtime validation explicit even when static callers are typed.""" - - return value diff --git a/src/docforge/adapter_sdk.py b/src/docforge/adapter_sdk.py deleted file mode 100644 index 888f7f2..0000000 --- a/src/docforge/adapter_sdk.py +++ /dev/null @@ -1,69 +0,0 @@ -"""Public typed surface for project adapter implementations and conformance checks.""" - -from .adapter_contract import ( - AdapterAssembly, - AdapterConformanceReport, - AdapterEdge, - AdapterImplementation, - AdapterLoader, - AdapterManifest, - AdapterNode, - AdapterProject, - AdapterProjection, - AdapterProjectSettings, - AdapterSource, - AdapterSourceProjection, - CompleteAdapterAssemblyLoader, - IncrementalAdapterAssembler, - IncrementalAdapterLoader, - verify_adapter_conformance, -) -from .adapter_validation import ( - validate_logic_projection, - validate_manifest, - validate_projection, - validate_source_projection, -) -from .models import ( - Edge, - Limits, - LogicEdge, - LogicNode, - LogicProjection, - Node, - ProposalWriter, - RenderConfig, - RenderView, -) - -__all__ = [ - "AdapterAssembly", - "AdapterConformanceReport", - "AdapterEdge", - "AdapterImplementation", - "AdapterLoader", - "AdapterManifest", - "AdapterNode", - "AdapterProject", - "AdapterProjection", - "AdapterProjectSettings", - "AdapterSource", - "AdapterSourceProjection", - "CompleteAdapterAssemblyLoader", - "Edge", - "IncrementalAdapterAssembler", - "IncrementalAdapterLoader", - "Limits", - "LogicEdge", - "LogicNode", - "LogicProjection", - "Node", - "ProposalWriter", - "RenderConfig", - "RenderView", - "validate_logic_projection", - "validate_manifest", - "validate_projection", - "validate_source_projection", - "verify_adapter_conformance", -] diff --git a/src/docforge/adapters/__init__.py b/src/docforge/adapters/__init__.py deleted file mode 100644 index 0ae5694..0000000 --- a/src/docforge/adapters/__init__.py +++ /dev/null @@ -1,25 +0,0 @@ -"""Repository-owned reference adapters built on the public adapter SDK.""" - -from .javascript import ( - JAVASCRIPT_ADAPTER_VERSION, - JAVASCRIPT_EXTRACTOR_VERSION, - JavaScriptReferenceAdapter, - JavaScriptUnsupportedFact, -) -from .python import ( - PYTHON_ADAPTER_VERSION, - PYTHON_EXTRACTOR_VERSION, - PythonReferenceAdapter, - PythonUnsupportedFact, -) - -__all__ = [ - "JAVASCRIPT_ADAPTER_VERSION", - "JAVASCRIPT_EXTRACTOR_VERSION", - "PYTHON_ADAPTER_VERSION", - "PYTHON_EXTRACTOR_VERSION", - "JavaScriptReferenceAdapter", - "JavaScriptUnsupportedFact", - "PythonReferenceAdapter", - "PythonUnsupportedFact", -] diff --git a/src/docforge/adapters/cpp.py b/src/docforge/adapters/cpp.py deleted file mode 100644 index 11e6425..0000000 --- a/src/docforge/adapters/cpp.py +++ /dev/null @@ -1,1349 +0,0 @@ -"""Deterministic, syntax-only C++ reference adapter. - -The adapter accepts one explicit, project-confined ``compile_commands.json`` as -the authoritative translation-unit inventory. The database, compiler command, -arguments, and C++ sources are parsed only as inert data. No compiler, build -tool, project binary, or project code is ever executed. - -Only quoted includes that resolve directly to a real, project-local header are -published as dependencies. Tree-sitter proves lexical declarations and -function control flow; compiler-dependent facts are deliberately omitted. -""" - -from __future__ import annotations - -import hashlib -import importlib -import json -import os -import stat -from collections.abc import Iterable, Sequence -from dataclasses import dataclass, replace -from functools import lru_cache -from pathlib import Path, PurePosixPath -from typing import TYPE_CHECKING, Protocol, cast - -from ..adapter_sdk import ( - AdapterAssembly, - AdapterEdge, - AdapterManifest, - AdapterNode, - AdapterProjection, - AdapterSource, - AdapterSourceProjection, - Edge, -) -from ..adapter_sdk import Node as GraphNode -from ..errors import DocForgeError -from ..treesitter_logic import ( - TreeSitterLogicOwner, - analyze_cpp_source, - discover_cpp_functions, -) - -if TYPE_CHECKING: - from tree_sitter import Language - -CPP_ADAPTER_ID = "docforge.reference.cpp" -CPP_ADAPTER_VERSION = "1" -CPP_EXTRACTOR_VERSION = "tree-sitter-cpp@1" -CPP_IDENTITY_VERSION = "cpp-reference-id@1" -CPP_SUPPORT_SCHEMA_VERSION = 1 - -_TRANSLATION_UNIT_SUFFIXES = frozenset({".c++", ".cc", ".cpp", ".cxx"}) -_HEADER_SUFFIXES = frozenset({".h", ".hh", ".hpp", ".hxx", ".inc", ".ipp", ".tpp"}) -_SCOPE_TYPES = { - "namespace_definition": "namespace", - "class_specifier": "class", - "struct_specifier": "struct", -} -_COMPILE_ENTRY_KEYS = frozenset({"arguments", "command", "directory", "file", "output"}) -_SUPPORTED_FACTS = ( - "cpp_file", - "cpp_translation_unit", - "cpp_namespace", - "cpp_class", - "cpp_struct", - "cpp_function", - "lexical_containment", - "project_local_quoted_include_dependency", - "function_logic", -) - - -@dataclass(frozen=True) -class CppUnsupportedFact: - """One compiler-dependent fact this syntax-only adapter does not claim.""" - - code: str - description: str - - def as_dict(self) -> dict[str, str]: - return {"code": self.code, "description": self.description} - - -_UNSUPPORTED_FACTS = ( - CppUnsupportedFact( - "call_resolution", - "Calls appear only inside function Logic and are not resolved to symbols.", - ), - CppUnsupportedFact( - "type_resolution", - "Types, templates, aliases, concepts, variables, and references are not resolved.", - ), - CppUnsupportedFact( - "inheritance_resolution", - "Base-specifier syntax is not resolved to local or external types.", - ), - CppUnsupportedFact( - "overload_ownership", - "Overload sets and out-of-line semantic ownership are not compiler-resolved.", - ), - CppUnsupportedFact( - "macro_semantics", - "Conditional compilation and macro expansion are not evaluated.", - ), - CppUnsupportedFact( - "compiler_command_execution", - "Compilation database commands and arguments are retained as fingerprints only.", - ), - CppUnsupportedFact( - "nonlocal_include_resolution", - "Angle-bracket, include-path, framework, generated, and compiler-provided " - "headers are omitted.", - ), -) - - -class _Point(Protocol): - row: int - - -class _TreeNode(Protocol): - type: str - has_error: bool - is_error: bool - is_missing: bool - start_byte: int - end_byte: int - start_point: _Point - named_children: list[_TreeNode] - - def child_by_field_name(self, name: str, /) -> _TreeNode | None: ... - - -@dataclass(frozen=True) -class _CompilationEntry: - source_path: str - directory: str - command: str | None - arguments: tuple[str, ...] | None - output: str | None - - def canonical_bytes(self) -> bytes: - value: dict[str, object] = { - "directory": self.directory, - "file": self.source_path, - } - if self.command is not None: - value["command"] = self.command - if self.arguments is not None: - value["arguments"] = list(self.arguments) - if self.output is not None: - value["output"] = self.output - return json.dumps( - value, - ensure_ascii=False, - separators=(",", ":"), - sort_keys=True, - ).encode("utf-8") - - -@dataclass(frozen=True) -class _SourceRecord: - source_id: str - source_path: str - translation_unit: bool - fingerprint: str - text: str - raw: bytes - root: _TreeNode - compilation_entry: _CompilationEntry | None = None - dependencies: tuple[str, ...] = () - - -@dataclass(frozen=True) -class _Definition: - kind: str - qualified_name: str - logic_qualified_name: str | None - line: int - parent_qualified_name: str | None - node: _TreeNode - node_id: str - - -@lru_cache(maxsize=1) -def _cpp_language() -> Language: - try: - tree_sitter = importlib.import_module("tree_sitter") - grammar = importlib.import_module("tree_sitter_cpp") - except ModuleNotFoundError as error: - raise DocForgeError( - "optional_dependency_missing", - "The requested language frontend is not installed", - extra="cpp", - install="docforge[cpp]", - missing_module=error.name, - ) from error - return cast("Language", tree_sitter.Language(grammar.language())) - - -class CppReferenceAdapter: - """Reference C++ adapter bound to one explicit compilation database.""" - - def __init__( - self, - root: Path, - *, - source_roots: Iterable[str | Path], - compilation_database: str | Path, - project_id: str = "cpp-reference", - title: str = "C++ reference project", - max_translation_units: int = 4_096, - max_headers: int = 8_192, - max_database_bytes: int = 8_000_000, - max_source_bytes: int = 2_000_000, - max_command_chars: int = 64_000, - max_arguments: int = 8_192, - max_logic_nodes_per_function: int = 2_000, - ) -> None: - resolved_root = root.resolve(strict=True) - if not resolved_root.is_dir(): - raise DocForgeError("invalid_adapter", "C++ adapter root must be a directory") - if ( - max_translation_units < 1 - or max_headers < 1 - or max_database_bytes < 1 - or max_source_bytes < 1 - or max_command_chars < 1 - or max_arguments < 1 - or max_logic_nodes_per_function < 2 - ): - raise ValueError("C++ adapter limits must be positive") - self.root = resolved_root - self.project_id = project_id - self.title = title - self.max_translation_units = max_translation_units - self.max_headers = max_headers - self.max_database_bytes = max_database_bytes - self.max_source_bytes = max_source_bytes - self.max_command_chars = max_command_chars - self.max_arguments = max_arguments - self.max_logic_nodes_per_function = max_logic_nodes_per_function - self.source_roots = self._normalize_source_roots(source_roots) - self.compilation_database = self._normalize_database_path(compilation_database) - - def support_report(self) -> dict[str, object]: - """Return deterministic scope, frontend, and limitation evidence.""" - - return { - "schema_version": CPP_SUPPORT_SCHEMA_VERSION, - "adapter_id": CPP_ADAPTER_ID, - "adapter_version": CPP_ADAPTER_VERSION, - "extractor_version": CPP_EXTRACTOR_VERSION, - "identity_version": CPP_IDENTITY_VERSION, - "frontend": "tree-sitter-cpp", - "compilation_database": self.compilation_database, - "compilation_database_is_authoritative": True, - "source_roots": list(self.source_roots), - "executes_compiler": False, - "executes_project_code": False, - "translation_unit_suffixes": sorted(_TRANSLATION_UNIT_SUFFIXES), - "header_suffixes": sorted(_HEADER_SUFFIXES), - "supported_facts": list(_SUPPORTED_FACTS), - "unsupported_facts": [fact.as_dict() for fact in _UNSUPPORTED_FACTS], - } - - def unsupported_facts(self) -> tuple[CppUnsupportedFact, ...]: - return _UNSUPPORTED_FACTS - - def load_manifest(self) -> AdapterManifest: - records = self._inventory() - path_sources = {record.source_path: record.source_id for record in records} - sources = tuple( - sorted( - ( - AdapterSource( - source_id=record.source_id, - source_path=record.source_path, - fingerprint=record.fingerprint, - extractor_version=CPP_EXTRACTOR_VERSION, - dependencies=tuple( - sorted( - path_sources[path] - for path in record.dependencies - if path_sources[path] != record.source_id - ) - ), - ) - for record in records - ), - key=lambda source: source.source_id, - ) - ) - source_hash = self._source_hash(sources) - return AdapterManifest( - project_id=self.project_id, - title=self.title, - adapter_id=CPP_ADAPTER_ID, - adapter_version=CPP_ADAPTER_VERSION, - root=self.root, - revision=source_hash[:12], - source_hash=source_hash, - families=("code",), - allowed_relations=("contains", "depends_on"), - sources=sources, - estimated_nodes=max(1, len(sources) * 10), - ) - - def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: - entry: _CompilationEntry | None = None - translation_unit = source.source_id.startswith("cpp.tu.") - if translation_unit: - entries = {item.source_path: item for item in self._compilation_entries()} - entry = entries.get(source.source_path) - if entry is None: - raise DocForgeError( - "stale_adapter_source", - "C++ translation unit is no longer in the compilation database", - source=source.source_path, - ) - record = self._read_source(source.source_path, compilation_entry=entry) - if ( - record.source_id != source.source_id - or record.fingerprint != source.fingerprint - or record.translation_unit != translation_unit - ): - raise DocForgeError( - "stale_adapter_source", - "C++ source or compilation entry changed after its manifest was captured", - source=source.source_path, - ) - return self._extract_record(record, source) - - def assemble_projection( - self, - manifest: AdapterManifest, - contributions: tuple[AdapterSourceProjection, ...], - ) -> AdapterAssembly: - expected = tuple(source.source_id for source in manifest.sources) - actual = tuple(sorted(contribution.source_id for contribution in contributions)) - if actual != expected: - raise DocForgeError( - "invalid_adapter", - "C++ assembly contributions do not match the current manifest", - ) - nodes = tuple( - sorted( - (node for contribution in contributions for node in contribution.nodes), - key=lambda item: item.node.node_id, - ) - ) - edges = tuple( - sorted( - (edge for contribution in contributions for edge in contribution.edges), - key=lambda item: ( - item.edge.source_id, - item.edge.relation, - item.edge.target_id, - ), - ) - ) - logic = tuple( - sorted( - (projection for contribution in contributions for projection in contribution.logic), - key=lambda projection: projection.owner_node_id, - ) - ) - return AdapterAssembly( - projection=AdapterProjection( - project_id=manifest.project_id, - title=manifest.title, - adapter_id=manifest.adapter_id, - adapter_version=manifest.adapter_version, - root=manifest.root, - revision=manifest.revision, - source_hash=manifest.source_hash, - nodes=nodes, - edges=edges, - ), - logic=logic, - ) - - def load_assembly(self) -> AdapterAssembly: - """Load one cache-independent complete graph and Logic assembly.""" - - manifest = self.load_manifest() - contributions = tuple(self.extract_source(source) for source in manifest.sources) - if self.load_manifest() != manifest: - raise DocForgeError( - "source_changed", - "C++ inputs changed during complete adapter extraction", - ) - return self.assemble_projection(manifest, contributions) - - def load_complete_assembly(self) -> AdapterAssembly: - """Implement the SDK's independent complete-assembly oracle.""" - - return self.load_assembly() - - def load_projection(self) -> AdapterProjection: - """Preserve the complete-projection compatibility contract.""" - - return self.load_assembly().projection - - def _normalize_database_path(self, value: str | Path) -> str: - supplied = Path(value) - candidate = supplied if supplied.is_absolute() else self.root / supplied - normalized = Path(os.path.abspath(candidate)) - if normalized.name != "compile_commands.json" or not normalized.is_relative_to(self.root): - raise DocForgeError( - "path_escape", - "C++ compilation database must be an explicit project-confined " - "compile_commands.json", - ) - relative = normalized.relative_to(self.root).as_posix() - self._validate_real_file( - normalized, - relative, - unavailable_code="invalid_compilation_database", - label="C++ compilation database", - ) - return relative - - def _normalize_source_roots( - self, - source_roots: Iterable[str | Path], - ) -> tuple[str, ...]: - normalized: list[str] = [] - for value in source_roots: - candidate = PurePosixPath(Path(value).as_posix()) - if ( - candidate.is_absolute() - or not candidate.parts - or ".." in candidate.parts - or str(candidate) in {"", "."} - ): - raise DocForgeError( - "path_escape", - "C++ source roots must be explicit project-relative directories", - ) - relative = candidate.as_posix() - absolute = self.root.joinpath(*candidate.parts) - if absolute.is_symlink(): - raise DocForgeError("path_escape", "C++ source root cannot be a symbolic link") - try: - resolved = absolute.resolve(strict=True) - except OSError as error: - raise DocForgeError( - "invalid_adapter", - "C++ source root does not exist", - source_root=relative, - ) from error - if ( - resolved != absolute - or not resolved.is_dir() - or not resolved.is_relative_to(self.root) - ): - raise DocForgeError( - "path_escape", - "C++ source root must be a confined real directory", - source_root=relative, - ) - normalized.append(relative) - result = tuple(sorted(set(normalized))) - if not result: - raise DocForgeError("invalid_adapter", "At least one C++ source root is required") - if len(result) != len(normalized): - raise DocForgeError("invalid_adapter", "C++ source roots must be unique") - paths = [PurePosixPath(value) for value in result] - for index, left in enumerate(paths): - for right in paths[index + 1 :]: - if left in right.parents or right in left.parents: - raise DocForgeError( - "invalid_adapter", - "C++ source roots must not overlap", - ) - return result - - def _source_root_for(self, path: PurePosixPath) -> str: - matches = [ - source_root - for source_root in self.source_roots - if path == PurePosixPath(source_root) or PurePosixPath(source_root) in path.parents - ] - if len(matches) != 1: - raise DocForgeError( - "path_escape", - "C++ source is outside the declared source roots", - source=path.as_posix(), - ) - return matches[0] - - def _compilation_entries(self) -> tuple[_CompilationEntry, ...]: - database = self.root.joinpath(*PurePosixPath(self.compilation_database).parts) - raw = self._read_real_file( - database, - self.compilation_database, - maximum=self.max_database_bytes, - unavailable_code="invalid_compilation_database", - label="C++ compilation database", - ) - try: - text = raw.decode("utf-8") - except UnicodeDecodeError as error: - raise DocForgeError( - "invalid_compilation_database", - "C++ compilation database must use UTF-8", - ) from error - try: - parsed = cast( - object, - json.loads(text, object_pairs_hook=self._unique_json_object), - ) - except (TypeError, ValueError, json.JSONDecodeError) as error: - raise DocForgeError( - "invalid_compilation_database", - "C++ compilation database is not valid duplicate-free JSON", - ) from error - if not isinstance(parsed, list) or not parsed: - raise DocForgeError( - "invalid_compilation_database", - "C++ compilation database must be a non-empty JSON array", - ) - parsed_items = cast("list[object]", parsed) - if len(parsed_items) > self.max_translation_units: - raise DocForgeError( - "adapter_too_large", - "C++ translation-unit inventory exceeds the configured limit", - maximum=self.max_translation_units, - ) - entries = tuple( - self._compilation_entry(value, index) for index, value in enumerate(parsed_items) - ) - paths = [entry.source_path for entry in entries] - if len(paths) != len(set(paths)): - duplicates = sorted(path for path in set(paths) if paths.count(path) > 1) - raise DocForgeError( - "duplicate_translation_unit", - "C++ compilation database repeats a normalized translation unit", - sources=duplicates, - ) - return tuple(sorted(entries, key=lambda entry: entry.source_path)) - - def _compilation_entry(self, value: object, index: int) -> _CompilationEntry: - if not isinstance(value, dict): - raise DocForgeError( - "invalid_compilation_database", - "Each C++ compilation entry must be a JSON object", - entry=index, - ) - raw_document = cast("dict[object, object]", value) - if any(not isinstance(key, str) for key in raw_document): - raise DocForgeError( - "invalid_compilation_database", - "Each C++ compilation entry must have string keys", - entry=index, - ) - document = cast("dict[str, object]", raw_document) - keys = set(document) - if ( - not keys <= _COMPILE_ENTRY_KEYS - or not {"directory", "file"} <= keys - or (("command" in keys) == ("arguments" in keys)) - ): - raise DocForgeError( - "invalid_compilation_database", - "Each C++ compilation entry requires directory, file, and exactly " - "one of command or arguments", - entry=index, - ) - directory_value = self._bounded_string( - document["directory"], - "directory", - index, - maximum=self.max_command_chars, - ) - file_value = self._bounded_string( - document["file"], - "file", - index, - maximum=self.max_command_chars, - ) - directory_path = Path(directory_value) - directory_candidate = ( - directory_path if directory_path.is_absolute() else self.root / directory_path - ) - directory = Path(os.path.abspath(directory_candidate)) - if not directory.is_relative_to(self.root): - raise DocForgeError( - "path_escape", - "C++ compilation entry directory escapes the project root", - entry=index, - ) - self._validate_real_directory(directory, index) - file_path = Path(file_value) - file_candidate = file_path if file_path.is_absolute() else directory / file_path - source = Path(os.path.abspath(file_candidate)) - if ( - not source.is_relative_to(self.root) - or source.suffix.casefold() not in _TRANSLATION_UNIT_SUFFIXES - ): - raise DocForgeError( - "path_escape", - "C++ compilation entry file must be a confined C++ translation unit", - entry=index, - ) - relative = source.relative_to(self.root).as_posix() - self._source_root_for(PurePosixPath(relative)) - self._validate_real_file( - source, - relative, - unavailable_code="invalid_compilation_database", - label="C++ translation unit", - ) - command: str | None = None - arguments: tuple[str, ...] | None = None - if "command" in value: - command = self._bounded_string( - document["command"], - "command", - index, - maximum=self.max_command_chars, - ) - else: - raw_arguments = document["arguments"] - if not isinstance(raw_arguments, list): - raise DocForgeError( - "invalid_compilation_database", - "C++ compilation arguments must be a non-empty bounded string array", - entry=index, - ) - argument_items = cast("list[object]", raw_arguments) - if not argument_items or len(argument_items) > self.max_arguments: - raise DocForgeError( - "invalid_compilation_database", - "C++ compilation arguments must be a non-empty bounded string array", - entry=index, - ) - arguments = tuple( - self._bounded_string( - argument, - "arguments", - index, - maximum=self.max_command_chars, - ) - for argument in argument_items - ) - output = ( - self._bounded_string( - document["output"], - "output", - index, - maximum=self.max_command_chars, - ) - if "output" in value - else None - ) - directory_relative = directory.relative_to(self.root).as_posix() or "." - return _CompilationEntry( - source_path=relative, - directory=directory_relative, - command=command, - arguments=arguments, - output=output, - ) - - @staticmethod - def _unique_json_object(pairs: list[tuple[str, object]]) -> dict[str, object]: - result: dict[str, object] = {} - for key, value in pairs: - if key in result: - raise ValueError(f"duplicate JSON key: {key}") - result[key] = value - return result - - @staticmethod - def _bounded_string( - value: object, - field: str, - index: int, - *, - maximum: int, - ) -> str: - if ( - not isinstance(value, str) - or not value - or len(value) > maximum - or any(ord(character) < 32 or ord(character) == 127 for character in value) - ): - raise DocForgeError( - "invalid_compilation_database", - f"C++ compilation entry {field} must be a non-empty bounded control-free string", - entry=index, - ) - return value - - def _validate_real_directory(self, path: Path, index: int) -> None: - try: - before = path.lstat() - resolved = path.resolve(strict=True) - except OSError as error: - raise DocForgeError( - "invalid_compilation_database", - "C++ compilation entry directory does not exist", - entry=index, - ) from error - if ( - stat.S_ISLNK(before.st_mode) - or not stat.S_ISDIR(before.st_mode) - or resolved != path - or not resolved.is_relative_to(self.root) - ): - raise DocForgeError( - "path_escape", - "C++ compilation entry directory must be a confined real directory", - entry=index, - ) - - def _inventory(self) -> tuple[_SourceRecord, ...]: - entries = self._compilation_entries() - records: dict[str, _SourceRecord] = {} - queue: list[tuple[str, _CompilationEntry | None]] = [ - (entry.source_path, entry) for entry in entries - ] - header_count = 0 - while queue: - source_path, compilation_entry = queue.pop(0) - existing = records.get(source_path) - if existing is not None: - if ( - compilation_entry is not None - and existing.compilation_entry != compilation_entry - ): - raise DocForgeError( - "duplicate_translation_unit", - "C++ source has conflicting compilation entries", - source=source_path, - ) - continue - record = self._read_source(source_path, compilation_entry=compilation_entry) - dependencies: list[str] = [] - for include in self._quoted_includes(record): - resolved = self._resolve_quoted_header(source_path, include) - if resolved is None or resolved == source_path: - continue - dependencies.append(resolved) - if resolved not in records and all(item[0] != resolved for item in queue): - header_count += 1 - if header_count > self.max_headers: - raise DocForgeError( - "adapter_too_large", - "C++ project-local header inventory exceeds the configured limit", - maximum=self.max_headers, - ) - queue.append((resolved, None)) - records[source_path] = replace( - record, - dependencies=tuple(sorted(set(dependencies))), - ) - queue.sort(key=lambda item: item[0]) - return tuple(sorted(records.values(), key=lambda record: record.source_path)) - - def _read_source( - self, - relative: str, - *, - compilation_entry: _CompilationEntry | None, - ) -> _SourceRecord: - path = PurePosixPath(relative) - translation_unit = compilation_entry is not None - allowed_suffixes = _TRANSLATION_UNIT_SUFFIXES if translation_unit else _HEADER_SUFFIXES - if ( - path.is_absolute() - or ".." in path.parts - or path.suffix.casefold() not in allowed_suffixes - ): - raise DocForgeError("path_escape", "C++ source path is unsafe", source=relative) - self._source_root_for(path) - candidate = self.root.joinpath(*path.parts) - raw = self._read_real_file( - candidate, - relative, - maximum=self.max_source_bytes, - unavailable_code="stale_adapter_source", - label="C++ source", - ) - try: - text = raw.decode("utf-8-sig") - except UnicodeDecodeError as error: - raise DocForgeError( - "invalid_cpp_source", - "C++ source must use UTF-8", - source=relative, - ) from error - parse_raw, root = self._parse(text) - if root.has_error: - raise DocForgeError( - "invalid_cpp_source", - "C++ source cannot be parsed by the reference adapter", - source=relative, - line=self._first_error_line(root) or 1, - ) - source_id = self._source_id(relative, translation_unit=translation_unit) - digest = hashlib.sha256() - digest.update(CPP_EXTRACTOR_VERSION.encode()) - digest.update(b"\0source\0") - digest.update(raw) - if compilation_entry is not None: - digest.update(b"\0compilation-entry\0") - digest.update(compilation_entry.canonical_bytes()) - return _SourceRecord( - source_id=source_id, - source_path=relative, - translation_unit=translation_unit, - fingerprint=digest.hexdigest(), - text=text, - raw=parse_raw, - root=root, - compilation_entry=compilation_entry, - ) - - @staticmethod - def _parse(source: str) -> tuple[bytes, _TreeNode]: - try: - tree_sitter = importlib.import_module("tree_sitter") - except ModuleNotFoundError as error: - raise DocForgeError( - "optional_dependency_missing", - "The requested language frontend is not installed", - extra="cpp", - install="docforge[cpp]", - missing_module=error.name, - ) from error - raw = source.encode("utf-8") - tree = tree_sitter.Parser(_cpp_language()).parse(raw) - return raw, cast("_TreeNode", tree.root_node) - - @classmethod - def _first_error_line(cls, node: _TreeNode) -> int | None: - if node.is_error or node.is_missing: - return node.start_point.row + 1 - for child in node.named_children: - line = cls._first_error_line(child) - if line is not None: - return line - return None - - def _resolve_quoted_header(self, source_path: str, include: str) -> str | None: - if ( - not include - or "\\" in include - or "\x00" in include - or PurePosixPath(include).is_absolute() - ): - return None - source_parent = self.root.joinpath(*PurePosixPath(source_path).parent.parts) - candidate = Path(os.path.abspath(source_parent / include)) - if not candidate.is_relative_to(self.root): - raise DocForgeError( - "path_escape", - "Quoted C++ include escapes the project root", - source=source_path, - include=include, - ) - if candidate.is_symlink(): - raise DocForgeError( - "path_escape", - "Quoted C++ include resolves through a symbolic link", - source=source_path, - include=include, - ) - if not candidate.exists(): - return None - if candidate.suffix.casefold() not in _HEADER_SUFFIXES: - return None - relative = candidate.relative_to(self.root).as_posix() - self._source_root_for(PurePosixPath(relative)) - self._validate_real_file( - candidate, - relative, - unavailable_code="invalid_cpp_source", - label="Quoted C++ header", - ) - return relative - - @classmethod - def _quoted_includes(cls, record: _SourceRecord) -> tuple[str, ...]: - result: set[str] = set() - stack = [record.root] - while stack: - node = stack.pop() - if node.type == "preproc_include": - path = node.child_by_field_name("path") - if path is not None: - value = cls._text(path, record.raw) - if ( - len(value) >= 2 - and value.startswith('"') - and value.endswith('"') - and "\\" not in value - and "\x00" not in value - ): - result.add(value[1:-1]) - stack.extend(node.named_children) - return tuple(sorted(result)) - - def _extract_record( - self, - record: _SourceRecord, - source: AdapterSource, - ) -> AdapterSourceProjection: - file_id = self._file_node_id(source.source_id) - root_id = file_id - nodes: list[AdapterNode] = [ - self._node( - node_id=file_id, - title=record.source_path, - kind="file", - qualified_name=record.source_path, - content=record.text, - source_path=record.source_path, - anchor="L1", - ) - ] - edges: list[AdapterEdge] = [] - if record.translation_unit: - root_id = self._translation_unit_node_id(source.source_id) - nodes.append( - self._node( - node_id=root_id, - title=record.source_path, - kind="translation_unit", - qualified_name=record.source_path, - content=f"C++ translation unit {record.source_path}.", - source_path=record.source_path, - anchor="L1", - ) - ) - edges.append(self._edge(root_id, "contains", file_id, "compilation_database")) - definitions = self._definitions(record) - definition_ids = { - definition.qualified_name: definition.node_id for definition in definitions - } - for definition in definitions: - parent_id = ( - definition_ids.get(definition.parent_qualified_name, file_id) - if definition.parent_qualified_name is not None - else file_id - ) - anchor = f"L{definition.line}" - nodes.append( - self._node( - node_id=definition.node_id, - title=definition.qualified_name, - kind=definition.kind, - qualified_name=definition.qualified_name, - content=self._text(definition.node, record.raw), - source_path=record.source_path, - anchor=anchor, - ) - ) - edges.append( - self._edge( - parent_id, - "contains", - definition.node_id, - "tree_sitter_syntax", - anchor=anchor, - ) - ) - for dependency in source.dependencies: - edges.append( - self._edge( - file_id, - "depends_on", - self._file_node_id(dependency), - "quoted_project_header", - ) - ) - owners = tuple( - TreeSitterLogicOwner( - owner_node_id=definition.node_id, - qualified_name=cast(str, definition.logic_qualified_name), - line=definition.line, - ) - for definition in definitions - if definition.kind == "function" - ) - logic = analyze_cpp_source( - record.text, - source_id=source.source_id, - owners=owners, - filename=record.source_path, - max_nodes_per_function=self.max_logic_nodes_per_function, - ) - return AdapterSourceProjection( - source_id=source.source_id, - fingerprint=source.fingerprint, - nodes=tuple(sorted(nodes, key=lambda item: item.node.node_id)), - edges=tuple( - sorted( - edges, - key=lambda item: ( - item.edge.source_id, - item.edge.relation, - item.edge.target_id, - ), - ) - ), - logic=logic, - ) - - def _definitions(self, record: _SourceRecord) -> tuple[_Definition, ...]: - definitions: list[_Definition] = [] - identities: set[tuple[str, int, str]] = set() - - def visit(node: _TreeNode, scopes: tuple[str, ...]) -> None: - definition = self._definition(record, node, scopes) - next_scopes = scopes - if definition is not None: - identity = ( - definition.qualified_name, - definition.line, - definition.kind, - ) - if identity in identities: - raise DocForgeError( - "ambiguous_cpp_symbol", - "C++ source repeats a syntax-local symbol identity", - source=record.source_path, - qualified_name=definition.qualified_name, - line=definition.line, - ) - identities.add(identity) - definitions.append(definition) - if definition.kind != "function": - next_scopes = tuple(definition.qualified_name.split(".")) - for child in node.named_children: - visit(child, next_scopes) - - visit(record.root, ()) - ordered = tuple(sorted(definitions, key=lambda item: item.node_id)) - public_functions = { - (function.qualified_name, function.line) - for function in discover_cpp_functions(record.text) - } - extracted_functions = { - (cast(str, definition.logic_qualified_name), definition.line) - for definition in ordered - if definition.kind == "function" - } - if extracted_functions != public_functions: - raise DocForgeError( - "invalid_adapter", - "C++ function discovery drifted from the Logic frontend", - source=record.source_path, - ) - if len(public_functions) != len( - [definition for definition in ordered if definition.kind == "function"] - ): - raise DocForgeError( - "ambiguous_cpp_symbol", - "C++ source has functions that cannot receive unique syntax-local Logic owners", - source=record.source_path, - ) - return ordered - - def _definition( - self, - record: _SourceRecord, - node: _TreeNode, - scopes: tuple[str, ...], - ) -> _Definition | None: - kind = _SCOPE_TYPES.get(node.type) - name: str | None = None - logic_name: str | None = None - if kind is not None: - name = self._field_text(node, "name", record.raw) - elif node.type == "function_definition": - declarator = node.child_by_field_name("declarator") - logic_name = self._declarator_name(declarator, record.raw) - name = logic_name - kind = "function" if name is not None else None - if kind is None or name is None: - return None - normalized_name = name.replace("::", ".") - if "::" in name: - qualified_name = normalized_name - else: - qualified_name = ".".join((*scopes, normalized_name)) - if kind == "function": - logic_name = ".".join((*scopes, name)) - parent = ".".join(scopes) or None - line = node.start_point.row + 1 - return _Definition( - kind=kind, - qualified_name=qualified_name, - logic_qualified_name=logic_name if kind == "function" else None, - line=line, - parent_qualified_name=parent, - node=node, - node_id=self._symbol_node_id( - kind, - record.source_id, - qualified_name, - line=line, - ), - ) - - @classmethod - def _declarator_name(cls, node: _TreeNode | None, raw: bytes) -> str | None: - if node is None: - return None - if node.type in { - "identifier", - "field_identifier", - "operator_name", - "destructor_name", - "qualified_identifier", - }: - return cls._text(node, raw) - for field in ("declarator", "name"): - child = node.child_by_field_name(field) - result = cls._declarator_name(child, raw) - if result: - return result - for child in node.named_children: - result = cls._declarator_name(child, raw) - if result: - return result - return None - - @staticmethod - def _source_id(relative: str, *, translation_unit: bool) -> str: - source_kind = "tu" if translation_unit else "header" - digest = hashlib.sha256( - f"{CPP_IDENTITY_VERSION}\0{source_kind}\0{relative}".encode() - ).hexdigest()[:24] - return f"cpp.{source_kind}.{digest}" - - @staticmethod - def _file_node_id(source_id: str) -> str: - if source_id.startswith("cpp.tu."): - return source_id.replace("cpp.tu.", "cpp.file.", 1) - return source_id.replace("cpp.header.", "cpp.file.", 1) - - @staticmethod - def _translation_unit_node_id(source_id: str) -> str: - return source_id.replace("cpp.tu.", "cpp.translation-unit.", 1) - - @staticmethod - def _symbol_node_id( - kind: str, - source_id: str, - qualified_name: str, - *, - line: int, - ) -> str: - digest = hashlib.sha256( - (f"{CPP_IDENTITY_VERSION}\0{kind}\0{source_id}\0{qualified_name}\0{line}").encode() - ).hexdigest()[:24] - return f"cpp.{kind}.{digest}" - - @staticmethod - def _source_hash(sources: Sequence[AdapterSource]) -> str: - digest = hashlib.sha256() - digest.update(CPP_ADAPTER_VERSION.encode()) - digest.update(CPP_EXTRACTOR_VERSION.encode()) - digest.update(CPP_IDENTITY_VERSION.encode()) - for source in sources: - digest.update(b"\0source\0") - digest.update(source.source_id.encode()) - digest.update(b"\0path\0") - digest.update(source.source_path.encode()) - digest.update(b"\0fingerprint\0") - digest.update(source.fingerprint.encode()) - for dependency in source.dependencies: - digest.update(b"\0dependency\0") - digest.update(dependency.encode()) - return digest.hexdigest() - - @staticmethod - def _text(node: _TreeNode, raw: bytes) -> str: - return raw[node.start_byte : node.end_byte].decode("utf-8") - - @classmethod - def _field_text(cls, node: _TreeNode, field: str, raw: bytes) -> str | None: - child = node.child_by_field_name(field) - return cls._text(child, raw) if child is not None else None - - @staticmethod - def _node( - *, - node_id: str, - title: str, - kind: str, - qualified_name: str, - content: str, - source_path: str, - anchor: str, - ) -> AdapterNode: - normalized = content.strip() or f"C++ {kind} {qualified_name}." - return AdapterNode( - node=GraphNode( - node_id=node_id, - title=title, - family="code", - authority="derived", - status="active", - tags=tuple(sorted({"cpp", kind})), - summary=f"C++ {kind} syntax fact for {qualified_name}.", - content=normalized, - source_path=source_path, - source_anchor=anchor, - content_hash=hashlib.sha256(normalized.encode()).hexdigest(), - ), - metadata=( - ("evidence", "tree-sitter-syntax"), - ("extractor", CPP_EXTRACTOR_VERSION), - ("identity", CPP_IDENTITY_VERSION), - ("kind", kind), - ("language", "cpp"), - ("qualified_name", qualified_name), - ), - ) - - @staticmethod - def _edge( - source_id: str, - relation: str, - target_id: str, - evidence: str, - *, - anchor: str | None = None, - ) -> AdapterEdge: - metadata = [("evidence", evidence)] - if anchor is not None: - metadata.append(("source_anchor", anchor)) - return AdapterEdge( - edge=Edge(source_id, relation, target_id), - metadata=tuple(metadata), - ) - - def _validate_real_file( - self, - path: Path, - relative: str, - *, - unavailable_code: str, - label: str, - ) -> os.stat_result: - try: - before = path.lstat() - resolved = path.resolve(strict=True) - except OSError as error: - raise DocForgeError( - unavailable_code, - f"{label} is unavailable", - source=relative, - ) from error - if ( - stat.S_ISLNK(before.st_mode) - or not stat.S_ISREG(before.st_mode) - or resolved != path - or not resolved.is_relative_to(self.root) - ): - raise DocForgeError( - "path_escape", - f"{label} must be a confined real regular file", - source=relative, - ) - return before - - def _read_real_file( - self, - path: Path, - relative: str, - *, - maximum: int, - unavailable_code: str, - label: str, - ) -> bytes: - before = self._validate_real_file( - path, - relative, - unavailable_code=unavailable_code, - label=label, - ) - if before.st_size > maximum: - raise DocForgeError( - "source_too_large", - f"{label} exceeds the configured adapter limit", - source=relative, - maximum=maximum, - ) - descriptor = -1 - try: - flags = os.O_RDONLY | getattr(os, "O_BINARY", 0) - flags |= getattr(os, "O_NOFOLLOW", 0) - descriptor = os.open(path, flags) - opened = os.fstat(descriptor) - if not stat.S_ISREG(opened.st_mode) or (opened.st_dev, opened.st_ino) != ( - before.st_dev, - before.st_ino, - ): - raise DocForgeError( - "path_escape", - f"{label} identity changed before it was opened", - source=relative, - ) - with os.fdopen(descriptor, "rb") as handle: - descriptor = -1 - raw = handle.read(maximum + 1) - after_open = os.fstat(handle.fileno()) - after = path.lstat() - except DocForgeError: - raise - except OSError as error: - raise DocForgeError( - unavailable_code, - f"{label} changed while being read", - source=relative, - ) from error - finally: - if descriptor >= 0: - os.close(descriptor) - identities = ( - (before.st_dev, before.st_ino, before.st_size, before.st_mtime_ns), - ( - after_open.st_dev, - after_open.st_ino, - after_open.st_size, - after_open.st_mtime_ns, - ), - (after.st_dev, after.st_ino, after.st_size, after.st_mtime_ns), - ) - if ( - len(raw) > maximum - or identities[0] != identities[1] - or identities[0] != identities[2] - or len(raw) != after.st_size - ): - raise DocForgeError( - unavailable_code, - f"{label} changed while being read", - source=relative, - ) - return raw diff --git a/src/docforge/adapters/javascript.py b/src/docforge/adapters/javascript.py deleted file mode 100644 index 37f9238..0000000 --- a/src/docforge/adapters/javascript.py +++ /dev/null @@ -1,1157 +0,0 @@ -"""Deterministic JavaScript and TypeScript reference adapter. - -The adapter parses explicitly declared source roots as untrusted data with the -optional pinned Tree-sitter frontends. It never imports, compiles, transpiles, -or executes project code. Only relative static imports and re-exports that -resolve to another inventoried source file become dependency evidence. -""" - -from __future__ import annotations - -import hashlib -import importlib -import os -import posixpath -import stat -from collections.abc import Iterable, Sequence -from dataclasses import dataclass -from functools import lru_cache -from pathlib import Path, PurePosixPath -from typing import TYPE_CHECKING, Protocol, cast - -from ..adapter_sdk import ( - AdapterAssembly, - AdapterEdge, - AdapterManifest, - AdapterNode, - AdapterProjection, - AdapterSource, - AdapterSourceProjection, - Edge, -) -from ..adapter_sdk import ( - Node as GraphNode, -) -from ..errors import DocForgeError -from ..treesitter_logic import ( - DiscoveredFunction, - TreeSitterLogicOwner, - analyze_javascript_source, - analyze_typescript_source, - discover_javascript_functions, - discover_typescript_functions, -) - -if TYPE_CHECKING: - from tree_sitter import Language - -JAVASCRIPT_ADAPTER_ID = "docforge.reference.javascript" -JAVASCRIPT_ADAPTER_VERSION = "1" -JAVASCRIPT_EXTRACTOR_VERSION = "tree-sitter-script@1" -JAVASCRIPT_IDENTITY_VERSION = "javascript-reference-id@1" -JAVASCRIPT_SUPPORT_SCHEMA_VERSION = 1 - -_JAVASCRIPT_SUFFIXES = frozenset({".js", ".mjs", ".cjs"}) -_TYPESCRIPT_SUFFIXES = frozenset({".ts", ".mts", ".cts"}) -_SOURCE_SUFFIXES = (*sorted(_JAVASCRIPT_SUFFIXES), *sorted(_TYPESCRIPT_SUFFIXES)) -_FUNCTION_TYPES = frozenset( - {"function_declaration", "generator_function_declaration", "method_definition"} -) - -_SUPPORTED_FACTS = ( - "script_file", - "script_module", - "script_class", - "script_function", - "lexical_containment", - "project_local_static_import_dependency", - "function_logic", -) - - -@dataclass(frozen=True) -class JavaScriptUnsupportedFact: - """One semantic fact this syntax-only reference adapter does not claim.""" - - code: str - description: str - - def as_dict(self) -> dict[str, str]: - return {"code": self.code, "description": self.description} - - -_UNSUPPORTED_FACTS = ( - JavaScriptUnsupportedFact( - "call_resolution", - "Calls are represented only inside function Logic and are not resolved to symbols.", - ), - JavaScriptUnsupportedFact( - "dynamic_module_resolution", - "Dynamic import(), require(), package exports, and loader hooks are not dependencies.", - ), - JavaScriptUnsupportedFact( - "inheritance_resolution", - "Class extends and implements clauses are not resolved.", - ), - JavaScriptUnsupportedFact( - "module_configuration_resolution", - "Bare specifiers, aliases, tsconfig paths, and JavaScript-to-TypeScript " - "remapping are omitted.", - ), - JavaScriptUnsupportedFact( - "runtime_generated_facts", - "Decorators, prototypes, proxies, eval, and executed module code are never evaluated.", - ), - JavaScriptUnsupportedFact( - "type_and_symbol_resolution", - "Types, interfaces, overloads, variables, references, and re-exported " - "symbols are not resolved.", - ), -) - - -class _Point(Protocol): - row: int - - -class _TreeNode(Protocol): - type: str - has_error: bool - is_error: bool - is_missing: bool - start_byte: int - end_byte: int - start_point: _Point - named_children: list[_TreeNode] - - def child_by_field_name(self, name: str, /) -> _TreeNode | None: ... - - -@dataclass(frozen=True) -class _SourceRecord: - source_id: str - source_path: str - module_name: str - language: str - fingerprint: str - text: str - raw: bytes - root: _TreeNode | None - - -@dataclass(frozen=True) -class _Definition: - kind: str - qualified_name: str - line: int - parent_qualified_name: str | None - node: _TreeNode - node_id: str - - -@lru_cache(maxsize=2) -def _language(language: str) -> Language: - if language == "javascript": - grammar_module = "tree_sitter_javascript" - grammar_function = "language" - extra = "javascript" - elif language == "typescript": - grammar_module = "tree_sitter_typescript" - grammar_function = "language_typescript" - extra = "typescript" - else: - raise ValueError(f"Unsupported script language: {language}") - try: - tree_sitter = importlib.import_module("tree_sitter") - grammar = importlib.import_module(grammar_module) - except ModuleNotFoundError as error: - raise DocForgeError( - "optional_dependency_missing", - "The requested language frontend is not installed", - extra=extra, - install=f"docforge[{extra}]", - missing_module=error.name, - ) from error - language_type = tree_sitter.Language - language_factory = getattr(grammar, grammar_function) - return cast("Language", language_type(language_factory())) - - -class JavaScriptReferenceAdapter: - """Reference JavaScript/TypeScript adapter over explicit source roots.""" - - def __init__( - self, - root: Path, - *, - source_roots: Iterable[str | Path], - project_id: str = "javascript-reference", - title: str = "JavaScript and TypeScript reference project", - max_sources: int = 4_096, - max_source_bytes: int = 1_000_000, - max_logic_nodes_per_function: int = 2_000, - ) -> None: - resolved_root = root.resolve(strict=True) - if not resolved_root.is_dir(): - raise DocForgeError("invalid_adapter", "Script adapter root must be a directory") - if max_sources < 1 or max_source_bytes < 1 or max_logic_nodes_per_function < 2: - raise ValueError("Script adapter limits must be positive") - self.root = resolved_root - self.project_id = project_id - self.title = title - self.max_sources = max_sources - self.max_source_bytes = max_source_bytes - self.max_logic_nodes_per_function = max_logic_nodes_per_function - self.source_roots = self._normalize_source_roots(source_roots) - - def support_report(self) -> dict[str, object]: - """Return deterministic scope, frontend, and limitation evidence.""" - - return { - "schema_version": JAVASCRIPT_SUPPORT_SCHEMA_VERSION, - "adapter_id": JAVASCRIPT_ADAPTER_ID, - "adapter_version": JAVASCRIPT_ADAPTER_VERSION, - "extractor_version": JAVASCRIPT_EXTRACTOR_VERSION, - "identity_version": JAVASCRIPT_IDENTITY_VERSION, - "frontends": { - "javascript": "tree-sitter-javascript", - "typescript": "tree-sitter-typescript", - }, - "imports_project_code": False, - "executes_project_code": False, - "source_roots": list(self.source_roots), - "source_suffixes": list(_SOURCE_SUFFIXES), - "supported_facts": list(_SUPPORTED_FACTS), - "unsupported_facts": [fact.as_dict() for fact in _UNSUPPORTED_FACTS], - } - - def unsupported_facts(self) -> tuple[JavaScriptUnsupportedFact, ...]: - return _UNSUPPORTED_FACTS - - def load_manifest(self) -> AdapterManifest: - records = self._inventory(parse=False) - path_sources = {record.source_path: record.source_id for record in records} - sources = tuple( - sorted( - ( - AdapterSource( - source_id=record.source_id, - source_path=record.source_path, - fingerprint=record.fingerprint, - extractor_version=(f"{JAVASCRIPT_EXTRACTOR_VERSION}:{record.language}"), - dependencies=self._local_dependencies(record, path_sources), - ) - for record in records - ), - key=lambda source: source.source_id, - ) - ) - source_hash = self._source_hash(sources) - return AdapterManifest( - project_id=self.project_id, - title=self.title, - adapter_id=JAVASCRIPT_ADAPTER_ID, - adapter_version=JAVASCRIPT_ADAPTER_VERSION, - root=self.root, - revision=source_hash[:12], - source_hash=source_hash, - families=("code",), - allowed_relations=("contains", "depends_on"), - sources=sources, - estimated_nodes=max(1, len(sources) * 8), - ) - - def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: - record = self._read_source(source.source_path) - if record.source_id != source.source_id or record.fingerprint != source.fingerprint: - raise DocForgeError( - "stale_adapter_source", - "Script source changed after its manifest was captured", - source=source.source_path, - ) - return self._extract_record(record, source) - - def assemble_projection( - self, - manifest: AdapterManifest, - contributions: tuple[AdapterSourceProjection, ...], - ) -> AdapterAssembly: - expected = tuple(source.source_id for source in manifest.sources) - actual = tuple(sorted(contribution.source_id for contribution in contributions)) - if actual != expected: - raise DocForgeError( - "invalid_adapter", - "Script assembly contributions do not match the current manifest", - ) - nodes = tuple( - sorted( - (node for contribution in contributions for node in contribution.nodes), - key=lambda item: item.node.node_id, - ) - ) - edges = tuple( - sorted( - (edge for contribution in contributions for edge in contribution.edges), - key=lambda item: ( - item.edge.source_id, - item.edge.relation, - item.edge.target_id, - ), - ) - ) - logic = tuple( - sorted( - (projection for contribution in contributions for projection in contribution.logic), - key=lambda projection: projection.owner_node_id, - ) - ) - return AdapterAssembly( - projection=AdapterProjection( - project_id=manifest.project_id, - title=manifest.title, - adapter_id=manifest.adapter_id, - adapter_version=manifest.adapter_version, - root=manifest.root, - revision=manifest.revision, - source_hash=manifest.source_hash, - nodes=nodes, - edges=edges, - ), - logic=logic, - ) - - def load_assembly(self) -> AdapterAssembly: - """Load one cache-independent complete graph and Logic assembly.""" - - manifest = self.load_manifest() - contributions = tuple(self.extract_source(source) for source in manifest.sources) - if self.load_manifest() != manifest: - raise DocForgeError( - "source_changed", - "Script sources changed during complete adapter extraction", - ) - return self.assemble_projection(manifest, contributions) - - def load_complete_assembly(self) -> AdapterAssembly: - return self.load_assembly() - - def load_projection(self) -> AdapterProjection: - return self.load_assembly().projection - - def _normalize_source_roots(self, source_roots: Iterable[str | Path]) -> tuple[str, ...]: - normalized: list[str] = [] - for value in source_roots: - candidate = PurePosixPath(Path(value).as_posix()) - if ( - candidate.is_absolute() - or not candidate.parts - or ".." in candidate.parts - or str(candidate) in {"", "."} - ): - raise DocForgeError( - "path_escape", - "Script source roots must be explicit project-relative directories", - ) - relative = candidate.as_posix() - absolute = self.root.joinpath(*candidate.parts) - if absolute.is_symlink(): - raise DocForgeError("path_escape", "Script source root cannot be a symbolic link") - try: - resolved = absolute.resolve(strict=True) - except OSError as error: - raise DocForgeError( - "invalid_adapter", - "Script source root does not exist", - source_root=relative, - ) from error - if ( - resolved != absolute - or not resolved.is_dir() - or not resolved.is_relative_to(self.root) - ): - raise DocForgeError( - "path_escape", - "Script source root must be a confined real directory", - source_root=relative, - ) - normalized.append(relative) - result = tuple(sorted(set(normalized))) - if not result: - raise DocForgeError("invalid_adapter", "At least one script source root is required") - if len(result) != len(normalized): - raise DocForgeError("invalid_adapter", "Script source roots must be unique") - paths = [PurePosixPath(value) for value in result] - for index, left in enumerate(paths): - for right in paths[index + 1 :]: - if left in right.parents or right in left.parents: - raise DocForgeError( - "invalid_adapter", - "Script source roots must not overlap", - ) - return result - - def _inventory(self, *, parse: bool = True) -> tuple[_SourceRecord, ...]: - paths: list[str] = [] - for source_root in self.source_roots: - absolute_root = self.root.joinpath(*PurePosixPath(source_root).parts) - if ( - absolute_root.is_symlink() - or not absolute_root.is_dir() - or absolute_root.resolve(strict=True) != absolute_root - ): - raise DocForgeError( - "path_escape", - "Script source root changed after adapter binding", - source_root=source_root, - ) - for directory, directory_names, file_names in os.walk( - absolute_root, - topdown=True, - followlinks=False, - ): - current = Path(directory) - for name in tuple(directory_names): - child = current / name - if child.is_symlink(): - raise DocForgeError( - "path_escape", - "Script source inventory contains a symbolic-link directory", - source=child.relative_to(self.root).as_posix(), - ) - directory_names[:] = sorted( - name for name in directory_names if name != "node_modules" - ) - for name in sorted(file_names): - if Path(name).suffix not in _SOURCE_SUFFIXES: - continue - child = current / name - relative = child.relative_to(self.root).as_posix() - if child.is_symlink(): - raise DocForgeError( - "path_escape", - "Script source inventory contains a symbolic-link file", - source=relative, - ) - paths.append(relative) - if len(paths) > self.max_sources: - raise DocForgeError( - "adapter_too_large", - "Script source inventory exceeds the configured limit", - maximum=self.max_sources, - ) - records = tuple(self._read_source(path, parse=parse) for path in sorted(paths)) - modules = [record.module_name for record in records] - if len(modules) != len(set(modules)): - duplicates = sorted(module for module in set(modules) if modules.count(module) > 1) - raise DocForgeError( - "ambiguous_script_module", - "Declared script source roots produce duplicate module names", - modules=duplicates, - ) - return records - - def _read_source(self, relative: str, *, parse: bool = True) -> _SourceRecord: - path = PurePosixPath(relative) - if path.is_absolute() or ".." in path.parts or path.suffix not in _SOURCE_SUFFIXES: - raise DocForgeError("path_escape", "Script source path is unsafe") - source_root = self._source_root_for(path) - candidate = self.root.joinpath(*path.parts) - try: - before = candidate.lstat() - except OSError as error: - raise DocForgeError( - "stale_adapter_source", - "Script source is unavailable", - source=relative, - ) from error - if ( - stat.S_ISLNK(before.st_mode) - or not stat.S_ISREG(before.st_mode) - or candidate.resolve(strict=True) != candidate - or not candidate.is_relative_to(self.root) - ): - raise DocForgeError( - "path_escape", - "Script source must be a confined regular file", - source=relative, - ) - if before.st_size > self.max_source_bytes: - raise DocForgeError( - "source_too_large", - "Script source exceeds the configured adapter limit", - source=relative, - maximum=self.max_source_bytes, - ) - descriptor = -1 - try: - flags = os.O_RDONLY | getattr(os, "O_BINARY", 0) - flags |= getattr(os, "O_NOFOLLOW", 0) - descriptor = os.open(candidate, flags) - opened = os.fstat(descriptor) - if not stat.S_ISREG(opened.st_mode) or (opened.st_dev, opened.st_ino) != ( - before.st_dev, - before.st_ino, - ): - raise DocForgeError( - "path_escape", - "Script source identity changed before it was opened", - source=relative, - ) - with os.fdopen(descriptor, "rb") as handle: - descriptor = -1 - raw = handle.read(self.max_source_bytes + 1) - after_open = os.fstat(handle.fileno()) - after = candidate.lstat() - except DocForgeError: - raise - except OSError as error: - raise DocForgeError( - "stale_adapter_source", - "Script source changed while being read", - source=relative, - ) from error - finally: - if descriptor >= 0: - os.close(descriptor) - identities = ( - (before.st_dev, before.st_ino, before.st_size, before.st_mtime_ns), - ( - after_open.st_dev, - after_open.st_ino, - after_open.st_size, - after_open.st_mtime_ns, - ), - (after.st_dev, after.st_ino, after.st_size, after.st_mtime_ns), - ) - if ( - len(raw) > self.max_source_bytes - or identities[0] != identities[1] - or identities[0] != identities[2] - or len(raw) != after.st_size - ): - raise DocForgeError( - "stale_adapter_source", - "Script source changed while being read", - source=relative, - ) - try: - text = raw.decode("utf-8-sig") - except UnicodeDecodeError as error: - raise DocForgeError( - "invalid_script_source", - "Script source must use UTF-8", - source=relative, - ) from error - language = self._language_for_suffix(path.suffix) - parse_raw = text.encode("utf-8") - root: _TreeNode | None = None - if parse: - parse_raw, root = self._parse(text, language) - if root.has_error: - raise DocForgeError( - "invalid_script_source", - f"{language.title()} source cannot be parsed by the reference adapter", - source=relative, - line=self._first_error_line(root) or 1, - ) - return _SourceRecord( - source_id=self._source_id(relative), - source_path=relative, - module_name=self._module_name(path, PurePosixPath(source_root)), - language=language, - fingerprint=hashlib.sha256(raw).hexdigest(), - text=text, - raw=parse_raw, - root=root, - ) - - @staticmethod - def _parse(source: str, language: str) -> tuple[bytes, _TreeNode]: - try: - tree_sitter = importlib.import_module("tree_sitter") - except ModuleNotFoundError as error: - raise DocForgeError( - "optional_dependency_missing", - "The requested language frontend is not installed", - extra=language, - install=f"docforge[{language}]", - missing_module=error.name, - ) from error - raw = source.encode("utf-8") - parser = tree_sitter.Parser(_language(language)) - tree = parser.parse(raw) - return raw, cast("_TreeNode", tree.root_node) - - @classmethod - def _first_error_line(cls, node: _TreeNode) -> int | None: - if node.is_error or node.is_missing: - return node.start_point.row + 1 - for child in node.named_children: - line = cls._first_error_line(child) - if line is not None: - return line - return None - - def _source_root_for(self, path: PurePosixPath) -> str: - matches = [ - source_root - for source_root in self.source_roots - if path == PurePosixPath(source_root) or PurePosixPath(source_root) in path.parents - ] - if len(matches) != 1: - raise DocForgeError( - "path_escape", - "Script source is outside the declared source roots", - source=path.as_posix(), - ) - return matches[0] - - @staticmethod - def _language_for_suffix(suffix: str) -> str: - if suffix in _JAVASCRIPT_SUFFIXES: - return "javascript" - if suffix in _TYPESCRIPT_SUFFIXES: - return "typescript" - raise DocForgeError("invalid_adapter", "Unsupported script source suffix") - - @staticmethod - def _module_name(path: PurePosixPath, source_root: PurePosixPath) -> str: - local = path.relative_to(source_root) - parts = list(local.with_suffix("").parts) - if parts and parts[-1] == "index": - parts.pop() - if not parts: - parts = list(source_root.parts) - return ".".join(parts) - - @staticmethod - def _source_id(relative: str) -> str: - digest = hashlib.sha256( - f"{JAVASCRIPT_IDENTITY_VERSION}\0source\0{relative}".encode() - ).hexdigest()[:24] - return f"javascript.source.{digest}" - - @staticmethod - def _file_node_id(source_id: str) -> str: - return source_id.replace("javascript.source.", "javascript.file.", 1) - - @staticmethod - def _module_node_id(source_id: str) -> str: - return source_id.replace("javascript.source.", "javascript.module.", 1) - - @staticmethod - def _symbol_node_id(kind: str, source_id: str, qualified_name: str) -> str: - digest = hashlib.sha256( - (f"{JAVASCRIPT_IDENTITY_VERSION}\0{kind}\0{source_id}\0{qualified_name}").encode() - ).hexdigest()[:24] - return f"javascript.{kind}.{digest}" - - @staticmethod - def _source_hash(sources: Sequence[AdapterSource]) -> str: - digest = hashlib.sha256() - digest.update(JAVASCRIPT_ADAPTER_VERSION.encode()) - digest.update(JAVASCRIPT_EXTRACTOR_VERSION.encode()) - digest.update(JAVASCRIPT_IDENTITY_VERSION.encode()) - for source in sources: - digest.update(b"\0source\0") - digest.update(source.source_id.encode()) - digest.update(b"\0path\0") - digest.update(source.source_path.encode()) - digest.update(b"\0fingerprint\0") - digest.update(source.fingerprint.encode()) - digest.update(b"\0extractor\0") - digest.update(source.extractor_version.encode()) - for dependency in source.dependencies: - digest.update(b"\0dependency\0") - digest.update(dependency.encode()) - return digest.hexdigest() - - def _local_dependencies( - self, - record: _SourceRecord, - path_sources: dict[str, str], - ) -> tuple[str, ...]: - dependencies: set[str] = set() - for specifier in self._static_specifiers(record): - target_path = self._resolve_relative_specifier( - record.source_path, specifier, path_sources - ) - if target_path is None: - continue - target = path_sources[target_path] - if target != record.source_id: - dependencies.add(target) - return tuple(sorted(dependencies)) - - @classmethod - def _static_specifiers(cls, record: _SourceRecord) -> tuple[str, ...]: - result = { - value - for value in cls._lexical_static_specifiers(record.text) - if value.startswith(("./", "../")) - } - return tuple(sorted(result)) - - @staticmethod - def _script_tokens(source: str) -> tuple[tuple[str, str], ...]: - """Tokenize only enough JavaScript syntax to inventory static module specifiers.""" - - tokens: list[tuple[str, str]] = [] - position = 0 - length = len(source) - while position < length: - character = source[position] - if character in " \t\f\v": - position += 1 - continue - if character in "\r\n": - if character == "\r" and position + 1 < length and source[position + 1] == "\n": - position += 1 - tokens.append(("newline", "\n")) - position += 1 - continue - if source.startswith("//", position): - position += 2 - while position < length and source[position] not in "\r\n": - position += 1 - continue - if source.startswith("/*", position): - position += 2 - while position < length and not source.startswith("*/", position): - if source[position] in "\r\n": - tokens.append(("newline", "\n")) - if ( - source[position] == "\r" - and position + 1 < length - and source[position + 1] == "\n" - ): - position += 1 - position += 1 - position = min(length, position + 2) - continue - if character in {'"', "'"}: - quote = character - position += 1 - value: list[str] = [] - escaped = False - closed = False - while position < length: - current = source[position] - if current == "\\": - escaped = True - position += 2 - continue - if current == quote: - position += 1 - closed = True - break - if current in "\r\n": - break - value.append(current) - position += 1 - tokens.append(("string" if closed and not escaped else "opaque", "".join(value))) - continue - if character == "`": - position += 1 - while position < length: - current = source[position] - if current == "\\": - position += 2 - continue - position += 1 - if current == "`": - break - tokens.append(("opaque", "template")) - continue - if character.isalpha() or character in {"_", "$"}: - start = position - position += 1 - while position < length and ( - source[position].isalnum() or source[position] in {"_", "$"} - ): - position += 1 - tokens.append(("name", source[start:position])) - continue - tokens.append(("punct", character)) - position += 1 - return tuple(tokens) - - @classmethod - def _lexical_static_specifiers(cls, source: str) -> tuple[str, ...]: - tokens = cls._script_tokens(source) - result: set[str] = set() - for position, (kind, value) in enumerate(tokens): - if kind != "name" or value not in {"import", "export"}: - continue - if position and tokens[position - 1] == ("punct", "."): - continue - keyword = value - index = position + 1 - depth = 0 - clause_seen = False - line_break = False - while index < len(tokens): - token_kind, token_value = tokens[index] - if token_kind == "newline" and depth == 0: - line_break = True - if keyword == "export" and not clause_seen: - break - index += 1 - continue - if token_kind == "punct": - if token_value in "([{": - depth += 1 - clause_seen = True - elif token_value in ")]}": - depth = max(0, depth - 1) - clause_seen = True - elif token_value == ";" and depth == 0: - break - elif token_value in {"*", ","}: - clause_seen = True - elif token_kind == "string": - if keyword == "import" and not clause_seen: - result.add(token_value) - break - elif token_kind == "name": - if line_break and token_value in { - "class", - "const", - "export", - "function", - "import", - "let", - "return", - "throw", - "var", - }: - break - if keyword == "export" and token_value in { - "async", - "class", - "const", - "default", - "function", - "let", - "var", - }: - break - if token_value == "from" and clause_seen: - source_index = index + 1 - while source_index < len(tokens) and tokens[source_index][0] == "newline": - source_index += 1 - if source_index < len(tokens) and tokens[source_index][0] == "string": - result.add(tokens[source_index][1]) - break - if keyword == "import" and token_value not in {"type"}: - clause_seen = True - if token_kind != "newline": - line_break = False - index += 1 - return tuple(sorted(result)) - - @staticmethod - def _resolve_relative_specifier( - source_path: str, - specifier: str, - path_sources: dict[str, str], - ) -> str | None: - parent = PurePosixPath(source_path).parent.as_posix() - normalized = posixpath.normpath(posixpath.join(parent, specifier)) - if normalized == ".." or normalized.startswith("../") or normalized.startswith("/"): - return None - candidate = PurePosixPath(normalized) - if candidate.suffix in _SOURCE_SUFFIXES: - return normalized if normalized in path_sources else None - candidates = [ - *(f"{normalized}{suffix}" for suffix in _SOURCE_SUFFIXES), - *(f"{normalized}/index{suffix}" for suffix in _SOURCE_SUFFIXES), - ] - matches = [value for value in candidates if value in path_sources] - return matches[0] if len(matches) == 1 else None - - def _extract_record( - self, - record: _SourceRecord, - source: AdapterSource, - ) -> AdapterSourceProjection: - if record.root is None: - raise DocForgeError( - "invalid_adapter", - "Script extraction requires a parsed source record", - source=record.source_path, - ) - file_id = self._file_node_id(source.source_id) - module_id = self._module_node_id(source.source_id) - nodes: list[AdapterNode] = [ - self._node( - node_id=file_id, - title=record.source_path, - kind="file", - qualified_name=record.source_path, - language=record.language, - content=record.text, - source_path=record.source_path, - anchor="L1", - ), - self._node( - node_id=module_id, - title=record.module_name, - kind="module", - qualified_name=record.module_name, - language=record.language, - content=f"{record.language.title()} module {record.module_name}.", - source_path=record.source_path, - anchor="L1", - ), - ] - edges: list[AdapterEdge] = [self._edge(file_id, "contains", module_id, "syntax")] - definitions = self._definitions(record) - definition_ids = { - definition.qualified_name: definition.node_id for definition in definitions - } - for definition in definitions: - parent_id = ( - definition_ids.get(definition.parent_qualified_name, module_id) - if definition.parent_qualified_name is not None - else module_id - ) - anchor = f"L{definition.line}" - content = self._text(definition.node, record.raw).strip() - nodes.append( - self._node( - node_id=definition.node_id, - title=definition.qualified_name, - kind=definition.kind, - qualified_name=f"{record.module_name}.{definition.qualified_name}", - language=record.language, - content=content, - source_path=record.source_path, - anchor=anchor, - ) - ) - edges.append( - self._edge( - parent_id, - "contains", - definition.node_id, - "tree_sitter_syntax", - anchor=anchor, - ) - ) - for dependency in source.dependencies: - edges.append( - self._edge( - module_id, - "depends_on", - self._module_node_id(dependency), - "static_relative_module", - ) - ) - functions = self._discovered_functions(record) - owners = tuple( - TreeSitterLogicOwner( - owner_node_id=self._function_definition(definitions, function).node_id, - qualified_name=function.qualified_name, - line=function.line, - ) - for function in functions - ) - analyzer = ( - analyze_javascript_source - if record.language == "javascript" - else analyze_typescript_source - ) - logic = analyzer( - record.text, - source_id=source.source_id, - owners=owners, - filename=record.source_path, - max_nodes_per_function=self.max_logic_nodes_per_function, - ) - return AdapterSourceProjection( - source_id=source.source_id, - fingerprint=source.fingerprint, - nodes=tuple(sorted(nodes, key=lambda item: item.node.node_id)), - edges=tuple( - sorted( - edges, - key=lambda item: ( - item.edge.source_id, - item.edge.relation, - item.edge.target_id, - ), - ) - ), - logic=logic, - ) - - def _definitions(self, record: _SourceRecord) -> tuple[_Definition, ...]: - root = record.root - if root is None: - raise DocForgeError( - "invalid_adapter", - "Script definition extraction requires a parsed source record", - source=record.source_path, - ) - definitions: list[_Definition] = [] - seen: set[str] = set() - - def visit(node: _TreeNode, scopes: tuple[str, ...]) -> None: - definition = self._definition(record, node, scopes) - next_scopes = scopes - if definition is not None: - if definition.qualified_name in seen: - raise DocForgeError( - "ambiguous_script_symbol", - "Script source repeats a class or function identity", - source=record.source_path, - qualified_name=definition.qualified_name, - ) - seen.add(definition.qualified_name) - definitions.append(definition) - next_scopes = (*scopes, definition.qualified_name.rsplit(".", 1)[-1]) - for child in node.named_children: - visit(child, next_scopes) - - visit(root, ()) - ordered = tuple(sorted(definitions, key=lambda item: item.node_id)) - public_functions = { - (function.qualified_name, function.line) - for function in self._discovered_functions(record) - } - extracted_functions = { - (definition.qualified_name, definition.line) - for definition in ordered - if definition.kind == "function" - } - if extracted_functions != public_functions: - raise DocForgeError( - "invalid_adapter", - "Reference function discovery drifted from the Logic frontend", - source=record.source_path, - ) - return ordered - - def _definition( - self, - record: _SourceRecord, - node: _TreeNode, - scopes: tuple[str, ...], - ) -> _Definition | None: - kind: str | None = None - name: str | None = None - definition_node = node - if node.type in {"class_declaration", "class"}: - name = self._field_text(node, "name", record.raw) - kind = "class" if name is not None else None - elif node.type in _FUNCTION_TYPES: - name = self._field_text(node, "name", record.raw) - kind = "function" if name is not None else None - elif node.type == "variable_declarator": - value = node.child_by_field_name("value") - if value is not None and value.type in {"arrow_function", "function_expression"}: - name = self._field_text(node, "name", record.raw) - definition_node = value - kind = "function" if name is not None else None - if kind is None or name is None: - return None - qualified_name = ".".join((*scopes, name)) - parent = ".".join(scopes) or None - return _Definition( - kind=kind, - qualified_name=qualified_name, - line=node.start_point.row + 1, - parent_qualified_name=parent, - node=definition_node, - node_id=self._symbol_node_id(kind, record.source_id, qualified_name), - ) - - @staticmethod - def _discovered_functions(record: _SourceRecord) -> tuple[DiscoveredFunction, ...]: - if record.language == "javascript": - return discover_javascript_functions(record.text) - return discover_typescript_functions(record.text) - - @staticmethod - def _function_definition( - definitions: tuple[_Definition, ...], - function: DiscoveredFunction, - ) -> _Definition: - matches = [ - definition - for definition in definitions - if definition.kind == "function" - and definition.qualified_name == function.qualified_name - and definition.line == function.line - ] - if len(matches) != 1: - raise DocForgeError( - "invalid_adapter", - "Logic owner does not match one extracted script function", - qualified_name=function.qualified_name, - line=function.line, - ) - return matches[0] - - @staticmethod - def _text(node: _TreeNode, raw: bytes) -> str: - return raw[node.start_byte : node.end_byte].decode("utf-8") - - @classmethod - def _field_text(cls, node: _TreeNode, field: str, raw: bytes) -> str | None: - child = node.child_by_field_name(field) - return cls._text(child, raw) if child is not None else None - - @staticmethod - def _node( - *, - node_id: str, - title: str, - kind: str, - qualified_name: str, - language: str, - content: str, - source_path: str, - anchor: str, - ) -> AdapterNode: - normalized = content.strip() or f"{language.title()} {kind} {qualified_name}." - return AdapterNode( - node=GraphNode( - node_id=node_id, - title=title, - family="code", - authority="derived", - status="active", - tags=tuple(sorted({language, kind})), - summary=f"{language.title()} {kind} fact for {qualified_name}.", - content=normalized, - source_path=source_path, - source_anchor=anchor, - content_hash=hashlib.sha256(normalized.encode()).hexdigest(), - ), - metadata=( - ("extractor", JAVASCRIPT_EXTRACTOR_VERSION), - ("identity", JAVASCRIPT_IDENTITY_VERSION), - ("kind", kind), - ("language", language), - ("qualified_name", qualified_name), - ), - ) - - @staticmethod - def _edge( - source_id: str, - relation: str, - target_id: str, - evidence: str, - *, - anchor: str | None = None, - ) -> AdapterEdge: - metadata = [("evidence", evidence)] - if anchor is not None: - metadata.append(("source_anchor", anchor)) - return AdapterEdge( - edge=Edge(source_id, relation, target_id), - metadata=tuple(metadata), - ) diff --git a/src/docforge/adapters/python.py b/src/docforge/adapters/python.py deleted file mode 100644 index 6a754d6..0000000 --- a/src/docforge/adapters/python.py +++ /dev/null @@ -1,948 +0,0 @@ -"""Deterministic stdlib-AST reference adapter for explicitly confined Python roots. - -The adapter parses Python source as untrusted data. It never imports or executes -project code. Its intentionally narrow dependency model publishes only imports -that resolve to another module in the same declared source inventory. -""" - -from __future__ import annotations - -import ast -import hashlib -import io -import os -import stat -import tokenize -from collections.abc import Iterable, Sequence -from dataclasses import dataclass -from pathlib import Path, PurePosixPath - -from ..adapter_sdk import ( - AdapterAssembly, - AdapterEdge, - AdapterManifest, - AdapterNode, - AdapterProjection, - AdapterSource, - AdapterSourceProjection, - Edge, - Node, -) -from ..errors import DocForgeError -from ..python_logic import PythonLogicOwner, analyze_python_source - -PYTHON_ADAPTER_ID = "docforge.reference.python" -PYTHON_ADAPTER_VERSION = "1" -PYTHON_EXTRACTOR_VERSION = "stdlib-ast@1" -PYTHON_IDENTITY_VERSION = "python-reference-id@1" -PYTHON_SUPPORT_SCHEMA_VERSION = 1 - -_SUPPORTED_FACTS = ( - "python_file", - "python_module", - "python_class", - "python_function", - "lexical_containment", - "local_import_dependency", - "function_logic", -) - - -@dataclass(frozen=True) -class PythonUnsupportedFact: - """One semantic fact this syntax-only reference adapter does not claim.""" - - code: str - description: str - - def as_dict(self) -> dict[str, str]: - return {"code": self.code, "description": self.description} - - -_UNSUPPORTED_FACTS = ( - PythonUnsupportedFact( - "call_resolution", - "Calls are represented only inside function Logic and are not resolved to symbols.", - ), - PythonUnsupportedFact( - "dynamic_import_resolution", - "Imports performed through runtime calls are not dependency evidence.", - ), - PythonUnsupportedFact( - "inheritance_resolution", - "Class bases are syntax only and are not resolved to local or external types.", - ), - PythonUnsupportedFact( - "runtime_generated_facts", - "Decorators, metaclasses, descriptors, and executed module code are never evaluated.", - ), - PythonUnsupportedFact( - "symbol_reference_resolution", - "Imported names, variable references, types, overloads, and re-exports are not resolved.", - ), -) - - -@dataclass(frozen=True) -class _SourceRecord: - source_id: str - source_path: str - module_name: str - fingerprint: str - raw: bytes - text: str - tree: ast.Module | None - dependencies: tuple[str, ...] = () - - -@dataclass(frozen=True) -class _Definition: - kind: str - qualified_name: str - parent_node_id: str - node: ast.ClassDef | ast.FunctionDef | ast.AsyncFunctionDef - node_id: str - - -class PythonReferenceAdapter: - """Reference Python adapter over explicit, non-overlapping source roots.""" - - def __init__( - self, - root: Path, - *, - source_roots: Iterable[str | Path], - project_id: str = "python-reference", - title: str = "Python reference project", - max_sources: int = 4_096, - max_source_bytes: int = 1_000_000, - max_logic_nodes_per_function: int = 2_000, - ) -> None: - resolved_root = root.resolve(strict=True) - if not resolved_root.is_dir(): - raise DocForgeError("invalid_adapter", "Python adapter root must be a directory") - if max_sources < 1 or max_source_bytes < 1 or max_logic_nodes_per_function < 2: - raise ValueError("Python adapter limits must be positive") - self.root = resolved_root - self.project_id = project_id - self.title = title - self.max_sources = max_sources - self.max_source_bytes = max_source_bytes - self.max_logic_nodes_per_function = max_logic_nodes_per_function - self.source_roots = self._normalize_source_roots(source_roots) - - def support_report(self) -> dict[str, object]: - """Return deterministic, machine-readable scope and limitation evidence.""" - - return { - "schema_version": PYTHON_SUPPORT_SCHEMA_VERSION, - "adapter_id": PYTHON_ADAPTER_ID, - "adapter_version": PYTHON_ADAPTER_VERSION, - "extractor_version": PYTHON_EXTRACTOR_VERSION, - "identity_version": PYTHON_IDENTITY_VERSION, - "frontend": "python-stdlib-ast", - "imports_project_code": False, - "executes_project_code": False, - "source_roots": list(self.source_roots), - "supported_facts": list(_SUPPORTED_FACTS), - "unsupported_facts": [fact.as_dict() for fact in _UNSUPPORTED_FACTS], - } - - def unsupported_facts(self) -> tuple[PythonUnsupportedFact, ...]: - """Return the adapter's fixed unsupported semantic fact inventory.""" - - return _UNSUPPORTED_FACTS - - def load_manifest(self) -> AdapterManifest: - records = self._inventory(parse=False) - module_sources = {record.module_name: record.source_id for record in records} - if len(module_sources) != len(records): - raise DocForgeError( - "ambiguous_python_module", - "Declared Python source roots produce duplicate module names", - ) - sources = tuple( - sorted( - ( - AdapterSource( - source_id=record.source_id, - source_path=record.source_path, - fingerprint=record.fingerprint, - extractor_version=PYTHON_EXTRACTOR_VERSION, - dependencies=self._local_dependencies(record, module_sources), - ) - for record in records - ), - key=lambda source: source.source_id, - ) - ) - source_hash = self._source_hash(sources) - return AdapterManifest( - project_id=self.project_id, - title=self.title, - adapter_id=PYTHON_ADAPTER_ID, - adapter_version=PYTHON_ADAPTER_VERSION, - root=self.root, - revision=source_hash[:12], - source_hash=source_hash, - families=("code",), - allowed_relations=("contains", "depends_on"), - sources=sources, - estimated_nodes=max(1, len(sources) * 8), - ) - - def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: - record = self._read_source(source.source_path) - if record.source_id != source.source_id or record.fingerprint != source.fingerprint: - raise DocForgeError( - "stale_adapter_source", - "Python source changed after its manifest was captured", - source=source.source_path, - ) - return self._extract_record(record, source) - - def assemble_projection( - self, - manifest: AdapterManifest, - contributions: tuple[AdapterSourceProjection, ...], - ) -> AdapterAssembly: - expected = tuple(source.source_id for source in manifest.sources) - actual = tuple(sorted(contribution.source_id for contribution in contributions)) - if actual != expected: - raise DocForgeError( - "invalid_adapter", - "Python assembly contributions do not match the current manifest", - ) - nodes = tuple( - sorted( - (node for contribution in contributions for node in contribution.nodes), - key=lambda item: item.node.node_id, - ) - ) - edges = tuple( - sorted( - (edge for contribution in contributions for edge in contribution.edges), - key=lambda item: ( - item.edge.source_id, - item.edge.relation, - item.edge.target_id, - ), - ) - ) - logic = tuple( - sorted( - (projection for contribution in contributions for projection in contribution.logic), - key=lambda projection: projection.owner_node_id, - ) - ) - return AdapterAssembly( - projection=AdapterProjection( - project_id=manifest.project_id, - title=manifest.title, - adapter_id=manifest.adapter_id, - adapter_version=manifest.adapter_version, - root=manifest.root, - revision=manifest.revision, - source_hash=manifest.source_hash, - nodes=nodes, - edges=edges, - ), - logic=logic, - ) - - def load_assembly(self) -> AdapterAssembly: - """Load one cache-independent complete graph and Logic assembly.""" - - manifest = self.load_manifest() - contributions = tuple(self.extract_source(source) for source in manifest.sources) - stable = self.load_manifest() - if stable != manifest: - raise DocForgeError( - "source_changed", - "Python sources changed during complete adapter extraction", - ) - return self.assemble_projection(manifest, contributions) - - def load_complete_assembly(self) -> AdapterAssembly: - """Implement the SDK's independent complete-assembly oracle.""" - - return self.load_assembly() - - def load_projection(self) -> AdapterProjection: - """Preserve the first-class legacy complete-projection contract.""" - - return self.load_assembly().projection - - def _normalize_source_roots(self, source_roots: Iterable[str | Path]) -> tuple[str, ...]: - normalized: list[str] = [] - for value in source_roots: - candidate = PurePosixPath(Path(value).as_posix()) - if ( - candidate.is_absolute() - or not candidate.parts - or ".." in candidate.parts - or str(candidate) in {"", "."} - ): - raise DocForgeError( - "path_escape", - "Python source roots must be explicit project-relative directories", - ) - relative = candidate.as_posix() - absolute = self.root.joinpath(*candidate.parts) - if absolute.is_symlink(): - raise DocForgeError("path_escape", "Python source root cannot be a symbolic link") - try: - resolved = absolute.resolve(strict=True) - except OSError as error: - raise DocForgeError( - "invalid_adapter", - "Python source root does not exist", - source_root=relative, - ) from error - if ( - resolved != absolute - or not resolved.is_dir() - or not resolved.is_relative_to(self.root) - ): - raise DocForgeError( - "path_escape", - "Python source root must be a confined real directory", - source_root=relative, - ) - normalized.append(relative) - result = tuple(sorted(set(normalized))) - if not result: - raise DocForgeError("invalid_adapter", "At least one Python source root is required") - if len(result) != len(normalized): - raise DocForgeError("invalid_adapter", "Python source roots must be unique") - paths = [PurePosixPath(value) for value in result] - for index, left in enumerate(paths): - for right in paths[index + 1 :]: - if left in right.parents or right in left.parents: - raise DocForgeError( - "invalid_adapter", - "Python source roots must not overlap", - ) - return result - - def _inventory(self, *, parse: bool = True) -> tuple[_SourceRecord, ...]: - paths: list[str] = [] - for source_root in self.source_roots: - absolute_root = self.root.joinpath(*PurePosixPath(source_root).parts) - if ( - absolute_root.is_symlink() - or not absolute_root.is_dir() - or absolute_root.resolve(strict=True) != absolute_root - ): - raise DocForgeError( - "path_escape", - "Python source root changed after adapter binding", - source_root=source_root, - ) - for directory, directory_names, file_names in os.walk( - absolute_root, - topdown=True, - followlinks=False, - ): - current = Path(directory) - for name in tuple(directory_names): - child = current / name - if child.is_symlink(): - raise DocForgeError( - "path_escape", - "Python source inventory contains a symbolic-link directory", - source=child.relative_to(self.root).as_posix(), - ) - directory_names[:] = sorted( - name for name in directory_names if name != "__pycache__" - ) - for name in sorted(file_names): - if not name.endswith(".py"): - continue - child = current / name - relative = child.relative_to(self.root).as_posix() - if child.is_symlink(): - raise DocForgeError( - "path_escape", - "Python source inventory contains a symbolic-link file", - source=relative, - ) - paths.append(relative) - if len(paths) > self.max_sources: - raise DocForgeError( - "adapter_too_large", - "Python source inventory exceeds the configured limit", - maximum=self.max_sources, - ) - records = tuple(self._read_source(path, parse=parse) for path in sorted(paths)) - modules = [record.module_name for record in records] - if len(modules) != len(set(modules)): - duplicates = sorted(module for module in set(modules) if modules.count(module) > 1) - raise DocForgeError( - "ambiguous_python_module", - "Declared Python source roots produce duplicate module names", - modules=duplicates, - ) - return records - - def _read_source(self, relative: str, *, parse: bool = True) -> _SourceRecord: - path = PurePosixPath(relative) - if path.is_absolute() or ".." in path.parts or path.suffix != ".py": - raise DocForgeError("path_escape", "Python source path is unsafe") - source_root = self._source_root_for(path) - candidate = self.root.joinpath(*path.parts) - try: - before = candidate.lstat() - except OSError as error: - raise DocForgeError( - "stale_adapter_source", - "Python source is unavailable", - source=relative, - ) from error - if ( - stat.S_ISLNK(before.st_mode) - or not stat.S_ISREG(before.st_mode) - or candidate.resolve(strict=True) != candidate - or not candidate.is_relative_to(self.root) - ): - raise DocForgeError( - "path_escape", - "Python source must be a confined regular file", - source=relative, - ) - if before.st_size > self.max_source_bytes: - raise DocForgeError( - "source_too_large", - "Python source exceeds the configured adapter limit", - source=relative, - maximum=self.max_source_bytes, - ) - descriptor = -1 - try: - flags = os.O_RDONLY | getattr(os, "O_BINARY", 0) - flags |= getattr(os, "O_NOFOLLOW", 0) - descriptor = os.open(candidate, flags) - opened = os.fstat(descriptor) - if not stat.S_ISREG(opened.st_mode) or (opened.st_dev, opened.st_ino) != ( - before.st_dev, - before.st_ino, - ): - raise DocForgeError( - "path_escape", - "Python source identity changed before it was opened", - source=relative, - ) - with os.fdopen(descriptor, "rb") as handle: - descriptor = -1 - raw = handle.read(self.max_source_bytes + 1) - after_open = os.fstat(handle.fileno()) - after = candidate.lstat() - except DocForgeError: - raise - except OSError as error: - raise DocForgeError( - "stale_adapter_source", - "Python source changed while being read", - source=relative, - ) from error - finally: - if descriptor >= 0: - os.close(descriptor) - identity_before = ( - before.st_dev, - before.st_ino, - before.st_size, - before.st_mtime_ns, - ) - identity_after = ( - after.st_dev, - after.st_ino, - after.st_size, - after.st_mtime_ns, - ) - identity_opened = ( - after_open.st_dev, - after_open.st_ino, - after_open.st_size, - after_open.st_mtime_ns, - ) - if ( - len(raw) > self.max_source_bytes - or identity_before != identity_opened - or identity_before != identity_after - or len(raw) != after.st_size - ): - raise DocForgeError( - "stale_adapter_source", - "Python source changed while being read", - source=relative, - ) - text = self._decode_source(raw, relative) - tree: ast.Module | None = None - if parse: - try: - tree = ast.parse(text, filename=relative, type_comments=True) - except SyntaxError as error: - raise DocForgeError( - "invalid_python_source", - "Python source cannot be parsed by the reference adapter", - source=relative, - line=error.lineno, - ) from error - module_name = self._module_name(path, PurePosixPath(source_root)) - return _SourceRecord( - source_id=self._source_id(relative), - source_path=relative, - module_name=module_name, - fingerprint=hashlib.sha256(raw).hexdigest(), - raw=raw, - text=text, - tree=tree, - ) - - def _source_root_for(self, path: PurePosixPath) -> str: - matches = [ - source_root - for source_root in self.source_roots - if path == PurePosixPath(source_root) or PurePosixPath(source_root) in path.parents - ] - if len(matches) != 1: - raise DocForgeError( - "path_escape", - "Python source is outside the declared source roots", - source=path.as_posix(), - ) - return matches[0] - - @staticmethod - def _decode_source(raw: bytes, source_path: str) -> str: - try: - encoding, _ = tokenize.detect_encoding(io.BytesIO(raw).readline) - return raw.decode(encoding) - except (LookupError, SyntaxError, UnicodeDecodeError) as error: - raise DocForgeError( - "invalid_python_source", - "Python source encoding is invalid", - source=source_path, - ) from error - - @staticmethod - def _module_name(path: PurePosixPath, source_root: PurePosixPath) -> str: - local = path.relative_to(source_root) - parts = list(local.with_suffix("").parts) - if parts and parts[-1] == "__init__": - parts.pop() - if not parts: - parts = list(source_root.parts) - return ".".join(parts) - - @staticmethod - def _source_id(relative: str) -> str: - digest = hashlib.sha256( - f"{PYTHON_IDENTITY_VERSION}\0source\0{relative}".encode() - ).hexdigest()[:24] - return f"python.source.{digest}" - - @staticmethod - def _file_node_id(source_id: str) -> str: - return source_id.replace("python.source.", "python.file.", 1) - - @staticmethod - def _module_node_id(source_id: str) -> str: - return source_id.replace("python.source.", "python.module.", 1) - - @staticmethod - def _symbol_node_id(kind: str, source_id: str, qualified_name: str) -> str: - digest = hashlib.sha256( - (f"{PYTHON_IDENTITY_VERSION}\0{kind}\0{source_id}\0{qualified_name}").encode() - ).hexdigest()[:24] - return f"python.{kind}.{digest}" - - @staticmethod - def _source_hash(sources: Sequence[AdapterSource]) -> str: - digest = hashlib.sha256() - digest.update(PYTHON_ADAPTER_VERSION.encode()) - digest.update(PYTHON_EXTRACTOR_VERSION.encode()) - digest.update(PYTHON_IDENTITY_VERSION.encode()) - for source in sources: - digest.update(b"\0source\0") - digest.update(source.source_id.encode()) - digest.update(b"\0path\0") - digest.update(source.source_path.encode()) - digest.update(b"\0fingerprint\0") - digest.update(source.fingerprint.encode()) - for dependency in source.dependencies: - digest.update(b"\0dependency\0") - digest.update(dependency.encode()) - return digest.hexdigest() - - def _local_dependencies( - self, - record: _SourceRecord, - module_sources: dict[str, str], - ) -> tuple[str, ...]: - dependencies: set[str] = set() - package = ( - record.module_name - if record.source_path.endswith("/__init__.py") - else record.module_name.rpartition(".")[0] - ) - for level, module, imported_names in self._lexical_imports(record): - candidates: list[str] = [] - if not imported_names and level == 0 and module is not None: - candidates.append(module) - else: - base = self._import_from_base(package, level, module) - if base: - candidates.append(base) - candidates.extend(f"{base}.{name}" for name in imported_names if name != "*") - for module_name in candidates: - target = module_sources.get(module_name) - if target is not None and target != record.source_id: - dependencies.add(target) - return tuple(sorted(dependencies)) - - @staticmethod - def _lexical_imports( - record: _SourceRecord, - ) -> tuple[tuple[int, str | None, tuple[str, ...]], ...]: - """Inventory import dependencies without constructing a Python AST.""" - - try: - tokens = tuple(tokenize.generate_tokens(io.StringIO(record.text).readline)) - except (IndentationError, tokenize.TokenError) as error: - line = error.args[1][0] if len(error.args) > 1 else None - raise DocForgeError( - "invalid_python_source", - "Python source cannot be tokenized by the reference adapter", - source=record.source_path, - line=line, - ) from error - - statements: list[list[tokenize.TokenInfo]] = [] - current: list[tokenize.TokenInfo] = [] - nesting = 0 - ignored = { - tokenize.ENCODING, - tokenize.INDENT, - tokenize.DEDENT, - tokenize.NL, - tokenize.COMMENT, - } - for token in tokens: - if token.type in ignored: - continue - if token.type == tokenize.OP: - if token.string in "([{": - nesting += 1 - elif token.string in ")]}": - nesting = max(0, nesting - 1) - elif token.string == ";" and nesting == 0: - if current: - statements.append(current) - current = [] - continue - if token.type in {tokenize.NEWLINE, tokenize.ENDMARKER} and nesting == 0: - if current: - statements.append(current) - current = [] - continue - current.append(token) - - imports: list[tuple[int, str | None, tuple[str, ...]]] = [] - for statement in statements: - for position, token in enumerate(statement): - if token.type != tokenize.NAME or token.string not in {"import", "from"}: - continue - if token.string == "import": - names = PythonReferenceAdapter._imported_names(statement[position + 1 :]) - imports.extend((0, name, ()) for name in names) - break - import_position = next( - ( - index - for index in range(position + 1, len(statement)) - if statement[index].type == tokenize.NAME - and statement[index].string == "import" - ), - None, - ) - if import_position is None: - break - prefix = statement[position + 1 : import_position] - level = 0 - for item in prefix: - if item.type == tokenize.NAME: - break - if item.type == tokenize.OP and set(item.string) == {"."}: - level += len(item.string) - module_parts = [item.string for item in prefix if item.type == tokenize.NAME] - module = ".".join(module_parts) or None - names = PythonReferenceAdapter._imported_names(statement[import_position + 1 :]) - imports.append((level, module, names)) - break - return tuple(imports) - - @staticmethod - def _imported_names(tokens: Sequence[tokenize.TokenInfo]) -> tuple[str, ...]: - names: list[str] = [] - current: list[str] = [] - skip_alias = False - for token in tokens: - if token.type == tokenize.NAME and token.string == "as": - skip_alias = True - continue - if token.type == tokenize.OP and token.string == ",": - if current: - names.append(".".join(current)) - current = [] - skip_alias = False - continue - if token.type == tokenize.OP and token.string == "*": - if not skip_alias: - current.append("*") - continue - if token.type == tokenize.NAME and not skip_alias: - current.append(token.string) - if current: - names.append(".".join(current)) - return tuple(name for name in names if name) - - @staticmethod - def _import_from_base(package: str, level: int, module: str | None) -> str: - if level == 0: - return module or "" - package_parts = package.split(".") if package else [] - keep = len(package_parts) - (level - 1) - if keep < 0: - return "" - prefix = package_parts[:keep] - if module: - prefix.extend(module.split(".")) - return ".".join(prefix) - - def _extract_record( - self, - record: _SourceRecord, - source: AdapterSource, - ) -> AdapterSourceProjection: - tree = record.tree - if tree is None: - raise DocForgeError( - "invalid_adapter", - "Python extraction requires a parsed source record", - source=record.source_path, - ) - file_id = self._file_node_id(source.source_id) - module_id = self._module_node_id(source.source_id) - nodes: list[AdapterNode] = [ - self._node( - node_id=file_id, - title=record.source_path, - kind="file", - qualified_name=record.source_path, - content=record.text, - source_path=record.source_path, - anchor="L1", - ), - self._node( - node_id=module_id, - title=record.module_name, - kind="module", - qualified_name=record.module_name, - content=( - ast.get_docstring(tree, clean=False) or f"Python module {record.module_name}." - ), - source_path=record.source_path, - anchor="L1", - ), - ] - edges: list[AdapterEdge] = [self._edge(file_id, "contains", module_id, "syntax")] - definitions = self._definitions(record, module_id) - owners: list[PythonLogicOwner] = [] - for definition in definitions: - content = ast.get_source_segment(record.text, definition.node) - if content is None or not content.strip(): - content = f"Python {definition.kind} {definition.qualified_name}." - anchor = f"L{definition.node.lineno}" - nodes.append( - self._node( - node_id=definition.node_id, - title=definition.qualified_name, - kind=definition.kind, - qualified_name=f"{record.module_name}.{definition.qualified_name}", - content=content, - source_path=record.source_path, - anchor=anchor, - asynchronous=isinstance(definition.node, ast.AsyncFunctionDef), - ) - ) - edges.append( - self._edge( - definition.parent_node_id, - "contains", - definition.node_id, - "syntax", - anchor=anchor, - ) - ) - if definition.kind == "function": - owners.append( - PythonLogicOwner( - owner_node_id=definition.node_id, - qualified_name=definition.qualified_name, - line=definition.node.lineno, - ) - ) - for dependency in source.dependencies: - edges.append( - self._edge( - module_id, - "depends_on", - self._module_node_id(dependency), - "local_import", - ) - ) - logic = analyze_python_source( - record.text, - source_id=source.source_id, - owners=owners, - filename=record.source_path, - max_nodes_per_function=self.max_logic_nodes_per_function, - ) - return AdapterSourceProjection( - source_id=source.source_id, - fingerprint=source.fingerprint, - nodes=tuple(sorted(nodes, key=lambda item: item.node.node_id)), - edges=tuple( - sorted( - edges, - key=lambda item: ( - item.edge.source_id, - item.edge.relation, - item.edge.target_id, - ), - ) - ), - logic=logic, - ) - - def _definitions( - self, - record: _SourceRecord, - module_node_id: str, - ) -> tuple[_Definition, ...]: - tree = record.tree - if tree is None: - raise DocForgeError( - "invalid_adapter", - "Python definition extraction requires a parsed source record", - source=record.source_path, - ) - definitions: list[_Definition] = [] - qualified_names: set[str] = set() - adapter = self - - class Collector(ast.NodeVisitor): - def __init__(self) -> None: - self.scope_names: list[str] = [] - self.scope_node_ids: list[str] = [module_node_id] - - def _definition( - self, - kind: str, - node: ast.ClassDef | ast.FunctionDef | ast.AsyncFunctionDef, - ) -> None: - qualified_name = ".".join((*self.scope_names, node.name)) - if qualified_name in qualified_names: - raise DocForgeError( - "ambiguous_python_symbol", - "Python source repeats a class or function identity", - source=record.source_path, - qualified_name=qualified_name, - ) - qualified_names.add(qualified_name) - node_id = adapter._symbol_node_id( - kind, - record.source_id, - qualified_name, - ) - definitions.append( - _Definition( - kind=kind, - qualified_name=qualified_name, - parent_node_id=self.scope_node_ids[-1], - node=node, - node_id=node_id, - ) - ) - self.scope_names.append(node.name) - self.scope_node_ids.append(node_id) - self.generic_visit(node) - self.scope_node_ids.pop() - self.scope_names.pop() - - def visit_ClassDef(self, node: ast.ClassDef) -> None: - self._definition("class", node) - - def visit_FunctionDef(self, node: ast.FunctionDef) -> None: - self._definition("function", node) - - def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None: - self._definition("function", node) - - Collector().visit(tree) - return tuple(sorted(definitions, key=lambda item: item.node_id)) - - @staticmethod - def _node( - *, - node_id: str, - title: str, - kind: str, - qualified_name: str, - content: str, - source_path: str, - anchor: str, - asynchronous: bool = False, - ) -> AdapterNode: - normalized = content.strip() or f"Python {kind} {qualified_name}." - tags = tuple(sorted({"python", kind, *(("async",) if asynchronous else ())})) - return AdapterNode( - node=Node( - node_id=node_id, - title=title, - family="code", - authority="derived", - status="active", - tags=tags, - summary=f"Python {kind} fact for {qualified_name}.", - content=normalized, - source_path=source_path, - source_anchor=anchor, - content_hash=hashlib.sha256(normalized.encode()).hexdigest(), - ), - metadata=( - ("extractor", PYTHON_EXTRACTOR_VERSION), - ("identity", PYTHON_IDENTITY_VERSION), - ("kind", kind), - ("qualified_name", qualified_name), - ), - ) - - @staticmethod - def _edge( - source_id: str, - relation: str, - target_id: str, - evidence: str, - *, - anchor: str | None = None, - ) -> AdapterEdge: - metadata = [("evidence", evidence)] - if anchor is not None: - metadata.append(("source_anchor", anchor)) - return AdapterEdge( - edge=Edge(source_id, relation, target_id), - metadata=tuple(metadata), - ) diff --git a/src/docforge/client_config.py b/src/docforge/client_config.py index d02e493..18c9de6 100644 --- a/src/docforge/client_config.py +++ b/src/docforge/client_config.py @@ -16,12 +16,6 @@ from dataclasses import dataclass from pathlib import Path from typing import Literal, cast -from .adapter_launcher import ( - AdapterLauncherV1, - AdapterSourceAvailabilityV1, - adapter_source_availability, - validate_adapter_launcher, -) from .changeset_contract import document_hash from .errors import DocForgeError from .models import ProjectDescriptor, ProjectService @@ -1085,450 +1079,3 @@ def generate_client_configuration( } _validate_configuration_result(result, trusted_descriptor=descriptor) return result - - -def generate_adapter_client_configuration( - project: ProjectService, - launcher: AdapterLauncherV1, - client: str, - *, - server_name: str | None = None, - capability_mode: str = "read", - proposal_writer: str | None = None, - canonical_applier: str | None = None, - no_ast: bool = False, - manual_render_policy: str | None = None, - portable_graph_policy: str | None = None, - live_viewer_policy: str | None = None, - startup_timeout: int = 30, - tool_timeout: int = 300, - output: Path | None = None, -) -> dict[str, object]: - """Generate one client fragment for an explicitly constructed project-owned adapter.""" - - validate_adapter_launcher(project, launcher) - selected_client = _client_name(client) - selected_mode = _capability_mode(capability_mode) - startup_seconds = _bounded_seconds( - startup_timeout, - field="startup_timeout", - maximum=3_600, - ) - tool_seconds = _bounded_seconds( - tool_timeout, - field="tool_timeout", - maximum=86_400, - ) - descriptor = project.descriptor - _validate_adapter_authority( - descriptor, - selected_mode=selected_mode, - proposal_writer=proposal_writer, - canonical_applier=canonical_applier, - ) - source_availability = adapter_source_availability(project, launcher) - fingerprint = project_root_fingerprint(descriptor.root) - selected_name = _validated_server_name( - server_name, - project_id=descriptor.project_id, - fingerprint=fingerprint, - ) - executable = Path(os.path.abspath(sys.executable)) - try: - executable_status = executable.stat() - except OSError as error: - raise DocForgeError( - "client_configuration_unavailable", - "Current Python executable cannot be inspected", - ) from error - if not stat.S_ISREG(executable_status.st_mode) or not os.access(executable, os.X_OK): - raise DocForgeError( - "client_configuration_unavailable", - "Current Python executable is not a runnable regular file", - ) - - policy = compose_effective_policy( - selected_mode=selected_mode, - capability_source="explicit", - no_ast=no_ast, - diagnostics=False, - render_configured=descriptor.render is not None, - application_enabled=canonical_applier is not None, - ) - projection_policy = compose_projection_policy( - manual=manual_render_policy, - portable_graph=portable_graph_policy, - live_viewer=live_viewer_policy, - manual_configured=descriptor.render is not None, - portable_graph_configured=descriptor.graph_render is not None, - application_enabled=canonical_applier is not None, - ) - default_projection_policy = compose_projection_policy( - manual_configured=descriptor.render is not None, - portable_graph_configured=descriptor.graph_render is not None, - application_enabled=canonical_applier is not None, - ) - arguments = [ - "-I", - "-m", - launcher.module, - "--project-root", - str(descriptor.root), - "--capability-mode", - selected_mode, - ] - if proposal_writer is not None: - arguments.extend(("--proposal-writer", proposal_writer)) - if canonical_applier is not None: - arguments.extend(("--canonical-applier", canonical_applier)) - for option, selected, default in ( - ( - "--manual-render-policy", - projection_policy.manual, - default_projection_policy.manual, - ), - ( - "--portable-graph-policy", - projection_policy.portable_graph, - default_projection_policy.portable_graph, - ), - ( - "--live-viewer-policy", - projection_policy.live_viewer, - default_projection_policy.live_viewer, - ), - ): - if selected != default: - arguments.extend((option, selected)) - if no_ast: - arguments.append("--no-ast") - - artifact_format, content, warning = _artifact( - selected_client, - server_name=selected_name, - command=str(executable), - arguments=arguments, - startup_timeout=startup_seconds, - tool_timeout=tool_seconds, - ) - - def validate_current_binding() -> None: - validate_adapter_launcher(project, launcher) - current = adapter_source_availability(project, launcher) - if current != source_availability: - raise DocForgeError( - "source_changed", - "Adapter sources changed during client configuration generation", - ) - - if output is None: - validate_current_binding() - write_state = "not_requested" - durability = "not_applicable" - publication_warning = None - output_path = None - else: - validate_current_binding() - write_state, durability, publication_warning, published_path = _atomic_write( - output, - content, - validate_binding=validate_current_binding, - ) - output_path = str(published_path) if published_path is not None else None - artifact = { - "format": artifact_format, - "content": content, - "content_sha256": hashlib.sha256(content.encode("utf-8")).hexdigest(), - "output_path": output_path, - "write_state": write_state, - "durability": durability, - } - binding = { - "transport": "stdio", - "capability_mode": selected_mode, - "adapter_policy": policy.adapter_policy(), - "render_policy": { - "manual": policy.manual_render, - "graph": policy.graph_render, - "live_viewer": policy.live_viewer, - }, - "command": str(executable), - "args": arguments, - "environment": {}, - "timeouts": { - "startup_seconds": startup_seconds, - "tool_seconds": tool_seconds, - }, - "launcher_hash": launcher.launcher_hash, - } - project_binding = { - "project_id": descriptor.project_id, - "project_root": str(descriptor.root), - "project_root_fingerprint": fingerprint, - "adapter": descriptor.adapter, - "descriptor_hash": descriptor.descriptor_hash, - } - projection_availability = { - "manual_configured": descriptor.render is not None, - "portable_graph_configured": descriptor.graph_render is not None, - "application_enabled": canonical_applier is not None, - "live_viewer_available": True, - } - policy_payload = policy.as_dict() - result: dict[str, object] = { - "status": "ok", - "schema_version": 1, - "operation": "adapter_client.configure", - "action": "write" if output is not None else "preview", - "client": selected_client, - "server_name": selected_name, - "project": project_binding, - "launcher": launcher.as_dict(), - "launcher_hash": launcher.launcher_hash, - "source_availability": source_availability.as_dict(), - "source_availability_hash": source_availability.availability_hash, - "binding": binding, - "effective_policy": policy_payload, - "projection_policy": projection_policy.as_dict(), - "projection_policy_hash": projection_policy.policy_hash, - "projection_availability": projection_availability, - "artifact": artifact, - "warnings": [ - *([] if warning is None else [{"code": "timeout_format_unverified"}]), - *([] if publication_warning is None else [{"code": publication_warning}]), - ], - } - result["configuration_hash"] = document_hash(_adapter_configuration_hash_payload(result)) - _validate_adapter_configuration_result( - result, - project=project, - launcher=launcher, - source_availability=source_availability, - ) - return result - - -def _validate_adapter_authority( - descriptor: ProjectDescriptor, - *, - selected_mode: CapabilityMode, - proposal_writer: str | None, - canonical_applier: str | None, -) -> None: - writer_ids = {writer.writer_id for writer in descriptor.proposal_writers} - if selected_mode == "read": - if proposal_writer is not None or canonical_applier is not None: - raise DocForgeError( - "invalid_capability_binding", - "Read configuration cannot bind proposal or application authority", - ) - elif selected_mode == "proposal": - if proposal_writer is None or proposal_writer not in writer_ids: - raise DocForgeError( - "capability_unavailable", - "Proposal configuration requires a descriptor-declared writer", - required="proposal_writer", - ) - if canonical_applier is not None: - raise DocForgeError( - "invalid_capability_binding", - "Proposal configuration cannot bind a canonical applier", - ) - elif ( - proposal_writer is None - or canonical_applier is None - or proposal_writer != canonical_applier - or proposal_writer not in writer_ids - ): - raise DocForgeError( - "capability_unavailable", - "Application configuration requires one declared writer/applier identity", - required="matching_declared_writer_and_applier", - ) - - -def _adapter_configuration_hash_payload(result: dict[str, object]) -> dict[str, object]: - artifact = cast(dict[str, object], result["artifact"]) - return { - "schema_version": 1, - "client": result["client"], - "server_name": result["server_name"], - "project": result["project"], - "launcher": result["launcher"], - "launcher_hash": result["launcher_hash"], - "source_availability": result["source_availability"], - "source_availability_hash": result["source_availability_hash"], - "binding": result["binding"], - "effective_policy": result["effective_policy"], - "projection_policy": result["projection_policy"], - "projection_policy_hash": result["projection_policy_hash"], - "projection_availability": result["projection_availability"], - "artifact_format": artifact["format"], - "artifact_content_sha256": artifact["content_sha256"], - } - - -def _validate_adapter_configuration_result( - result: dict[str, object], - *, - project: ProjectService, - launcher: AdapterLauncherV1, - source_availability: AdapterSourceAvailabilityV1, -) -> None: - validate_adapter_launcher(project, launcher) - descriptor = project.descriptor - project_binding = cast(dict[str, object], result["project"]) - if ( - result["launcher"] != launcher.as_dict() - or result["launcher_hash"] != launcher.launcher_hash - or result["source_availability"] != source_availability.as_dict() - or result["source_availability_hash"] != source_availability.availability_hash - or project_binding - != { - "project_id": descriptor.project_id, - "project_root": str(descriptor.root), - "project_root_fingerprint": project_root_fingerprint(descriptor.root), - "adapter": descriptor.adapter, - "descriptor_hash": descriptor.descriptor_hash, - } - ): - raise AssertionError("Generated adapter client identity drifted") - - artifact = cast(dict[str, object], result["artifact"]) - binding = cast(dict[str, object], result["binding"]) - content = cast(str, artifact["content"]) - if artifact["content_sha256"] != hashlib.sha256(content.encode("utf-8")).hexdigest(): - raise AssertionError("Generated adapter client content hash drifted") - timeouts = cast(dict[str, object], binding["timeouts"]) - expected_format, expected_content, _ = _artifact( - cast(ClientName, result["client"]), - server_name=cast(str, result["server_name"]), - command=cast(str, binding["command"]), - arguments=cast(list[str], binding["args"]), - startup_timeout=cast(int, timeouts["startup_seconds"]), - tool_timeout=cast(int, timeouts["tool_seconds"]), - ) - if artifact["format"] != expected_format or content != expected_content: - raise AssertionError("Generated adapter client artifact drifted from its binding") - if binding["environment"] != {} or binding["launcher_hash"] != launcher.launcher_hash: - raise AssertionError("Generated adapter launch boundary drifted") - - adapter_policy = cast(dict[str, object], binding["adapter_policy"]) - render_policy = cast(dict[str, object], binding["render_policy"]) - projection_policy = cast(dict[str, object], result["projection_policy"]) - projection_availability = cast(dict[str, object], result["projection_availability"]) - arguments = cast(list[str], binding["args"]) - prefix = [ - "-I", - "-m", - launcher.module, - "--project-root", - str(descriptor.root), - "--capability-mode", - cast(str, binding["capability_mode"]), - ] - if arguments[:7] != prefix: - raise AssertionError("Generated adapter client arguments drifted from their binding") - remaining = arguments[7:] - no_ast_argument = "--no-ast" in arguments - if no_ast_argument: - if remaining[-1:] != ["--no-ast"] or arguments.count("--no-ast") != 1: - raise AssertionError("Generated adapter no-AST argument layout drifted") - remaining = remaining[:-1] - projection_arguments: dict[str, str] = {} - authority_arguments: list[str] = [] - projection_options = { - "--manual-render-policy": "manual", - "--portable-graph-policy": "portable_graph", - "--live-viewer-policy": "live_viewer", - } - position = 0 - while position < len(remaining): - option = remaining[position] - field = projection_options.get(option) - if field is None: - authority_arguments.append(option) - position += 1 - continue - if position + 1 >= len(remaining) or option in projection_arguments: - raise AssertionError("Generated adapter projection argument layout drifted") - value = remaining[position + 1] - projection_arguments[option] = value - if projection_policy[field] != value: - raise AssertionError("Generated adapter projection policy argument drifted") - position += 2 - mode = binding["capability_mode"] - if ( - (mode == "read" and authority_arguments) - or ( - mode == "proposal" - and ( - len(authority_arguments) != 2 - or authority_arguments[0] != "--proposal-writer" - or not authority_arguments[1] - ) - ) - or ( - mode == "application" - and ( - len(authority_arguments) != 4 - or authority_arguments[0] != "--proposal-writer" - or authority_arguments[2] != "--canonical-applier" - or not authority_arguments[1] - or authority_arguments[1] != authority_arguments[3] - ) - ) - ): - raise AssertionError("Generated adapter authority argument layout drifted") - - expected_projection_policy = compose_projection_policy( - manual=projection_arguments.get("--manual-render-policy"), - portable_graph=projection_arguments.get("--portable-graph-policy"), - live_viewer=projection_arguments.get("--live-viewer-policy"), - manual_configured=cast(bool, projection_availability["manual_configured"]), - portable_graph_configured=cast( - bool, - projection_availability["portable_graph_configured"], - ), - application_enabled=cast( - bool, - projection_availability["application_enabled"], - ), - live_viewer_available=cast( - bool, - projection_availability["live_viewer_available"], - ), - ) - policy = cast(dict[str, object], result["effective_policy"]) - composed_policy = compose_effective_policy( - selected_mode=cast(CapabilityMode, mode), - capability_source="explicit", - no_ast=adapter_policy["mode"] == "preserve-no-ast", - diagnostics=False, - render_configured=render_policy["manual"] != "disabled", - application_enabled=mode == "application", - ) - if ( - projection_policy != expected_projection_policy.as_dict() - or projection_availability["manual_configured"] != (descriptor.render is not None) - or projection_availability["portable_graph_configured"] - != (descriptor.graph_render is not None) - or projection_availability["application_enabled"] != (mode == "application") - or projection_availability["live_viewer_available"] is not True - or policy != composed_policy.as_dict() - or adapter_policy != composed_policy.adapter_policy() - or no_ast_argument != (adapter_policy["mode"] == "preserve-no-ast") - or binding["launcher_hash"] != result["launcher_hash"] - ): - raise AssertionError("Generated adapter client policy drifted") - if result["projection_policy_hash"] != hashlib.sha256( - json.dumps( - projection_policy, - sort_keys=True, - separators=(",", ":"), - ensure_ascii=False, - ).encode("utf-8") - ).hexdigest() or result["configuration_hash"] != document_hash( - _adapter_configuration_hash_payload(result) - ): - raise AssertionError("Generated adapter client hash drifted") diff --git a/src/docforge/command_reference.py b/src/docforge/command_reference.py deleted file mode 100644 index 909422e..0000000 --- a/src/docforge/command_reference.py +++ /dev/null @@ -1,242 +0,0 @@ -"""Deterministic command references derived from live CLI and MCP registrations.""" - -from __future__ import annotations - -import argparse -import hashlib -import json -from collections.abc import Collection, Iterable, Mapping -from dataclasses import dataclass -from typing import Any, cast - -from mcp.types import Tool - - -@dataclass(frozen=True) -class CliCommandReference: - """One CLI command and the exact normalized usage emitted by argparse.""" - - name: str - invocation: str - - -@dataclass(frozen=True) -class McpToolReference: - """One registered MCP tool with arguments derived from its input schema.""" - - surface: str - name: str - required_arguments: tuple[str, ...] - optional_arguments: tuple[str, ...] - description: str - input_schema_hash: str - - -def cli_command_references( - parser: argparse.ArgumentParser | None = None, -) -> tuple[CliCommandReference, ...]: - """Read CLI reference rows from the parser used by ``docforge``.""" - - effective_parser = parser or _docforge_parser() - commands = _command_parsers(effective_parser) - return tuple( - CliCommandReference( - name=name, - invocation=_normalize_usage(command_parser.format_usage()), - ) - for name, command_parser in sorted(commands.items()) - ) - - -def mcp_tool_references( - tools: Iterable[Tool], - *, - expected_names: Collection[str] | None = None, -) -> tuple[McpToolReference, ...]: - """Read MCP reference rows from registered tools returned by ``list_tools``. - - ``expected_names`` makes documentation generation fail closed when the selected - server surface is incomplete or has drifted. - """ - - surface_by_name = _mcp_surface_by_name() - references: list[McpToolReference] = [] - observed: set[str] = set() - for tool in tools: - if tool.name in observed: - raise ValueError(f"MCP tool registration repeats {tool.name!r}") - observed.add(tool.name) - try: - surface = surface_by_name[tool.name] - except KeyError as error: - raise ValueError( - f"MCP tool {tool.name!r} has no declared capability surface" - ) from error - schema = tool.inputSchema - properties = _schema_properties(schema) - required = _schema_required(schema, properties) - references.append( - McpToolReference( - surface=surface, - name=tool.name, - required_arguments=tuple(sorted(required)), - optional_arguments=tuple(sorted(set(properties) - required)), - description=_normalize_text(tool.description or ""), - input_schema_hash=_canonical_hash(schema), - ) - ) - if expected_names is not None: - expected = set(expected_names) - if observed != expected: - missing = sorted(expected - observed) - unexpected = sorted(observed - expected) - raise ValueError( - "Registered MCP tools do not match the requested reference surface: " - f"missing={missing!r}, unexpected={unexpected!r}" - ) - surface_rank = {"read": 0, "proposal": 1, "application": 2} - return tuple(sorted(references, key=lambda item: (surface_rank[item.surface], item.name))) - - -def render_cli_reference_markdown(references: Iterable[CliCommandReference]) -> str: - """Render a deterministic Markdown table for CLI commands.""" - - rows = sorted(references, key=lambda item: item.name) - lines = [ - "| Command | Invocation |", - "|---|---|", - ] - lines.extend( - f"| `{_escape_markdown(item.name)}` | `{_escape_markdown(item.invocation)}` |" - for item in rows - ) - return "\n".join(lines) + "\n" - - -def render_mcp_reference_markdown(references: Iterable[McpToolReference]) -> str: - """Render a deterministic Markdown table for registered MCP tools.""" - - surface_rank = {"read": 0, "proposal": 1, "application": 2} - rows = sorted(references, key=lambda item: (surface_rank[item.surface], item.name)) - lines = [ - "| Surface | Tool | Required arguments | Optional arguments | " - "Input schema SHA-256 | Description |", - "|---|---|---|---|---|---|", - ] - lines.extend( - "| " - f"{_escape_markdown(item.surface)} | " - f"`{_escape_markdown(item.name)}` | " - f"{_argument_list(item.required_arguments)} | " - f"{_argument_list(item.optional_arguments)} | " - f"`{item.input_schema_hash}` | " - f"{_escape_markdown(item.description) or '—'} |" - for item in rows - ) - return "\n".join(lines) + "\n" - - -def render_command_reference_markdown( - cli_references: Iterable[CliCommandReference], - mcp_references: Iterable[McpToolReference], -) -> str: - """Render the complete deterministic CLI and MCP command reference.""" - - return ( - "# DocForge command reference\n\n" - "> Generated from the live CLI parser and MCP registrations. Do not edit this file " - "by hand.\n\n" - "Global CLI options are documented in `docforge --help` and are not repeated in each " - "command row. The MCP table is the complete generic application-enabled surface; " - "the fixed reference-adapter server exposes only its read rows.\n\n" - "## CLI commands\n\n" - f"{render_cli_reference_markdown(cli_references)}" - "\n## MCP tools\n\n" - f"{render_mcp_reference_markdown(mcp_references)}" - ) - - -def _docforge_parser() -> argparse.ArgumentParser: - from .cli import _parser # pyright: ignore[reportPrivateUsage] - - return _parser() - - -def _command_parsers( - parser: argparse.ArgumentParser, -) -> Mapping[str, argparse.ArgumentParser]: - actions = parser._actions # pyright: ignore[reportPrivateUsage] - subparsers = [ - action - for action in actions - if action.dest == "command" and isinstance(action.choices, dict) - ] - if len(subparsers) != 1: - raise ValueError("CLI parser must define exactly one command subparser") - return cast(Mapping[str, argparse.ArgumentParser], subparsers[0].choices) - - -def _mcp_surface_by_name() -> dict[str, str]: - from .mcp_server import APPLICATION_TOOLS, PROPOSAL_TOOLS, READ_TOOLS - - groups = { - "read": READ_TOOLS, - "proposal": PROPOSAL_TOOLS, - "application": APPLICATION_TOOLS, - } - surface_by_name: dict[str, str] = {} - for surface, names in groups.items(): - for name in names: - if name in surface_by_name: - raise ValueError(f"MCP tool surface declaration repeats {name!r}") - surface_by_name[name] = surface - return surface_by_name - - -def _normalize_usage(usage: str) -> str: - return _normalize_text(usage.removeprefix("usage: ")) - - -def _normalize_text(value: str) -> str: - return " ".join(value.split()) - - -def _schema_properties(schema: Mapping[str, Any]) -> dict[str, Any]: - raw_properties: object = schema.get("properties", {}) - if not isinstance(raw_properties, dict): - raise ValueError("MCP tool input schema properties must be a string-keyed object") - properties: dict[str, Any] = {} - for name, value in cast(dict[object, object], raw_properties).items(): - if not isinstance(name, str): - raise ValueError("MCP tool input schema properties must be a string-keyed object") - properties[name] = value - return properties - - -def _schema_required(schema: Mapping[str, Any], properties: Mapping[str, Any]) -> set[str]: - raw_required: object = schema.get("required", []) - if not isinstance(raw_required, list): - raise ValueError("MCP tool input schema required arguments must be strings") - required_names: set[str] = set() - for name in cast(list[object], raw_required): - if not isinstance(name, str): - raise ValueError("MCP tool input schema required arguments must be strings") - required_names.add(name) - if not required_names <= set(properties): - raise ValueError("MCP tool input schema requires an undeclared argument") - return required_names - - -def _canonical_hash(payload: Mapping[str, Any]) -> str: - encoded = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode() - return hashlib.sha256(encoded).hexdigest() - - -def _argument_list(arguments: tuple[str, ...]) -> str: - if not arguments: - return "—" - return ", ".join(f"`{_escape_markdown(argument)}`" for argument in arguments) - - -def _escape_markdown(value: str) -> str: - return value.replace("\\", "\\\\").replace("|", "\\|").replace("`", "\\`") diff --git a/src/docforge/incremental.py b/src/docforge/incremental.py index 7f5f511..776111d 100644 --- a/src/docforge/incremental.py +++ b/src/docforge/incremental.py @@ -4,7 +4,6 @@ from __future__ import annotations import json import os -import stat import tempfile from dataclasses import dataclass from pathlib import Path @@ -13,8 +12,6 @@ from typing import Any, cast from .errors import DocForgeError EXTRACTION_CACHE_SCHEMA_VERSION = 1 -MAX_EXTRACTION_CACHE_BYTES = 64_000_000 -MAX_EXTRACTION_CACHE_SOURCES = 10_000 @dataclass(frozen=True) @@ -45,39 +42,13 @@ def load_extraction_cache( project_id: str, adapter_id: str, adapter_version: str, - max_bytes: int = MAX_EXTRACTION_CACHE_BYTES, - max_sources: int = MAX_EXTRACTION_CACHE_SOURCES, ) -> ExtractionCache | None: """Read a cache generation, treating malformed or incompatible data as a miss.""" - if max_bytes < 1 or max_sources < 1: - raise ValueError("Extraction cache limits must be positive") - try: - descriptor = os.open( - path, - os.O_RDONLY - | getattr(os, "O_CLOEXEC", 0) - | getattr(os, "O_NOFOLLOW", 0) - | getattr(os, "O_NONBLOCK", 0), - ) - except OSError: + if not path.is_file() or path.is_symlink(): return None try: - with os.fdopen(descriptor, "rb") as handle: - before = os.fstat(handle.fileno()) - if not stat.S_ISREG(before.st_mode) or before.st_size > max_bytes: - return None - encoded = handle.read(max_bytes + 1) - after = os.fstat(handle.fileno()) - if ( - len(encoded) > max_bytes - or before.st_dev != after.st_dev - or before.st_ino != after.st_ino - or before.st_size != after.st_size - or before.st_mtime_ns != after.st_mtime_ns - ): - return None - raw = json.loads(encoded.decode("utf-8")) + raw = json.loads(path.read_text(encoding="utf-8")) if not isinstance(raw, dict): return None document = cast(dict[str, Any], raw) @@ -91,11 +62,8 @@ def load_extraction_cache( raw_sources = document.get("sources") if not isinstance(raw_sources, list): return None - source_items = cast(list[object], raw_sources) - if len(source_items) > max_sources: - return None sources: list[CachedSource] = [] - for raw_source in source_items: + for raw_source in cast(list[object], raw_sources): if not isinstance(raw_source, dict): return None item = cast(dict[str, object], raw_source) @@ -142,24 +110,9 @@ def load_extraction_cache( return None -def write_extraction_cache( - path: Path, - cache: ExtractionCache, - *, - max_bytes: int = MAX_EXTRACTION_CACHE_BYTES, - max_sources: int = MAX_EXTRACTION_CACHE_SOURCES, -) -> None: +def write_extraction_cache(path: Path, cache: ExtractionCache) -> None: """Atomically publish one validated extraction-cache generation.""" - if max_bytes < 1 or max_sources < 1: - raise ValueError("Extraction cache limits must be positive") - if len(cache.sources) > max_sources: - raise DocForgeError( - "cache_limit", - "Incremental extraction cache exceeds its source limit", - maximum=max_sources, - actual=len(cache.sources), - ) path.parent.mkdir(parents=True, exist_ok=True) document = { "schema_version": EXTRACTION_CACHE_SCHEMA_VERSION, @@ -178,23 +131,17 @@ def write_extraction_cache( for source in cache.sources ], } - encoded = json.dumps(document, sort_keys=True, separators=(",", ":")).encode("utf-8") + b"\n" - if len(encoded) > max_bytes: - raise DocForgeError( - "cache_limit", - "Incremental extraction cache exceeds its byte limit", - maximum=max_bytes, - actual=len(encoded), - ) with tempfile.NamedTemporaryFile( - mode="wb", + mode="w", + encoding="utf-8", prefix="extractions-", suffix=".json", dir=path.parent, delete=False, ) as descriptor: temporary = Path(descriptor.name) - descriptor.write(encoded) + json.dump(document, descriptor, sort_keys=True, separators=(",", ":")) + descriptor.write("\n") descriptor.flush() os.fsync(descriptor.fileno()) try: diff --git a/src/docforge/reference_config.py b/src/docforge/reference_config.py deleted file mode 100644 index c6557a2..0000000 --- a/src/docforge/reference_config.py +++ /dev/null @@ -1,428 +0,0 @@ -"""Strict, data-only configuration for the runnable reference adapters.""" - -from __future__ import annotations - -import hashlib -import os -import stat -import tomllib -from dataclasses import dataclass -from pathlib import Path, PurePosixPath -from typing import Literal, cast - -from .config_validation import ID_PATTERN -from .errors import DocForgeError - -REFERENCE_ADAPTER_SCHEMA_VERSION = 1 -REFERENCE_ADAPTER_CONFIG = PurePosixPath(".docforge/reference-adapter.toml") -MAX_REFERENCE_CONFIG_BYTES = 65_536 -MAX_REFERENCE_INVENTORY_ENTRIES = 65_536 - -ReferenceLanguage = Literal["python", "javascript", "typescript", "cpp"] -REFERENCE_LANGUAGES: tuple[ReferenceLanguage, ...] = ( - "python", - "javascript", - "typescript", - "cpp", -) - -_CONFIG_FIELDS = frozenset( - { - "schema_version", - "project_id", - "title", - "language", - "source_roots", - "compilation_database", - } -) -_LANGUAGE_SUFFIXES: dict[ReferenceLanguage, frozenset[str]] = { - "python": frozenset({".py"}), - "javascript": frozenset({".cjs", ".js", ".mjs"}), - "typescript": frozenset({".cts", ".mts", ".ts"}), - "cpp": frozenset({".c", ".cc", ".cpp", ".cxx", ".h", ".hh", ".hpp", ".hxx"}), -} -_KNOWN_SOURCE_SUFFIXES = frozenset( - suffix for suffixes in _LANGUAGE_SUFFIXES.values() for suffix in suffixes -) - -__all__ = [ - "MAX_REFERENCE_CONFIG_BYTES", - "MAX_REFERENCE_INVENTORY_ENTRIES", - "REFERENCE_ADAPTER_CONFIG", - "REFERENCE_ADAPTER_SCHEMA_VERSION", - "REFERENCE_LANGUAGES", - "ReferenceAdapterConfigV1", - "ReferenceLanguage", - "load_reference_adapter_config", -] - - -@dataclass(frozen=True) -class ReferenceAdapterConfigV1: - """One immutable, project-confined reference adapter selection.""" - - schema_version: Literal[1] - project_id: str - title: str - language: ReferenceLanguage - source_roots: tuple[Path, ...] - compilation_database: Path | None - project_root: Path - config_path: Path - config_hash: str - - def as_dict(self) -> dict[str, object]: - document: dict[str, object] = { - "schema_version": self.schema_version, - "project_id": self.project_id, - "title": self.title, - "language": self.language, - "source_roots": [ - path.relative_to(self.project_root).as_posix() for path in self.source_roots - ], - } - if self.compilation_database is not None: - document["compilation_database"] = self.compilation_database.relative_to( - self.project_root - ).as_posix() - return document - - -def load_reference_adapter_config(project_root: str | Path) -> ReferenceAdapterConfigV1: - """Load the fixed reference-adapter descriptor without executing project content.""" - - root = _project_root(project_root) - config_path = root.joinpath(*REFERENCE_ADAPTER_CONFIG.parts) - try: - resolved_config = config_path.resolve(strict=True) - except (OSError, ValueError) as error: - raise DocForgeError( - "missing_path", - "Reference adapter configuration does not exist", - path=str(config_path), - ) from error - if resolved_config != config_path or not resolved_config.is_relative_to(root): - raise DocForgeError( - "path_escape", - "Reference adapter configuration must not traverse symbolic links", - ) - raw = _read_regular_file( - config_path, - field="reference adapter configuration", - maximum_bytes=MAX_REFERENCE_CONFIG_BYTES, - ) - try: - document = cast(dict[str, object], tomllib.loads(raw.decode("utf-8"))) - except UnicodeDecodeError as error: - raise DocForgeError( - "invalid_reference_config", - "Reference adapter configuration must be UTF-8", - ) from error - except tomllib.TOMLDecodeError as error: - raise DocForgeError( - "invalid_reference_config", - f"Invalid reference adapter configuration: {error}", - ) from error - - unknown = sorted(set(document) - _CONFIG_FIELDS) - if unknown: - raise DocForgeError( - "invalid_reference_config", - "Reference adapter configuration has unknown fields", - fields=unknown, - ) - if document.get("schema_version") != REFERENCE_ADAPTER_SCHEMA_VERSION: - raise DocForgeError( - "invalid_reference_config", - "Reference adapter schema_version must be 1", - ) - project_id = _string(document, "project_id") - if ID_PATTERN.fullmatch(project_id) is None: - raise DocForgeError( - "invalid_reference_config", - "Reference adapter project_id is not a stable ID", - project_id=project_id, - ) - title = _string(document, "title") - if len(title) > 256: - raise DocForgeError( - "invalid_reference_config", - "Reference adapter title exceeds its size limit", - maximum_characters=256, - ) - raw_language = _string(document, "language") - if raw_language not in REFERENCE_LANGUAGES: - raise DocForgeError( - "unsupported_reference_language", - "Reference adapter language is not supported", - language=raw_language, - supported=list(REFERENCE_LANGUAGES), - ) - language = raw_language - source_roots = _source_roots(root, document.get("source_roots")) - - raw_compilation_database = document.get("compilation_database") - compilation_database: Path | None = None - if language == "cpp": - if raw_compilation_database is None: - raise DocForgeError( - "invalid_reference_config", - "C++ reference adapters require compilation_database", - ) - compilation_database = _confined_path( - root, - raw_compilation_database, - field="compilation_database", - expected="file", - ) - elif raw_compilation_database is not None: - raise DocForgeError( - "invalid_reference_config", - "compilation_database is allowed only for the C++ reference adapter", - language=language, - ) - - _validate_language_inventory(source_roots, language) - return ReferenceAdapterConfigV1( - schema_version=1, - project_id=project_id, - title=title, - language=language, - source_roots=source_roots, - compilation_database=compilation_database, - project_root=root, - config_path=config_path, - config_hash=hashlib.sha256(raw).hexdigest(), - ) - - -def _project_root(raw: str | Path) -> Path: - path = Path(raw) - if path.is_symlink(): - raise DocForgeError("path_escape", "Reference project root must not be a symlink") - try: - root = path.resolve(strict=True) - except (OSError, ValueError) as error: - raise DocForgeError( - "missing_path", - "Reference project root does not exist", - path=str(path), - ) from error - if not root.is_dir(): - raise DocForgeError("invalid_path", "Reference project root must be a directory") - return root - - -def _read_regular_file(path: Path, *, field: str, maximum_bytes: int) -> bytes: - try: - before = path.lstat() - except OSError as error: - raise DocForgeError("missing_path", f"{field} does not exist", path=str(path)) from error - if stat.S_ISLNK(before.st_mode) or not stat.S_ISREG(before.st_mode): - raise DocForgeError("invalid_path", f"{field} must be a regular non-symlink file") - if before.st_size > maximum_bytes: - raise DocForgeError( - "invalid_reference_config", - f"{field} exceeds its size limit", - maximum_bytes=maximum_bytes, - ) - - def identity(value: os.stat_result) -> tuple[int, int, int, int, int, int]: - return ( - value.st_dev, - value.st_ino, - value.st_mode, - value.st_size, - value.st_mtime_ns, - value.st_ctime_ns, - ) - - flags = os.O_RDONLY | getattr(os, "O_CLOEXEC", 0) | getattr(os, "O_NOFOLLOW", 0) - try: - descriptor = os.open(path, flags) - except OSError as error: - raise DocForgeError( - "reference_config_changed", - f"{field} changed while it was opened", - ) from error - try: - opened = os.fstat(descriptor) - chunks: list[bytes] = [] - remaining = maximum_bytes + 1 - while remaining: - chunk = os.read(descriptor, min(remaining, 65_536)) - if not chunk: - break - chunks.append(chunk) - remaining -= len(chunk) - raw = b"".join(chunks) - opened_after = os.fstat(descriptor) - except OSError as error: - raise DocForgeError( - "reference_config_changed", - f"{field} changed while it was read", - ) from error - finally: - os.close(descriptor) - try: - after = path.lstat() - except OSError as error: - raise DocForgeError( - "reference_config_changed", - f"{field} changed while it was read", - ) from error - - if ( - identity(before) != identity(opened) - or identity(opened) != identity(opened_after) - or identity(opened_after) != identity(after) - or len(raw) != opened_after.st_size - or len(raw) > maximum_bytes - ): - raise DocForgeError( - "reference_config_changed", - f"{field} changed while it was read", - ) - return raw - - -def _string(document: dict[str, object], field: str) -> str: - value = document.get(field) - if not isinstance(value, str) or not value or value != value.strip(): - raise DocForgeError( - "invalid_reference_config", - f"Reference adapter {field} must be a non-empty normalized string", - ) - return value - - -def _source_roots(root: Path, raw: object) -> tuple[Path, ...]: - if not isinstance(raw, list) or not raw: - raise DocForgeError( - "invalid_reference_config", - "Reference adapter source_roots must be a non-empty string list", - ) - values = cast(list[object], raw) - if len(values) > 64: - raise DocForgeError( - "invalid_reference_config", - "Reference adapter source_roots exceeds its item limit", - maximum_items=64, - ) - paths = tuple( - _confined_path(root, value, field="source_roots", expected="directory") for value in values - ) - if len(paths) != len(set(paths)): - raise DocForgeError( - "invalid_reference_config", - "Reference adapter source_roots contains duplicates", - ) - ordered = tuple(sorted(paths)) - if paths != ordered: - raise DocForgeError( - "invalid_reference_config", - "Reference adapter source_roots must be sorted", - ) - private_root = root / ".docforge" - if any( - path == private_root - or path.is_relative_to(private_root) - or any( - path.is_relative_to(other) or other.is_relative_to(path) - for other in ordered - if other != path - ) - for path in ordered - ): - raise DocForgeError( - "invalid_reference_config", - "Reference adapter source_roots must be non-overlapping and outside .docforge", - ) - return ordered - - -def _confined_path(root: Path, raw: object, *, field: str, expected: str) -> Path: - if not isinstance(raw, str) or not raw or raw != raw.strip(): - raise DocForgeError( - "invalid_reference_config", - f"{field} must be a non-empty normalized relative path", - ) - relative = Path(raw) - if relative.is_absolute() or ".." in relative.parts or relative == Path("."): - raise DocForgeError("path_escape", f"{field} must stay inside the project root", path=raw) - candidate = root / relative - if candidate.is_symlink(): - raise DocForgeError("path_escape", f"{field} must not be a symlink", path=raw) - try: - resolved = candidate.resolve(strict=True) - except (OSError, ValueError) as error: - raise DocForgeError("missing_path", f"{field} does not exist", path=raw) from error - if resolved != candidate or not resolved.is_relative_to(root): - raise DocForgeError( - "path_escape", - f"{field} must not traverse symlinks or leave the project root", - path=raw, - ) - if expected == "directory" and not resolved.is_dir(): - raise DocForgeError("invalid_path", f"{field} must identify a directory", path=raw) - if expected == "file" and (not resolved.is_file() or resolved.is_symlink()): - raise DocForgeError( - "invalid_path", - f"{field} must identify a regular non-symlink file", - path=raw, - ) - return resolved - - -def _validate_language_inventory( - source_roots: tuple[Path, ...], - language: ReferenceLanguage, -) -> None: - expected = _LANGUAGE_SUFFIXES[language] - selected = 0 - examined = 0 - other_languages: set[str] = set() - for source_root in source_roots: - for directory, names, files in os.walk( - source_root, - followlinks=False, - onerror=_inventory_error, - ): - base = Path(directory) - for name in (*names, *files): - examined += 1 - if examined > MAX_REFERENCE_INVENTORY_ENTRIES: - raise DocForgeError( - "reference_inventory_too_large", - "Reference source inventory exceeds its entry limit", - maximum_entries=MAX_REFERENCE_INVENTORY_ENTRIES, - ) - candidate = base / name - if candidate.is_symlink(): - raise DocForgeError( - "path_escape", - "Reference source roots must not contain symlinks", - path=candidate.as_posix(), - ) - for name in files: - suffix = Path(name).suffix - if suffix in expected: - selected += 1 - elif suffix in _KNOWN_SOURCE_SUFFIXES: - other_languages.add(suffix) - if selected == 0: - raise DocForgeError( - "reference_language_mismatch", - "Declared source roots contain no sources for the selected language", - language=language, - observed_suffixes=sorted(other_languages), - ) - - -def _inventory_error(error: OSError) -> None: - raise DocForgeError( - "invalid_path", - "Reference source inventory cannot be read safely", - path=error.filename, - ) from error diff --git a/src/docforge/reference_mcp.py b/src/docforge/reference_mcp.py deleted file mode 100644 index bf4a76e..0000000 --- a/src/docforge/reference_mcp.py +++ /dev/null @@ -1,196 +0,0 @@ -"""Runnable, read-only MCP binding for fixed in-repository reference adapters.""" - -from __future__ import annotations - -import argparse -import importlib -from pathlib import Path -from typing import cast - -from mcp.server.fastmcp import FastMCP - -from .adapter_sdk import ( - AdapterImplementation, - AdapterLoader, - AdapterProject, - AdapterProjectSettings, -) -from .errors import DocForgeError -from .mcp_server import create_read_only_server -from .reference_config import ReferenceAdapterConfigV1, load_reference_adapter_config - -REFERENCE_MCP_MODULE = "docforge.reference_mcp" -_FIXED_PROVIDERS = { - "python": ("docforge.adapters.python", "PythonReferenceAdapter"), - "javascript": ("docforge.adapters.javascript", "JavaScriptReferenceAdapter"), - "typescript": ("docforge.adapters.javascript", "JavaScriptReferenceAdapter"), - "cpp": ("docforge.adapters.cpp", "CppReferenceAdapter"), -} - -__all__ = [ - "REFERENCE_MCP_MODULE", - "create_reference_project", - "create_reference_server", - "main", -] - - -def create_reference_project(project_root: str | Path) -> AdapterProject: - """Construct one configured adapter project from fixed internal providers.""" - - config = load_reference_adapter_config(project_root) - return _project_from_config(config) - - -def create_reference_server( - project_root: str | Path, - *, - no_ast: bool = False, - diagnostics: bool = False, - capability_mode: str | None = None, - manual_projection_policy: str | None = None, - portable_graph_policy: str | None = None, - live_viewer_policy: str | None = None, -) -> FastMCP: - """Create the fixed read-only MCP surface for one reference project.""" - - config = load_reference_adapter_config(project_root) - project = _project_from_config(config) - return create_read_only_server( - project, - binding_metadata={ - "server_module": REFERENCE_MCP_MODULE, - "adapter_mode": "reference", - "reference_language": config.language, - "reference_config_hash": config.config_hash, - }, - no_ast=no_ast, - diagnostics=diagnostics, - capability_mode=capability_mode, - manual_projection_policy=manual_projection_policy, - portable_graph_policy=portable_graph_policy, - live_viewer_policy=live_viewer_policy, - ) - - -def _project_from_config(config: ReferenceAdapterConfigV1) -> AdapterProject: - loader = _adapter_loader(config) - cache_root = config.project_root / ".docforge" / "cache" / "reference-adapter" / config.language - return AdapterProject( - loader, - cache_root=cache_root, - settings=AdapterProjectSettings( - descriptor_path=config.config_path, - implementation=AdapterImplementation(files=(config.config_path,)), - ), - ) - - -def _adapter_loader(config: ReferenceAdapterConfigV1) -> AdapterLoader: - module_name, class_name = _FIXED_PROVIDERS[config.language] - try: - module = importlib.import_module(module_name) - except ModuleNotFoundError as error: - if error.name != module_name: - raise DocForgeError( - "optional_dependency_missing", - "The selected reference adapter dependency is not installed", - language=config.language, - missing_module=error.name, - install=_install_target(config.language), - ) from error - raise DocForgeError( - "reference_adapter_unavailable", - "The selected fixed reference adapter is not available", - language=config.language, - ) from error - constructor = getattr(module, class_name, None) - if not callable(constructor): - raise DocForgeError( - "reference_adapter_unavailable", - "The selected fixed reference adapter is not available", - language=config.language, - ) - arguments: dict[str, object] = { - "source_roots": tuple( - path.relative_to(config.project_root).as_posix() for path in config.source_roots - ), - "project_id": config.project_id, - "title": config.title, - } - if config.language == "cpp": - if config.compilation_database is None: - raise DocForgeError( - "invalid_reference_config", - "C++ reference adapters require compilation_database", - ) - arguments["compilation_database"] = config.compilation_database.relative_to( - config.project_root - ).as_posix() - loader = constructor(config.project_root, **arguments) - if not callable(getattr(loader, "load_projection", None)): - raise DocForgeError( - "invalid_adapter", - "The selected fixed provider does not implement the adapter contract", - language=config.language, - ) - return cast(AdapterLoader, loader) - - -def _install_target(language: str) -> str: - if language in {"javascript", "typescript"}: - return f"docforge[{language}]" - if language == "cpp": - return "docforge[cpp]" - return "docforge" - - -def main() -> None: - """Run the fixed reference binding over stdio.""" - - parser = argparse.ArgumentParser(prog="python -m docforge.reference_mcp") - parser.add_argument("--project-root", type=Path, required=True) - parser.add_argument( - "--no-ast", - action="store_true", - help=( - "Preserve the existing adapter and forbid AST, Tree-sitter, compiler-AST, " - "and function-Logic extraction changes" - ), - ) - parser.add_argument( - "--diagnostics", - action="store_true", - help="Attach bounded request-local stage timings and counters", - ) - parser.add_argument( - "--capability-mode", - choices=("read",), - help="Expose the fixed read-only project-bound capability surface", - ) - parser.add_argument( - "--manual-render-policy", - choices=("auto", "explicit", "disabled"), - ) - parser.add_argument( - "--portable-graph-policy", - choices=("explicit", "disabled"), - ) - parser.add_argument( - "--live-viewer-policy", - choices=("on-demand", "disabled"), - ) - arguments = parser.parse_args() - create_reference_server( - arguments.project_root, - no_ast=arguments.no_ast, - diagnostics=arguments.diagnostics, - capability_mode=arguments.capability_mode, - manual_projection_policy=arguments.manual_render_policy, - portable_graph_policy=arguments.portable_graph_policy, - live_viewer_policy=arguments.live_viewer_policy, - ).run(transport="stdio") - - -if __name__ == "__main__": - main() diff --git a/src/docforge/treesitter_logic.py b/src/docforge/treesitter_logic.py index c6892bb..41f981a 100644 --- a/src/docforge/treesitter_logic.py +++ b/src/docforge/treesitter_logic.py @@ -9,20 +9,18 @@ data; it is never imported, compiled, or executed. from __future__ import annotations import hashlib -import importlib from collections.abc import Iterable from dataclasses import dataclass from functools import lru_cache -from typing import TYPE_CHECKING + +import tree_sitter_cpp +import tree_sitter_javascript +from tree_sitter import Language, Node, Parser from .errors import DocForgeError from .models import LogicEdge, LogicNode, LogicProjection -if TYPE_CHECKING: - from tree_sitter import Language, Node - _COMMENT_NODE_TYPES = frozenset({"comment"}) -_SCRIPT_LANGUAGES = frozenset({"javascript", "typescript"}) @dataclass(frozen=True) @@ -81,34 +79,7 @@ class _LanguageProfile: def _javascript_profile() -> _LanguageProfile: return _LanguageProfile( name="javascript", - language=_optional_language( - grammar_module="tree_sitter_javascript", - grammar_function="language", - extra="javascript", - ), - root_type="program", - block_types=frozenset({"program", "statement_block"}), - function_types=frozenset( - {"function_declaration", "generator_function_declaration", "method_definition"} - ), - loop_types=frozenset( - {"while_statement", "do_statement", "for_statement", "for_in_statement"} - ), - return_types=frozenset({"return_statement"}), - raise_types=frozenset({"throw_statement"}), - switch_case_types=frozenset({"switch_case", "switch_default"}), - ) - - -@lru_cache(maxsize=1) -def _typescript_profile() -> _LanguageProfile: - return _LanguageProfile( - name="typescript", - language=_optional_language( - grammar_module="tree_sitter_typescript", - grammar_function="language_typescript", - extra="typescript", - ), + language=Language(tree_sitter_javascript.language()), root_type="program", block_types=frozenset({"program", "statement_block"}), function_types=frozenset( @@ -127,11 +98,7 @@ def _typescript_profile() -> _LanguageProfile: def _cpp_profile() -> _LanguageProfile: return _LanguageProfile( name="cpp", - language=_optional_language( - grammar_module="tree_sitter_cpp", - grammar_function="language", - extra="cpp", - ), + language=Language(tree_sitter_cpp.language()), root_type="translation_unit", block_types=frozenset({"translation_unit", "compound_statement"}), function_types=frozenset({"function_definition"}), @@ -149,28 +116,6 @@ def _cpp_profile() -> _LanguageProfile: ) -def _optional_language( - *, - grammar_module: str, - grammar_function: str, - extra: str, -) -> Language: - try: - tree_sitter = importlib.import_module("tree_sitter") - grammar = importlib.import_module(grammar_module) - except ModuleNotFoundError as error: - raise DocForgeError( - "optional_dependency_missing", - "The requested language frontend is not installed", - extra=extra, - install=f"docforge[{extra}]", - missing_module=error.name, - ) from error - language_type = tree_sitter.Language - language_factory = getattr(grammar, grammar_function) - return language_type(language_factory()) - - def analyze_javascript_source( source: str, *, @@ -191,38 +136,12 @@ def analyze_javascript_source( ) -def analyze_typescript_source( - source: str, - *, - source_id: str, - owners: Iterable[TreeSitterLogicOwner], - filename: str = "", - max_nodes_per_function: int = 2_000, -) -> tuple[LogicProjection, ...]: - """Build control-flow projections for named TypeScript functions and methods.""" - - return _analyze_tree_sitter_source( - source, - source_id=source_id, - owners=owners, - filename=filename, - profile=_typescript_profile(), - max_nodes_per_function=max_nodes_per_function, - ) - - def discover_javascript_functions(source: str) -> tuple[DiscoveredFunction, ...]: """Return named JavaScript functions, methods, and assigned arrow functions.""" return _discover_functions(source, _javascript_profile()) -def discover_typescript_functions(source: str) -> tuple[DiscoveredFunction, ...]: - """Return named TypeScript functions, methods, and assigned arrow functions.""" - - return _discover_functions(source, _typescript_profile()) - - def discover_cpp_functions(source: str) -> tuple[DiscoveredFunction, ...]: """Return named C++ functions and methods.""" @@ -253,8 +172,6 @@ def _discover_functions( source: str, profile: _LanguageProfile, ) -> tuple[DiscoveredFunction, ...]: - from tree_sitter import Parser - raw = source.encode("utf-8") parser = Parser(profile.language) tree = parser.parse(raw) @@ -295,8 +212,6 @@ def _analyze_tree_sitter_source( profile: _LanguageProfile, max_nodes_per_function: int, ) -> tuple[LogicProjection, ...]: - from tree_sitter import Parser - if max_nodes_per_function < 2: raise ValueError("max_nodes_per_function must allow entry and exit nodes") raw = source.encode("utf-8") @@ -378,7 +293,7 @@ def _function_definitions( def _scope_name(node: Node, raw: bytes, profile: _LanguageProfile) -> str | None: - if profile.name in _SCRIPT_LANGUAGES and node.type in {"class_declaration", "class"}: + if profile.name == "javascript" and node.type in {"class_declaration", "class"}: return _field_text(node, "name", raw) if profile.name == "cpp" and node.type in { "namespace_definition", @@ -392,11 +307,11 @@ def _scope_name(node: Node, raw: bytes, profile: _LanguageProfile) -> str | None def _function_name(node: Node, raw: bytes, profile: _LanguageProfile) -> str | None: if node.type in profile.function_types: - if profile.name in _SCRIPT_LANGUAGES: + if profile.name == "javascript": return _field_text(node, "name", raw) declarator = node.child_by_field_name("declarator") return _declarator_name(declarator, raw) if declarator is not None else None - if profile.name not in _SCRIPT_LANGUAGES or node.type != "variable_declarator": + if profile.name != "javascript" or node.type != "variable_declarator": return None value = node.child_by_field_name("value") if value is None or value.type not in {"arrow_function", "function_expression"}: diff --git a/tests/fixtures/reference-cpp/compile_commands.json b/tests/fixtures/reference-cpp/compile_commands.json deleted file mode 100644 index a34b931..0000000 --- a/tests/fixtures/reference-cpp/compile_commands.json +++ /dev/null @@ -1,19 +0,0 @@ -[ - { - "arguments": [ - "c++", - "-std=c++20", - "-Iinclude", - "-c", - "src/main.cpp" - ], - "directory": ".", - "file": "src/main.cpp" - }, - { - "command": "c++ -std=c++20 -Iinclude -c src/worker.cpp", - "directory": ".", - "file": "src/worker.cpp", - "output": "worker.o" - } -] diff --git a/tests/fixtures/reference-cpp/include/config.hpp b/tests/fixtures/reference-cpp/include/config.hpp deleted file mode 100644 index 7e57679..0000000 --- a/tests/fixtures/reference-cpp/include/config.hpp +++ /dev/null @@ -1,15 +0,0 @@ -#pragma once - -#include "detail.hpp" - -namespace demo { - -struct Config { - int limit; -}; - -inline int effective_limit(const Config& config) { - return normalize(config.limit); -} - -} // namespace demo diff --git a/tests/fixtures/reference-cpp/include/detail.hpp b/tests/fixtures/reference-cpp/include/detail.hpp deleted file mode 100644 index 63a76f6..0000000 --- a/tests/fixtures/reference-cpp/include/detail.hpp +++ /dev/null @@ -1,12 +0,0 @@ -#pragma once - -namespace demo { - -inline int normalize(int value) { - if (value < 0) { - return 0; - } - return value; -} - -} // namespace demo diff --git a/tests/fixtures/reference-cpp/src/main.cpp b/tests/fixtures/reference-cpp/src/main.cpp deleted file mode 100644 index 6f298ae..0000000 --- a/tests/fixtures/reference-cpp/src/main.cpp +++ /dev/null @@ -1,17 +0,0 @@ -#include "../include/config.hpp" - -namespace demo { - -int execute(int value) { - Config config{value}; - if (value == 0) { - return effective_limit(config); - } - return normalize(value); -} - -} // namespace demo - -int main() { - return demo::execute(3); -} diff --git a/tests/fixtures/reference-cpp/src/worker.cpp b/tests/fixtures/reference-cpp/src/worker.cpp deleted file mode 100644 index 1d1cbcb..0000000 --- a/tests/fixtures/reference-cpp/src/worker.cpp +++ /dev/null @@ -1,17 +0,0 @@ -#include "../include/config.hpp" - -namespace demo { - -class Worker { -public: - int run(int value) { - for (int attempt = 0; attempt < 2; ++attempt) { - if (value > attempt) { - return effective_limit(Config{value}); - } - } - return 0; - } -}; - -} // namespace demo diff --git a/tests/fixtures/reference-javascript/src/app/index.js b/tests/fixtures/reference-javascript/src/app/index.js deleted file mode 100644 index 4721b77..0000000 --- a/tests/fixtures/reference-javascript/src/app/index.js +++ /dev/null @@ -1,8 +0,0 @@ -import { Service } from "./service.mjs"; -export { clamp } from "./shared.js"; - -throw new Error("The reference adapter must never execute project code"); - -export function applicationName() { - return Service.name; -} diff --git a/tests/fixtures/reference-javascript/src/app/service.mjs b/tests/fixtures/reference-javascript/src/app/service.mjs deleted file mode 100644 index 80a7882..0000000 --- a/tests/fixtures/reference-javascript/src/app/service.mjs +++ /dev/null @@ -1,14 +0,0 @@ -import { clamp } from "./shared.js"; - -export class Service { - run(value) { - return clamp(value); - } -} - -export const buildService = (enabled) => { - if (enabled) { - return new Service(); - } - return null; -}; diff --git a/tests/fixtures/reference-javascript/src/app/shared.js b/tests/fixtures/reference-javascript/src/app/shared.js deleted file mode 100644 index e502fbe..0000000 --- a/tests/fixtures/reference-javascript/src/app/shared.js +++ /dev/null @@ -1,11 +0,0 @@ -export const DEFAULT_LIMIT = 3; - -export function clamp(value, limit = DEFAULT_LIMIT) { - if (value < 0) { - return 0; - } - if (value > limit) { - return limit; - } - return value; -} diff --git a/tests/fixtures/reference-javascript/src/app/worker.cjs b/tests/fixtures/reference-javascript/src/app/worker.cjs deleted file mode 100644 index 30592c9..0000000 --- a/tests/fixtures/reference-javascript/src/app/worker.cjs +++ /dev/null @@ -1,5 +0,0 @@ -export { Service } from "./service.mjs"; - -export function execute(worker, value) { - return worker.run(value); -} diff --git a/tests/fixtures/reference-python/src/sample/__init__.py b/tests/fixtures/reference-python/src/sample/__init__.py deleted file mode 100644 index cc3b58a..0000000 --- a/tests/fixtures/reference-python/src/sample/__init__.py +++ /dev/null @@ -1,7 +0,0 @@ -"""Small multi-file package for the Python reference adapter.""" - -from .service import Service - - -def package_name() -> str: - return Service.__name__ diff --git a/tests/fixtures/reference-python/src/sample/service.py b/tests/fixtures/reference-python/src/sample/service.py deleted file mode 100644 index b559183..0000000 --- a/tests/fixtures/reference-python/src/sample/service.py +++ /dev/null @@ -1,18 +0,0 @@ -"""Service behavior with class, method, function, and local import evidence.""" - -import json - -from .shared import clamp - -raise RuntimeError("The reference adapter must never execute project code") - - -class Service: - def run(self, value: int) -> int: - return clamp(value) if json.__name__ else value - - -async def build_service(enabled: bool) -> Service | None: - if enabled: - return Service() - return None diff --git a/tests/fixtures/reference-python/src/sample/shared.py b/tests/fixtures/reference-python/src/sample/shared.py deleted file mode 100644 index 42a1fe9..0000000 --- a/tests/fixtures/reference-python/src/sample/shared.py +++ /dev/null @@ -1,11 +0,0 @@ -"""Shared values used by dependent modules.""" - -DEFAULT_LIMIT = 3 - - -def clamp(value: int, limit: int = DEFAULT_LIMIT) -> int: - if value < 0: - return 0 - if value > limit: - return limit - return value diff --git a/tests/fixtures/reference-python/src/sample/worker.py b/tests/fixtures/reference-python/src/sample/worker.py deleted file mode 100644 index 7d1962d..0000000 --- a/tests/fixtures/reference-python/src/sample/worker.py +++ /dev/null @@ -1,8 +0,0 @@ -"""A transitive dependent used to prove reverse invalidation.""" - -from sample.service import Service - - -def execute(value: int) -> int: - worker = Service() - return worker.run(value) diff --git a/tests/fixtures/reference-typescript/src/app/index.ts b/tests/fixtures/reference-typescript/src/app/index.ts deleted file mode 100644 index 30e3fb4..0000000 --- a/tests/fixtures/reference-typescript/src/app/index.ts +++ /dev/null @@ -1,8 +0,0 @@ -import { Service } from "./service"; -export type { Choice } from "./types"; - -throw new Error("The reference adapter must never execute project code"); - -export function applicationName(): string { - return Service.name; -} diff --git a/tests/fixtures/reference-typescript/src/app/service.ts b/tests/fixtures/reference-typescript/src/app/service.ts deleted file mode 100644 index aec4304..0000000 --- a/tests/fixtures/reference-typescript/src/app/service.ts +++ /dev/null @@ -1,17 +0,0 @@ -import type { Choice } from "./types"; - -export class Service { - run(choice: Choice): number { - if (choice.enabled) { - return choice.value; - } - return 0; - } -} - -export const buildService = (choice: Choice): Service | null => { - if (choice.enabled) { - return new Service(); - } - return null; -}; diff --git a/tests/fixtures/reference-typescript/src/app/types.mts b/tests/fixtures/reference-typescript/src/app/types.mts deleted file mode 100644 index 9d09da8..0000000 --- a/tests/fixtures/reference-typescript/src/app/types.mts +++ /dev/null @@ -1,6 +0,0 @@ -export interface Choice { - enabled: boolean; - value: number; -} - -export const DEFAULT_LIMIT: number = 3; diff --git a/tests/fixtures/reference-typescript/src/app/worker.cts b/tests/fixtures/reference-typescript/src/app/worker.cts deleted file mode 100644 index 4fc33e1..0000000 --- a/tests/fixtures/reference-typescript/src/app/worker.cts +++ /dev/null @@ -1,6 +0,0 @@ -import { Service } from "./service"; -import type { Choice } from "./types"; - -export function execute(worker: Service, choice: Choice): number { - return worker.run(choice); -} diff --git a/tests/test_adapter_contract.py b/tests/test_adapter_contract.py index ee1b183..e4bc3f5 100644 --- a/tests/test_adapter_contract.py +++ b/tests/test_adapter_contract.py @@ -39,7 +39,6 @@ from docforge.mcp_server import ( ) from docforge.models import ( Edge, - Limits, LogicEdge, LogicNode, LogicProjection, @@ -249,36 +248,6 @@ class NonLogicIncrementalLoader(IncrementalLoader): return replace(super().extract_source(source), logic=()) -class CompleteLogicIncrementalLoader(IncrementalLoader): - """Supply an independent complete Logic oracle alongside the legacy graph oracle.""" - - def __init__(self, root: Path, *, mismatched_logic: bool = False) -> None: - super().__init__(root) - self.mismatched_logic = mismatched_logic - - def load_complete_assembly(self) -> AdapterAssembly: - projection = super().load_projection() - if "guide.workflow" not in self.sources: - return AdapterAssembly(projection) - manifest = self.load_manifest() - workflow = next( - source for source in manifest.sources if source.source_id == "guide.workflow" - ) - logic = self.extract_source(workflow).logic - if self.mismatched_logic: - current = logic[0] - logic = ( - replace( - current, - nodes=( - current.nodes[0], - replace(current.nodes[1], label="Different complete return"), - ), - ), - ) - return AdapterAssembly(projection, logic) - - class AdapterContractTests(unittest.TestCase): def projection(self, root: Path) -> AdapterProjection: foundation = Node( @@ -342,23 +311,6 @@ class AdapterContractTests(unittest.TestCase): self.assertEqual("Workflow", index.get_node("guide.workflow")["node"]["title"]) self.assertEqual(projection.identity(), projection.identity()) - def test_adapter_project_enforces_graph_bounds_before_publication(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = Path(directory).resolve() - project = AdapterProject( - Loader(self.projection(root)), - cache_root=root / ".cache" / "bounded", - settings=AdapterProjectSettings(limits=Limits(max_nodes=1)), - ) - - with self.assertRaises(DocForgeError) as captured: - project.load() - - self.assertEqual("adapter_limit", captured.exception.code) - self.assertEqual("nodes", captured.exception.details["kind"]) - self.assertEqual(1, captured.exception.details["maximum"]) - self.assertEqual(2, captured.exception.details["actual"]) - def test_projection_rejects_unsorted_metadata_graph_and_identity_changes(self) -> None: with tempfile.TemporaryDirectory() as directory: root = Path(directory).resolve() @@ -830,46 +782,6 @@ class AdapterContractTests(unittest.TestCase): self.assertEqual("ok", equivalent["status"]) self.assertEqual(1, equivalent["node_count"]) - def test_logic_equivalence_requires_an_independent_complete_oracle(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = Path(directory).resolve() - project = AdapterProject( - IncrementalLoader(root), - cache_root=root / ".cache" / "logic-without-oracle", - ) - - with self.assertRaises(DocForgeError) as captured: - project.verify_incremental_equivalence() - - self.assertEqual("complete_logic_oracle_required", captured.exception.code) - - def test_logic_equivalence_uses_the_complete_assembly_oracle(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = Path(directory).resolve() - project = AdapterProject( - CompleteLogicIncrementalLoader(root), - cache_root=root / ".cache" / "logic-oracle", - ) - - result = project.verify_incremental_equivalence() - - self.assertEqual("ok", result["status"]) - self.assertEqual(1, result["logic_projection_count"]) - - def test_logic_equivalence_reports_exact_mismatch_field(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = Path(directory).resolve() - project = AdapterProject( - CompleteLogicIncrementalLoader(root, mismatched_logic=True), - cache_root=root / ".cache" / "logic-mismatch", - ) - - with self.assertRaises(DocForgeError) as captured: - project.verify_incremental_equivalence() - - self.assertEqual("incremental_mismatch", captured.exception.code) - self.assertEqual(["logic"], captured.exception.details["fields"]) - def test_incremental_adapter_can_assemble_overlapping_source_evidence(self) -> None: with tempfile.TemporaryDirectory() as directory: root = Path(directory).resolve() diff --git a/tests/test_adapter_launcher.py b/tests/test_adapter_launcher.py deleted file mode 100644 index e03df33..0000000 --- a/tests/test_adapter_launcher.py +++ /dev/null @@ -1,475 +0,0 @@ -from __future__ import annotations - -import asyncio -import json -import os -import shutil -import subprocess -import sys -import tempfile -import tomllib -import unittest -import venv -from dataclasses import replace -from pathlib import Path -from unittest import mock - -from jsonschema import Draft202012Validator -from mcp import ClientSession, StdioServerParameters -from mcp.client.stdio import stdio_client -from referencing import Registry, Resource - -from docforge.adapter_contract import AdapterNode, AdapterProject, AdapterProjection -from docforge.adapter_launcher import AdapterLauncherV1 -from docforge.client_config import ( - _validate_adapter_configuration_result, - generate_adapter_client_configuration, - generate_client_configuration, -) -from docforge.errors import DocForgeError -from docforge.models import Node -from docforge.project import Project -from docforge.reference_mcp import ( - REFERENCE_MCP_MODULE, - create_reference_project, -) - -ROOT = Path(__file__).resolve().parents[1] -FIXTURES = ROOT / "tests" / "fixtures" -SCHEMAS = ROOT / "schemas" -LAUNCHER_SCHEMA = json.loads((SCHEMAS / "adapter-launcher.schema.json").read_text(encoding="utf-8")) -ADAPTER_CONFIGURATION_SCHEMA = json.loads( - (SCHEMAS / "adapter-client-configuration.schema.json").read_text(encoding="utf-8") -) -GENERIC_CONFIGURATION_SCHEMA = json.loads( - (SCHEMAS / "client-configuration.schema.json").read_text(encoding="utf-8") -) -SCHEMA_REGISTRY = Registry().with_resources( - ( - (LAUNCHER_SCHEMA["$id"], Resource.from_contents(LAUNCHER_SCHEMA)), - ( - GENERIC_CONFIGURATION_SCHEMA["$id"], - Resource.from_contents(GENERIC_CONFIGURATION_SCHEMA), - ), - ( - ADAPTER_CONFIGURATION_SCHEMA["$id"], - Resource.from_contents(ADAPTER_CONFIGURATION_SCHEMA), - ), - ) -) -ADAPTER_CONFIGURATION_VALIDATOR = Draft202012Validator( - ADAPTER_CONFIGURATION_SCHEMA, - registry=SCHEMA_REGISTRY, -) - - -class Loader: - def __init__(self, projection: AdapterProjection) -> None: - self.projection = projection - self.load_calls = 0 - - def load_projection(self) -> AdapterProjection: - self.load_calls += 1 - return self.projection - - -class DriftingLoader(Loader): - def __init__( - self, - projection: AdapterProjection, - changed: AdapterProjection, - ) -> None: - super().__init__(projection) - self.changed = changed - - def load_projection(self) -> AdapterProjection: - self.load_calls += 1 - return self.projection if self.load_calls < 3 else self.changed - - -class AdapterLauncherTests(unittest.TestCase): - def copy_fixture(self, destination: Path) -> Path: - root = destination / "alpha" - shutil.copytree(FIXTURES / "alpha", root) - return root - - def copy_reference_fixture(self, destination: Path) -> Path: - root = destination / "reference-python" - shutil.copytree(FIXTURES / "reference-python", root) - config = root / ".docforge" / "reference-adapter.toml" - config.parent.mkdir(parents=True) - config.write_text( - "\n".join( - ( - "schema_version = 1", - 'project_id = "reference-python"', - 'title = "Runnable Python reference"', - 'language = "python"', - 'source_roots = ["src"]', - "", - ) - ), - encoding="utf-8", - ) - return root.resolve() - - def projection( - self, - root: Path, - *, - source_hash: str = "3" * 64, - ) -> AdapterProjection: - node = Node( - node_id="source.entry", - title="Entry", - family="source", - authority="derived", - status="active", - tags=("python",), - summary="Reference adapter entry.", - content="Reference adapter content.", - source_path="src/entry.py", - source_anchor="L1", - content_hash="2" * 64, - ) - return AdapterProjection( - project_id="adapter-client-fixture", - title="Adapter client fixture", - adapter_id="fixture-client", - adapter_version="1", - root=root, - revision="fixture-revision", - source_hash=source_hash, - nodes=(AdapterNode(node),), - edges=(), - ) - - def project( - self, - root: Path, - *, - loader: Loader | None = None, - ) -> tuple[AdapterProject, Loader]: - effective_loader = loader or Loader(self.projection(root)) - return ( - AdapterProject( - effective_loader, - cache_root=root / ".docforge-cache" / "adapter-client", - ), - effective_loader, - ) - - def test_launcher_and_all_client_fragments_are_deterministic_and_schema_valid(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = Path(directory).resolve() - project, _ = self.project(root) - launcher = AdapterLauncherV1.for_project( - project, - module=REFERENCE_MCP_MODULE, - ) - Draft202012Validator(LAUNCHER_SCHEMA).validate(launcher.as_dict()) - previous = os.environ.get("DOCFORGE_ADAPTER_LAUNCHER_SECRET") - os.environ["DOCFORGE_ADAPTER_LAUNCHER_SECRET"] = "must-not-appear" - try: - with mock.patch("docforge.client_config.subprocess.run") as executed: - results: dict[str, dict[str, object]] = {} - for client in ("codex", "claude", "openclaw"): - first = generate_adapter_client_configuration( - project, - launcher, - client, - no_ast=True, - ) - second = generate_adapter_client_configuration( - project, - launcher, - client, - no_ast=True, - ) - self.assertEqual(first, second) - ADAPTER_CONFIGURATION_VALIDATOR.validate(first) - self.assertEqual(launcher.launcher_hash, first["launcher_hash"]) - self.assertEqual( - launcher.launcher_hash, - first["binding"]["launcher_hash"], - ) - self.assertEqual( - [ - "-I", - "-m", - REFERENCE_MCP_MODULE, - "--project-root", - str(root), - "--capability-mode", - "read", - "--no-ast", - ], - first["binding"]["args"], - ) - self.assertEqual({}, first["binding"]["environment"]) - self.assertNotIn("cwd", first["binding"]) - self.assertNotIn( - "must-not-appear", - json.dumps(first, sort_keys=True), - ) - results[client] = first - executed.assert_not_called() - finally: - if previous is None: - os.environ.pop("DOCFORGE_ADAPTER_LAUNCHER_SECRET", None) - else: - os.environ["DOCFORGE_ADAPTER_LAUNCHER_SECRET"] = previous - - self.assertIn( - "mcp_servers", - tomllib.loads(results["codex"]["artifact"]["content"]), - ) - self.assertIn( - "mcpServers", - json.loads(results["claude"]["artifact"]["content"]), - ) - self.assertIn( - "mcp", - json.loads(results["openclaw"]["artifact"]["content"]), - ) - - def test_launcher_rejects_module_path_entry_point_and_argument_injection(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = Path(directory).resolve() - project, _ = self.project(root) - launcher = AdapterLauncherV1.for_project( - project, - module=REFERENCE_MCP_MODULE, - ) - for module in ( - "-m", - "fixture_adapter.mcp_server", - "fixture_adapter.mcp_server --debug", - "fixture_adapter:mcp_server", - "fixture_adapter/mcp_server", - ".fixture_adapter", - "docforge.mcp_server", - "fixture_adapter.$server", - ): - with self.subTest(module=module), self.assertRaises(DocForgeError) as captured: - replace(launcher, module=module) - self.assertEqual("invalid_adapter_launcher", captured.exception.code) - - with self.assertRaises(DocForgeError): - replace(launcher, project_root=Path("relative/project")) - with self.assertRaises(DocForgeError): - replace(launcher, entry_point="console-script") # type: ignore[arg-type] - payload = launcher.as_dict() - payload["arguments"] = ["--shell", "command"] - with self.assertRaises(TypeError): - AdapterLauncherV1(**payload) # type: ignore[arg-type] - - result = generate_adapter_client_configuration(project, launcher, "codex") - result["binding"]["args"].append("--arbitrary") - with self.assertRaisesRegex(AssertionError, "Generated adapter client"): - _validate_adapter_configuration_result( - result, - project=project, - launcher=launcher, - source_availability=project_source_availability(result), - ) - - def test_wrong_project_descriptor_and_source_drift_fail_closed(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - first_root = (parent / "first").resolve() - second_root = (parent / "second").resolve() - first_root.mkdir() - second_root.mkdir() - first, _ = self.project(first_root) - second, _ = self.project(second_root) - launcher = AdapterLauncherV1.for_project( - first, - module=REFERENCE_MCP_MODULE, - ) - - with self.assertRaises(DocForgeError) as wrong_project: - generate_adapter_client_configuration(second, launcher, "codex") - self.assertEqual("adapter_launcher_mismatch", wrong_project.exception.code) - with self.assertRaises(DocForgeError) as descriptor_drift: - generate_adapter_client_configuration( - first, - replace(launcher, descriptor_hash="0" * 64), - "codex", - ) - self.assertEqual("adapter_launcher_mismatch", descriptor_drift.exception.code) - - base = self.projection(first_root) - changed = replace(base, source_hash="4" * 64) - drifting, _ = self.project( - first_root, - loader=DriftingLoader(base, changed), - ) - drifting_launcher = AdapterLauncherV1.for_project( - drifting, - module=REFERENCE_MCP_MODULE, - ) - with self.assertRaises(DocForgeError) as source_drift: - generate_adapter_client_configuration( - drifting, - drifting_launcher, - "openclaw", - ) - self.assertEqual("source_changed", source_drift.exception.code) - - def test_generic_configuration_remains_separate_and_compatible(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - generic = Project.open(root) - result = generate_client_configuration(generic, "codex", no_ast=True) - Draft202012Validator(GENERIC_CONFIGURATION_SCHEMA).validate(result) - self.assertEqual("client.configure", result["operation"]) - self.assertNotIn("launcher", result) - with self.assertRaises(DocForgeError) as generic_launcher: - AdapterLauncherV1.for_project( - generic, - module=REFERENCE_MCP_MODULE, - ) - self.assertEqual("adapter_launcher_unavailable", generic_launcher.exception.code) - - custom_root = (Path(directory) / "custom").resolve() - custom_root.mkdir() - custom, _ = self.project(custom_root) - with self.assertRaises(DocForgeError) as generic_api: - generate_client_configuration(custom, "codex") - self.assertEqual("missing_config", generic_api.exception.code) - - def test_installed_project_owned_top_level_module_launches_without_probe_execution( - self, - ) -> None: - with tempfile.TemporaryDirectory() as directory: - root = Path(directory).resolve() - environment = root / ".launcher-venv" - venv.EnvBuilder(with_pip=False).create(environment) - executable = environment / "bin" / "python" - site_packages = Path( - subprocess.run( - [ - str(executable), - "-I", - "-c", - "import site; print(site.getsitepackages()[0])", - ], - check=True, - capture_output=True, - text=True, - ).stdout.strip() - ) - (site_packages / "fixture-adapter.pth").write_text( - f"{root}\n", - encoding="utf-8", - ) - sentinel = root / "fixture-adapter-executed" - (root / "fixture_adapter.py").write_text( - "\n".join( - ( - "import argparse", - "import json", - "from pathlib import Path", - "parser = argparse.ArgumentParser()", - 'parser.add_argument("--project-root", required=True)', - ( - 'parser.add_argument("--capability-mode", ' - 'choices=("read",), required=True)' - ), - "arguments = parser.parse_args()", - f"Path({str(sentinel)!r}).write_text('executed', encoding='utf-8')", - ( - "print(json.dumps({'project_root': arguments.project_root}, " - "sort_keys=True))" - ), - "", - ) - ), - encoding="utf-8", - ) - - project, _ = self.project(root) - with mock.patch.object(sys, "executable", str(executable)): - launcher = AdapterLauncherV1.for_project( - project, - module="fixture_adapter", - ) - result = generate_adapter_client_configuration( - project, - launcher, - "codex", - ) - self.assertFalse(sentinel.exists()) - Draft202012Validator(LAUNCHER_SCHEMA).validate(launcher.as_dict()) - binding = result["binding"] - completed = subprocess.run( - [binding["command"], *binding["args"]], - check=True, - capture_output=True, - text=True, - timeout=10, - ) - self.assertEqual({"project_root": str(root)}, json.loads(completed.stdout)) - self.assertEqual("executed", sentinel.read_text(encoding="utf-8")) - - def test_uninstalled_packages_and_non_project_modules_fail_closed(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = Path(directory).resolve() - project, _ = self.project(root) - for module, code in ( - ("fixture_adapter", "adapter_launcher_unavailable"), - ("os", "adapter_launcher_unavailable"), - ("json", "invalid_adapter_launcher"), - ): - with self.subTest(module=module), self.assertRaises(DocForgeError) as captured: - AdapterLauncherV1.for_project(project, module=module) - self.assertEqual(code, captured.exception.code) - - def test_generated_reference_binding_starts_a_real_isolated_mcp_process(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_reference_fixture(Path(directory)) - project = create_reference_project(root) - launcher = AdapterLauncherV1.for_project( - project, - module=REFERENCE_MCP_MODULE, - ) - result = generate_adapter_client_configuration( - project, - launcher, - "codex", - ) - binding = result["binding"] - - async def inspect() -> tuple[str, ...]: - parameters = StdioServerParameters( - command=binding["command"], - args=binding["args"], - env=binding["environment"], - ) - async with ( - stdio_client(parameters) as streams, - ClientSession(*streams) as session, - ): - await session.initialize() - return tuple(tool.name for tool in (await session.list_tools()).tools) - - tools = asyncio.run(inspect()) - self.assertIn("docforge_bootstrap", tools) - - -def project_source_availability(result: dict[str, object]): - from docforge.adapter_launcher import AdapterSourceAvailabilityV1 - - payload = result["source_availability"] - return AdapterSourceAvailabilityV1( - schema_version=payload["schema_version"], - status=payload["status"], - method=payload["method"], - revision=payload["revision"], - source_hash=payload["source_hash"], - ) - - -if __name__ == "__main__": - unittest.main() diff --git a/tests/test_adapter_sdk.py b/tests/test_adapter_sdk.py deleted file mode 100644 index 059d095..0000000 --- a/tests/test_adapter_sdk.py +++ /dev/null @@ -1,261 +0,0 @@ -from __future__ import annotations - -import hashlib -import tempfile -import unittest -from dataclasses import replace -from pathlib import Path - -import docforge.adapter_sdk as sdk -from docforge.errors import DocForgeError - - -class ReferenceIncrementalAdapter: - """Small public-SDK-only adapter used to prove the conformance entry point.""" - - def __init__(self, root: Path) -> None: - self.root = root - - @staticmethod - def _hash(value: str) -> str: - return hashlib.sha256(value.encode()).hexdigest() - - def _node(self) -> sdk.AdapterNode: - content = "A deterministic reference function." - return sdk.AdapterNode( - sdk.Node( - node_id="reference.run", - title="Reference run", - family="source", - authority="derived", - status="current", - tags=("python",), - summary=content, - content=content, - source_path="src/reference.py", - source_anchor="L1", - content_hash=self._hash(content), - ) - ) - - @staticmethod - def _logic() -> sdk.LogicProjection: - return sdk.LogicProjection( - owner_node_id="reference.run", - source_id="source.reference", - nodes=( - sdk.LogicNode("entry", "entry", "Enter run", "L1"), - sdk.LogicNode("return", "return", "Return", "L2"), - ), - edges=(sdk.LogicEdge("entry", "return", "return", "RETURN", 0),), - ) - - def load_manifest(self) -> sdk.AdapterManifest: - fingerprint = self._hash("def run(): return True\n") - return sdk.AdapterManifest( - project_id="sdk-reference", - title="SDK reference", - adapter_id="python-reference", - adapter_version="1", - root=self.root, - revision=fingerprint[:12], - source_hash=fingerprint, - families=("source",), - allowed_relations=(), - sources=( - sdk.AdapterSource( - source_id="source.reference", - source_path="src/reference.py", - fingerprint=fingerprint, - extractor_version="python-reference@1", - ), - ), - estimated_nodes=1, - ) - - def extract_source(self, source: sdk.AdapterSource) -> sdk.AdapterSourceProjection: - return sdk.AdapterSourceProjection( - source_id=source.source_id, - fingerprint=source.fingerprint, - nodes=(self._node(),), - edges=(), - logic=(self._logic(),), - ) - - def load_projection(self) -> sdk.AdapterProjection: - manifest = self.load_manifest() - return sdk.AdapterProjection( - project_id=manifest.project_id, - title=manifest.title, - adapter_id=manifest.adapter_id, - adapter_version=manifest.adapter_version, - root=manifest.root, - revision=manifest.revision, - source_hash=manifest.source_hash, - nodes=(self._node(),), - edges=(), - ) - - def load_complete_assembly(self) -> sdk.AdapterAssembly: - return sdk.AdapterAssembly( - projection=self.load_projection(), - logic=(self._logic(),), - ) - - -class LegacyAdapter: - def __init__(self, projection: sdk.AdapterProjection) -> None: - self.projection = projection - - def load_projection(self) -> sdk.AdapterProjection: - return self.projection - - -class LogicWithoutCompleteAdapter: - def __init__(self, root: Path) -> None: - self.reference = ReferenceIncrementalAdapter(root) - - def load_manifest(self) -> sdk.AdapterManifest: - return self.reference.load_manifest() - - def extract_source(self, source: sdk.AdapterSource) -> sdk.AdapterSourceProjection: - return self.reference.extract_source(source) - - def load_projection(self) -> sdk.AdapterProjection: - return self.reference.load_projection() - - -class NondeterministicCompleteAdapter(ReferenceIncrementalAdapter): - def __init__(self, root: Path) -> None: - super().__init__(root) - self.complete_calls = 0 - - def load_complete_assembly(self) -> sdk.AdapterAssembly: - self.complete_calls += 1 - assembly = super().load_complete_assembly() - if self.complete_calls % 2 == 0: - logic = assembly.logic[0] - return replace( - assembly, - logic=( - replace( - logic, - nodes=( - logic.nodes[0], - replace(logic.nodes[1], label="Unstable return"), - ), - ), - ), - ) - return assembly - - -class AdapterSdkTests(unittest.TestCase): - def test_public_facade_contains_the_complete_authoring_surface(self) -> None: - expected = { - "AdapterAssembly", - "AdapterConformanceReport", - "AdapterManifest", - "AdapterNode", - "AdapterProjection", - "AdapterSource", - "AdapterSourceProjection", - "CompleteAdapterAssemblyLoader", - "Edge", - "LogicEdge", - "LogicNode", - "LogicProjection", - "Node", - "verify_adapter_conformance", - } - - self.assertLessEqual(expected, set(sdk.__all__)) - for name in expected: - with self.subTest(name=name): - self.assertTrue(hasattr(sdk, name)) - - def test_conformance_proves_deterministic_complete_graph_and_logic_parity(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = Path(directory).resolve() - report = sdk.verify_adapter_conformance( - ReferenceIncrementalAdapter(root), - cache_root=root / ".cache" / "conformance", - ) - - self.assertEqual(1, report.schema_version) - self.assertTrue(report.incremental) - self.assertEqual(1, report.node_count) - self.assertEqual(0, report.edge_count) - self.assertEqual(1, report.logic_projection_count) - self.assertEqual(64, len(report.assembly_hash)) - self.assertEqual(report.assembly_hash, report.as_dict()["assembly_hash"]) - - def test_conformance_keeps_load_projection_only_adapters_first_class(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = Path(directory).resolve() - content = "Legacy adapter content." - projection = sdk.AdapterProjection( - project_id="legacy-sdk", - title="Legacy SDK", - adapter_id="legacy", - adapter_version="1", - root=root, - revision="legacy", - source_hash=hashlib.sha256(content.encode()).hexdigest(), - nodes=( - sdk.AdapterNode( - sdk.Node( - node_id="legacy.node", - title="Legacy node", - family="guide", - authority="authoritative", - status="current", - tags=("legacy",), - summary=content, - content=content, - source_path="docs/legacy.md", - source_anchor=None, - content_hash=hashlib.sha256(content.encode()).hexdigest(), - ) - ), - ), - edges=(), - ) - - report = sdk.verify_adapter_conformance( - LegacyAdapter(projection), - cache_root=root / ".cache" / "legacy-conformance", - ) - - self.assertFalse(report.incremental) - self.assertEqual(0, report.logic_projection_count) - self.assertEqual(projection.identity(), projection.identity()) - - def test_conformance_requires_a_complete_oracle_for_incremental_logic(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = Path(directory).resolve() - - with self.assertRaises(DocForgeError) as captured: - sdk.verify_adapter_conformance( - LogicWithoutCompleteAdapter(root), - cache_root=root / ".cache" / "missing-logic-oracle", - ) - - self.assertEqual("complete_logic_oracle_required", captured.exception.code) - - def test_conformance_rejects_nondeterministic_complete_logic(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = Path(directory).resolve() - - with self.assertRaises(DocForgeError) as captured: - sdk.verify_adapter_conformance( - NondeterministicCompleteAdapter(root), - cache_root=root / ".cache" / "nondeterministic", - ) - - self.assertEqual("nondeterministic_adapter", captured.exception.code) - self.assertEqual(["logic"], captured.exception.details["fields"]) - - -if __name__ == "__main__": - unittest.main() diff --git a/tests/test_command_reference.py b/tests/test_command_reference.py deleted file mode 100644 index e0f1741..0000000 --- a/tests/test_command_reference.py +++ /dev/null @@ -1,141 +0,0 @@ -from __future__ import annotations - -import argparse -import hashlib -import json -import shutil -import tempfile -import unittest -from pathlib import Path - -from mcp.shared.memory import create_connected_server_and_client_session -from mcp.types import Tool - -from docforge.cli import _parser -from docforge.command_reference import ( - cli_command_references, - mcp_tool_references, - render_cli_reference_markdown, - render_command_reference_markdown, - render_mcp_reference_markdown, -) -from docforge.index import ProjectIndex -from docforge.mcp_server import ALL_TOOLS, APPLICATION_TOOLS, create_server -from docforge.project import Project - -ROOT = Path(__file__).resolve().parents[1] -FIXTURES = ROOT / "tests" / "fixtures" - - -class CommandReferenceTests(unittest.IsolatedAsyncioTestCase): - def copy_fixture(self, destination: Path) -> Path: - root = destination / "alpha" - shutil.copytree(FIXTURES / "alpha", root) - return root - - def test_cli_reference_is_derived_from_the_real_parser(self) -> None: - parser = _parser() - references = cli_command_references(parser) - subparsers = next( - action - for action in parser._actions # pyright: ignore[reportPrivateUsage] - if isinstance( - action, - argparse._SubParsersAction, # pyright: ignore[reportPrivateUsage] - ) - ) - expected = { - name: " ".join(command.format_usage().removeprefix("usage: ").split()) - for name, command in subparsers.choices.items() - } - - self.assertEqual(tuple(sorted(expected)), tuple(item.name for item in references)) - self.assertEqual(expected, {item.name: item.invocation for item in references}) - rendered = render_cli_reference_markdown(references) - self.assertEqual(rendered, render_cli_reference_markdown(reversed(references))) - self.assertIn("| `search` | `docforge search [-h] [--limit LIMIT] query` |", rendered) - self.assertIn( - "| `graph-render-status` | `docforge graph-render-status [-h] [view_id]` |", - rendered, - ) - - async def test_mcp_reference_is_derived_from_registered_tool_metadata(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - ProjectIndex(Project.open(root)).build() - async with create_connected_server_and_client_session( - create_server( - root, - "alpha-editor", - canonical_applier_id="alpha-editor", - ), - raise_exceptions=True, - ) as session: - tools = (await session.list_tools()).tools - - expected_names = (*ALL_TOOLS, *APPLICATION_TOOLS) - references = mcp_tool_references(tools, expected_names=expected_names) - by_name = {item.name: item for item in references} - tools_by_name = {tool.name: tool for tool in tools} - - self.assertEqual(set(expected_names), set(by_name)) - for name, tool in tools_by_name.items(): - with self.subTest(tool=name): - schema_hash = hashlib.sha256( - json.dumps( - tool.inputSchema, - sort_keys=True, - separators=(",", ":"), - ).encode() - ).hexdigest() - properties = set(tool.inputSchema.get("properties", {})) - required = set(tool.inputSchema.get("required", [])) - reference = by_name[name] - self.assertEqual(" ".join((tool.description or "").split()), reference.description) - self.assertEqual(tuple(sorted(required)), reference.required_arguments) - self.assertEqual( - tuple(sorted(properties - required)), - reference.optional_arguments, - ) - self.assertEqual(schema_hash, reference.input_schema_hash) - - rendered = render_mcp_reference_markdown(references) - self.assertEqual(rendered, render_mcp_reference_markdown(reversed(references))) - self.assertIn("| read | `docforge_bootstrap` | — | — |", rendered) - self.assertIn( - "| application | `docforge_apply_changeset` | " - "`changeset_id`, `expected_changeset_hash` | — |", - rendered, - ) - complete = render_command_reference_markdown( - cli_command_references(), - references, - ) - self.assertEqual(1, complete.count("## CLI commands")) - self.assertEqual(1, complete.count("## MCP tools")) - self.assertTrue(complete.startswith("# DocForge command reference\n")) - self.assertIn("Do not edit this file by hand.", complete) - self.assertIn("fixed reference-adapter server exposes only its read rows", complete) - - def test_mcp_reference_fails_closed_on_registration_drift(self) -> None: - known = Tool( - name="docforge_bootstrap", - description="Bootstrap.", - inputSchema={"type": "object", "properties": {}}, - ) - unknown = Tool( - name="docforge_unregistered", - description="Unknown.", - inputSchema={"type": "object", "properties": {}}, - ) - - with self.assertRaisesRegex(ValueError, "no declared capability surface"): - mcp_tool_references((unknown,)) - with self.assertRaisesRegex(ValueError, "missing="): - mcp_tool_references((known,), expected_names=ALL_TOOLS) - with self.assertRaisesRegex(ValueError, "repeats"): - mcp_tool_references((known, known)) - - -if __name__ == "__main__": - unittest.main() diff --git a/tests/test_command_reference_tool.py b/tests/test_command_reference_tool.py deleted file mode 100644 index 6498087..0000000 --- a/tests/test_command_reference_tool.py +++ /dev/null @@ -1,404 +0,0 @@ -from __future__ import annotations - -import asyncio -import contextlib -import hashlib -import io -import os -import tempfile -import threading -import unittest -from pathlib import Path -from unittest import mock - -from docforge.mcp_server import ALL_TOOLS, APPLICATION_TOOLS -from tools import generate_command_reference as command_reference_tool -from tools.generate_command_reference import ( - EXPECTED_CLI_ROWS, - EXPECTED_MCP_ROWS, - CommandReferenceDrift, - CommandReferenceToolError, - collect_command_reference_rows, - generate_command_reference_bytes, - main, - publish_or_check_command_reference, -) - -ROOT = Path(__file__).resolve().parents[1] -ALPHA = ROOT / "tests" / "fixtures" / "alpha" - - -def _tree_snapshot(root: Path) -> tuple[tuple[str, str, str], ...]: - records: list[tuple[str, str, str]] = [] - for path in sorted(root.rglob("*")): - relative = path.relative_to(root).as_posix() - if path.is_symlink(): - records.append((relative, "symlink", os.readlink(path))) - elif path.is_dir(): - records.append((relative, "directory", "")) - else: - records.append( - ( - relative, - "file", - hashlib.sha256(path.read_bytes()).hexdigest(), - ) - ) - return tuple(records) - - -class CommandReferenceToolTests(unittest.TestCase): - def test_generated_bytes_are_stable_and_never_mutate_the_fixture(self) -> None: - before = _tree_snapshot(ALPHA) - - first = generate_command_reference_bytes(ALPHA) - middle = _tree_snapshot(ALPHA) - second = generate_command_reference_bytes(ALPHA) - - self.assertEqual(first, second) - self.assertEqual(before, middle) - self.assertEqual(before, _tree_snapshot(ALPHA)) - self.assertTrue(first.endswith(b"\n")) - self.assertTrue(first.startswith(b"# DocForge command reference\n")) - self.assertIn(b"Do not edit this file by hand.", first) - self.assertEqual(1, first.count(b"## CLI commands")) - self.assertEqual(1, first.count(b"## MCP tools")) - - def test_registration_collection_is_complete(self) -> None: - cli_rows, mcp_rows = asyncio.run( - collect_command_reference_rows( - ALPHA, - proposal_writer="alpha-editor", - canonical_applier="alpha-editor", - ) - ) - - self.assertEqual(EXPECTED_CLI_ROWS, len(cli_rows)) - self.assertEqual(EXPECTED_MCP_ROWS, len(mcp_rows)) - self.assertEqual(len(cli_rows), len({item.name for item in cli_rows})) - self.assertEqual( - set((*ALL_TOOLS, *APPLICATION_TOOLS)), - {item.name for item in mcp_rows}, - ) - self.assertEqual( - {"read", "proposal", "application"}, - {item.surface for item in mcp_rows}, - ) - - def test_atomic_write_check_and_drift_detection(self) -> None: - content = generate_command_reference_bytes(ALPHA) - with tempfile.TemporaryDirectory() as directory: - repository = Path(directory).resolve() - (repository / "generated").mkdir() - output = Path("generated/command-reference.md") - target = repository / output - - self.assertEqual( - "written", - publish_or_check_command_reference( - content, - repository_root=repository, - project_root=ALPHA, - output=output, - check=False, - ), - ) - self.assertEqual(content, target.read_bytes()) - self.assertEqual( - "current", - publish_or_check_command_reference( - content, - repository_root=repository, - project_root=ALPHA, - output=output, - check=True, - ), - ) - target.write_text("stale\n", encoding="utf-8") - with self.assertRaises(CommandReferenceDrift): - publish_or_check_command_reference( - content, - repository_root=repository, - project_root=ALPHA, - output=output, - check=True, - ) - self.assertEqual(b"stale\n", target.read_bytes()) - self.assertEqual( - "written", - publish_or_check_command_reference( - content, - repository_root=repository, - project_root=ALPHA, - output=output, - check=False, - ), - ) - self.assertEqual(content, target.read_bytes()) - self.assertEqual( - "unchanged", - publish_or_check_command_reference( - content, - repository_root=repository, - project_root=ALPHA, - output=output, - check=False, - ), - ) - - def test_output_confinement_rejects_escapes_and_symlinks(self) -> None: - content = b"reference\n" - with tempfile.TemporaryDirectory() as directory: - repository = Path(directory).resolve() - generated = repository / "generated" - generated.mkdir() - outside = repository.parent / f"{repository.name}-outside.md" - symlink = generated / "reference.md" - symlink.symlink_to(outside) - - for output in (Path("../escape.md"), Path("generated/reference.md")): - with ( - self.subTest(output=output), - self.assertRaises(CommandReferenceToolError), - ): - publish_or_check_command_reference( - content, - repository_root=repository, - project_root=ALPHA, - output=output, - check=False, - ) - symlink.unlink() - (repository / "linked").symlink_to(generated, target_is_directory=True) - with self.assertRaises(CommandReferenceToolError): - publish_or_check_command_reference( - content, - repository_root=repository, - project_root=ALPHA, - output=Path("linked/reference.md"), - check=False, - ) - - with self.assertRaises(CommandReferenceToolError): - publish_or_check_command_reference( - content, - repository_root=ROOT, - project_root=ALPHA, - output=Path("tests/fixtures/alpha/generated-reference.md"), - check=False, - ) - - def test_existing_target_race_is_restored_without_overwrite(self) -> None: - with tempfile.TemporaryDirectory() as directory: - repository = Path(directory).resolve() - generated = repository / "generated" - generated.mkdir() - output = Path("generated/reference.md") - target = repository / output - target.write_bytes(b"initial\n") - original_exchange = command_reference_tool._rename_exchange - injected = False - - def exchange_after_race(directory_fd: int, first: str, second: str) -> None: - nonlocal injected - if not injected: - injected = True - target.write_bytes(b"concurrent\n") - original_exchange(directory_fd, first, second) - - with ( - mock.patch.object( - command_reference_tool, - "_rename_exchange", - side_effect=exchange_after_race, - ), - self.assertRaisesRegex( - CommandReferenceToolError, - "changed during atomic publication", - ), - ): - publish_or_check_command_reference( - b"generated\n", - repository_root=repository, - project_root=ALPHA, - output=output, - check=False, - ) - - self.assertEqual(b"concurrent\n", target.read_bytes()) - self.assertEqual( - [], - list(generated.glob(".reference.md.docforge-command-reference-*")), - ) - - def test_missing_target_race_is_never_clobbered(self) -> None: - with tempfile.TemporaryDirectory() as directory: - repository = Path(directory).resolve() - generated = repository / "generated" - generated.mkdir() - output = Path("generated/reference.md") - target = repository / output - original_link = command_reference_tool._link_no_replace - injected = False - - def link_after_race(directory_fd: int, source: str, destination: str) -> None: - nonlocal injected - if not injected: - injected = True - target.write_bytes(b"concurrent\n") - original_link(directory_fd, source, destination) - - with ( - mock.patch.object( - command_reference_tool, - "_link_no_replace", - side_effect=link_after_race, - ), - self.assertRaisesRegex(CommandReferenceToolError, "appeared"), - ): - publish_or_check_command_reference( - b"generated\n", - repository_root=repository, - project_root=ALPHA, - output=output, - check=False, - ) - - self.assertEqual(b"concurrent\n", target.read_bytes()) - self.assertEqual( - [], - list(generated.glob(".reference.md.docforge-command-reference-*")), - ) - - def test_concurrent_generators_are_serialized_around_inspection_and_write(self) -> None: - with tempfile.TemporaryDirectory() as directory: - repository = Path(directory).resolve() - (repository / "generated").mkdir() - output = Path("generated/reference.md") - target = repository / output - target.write_bytes(b"initial\n") - first_entered = threading.Event() - release_first = threading.Event() - second_started = threading.Event() - second_entered = threading.Event() - results: list[str] = [] - errors: list[BaseException] = [] - original_replace = command_reference_tool._atomic_replace - invocation_count = 0 - invocation_lock = threading.Lock() - - def blocking_replace( - directory_fd: int, - name: str, - content: bytes, - *, - expected: object, - expected_content: bytes | None, - ) -> None: - nonlocal invocation_count - with invocation_lock: - invocation_count += 1 - invocation = invocation_count - if invocation == 1: - first_entered.set() - if not release_first.wait(timeout=5): - raise AssertionError("Timed out waiting to release first generator") - else: - second_entered.set() - original_replace( - directory_fd, - name, - content, - expected=expected, # type: ignore[arg-type] - expected_content=expected_content, - ) - - def publish(content: bytes, *, started: threading.Event | None = None) -> None: - if started is not None: - started.set() - try: - results.append( - publish_or_check_command_reference( - content, - repository_root=repository, - project_root=ALPHA, - output=output, - check=False, - ) - ) - except BaseException as error: - errors.append(error) - - with mock.patch.object( - command_reference_tool, - "_atomic_replace", - side_effect=blocking_replace, - ): - first = threading.Thread(target=publish, args=(b"first\n",)) - second = threading.Thread( - target=publish, - args=(b"second\n",), - kwargs={"started": second_started}, - ) - first.start() - self.assertTrue(first_entered.wait(timeout=5)) - second.start() - self.assertTrue(second_started.wait(timeout=5)) - self.assertFalse(second_entered.wait(timeout=0.1)) - release_first.set() - first.join(timeout=5) - second.join(timeout=5) - - self.assertFalse(first.is_alive()) - self.assertFalse(second.is_alive()) - self.assertEqual([], errors) - self.assertEqual(["written", "written"], results) - self.assertEqual(b"second\n", target.read_bytes()) - - def test_command_runs_with_explicit_repository_and_project_roots(self) -> None: - with tempfile.TemporaryDirectory() as directory: - repository = Path(directory).resolve() - (repository / "generated").mkdir() - output = Path("generated/reference.md") - stdout = io.StringIO() - - with contextlib.redirect_stdout(stdout): - status = main( - ( - "--repository-root", - str(repository), - "--project-root", - str(ALPHA), - "--output", - str(output), - ) - ) - - self.assertEqual(0, status) - self.assertTrue((repository / output).is_file()) - self.assertIn('"cli_rows":28', stdout.getvalue()) - self.assertIn('"mcp_rows":36', stdout.getvalue()) - - target = repository / output - target.write_text("drift\n", encoding="utf-8") - stderr = io.StringIO() - with contextlib.redirect_stderr(stderr): - check_status = main( - ( - "--repository-root", - str(repository), - "--project-root", - str(ALPHA), - "--output", - str(output), - "--check", - ) - ) - - self.assertEqual(1, check_status) - self.assertEqual("drift\n", target.read_text(encoding="utf-8")) - self.assertIn("missing or stale", stderr.getvalue()) - - -if __name__ == "__main__": - unittest.main() diff --git a/tests/test_cpp_reference_adapter.py b/tests/test_cpp_reference_adapter.py deleted file mode 100644 index 70c3bdf..0000000 --- a/tests/test_cpp_reference_adapter.py +++ /dev/null @@ -1,486 +0,0 @@ -from __future__ import annotations - -import json -import shutil -import tempfile -import unittest -from pathlib import Path -from typing import cast -from unittest.mock import patch - -from docforge.adapter_sdk import ( - AdapterProject, - AdapterSource, - AdapterSourceProjection, - verify_adapter_conformance, -) -from docforge.adapters.cpp import ( - CPP_ADAPTER_ID, - CPP_ADAPTER_VERSION, - CPP_EXTRACTOR_VERSION, - CppReferenceAdapter, -) -from docforge.errors import DocForgeError -from docforge.index import ProjectIndex - -ROOT = Path(__file__).resolve().parents[1] -FIXTURE = ROOT / "tests" / "fixtures" / "reference-cpp" - - -class RecordingCppAdapter(CppReferenceAdapter): - def __init__(self, root: Path) -> None: - super().__init__( - root, - source_roots=("include", "src"), - compilation_database="compile_commands.json", - project_id="reference-cpp", - title="Reference C++ fixture", - ) - self.extracted_paths: list[str] = [] - - def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: - self.extracted_paths.append(source.source_path) - return super().extract_source(source) - - -class CppReferenceAdapterTests(unittest.TestCase): - def copy_fixture(self, parent: Path) -> Path: - root = parent / "reference-cpp" - shutil.copytree(FIXTURE, root) - return root.resolve() - - def adapter(self, root: Path) -> RecordingCppAdapter: - return RecordingCppAdapter(root) - - @staticmethod - def build_metrics(result: dict[str, object]) -> dict[str, object]: - return cast(dict[str, object], result["build"]) - - @staticmethod - def read_database(root: Path) -> list[dict[str, object]]: - return cast( - list[dict[str, object]], - json.loads((root / "compile_commands.json").read_text(encoding="utf-8")), - ) - - @staticmethod - def write_database(root: Path, entries: list[dict[str, object]]) -> None: - (root / "compile_commands.json").write_text( - json.dumps(entries, indent=2, sort_keys=True) + "\n", - encoding="utf-8", - ) - - def test_complete_assembly_is_deterministic_conformant_and_syntax_scoped(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - adapter = self.adapter(root) - - first = adapter.load_assembly() - second = adapter.load_complete_assembly() - projection = adapter.load_projection() - - self.assertEqual(first, second) - self.assertEqual(first.projection, projection) - self.assertEqual(CPP_ADAPTER_ID, projection.adapter_id) - self.assertEqual(CPP_ADAPTER_VERSION, projection.adapter_version) - self.assertEqual(17, len(projection.nodes)) - self.assertEqual(16, len(projection.edges)) - self.assertEqual(5, len(first.logic)) - self.assertEqual( - list(projection.nodes), - sorted(projection.nodes, key=lambda item: item.node.node_id), - ) - self.assertEqual( - {"contains", "depends_on"}, - {item.edge.relation for item in projection.edges}, - ) - self.assertTrue( - any( - item.node.title == "demo.Worker" and ("kind", "class") in item.metadata - for item in projection.nodes - ) - ) - self.assertTrue( - any( - item.node.title == "demo.Config" and ("kind", "struct") in item.metadata - for item in projection.nodes - ) - ) - self.assertTrue( - any( - item.node.title == "demo.Worker.run" and ("kind", "function") in item.metadata - for item in projection.nodes - ) - ) - self.assertEqual( - {"include/config.hpp", "include/detail.hpp", "src/main.cpp", "src/worker.cpp"}, - {item.node.source_path for item in projection.nodes}, - ) - self.assertFalse( - {"calls", "inherits", "type_of"} & {item.edge.relation for item in projection.edges} - ) - - report = adapter.support_report() - self.assertEqual(1, report["schema_version"]) - self.assertEqual(CPP_EXTRACTOR_VERSION, report["extractor_version"]) - self.assertEqual(["include", "src"], report["source_roots"]) - self.assertTrue(report["compilation_database_is_authoritative"]) - self.assertFalse(report["executes_compiler"]) - self.assertFalse(report["executes_project_code"]) - self.assertEqual( - [ - "call_resolution", - "type_resolution", - "inheritance_resolution", - "overload_ownership", - "macro_semantics", - "compiler_command_execution", - "nonlocal_include_resolution", - ], - [item.code for item in adapter.unsupported_facts()], - ) - self.assertFalse((root / ".cache").exists()) - - conformance = verify_adapter_conformance( - adapter, - cache_root=root / ".cache" / "conformance", - ) - self.assertEqual("reference-cpp", conformance.project_id) - self.assertEqual(17, conformance.node_count) - self.assertEqual(16, conformance.edge_count) - self.assertEqual(5, conformance.logic_projection_count) - self.assertTrue(conformance.incremental) - - def test_incremental_cold_warm_source_header_and_flag_invalidation(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - adapter = self.adapter(root) - project = AdapterProject(adapter, cache_root=root / ".cache" / "incremental") - index = ProjectIndex(project) - - cold = index.build() - self.assertEqual(4, self.build_metrics(cold)["reparsed_sources"]) - self.assertEqual( - [ - "include/config.hpp", - "include/detail.hpp", - "src/main.cpp", - "src/worker.cpp", - ], - sorted(adapter.extracted_paths), - ) - - adapter.extracted_paths.clear() - warm = index.build() - self.assertEqual(4, self.build_metrics(warm)["cache_hits"]) - self.assertEqual(0, self.build_metrics(warm)["reparsed_sources"]) - self.assertEqual([], adapter.extracted_paths) - - main = root / "src" / "main.cpp" - main.write_text( - main.read_text(encoding="utf-8").replace( - "return demo::execute(3);", - "return demo::execute(4);", - ), - encoding="utf-8", - ) - adapter.extracted_paths.clear() - source_changed = index.build() - self.assertEqual(1, self.build_metrics(source_changed)["invalidated_sources"]) - self.assertEqual(["src/main.cpp"], adapter.extracted_paths) - - detail = root / "include" / "detail.hpp" - detail.write_text( - detail.read_text(encoding="utf-8").replace( - "return value;", - "return value + 0;", - ), - encoding="utf-8", - ) - adapter.extracted_paths.clear() - header_changed = index.build() - self.assertEqual(4, self.build_metrics(header_changed)["invalidated_sources"]) - self.assertEqual( - [ - "include/config.hpp", - "include/detail.hpp", - "src/main.cpp", - "src/worker.cpp", - ], - sorted(adapter.extracted_paths), - ) - - entries = self.read_database(root) - arguments = cast(list[str], entries[0]["arguments"]) - arguments.append("-DREFERENCE_FLAG=1") - self.write_database(root, entries) - adapter.extracted_paths.clear() - flag_changed = index.build() - self.assertEqual(1, self.build_metrics(flag_changed)["invalidated_sources"]) - self.assertEqual(["src/main.cpp"], adapter.extracted_paths) - - equivalent = project.verify_incremental_equivalence() - self.assertEqual("ok", equivalent["status"]) - self.assertEqual(17, equivalent["node_count"]) - self.assertEqual(16, equivalent["edge_count"]) - self.assertEqual(5, equivalent["logic_projection_count"]) - - def test_add_delete_and_corrupt_cache_recover_without_stale_facts(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - adapter = self.adapter(root) - cache_root = root / ".cache" / "incremental" - project = AdapterProject(adapter, cache_root=cache_root) - index = ProjectIndex(project) - index.build() - - extra = root / "src" / "extra.cpp" - extra.write_text( - '#include "../include/config.hpp"\n\n' - "int extra(int value) {\n" - " return demo::normalize(value);\n" - "}\n", - encoding="utf-8", - ) - entries = self.read_database(root) - entries.append( - { - "arguments": ["c++", "-std=c++20", "-c", "src/extra.cpp"], - "directory": ".", - "file": "src/extra.cpp", - } - ) - self.write_database(root, entries) - adapter.extracted_paths.clear() - added = index.build() - self.assertEqual(1, self.build_metrics(added)["reparsed_sources"]) - self.assertEqual(["src/extra.cpp"], adapter.extracted_paths) - self.assertIn( - "src/extra.cpp", - {node.source_path for node in project.load().nodes}, - ) - - worker = root / "src" / "worker.cpp" - entries = [ - entry for entry in self.read_database(root) if entry["file"] != "src/worker.cpp" - ] - self.write_database(root, entries) - worker.unlink() - adapter.extracted_paths.clear() - deleted = index.build() - self.assertEqual(1, self.build_metrics(deleted)["deleted_sources"]) - self.assertNotIn( - "src/worker.cpp", - {node.source_path for node in project.load().nodes}, - ) - - (cache_root / "extractions.json").write_text("{broken", encoding="utf-8") - adapter.extracted_paths.clear() - recovered = index.build() - self.assertEqual(4, self.build_metrics(recovered)["reparsed_sources"]) - self.assertEqual( - [ - "include/config.hpp", - "include/detail.hpp", - "src/extra.cpp", - "src/main.cpp", - ], - sorted(adapter.extracted_paths), - ) - self.assertEqual("ok", project.verify_incremental_equivalence()["status"]) - - def test_database_shape_bounds_duplicates_and_paths_fail_closed(self) -> None: - invalid_documents: list[tuple[str, list[dict[str, object]], str]] = [] - baseline = cast( - list[dict[str, object]], - json.loads((FIXTURE / "compile_commands.json").read_text(encoding="utf-8")), - ) - - both = json.loads(json.dumps(baseline)) - both[0]["command"] = "c++ -c src/main.cpp" - invalid_documents.append(("both command forms", both, "invalid_compilation_database")) - - neither = json.loads(json.dumps(baseline)) - del neither[1]["command"] - invalid_documents.append(("neither command form", neither, "invalid_compilation_database")) - - bad_argument = json.loads(json.dumps(baseline)) - bad_argument[0]["arguments"] = ["c++", 7] - invalid_documents.append( - ("non-string argument", bad_argument, "invalid_compilation_database") - ) - - control_command = json.loads(json.dumps(baseline)) - control_command[1]["command"] = "c++ -c src/worker.cpp\nmalicious" - invalid_documents.append( - ("control command", control_command, "invalid_compilation_database") - ) - - unknown_key = json.loads(json.dumps(baseline)) - unknown_key[0]["environment"] = {"PATH": "/tmp"} - invalid_documents.append(("unknown key", unknown_key, "invalid_compilation_database")) - - duplicate = json.loads(json.dumps(baseline)) - duplicate.append(json.loads(json.dumps(duplicate[0]))) - invalid_documents.append(("duplicate TU", duplicate, "duplicate_translation_unit")) - - escaped_directory = json.loads(json.dumps(baseline)) - escaped_directory[0]["directory"] = ".." - invalid_documents.append(("escaped directory", escaped_directory, "path_escape")) - - escaped_file = json.loads(json.dumps(baseline)) - escaped_file[0]["file"] = "../outside.cpp" - invalid_documents.append(("escaped file", escaped_file, "path_escape")) - - for name, document, code in invalid_documents: - with self.subTest(name=name), tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - self.write_database(root, document) - adapter = self.adapter(root) - with self.assertRaises(DocForgeError) as captured: - adapter.load_manifest() - self.assertEqual(code, captured.exception.code) - - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - outside_database = parent / "compile_commands.json" - outside_database.write_text("[]\n", encoding="utf-8") - database = root / "compile_commands.json" - database.unlink() - database.symlink_to(outside_database) - with self.assertRaises(DocForgeError) as database_link: - self.adapter(root) - self.assertEqual("path_escape", database_link.exception.code) - - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - outside_source = parent / "worker.cpp" - outside_source.write_text("int outside() { return 1; }\n", encoding="utf-8") - worker = root / "src" / "worker.cpp" - worker.unlink() - worker.symlink_to(outside_source) - adapter = self.adapter(root) - with self.assertRaises(DocForgeError) as source_link: - adapter.load_manifest() - self.assertEqual("path_escape", source_link.exception.code) - - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - bounded_adapters = ( - ( - CppReferenceAdapter( - root, - source_roots=("include", "src"), - compilation_database="compile_commands.json", - max_database_bytes=1, - ), - "source_too_large", - ), - ( - CppReferenceAdapter( - root, - source_roots=("include", "src"), - compilation_database="compile_commands.json", - max_source_bytes=1, - ), - "source_too_large", - ), - ( - CppReferenceAdapter( - root, - source_roots=("include", "src"), - compilation_database="compile_commands.json", - max_command_chars=32, - ), - "invalid_compilation_database", - ), - ( - CppReferenceAdapter( - root, - source_roots=("include", "src"), - compilation_database="compile_commands.json", - max_arguments=2, - ), - "invalid_compilation_database", - ), - ( - CppReferenceAdapter( - root, - source_roots=("include", "src"), - compilation_database="compile_commands.json", - max_translation_units=1, - ), - "adapter_too_large", - ), - ( - CppReferenceAdapter( - root, - source_roots=("include", "src"), - compilation_database="compile_commands.json", - max_headers=1, - ), - "adapter_too_large", - ), - ) - for adapter, code in bounded_adapters: - with self.subTest(limit=code), self.assertRaises(DocForgeError) as bounded: - adapter.load_manifest() - self.assertEqual(code, bounded.exception.code) - - def test_source_roots_no_ast_and_optional_extra_fail_closed(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - with self.assertRaises(DocForgeError) as translation_unit_outside: - CppReferenceAdapter( - root, - source_roots=("include",), - compilation_database="compile_commands.json", - ).load_manifest() - self.assertEqual("path_escape", translation_unit_outside.exception.code) - - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - other = root / "other" - other.mkdir() - (other / "outside.hpp").write_text( - "#pragma once\nstruct Outside {};\n", - encoding="utf-8", - ) - main = root / "src" / "main.cpp" - main.write_text( - '#include "../other/outside.hpp"\n' + main.read_text(encoding="utf-8"), - encoding="utf-8", - ) - with self.assertRaises(DocForgeError) as header_outside: - self.adapter(root).load_manifest() - self.assertEqual("path_escape", header_outside.exception.code) - - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - adapter = self.adapter(root) - project = AdapterProject(adapter, cache_root=root / ".cache" / "no-ast") - with self.assertRaises(DocForgeError) as no_ast: - ProjectIndex(project, allow_logic=False).build() - self.assertEqual("adapter_policy_forbids_logic", no_ast.exception.code) - self.assertFalse(project.descriptor.index_path.exists()) - - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - adapter = self.adapter(root) - missing = ModuleNotFoundError( - "No module named 'tree_sitter'", - name="tree_sitter", - ) - with ( - patch( - "docforge.adapters.cpp.importlib.import_module", - side_effect=missing, - ), - self.assertRaises(DocForgeError) as optional, - ): - adapter.load_manifest() - self.assertEqual("optional_dependency_missing", optional.exception.code) - self.assertEqual("cpp", optional.exception.details["extra"]) - self.assertEqual("docforge[cpp]", optional.exception.details["install"]) diff --git a/tests/test_docs_check.py b/tests/test_docs_check.py deleted file mode 100644 index d4cb2e4..0000000 --- a/tests/test_docs_check.py +++ /dev/null @@ -1,198 +0,0 @@ -from __future__ import annotations - -import shutil -import tempfile -import unittest -from pathlib import Path - -from tools.check_documentation import ( - COMMAND_REFERENCE_NOTICE, - MAX_DIAGNOSTICS, - DocumentationPolicy, - check_documentation, -) - -ROOT = Path(__file__).resolve().parents[1] -SCHEMA = ROOT / "schemas/reference-adapter.schema.json" - - -class DocumentationCheckTests(unittest.TestCase): - def repository(self) -> Path: - root = Path(self.enterContext(tempfile.TemporaryDirectory())) - (root / "docs").mkdir() - (root / "schemas").mkdir() - shutil.copyfile(SCHEMA, root / "schemas/reference-adapter.schema.json") - (root / "pending.txt").write_text("", encoding="utf-8") - return root - - @staticmethod - def policy(*required: str) -> DocumentationPolicy: - return DocumentationPolicy( - required_pages=tuple(Path(item) for item in required), - pending_inventory=Path("pending.txt"), - ) - - @staticmethod - def command_reference() -> str: - return f"# DocForge command reference\n\n{COMMAND_REFERENCE_NOTICE}\n\n## CLI commands\n" - - def test_valid_graph_h1_anchors_and_reference_adapter_toml_pass(self) -> None: - root = self.repository() - (root / "README.md").write_text( - "# Documentation\n\n" - "[Guide](docs/GUIDE.md#adapter-setup) and " - "[commands](docs/COMMAND_REFERENCE.md). [Top](#documentation)\n", - encoding="utf-8", - ) - (root / "docs/GUIDE.md").write_text( - "# Guide\n\n" - "## Adapter *setup*\n\n" - "Reference adapter configuration `.docforge/reference-adapter.toml`:\n\n" - "```toml\n" - "schema_version = 1\n" - 'project_id = "example-project"\n' - 'title = "Example"\n' - 'language = "python"\n' - 'source_roots = ["src"]\n' - "```\n\n" - "[Back](../README.md#documentation)\n", - encoding="utf-8", - ) - (root / "docs/COMMAND_REFERENCE.md").write_text( - self.command_reference(), - encoding="utf-8", - ) - - report = check_documentation( - root, - policy=self.policy( - "README.md", - "docs/GUIDE.md", - "docs/COMMAND_REFERENCE.md", - ), - ) - - self.assertTrue(report.ok, [item.render() for item in report.diagnostics]) - self.assertEqual(3, report.markdown_pages) - self.assertEqual(1, report.reference_adapter_examples) - - def test_links_anchors_h1_and_generated_notice_fail_closed(self) -> None: - root = self.repository() - (root / "README.md").write_text( - "# Documentation\n\n" - "[bad anchor](docs/GUIDE.md#missing) " - "[missing file](docs/ABSENT.md) " - "[commands](docs/COMMAND_REFERENCE.md).\n", - encoding="utf-8", - ) - (root / "docs/GUIDE.md").write_text( - "# Guide\n\n# Duplicate\n", - encoding="utf-8", - ) - (root / "docs/COMMAND_REFERENCE.md").write_text( - "# DocForge command reference\n\n## CLI commands\n", - encoding="utf-8", - ) - - report = check_documentation( - root, - policy=self.policy( - "README.md", - "docs/GUIDE.md", - "docs/COMMAND_REFERENCE.md", - ), - ) - - self.assertFalse(report.ok) - self.assertEqual( - {"DOC016", "DOC017", "DOC020", "DOC028"}, - {item.code for item in report.diagnostics}, - ) - - def test_only_prior_milestone_records_are_reachability_exempt(self) -> None: - root = self.repository() - (root / "README.md").write_text("# Documentation\n", encoding="utf-8") - (root / "docs/MILESTONE_2_BASELINE.md").write_text( - "# Historical evidence\n", - encoding="utf-8", - ) - (root / "docs/ORPHAN.md").write_text("# Orphan\n", encoding="utf-8") - - report = check_documentation( - root, - policy=self.policy("README.md"), - ) - - orphans = [item.path for item in report.diagnostics if item.code == "DOC023"] - self.assertEqual(["docs/ORPHAN.md"], orphans) - - def test_pending_page_becomes_a_hard_failure_as_soon_as_it_exists(self) -> None: - root = self.repository() - (root / "README.md").write_text("# Documentation\n", encoding="utf-8") - (root / "pending.txt").write_text("docs/FUTURE.md\n", encoding="utf-8") - policy = self.policy("README.md", "docs/FUTURE.md") - - pending = check_documentation(root, policy=policy) - - self.assertTrue(pending.ok) - self.assertEqual(1, pending.pending_pages) - - (root / "docs/FUTURE.md").write_text("# Future\n", encoding="utf-8") - stale = check_documentation(root, policy=policy) - - self.assertIn("DOC014", {item.code for item in stale.diagnostics}) - self.assertIn("DOC023", {item.code for item in stale.diagnostics}) - - def test_unknown_pending_page_and_invalid_toml_are_reported(self) -> None: - root = self.repository() - (root / "README.md").write_text( - "# Documentation\n\n[Adapter](docs/ADAPTER.md)\n", - encoding="utf-8", - ) - (root / "docs/ADAPTER.md").write_text( - "# Adapter\n\n" - "```toml reference-adapter\n" - "schema_version = 1\n" - 'project_id = "cpp-example"\n' - 'title = "C++"\n' - 'language = "cpp"\n' - 'source_roots = ["src"]\n' - "```\n", - encoding="utf-8", - ) - (root / "pending.txt").write_text("docs/UNKNOWN.md\n", encoding="utf-8") - - report = check_documentation( - root, - policy=self.policy("README.md", "docs/ADAPTER.md"), - ) - - self.assertIn("DOC012", {item.code for item in report.diagnostics}) - schema_failures = [item for item in report.diagnostics if item.code == "DOC026"] - self.assertEqual(1, len(schema_failures)) - self.assertIn("compilation_database", schema_failures[0].message) - - def test_diagnostics_are_deterministically_bounded(self) -> None: - root = self.repository() - (root / "README.md").write_text("# Documentation\n", encoding="utf-8") - for index in range(MAX_DIAGNOSTICS + 7): - (root / "docs" / f"ORPHAN_{index:03d}.md").write_text( - f"# Orphan {index}\n", - encoding="utf-8", - ) - - report = check_documentation( - root, - policy=self.policy("README.md"), - ) - - self.assertEqual(MAX_DIAGNOSTICS, len(report.diagnostics)) - self.assertEqual(7, report.omitted_diagnostics) - self.assertEqual( - sorted(report.diagnostics), - list(report.diagnostics), - ) - - -if __name__ == "__main__": - unittest.main() diff --git a/tests/test_documentation_check.py b/tests/test_documentation_check.py deleted file mode 100644 index de2d851..0000000 --- a/tests/test_documentation_check.py +++ /dev/null @@ -1,176 +0,0 @@ -from __future__ import annotations - -import json -import tempfile -import unittest -from pathlib import Path - -from tools.check_documentation import ( - DocumentationPolicy, - check_documentation, -) - - -class DocumentationCheckTests(unittest.TestCase): - def repository(self, parent: Path) -> Path: - root = parent / "repository" - (root / "docs").mkdir(parents=True) - (root / "schemas").mkdir() - (root / "schemas" / "reference-adapter.schema.json").write_text( - json.dumps( - { - "$schema": "https://json-schema.org/draft/2020-12/schema", - "type": "object", - "additionalProperties": False, - "required": [ - "schema_version", - "project_id", - "title", - "language", - "source_roots", - ], - "properties": { - "schema_version": {"const": 1}, - "project_id": {"type": "string"}, - "title": {"type": "string"}, - "language": {"enum": ["python", "javascript", "typescript", "cpp"]}, - "source_roots": { - "type": "array", - "minItems": 1, - "items": {"type": "string"}, - }, - "compilation_database": {"type": "string"}, - }, - } - ), - encoding="utf-8", - ) - return root - - def test_linked_pages_h1_anchors_and_reference_example_pass(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.repository(Path(directory)) - (root / "README.md").write_text( - "# Project\n\n[Guide](docs/GUIDE.md#reference-adapter-configuration)\n", - encoding="utf-8", - ) - (root / "docs" / "GUIDE.md").write_text( - "\n".join( - ( - "# Guide", - "", - "## Reference adapter configuration", - "", - "`.docforge/reference-adapter.toml`:", - "", - "```toml", - "schema_version = 1", - 'project_id = "fixture"', - 'title = "Fixture"', - 'language = "python"', - 'source_roots = ["src"]', - "```", - "", - ) - ), - encoding="utf-8", - ) - - report = check_documentation( - root, - policy=DocumentationPolicy( - required_pages=(Path("README.md"), Path("docs/GUIDE.md")), - pending_inventory=None, - ), - ) - - self.assertTrue(report.ok, [item.render() for item in report.diagnostics]) - self.assertEqual(2, report.markdown_pages) - self.assertEqual(1, report.reference_adapter_examples) - - def test_missing_anchor_orphan_and_duplicate_h1_fail_deterministically(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.repository(Path(directory)) - (root / "README.md").write_text( - "# Project\n\n[Broken](docs/GUIDE.md#missing)\n", - encoding="utf-8", - ) - (root / "docs" / "GUIDE.md").write_text( - "# Guide\n\n# Duplicate\n", - encoding="utf-8", - ) - (root / "docs" / "ORPHAN.md").write_text("# Orphan\n", encoding="utf-8") - - report = check_documentation( - root, - policy=DocumentationPolicy( - required_pages=(Path("README.md"), Path("docs/GUIDE.md")), - pending_inventory=None, - ), - ) - - self.assertFalse(report.ok) - codes = [item.code for item in report.diagnostics] - self.assertEqual(["DOC016", "DOC017", "DOC023"], sorted(codes)) - - def test_pending_inventory_allows_absence_but_rejects_existing_page(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.repository(Path(directory)) - (root / "README.md").write_text("# Project\n", encoding="utf-8") - pending_path = root / "pending.txt" - pending_path.write_text("docs/FUTURE.md\n", encoding="utf-8") - policy = DocumentationPolicy( - required_pages=(Path("README.md"), Path("docs/FUTURE.md")), - pending_inventory=Path("pending.txt"), - ) - - missing = check_documentation(root, policy=policy) - self.assertTrue(missing.ok) - self.assertEqual(1, missing.pending_pages) - - (root / "docs" / "FUTURE.md").write_text("# Future\n", encoding="utf-8") - stale = check_documentation(root, policy=policy) - self.assertEqual(["DOC014", "DOC023"], [item.code for item in stale.diagnostics]) - - def test_invalid_reference_example_reports_schema_path(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.repository(Path(directory)) - (root / "README.md").write_text( - "# Project\n\n[Guide](docs/GUIDE.md)\n", - encoding="utf-8", - ) - (root / "docs" / "GUIDE.md").write_text( - "\n".join( - ( - "# Guide", - "", - "Reference adapter configuration:", - "", - "```toml", - 'language = "python"', - 'source_roots = ["src"]', - "```", - "", - ) - ), - encoding="utf-8", - ) - - report = check_documentation( - root, - policy=DocumentationPolicy( - required_pages=(Path("README.md"), Path("docs/GUIDE.md")), - pending_inventory=None, - ), - ) - - schema_errors = [item for item in report.diagnostics if item.code == "DOC026"] - self.assertEqual(3, len(schema_errors)) - messages = " ".join(item.message for item in schema_errors) - self.assertIn("schema_version", messages) - self.assertIn("project_id", messages) - self.assertIn("title", messages) - - -if __name__ == "__main__": - unittest.main() diff --git a/tests/test_incremental_cache.py b/tests/test_incremental_cache.py deleted file mode 100644 index cfd2a9d..0000000 --- a/tests/test_incremental_cache.py +++ /dev/null @@ -1,102 +0,0 @@ -from __future__ import annotations - -import hashlib -import os -import tempfile -import unittest -from pathlib import Path - -from docforge.errors import DocForgeError -from docforge.incremental import ( - CachedSource, - ExtractionCache, - load_extraction_cache, - write_extraction_cache, -) - - -class ExtractionCacheBoundsTests(unittest.TestCase): - @staticmethod - def cache(*, payload: str = "ok", count: int = 1) -> ExtractionCache: - sources = tuple( - CachedSource( - source_id=f"source-{index}", - source_path=f"src/{index}.txt", - fingerprint=hashlib.sha256(str(index).encode()).hexdigest(), - extractor_version="fixture@1", - dependencies=(), - payload={"content": payload}, - ) - for index in range(count) - ) - return ExtractionCache("fixture", "fixture-adapter", "1", sources) - - def test_bounded_round_trip_and_oversized_read_is_a_cache_miss(self) -> None: - with tempfile.TemporaryDirectory() as directory: - path = Path(directory) / "extractions.json" - write_extraction_cache(path, self.cache(), max_bytes=1_000, max_sources=2) - loaded = load_extraction_cache( - path, - project_id="fixture", - adapter_id="fixture-adapter", - adapter_version="1", - max_bytes=1_000, - max_sources=2, - ) - self.assertEqual(self.cache(), loaded) - - with path.open("ab") as handle: - handle.write(b" " * 1_000) - self.assertIsNone( - load_extraction_cache( - path, - project_id="fixture", - adapter_id="fixture-adapter", - adapter_version="1", - max_bytes=1_000, - max_sources=2, - ) - ) - - def test_write_rejects_byte_and_source_limits_without_replacing_target(self) -> None: - with tempfile.TemporaryDirectory() as directory: - path = Path(directory) / "extractions.json" - path.write_bytes(b"preserved\n") - for cache, maximum_bytes, maximum_sources in ( - (self.cache(payload="x" * 500), 100, 2), - (self.cache(count=2), 1_000, 1), - ): - with self.assertRaises(DocForgeError) as captured: - write_extraction_cache( - path, - cache, - max_bytes=maximum_bytes, - max_sources=maximum_sources, - ) - self.assertEqual("cache_limit", captured.exception.code) - self.assertEqual(b"preserved\n", path.read_bytes()) - - def test_symlink_and_non_regular_targets_are_cache_misses(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = Path(directory) - real = root / "real.json" - write_extraction_cache(real, self.cache(), max_bytes=1_000, max_sources=2) - linked = root / "linked.json" - linked.symlink_to(real) - fifo = root / "fifo" - os.mkfifo(fifo) - for path in (linked, fifo): - self.assertIsNone( - load_extraction_cache( - path, - project_id="fixture", - adapter_id="fixture-adapter", - adapter_version="1", - max_bytes=1_000, - max_sources=2, - ) - ) - - -if __name__ == "__main__": - unittest.main() diff --git a/tests/test_javascript_reference_adapter.py b/tests/test_javascript_reference_adapter.py deleted file mode 100644 index 14c5b96..0000000 --- a/tests/test_javascript_reference_adapter.py +++ /dev/null @@ -1,267 +0,0 @@ -from __future__ import annotations - -import shutil -import tempfile -import unittest -from pathlib import Path -from typing import cast -from unittest import mock - -import docforge.adapters.javascript as javascript_adapter -from docforge.adapter_sdk import ( - AdapterProject, - AdapterSource, - AdapterSourceProjection, - verify_adapter_conformance, -) -from docforge.adapters.javascript import ( - JAVASCRIPT_ADAPTER_ID, - JAVASCRIPT_ADAPTER_VERSION, - JAVASCRIPT_EXTRACTOR_VERSION, - JavaScriptReferenceAdapter, -) -from docforge.errors import DocForgeError -from docforge.index import ProjectIndex - -ROOT = Path(__file__).resolve().parents[1] -FIXTURES = ROOT / "tests" / "fixtures" - - -class RecordingScriptAdapter(JavaScriptReferenceAdapter): - def __init__(self, root: Path, project_id: str) -> None: - super().__init__( - root, - source_roots=("src",), - project_id=project_id, - title=f"{project_id} fixture", - ) - self.extracted_paths: list[str] = [] - - def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: - self.extracted_paths.append(source.source_path) - return super().extract_source(source) - - -class JavaScriptReferenceAdapterTests(unittest.TestCase): - CASES = ( - ("reference-javascript", "javascript", 14, 14, 5, "src/app/shared.js"), - ("reference-typescript", "typescript", 13, 14, 4, "src/app/types.mts"), - ) - - def copy_fixture(self, parent: Path, fixture: str) -> Path: - root = parent / fixture - shutil.copytree(FIXTURES / fixture, root) - return root.resolve() - - @staticmethod - def build_metrics(result: dict[str, object]) -> dict[str, object]: - return cast(dict[str, object], result["build"]) - - def test_both_grammars_produce_deterministic_complete_logic_assemblies(self) -> None: - for fixture, language, nodes, edges, logic, _shared in self.CASES: - with self.subTest(language=language), tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory), fixture) - adapter = RecordingScriptAdapter(root, fixture) - - first = adapter.load_assembly() - second = adapter.load_complete_assembly() - projection = adapter.load_projection() - - self.assertEqual(first, second) - self.assertEqual(first.projection, projection) - self.assertEqual(JAVASCRIPT_ADAPTER_ID, projection.adapter_id) - self.assertEqual(JAVASCRIPT_ADAPTER_VERSION, projection.adapter_version) - self.assertEqual(nodes, len(projection.nodes)) - self.assertEqual(edges, len(projection.edges)) - self.assertEqual(logic, len(first.logic)) - self.assertEqual( - {language}, - {dict(item.metadata)["language"] for item in projection.nodes}, - ) - self.assertEqual( - {"contains", "depends_on"}, - {item.edge.relation for item in projection.edges}, - ) - self.assertTrue( - any( - item.node.title == "Service.run" - and dict(item.metadata)["kind"] == "function" - for item in projection.nodes - ) - ) - self.assertFalse((root / ".cache").exists()) - - report = adapter.support_report() - self.assertEqual(1, report["schema_version"]) - self.assertEqual(JAVASCRIPT_EXTRACTOR_VERSION, report["extractor_version"]) - self.assertFalse(report["imports_project_code"]) - self.assertFalse(report["executes_project_code"]) - self.assertEqual( - [ - "call_resolution", - "dynamic_module_resolution", - "inheritance_resolution", - "module_configuration_resolution", - "runtime_generated_facts", - "type_and_symbol_resolution", - ], - [item.code for item in adapter.unsupported_facts()], - ) - - conformance = verify_adapter_conformance( - adapter, - cache_root=root / ".cache" / "conformance", - ) - self.assertEqual(nodes, conformance.node_count) - self.assertEqual(edges, conformance.edge_count) - self.assertEqual(logic, conformance.logic_projection_count) - self.assertTrue(conformance.incremental) - - def test_cold_warm_reverse_dependency_and_exact_equivalence_for_both_grammars( - self, - ) -> None: - for fixture, language, nodes, edges, logic, shared_path in self.CASES: - with self.subTest(language=language), tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory), fixture) - adapter = RecordingScriptAdapter(root, fixture) - project = AdapterProject(adapter, cache_root=root / ".cache" / "incremental") - index = ProjectIndex(project) - - cold = index.build() - self.assertEqual(4, self.build_metrics(cold)["reparsed_sources"]) - self.assertEqual(4, len(adapter.extracted_paths)) - - adapter.extracted_paths.clear() - with mock.patch.object( - adapter, - "_parse", - wraps=adapter._parse, # pyright: ignore[reportPrivateUsage] - ) as parsed: - warm = index.build() - self.assertEqual(4, self.build_metrics(warm)["cache_hits"]) - self.assertEqual(0, self.build_metrics(warm)["reparsed_sources"]) - self.assertEqual([], adapter.extracted_paths) - parsed.assert_not_called() - - shared = root / shared_path - shared.write_text( - shared.read_text(encoding="utf-8") + "\n// changed dependency evidence\n", - encoding="utf-8", - ) - adapter.extracted_paths.clear() - with mock.patch.object( - adapter, - "_parse", - wraps=adapter._parse, # pyright: ignore[reportPrivateUsage] - ) as parsed: - changed = index.build() - self.assertEqual(4, self.build_metrics(changed)["invalidated_sources"]) - self.assertEqual(4, len(adapter.extracted_paths)) - self.assertEqual(4, parsed.call_count) - - equivalent = project.verify_incremental_equivalence() - self.assertEqual("ok", equivalent["status"]) - self.assertEqual(nodes, equivalent["node_count"]) - self.assertEqual(edges, equivalent["edge_count"]) - self.assertEqual(logic, equivalent["logic_projection_count"]) - - def test_add_delete_and_corrupt_cache_recovery_for_both_grammars(self) -> None: - additions = { - "javascript": ( - "src/app/extra.js", - "export function extra() { return 'extra'; }\n", - ), - "typescript": ( - "src/app/extra.ts", - "export function extra(): string { return 'extra'; }\n", - ), - } - for fixture, language, _nodes, _edges, _logic, _shared_path in self.CASES: - with self.subTest(language=language), tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory), fixture) - adapter = RecordingScriptAdapter(root, fixture) - cache_root = root / ".cache" / "incremental" - project = AdapterProject(adapter, cache_root=cache_root) - index = ProjectIndex(project) - index.build() - - relative, content = additions[language] - added_path = root / relative - added_path.write_text(content, encoding="utf-8") - adapter.extracted_paths.clear() - added = index.build() - self.assertEqual(1, self.build_metrics(added)["reparsed_sources"]) - self.assertEqual([relative], adapter.extracted_paths) - self.assertIn(relative, {node.source_path for node in project.load().nodes}) - - added_path.unlink() - adapter.extracted_paths.clear() - deleted = index.build() - self.assertEqual(1, self.build_metrics(deleted)["deleted_sources"]) - self.assertNotIn(relative, {node.source_path for node in project.load().nodes}) - - (cache_root / "extractions.json").write_text("{broken", encoding="utf-8") - adapter.extracted_paths.clear() - recovered = index.build() - self.assertEqual(4, self.build_metrics(recovered)["reparsed_sources"]) - self.assertEqual(4, len(adapter.extracted_paths)) - self.assertEqual("ok", project.verify_incremental_equivalence()["status"]) - - def test_confinement_and_no_ast_reject_logic_for_both_grammars(self) -> None: - for fixture, language, _nodes, _edges, _logic, _shared_path in self.CASES: - with self.subTest(language=language), tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent, fixture) - - for source_root in (root / "src", Path("../outside")): - with self.assertRaises(DocForgeError) as captured: - JavaScriptReferenceAdapter(root, source_roots=(source_root,)) - self.assertEqual("path_escape", captured.exception.code) - - outside = parent / "outside" - outside.mkdir() - linked_root = root / "linked" - linked_root.symlink_to(outside, target_is_directory=True) - with self.assertRaises(DocForgeError) as linked: - JavaScriptReferenceAdapter(root, source_roots=("linked",)) - self.assertEqual("path_escape", linked.exception.code) - - adapter = RecordingScriptAdapter(root, fixture) - project = AdapterProject(adapter, cache_root=root / ".cache" / "no-ast") - with self.assertRaises(DocForgeError) as no_ast: - ProjectIndex(project, allow_logic=False).build() - self.assertEqual("adapter_policy_forbids_logic", no_ast.exception.code) - self.assertFalse(project.descriptor.index_path.exists()) - - def test_missing_optional_frontends_report_exact_install_remediation(self) -> None: - for fixture, language, _nodes, _edges, _logic, _shared_path in self.CASES: - grammar_module = ( - "tree_sitter_javascript" if language == "javascript" else "tree_sitter_typescript" - ) - with self.subTest(language=language), tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory), fixture) - adapter = RecordingScriptAdapter(root, fixture) - missing = ModuleNotFoundError( - f"No module named {grammar_module!r}", - name=grammar_module, - ) - with ( - mock.patch.object( - javascript_adapter.importlib, - "import_module", - side_effect=missing, - ), - self.assertRaises(DocForgeError) as captured, - ): - source = adapter.load_manifest().sources[0] - adapter.extract_source(source) - self.assertEqual("optional_dependency_missing", captured.exception.code) - self.assertEqual(language, captured.exception.details["extra"]) - self.assertEqual( - f"docforge[{language}]", - captured.exception.details["install"], - ) - self.assertEqual( - grammar_module, - captured.exception.details["missing_module"], - ) diff --git a/tests/test_milestone4_adoption.py b/tests/test_milestone4_adoption.py deleted file mode 100644 index af13c3e..0000000 --- a/tests/test_milestone4_adoption.py +++ /dev/null @@ -1,137 +0,0 @@ -from __future__ import annotations - -import json -import subprocess -import tempfile -import unittest -from pathlib import Path -from unittest import mock - -from tools.milestone4_adoption import ( - AdoptionProofError, - find_wheel, - parse_evidence, - run_adoption_proof, -) - - -def _evidence() -> dict[str, object]: - return { - "schema_version": 1, - "base_frontend_modules": [], - "base_frontend_distributions": [], - "python": {"build_status": "ok", "check_status": "ok", "node_count": 3}, - "mcp": { - "bootstrap_status": "ok", - "node_id": "python.class.service", - "read_tool_count": 20, - }, - "cpp_without_extra": { - "code": "optional_dependency_missing", - "install": "docforge[cpp]", - "missing_module": "tree_sitter", - }, - } - - -class Milestone4AdoptionTests(unittest.TestCase): - def test_evidence_requires_base_isolation_python_mcp_and_actionable_cpp_failure( - self, - ) -> None: - self.assertEqual(_evidence(), parse_evidence(json.dumps(_evidence()))) - - for mutation in ( - {"base_frontend_modules": ["tree_sitter"]}, - {"python": {"build_status": "error", "check_status": "ok"}}, - {"mcp": {"bootstrap_status": "error"}}, - { - "cpp_without_extra": { - "code": "optional_dependency_missing", - "install": "docforge[languages]", - } - }, - ): - evidence = {**_evidence(), **mutation} - with self.subTest(mutation=mutation), self.assertRaises(AdoptionProofError): - parse_evidence(json.dumps(evidence)) - - def test_wheel_discovery_rejects_missing_or_ambiguous_artifacts(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = Path(directory) - with self.assertRaises(AdoptionProofError): - find_wheel(root) - first = root / "docforge-1-py3-none-any.whl" - first.touch() - self.assertEqual(first, find_wheel(root)) - (root / "docforge-2-py3-none-any.whl").touch() - with self.assertRaises(AdoptionProofError): - find_wheel(root) - - def test_orchestration_is_offline_locked_and_installs_the_wheel_without_extras( - self, - ) -> None: - with tempfile.TemporaryDirectory() as directory: - root = Path(directory).resolve() - (root / "pyproject.toml").write_text("[project]\n", encoding="utf-8") - (root / "uv.lock").write_text("version = 1\n", encoding="utf-8") - commands: list[list[str]] = [] - - def execute( - arguments: list[str], - *, - cwd: Path, - timeout: int, - environment: dict[str, str] | None = None, - ) -> subprocess.CompletedProcess[str]: - del cwd, timeout - commands.append(arguments) - self.assertIsNotNone(environment) - assert environment is not None - self.assertEqual("1", environment["UV_OFFLINE"]) - self.assertNotIn("PYTHONPATH", environment) - if arguments[1] == "build": - output = Path(arguments[arguments.index("--out-dir") + 1]) - output.mkdir() - (output / "docforge-1-py3-none-any.whl").write_bytes(b"wheel") - elif arguments[1] == "export": - output = Path(arguments[arguments.index("--output-file") + 1]) - output.write_text("mcp==1\n", encoding="utf-8") - elif arguments[1] == "venv": - virtual_environment = Path(arguments[-1]) - (virtual_environment / "bin").mkdir(parents=True) - (virtual_environment / "bin" / "python").touch() - elif arguments[0].endswith("/bin/python"): - return subprocess.CompletedProcess( - arguments, - 0, - stdout=json.dumps(_evidence()), - stderr="", - ) - return subprocess.CompletedProcess(arguments, 0, stdout="", stderr="") - - with mock.patch( - "tools.milestone4_adoption._run_checked", - side_effect=execute, - ): - result = run_adoption_proof(root, uv="uv") - - self.assertEqual("offline", result["network"]) - wheel = result["wheel"] - self.assertIsInstance(wheel, dict) - assert isinstance(wheel, dict) - self.assertEqual("docforge-1-py3-none-any.whl", wheel["filename"]) - network_commands = ( - command - for command in commands - if command[0] == "uv" and command[1] in {"build", "export", "pip"} - ) - self.assertTrue(all("--offline" in command for command in network_commands)) - export = next(command for command in commands if command[1] == "export") - self.assertIn("--frozen", export) - self.assertIn("--no-dev", export) - wheel_install = next(command for command in commands if command[1:3] == ["pip", "install"]) - self.assertIn("--no-deps", wheel_install) - - -if __name__ == "__main__": - unittest.main() diff --git a/tests/test_public_contract.py b/tests/test_public_contract.py index 6aba6e8..e32af11 100644 --- a/tests/test_public_contract.py +++ b/tests/test_public_contract.py @@ -43,7 +43,6 @@ PUBLIC_IMPORTS = { ), "docforge.adapter_contract": ( "AdapterAssembly", - "AdapterConformanceReport", "AdapterEdge", "AdapterImplementation", "AdapterLoader", @@ -54,41 +53,15 @@ PUBLIC_IMPORTS = { "AdapterProjectSettings", "AdapterSource", "AdapterSourceProjection", - "CompleteAdapterAssemblyLoader", "IncrementalAdapterAssembler", "IncrementalAdapterLoader", - "verify_adapter_conformance", - ), - "docforge.adapter_sdk": ( - "AdapterAssembly", - "AdapterConformanceReport", - "AdapterLoader", - "AdapterManifest", - "AdapterProject", - "AdapterProjection", - "AdapterSourceProjection", - "CompleteAdapterAssemblyLoader", - "IncrementalAdapterLoader", - "LogicProjection", - "Node", - "verify_adapter_conformance", ), "docforge.application": ( "CanonicalApplier", "CanonicalApplicationService", "GenericCanonicalApplier", ), - "docforge.adapter_launcher": ( - "ADAPTER_LAUNCHER_SCHEMA_VERSION", - "AdapterLauncherV1", - "AdapterSourceAvailabilityV1", - "adapter_source_availability", - "validate_adapter_launcher", - ), - "docforge.client_config": ( - "generate_adapter_client_configuration", - "generate_client_configuration", - ), + "docforge.client_config": ("generate_client_configuration",), "docforge.doctor": ("run_doctor",), "docforge.index": ("ProjectIndex",), "docforge.graph_projection": ( @@ -145,18 +118,6 @@ PUBLIC_IMPORTS = { "projection_hash", ), "docforge.projection_worker": ("render_projection_in_worker",), - "docforge.reference_config": ( - "REFERENCE_ADAPTER_CONFIG", - "REFERENCE_ADAPTER_SCHEMA_VERSION", - "REFERENCE_LANGUAGES", - "ReferenceAdapterConfigV1", - "load_reference_adapter_config", - ), - "docforge.reference_mcp": ( - "REFERENCE_MCP_MODULE", - "create_reference_project", - "create_reference_server", - ), "docforge.retrieval": ( "ContextCapsuleV1", "RetrievalPlanV1", diff --git a/tests/test_python_reference_adapter.py b/tests/test_python_reference_adapter.py deleted file mode 100644 index 0755321..0000000 --- a/tests/test_python_reference_adapter.py +++ /dev/null @@ -1,282 +0,0 @@ -from __future__ import annotations - -import shutil -import tempfile -import unittest -from pathlib import Path -from typing import cast -from unittest import mock - -import docforge.adapters.python as python_adapter -from docforge.adapter_sdk import ( - AdapterProject, - AdapterSource, - AdapterSourceProjection, - verify_adapter_conformance, -) -from docforge.adapters.python import ( - PYTHON_ADAPTER_ID, - PYTHON_ADAPTER_VERSION, - PYTHON_EXTRACTOR_VERSION, - PythonReferenceAdapter, -) -from docforge.errors import DocForgeError -from docforge.index import ProjectIndex - -ROOT = Path(__file__).resolve().parents[1] -FIXTURE = ROOT / "tests" / "fixtures" / "reference-python" - - -class RecordingPythonAdapter(PythonReferenceAdapter): - def __init__(self, root: Path) -> None: - super().__init__( - root, - source_roots=("src",), - project_id="reference-python", - title="Reference Python fixture", - ) - self.extracted_paths: list[str] = [] - - def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: - self.extracted_paths.append(source.source_path) - return super().extract_source(source) - - -class PythonReferenceAdapterTests(unittest.TestCase): - def copy_fixture(self, parent: Path) -> Path: - root = parent / "reference-python" - shutil.copytree(FIXTURE, root) - return root.resolve() - - def adapter(self, root: Path) -> RecordingPythonAdapter: - return RecordingPythonAdapter(root) - - @staticmethod - def build_metrics(result: dict[str, object]) -> dict[str, object]: - return cast(dict[str, object], result["build"]) - - def test_complete_assembly_is_exact_ordered_cache_independent_and_scoped(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - adapter = self.adapter(root) - - first = adapter.load_assembly() - second = adapter.load_complete_assembly() - projection = adapter.load_projection() - - self.assertEqual(first, second) - self.assertEqual(first.projection, projection) - self.assertEqual(PYTHON_ADAPTER_ID, projection.adapter_id) - self.assertEqual(PYTHON_ADAPTER_VERSION, projection.adapter_version) - self.assertEqual(14, len(projection.nodes)) - self.assertEqual(13, len(projection.edges)) - self.assertEqual(5, len(first.logic)) - self.assertEqual( - list(projection.nodes), - sorted(projection.nodes, key=lambda item: item.node.node_id), - ) - self.assertEqual( - {"contains", "depends_on"}, {item.edge.relation for item in projection.edges} - ) - self.assertFalse(any(item.node.title == "json" for item in projection.nodes)) - self.assertTrue( - any( - item.node.title == "Service" and ("kind", "class") in item.metadata - for item in projection.nodes - ) - ) - self.assertTrue( - any( - item.node.title == "Service.run" and ("kind", "function") in item.metadata - for item in projection.nodes - ) - ) - - report = adapter.support_report() - self.assertEqual(1, report["schema_version"]) - self.assertEqual(PYTHON_EXTRACTOR_VERSION, report["extractor_version"]) - self.assertFalse(report["imports_project_code"]) - self.assertFalse(report["executes_project_code"]) - self.assertEqual( - [ - "call_resolution", - "dynamic_import_resolution", - "inheritance_resolution", - "runtime_generated_facts", - "symbol_reference_resolution", - ], - [item.code for item in adapter.unsupported_facts()], - ) - self.assertFalse((root / ".cache").exists()) - - conformance = verify_adapter_conformance( - adapter, - cache_root=root / ".cache" / "conformance", - ) - self.assertEqual("reference-python", conformance.project_id) - self.assertEqual(14, conformance.node_count) - self.assertEqual(13, conformance.edge_count) - self.assertEqual(5, conformance.logic_projection_count) - self.assertTrue(conformance.incremental) - - def test_incremental_cold_warm_reverse_dependencies_and_logic_equivalence(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - adapter = self.adapter(root) - project = AdapterProject(adapter, cache_root=root / ".cache" / "incremental") - index = ProjectIndex(project) - - cold = index.build() - self.assertEqual(4, self.build_metrics(cold)["reparsed_sources"]) - self.assertEqual( - [ - "src/sample/__init__.py", - "src/sample/service.py", - "src/sample/shared.py", - "src/sample/worker.py", - ], - sorted(adapter.extracted_paths), - ) - - adapter.extracted_paths.clear() - original_parse = python_adapter.ast.parse - with mock.patch.object( - python_adapter.ast, - "parse", - wraps=original_parse, - ) as parsed: - warm = index.build() - self.assertEqual(4, self.build_metrics(warm)["cache_hits"]) - self.assertEqual(0, self.build_metrics(warm)["reparsed_sources"]) - self.assertEqual([], adapter.extracted_paths) - parsed.assert_not_called() - - shared = root / "src" / "sample" / "shared.py" - shared.write_text( - shared.read_text(encoding="utf-8").replace( - "DEFAULT_LIMIT = 3", "DEFAULT_LIMIT = 5" - ), - encoding="utf-8", - ) - adapter.extracted_paths.clear() - with mock.patch.object( - python_adapter.ast, - "parse", - wraps=original_parse, - ) as parsed: - changed = index.build() - self.assertEqual(4, self.build_metrics(changed)["invalidated_sources"]) - self.assertGreater(parsed.call_count, 0) - self.assertEqual( - [ - "src/sample/__init__.py", - "src/sample/service.py", - "src/sample/shared.py", - "src/sample/worker.py", - ], - sorted(adapter.extracted_paths), - ) - - equivalent = project.verify_incremental_equivalence() - self.assertEqual("ok", equivalent["status"]) - self.assertEqual(14, equivalent["node_count"]) - self.assertEqual(13, equivalent["edge_count"]) - self.assertEqual(5, equivalent["logic_projection_count"]) - - def test_add_rename_delete_and_corrupt_cache_recover_without_stale_facts(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - adapter = self.adapter(root) - cache_root = root / ".cache" / "incremental" - project = AdapterProject(adapter, cache_root=cache_root) - index = ProjectIndex(project) - index.build() - - extra = root / "src" / "sample" / "extra.py" - extra.write_text( - '"""Added source."""\n\n\ndef extra() -> str:\n return "extra"\n', - encoding="utf-8", - ) - adapter.extracted_paths.clear() - added = index.build() - self.assertEqual(1, self.build_metrics(added)["reparsed_sources"]) - self.assertEqual(["src/sample/extra.py"], adapter.extracted_paths) - self.assertIn( - "src/sample/extra.py", - {node.source_path for node in project.load().nodes}, - ) - - worker = root / "src" / "sample" / "worker.py" - runner = root / "src" / "sample" / "runner.py" - worker.rename(runner) - adapter.extracted_paths.clear() - renamed = index.build() - self.assertEqual(1, self.build_metrics(renamed)["deleted_sources"]) - self.assertEqual(1, self.build_metrics(renamed)["reparsed_sources"]) - renamed_paths = {node.source_path for node in project.load().nodes} - self.assertNotIn("src/sample/worker.py", renamed_paths) - self.assertIn("src/sample/runner.py", renamed_paths) - - extra.unlink() - adapter.extracted_paths.clear() - deleted = index.build() - self.assertEqual(1, self.build_metrics(deleted)["deleted_sources"]) - self.assertNotIn( - "src/sample/extra.py", - {node.source_path for node in project.load().nodes}, - ) - - (cache_root / "extractions.json").write_text("{broken", encoding="utf-8") - adapter.extracted_paths.clear() - recovered = index.build() - self.assertEqual(4, self.build_metrics(recovered)["reparsed_sources"]) - self.assertEqual( - [ - "src/sample/__init__.py", - "src/sample/runner.py", - "src/sample/service.py", - "src/sample/shared.py", - ], - sorted(adapter.extracted_paths), - ) - self.assertEqual("ok", project.verify_incremental_equivalence()["status"]) - - def test_confinement_and_no_ast_policy_reject_logic_without_importing_code(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - - for source_root in (root / "src", Path("../outside")): - with self.subTest(source_root=source_root): - with self.assertRaises(DocForgeError) as captured: - PythonReferenceAdapter(root, source_roots=(source_root,)) - self.assertEqual("path_escape", captured.exception.code) - - outside = parent / "outside" - outside.mkdir() - linked_root = root / "linked" - linked_root.symlink_to(outside, target_is_directory=True) - with self.assertRaises(DocForgeError) as linked: - PythonReferenceAdapter(root, source_roots=("linked",)) - self.assertEqual("path_escape", linked.exception.code) - - external_source = outside / "external.py" - external_source.write_text("def outside():\n return True\n", encoding="utf-8") - worker = root / "src" / "sample" / "worker.py" - worker.unlink() - worker.symlink_to(external_source) - adapter = self.adapter(root) - with self.assertRaises(DocForgeError) as source_link: - adapter.load_manifest() - self.assertEqual("path_escape", source_link.exception.code) - - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - adapter = self.adapter(root) - project = AdapterProject(adapter, cache_root=root / ".cache" / "no-ast") - - with self.assertRaises(DocForgeError) as no_ast: - ProjectIndex(project, allow_logic=False).build() - - self.assertEqual("adapter_policy_forbids_logic", no_ast.exception.code) - self.assertFalse(project.descriptor.index_path.exists()) diff --git a/tests/test_reference_mcp.py b/tests/test_reference_mcp.py deleted file mode 100644 index 5b8ad06..0000000 --- a/tests/test_reference_mcp.py +++ /dev/null @@ -1,296 +0,0 @@ -from __future__ import annotations - -import io -import json -import shutil -import sys -import tempfile -import unittest -from contextlib import redirect_stderr -from pathlib import Path -from unittest import mock - -from jsonschema import Draft202012Validator -from mcp.shared.memory import create_connected_server_and_client_session - -from docforge.adapter_launcher import AdapterLauncherV1 -from docforge.client_config import generate_adapter_client_configuration -from docforge.errors import DocForgeError -from docforge.index import ProjectIndex -from docforge.mcp_server import READ_TOOLS -from docforge.reference_config import ( - REFERENCE_ADAPTER_CONFIG, - load_reference_adapter_config, -) -from docforge.reference_mcp import ( - REFERENCE_MCP_MODULE, - create_reference_project, - create_reference_server, - main, -) - -ROOT = Path(__file__).resolve().parents[1] -FIXTURE = ROOT / "tests" / "fixtures" / "reference-python" -CPP_FIXTURE = ROOT / "tests" / "fixtures" / "reference-cpp" -SCHEMA = json.loads( - (ROOT / "schemas" / "reference-adapter.schema.json").read_text(encoding="utf-8") -) - - -class ReferenceMcpTests(unittest.IsolatedAsyncioTestCase): - def copy_fixture(self, parent: Path) -> Path: - root = parent / "reference-python" - shutil.copytree(FIXTURE, root) - return root.resolve() - - def copy_cpp_fixture(self, parent: Path) -> Path: - root = parent / "reference-cpp" - shutil.copytree(CPP_FIXTURE, root) - return root.resolve() - - @staticmethod - def write_config( - root: Path, - *, - language: str = "python", - source_roots: tuple[str, ...] = ("src",), - extra: str = "", - ) -> Path: - config = root.joinpath(*REFERENCE_ADAPTER_CONFIG.parts) - config.parent.mkdir(parents=True, exist_ok=True) - quoted_roots = ", ".join(json.dumps(item) for item in source_roots) - config.write_text( - "\n".join( - ( - "schema_version = 1", - 'project_id = "reference-python"', - 'title = "Runnable Python reference"', - f"language = {json.dumps(language)}", - f"source_roots = [{quoted_roots}]", - extra, - "", - ) - ), - encoding="utf-8", - ) - return config - - async def test_python_config_build_check_and_real_mcp_retrieval(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - self.write_config(root) - config = load_reference_adapter_config(root) - - Draft202012Validator(SCHEMA).validate(config.as_dict()) - self.assertEqual("python", config.language) - self.assertEqual((root / "src",), config.source_roots) - self.assertIsNone(config.compilation_database) - self.assertEqual(64, len(config.config_hash)) - - project = create_reference_project(root) - index = ProjectIndex(project) - built = index.build() - checked = index.check() - self.assertEqual("ok", built["status"]) - self.assertEqual("ok", checked["status"]) - self.assertTrue(project.descriptor.index_path.is_relative_to(root / ".docforge")) - - async with create_connected_server_and_client_session( - create_reference_server(root, capability_mode="read"), - raise_exceptions=True, - ) as session: - tools = tuple(tool.name for tool in (await session.list_tools()).tools) - bootstrap = await session.call_tool("docforge_bootstrap", {}) - search = await session.call_tool( - "docforge_search", - {"query": "Service", "limit": 5}, - ) - node_id = search.structuredContent["results"][0]["node_id"] - node = await session.call_tool("docforge_get_node", {"node_id": node_id}) - - self.assertEqual(READ_TOOLS, tools) - self.assertEqual("ok", bootstrap.structuredContent["status"]) - self.assertEqual( - REFERENCE_MCP_MODULE, - bootstrap.structuredContent["binding"]["server_module"], - ) - self.assertEqual( - "python", - bootstrap.structuredContent["binding"]["reference_language"], - ) - self.assertGreater(search.structuredContent["count"], 0) - self.assertEqual(node_id, node.structuredContent["node"]["node_id"]) - - async def test_cpp_config_build_check_and_real_mcp_retrieval(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_cpp_fixture(Path(directory)) - self.write_config( - root, - language="cpp", - source_roots=("include", "src"), - extra='compilation_database = "compile_commands.json"', - ) - config = load_reference_adapter_config(root) - Draft202012Validator(SCHEMA).validate(config.as_dict()) - - project = create_reference_project(root) - index = ProjectIndex(project) - self.assertEqual("ok", index.build()["status"]) - self.assertEqual("ok", index.check()["status"]) - - async with create_connected_server_and_client_session( - create_reference_server(root, capability_mode="read"), - raise_exceptions=True, - ) as session: - bootstrap = await session.call_tool("docforge_bootstrap", {}) - search = await session.call_tool( - "docforge_search", - {"query": "worker", "limit": 5}, - ) - node_id = search.structuredContent["results"][0]["node_id"] - node = await session.call_tool("docforge_get_node", {"node_id": node_id}) - - self.assertEqual("cpp", config.language) - self.assertEqual(root / "compile_commands.json", config.compilation_database) - self.assertEqual( - "cpp", - bootstrap.structuredContent["binding"]["reference_language"], - ) - self.assertEqual(node_id, node.structuredContent["node"]["node_id"]) - - def test_launcher_generates_exact_isolated_module_invocation(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - self.write_config(root) - project = create_reference_project(root) - launcher = AdapterLauncherV1.for_project(project, module=REFERENCE_MCP_MODULE) - - with mock.patch("docforge.client_config.subprocess.run") as executed: - result = generate_adapter_client_configuration( - project, - launcher, - "codex", - capability_mode="read", - ) - - self.assertEqual( - [ - "-I", - "-m", - REFERENCE_MCP_MODULE, - "--project-root", - str(root), - "--capability-mode", - "read", - ], - result["binding"]["args"], - ) - self.assertEqual({}, result["binding"]["environment"]) - self.assertNotIn("cwd", result["binding"]) - executed.assert_not_called() - - def test_config_drift_requires_runtime_restart(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - config_path = self.write_config(root) - project = create_reference_project(root) - project.validate_runtime() - - config_path.write_text( - config_path.read_text(encoding="utf-8").replace( - "Runnable Python reference", - "Changed Python reference", - ), - encoding="utf-8", - ) - with self.assertRaises(DocForgeError) as captured: - project.validate_runtime() - self.assertEqual("adapter_restart_required", captured.exception.code) - self.assertEqual( - [REFERENCE_ADAPTER_CONFIG.as_posix()], - captured.exception.details["changed"], - ) - - def test_config_rejects_unknown_fields_languages_and_language_mismatch(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - for language, extra, code in ( - ("ruby", "", "unsupported_reference_language"), - ("javascript", "", "reference_language_mismatch"), - ("python", 'command = "python project.py"', "invalid_reference_config"), - ( - "python", - 'compilation_database = "compile_commands.json"', - "invalid_reference_config", - ), - ("cpp", "", "invalid_reference_config"), - ): - with self.subTest(language=language, extra=extra): - self.write_config(root, language=language, extra=extra) - with self.assertRaises(DocForgeError) as captured: - load_reference_adapter_config(root) - self.assertEqual(code, captured.exception.code) - - def test_config_rejects_path_escape_and_source_symlinks(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - outside = parent / "outside" - outside.mkdir() - - self.write_config(root, source_roots=("../outside",)) - with self.assertRaises(DocForgeError) as escaped: - load_reference_adapter_config(root) - self.assertEqual("path_escape", escaped.exception.code) - - linked = root / "linked" - linked.symlink_to(outside, target_is_directory=True) - self.write_config(root, source_roots=("linked",)) - with self.assertRaises(DocForgeError) as linked_root: - load_reference_adapter_config(root) - self.assertEqual("path_escape", linked_root.exception.code) - - self.write_config(root) - external_source = outside / "external.py" - external_source.write_text("def external(): pass\n", encoding="utf-8") - source_link = root / "src" / "external.py" - source_link.symlink_to(external_source) - with self.assertRaises(DocForgeError) as linked_source: - load_reference_adapter_config(root) - self.assertEqual("path_escape", linked_source.exception.code) - - def test_config_file_must_be_a_regular_non_symlink(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - config_path = self.write_config(root) - outside = parent / "reference-adapter.toml" - shutil.copyfile(config_path, outside) - config_path.unlink() - config_path.symlink_to(outside) - - with self.assertRaises(DocForgeError) as captured: - load_reference_adapter_config(root) - self.assertEqual("path_escape", captured.exception.code) - - def test_reference_module_cli_advertises_read_mode_only(self) -> None: - with ( - redirect_stderr(io.StringIO()), - mock.patch.object( - sys, - "argv", - [ - REFERENCE_MCP_MODULE, - "--project-root", - "/irrelevant", - "--capability-mode", - "proposal", - ], - ), - mock.patch("docforge.reference_mcp.create_reference_server") as create_server, - self.assertRaises(SystemExit) as captured, - ): - main() - - self.assertEqual(2, captured.exception.code) - create_server.assert_not_called() diff --git a/tests/test_treesitter_logic.py b/tests/test_treesitter_logic.py index dad4da3..51913b3 100644 --- a/tests/test_treesitter_logic.py +++ b/tests/test_treesitter_logic.py @@ -7,8 +7,6 @@ from docforge.treesitter_logic import ( TreeSitterLogicOwner, analyze_cpp_source, analyze_javascript_source, - analyze_typescript_source, - discover_typescript_functions, ) @@ -202,44 +200,5 @@ public: self.assertEqual(missing.exception.code, "missing_logic_owner") -class TypeScriptLogicTests(unittest.TestCase): - def test_typed_function_and_method_use_the_typescript_grammar(self) -> None: - source = """ -interface Choice { - enabled: boolean; -} - -function choose(choice: Choice): number { - if (choice.enabled) { - return 1; - } - return 0; -} - -class Worker { - run(choice: Choice): number { - return choose(choice); - } -} -""".strip() - - discovered = discover_typescript_functions(source) - self.assertEqual( - [("Worker.run", 13), ("choose", 5)], - [(item.qualified_name, item.line) for item in discovered], - ) - projection = analyze_typescript_source( - source, - source_id="source.typescript", - owners=(TreeSitterLogicOwner("ts.symbol.choose", "choose", 5),), - )[0] - - self.assertIn("choice.enabled", {node.label for node in projection.nodes}) - self.assertEqual( - {"return"}, - {edge.relation for edge in projection.edges if edge.relation == "return"}, - ) - - if __name__ == "__main__": unittest.main() diff --git a/tools/check_documentation.py b/tools/check_documentation.py deleted file mode 100644 index df32b56..0000000 --- a/tools/check_documentation.py +++ /dev/null @@ -1,698 +0,0 @@ -"""Validate the maintained DocForge Markdown documentation graph.""" - -from __future__ import annotations - -import argparse -import html -import json -import re -import stat -import sys -import tomllib -import unicodedata -from collections import deque -from collections.abc import Iterable, Sequence -from dataclasses import dataclass -from pathlib import Path -from typing import Protocol, cast -from urllib.parse import unquote, urlsplit - -from jsonschema import Draft202012Validator -from jsonschema.exceptions import ValidationError -from markdown_it import MarkdownIt -from markdown_it.token import Token - -REPOSITORY_ROOT = Path(__file__).resolve().parents[1] -DEFAULT_PENDING_INVENTORY = Path("tools/docs_pending_m4_pages.txt") -COMMAND_REFERENCE = Path("docs/COMMAND_REFERENCE.md") -COMMAND_REFERENCE_H1 = "DocForge command reference" -COMMAND_REFERENCE_NOTICE = ( - "> Generated from the live CLI parser and MCP registrations. Do not edit this file by hand." -) -REFERENCE_ADAPTER_SCHEMA = Path("schemas/reference-adapter.schema.json") -MAX_MARKDOWN_PAGES = 256 -MAX_PAGE_BYTES = 2_000_000 -MAX_TOTAL_BYTES = 20_000_000 -MAX_DIAGNOSTICS = 100 - -REQUIRED_MILESTONE_4_PAGES = ( - Path("README.md"), - Path("docs/ADAPTER_AUTHORING_GUIDE.md"), - Path("docs/AGENT_INTEGRATION.md"), - Path("docs/APPLICATION_DECISION.md"), - Path("docs/COMMAND_REFERENCE.md"), - Path("docs/COMPATIBILITY.md"), - Path("docs/CONTRACT.md"), - Path("docs/CORE_CONCEPTS_AND_AUTHORITY.md"), - Path("docs/INCREMENTAL_INDEXING.md"), - Path("docs/LEGACY_AND_NO_AST.md"), - Path("docs/MCP_CONTRACT.md"), - Path("docs/MIGRATING_FROM_V1.md"), - Path("docs/MILESTONE_4_BASELINE.md"), - Path("docs/MILESTONE_4_CLOSEOUT.md"), - Path("docs/NEW_PROJECT_QUICKSTART.md"), - Path("docs/POLICY_PRECEDENCE.md"), - Path("docs/PROJECT_DESCRIPTOR.md"), - Path("docs/PROJECT_ONBOARDING.md"), - Path("docs/RECOVERY_AND_PERFORMANCE.md"), - Path("docs/REFERENCE_ADAPTERS.md"), - Path("docs/RENDERING_AND_VISUALIZATION.md"), - Path("docs/SECURITY.md"), - Path("docs/USER_MANUAL.md"), - Path("docs/VIEWER_MANAGER.md"), -) - -_HISTORICAL_RECORD = re.compile(r"^docs/MILESTONE_[0-3]_(?:BASELINE|CLOSEOUT)\.md$") -_HTML_ID = re.compile( - r"""\bid\s*=\s*(?:"([^"]+)"|'([^']+)')""", - re.IGNORECASE, -) -_REFERENCE_ADAPTER_KEY = re.compile(r"(?m)^(?:language|source_roots|compilation_database)\s*=") -_URI_SCHEME = re.compile(r"^[A-Za-z][A-Za-z0-9+.-]*:") - - -@dataclass(frozen=True, order=True) -class Diagnostic: - """One deterministic documentation validation failure.""" - - path: str - line: int - code: str - message: str - - def render(self) -> str: - """Render one bounded compiler-style diagnostic.""" - - return f"{self.path}:{self.line}: {self.code}: {self.message}" - - -@dataclass(frozen=True) -class DocumentationPolicy: - """Repository-relative pages and temporary inventory accepted by one run.""" - - required_pages: tuple[Path, ...] = REQUIRED_MILESTONE_4_PAGES - pending_inventory: Path | None = DEFAULT_PENDING_INVENTORY - - -@dataclass(frozen=True) -class DocumentationReport: - """Bounded deterministic result from one complete documentation check.""" - - diagnostics: tuple[Diagnostic, ...] - omitted_diagnostics: int - markdown_pages: int - pending_pages: int - reference_adapter_examples: int - - @property - def ok(self) -> bool: - """Return whether every documentation invariant passed.""" - - return not self.diagnostics and self.omitted_diagnostics == 0 - - -@dataclass(frozen=True) -class _Page: - relative: Path - source: str - tokens: tuple[Token, ...] - anchors: frozenset[str] - - -class _Diagnostics: - def __init__(self) -> None: - self._items: list[Diagnostic] = [] - self._omitted = 0 - - def add(self, path: Path | str, line: int, code: str, message: str) -> None: - relative = path.as_posix() if isinstance(path, Path) else path - item = Diagnostic(relative, max(1, line), code, message) - if len(self._items) < MAX_DIAGNOSTICS: - self._items.append(item) - else: - self._omitted += 1 - - def result(self) -> tuple[tuple[Diagnostic, ...], int]: - return tuple(sorted(self._items)), self._omitted - - -class _Validator(Protocol): - def iter_errors(self, instance: object) -> Iterable[ValidationError]: ... - - -def check_documentation( - repository_root: Path, - *, - policy: DocumentationPolicy | None = None, -) -> DocumentationReport: - """Check one repository's maintained Markdown pages without modifying it.""" - - root = repository_root.resolve(strict=True) - if not root.is_dir(): - raise ValueError(f"repository root is not a directory: {root}") - effective_policy = policy or DocumentationPolicy() - - diagnostics = _Diagnostics() - parser = MarkdownIt("commonmark", {"html": True, "typographer": False}) - page_paths = _discover_pages(root, diagnostics) - pages = _load_pages(root, page_paths, parser, diagnostics) - pending = _load_pending_inventory(root, effective_policy, diagnostics) - _check_required_inventory( - root, - effective_policy.required_pages, - pending, - diagnostics, - ) - _check_h1s(pages, diagnostics) - link_graph = _check_links(root, pages, parser, diagnostics) - _check_reachability(pages, link_graph, diagnostics) - reference_examples = _check_reference_adapter_examples(root, pages, diagnostics) - _check_command_reference(pages, diagnostics) - items, omitted = diagnostics.result() - return DocumentationReport( - diagnostics=items, - omitted_diagnostics=omitted, - markdown_pages=len(pages), - pending_pages=len(pending), - reference_adapter_examples=reference_examples, - ) - - -def _discover_pages(root: Path, diagnostics: _Diagnostics) -> tuple[Path, ...]: - candidates = [Path("README.md")] - docs = root / "docs" - if not docs.is_dir(): - diagnostics.add("docs", 1, "DOC001", "maintained documentation directory is missing") - return tuple(candidates) - - candidates.extend( - path.relative_to(root) - for path in docs.rglob("*.md") - if not path.is_symlink() and path.is_file() - ) - unique = tuple(sorted(set(candidates), key=Path.as_posix)) - if len(unique) > MAX_MARKDOWN_PAGES: - diagnostics.add( - "docs", - 1, - "DOC002", - f"found {len(unique)} Markdown pages; limit is {MAX_MARKDOWN_PAGES}", - ) - return unique[:MAX_MARKDOWN_PAGES] - return unique - - -def _load_pages( - root: Path, - paths: tuple[Path, ...], - parser: MarkdownIt, - diagnostics: _Diagnostics, -) -> dict[Path, _Page]: - pages: dict[Path, _Page] = {} - total_bytes = 0 - for relative in paths: - absolute = root / relative - try: - metadata = absolute.lstat() - except FileNotFoundError: - diagnostics.add(relative, 1, "DOC003", "maintained Markdown page is missing") - continue - if not stat.S_ISREG(metadata.st_mode): - diagnostics.add(relative, 1, "DOC004", "maintained Markdown page is not a regular file") - continue - if metadata.st_size > MAX_PAGE_BYTES: - diagnostics.add( - relative, - 1, - "DOC005", - f"page is {metadata.st_size} bytes; limit is {MAX_PAGE_BYTES}", - ) - continue - total_bytes += metadata.st_size - if total_bytes > MAX_TOTAL_BYTES: - diagnostics.add( - relative, - 1, - "DOC006", - f"total Markdown input exceeds {MAX_TOTAL_BYTES} bytes", - ) - break - try: - source = absolute.read_text(encoding="utf-8") - except UnicodeDecodeError: - diagnostics.add(relative, 1, "DOC007", "page is not valid UTF-8") - continue - tokens = tuple(parser.parse(source)) - pages[relative] = _Page( - relative=relative, - source=source, - tokens=tokens, - anchors=_anchors(tokens), - ) - return pages - - -def _load_pending_inventory( - root: Path, - policy: DocumentationPolicy, - diagnostics: _Diagnostics, -) -> frozenset[Path]: - inventory = policy.pending_inventory - if inventory is None: - return frozenset() - if inventory.is_absolute(): - diagnostics.add( - inventory.as_posix(), - 1, - "DOC008", - "pending inventory path must be repository-relative", - ) - return frozenset() - absolute = root / inventory - try: - lines = absolute.read_text(encoding="utf-8").splitlines() - except FileNotFoundError: - diagnostics.add(inventory, 1, "DOC009", "pending inventory file is missing") - return frozenset() - except UnicodeDecodeError: - diagnostics.add(inventory, 1, "DOC010", "pending inventory is not valid UTF-8") - return frozenset() - - required = set(policy.required_pages) - pending: set[Path] = set() - for line_number, raw in enumerate(lines, start=1): - value = raw.strip() - if not value or value.startswith("#"): - continue - path = Path(value) - if path.is_absolute() or ".." in path.parts or path.as_posix() != value: - diagnostics.add( - inventory, - line_number, - "DOC011", - f"unsafe pending inventory path: {value!r}", - ) - continue - if path not in required: - diagnostics.add( - inventory, - line_number, - "DOC012", - f"pending page is not in the required inventory: {value}", - ) - continue - if path in pending: - diagnostics.add( - inventory, - line_number, - "DOC013", - f"duplicate pending page: {value}", - ) - continue - pending.add(path) - return frozenset(pending) - - -def _check_required_inventory( - root: Path, - required_pages: tuple[Path, ...], - pending: frozenset[Path], - diagnostics: _Diagnostics, -) -> None: - for relative in sorted(set(required_pages), key=Path.as_posix): - exists = (root / relative).is_file() - if relative in pending: - if exists: - diagnostics.add( - relative, - 1, - "DOC014", - "page exists but remains in the pending inventory", - ) - continue - if not exists: - diagnostics.add(relative, 1, "DOC015", "required Milestone 4 page is missing") - - -def _check_h1s(pages: dict[Path, _Page], diagnostics: _Diagnostics) -> None: - for relative, page in pages.items(): - h1_lines = [ - _token_line(token) - for token in page.tokens - if token.type == "heading_open" and token.tag == "h1" - ] - if len(h1_lines) != 1: - rendered = ", ".join(str(line) for line in h1_lines) or "none" - diagnostics.add( - relative, - h1_lines[1] if len(h1_lines) > 1 else 1, - "DOC016", - f"expected exactly one H1; found {len(h1_lines)} (lines: {rendered})", - ) - - -def _check_links( - root: Path, - pages: dict[Path, _Page], - parser: MarkdownIt, - diagnostics: _Diagnostics, -) -> dict[Path, frozenset[Path]]: - graph: dict[Path, frozenset[Path]] = {} - anchor_cache = {relative: page.anchors for relative, page in pages.items()} - for relative, page in pages.items(): - destinations: set[Path] = set() - for href, line in _page_links(page.tokens): - target = _resolve_local_link(root, relative, href, line, diagnostics) - if target is None: - continue - target_relative, fragment = target - if target_relative in pages: - destinations.add(target_relative) - if not fragment: - continue - anchors = anchor_cache.get(target_relative) - if anchors is None: - anchors = _load_link_target_anchors( - root, - target_relative, - parser, - diagnostics, - ) - if anchors is None: - continue - anchor_cache[target_relative] = anchors - if fragment not in anchors: - diagnostics.add( - relative, - line, - "DOC017", - f"missing anchor #{fragment} in {target_relative.as_posix()}", - ) - graph[relative] = frozenset(destinations) - return graph - - -def _page_links(tokens: tuple[Token, ...]) -> Iterable[tuple[str, int]]: - for token in tokens: - children = token.children or [] - for child in children: - attribute = "href" if child.type == "link_open" else "src" - if child.type not in {"link_open", "image"}: - continue - raw = child.attrs.get(attribute) - if isinstance(raw, str): - yield raw, _token_line(token) - - -def _resolve_local_link( - root: Path, - source: Path, - href: str, - line: int, - diagnostics: _Diagnostics, -) -> tuple[Path, str] | None: - if not href or href.startswith("//") or _URI_SCHEME.match(href): - return None - try: - parsed = urlsplit(href) - except ValueError as error: - diagnostics.add(source, line, "DOC018", f"invalid link {href!r}: {error}") - return None - if parsed.netloc: - return None - decoded_path = unquote(parsed.path) - fragment = unquote(parsed.fragment) - if "\x00" in decoded_path or decoded_path.startswith("/"): - diagnostics.add(source, line, "DOC019", f"local link escapes the repository: {href!r}") - return None - absolute = (root / source if not decoded_path else root / source.parent / decoded_path).resolve( - strict=False - ) - try: - target = absolute.relative_to(root) - except ValueError: - diagnostics.add(source, line, "DOC019", f"local link escapes the repository: {href!r}") - return None - if not absolute.is_file(): - diagnostics.add(source, line, "DOC020", f"missing local link target: {target.as_posix()}") - return None - return target, fragment - - -def _load_link_target_anchors( - root: Path, - relative: Path, - parser: MarkdownIt, - diagnostics: _Diagnostics, -) -> frozenset[str] | None: - if relative.suffix.lower() not in {".md", ".markdown"}: - diagnostics.add( - relative, - 1, - "DOC021", - "link uses an anchor on a non-Markdown target", - ) - return None - absolute = root / relative - try: - metadata = absolute.lstat() - if not stat.S_ISREG(metadata.st_mode) or metadata.st_size > MAX_PAGE_BYTES: - raise ValueError("target is not a bounded regular Markdown file") - source = absolute.read_text(encoding="utf-8") - except (OSError, UnicodeDecodeError, ValueError) as error: - diagnostics.add(relative, 1, "DOC022", f"cannot inspect linked Markdown anchors: {error}") - return None - return _anchors(tuple(parser.parse(source))) - - -def _anchors(tokens: tuple[Token, ...]) -> frozenset[str]: - anchors: set[str] = set() - slug_counts: dict[str, int] = {} - for index, token in enumerate(tokens): - if token.type in {"html_block", "html_inline"}: - anchors.update(_html_ids(token.content)) - if token.children: - for child in token.children: - if child.type == "html_inline": - anchors.update(_html_ids(child.content)) - if token.type != "heading_open" or index + 1 >= len(tokens): - continue - inline = tokens[index + 1] - base = _heading_slug(_inline_text(inline)) - duplicate = slug_counts.get(base, 0) - slug_counts[base] = duplicate + 1 - anchors.add(base if duplicate == 0 else f"{base}-{duplicate}") - return frozenset(anchors) - - -def _inline_text(token: Token) -> str: - if not token.children: - return token.content - pieces: list[str] = [] - for child in token.children: - if child.type in {"text", "code_inline"}: - pieces.append(child.content) - elif child.type in {"softbreak", "hardbreak"}: - pieces.append(" ") - elif child.type == "image": - pieces.append(child.content) - elif child.type == "html_inline": - pieces.append(re.sub(r"<[^>]*>", "", child.content)) - return "".join(pieces) - - -def _heading_slug(value: str) -> str: - characters: list[str] = [] - for character in html.unescape(value).casefold().strip(): - category = unicodedata.category(character) - if category[0] in {"L", "M", "N"} or character in {"-", "_"}: - characters.append(character) - elif character.isspace(): - characters.append("-") - return "".join(characters) - - -def _html_ids(value: str) -> set[str]: - return {first or second for first, second in _HTML_ID.findall(value)} - - -def _check_reachability( - pages: dict[Path, _Page], - graph: dict[Path, frozenset[Path]], - diagnostics: _Diagnostics, -) -> None: - root_page = Path("README.md") - if root_page not in pages: - return - reached = {root_page} - queue = deque([root_page]) - while queue: - current = queue.popleft() - for target in sorted(graph.get(current, frozenset()), key=Path.as_posix): - if target not in reached: - reached.add(target) - queue.append(target) - for relative in sorted(set(pages) - reached, key=Path.as_posix): - if _HISTORICAL_RECORD.fullmatch(relative.as_posix()): - continue - diagnostics.add( - relative, - 1, - "DOC023", - "maintained page is not reachable from README.md", - ) - - -def _check_reference_adapter_examples( - root: Path, - pages: dict[Path, _Page], - diagnostics: _Diagnostics, -) -> int: - schema_path = root / REFERENCE_ADAPTER_SCHEMA - try: - schema_document = cast( - dict[str, object], - json.loads(schema_path.read_text(encoding="utf-8")), - ) - Draft202012Validator.check_schema(schema_document) - except (OSError, UnicodeDecodeError, json.JSONDecodeError, TypeError) as error: - diagnostics.add( - REFERENCE_ADAPTER_SCHEMA, - 1, - "DOC024", - f"cannot load reference-adapter schema: {error}", - ) - return 0 - validator = cast(_Validator, Draft202012Validator(schema_document)) - count = 0 - for relative, page in pages.items(): - previous_inline: Token | None = None - for token in page.tokens: - if token.type == "inline": - previous_inline = token - continue - if token.type != "fence" or token.info.split(maxsplit=1)[:1] != ["toml"]: - continue - if not _is_reference_adapter_example(token, previous_inline): - continue - count += 1 - line = _token_line(token) - try: - document = tomllib.loads(token.content) - except tomllib.TOMLDecodeError as error: - diagnostics.add( - relative, - line, - "DOC025", - f"invalid reference-adapter TOML example: {error}", - ) - continue - errors = sorted( - validator.iter_errors(document), - key=lambda error: tuple(str(part) for part in error.absolute_path), - ) - for error in errors: - location = ".".join(str(part) for part in error.absolute_path) or "" - diagnostics.add( - relative, - line, - "DOC026", - f"reference-adapter example {location}: {error.message}", - ) - return count - - -def _is_reference_adapter_example(token: Token, previous_inline: Token | None) -> bool: - info = token.info.casefold() - if "reference-adapter" in info or "reference_adapter" in info: - return True - if previous_inline is not None and previous_inline.map is not None and token.map is not None: - adjacent = token.map[0] - previous_inline.map[1] <= 1 - context = previous_inline.content.casefold() - if adjacent and ( - "reference-adapter.toml" in context or "reference adapter configuration" in context - ): - return True - keys = set(_REFERENCE_ADAPTER_KEY.findall(token.content)) - return {"language", "source_roots"}.issubset(keys) - - -def _check_command_reference( - pages: dict[Path, _Page], - diagnostics: _Diagnostics, -) -> None: - page = pages.get(COMMAND_REFERENCE) - if page is None: - return - lines = page.source.splitlines() - if not lines or lines[0] != f"# {COMMAND_REFERENCE_H1}": - diagnostics.add( - COMMAND_REFERENCE, - 1, - "DOC027", - f"generated reference must begin with '# {COMMAND_REFERENCE_H1}'", - ) - if COMMAND_REFERENCE_NOTICE not in lines[:5]: - diagnostics.add( - COMMAND_REFERENCE, - 2, - "DOC028", - "generated reference notice is missing or changed", - ) - - -def _token_line(token: Token) -> int: - return token.map[0] + 1 if token.map else 1 - - -def _parser() -> argparse.ArgumentParser: - parser = argparse.ArgumentParser( - description="Check deterministic DocForge documentation invariants." - ) - parser.add_argument("--repository-root", type=Path, default=REPOSITORY_ROOT) - parser.add_argument( - "--pending-inventory", - type=Path, - default=DEFAULT_PENDING_INVENTORY, - help="repository-relative list of required pages not authored yet", - ) - return parser - - -def main(arguments: Sequence[str] | None = None) -> int: - parsed = _parser().parse_args(arguments) - try: - report = check_documentation( - parsed.repository_root, - policy=DocumentationPolicy(pending_inventory=parsed.pending_inventory), - ) - except Exception as error: - print(f"documentation check failed: {error}", file=sys.stderr) - return 2 - if not report.ok: - for diagnostic in report.diagnostics: - print(diagnostic.render(), file=sys.stderr) - if report.omitted_diagnostics: - print( - f"... {report.omitted_diagnostics} additional diagnostics omitted " - f"(limit {MAX_DIAGNOSTICS})", - file=sys.stderr, - ) - return 1 - print( - json.dumps( - { - "markdown_pages": report.markdown_pages, - "pending_pages": report.pending_pages, - "reference_adapter_examples": report.reference_adapter_examples, - "status": "ok", - }, - sort_keys=True, - separators=(",", ":"), - ) - ) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/tools/docs_pending_m4_pages.txt b/tools/docs_pending_m4_pages.txt deleted file mode 100644 index 6b179f8..0000000 --- a/tools/docs_pending_m4_pages.txt +++ /dev/null @@ -1 +0,0 @@ -# Milestone 4 required-page inventory is complete. diff --git a/tools/generate_command_reference.py b/tools/generate_command_reference.py deleted file mode 100644 index 71d0c9f..0000000 --- a/tools/generate_command_reference.py +++ /dev/null @@ -1,533 +0,0 @@ -"""Generate or verify the deterministic DocForge CLI and MCP command reference.""" - -from __future__ import annotations - -import argparse -import asyncio -import contextlib -import ctypes -import errno -import fcntl -import json -import os -import secrets -import shutil -import stat -import sys -import tempfile -from collections.abc import Sequence -from dataclasses import dataclass -from pathlib import Path -from typing import Literal, Protocol, cast - -from mcp.shared.memory import create_connected_server_and_client_session - -from docforge.command_reference import ( - CliCommandReference, - McpToolReference, - cli_command_references, - mcp_tool_references, - render_command_reference_markdown, -) -from docforge.index import ProjectIndex -from docforge.mcp_server import ALL_TOOLS, APPLICATION_TOOLS, create_server -from docforge.project import Project - -REPOSITORY_ROOT = Path(__file__).resolve().parents[1] -DEFAULT_PROJECT_ROOT = REPOSITORY_ROOT / "tests" / "fixtures" / "alpha" -EXPECTED_CLI_ROWS = 28 -EXPECTED_MCP_ROWS = 36 -MAX_REFERENCE_BYTES = 5_000_000 -RENAME_EXCHANGE = 2 - - -class CommandReferenceToolError(RuntimeError): - """One safe repository-tool validation or publication failure.""" - - -class CommandReferenceDrift(CommandReferenceToolError): - """The checked output does not match current registered command metadata.""" - - -@dataclass(frozen=True) -class _FileIdentity: - device: int - inode: int - mode: int - owner: int - group: int - size: int - modified_ns: int - changed_ns: int - - -class _RenameAt2(Protocol): - argtypes: list[object] - restype: object - - def __call__( - self, - old_directory_fd: int, - old_name: bytes, - new_directory_fd: int, - new_name: bytes, - flags: int, - /, - ) -> int: ... - - -async def collect_command_reference_rows( - project_root: Path, - *, - proposal_writer: str, - canonical_applier: str, -) -> tuple[tuple[CliCommandReference, ...], tuple[McpToolReference, ...]]: - """Collect the real CLI parser and full project-bound MCP registration surface.""" - - source = _safe_project_root(project_root) - with tempfile.TemporaryDirectory(prefix="docforge-command-reference-") as directory: - copied = Path(directory) / "project" - shutil.copytree(source, copied, symlinks=True) - ProjectIndex(Project.open(copied)).build() - server = create_server( - copied, - proposal_writer, - canonical_applier_id=canonical_applier, - ) - async with create_connected_server_and_client_session( - server, - raise_exceptions=True, - ) as session: - tools = tuple((await session.list_tools()).tools) - - cli_rows = cli_command_references() - expected_mcp_names = (*ALL_TOOLS, *APPLICATION_TOOLS) - mcp_rows = mcp_tool_references(tools, expected_names=expected_mcp_names) - if len(cli_rows) != EXPECTED_CLI_ROWS: - raise CommandReferenceToolError( - f"Expected {EXPECTED_CLI_ROWS} CLI rows, found {len(cli_rows)}" - ) - if len(mcp_rows) != EXPECTED_MCP_ROWS: - raise CommandReferenceToolError( - f"Expected {EXPECTED_MCP_ROWS} MCP rows, found {len(mcp_rows)}" - ) - return cli_rows, mcp_rows - - -def generate_command_reference_bytes( - project_root: Path, - *, - proposal_writer: str = "alpha-editor", - canonical_applier: str = "alpha-editor", -) -> bytes: - """Render one stable Markdown document from an isolated project copy.""" - - cli_rows, mcp_rows = asyncio.run( - collect_command_reference_rows( - project_root, - proposal_writer=proposal_writer, - canonical_applier=canonical_applier, - ) - ) - rendered = render_command_reference_markdown(cli_rows, mcp_rows).encode("utf-8") - if len(rendered) > MAX_REFERENCE_BYTES: - raise CommandReferenceToolError("Generated command reference exceeds its byte limit") - return rendered - - -def publish_or_check_command_reference( - content: bytes, - *, - repository_root: Path, - project_root: Path, - output: Path, - check: bool, -) -> Literal["current", "unchanged", "written"]: - """Safely check or atomically publish one repository-confined Markdown file.""" - - if len(content) > MAX_REFERENCE_BYTES: - raise CommandReferenceToolError("Command reference exceeds its byte limit") - root = _safe_repository_root(repository_root) - source = _safe_project_root(project_root) - relative = _safe_output_relative(root, source, output) - parent_parts = relative.parts[:-1] - target_name = relative.name - parent_fd = _open_relative_directory(root, parent_parts) - try: - fcntl.flock(parent_fd, fcntl.LOCK_EX) - try: - initial = _file_identity(parent_fd, target_name) - existing = ( - _read_regular_file(parent_fd, target_name, limit=MAX_REFERENCE_BYTES) - if initial is not None - else None - ) - if _file_identity(parent_fd, target_name) != initial: - raise CommandReferenceToolError("Output target changed during inspection") - if check: - if existing != content: - raise CommandReferenceDrift( - f"Command reference is missing or stale: {relative.as_posix()}" - ) - return "current" - if existing == content: - return "unchanged" - _atomic_replace( - parent_fd, - target_name, - content, - expected=initial, - expected_content=existing, - ) - return "written" - finally: - fcntl.flock(parent_fd, fcntl.LOCK_UN) - finally: - os.close(parent_fd) - - -def _safe_repository_root(path: Path) -> Path: - if path.is_symlink(): - raise CommandReferenceToolError("Repository root must not be a symbolic link") - try: - resolved = path.resolve(strict=True) - except OSError as error: - raise CommandReferenceToolError("Repository root does not exist") from error - if not resolved.is_dir(): - raise CommandReferenceToolError("Repository root must be a directory") - return resolved - - -def _safe_project_root(path: Path) -> Path: - if path.is_symlink(): - raise CommandReferenceToolError("Project fixture root must not be a symbolic link") - try: - resolved = path.resolve(strict=True) - except OSError as error: - raise CommandReferenceToolError("Project fixture root does not exist") from error - if not resolved.is_dir(): - raise CommandReferenceToolError("Project fixture root must be a directory") - return resolved - - -def _safe_output_relative(root: Path, project_root: Path, output: Path) -> Path: - candidate = output if output.is_absolute() else root / output - absolute = Path(os.path.abspath(candidate)) - if absolute == root or not absolute.is_relative_to(root): - raise CommandReferenceToolError("Output must remain inside the repository root") - if absolute == project_root or absolute.is_relative_to(project_root): - raise CommandReferenceToolError("Output must not modify the source project fixture") - relative = absolute.relative_to(root) - if relative.suffix != ".md": - raise CommandReferenceToolError("Output must be one Markdown file") - if not relative.name or len(relative.parts) < 2: - raise CommandReferenceToolError("Output must be below an existing repository directory") - return relative - - -def _open_relative_directory(root: Path, parts: tuple[str, ...]) -> int: - flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW - descriptor = os.open(root, flags) - try: - for part in parts: - if part in {"", ".", ".."}: - raise CommandReferenceToolError("Output directory is unsafe") - try: - next_descriptor = os.open(part, flags, dir_fd=descriptor) - except OSError as error: - raise CommandReferenceToolError( - "Output parent must be an existing non-symlink directory" - ) from error - os.close(descriptor) - descriptor = next_descriptor - return descriptor - except Exception: - os.close(descriptor) - raise - - -def _file_identity(directory_fd: int, name: str) -> _FileIdentity | None: - try: - status = os.stat(name, dir_fd=directory_fd, follow_symlinks=False) - except FileNotFoundError: - return None - if not stat.S_ISREG(status.st_mode): - raise CommandReferenceToolError("Output target must be a regular file") - return _FileIdentity( - device=status.st_dev, - inode=status.st_ino, - mode=status.st_mode, - owner=status.st_uid, - group=status.st_gid, - size=status.st_size, - modified_ns=status.st_mtime_ns, - changed_ns=status.st_ctime_ns, - ) - - -def _read_regular_file(directory_fd: int, name: str, *, limit: int) -> bytes: - try: - descriptor = os.open(name, os.O_RDONLY | os.O_NOFOLLOW, dir_fd=directory_fd) - except OSError as error: - raise CommandReferenceToolError("Output target could not be opened safely") from error - try: - before = os.fstat(descriptor) - if not stat.S_ISREG(before.st_mode) or before.st_size > limit: - raise CommandReferenceToolError("Output target is not a bounded regular file") - chunks: list[bytes] = [] - remaining = limit + 1 - while remaining: - chunk = os.read(descriptor, min(remaining, 64 * 1024)) - if not chunk: - break - chunks.append(chunk) - remaining -= len(chunk) - content = b"".join(chunks) - after = os.fstat(descriptor) - if len(content) > limit or _identity_from_stat(before) != _identity_from_stat(after): - raise CommandReferenceToolError("Output target changed during inspection") - return content - finally: - os.close(descriptor) - - -def _identity_from_stat(status: os.stat_result) -> _FileIdentity: - return _FileIdentity( - device=status.st_dev, - inode=status.st_ino, - mode=status.st_mode, - owner=status.st_uid, - group=status.st_gid, - size=status.st_size, - modified_ns=status.st_mtime_ns, - changed_ns=status.st_ctime_ns, - ) - - -def _atomic_replace( - directory_fd: int, - name: str, - content: bytes, - *, - expected: _FileIdentity | None, - expected_content: bytes | None, -) -> None: - temporary_name = f".{name}.docforge-command-reference-{secrets.token_hex(12)}" - flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW - try: - descriptor = os.open(temporary_name, flags, 0o644, dir_fd=directory_fd) - except OSError as error: - raise CommandReferenceToolError("Could not create atomic output temporary") from error - try: - view = memoryview(content) - while view: - written = os.write(descriptor, view) - if written <= 0: - raise CommandReferenceToolError("Could not write command reference output") - view = view[written:] - os.fsync(descriptor) - except Exception: - os.close(descriptor) - _unlink_at(directory_fd, temporary_name) - raise - else: - os.close(descriptor) - temporary_contains_only_staged_content = True - try: - staged = _file_identity(directory_fd, temporary_name) - if staged is None: - raise CommandReferenceToolError("Atomic output temporary disappeared") - if _file_identity(directory_fd, name) != expected: - raise CommandReferenceToolError("Output target changed before atomic publication") - if expected is None: - _link_no_replace(directory_fd, temporary_name, name) - _unlink_at(directory_fd, temporary_name) - os.fsync(directory_fd) - return - if expected_content is None: - raise CommandReferenceToolError("Expected output content was not captured") - - _rename_exchange(directory_fd, temporary_name, name) - temporary_contains_only_staged_content = False - displaced = _file_identity(directory_fd, temporary_name) - published = _file_identity(directory_fd, name) - displaced_content = _read_regular_file( - directory_fd, - temporary_name, - limit=MAX_REFERENCE_BYTES, - ) - if ( - displaced is None - or published is None - or not _same_identity_after_rename(displaced, expected) - or displaced_content != expected_content - or not _same_identity_after_rename(published, staged) - ): - try: - _rename_exchange(directory_fd, temporary_name, name) - except Exception as error: - raise CommandReferenceToolError( - "Output target raced publication; displaced data was retained " - f"in {temporary_name}" - ) from error - temporary_contains_only_staged_content = True - if not _same_identity_after_rename( - _file_identity(directory_fd, name), - displaced, - ) or not _same_identity_after_rename( - _file_identity(directory_fd, temporary_name), - staged, - ): - raise CommandReferenceToolError( - "Output target raced publication and could not be safely restored" - ) - raise CommandReferenceToolError("Output target changed during atomic publication") - - _unlink_at(directory_fd, temporary_name) - os.fsync(directory_fd) - except Exception: - if temporary_contains_only_staged_content: - _unlink_at(directory_fd, temporary_name) - raise - - -def _same_identity_after_rename( - actual: _FileIdentity | None, - expected: _FileIdentity | None, -) -> bool: - if actual is None or expected is None: - return False - return ( - actual.device, - actual.inode, - actual.mode, - actual.owner, - actual.group, - actual.size, - actual.modified_ns, - ) == ( - expected.device, - expected.inode, - expected.mode, - expected.owner, - expected.group, - expected.size, - expected.modified_ns, - ) - - -def _link_no_replace(directory_fd: int, source: str, target: str) -> None: - try: - os.link( - source, - target, - src_dir_fd=directory_fd, - dst_dir_fd=directory_fd, - follow_symlinks=False, - ) - except FileExistsError as error: - raise CommandReferenceToolError( - "Output target appeared during atomic publication" - ) from error - except OSError as error: - raise CommandReferenceToolError("Could not publish atomic output") from error - - -def _rename_exchange(directory_fd: int, first: str, second: str) -> None: - rename_at2 = _load_rename_at2() - ctypes.set_errno(0) - result = rename_at2( - directory_fd, - os.fsencode(first), - directory_fd, - os.fsencode(second), - RENAME_EXCHANGE, - ) - if result == 0: - return - error_number = ctypes.get_errno() - if error_number in {errno.ENOSYS, errno.EINVAL, errno.EOPNOTSUPP}: - raise CommandReferenceToolError( - "Atomic exchange publication is unavailable on this filesystem" - ) - raise CommandReferenceToolError("Could not exchange atomic output") from OSError( - error_number, - os.strerror(error_number), - ) - - -def _load_rename_at2() -> _RenameAt2: - library = ctypes.CDLL(None, use_errno=True) - try: - rename_at2 = cast(_RenameAt2, library.renameat2) - except AttributeError as error: - raise CommandReferenceToolError( - "Atomic exchange publication is unavailable on this platform" - ) from error - rename_at2.argtypes = [ - ctypes.c_int, - ctypes.c_char_p, - ctypes.c_int, - ctypes.c_char_p, - ctypes.c_uint, - ] - rename_at2.restype = ctypes.c_int - return rename_at2 - - -def _unlink_at(directory_fd: int, name: str) -> None: - with contextlib.suppress(FileNotFoundError): - os.unlink(name, dir_fd=directory_fd) - - -def _parser() -> argparse.ArgumentParser: - parser = argparse.ArgumentParser( - description="Generate or check deterministic DocForge command-reference Markdown." - ) - parser.add_argument("--output", type=Path, required=True) - parser.add_argument("--check", action="store_true") - parser.add_argument("--repository-root", type=Path, default=REPOSITORY_ROOT) - parser.add_argument("--project-root", type=Path, default=DEFAULT_PROJECT_ROOT) - parser.add_argument("--proposal-writer", default="alpha-editor") - parser.add_argument("--canonical-applier", default="alpha-editor") - return parser - - -def main(arguments: Sequence[str] | None = None) -> int: - parsed = _parser().parse_args(arguments) - try: - content = generate_command_reference_bytes( - parsed.project_root, - proposal_writer=parsed.proposal_writer, - canonical_applier=parsed.canonical_applier, - ) - state = publish_or_check_command_reference( - content, - repository_root=parsed.repository_root, - project_root=parsed.project_root, - output=parsed.output, - check=parsed.check, - ) - except CommandReferenceDrift as error: - print(str(error), file=sys.stderr) - return 1 - except Exception as error: - print(f"command-reference generation failed: {error}", file=sys.stderr) - return 2 - print( - json.dumps( - { - "status": state, - "output": str(parsed.output), - "cli_rows": EXPECTED_CLI_ROWS, - "mcp_rows": EXPECTED_MCP_ROWS, - }, - sort_keys=True, - separators=(",", ":"), - ) - ) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/tools/milestone4_adoption.py b/tools/milestone4_adoption.py deleted file mode 100644 index 3e4d47e..0000000 --- a/tools/milestone4_adoption.py +++ /dev/null @@ -1,430 +0,0 @@ -#!/usr/bin/env python3 -"""Run the offline Milestone 4 fresh-wheel adoption proof.""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import os -import shutil -import subprocess -import sys -import tempfile -from pathlib import Path -from typing import cast - -PROOF_SCHEMA_VERSION = 1 -COMMAND_TIMEOUT_SECONDS = 120 -VALIDATION_TIMEOUT_SECONDS = 45 -OPTIONAL_FRONTEND_MODULES = ( - "tree_sitter", - "tree_sitter_cpp", - "tree_sitter_javascript", - "tree_sitter_typescript", -) - -_VALIDATION_PROGRAM = r""" -from __future__ import annotations - -import asyncio -import importlib.metadata -import importlib.util -import json -import sys -from pathlib import Path - -from docforge.errors import DocForgeError -from docforge.index import ProjectIndex -from docforge.reference_mcp import create_reference_project -from mcp import ClientSession, StdioServerParameters -from mcp.client.stdio import stdio_client - - -def write_config( - root: Path, - *, - language: str, - source_roots: tuple[str, ...], - compilation_database: str | None = None, -) -> None: - config = root / ".docforge" / "reference-adapter.toml" - config.parent.mkdir(parents=True) - roots = ", ".join(json.dumps(item) for item in source_roots) - lines = [ - "schema_version = 1", - 'project_id = "fresh-wheel-adoption"', - 'title = "Fresh wheel adoption proof"', - f"language = {json.dumps(language)}", - f"source_roots = [{roots}]", - ] - if compilation_database is not None: - lines.append(f"compilation_database = {json.dumps(compilation_database)}") - config.write_text("\n".join((*lines, "")), encoding="utf-8") - - -async def retrieve(python: Path, root: Path) -> dict[str, object]: - parameters = StdioServerParameters( - command=str(python), - args=[ - "-I", - "-m", - "docforge.reference_mcp", - "--project-root", - str(root), - "--capability-mode", - "read", - ], - env={}, - ) - async with ( - stdio_client(parameters) as streams, - ClientSession(*streams) as session, - ): - await session.initialize() - tools = tuple(tool.name for tool in (await session.list_tools()).tools) - bootstrap = await session.call_tool("docforge_bootstrap", {}) - search = await session.call_tool( - "docforge_search", - {"query": "Service", "limit": 5}, - ) - if search.isError or not search.structuredContent: - raise RuntimeError("fresh-wheel reference MCP search failed") - results = search.structuredContent.get("results") - if not isinstance(results, list) or not results: - raise RuntimeError("fresh-wheel reference MCP search returned no evidence") - first = results[0] - if not isinstance(first, dict) or not isinstance(first.get("node_id"), str): - raise RuntimeError("fresh-wheel reference MCP search returned invalid evidence") - node_id = first["node_id"] - node = await session.call_tool("docforge_get_node", {"node_id": node_id}) - if bootstrap.isError or not bootstrap.structuredContent: - raise RuntimeError("fresh-wheel reference MCP bootstrap failed") - if node.isError or not node.structuredContent: - raise RuntimeError("fresh-wheel reference MCP node retrieval failed") - retrieved = node.structuredContent.get("node") - if not isinstance(retrieved, dict) or retrieved.get("node_id") != node_id: - raise RuntimeError("fresh-wheel reference MCP returned the wrong node") - return { - "bootstrap_status": bootstrap.structuredContent.get("status"), - "node_id": node_id, - "read_tool_count": len(tools), - } - - -workspace = Path(sys.argv[1]).resolve(strict=True) -python = Path(sys.executable).absolute() - -present_modules = [ - name - for name in ( - "tree_sitter", - "tree_sitter_cpp", - "tree_sitter_javascript", - "tree_sitter_typescript", - ) - if importlib.util.find_spec(name) is not None -] -installed_frontends = sorted( - distribution.metadata["Name"] - for distribution in importlib.metadata.distributions() - if distribution.metadata["Name"].casefold().startswith("tree-sitter") -) -if present_modules or installed_frontends: - raise RuntimeError( - f"base wheel unexpectedly installed language frontends: " - f"{present_modules or installed_frontends}" - ) - -python_root = workspace / "python-project" -(python_root / "src" / "sample").mkdir(parents=True) -(python_root / "src" / "sample" / "__init__.py").write_text("", encoding="utf-8") -(python_root / "src" / "sample" / "service.py").write_text( - "class Service:\n" - " def execute(self, ready: bool) -> str:\n" - " if ready:\n" - ' return "ready"\n' - ' return "waiting"\n', - encoding="utf-8", -) -write_config(python_root, language="python", source_roots=("src",)) - -project = create_reference_project(python_root) -index = ProjectIndex(project) -built = index.build() -checked = index.check() -if built.get("status") != "ok" or checked.get("status") != "ok": - raise RuntimeError("fresh-wheel Python reference build/check failed") -mcp = asyncio.run(retrieve(python, python_root)) -if mcp["bootstrap_status"] != "ok": - raise RuntimeError("fresh-wheel reference MCP bootstrap did not report ok") - -cpp_root = workspace / "cpp-project" -(cpp_root / "src").mkdir(parents=True) -(cpp_root / "src" / "main.cpp").write_text("int main() { return 0; }\n", encoding="utf-8") -(cpp_root / "compile_commands.json").write_text( - json.dumps( - [ - { - "directory": str(cpp_root), - "file": "src/main.cpp", - "arguments": ["c++", "-c", "src/main.cpp"], - } - ], - sort_keys=True, - ) - + "\n", - encoding="utf-8", -) -write_config( - cpp_root, - language="cpp", - source_roots=("src",), - compilation_database="compile_commands.json", -) - -try: - ProjectIndex(create_reference_project(cpp_root)).build() -except DocForgeError as error: - if error.code != "optional_dependency_missing": - raise - if error.details.get("install") != "docforge[cpp]": - raise RuntimeError("C++ missing-extra failure was not actionable") from error - cpp_failure = { - "code": error.code, - "install": error.details["install"], - "missing_module": error.details.get("missing_module"), - } -else: - raise RuntimeError("C++ reference build unexpectedly worked without docforge[cpp]") - -print( - json.dumps( - { - "schema_version": 1, - "base_frontend_modules": present_modules, - "base_frontend_distributions": installed_frontends, - "python": { - "build_status": built["status"], - "check_status": checked["status"], - "node_count": checked["node_count"], - }, - "mcp": mcp, - "cpp_without_extra": cpp_failure, - }, - sort_keys=True, - ) -) -""" - - -class AdoptionProofError(RuntimeError): - """The disposable adoption proof could not be completed.""" - - -def _run_checked( - arguments: list[str], - *, - cwd: Path, - timeout: int, - environment: dict[str, str] | None = None, -) -> subprocess.CompletedProcess[str]: - result = subprocess.run( - arguments, - cwd=cwd, - env=environment, - text=True, - capture_output=True, - check=False, - timeout=timeout, - ) - if result.returncode != 0: - detail = result.stderr.strip() or result.stdout.strip() or "no command output" - raise AdoptionProofError( - f"command failed with exit {result.returncode}: {arguments[0]}\n{detail}" - ) - return result - - -def find_wheel(directory: Path) -> Path: - wheels = tuple(sorted(directory.glob("docforge-*.whl"))) - if len(wheels) != 1: - raise AdoptionProofError(f"expected exactly one DocForge wheel, found {len(wheels)}") - return wheels[0] - - -def parse_evidence(raw: str) -> dict[str, object]: - try: - loaded: object = json.loads(raw) - except json.JSONDecodeError as error: - raise AdoptionProofError("fresh-wheel validation returned invalid JSON") from error - if not isinstance(loaded, dict): - raise AdoptionProofError("fresh-wheel validation returned the wrong schema") - document = cast(dict[str, object], loaded) - if document.get("schema_version") != PROOF_SCHEMA_VERSION: - raise AdoptionProofError("fresh-wheel validation returned the wrong schema") - if document.get("base_frontend_modules") != []: - raise AdoptionProofError("fresh-wheel validation found optional frontend modules") - if document.get("base_frontend_distributions") != []: - raise AdoptionProofError("fresh-wheel validation found optional frontend distributions") - raw_python = document.get("python") - raw_mcp = document.get("mcp") - raw_cpp = document.get("cpp_without_extra") - python = cast(dict[str, object], raw_python) if isinstance(raw_python, dict) else None - mcp = cast(dict[str, object], raw_mcp) if isinstance(raw_mcp, dict) else None - cpp = cast(dict[str, object], raw_cpp) if isinstance(raw_cpp, dict) else None - if ( - python is None - or python.get("build_status") != "ok" - or python.get("check_status") != "ok" - or mcp is None - or mcp.get("bootstrap_status") != "ok" - or cpp is None - or cpp.get("code") != "optional_dependency_missing" - or cpp.get("install") != "docforge[cpp]" - ): - raise AdoptionProofError("fresh-wheel validation evidence is incomplete") - return document - - -def run_adoption_proof(repository_root: Path, *, uv: str = "uv") -> dict[str, object]: - """Build and exercise a base wheel without using the network.""" - - root = repository_root.resolve(strict=True) - if not root.joinpath("pyproject.toml").is_file() or not root.joinpath("uv.lock").is_file(): - raise AdoptionProofError("repository root lacks pyproject.toml or uv.lock") - - offline_environment = { - **os.environ, - "UV_OFFLINE": "1", - "UV_PYTHON_DOWNLOADS": "never", - } - offline_environment.pop("PYTHONPATH", None) - with tempfile.TemporaryDirectory(prefix="docforge-m4-adoption-") as directory: - workspace = Path(directory).resolve() - distributions = workspace / "dist" - requirements = workspace / "base-requirements.txt" - virtual_environment = workspace / "venv" - python = virtual_environment / "bin" / "python" - - _run_checked( - [ - uv, - "build", - "--wheel", - "--offline", - "--no-python-downloads", - "--out-dir", - str(distributions), - str(root), - ], - cwd=root, - timeout=COMMAND_TIMEOUT_SECONDS, - environment=offline_environment, - ) - wheel = find_wheel(distributions) - _run_checked( - [ - uv, - "export", - "--frozen", - "--offline", - "--no-dev", - "--no-emit-project", - "--output-file", - str(requirements), - ], - cwd=root, - timeout=COMMAND_TIMEOUT_SECONDS, - environment=offline_environment, - ) - exported = requirements.read_text(encoding="utf-8").casefold() - if any(module.replace("_", "-") in exported for module in OPTIONAL_FRONTEND_MODULES): - raise AdoptionProofError("locked base dependency export contains a language frontend") - _run_checked( - [ - uv, - "venv", - "--python", - sys.executable, - "--no-project", - "--no-python-downloads", - str(virtual_environment), - ], - cwd=workspace, - timeout=COMMAND_TIMEOUT_SECONDS, - environment=offline_environment, - ) - _run_checked( - [ - uv, - "pip", - "sync", - "--python", - str(python), - "--offline", - "--strict", - str(requirements), - ], - cwd=workspace, - timeout=COMMAND_TIMEOUT_SECONDS, - environment=offline_environment, - ) - _run_checked( - [ - uv, - "pip", - "install", - "--python", - str(python), - "--offline", - "--strict", - "--no-deps", - str(wheel), - ], - cwd=workspace, - timeout=COMMAND_TIMEOUT_SECONDS, - environment=offline_environment, - ) - validation = _run_checked( - [str(python), "-I", "-c", _VALIDATION_PROGRAM, str(workspace)], - cwd=workspace, - timeout=VALIDATION_TIMEOUT_SECONDS, - environment=offline_environment, - ) - evidence = parse_evidence(validation.stdout.strip()) - evidence["wheel"] = { - "filename": wheel.name, - "sha256": hashlib.sha256(wheel.read_bytes()).hexdigest(), - } - evidence["network"] = "offline" - return evidence - - -def main() -> None: - parser = argparse.ArgumentParser( - description="Run the offline Milestone 4 fresh-wheel adoption proof" - ) - parser.add_argument( - "--repository-root", - type=Path, - default=Path(__file__).resolve().parents[1], - ) - parser.add_argument("--uv", default=shutil.which("uv") or "uv") - parser.add_argument("--output", type=Path) - arguments = parser.parse_args() - - try: - evidence = run_adoption_proof(arguments.repository_root, uv=arguments.uv) - except (AdoptionProofError, OSError, subprocess.SubprocessError) as error: - print(json.dumps({"status": "error", "error": str(error)}, sort_keys=True)) - raise SystemExit(1) from error - - payload = json.dumps({"status": "ok", **evidence}, indent=2, sort_keys=True) + "\n" - if arguments.output is not None: - arguments.output.write_text(payload, encoding="utf-8") - print(payload, end="") - - -if __name__ == "__main__": - main() diff --git a/tools/milestone4_benchmark.py b/tools/milestone4_benchmark.py deleted file mode 100644 index c6a41c1..0000000 --- a/tools/milestone4_benchmark.py +++ /dev/null @@ -1,498 +0,0 @@ -"""Milestone 4 Python adapter SDK, incremental, and recovery benchmark gates.""" - -from __future__ import annotations - -import argparse -import ast -import gc -import hashlib -import json -import math -import platform -import resource -import statistics -import subprocess -import sys -import tempfile -import time -import tracemalloc -from collections.abc import Callable, Generator, Mapping -from contextlib import contextmanager -from dataclasses import dataclass -from pathlib import Path -from typing import cast -from unittest import mock - -from docforge.adapter_sdk import ( - AdapterAssembly, - AdapterProject, - AdapterSource, - AdapterSourceProjection, -) -from docforge.adapters.python import PythonReferenceAdapter -from docforge.index import ProjectIndex - -ROOT = Path(__file__).resolve().parents[1] -FULL_SOURCE_COUNT = 334 -SMOKE_SOURCE_COUNT = 12 -NODES_PER_SOURCE = 3 -DEFAULT_FULL_SAMPLES = 3 -DEFAULT_SMOKE_SAMPLES = 1 - -# These are regression tripwires, not performance targets. They deliberately -# leave several times the Milestone 3 1,000-node in-process allowances for the -# additional AST, Logic, extraction-cache, and SQLite work in this gate. -MAX_TRACED_PEAK_BYTES = 256 * 1024 * 1024 -MAX_PROCESS_PEAK_BYTES = 512 * 1024 * 1024 -MAX_RESPONSE_BYTES = 256 * 1024 -COLD_LIMIT_MS = 30_000.0 -WARM_LIMIT_MS = 20_000.0 -EQUIVALENCE_LIMIT_MS = 30_000.0 -RECOVERY_LIMIT_MS = 30_000.0 - - -@dataclass(frozen=True) -class _InstrumentedResult: - value: Mapping[str, object] - ast_parse_calls: int - extraction_calls: int - - -class _RecordingPythonAdapter(PythonReferenceAdapter): - """Count extraction entry points without weakening the production adapter.""" - - def __init__(self, root: Path) -> None: - super().__init__( - root, - source_roots=("src",), - project_id="milestone4-python-benchmark", - title="Milestone 4 Python adapter benchmark", - ) - self.extraction_calls = 0 - - def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: - self.extraction_calls += 1 - return super().extract_source(source) - - -def _parser() -> argparse.ArgumentParser: - parser = argparse.ArgumentParser( - description=("Gate DocForge2 Milestone 4 adapter behavior on a disposable Python project.") - ) - parser.add_argument("--mode", choices=("smoke", "full"), default="full") - parser.add_argument("--sources", type=int) - parser.add_argument("--samples", type=int) - parser.add_argument("--output", type=Path) - return parser - - -def encode_report(value: object) -> str: - """Serialize benchmark evidence in one deterministic JSON representation.""" - - return json.dumps(value, sort_keys=True, indent=2, ensure_ascii=False) + "\n" - - -def _compact_json(value: object) -> bytes: - return json.dumps( - value, - sort_keys=True, - separators=(",", ":"), - ensure_ascii=False, - ).encode("utf-8") - - -def _sha256(value: object) -> str: - return hashlib.sha256(_compact_json(value)).hexdigest() - - -def _git(arguments: list[str]) -> str: - return subprocess.run( - ["git", *arguments], - cwd=ROOT, - check=True, - capture_output=True, - text=True, - ).stdout.strip() - - -def write_python_fixture(root: Path, source_count: int) -> None: - """Write a deterministic project with three primary nodes per source.""" - - if source_count < 2: - raise ValueError("The Milestone 4 fixture requires at least two sources") - package = root / "src" / "benchmark" - package.mkdir(parents=True) - for index in range(source_count): - dependency = "" - call = f"value = {index}" - if index: - dependency = f"from benchmark.module_{index - 1:04d} import compute_{index - 1:04d}\n\n" - call = f"value = compute_{index - 1:04d}(value)" - source = ( - f'"""Synthetic adapter benchmark module {index:04d}."""\n\n' - f"{dependency}" - f"def compute_{index:04d}(value: int) -> int:\n" - f' """Return one deterministic branch result."""\n' - f" {call}\n" - " if value % 2:\n" - f" return value + {index + 1}\n" - " return value\n" - ) - (package / f"module_{index:04d}.py").write_text(source, encoding="utf-8") - - -@contextmanager -def _count_ast_parses() -> Generator[mock.MagicMock]: - with mock.patch.object(ast, "parse", wraps=ast.parse) as parsed: - yield parsed - - -def _instrument( - adapter: _RecordingPythonAdapter, - operation: Callable[[], Mapping[str, object]], -) -> _InstrumentedResult: - adapter.extraction_calls = 0 - with _count_ast_parses() as counter: - value = operation() - return _InstrumentedResult( - value=value, - ast_parse_calls=counter.call_count, - extraction_calls=adapter.extraction_calls, - ) - - -def _build_summary(sample: _InstrumentedResult) -> dict[str, object]: - value = sample.value - build = cast(Mapping[str, object], value["build"]) - return { - "status": value["status"], - "revision": value["revision"], - "source_hash": value["source_hash"], - "node_hash": value["node_hash"], - "node_count": value["node_count"], - "edge_hash": value["edge_hash"], - "edge_count": value["edge_count"], - "logic_hash": value["logic_hash"], - "logic_projection_count": value["logic_projection_count"], - "logic_node_count": value["logic_node_count"], - "logic_edge_count": value["logic_edge_count"], - "cache_hits": build["cache_hits"], - "reparsed_sources": build["reparsed_sources"], - "invalidated_sources": build["invalidated_sources"], - "deleted_sources": build["deleted_sources"], - "total_sources": build["total_sources"], - "ast_parse_calls": sample.ast_parse_calls, - "extraction_calls": sample.extraction_calls, - } - - -def _equivalence_summary(sample: _InstrumentedResult) -> dict[str, object]: - return { - **sample.value, - "ast_parse_calls": sample.ast_parse_calls, - "extraction_calls": sample.extraction_calls, - } - - -def _synchronization_summary(sample: _InstrumentedResult) -> dict[str, object]: - synchronization = cast(Mapping[str, object], sample.value["synchronization"]) - build = cast(Mapping[str, object], synchronization["build"]) - return { - "status": sample.value["status"], - "revision": sample.value["revision"], - "source_hash": sample.value["source_hash"], - "node_hash": sample.value["node_hash"], - "node_count": sample.value["node_count"], - "edge_hash": sample.value["edge_hash"], - "edge_count": sample.value["edge_count"], - "logic_hash": sample.value["logic_hash"], - "logic_projection_count": sample.value["logic_projection_count"], - "action": synchronization["action"], - "initial_error_code": cast(Mapping[str, object], synchronization["initial_error"])["code"], - "cache_hits": build["cache_hits"], - "reparsed_sources": build["reparsed_sources"], - "ast_parse_calls": sample.ast_parse_calls, - "extraction_calls": sample.extraction_calls, - } - - -def _measure( - operation: Callable[[], _InstrumentedResult], - *, - samples: int, - p95_limit_ms: float, - summary: Callable[[_InstrumentedResult], Mapping[str, object]], -) -> tuple[dict[str, object], _InstrumentedResult]: - durations: list[float] = [] - traced_peaks: list[int] = [] - response_sizes: list[int] = [] - stable_summary: dict[str, object] | None = None - last: _InstrumentedResult | None = None - for _ in range(samples): - gc.collect() - tracemalloc.start() - started = time.perf_counter_ns() - try: - current = operation() - elapsed_ms = (time.perf_counter_ns() - started) / 1_000_000 - _, traced_peak = tracemalloc.get_traced_memory() - finally: - tracemalloc.stop() - current_summary = dict(summary(current)) - if stable_summary is None: - stable_summary = current_summary - elif current_summary != stable_summary: - raise RuntimeError("Milestone 4 operation changed its deterministic result") - response_bytes = len(_compact_json(current.value)) - if response_bytes > MAX_RESPONSE_BYTES: - raise RuntimeError( - "Milestone 4 response exceeded its fixed byte boundary: " - f"{response_bytes} > {MAX_RESPONSE_BYTES}" - ) - if traced_peak > MAX_TRACED_PEAK_BYTES: - raise RuntimeError( - "Milestone 4 operation exceeded its traced-memory boundary: " - f"{traced_peak} > {MAX_TRACED_PEAK_BYTES}" - ) - durations.append(elapsed_ms) - traced_peaks.append(traced_peak) - response_sizes.append(response_bytes) - last = current - ordered = sorted(durations) - p95_index = max(0, math.ceil(len(ordered) * 0.95) - 1) - p95 = ordered[p95_index] - if p95 > p95_limit_ms: - raise RuntimeError(f"Milestone 4 operation p95 {p95:.3f} ms exceeds {p95_limit_ms:.3f} ms") - assert stable_summary is not None - assert last is not None - return ( - { - "samples": samples, - "median_ms": round(statistics.median(ordered), 3), - "p95_ms": round(p95, 3), - "min_ms": round(ordered[0], 3), - "max_ms": round(ordered[-1], 3), - "p95_limit_ms": p95_limit_ms, - "maximum_response_bytes": max(response_sizes), - "response_limit_bytes": MAX_RESPONSE_BYTES, - "maximum_traced_peak_bytes": max(traced_peaks), - "traced_peak_limit_bytes": MAX_TRACED_PEAK_BYTES, - "stable_result": stable_summary, - }, - last, - ) - - -def _semantic_assembly_hash(assembly: AdapterAssembly) -> str: - projection = assembly.projection - return _sha256( - { - "project_id": projection.project_id, - "title": projection.title, - "adapter_id": projection.adapter_id, - "adapter_version": projection.adapter_version, - "revision": projection.revision, - "source_hash": projection.source_hash, - "nodes": [item.as_dict() for item in projection.nodes], - "edges": [item.as_dict() for item in projection.edges], - "logic": [item.as_dict() for item in assembly.logic], - } - ) - - -def _benchmark(root: Path, source_count: int, samples: int) -> dict[str, object]: - write_python_fixture(root, source_count) - adapter = _RecordingPythonAdapter(root) - cache_root = root / ".docforge" / "benchmark" - project = AdapterProject(adapter, cache_root=cache_root) - index = ProjectIndex(project) - - cold, cold_sample = _measure( - lambda: _instrument(adapter, index.build), - samples=1, - p95_limit_ms=COLD_LIMIT_MS, - summary=_build_summary, - ) - cold_summary = _build_summary(cold_sample) - expected_nodes = source_count * NODES_PER_SOURCE - if ( - cold_summary["node_count"] != expected_nodes - or cold_summary["logic_projection_count"] != source_count - or cold_summary["reparsed_sources"] != source_count - or cast(int, cold_summary["ast_parse_calls"]) < source_count - or cold_summary["extraction_calls"] != source_count - ): - raise RuntimeError("Cold adapter build did not cover the complete synthetic fixture") - - warm, warm_sample = _measure( - lambda: _instrument(adapter, index.build), - samples=samples, - p95_limit_ms=WARM_LIMIT_MS, - summary=_build_summary, - ) - warm_summary = _build_summary(warm_sample) - if ( - warm_summary["cache_hits"] != source_count - or warm_summary["reparsed_sources"] != 0 - or warm_summary["ast_parse_calls"] != 0 - or warm_summary["extraction_calls"] != 0 - ): - raise RuntimeError("Warm adapter build performed forbidden parsing or extraction") - - equivalence, equivalence_sample = _measure( - lambda: _instrument(adapter, project.verify_incremental_equivalence), - samples=1, - p95_limit_ms=EQUIVALENCE_LIMIT_MS, - summary=_equivalence_summary, - ) - equivalence_summary = _equivalence_summary(equivalence_sample) - if ( - equivalence_summary["status"] != "ok" - or equivalence_summary["node_count"] != expected_nodes - or equivalence_summary["logic_projection_count"] != source_count - ): - raise RuntimeError("Full and incremental graph plus Logic output was not equivalent") - complete_assembly = adapter.load_complete_assembly() - assembly_hash = _semantic_assembly_hash(complete_assembly) - - extraction_cache = cache_root / "extractions.json" - extraction_cache.write_bytes(b"{broken") - cache_recovery, cache_recovery_sample = _measure( - lambda: _instrument(adapter, index.build), - samples=1, - p95_limit_ms=RECOVERY_LIMIT_MS, - summary=_build_summary, - ) - cache_recovery_summary = _build_summary(cache_recovery_sample) - if ( - cache_recovery_summary["reparsed_sources"] != source_count - or cast(int, cache_recovery_summary["ast_parse_calls"]) < source_count - or cache_recovery_summary["node_hash"] != cold_summary["node_hash"] - or cache_recovery_summary["logic_hash"] != cold_summary["logic_hash"] - ): - raise RuntimeError("Corrupt extraction-cache recovery changed adapter output") - - index.path.write_bytes(b"not-a-sqlite-index") - index_recovery, index_recovery_sample = _measure( - lambda: _instrument(adapter, index.synchronize), - samples=1, - p95_limit_ms=RECOVERY_LIMIT_MS, - summary=_synchronization_summary, - ) - index_recovery_summary = _synchronization_summary(index_recovery_sample) - if ( - index_recovery_summary["action"] != "rebuilt" - or index_recovery_summary["cache_hits"] != source_count - or index_recovery_summary["reparsed_sources"] != 0 - or index_recovery_summary["ast_parse_calls"] != 0 - or index_recovery_summary["extraction_calls"] != 0 - or index_recovery_summary["node_hash"] != cold_summary["node_hash"] - or index_recovery_summary["logic_hash"] != cold_summary["logic_hash"] - ): - raise RuntimeError("Corrupt derived-index recovery changed adapter output") - - evidence = { - "source_count": source_count, - "node_count": expected_nodes, - "edge_count": cold_summary["edge_count"], - "logic_projection_count": source_count, - "logic_node_count": cold_summary["logic_node_count"], - "logic_edge_count": cold_summary["logic_edge_count"], - "source_hash": cold_summary["source_hash"], - "node_hash": cold_summary["node_hash"], - "edge_hash": cold_summary["edge_hash"], - "logic_hash": cold_summary["logic_hash"], - "complete_assembly_hash": assembly_hash, - "full_incremental_graph_and_logic_exact": True, - "warm_zero_ast_parse": True, - "warm_zero_source_extraction": True, - "cache_recovery_exact": True, - "index_recovery_exact": True, - } - return { - "fixture": { - "kind": "synthetic_python_reference_adapter", - "source_count": source_count, - "nodes_per_source": NODES_PER_SOURCE, - "expected_node_count": expected_nodes, - }, - "operations": { - "cold_incremental_build": cold, - "warm_incremental_build": warm, - "full_incremental_equivalence": equivalence, - "corrupt_extraction_cache_recovery": cache_recovery, - "corrupt_index_recovery": index_recovery, - }, - "evidence": evidence, - "evidence_sha256": _sha256(evidence), - } - - -def main() -> int: - arguments = _parser().parse_args() - default_sources = FULL_SOURCE_COUNT if arguments.mode == "full" else SMOKE_SOURCE_COUNT - default_samples = DEFAULT_FULL_SAMPLES if arguments.mode == "full" else DEFAULT_SMOKE_SAMPLES - source_count = default_sources if arguments.sources is None else arguments.sources - samples = default_samples if arguments.samples is None else arguments.samples - if not 2 <= source_count <= FULL_SOURCE_COUNT: - raise SystemExit(f"--sources must be between 2 and {FULL_SOURCE_COUNT}") - if arguments.mode == "full" and source_count != FULL_SOURCE_COUNT: - raise SystemExit(f"--mode full requires exactly {FULL_SOURCE_COUNT} sources") - if samples < 1: - raise SystemExit("--samples must be positive") - - with tempfile.TemporaryDirectory(prefix="docforge-milestone4-") as directory: - measurement = _benchmark(Path(directory).resolve(), source_count, samples) - - process_peak_bytes = int(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss) * 1024 - if process_peak_bytes > MAX_PROCESS_PEAK_BYTES: - raise RuntimeError( - "Milestone 4 process peak exceeded its memory boundary: " - f"{process_peak_bytes} > {MAX_PROCESS_PEAK_BYTES}" - ) - status = _git(["status", "--porcelain"]) - report: dict[str, object] = { - "schema_version": 1, - "benchmark": "docforge2_milestone4", - "mode": arguments.mode, - "source": { - "revision": _git(["rev-parse", "HEAD"]), - "dirty": bool(status), - }, - "environment": { - "platform": platform.platform(), - "machine": platform.machine(), - "python": platform.python_version(), - "implementation": platform.python_implementation(), - }, - "method": { - "clock": "time.perf_counter_ns", - "in_process_peak_memory": "tracemalloc per measured invocation", - "process_peak_memory": "resource.getrusage(RUSAGE_SELF).ru_maxrss", - "response_size": "UTF-8 bytes of compact sorted JSON", - "ast_instrumentation": "temporary counter around stdlib ast.parse", - "extraction_instrumentation": "reference adapter extract_source entry counter", - "samples": samples, - "full_mode_source_requirement": FULL_SOURCE_COUNT, - "threshold_basis": ( - "Regression tripwires leave several times the Milestone 3 1,000-node " - "allowances for AST, Logic, extraction-cache, and SQLite work." - ), - }, - **measurement, - "memory": { - "process_peak_bytes": process_peak_bytes, - "process_peak_limit_bytes": MAX_PROCESS_PEAK_BYTES, - }, - } - encoded = encode_report(report) - if arguments.output is not None: - output = arguments.output.resolve() - output.parent.mkdir(parents=True, exist_ok=True) - output.write_text(encoded, encoding="utf-8") - sys.stdout.write(encoded) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/uv.lock b/uv.lock index de8ab13..21b0bfd 100644 --- a/uv.lock +++ b/uv.lock @@ -211,26 +211,9 @@ source = { editable = "." } dependencies = [ { name = "markdown-it-py" }, { name = "mcp" }, -] - -[package.optional-dependencies] -cpp = [ - { name = "tree-sitter" }, - { name = "tree-sitter-cpp" }, -] -javascript = [ - { name = "tree-sitter" }, - { name = "tree-sitter-javascript" }, -] -languages = [ { name = "tree-sitter" }, { name = "tree-sitter-cpp" }, { name = "tree-sitter-javascript" }, - { name = "tree-sitter-typescript" }, -] -typescript = [ - { name = "tree-sitter" }, - { name = "tree-sitter-typescript" }, ] [package.dev-dependencies] @@ -238,38 +221,22 @@ dev = [ { name = "jsonschema" }, { name = "pytest" }, { name = "ruff" }, - { name = "tree-sitter" }, - { name = "tree-sitter-cpp" }, - { name = "tree-sitter-javascript" }, - { name = "tree-sitter-typescript" }, ] [package.metadata] requires-dist = [ { name = "markdown-it-py", specifier = ">=4.2,<5" }, { name = "mcp", specifier = ">=1.28,<2" }, - { name = "tree-sitter", marker = "extra == 'cpp'", specifier = ">=0.25,<0.26" }, - { name = "tree-sitter", marker = "extra == 'javascript'", specifier = ">=0.25,<0.26" }, - { name = "tree-sitter", marker = "extra == 'languages'", specifier = ">=0.25,<0.26" }, - { name = "tree-sitter", marker = "extra == 'typescript'", specifier = ">=0.25,<0.26" }, - { name = "tree-sitter-cpp", marker = "extra == 'cpp'", specifier = ">=0.23,<0.24" }, - { name = "tree-sitter-cpp", marker = "extra == 'languages'", specifier = ">=0.23,<0.24" }, - { name = "tree-sitter-javascript", marker = "extra == 'javascript'", specifier = ">=0.25,<0.26" }, - { name = "tree-sitter-javascript", marker = "extra == 'languages'", specifier = ">=0.25,<0.26" }, - { name = "tree-sitter-typescript", marker = "extra == 'languages'", specifier = ">=0.23,<0.24" }, - { name = "tree-sitter-typescript", marker = "extra == 'typescript'", specifier = ">=0.23,<0.24" }, + { name = "tree-sitter", specifier = ">=0.25,<0.26" }, + { name = "tree-sitter-cpp", specifier = ">=0.23,<0.24" }, + { name = "tree-sitter-javascript", specifier = ">=0.25,<0.26" }, ] -provides-extras = ["javascript", "typescript", "cpp", "languages"] [package.metadata.requires-dev] dev = [ { name = "jsonschema", specifier = ">=4.25,<5" }, { name = "pytest", specifier = ">=9.1,<10" }, { name = "ruff", specifier = ">=0.15,<1" }, - { name = "tree-sitter", specifier = ">=0.25,<0.26" }, - { name = "tree-sitter-cpp", specifier = ">=0.23,<0.24" }, - { name = "tree-sitter-javascript", specifier = ">=0.25,<0.26" }, - { name = "tree-sitter-typescript", specifier = ">=0.23,<0.24" }, ] [[package]] @@ -837,21 +804,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/2e/1f/f9eba1038b7d4394410f3c0a6ec2122b590cd7acb03f196e52fa57ebbe72/tree_sitter_javascript-0.25.0-cp310-abi3-win_arm64.whl", hash = "sha256:622a69d677aa7f6ee2931d8c77c981a33f0ebb6d275aa9d43d3397c879a9bb0b", size = 61668, upload-time = "2025-09-01T07:13:43.803Z" }, ] -[[package]] -name = "tree-sitter-typescript" -version = "0.23.2" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/1e/fc/bb52958f7e399250aee093751e9373a6311cadbe76b6e0d109b853757f35/tree_sitter_typescript-0.23.2.tar.gz", hash = "sha256:7b167b5827c882261cb7a50dfa0fb567975f9b315e87ed87ad0a0a3aedb3834d", size = 773053, upload-time = "2024-11-11T02:36:11.396Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/28/95/4c00680866280e008e81dd621fd4d3f54aa3dad1b76b857a19da1b2cc426/tree_sitter_typescript-0.23.2-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:3cd752d70d8e5371fdac6a9a4df9d8924b63b6998d268586f7d374c9fba2a478", size = 286677, upload-time = "2024-11-11T02:35:58.839Z" }, - { url = "https://files.pythonhosted.org/packages/8f/2f/1f36fda564518d84593f2740d5905ac127d590baf5c5753cef2a88a89c15/tree_sitter_typescript-0.23.2-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:c7cc1b0ff5d91bac863b0e38b1578d5505e718156c9db577c8baea2557f66de8", size = 302008, upload-time = "2024-11-11T02:36:00.733Z" }, - { url = "https://files.pythonhosted.org/packages/96/2d/975c2dad292aa9994f982eb0b69cc6fda0223e4b6c4ea714550477d8ec3a/tree_sitter_typescript-0.23.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4b1eed5b0b3a8134e86126b00b743d667ec27c63fc9de1b7bb23168803879e31", size = 351987, upload-time = "2024-11-11T02:36:02.669Z" }, - { url = "https://files.pythonhosted.org/packages/49/d1/a71c36da6e2b8a4ed5e2970819b86ef13ba77ac40d9e333cb17df6a2c5db/tree_sitter_typescript-0.23.2-cp39-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e96d36b85bcacdeb8ff5c2618d75593ef12ebaf1b4eace3477e2bdb2abb1752c", size = 344960, upload-time = "2024-11-11T02:36:04.443Z" }, - { url = "https://files.pythonhosted.org/packages/7f/cb/f57b149d7beed1a85b8266d0c60ebe4c46e79c9ba56bc17b898e17daf88e/tree_sitter_typescript-0.23.2-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:8d4f0f9bcb61ad7b7509d49a1565ff2cc363863644a234e1e0fe10960e55aea0", size = 340245, upload-time = "2024-11-11T02:36:06.473Z" }, - { url = "https://files.pythonhosted.org/packages/8b/ab/dd84f0e2337296a5f09749f7b5483215d75c8fa9e33738522e5ed81f7254/tree_sitter_typescript-0.23.2-cp39-abi3-win_amd64.whl", hash = "sha256:3f730b66396bc3e11811e4465c41ee45d9e9edd6de355a58bbbc49fa770da8f9", size = 278015, upload-time = "2024-11-11T02:36:07.631Z" }, - { url = "https://files.pythonhosted.org/packages/9f/e4/81f9a935789233cf412a0ed5fe04c883841d2c8fb0b7e075958a35c65032/tree_sitter_typescript-0.23.2-cp39-abi3-win_arm64.whl", hash = "sha256:05db58f70b95ef0ea126db5560f3775692f609589ed6f8dd0af84b7f19f1cbb7", size = 274052, upload-time = "2024-11-11T02:36:09.514Z" }, -] - [[package]] name = "typing-extensions" version = "4.16.0"