diff --git a/ACTIVE_SLICE.md b/ACTIVE_SLICE.md index 71c0e5e..3dbf6b3 100644 --- a/ACTIVE_SLICE.md +++ b/ACTIVE_SLICE.md @@ -6,14 +6,7 @@ Goal: Let one project-bound server return compact, task-shaped, explainable cont In scope: Capability modes; capability-aware bootstrap; versioned retrieval plans and context capsules; task-shaped context; generation diffs; evidence-gap diagnostics; generated client configuration; doctor checks. Out of scope: Independent render-plan packages; adapter SDK expansion; self-hosting; storage replacement; embeddings; WorldForge or ScrapeStation changes; production MCP repointing; tags and releases. Done when: Policy and capabilities are explicit; bootstrap recommends only available actions; task context is compact, deterministic, provenance-bearing, and bounded; generation and evidence gaps are explainable; generated configuration and doctor checks are safe and tested; the complete repository gate and Milestone 2 benchmark pass. -Status: Complete. Effective policy, versioned task retrieval, latest-generation diff receipts, -logarithmic bounded page packing, deterministic client configuration, and the read-only integration -doctor are implemented and contract-tested. The complete repository gate passes with 205 tests and -120 subtests. Three independent adversarial audits found no remaining implementation blocker. The -clean 1,000-node baseline is recorded against candidate commit -`fb0df5e4a1c591c2a84788fd4814d98550f11863`, including task/generation reconstruction, -response-size behavior, zero-hidden-work counters, and isolated memory. No tag or release was -created, no production integration was repointed, and self-hosting remains out of scope. +Status: Active. Read-only contract audits begin from the verified Milestone 1 boundary. ``` Milestones 3–5 remain directional context and are not active. diff --git a/DEVELOPMENT_NOTES.md b/DEVELOPMENT_NOTES.md index 8e15aea..d57fc78 100644 --- a/DEVELOPMENT_NOTES.md +++ b/DEVELOPMENT_NOTES.md @@ -344,269 +344,3 @@ These are notes, not commitments: - Cursor authentication remains deliberately absent. If read cursors ever carry authority rather than bounded positions, they will need a different versioned security contract and persisted key lifecycle. - -## Milestone 2 — active: agent retrieval and MCP experience - -### Audit reconciliation - -Three independent read-only audits covered effective policy and bootstrap, task-shaped retrieval -and context capsules, and generation diffs plus client configuration and doctor checks. - -They agreed on these boundaries: - -- Keep the project descriptor at schema version 1. Process capability and client configuration are - machine-specific bindings, not canonical project content. -- Preserve the legacy adapter-policy payload, no-AST shorthand, tool names, default tool ordering, - one-method adapters, and custom context provider. -- Add one versioned effective-policy authority and derive bootstrap, contract, instructions, and - access reporting from it. -- Add one task-context operation with a closed task-kind vocabulary and one immutable, - generation-pinned retrieval plan. Do not create a tool for every task kind. -- Produce evidence gaps only from declared plan requirements and completed bounded checks. Never - infer missing facts from arbitrary project naming. -- Record only the latest bounded generation transition as disposable evidence. Do not add a - history database. -- Preview client configuration by default. Any write must be explicit, atomic, merge-preserving, - and backed by a verified client-format driver. -- Keep doctor strictly read-only. It must not bootstrap, synchronize, build, render, start a - viewer, or rewrite client configuration. - -### Versioned effective policy and session contract - -The binding now composes an immutable version-1 policy containing capability mode, adapter -evolution, AST and Logic behavior, synchronization and integrity levels, render and viewer -behavior, profiling, blocked tools, prohibitions, and explicit precedence. `--no-ast` is a -restrictive override. The exact legacy `adapter_policy` response remains a projection of the new -object. - -Bootstrap reuses the identity already proven by synchronization and no longer reloads the complete -project. Its additive version-1 session contract reports binding, generation, effective policy, -actual registered surfaces and mutation access, render policies, first operation, filtered -workflow, and prohibitions. Read mode does not recommend proposals. Proposal mode recommends -registration and review only with writer access. Application is recommended only when the -exact-hash applier is enabled. - -Existing factory defaults and tool order remain unchanged. Explicit application mode fails closed -without an applier. Operator mode is reserved and currently adds no tools. - -### Versioned task retrieval and context capsules - -The first Milestone 2 retrieval slice adds one `docforge_get_task_context` read tool rather than a -family of task-specific tools. Its closed task kinds are change, implementation, failure, -ownership, test, operation, and release. One immutable `RetrievalPlanV1` derives exact or lexical -focus, bounded bidirectional graph traversal, metadata hydration, required evidence categories, -and fixed work budgets from the project descriptor and effective policy. - -The public executor re-derives every submitted plan before opening SQLite. It rejects modified -steps, task identity, requirements, category order, bounds, policy identity, or hashes as -`invalid_retrieval_plan`. Traversal binds the project relation set by canonical hash and queries -the already-validated edge table by endpoint, avoiding relation-sized SQL parameter lists. -Version-1 internal ceilings are 1,000 evidence items, 100,000 candidate edges, and 10,000 task -query characters. - -The executor uses one immutable SQLite read generation. It rejects missing explicit focus, blocks -unresolved or tied lexical focus, stops at deterministic evidence and candidate-edge limits, and -checks source identity again when the transaction closes. `ContextCapsuleV1` binds the generation, -policy, request, plan, evidence collection, and complete capsule with canonical hashes. - -Project relation names remain authoritative. The core recognizes only a versioned alias map for -structure, implementation, dependency, execution, data, evidence, and context. Unknown allowed -relations stay visible under their raw names as `unclassified`. Required evidence diagnostics -distinguish categories the project never declared, completed bounded checks with no selected -evidence, and incomplete checks caused by a result, work, token, or response limit. - -Each evidence item carries a stable content hash, confined source identity, shortest selected graph -path, every additional qualifying relationship reason observed during traversal, and explicit -limitations where the current graph cannot prove evidence type, extractor identity, relationship -source provenance, or observation time. The planner contains no Logic operation, so no-AST -bindings can use task context without weakening their existing Logic prohibition. - -Path direction is relative to the preceding traversal node. Additional relationship reasons use -the evidence node as their direction subject. Candidate-edge and unclassified-relation ceilings -produce explicit omissions and bounded summaries. - -MCP pagination preserves the complete plan, collection, and capsule hashes while returning bounded -pages. Its cursor additionally binds the effective policy and task request. An individually -oversized item advances once as a hash-identified omission. A later generation or policy change -fails closed as `stale_cursor`. - -The legacy profile-context contract remains intact. A custom context provider does not silently -gain core task planning. Version 1 defines no custom task-planner extension, so the additive tool -returns `task_context_unavailable` without synchronization or a complete projection load. - -Two independent pre-commit audits reproduced and closed plan-forgery, relation-sized SQL, -SQLite-parameter portability, ambiguous relationship-direction, missing work-limit evidence, -schema/runtime drift, incomplete page hashing, and custom-provider hidden-load defects. Regression -coverage includes 33,005 valid relation names, fixed extreme project limits, tampered plans, -evidence-relative directions, edge and unclassified limits, schema-valid pages, changed cursor -semantics, oversized evidence advancement, no-AST retrieval, and legacy complete-projection -adapters. - -The complete repository gate passes with 158 tests and 101 subtests, zero Pyright diagnostics, -warning-strict execution, package builds, public-contract validation, and the maintained Milestone -0 and Milestone 1 smoke benchmarks. Gitleaks 8.30.1 reports no secret findings in the working tree. - -### Latest-generation diff receipt - -Three read-only audits reconciled the index publication, public transport, compatibility, and -no-AST boundaries before implementation. The selected design stores one disposable -`generation-diff.json` receipt. It does not add a history database, arbitrary generation -selectors, source text, rendered content, or Logic details. - -Before a build loads current source, it accepts an existing index only when its exact main-file -inode has a matching stable whole-file attestation and no WAL, journal, or shared-memory sidecar. -It then captures that predecessor through an immutable main-file transaction. The capture validates -the SQLite application and schema IDs, project/root/adapter binding, integrity, complete node and -edge rows, Logic aggregate identity, FTS count, metadata hashes and counts, and final file -signature. It never calls normal check or synchronization and never repairs predecessor evidence. - -The final source revalidation now compares exact nodes and edges in addition to source hash, -revision, and Logic. A verified predecessor that maps the same source identity to different graph -content fails before publication as `generation_collision`. This closes a pre-existing adapter -determinism gap found during the generation-diff audit. - -SQLite replacement is now the explicit derived mutation commit point. Whole-file attestation, -cheap source-generation, and generation-diff receipts publish independently afterward. Any -post-commit receipt failure returns `status = ok`, `index = published`, a bounded degraded -publication record, and receipt-stage names. It never rolls back the new index or reports a false failed -mutation. Attestation hashing checks the exact index signature before, during, and immediately -before receipt publication. - -Version-1 diff semantics compare every core `Node` field by stable node ID and exact edge triples. -Node renames are removal plus addition. Edge changes are removal plus addition. Exact summary -counts and a full ordered item-hash collection cover every change. Retained details are -deterministically ordered and independently capped at 1,000 items and 1 MiB with explicit item- or -byte-limit evidence. A first build or untrusted predecessor is a baseline with no fabricated -all-added result. A same-generation reindex republishes the existing meaningful transition against -the new index file identity instead of erasing it with an empty diff. - -The additive public surfaces are: - -- CLI `generation-diff [--limit N] [--cursor OPAQUE]`. -- MCP `docforge_get_generation_diff(limit=None, cursor=None)`. -- Telemetry operations `cli.generation-diff` and `mcp.generation_diff`. - -Public reads do not open SQLite, call `project.load()`, extract an adapter projection, parse source, -check, synchronize, build, or repair. They strictly validate the bounded receipt, compare stable -receipt and index file identities, require two matching cheap source-generation checks, and report -unknown for legacy adapters without that capability. Missing, corrupt, foreign, oversized, -symlinked, stale, or concurrently changed evidence remains a read-only status outcome. - -Generation-diff pagination binds the complete stored receipt hash and effective policy. That hash -already covers project, generation, graph, collection, and committed-index identity. Page size may -change. A replaced receipt returns `stale_cursor`. One top-level pagination object owns the only -cursor. The nested version-1 page uses `receipt_header.stored_receipt_hash` so it never -misrepresents the complete receipt hash as the hash of a partial header. The summary distinguishes -additional retained pages from details permanently omitted by the fixed publication limits. - -Adversarial coverage now includes strict runtime/schema rejection, predecessor attestation and -generation identity, live and synthetic SQLite sidecars, cache-root symlink substitution, -source/sidecar changes during diff preparation, independent receipt failures, and degraded -post-commit identity and durability failures. Focused verification passes the direct, CLI, MCP, -schema, pagination, legacy, incremental no-AST, and zero-work suites. The complete repository gate -passes with 176 tests and 113 subtests, zero Pyright diagnostics, package builds, web checks, and -the maintained Milestone 0 and Milestone 1 smoke benchmarks. Final independent re-audit is in -progress before this slice is committed. - -The final dense benchmark uses a 1,000-node transition with 1,000 changed details. Its receipt is -775,663 bytes. Direct status is 37.659 ms median and 39.108 ms p95. A maximum-size MCP request -returns 307 items in 199,754 bytes at 54.037 ms median and 56.617 ms p95. Four pages reconstruct -all 1,000 retained details in 652,798 bytes at 201.55 ms median. Peak RSS is 79,096 KiB. -Every hidden-work counter remains zero; the read performs exactly two cheap source-generation -checks. - -Measurement found and removed two avoidable costs before commit. Receipt loading had repeated the -complete 1,000-item validator solely to check project identity; it now validates once and compares -the three binding fields directly. Page fitting had encoded every growing prefix; it now uses an -exact logarithmic search and retains the hash-only oversized-item omission path. The maximum page -fell from 272.06 ms p95 to 56.617 ms p95, while full traversal fell from roughly 859 ms to -203.55 ms p95. Regression tests require one receipt validation and at most 15 response encodes for -1,000 page candidates. Final independent publication, contract, and performance audits approve -the slice for commit. - -### Deterministic client configuration and read-only doctor - -Client integration remains an explicit machine-local boundary rather than canonical project -content. `docforge configure {codex,claude,openclaw} --project PATH` previews a deterministic -version-1 fragment by default. An optional output path publishes only a standalone fragment into -an existing real directory. Publication is create-only, private-mode, no-follow, bounded, and -conflict-aware. Existing differing client configuration is never merged, replaced, or silently -overwritten. - -Generated commands use the exact current virtual-environment Python executable with isolated -module startup. The binding records explicit read, proposal, or application mode, no-AST policy, -render policy, empty environment, and bounded timeouts. Proposal and application generation fail -closed unless the descriptor declares the required writer and matching applier identity. A generic -CLI cannot reconstruct project-owned adapter composition, so custom adapters return an explicit -unavailable result instead of generating a misleading command. - -Codex and OpenClaw fragments include their verified timeout fields. Claude JSON fragment syntax is -supported, while its timeout representation remains an explicit warning. The configuration result -has a strict JSON schema and canonical plan hash. Diagnostics are additive and remain disabled by -default. - -`docforge doctor --client CLIENT` performs bounded, non-mutating inspection only. It reads the -project descriptor and selected client file through stable, directory-bound, no-follow handles; -parses at most 1 MiB and 256 server entries; selects at most one exact project binding; validates -the closed server argument set; checks executable, capability, declared authority, no-AST, -timeouts, environment-key names, and tool-filter presence; and performs only a stat-level index -presence check. It never loads canonical sources, opens SQLite, starts MCP, executes the configured -command, synchronizes, builds, renders, starts a viewer, or writes client configuration. - -Doctor reports healthy, degraded, or unhealthy with stable process exit codes 0, 1, and 2. Secret -environment values are parsed only to enforce bounded string limits and are never returned. -Unknown or unverified client tool filtering, Claude timeout representation, implicit legacy -capability mode, shadowed authority, and missing disposable indexes are warnings. Unsafe paths, -malformed matching entries, unexpected executables, wrong project roots, invalid authorities, and -missing configuration are failures. - -The first benchmark smoke failed for the correct product reason: its disposable doctor fragment -used the shared path `/tmp/doctor-codex.toml`, where a previous run had left different content. The -harness now creates a project subdirectory inside one unique temporary root and places the client -fragment beside it. This preserves create-only conflict safety and makes every run disposable. - -The final pre-commit 1,000-node audit sample passes every provisional Milestone 2 gate. Task -context reconstructs 1,000 candidates as 108 cited evidence records and 892 explicit bounded -omissions across 11 pages in 703.808 ms. Generation diff reconstructs 1,000 changed details across -10 pages in 427.450 ms. Maximum pages remain below the 200,000-byte MCP budget; generation diff -uses 199,566 bytes and proves that diagnostics are discarded before the primary result. Isolated -peak RSS is 86,168 KiB. - -Configuration preview now includes a bounded real import probe of the exact isolated interpreter, -so its provisional single-sample latency is about 315 ms rather than the earlier sub-millisecond -derivation-only figure. Doctor remains below 1 ms on generated disposable configurations. -Every configuration and doctor hidden-work counter is zero. - -The aggregate `make gate` includes the Milestone 2 smoke benchmark. The frozen candidate passes -205 tests and 120 schema subtests, strict warnings, Ruff, formatting, Pyright, web checks, -compilation, lock and dependency checks, package builds, and all three milestone smoke benchmarks. -Three independent final audits approve client publication and policy binding, doctor fail-closed -behavior, and benchmark/contract coverage. Clean-revision benchmark evidence is still required -before closeout. - -### Milestone 2 closeout - -Candidate commit `fb0df5e4a1c591c2a84788fd4814d98550f11863` passed the clean ten-sample -Milestone 2 benchmark. Task-context complete traversal measured 703.561 ms median and 721.847 ms -p95 across 11 bounded pages. It reconstructed the exact 1,000-candidate collection from 108 cited -evidence records, 891 original token-budget omissions, and one hash-attested response-limit -surrogate. Generation-diff complete traversal measured 418.607 ms median and 425.315 ms p95 across -10 pages. - -Read and no-AST bootstrap remained below 10 ms p95. The maximum generation page used 199,566 bytes -of the 200,000-byte budget and correctly discarded diagnostics before primary evidence. -Configuration preview measured about 314 ms median and 365 ms p95 because it proves the real -isolated interpreter import on every invocation. Codex and OpenClaw doctor checks remained below -0.6 ms p95; Claude remained explicitly degraded because its timeout format is unverified. -Isolated-process peak RSS was 86,448 KiB against the 262,144 KiB gate. - -All measured configuration and doctor counters were zero. Task-context pages performed one index -check and two cheap generation checks with no loads, parses, synchronization, builds, extraction, -rendering, or viewer work. Generation-diff pages performed two cheap generation checks and no -index check. The canonical machine-readable result is -`benchmarks/milestone2-2026-07-29.json`. - -Milestone 2 is complete. Follow-up ideas stay explicitly later-scope: avoid recomputing the -task-shaped capsule for every continuation page, add authenticated continuation when the threat -model requires it, verify a native Claude timeout representation, and introduce adapter-owned -launcher metadata before generating configurations for custom adapters. diff --git a/Makefile b/Makefile index ab909e4..572cf59 100644 --- a/Makefile +++ b/Makefile @@ -5,7 +5,7 @@ NPM := npm PYTHONPYCACHEPREFIX := /tmp/docforge-quality-pycache PYTEST_BASETEMP := /tmp/docforge-quality-pytest -.PHONY: benchmark benchmark-m1 benchmark-m1-smoke benchmark-m2 benchmark-m2-smoke benchmark-smoke build compile contract dependencies format-check gate lint lock test type +.PHONY: benchmark benchmark-m1 benchmark-m1-smoke benchmark-smoke build compile contract dependencies format-check gate lint lock test type format-check: $(PYTHON) -m ruff format --check src tests tools @@ -24,11 +24,6 @@ contract: PYTHONPYCACHEPREFIX=$(PYTHONPYCACHEPREFIX) $(PYTHON) -m pytest -q \ -p no:cacheprovider --basetemp=$(PYTEST_BASETEMP) \ tests/test_public_contract.py \ - tests/test_policy.py \ - tests/test_retrieval.py \ - tests/test_generation_diff.py \ - tests/test_client_integration.py \ - tests/test_observability.py::TelemetryContractTests::test_schema_fixed_names_match_the_implementation \ tests/test_adapter_contract.py::AdapterContractTests::test_no_ast_index_policy_rejects_logic_publication \ tests/test_adapter_contract.py::AdapterContractTests::test_no_ast_index_accepts_legacy_and_non_logic_incremental_adapters \ tests/test_adapter_contract.py::AdapterContractTests::test_no_ast_rejects_preexisting_logic_index_and_viewer_snapshot \ @@ -61,11 +56,4 @@ benchmark-m1-smoke: benchmark-m1: $(PYTHON) tools/milestone1_benchmark.py --nodes 1000 --samples 10 -benchmark-m2-smoke: - $(PYTHON) tools/milestone2_benchmark.py --nodes 25 --samples 1 \ - --output /tmp/docforge-milestone2-smoke.json > /dev/null - -benchmark-m2: - $(PYTHON) tools/milestone2_benchmark.py --nodes 1000 --samples 10 - -gate: format-check lint type compile contract test lock dependencies build benchmark-smoke benchmark-m1-smoke benchmark-m2-smoke +gate: format-check lint type compile contract test lock dependencies build benchmark-smoke benchmark-m1-smoke diff --git a/README.md b/README.md index 16a03a6..1fe2f62 100644 --- a/README.md +++ b/README.md @@ -9,14 +9,6 @@ declared manuals, visualizes project structure, and manages reviewable documenta - Validates stable Markdown/TOML nodes and typed relationships. - Builds a deterministic SQLite search and graph index. - Exposes project-bound CLI and MCP query surfaces. -- Compiles versioned, generation-bound task context with cited evidence, explicit gaps, and bounded - continuation. -- Records one bounded, versioned latest-generation graph transition without creating a history - database. -- Generates deterministic project-bound Codex, Claude, and OpenClaw client fragments without - copying ambient secrets. -- Diagnoses one client binding through bounded read-only checks without starting MCP or rebuilding - project state. - Automatically synchronizes disposable indexes before MCP work. - Creates, validates, diffs, and previews isolated changesets. - Registers complete proposals atomically without caller-managed hash chaining. @@ -56,23 +48,6 @@ AST, Tree-sitter, compiler-AST, or function-Logic extraction, blocks the Logic t nonempty Logic publication. Complete-projection adapters continue unchanged, and non-AST incremental fingerprinting and caching remain allowed. -DocForge2 bindings may also declare -`--capability-mode read|proposal|application|operator`. Bootstrap returns one versioned effective -policy and the actual startup-gated capabilities. Existing tool surfaces and the legacy no-AST -payload remain compatible. - -`docforge_get_task_context` is an additive read tool for `change`, `implementation`, `failure`, -`ownership`, `test`, `operation`, and `release` work. It derives a closed version-1 retrieval plan, -executes it against one immutable index generation, and returns a hash-bound context capsule. -Project relation names remain authoritative. DocForge classifies only its versioned alias set and -preserves every unknown relation as `unclassified` instead of guessing semantics. - -`docforge_get_generation_diff` reports the latest verified primary-graph transition through one -bounded disposable receipt. It includes exact node and edge change counts, hash-bound retained -details, and explicit truncation. Paged results use one top-level cursor and a versioned -`receipt_header`; `stored_receipt_hash` identifies the complete persisted receipt. The read never -exposes Logic details, loads canonical source, repairs derived state, or invents history. - ## Graph views The browser presents the primary architecture graph through three complementary views and loads a @@ -139,16 +114,6 @@ Start an MCP server for one project: Add `--canonical-applier project-editor` only when that MCP integration should expose the hash-bound `docforge_apply_changeset` tool. -Preview a read-only Codex fragment and diagnose an installed binding: - -```bash -.venv/bin/docforge configure codex --project "$PROJECT" -.venv/bin/docforge doctor --client codex --project "$PROJECT" -``` - -Pass `--output /absolute/path/docforge.toml` only when creating a standalone fragment. DocForge -never replaces or merges an existing different client file. - For an unconfigured codebase, begin with a read-only language and documentation assessment: ```bash @@ -170,10 +135,6 @@ DocForge describes them as a source graph. performance, memory, rendering and response sizes, bottlenecks, and missing coverage. - [Milestone 0 closeout](docs/MILESTONE_0_CLOSEOUT.md) — lineage, migration, security scan, repository state, and fresh-clone proof. -- [Milestone 2 baseline](docs/MILESTONE_2_BASELINE.md) — task context, generation diff, client - configuration, doctor, response-size, counter, and memory measurements. -- [Milestone 2 closeout](docs/MILESTONE_2_CLOSEOUT.md) — implemented contracts, adversarial - validation, exclusions, and exact candidate evidence. - [MCP contract](docs/MCP_CONTRACT.md) — exact tool and process boundary. - [Viewer manager](docs/VIEWER_MANAGER.md) — native service setup and lifecycle. - [Adapter decision](docs/APPLICATION_DECISION.md) — why custom adapters own canonical @@ -196,7 +157,7 @@ make gate Focused entry points are available as `make contract`, `make test`, `make type`, `make benchmark-smoke`, `make benchmark`, `make benchmark-m1-smoke`, and -`make benchmark-m1`. Milestone 2 adds `make benchmark-m2-smoke` and `make benchmark-m2`. +`make benchmark-m1`. The committed 1,000-node baseline and its measurement method are under `benchmarks/`. diff --git a/benchmarks/README.md b/benchmarks/README.md index 9d78eda..05f5c55 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -27,13 +27,6 @@ Run the maintained 1,000-node Milestone 1 benchmark: make benchmark-m1 ``` -Run the Milestone 2 agent-workflow smoke and full gates: - -```bash -make benchmark-m2-smoke -make benchmark-m2 -``` - The benchmark creates canonical sources, derived state, changesets, rendered output, and caches only in a disposable temporary directory. It does not read another project, self-host DocForge, or mutate repository content. @@ -49,13 +42,6 @@ harness enforces operation-specific p95 ceilings and fixed zero-work counter inv human-readable interpretation is in [`docs/MILESTONE_1_BASELINE.md`](../docs/MILESTONE_1_BASELINE.md). -`milestone2-2026-07-29.json` is the clean-tree agent-retrieval and client-integration baseline -captured from commit `fb0df5e4a1c591c2a84788fd4814d98550f11863`. It gates every warmup and -sample, reconstructs complete task-context and generation-diff collections across bounded pages, -records whether diagnostics were dropped for response budget, checks all hidden-work counters, -and measures isolated-process peak RSS. Its interpretation is in -[`docs/MILESTONE_2_BASELINE.md`](../docs/MILESTONE_2_BASELINE.md). - The generic fixture exposes whole-source scaling. It does not replace the incremental adapter equivalence tests and does not claim to measure a portable graph renderer, because Milestone 0 has no portable graph-planning or graph-rendering contract. diff --git a/benchmarks/milestone2-2026-07-29.json b/benchmarks/milestone2-2026-07-29.json deleted file mode 100644 index f4196ac..0000000 --- a/benchmarks/milestone2-2026-07-29.json +++ /dev/null @@ -1,457 +0,0 @@ -{ - "benchmark": "docforge2_milestone2", - "environment": { - "implementation": "CPython", - "machine": "x86_64", - "platform": "Linux-7.1.3-200.nobara.fc44.x86_64-x86_64-with-glibc2.43", - "python": "3.14.6" - }, - "fixture": { - "edge_count": 999, - "kind": "synthetic_generic_focus_fan_in", - "max_tool_output_chars": 200000, - "node_count": 1000, - "source_file_count": 1000 - }, - "isolated_process_peak_rss_kib": 86448, - "method": { - "clock": "time.perf_counter_ns", - "memory": "isolated child-process resource.getrusage(RUSAGE_SELF).ru_maxrss", - "memory_limit_kib": 262144, - "memory_probe_samples": 10, - "percentile": "nearest-rank", - "response_size": "UTF-8 bytes of compact sorted JSON", - "samples": 10, - "warmups": 1, - "zero_work_counters": [ - "project_loads", - "source_files_parsed", - "source_bytes_parsed", - "adapter_projection_loads", - "adapter_source_extractions", - "index_synchronizations", - "index_builds", - "render_prepare_calls", - "render_output_bytes_built", - "render_output_bytes_hashed", - "viewer_manager_requests" - ] - }, - "operations": { - "bootstrap_no_ast": { - "counter_ranges": { - "adapter_projection_loads": {"maximum": 0, "minimum": 0}, - "adapter_source_extractions": {"maximum": 0, "minimum": 0}, - "index_builds": {"maximum": 0, "minimum": 0}, - "index_checks": {"maximum": 1, "minimum": 1}, - "index_synchronizations": {"maximum": 1, "minimum": 1}, - "project_loads": {"maximum": 0, "minimum": 0}, - "render_output_bytes_built": {"maximum": 0, "minimum": 0}, - "render_output_bytes_hashed": {"maximum": 0, "minimum": 0}, - "render_prepare_calls": {"maximum": 0, "minimum": 0}, - "source_bytes_parsed": {"maximum": 0, "minimum": 0}, - "source_files_parsed": {"maximum": 0, "minimum": 0}, - "source_generation_checks": {"maximum": 1, "minimum": 1}, - "viewer_manager_requests": {"maximum": 0, "minimum": 0} - }, - "max_ms": 9.379, - "maximum_response_bytes": 8796, - "median_ms": 9.153, - "min_ms": 9.026, - "p95_limit_ms": 100, - "p95_ms": 9.379, - "response_bytes": 8795, - "response_limit_bytes": 32768, - "samples": 10, - "validated_invocations": 11 - }, - "bootstrap_read": { - "counter_ranges": { - "adapter_projection_loads": {"maximum": 0, "minimum": 0}, - "adapter_source_extractions": {"maximum": 0, "minimum": 0}, - "index_builds": {"maximum": 0, "minimum": 0}, - "index_checks": {"maximum": 1, "minimum": 1}, - "index_synchronizations": {"maximum": 1, "minimum": 1}, - "project_loads": {"maximum": 0, "minimum": 0}, - "render_output_bytes_built": {"maximum": 0, "minimum": 0}, - "render_output_bytes_hashed": {"maximum": 0, "minimum": 0}, - "render_prepare_calls": {"maximum": 0, "minimum": 0}, - "source_bytes_parsed": {"maximum": 0, "minimum": 0}, - "source_files_parsed": {"maximum": 0, "minimum": 0}, - "source_generation_checks": {"maximum": 1, "minimum": 1}, - "viewer_manager_requests": {"maximum": 0, "minimum": 0} - }, - "max_ms": 9.884, - "maximum_response_bytes": 7495, - "median_ms": 9.406, - "min_ms": 9.053, - "p95_limit_ms": 100, - "p95_ms": 9.884, - "response_bytes": 7491, - "response_limit_bytes": 32768, - "samples": 10, - "validated_invocations": 11 - }, - "configuration_preview": { - "claude": { - "artifact_format": "claude-json-fragment-v1", - "configuration_hash": "4dbb4ed0f38264fdba350de8904cc898493d620194b5105b7889c54bd5913c9c", - "counter_ranges": { - "adapter_projection_loads": {"maximum": 0, "minimum": 0}, - "adapter_source_extractions": {"maximum": 0, "minimum": 0}, - "index_builds": {"maximum": 0, "minimum": 0}, - "index_checks": {"maximum": 0, "minimum": 0}, - "index_synchronizations": {"maximum": 0, "minimum": 0}, - "project_loads": {"maximum": 0, "minimum": 0}, - "render_output_bytes_built": {"maximum": 0, "minimum": 0}, - "render_output_bytes_hashed": {"maximum": 0, "minimum": 0}, - "render_prepare_calls": {"maximum": 0, "minimum": 0}, - "source_bytes_parsed": {"maximum": 0, "minimum": 0}, - "source_files_parsed": {"maximum": 0, "minimum": 0}, - "source_generation_checks": {"maximum": 0, "minimum": 0}, - "viewer_manager_requests": {"maximum": 0, "minimum": 0} - }, - "max_ms": 364.365, - "maximum_response_bytes": 2748, - "median_ms": 314.326, - "min_ms": 314.278, - "p95_limit_ms": 500, - "p95_ms": 364.365, - "response_bytes": 2748, - "response_limit_bytes": 32768, - "samples": 10, - "validated_invocations": 11 - }, - "codex": { - "artifact_format": "codex-toml-fragment-v1", - "configuration_hash": "3e1dd5191021da1778cd1c4f4658768537775e5e16252e42d4c80e328841145b", - "counter_ranges": { - "adapter_projection_loads": {"maximum": 0, "minimum": 0}, - "adapter_source_extractions": {"maximum": 0, "minimum": 0}, - "index_builds": {"maximum": 0, "minimum": 0}, - "index_checks": {"maximum": 0, "minimum": 0}, - "index_synchronizations": {"maximum": 0, "minimum": 0}, - "project_loads": {"maximum": 0, "minimum": 0}, - "render_output_bytes_built": {"maximum": 0, "minimum": 0}, - "render_output_bytes_hashed": {"maximum": 0, "minimum": 0}, - "render_prepare_calls": {"maximum": 0, "minimum": 0}, - "source_bytes_parsed": {"maximum": 0, "minimum": 0}, - "source_files_parsed": {"maximum": 0, "minimum": 0}, - "source_generation_checks": {"maximum": 0, "minimum": 0}, - "viewer_manager_requests": {"maximum": 0, "minimum": 0} - }, - "max_ms": 364.383, - "maximum_response_bytes": 2627, - "median_ms": 314.365, - "min_ms": 314.248, - "p95_limit_ms": 500, - "p95_ms": 364.383, - "response_bytes": 2627, - "response_limit_bytes": 32768, - "samples": 10, - "validated_invocations": 11 - }, - "openclaw": { - "artifact_format": "openclaw-json-fragment-v1", - "configuration_hash": "6f269e90a55088c5d517f91761c53a3b90036d62fd80b5b9d094668267257b99", - "counter_ranges": { - "adapter_projection_loads": {"maximum": 0, "minimum": 0}, - "adapter_source_extractions": {"maximum": 0, "minimum": 0}, - "index_builds": {"maximum": 0, "minimum": 0}, - "index_checks": {"maximum": 0, "minimum": 0}, - "index_synchronizations": {"maximum": 0, "minimum": 0}, - "project_loads": {"maximum": 0, "minimum": 0}, - "render_output_bytes_built": {"maximum": 0, "minimum": 0}, - "render_output_bytes_hashed": {"maximum": 0, "minimum": 0}, - "render_prepare_calls": {"maximum": 0, "minimum": 0}, - "source_bytes_parsed": {"maximum": 0, "minimum": 0}, - "source_files_parsed": {"maximum": 0, "minimum": 0}, - "source_generation_checks": {"maximum": 0, "minimum": 0}, - "viewer_manager_requests": {"maximum": 0, "minimum": 0} - }, - "max_ms": 364.532, - "maximum_response_bytes": 2869, - "median_ms": 314.401, - "min_ms": 314.251, - "p95_limit_ms": 500, - "p95_ms": 364.532, - "response_bytes": 2869, - "response_limit_bytes": 32768, - "samples": 10, - "validated_invocations": 11 - } - }, - "doctor": { - "claude": { - "counter_ranges": { - "adapter_projection_loads": {"maximum": 0, "minimum": 0}, - "adapter_source_extractions": {"maximum": 0, "minimum": 0}, - "index_builds": {"maximum": 0, "minimum": 0}, - "index_checks": {"maximum": 0, "minimum": 0}, - "index_synchronizations": {"maximum": 0, "minimum": 0}, - "project_loads": {"maximum": 0, "minimum": 0}, - "render_output_bytes_built": {"maximum": 0, "minimum": 0}, - "render_output_bytes_hashed": {"maximum": 0, "minimum": 0}, - "render_prepare_calls": {"maximum": 0, "minimum": 0}, - "source_bytes_parsed": {"maximum": 0, "minimum": 0}, - "source_files_parsed": {"maximum": 0, "minimum": 0}, - "source_generation_checks": {"maximum": 0, "minimum": 0}, - "viewer_manager_requests": {"maximum": 0, "minimum": 0} - }, - "doctor_state": "degraded", - "max_ms": 0.446, - "maximum_response_bytes": 3669, - "median_ms": 0.364, - "min_ms": 0.352, - "p95_limit_ms": 100, - "p95_ms": 0.446, - "response_bytes": 3669, - "response_limit_bytes": 32768, - "samples": 10, - "summary": {"failed": 0, "passed": 11, "skipped": 1, "warning": 2}, - "validated_invocations": 11 - }, - "codex": { - "counter_ranges": { - "adapter_projection_loads": {"maximum": 0, "minimum": 0}, - "adapter_source_extractions": {"maximum": 0, "minimum": 0}, - "index_builds": {"maximum": 0, "minimum": 0}, - "index_checks": {"maximum": 0, "minimum": 0}, - "index_synchronizations": {"maximum": 0, "minimum": 0}, - "project_loads": {"maximum": 0, "minimum": 0}, - "render_output_bytes_built": {"maximum": 0, "minimum": 0}, - "render_output_bytes_hashed": {"maximum": 0, "minimum": 0}, - "render_prepare_calls": {"maximum": 0, "minimum": 0}, - "source_bytes_parsed": {"maximum": 0, "minimum": 0}, - "source_files_parsed": {"maximum": 0, "minimum": 0}, - "source_generation_checks": {"maximum": 0, "minimum": 0}, - "viewer_manager_requests": {"maximum": 0, "minimum": 0} - }, - "doctor_state": "healthy", - "max_ms": 0.556, - "maximum_response_bytes": 3595, - "median_ms": 0.421, - "min_ms": 0.404, - "p95_limit_ms": 100, - "p95_ms": 0.556, - "response_bytes": 3595, - "response_limit_bytes": 32768, - "samples": 10, - "summary": {"failed": 0, "passed": 13, "skipped": 1, "warning": 0}, - "validated_invocations": 11 - }, - "openclaw": { - "counter_ranges": { - "adapter_projection_loads": {"maximum": 0, "minimum": 0}, - "adapter_source_extractions": {"maximum": 0, "minimum": 0}, - "index_builds": {"maximum": 0, "minimum": 0}, - "index_checks": {"maximum": 0, "minimum": 0}, - "index_synchronizations": {"maximum": 0, "minimum": 0}, - "project_loads": {"maximum": 0, "minimum": 0}, - "render_output_bytes_built": {"maximum": 0, "minimum": 0}, - "render_output_bytes_hashed": {"maximum": 0, "minimum": 0}, - "render_prepare_calls": {"maximum": 0, "minimum": 0}, - "source_bytes_parsed": {"maximum": 0, "minimum": 0}, - "source_files_parsed": {"maximum": 0, "minimum": 0}, - "source_generation_checks": {"maximum": 0, "minimum": 0}, - "viewer_manager_requests": {"maximum": 0, "minimum": 0} - }, - "doctor_state": "healthy", - "max_ms": 0.484, - "maximum_response_bytes": 3602, - "median_ms": 0.384, - "min_ms": 0.353, - "p95_limit_ms": 100, - "p95_ms": 0.484, - "response_bytes": 3602, - "response_limit_bytes": 32768, - "samples": 10, - "summary": {"failed": 0, "passed": 13, "skipped": 1, "warning": 0}, - "validated_invocations": 11 - } - }, - "generation_diff_complete": { - "max_ms": 425.315, - "maximum_response_bytes": 984, - "median_ms": 418.607, - "min_ms": 410.974, - "p95_limit_ms": 500, - "p95_ms": 425.315, - "response_bytes": 983, - "response_limit_bytes": 32768, - "result_summary": { - "aggregate_page_bytes": 664715, - "counter_ranges": { - "adapter_projection_loads": {"maximum": 0, "minimum": 0}, - "adapter_source_extractions": {"maximum": 0, "minimum": 0}, - "index_builds": {"maximum": 0, "minimum": 0}, - "index_checks": {"maximum": 0, "minimum": 0}, - "index_synchronizations": {"maximum": 0, "minimum": 0}, - "project_loads": {"maximum": 0, "minimum": 0}, - "render_output_bytes_built": {"maximum": 0, "minimum": 0}, - "render_output_bytes_hashed": {"maximum": 0, "minimum": 0}, - "render_prepare_calls": {"maximum": 0, "minimum": 0}, - "source_bytes_parsed": {"maximum": 0, "minimum": 0}, - "source_files_parsed": {"maximum": 0, "minimum": 0}, - "source_generation_checks": {"maximum": 2, "minimum": 2}, - "viewer_manager_requests": {"maximum": 0, "minimum": 0} - }, - "elapsed_ms": 419.04, - "item_count": 1000, - "maximum_cursor_bytes": 448, - "maximum_page_bytes": 66516, - "ordered_item_hash": "1ac48cc72532809ef5d3e949756e536eec819f348eaf06338c9b39b14e63b2c7", - "page_count": 10, - "receipt_hash": "6913c962972d8255f56966e5cfab5ac8293e41bdfb5f39ef91a4f33a3b092f88", - "status": "ok" - }, - "samples": 10, - "validated_invocations": 11 - }, - "generation_diff_diagnostic_page": { - "counter_ranges": { - "adapter_projection_loads": {"maximum": 0, "minimum": 0}, - "adapter_source_extractions": {"maximum": 0, "minimum": 0}, - "index_builds": {"maximum": 0, "minimum": 0}, - "index_checks": {"maximum": 0, "minimum": 0}, - "index_synchronizations": {"maximum": 0, "minimum": 0}, - "project_loads": {"maximum": 0, "minimum": 0}, - "render_output_bytes_built": {"maximum": 0, "minimum": 0}, - "render_output_bytes_hashed": {"maximum": 0, "minimum": 0}, - "render_prepare_calls": {"maximum": 0, "minimum": 0}, - "source_bytes_parsed": {"maximum": 0, "minimum": 0}, - "source_files_parsed": {"maximum": 0, "minimum": 0}, - "source_generation_checks": {"maximum": 2, "minimum": 2}, - "viewer_manager_requests": {"maximum": 0, "minimum": 0} - }, - "max_ms": 41.65, - "maximum_response_bytes": 66516, - "median_ms": 40.961, - "min_ms": 40.239, - "p95_limit_ms": 100, - "p95_ms": 41.65, - "response_bytes": 66516, - "response_limit_bytes": 200000, - "samples": 10, - "validated_invocations": 11 - }, - "generation_diff_maximum_page": { - "diagnostics_dropped_for_budget": true, - "max_ms": 59.184, - "maximum_response_bytes": 199566, - "median_ms": 55.565, - "min_ms": 54.68, - "p95_limit_ms": 100, - "p95_ms": 59.184, - "response_bytes": 199566, - "response_limit_bytes": 200000, - "samples": 10, - "validated_invocations": 11 - }, - "task_context_complete": { - "max_ms": 721.847, - "maximum_response_bytes": 1325, - "median_ms": 703.561, - "min_ms": 688.975, - "p95_limit_ms": 2500, - "p95_ms": 721.847, - "response_bytes": 1324, - "response_limit_bytes": 32768, - "result_summary": { - "aggregate_page_bytes": 348845, - "capsule_hash": "20663ed685a255f7cb8a0e8d78262bf7bf26863d728f0159e0b29000f6a52b0a", - "collection_hash": "9dbc46eb5b1ac8c8340bb149205d3599fffaf67e3b363de2d035475da274857c", - "collection_hash_reconstructed": true, - "counter_ranges": { - "adapter_projection_loads": {"maximum": 0, "minimum": 0}, - "adapter_source_extractions": {"maximum": 0, "minimum": 0}, - "index_builds": {"maximum": 0, "minimum": 0}, - "index_checks": {"maximum": 1, "minimum": 1}, - "index_synchronizations": {"maximum": 0, "minimum": 0}, - "project_loads": {"maximum": 0, "minimum": 0}, - "render_output_bytes_built": {"maximum": 0, "minimum": 0}, - "render_output_bytes_hashed": {"maximum": 0, "minimum": 0}, - "render_prepare_calls": {"maximum": 0, "minimum": 0}, - "source_bytes_parsed": {"maximum": 0, "minimum": 0}, - "source_files_parsed": {"maximum": 0, "minimum": 0}, - "source_generation_checks": {"maximum": 2, "minimum": 2}, - "viewer_manager_requests": {"maximum": 0, "minimum": 0} - }, - "elapsed_ms": 719.55, - "evidence_count": 108, - "item_count": 1000, - "maximum_cursor_bytes": 1066, - "maximum_page_bytes": 151172, - "omission_count": 892, - "ordered_candidate_hash": "bdeb3a8f4018000f72a5ff1891aa800b6edf98070b9814f443c6bac4e52c38f3", - "ordered_evidence_hash": "ec07bac7f528f5a3afbc083ea5ad60541335c8789d42980fe7bae0476e0a5331", - "page_count": 11, - "plan_hash": "84dbe60267e5f8359adcf30d4d395f1ac5d5bea957beaa3888d2938120860a9c", - "status": "ok" - }, - "samples": 10, - "validated_invocations": 11 - }, - "task_context_diagnostic_page": { - "counter_ranges": { - "adapter_projection_loads": {"maximum": 0, "minimum": 0}, - "adapter_source_extractions": {"maximum": 0, "minimum": 0}, - "index_builds": {"maximum": 0, "minimum": 0}, - "index_checks": {"maximum": 1, "minimum": 1}, - "index_synchronizations": {"maximum": 0, "minimum": 0}, - "project_loads": {"maximum": 0, "minimum": 0}, - "render_output_bytes_built": {"maximum": 0, "minimum": 0}, - "render_output_bytes_hashed": {"maximum": 0, "minimum": 0}, - "render_prepare_calls": {"maximum": 0, "minimum": 0}, - "source_bytes_parsed": {"maximum": 0, "minimum": 0}, - "source_files_parsed": {"maximum": 0, "minimum": 0}, - "source_generation_checks": {"maximum": 2, "minimum": 2}, - "viewer_manager_requests": {"maximum": 0, "minimum": 0} - }, - "max_ms": 93.115, - "maximum_response_bytes": 7190, - "median_ms": 73.326, - "min_ms": 72.264, - "p95_limit_ms": 500, - "p95_ms": 93.115, - "response_bytes": 7190, - "response_limit_bytes": 200000, - "samples": 10, - "validated_invocations": 11 - }, - "task_context_maximum_page": { - "counter_ranges": { - "adapter_projection_loads": {"maximum": 0, "minimum": 0}, - "adapter_source_extractions": {"maximum": 0, "minimum": 0}, - "index_builds": {"maximum": 0, "minimum": 0}, - "index_checks": {"maximum": 1, "minimum": 1}, - "index_synchronizations": {"maximum": 0, "minimum": 0}, - "project_loads": {"maximum": 0, "minimum": 0}, - "render_output_bytes_built": {"maximum": 0, "minimum": 0}, - "render_output_bytes_hashed": {"maximum": 0, "minimum": 0}, - "render_prepare_calls": {"maximum": 0, "minimum": 0}, - "source_bytes_parsed": {"maximum": 0, "minimum": 0}, - "source_files_parsed": {"maximum": 0, "minimum": 0}, - "source_generation_checks": {"maximum": 2, "minimum": 2}, - "viewer_manager_requests": {"maximum": 0, "minimum": 0} - }, - "diagnostics_dropped_for_budget": false, - "max_ms": 87.372, - "maximum_response_bytes": 7192, - "median_ms": 84.933, - "min_ms": 83.652, - "p95_limit_ms": 500, - "p95_ms": 87.372, - "response_bytes": 7192, - "response_limit_bytes": 200000, - "samples": 10, - "validated_invocations": 11 - } - }, - "process_peak_rss_kib": 86072, - "schema_version": 1, - "source": { - "dirty": false, - "revision": "fb0df5e4a1c591c2a84788fd4814d98550f11863" - } -} diff --git a/docs/COMPATIBILITY.md b/docs/COMPATIBILITY.md index 12c6247..c92004f 100644 --- a/docs/COMPATIBILITY.md +++ b/docs/COMPATIBILITY.md @@ -35,7 +35,6 @@ remain supported: - `docforge.index` - `docforge.mcp_server` - `docforge.models` -- `docforge.policy` - `docforge.render_contract` Names beginning with an underscore are implementation details. New public names may be added @@ -60,17 +59,6 @@ MCP results retain: The result schema describes the common envelope. Operation-specific fields are additive and remain bounded by the configured tool-output limit. -The following Milestone 2 CLI additions do not change existing command signatures: - -- `docforge configure codex|claude|openclaw --project ROOT` -- `docforge doctor --client codex|claude|openclaw` - -Configuration output is a new version-1 machine-local contract. It preserves the `docforge` -package and executable names and emits the existing `docforge.mcp_server` module entrypoint. -Existing hand-written client configurations remain valid and are never rewritten automatically. -Doctor is inspection-only and does not become a hidden bootstrap, synchronization, or migration -path. - ## Versioned data contracts Milestone 0 preserves: @@ -84,18 +72,10 @@ Milestone 0 preserves: version 3 adds a source-ordered incoming-edge index for bounded impact traversal. - Index-attestation schema version 1. - Incremental extraction-cache schema version 1. -- Effective process-policy schema version 1. The project descriptor remains schema version 1; - machine-specific capability selection is a startup binding, not canonical project content. - Read-pagination schema version 1. Existing tool names and required arguments are unchanged. Context and changeset MCP reads accept optional limits and opaque generation-bound cursors. Direct Python changeset methods and the ordinary CLI context command retain full legacy results when pagination is not requested. -- Latest-generation-diff receipt schema version 1. The additive `generation-diff` CLI command and - `docforge_get_generation_diff` MCP read accept only optional pagination fields. They record one - primary-graph transition and do not create a history store or expose Logic details. -- Latest-generation-diff page schema version 1. Pages use one top-level pagination object and a - nested `receipt_header`. `stored_receipt_hash` names the complete stored receipt. Opaque cursors - may be restarted after a server or receipt change and are not durable public identifiers. Indexes, attestations, extraction caches, previews, and rendered artifacts are disposable. A schema change may rebuild them. Canonical project content and stored proposals may not be silently @@ -116,10 +96,6 @@ must retain `load_projection()` as their independent clean-build and equivalence Project adapters remain explicitly composed. Generic DocForge does not discover arbitrary adapter modules or choose a project globally. -The supported generation-diff Python boundary is `ProjectIndex.generation_diff()`. Helpers in the -`docforge.generation_diff` module implement the disposable publication contract and are internal; -they are not frozen as adapter-authoring imports. - ## Preserved no-AST binding `docforge-mcp --project-root /project --no-ast` is a stable shorthand for the @@ -136,10 +112,6 @@ The binding: - Applies the same restriction during hash-bound canonical-application refresh. - Reports the effective policy through bootstrap and contract results. -The legacy `adapter_policy` payload and error codes remain unchanged. The version-1 -`effective_policy` is additive and makes precedence, capability mode, render behavior, blocked -tools, and prohibitions machine-readable. - DocForge does not inspect arbitrary adapter source to prove which parser implementation it uses. The no-AST binding is an owner-selected process policy backed by Logic publication and retrieval enforcement. It is not a filesystem sandbox and cannot stop an unrelated process with repository @@ -168,37 +140,6 @@ and side-effect-free status. The live graph viewer remains a read-only consumer of a generation-pinned validated index. It does not become project authority or MCP retrieval authority. -## Task-context compatibility - -`docforge_get_task_context` is an additive MCP read tool. The legacy `docforge_get_context` -signature, profile compiler, direct Python results, and custom three-argument context-provider -contract remain unchanged. - -The new `ContextCapsuleV1` and `RetrievalPlanV1` types live in the public -`docforge.retrieval` submodule. Version 1 guarantees: - -- A closed task-kind vocabulary and core-derived plan. Callers cannot inject arbitrary operations, - SQL, paths, relations, or Logic requests. -- One immutable index transaction and one exact project, adapter, revision, source, policy, - request, plan, collection, and capsule identity. -- Deterministic bounded focus, traversal, hydration, token accounting, response packing, and - continuation, with fixed version-1 ceilings of 1,000 evidence items, 100,000 candidate edges, and - 10,000 task-query characters. -- Raw preservation of project-owned relation names. Only the documented versioned alias map gains - task semantics; all other relations remain `unclassified`. -- Separate missing, incomplete, blocked, and provenance-limitation evidence. -- No-AST bindings retain task context but never add a Logic retrieval step or weaken the existing - Logic prohibition. - -An integration that replaces the legacy context provider does not silently receive the core task -planner. Version 1 has no custom task-planner protocol. The task-context tool remains registered -for additive name compatibility but returns `task_context_unavailable` without loading or -synchronizing the custom projection. - -The exact version-1 relation aliases are frozen by the MCP contract and repository contract tests. -Changing an alias category requires a new planner version; it is not a silent implementation -detail. - `ManualRenderPlan`, `GraphViewPlan`, a portable graph renderer, and independently packaged renderers are later-milestone direction. Milestone 0 does not claim that those contracts exist. diff --git a/docs/CONTRACT.md b/docs/CONTRACT.md index ec2b062..8406be4 100644 --- a/docs/CONTRACT.md +++ b/docs/CONTRACT.md @@ -18,13 +18,7 @@ commit when Git is available; it cannot change repository state. - Edge schema: `schemas/edge.schema.json`, version 1. - Result envelope: `schemas/result.schema.json`, version 1. - Changeset schema: `schemas/changeset.schema.json`, version 1. -- Effective policy: `schemas/policy.schema.json`, version 1. -- Task context capsule: `schemas/context-capsule.schema.json`, version 1. -- Latest generation diff: `schemas/generation-diff.schema.json`, version 1. -- Latest generation-diff page: `schemas/generation-diff-page.schema.json`, version 1. -- Generated client configuration: `schemas/client-configuration.schema.json`, version 1. -- Client doctor result: `schemas/doctor-result.schema.json`, version 1. -- Index schema: version 3, disposable and reproducible. +- Index schema: version 2, disposable and reproducible. - Index attestation: schema version 1, disposable and reproducible. - Core, CLI, and MCP server: version 1.3.0.dev0. - Incremental extraction cache: version 1, disposable and reproducible. @@ -50,52 +44,11 @@ source hash. Errors use a stable code, direct message, structured details, and a tool when recovery is safe. MCP operations synchronize disposable index state under a project lock before reading or proposing. Canonical source validation remains fail-closed. -Task-context retrieval derives a closed version-1 plan from a bounded task kind and the effective -process policy. It executes against one immutable index transaction and returns generation-bound, -hash-identified evidence, gaps, omissions, and provenance limitations. Project relation names -remain authoritative. The core applies task semantics only to its versioned alias set and preserves -every other allowed relation as unclassified. - An atomic index build writes a whole-file SHA-256 attestation after complete graph, row, FTS, and SQLite integrity verification. A fresh process may use that receipt to verify an unchanged index without reconstructing all graph rows. A missing, malformed, or mismatched receipt falls back to complete verification and is repaired only after that verification succeeds. -Index replacement is the derived publication commit point. Attestation, cheap source-generation, -and latest-generation-diff receipts are independent post-commit evidence. Their failure produces -bounded degraded success and never falsely reports that a committed index mutation failed. - -Before replacement, a build accepts a predecessor only when its exact main-file inode has a -matching whole-file attestation, has no WAL, journal, or shared-memory sidecar, and passes the -published SQLite identity, row, hash, FTS, integrity, and policy checks. It uses an immutable -main-file read and never repairs predecessor evidence. The build then revalidates the new source -snapshot including exact node and edge equality and rejects a stable source identity that produces -different graph content as `generation_collision`. - -The version-1 generation-diff receipt stores one bounded latest primary-graph transition. It is -not history and contains no Logic details or source text. Public pages carry one -`receipt_header`; its `stored_receipt_hash` identifies the complete persisted receipt rather than -the header alone. One top-level pagination object carries the only continuation cursor. - -## Machine-local client integration - -Generated Codex, Claude, and OpenClaw fragments are machine-local projections. They are not -canonical project content. Version 1 binds the selected project, exact isolated Python -interpreter, canonical argument layout, effective policy, no-AST projection, render policy, -timeouts, artifact bytes, and configuration hash. - -Preview is side-effect free. Explicit publication creates only one new private standalone -fragment in an existing real directory. It never merges or replaces different content. Descriptor, -parent, target, content, ownership, permission, and link identities are checked before and after -the directory durability boundary. A failure rolls back when that can be proven and otherwise -returns bounded unconfirmed publication evidence. - -Doctor is a bounded read-only inspector with one fixed check inventory. It uses stable no-follow -descriptor and configuration reads plus stat-only derived-index evidence. It never loads a -complete projection, opens SQLite, starts MCP, executes the configured command, synchronizes, -builds, renders, starts a viewer, or writes configuration. Unprovable client behavior is a warning, -not an invented success. - ## Isolated proposal model Create, update, move, and delete are ordered node operations inside an isolated changeset. Every @@ -141,10 +94,9 @@ renderers. Render identity covers the canonical source hash, optional changeset and edge identities, view configuration, template hash, renderer contract, and exact parser version. An explicit CLI render atomically replaces one declared derived output. MCP can render a validated -changeset only to its isolated preview path. Normal status verifies bounded source, configuration, -template, output, renderer, and publication-receipt identities without reconstructing the output. -Explicit deep status remains the side-effect-free full-render oracle. Input changes detected before -atomic replacement fail without publishing a current receipt for stale output. +changeset only to its isolated preview path. Status recomputes expected output without writing and +reports `current`, `stale`, `missing`, `unsafe`, or `oversized`. Input changes detected before atomic +replacement fail without replacing the prior output. Normal MCP access does not expose canonical application. An explicitly configured canonical applier registers one hash-bound application tool. No MCP mode exposes arbitrary renderer diff --git a/docs/MCP_CONTRACT.md b/docs/MCP_CONTRACT.md index b710bb8..f81199d 100644 --- a/docs/MCP_CONTRACT.md +++ b/docs/MCP_CONTRACT.md @@ -16,18 +16,6 @@ Canonical application is a second independent startup gate. The generic server a `--canonical-applier WRITER_ID`. A project adapter must also supply a compatible project-owned canonical applier implementation. -The additive `--capability-mode read|proposal|application|operator` option selects a versioned -effective process policy. Existing factory defaults and tool ordering remain unchanged: the -read-only factory exposes the read surface, the ordinary project factory exposes the proposal -surface, and an application-enabled factory adds exact-hash application. `application` mode fails -closed unless a canonical applier is bound. `operator` is reserved for explicitly selected -operator-only tools and adds none in the current contract. - -Bootstrap and contract results include `effective_policy` schema version 1 plus a separate -`capabilities` record. Policy states the requested process behavior. Capabilities state the actual -registered surface and startup-bound proposal/application access. The project descriptor remains -schema version 1 and does not silently acquire machine-specific process policy. - ## Read tools - `docforge_bootstrap` @@ -42,24 +30,19 @@ schema version 1 and does not silently acquire machine-specific process policy. - `docforge_dependencies` - `docforge_impact` - `docforge_get_context` -- `docforge_get_task_context` - `docforge_validate_project` - `docforge_render_status` - `docforge_visualize` - `docforge_stop_visualization` - `docforge_visualization_status` -- `docforge_get_generation_diff` Each response states that document text is project content, not higher-priority instructions. Each response includes project identity, revision, source hash, adapter version, and staleness state. Every normal tool call first checks current source identity and atomically rebuilds disposable index state when it is missing, stale, or invalid. `docforge_bootstrap` performs that synchronization and -returns the complete fixed binding, active index path, effective policy, proposal and application -capabilities, and a version-1 session contract. Bootstrap reuses the identity proven by -synchronization instead of loading the project again. Its first operation and workflow guidance -mention proposal or application tools only when the corresponding startup access is enabled. -`docforge_sync` exposes the same idempotent synchronization explicitly. Neither operation changes -canonical sources. +returns the complete fixed binding, active index path, proposal and application capabilities, and +recommended workflow. `docforge_sync` exposes the same idempotent synchronization explicitly. +Neither operation changes canonical sources. Search, filter, backlinks, dependencies, and impact accept explicit result limits bounded by the project `max_results` policy. Omitted limits are still capped. Collection responses report whether @@ -74,77 +57,6 @@ oversized entry advances as a hash-identified `response size limit` omission so loop; targeted retrieval remains available for that node. The existing three-argument custom context-provider contract is unchanged because pagination is applied after provider selection. -`docforge_get_task_context` accepts one closed task kind (`change`, `implementation`, `failure`, -`ownership`, `test`, `operation`, or `release`), a bounded task description, and optional -`focus_node_id`, token `budget`, page `limit`, and opaque `cursor`. It derives, rather than accepts, -a version-1 retrieval plan. The plan contains only exact or lexical focus, bounded outgoing and -incoming graph traversal, and metadata hydration. It cannot request arbitrary SQL, paths, relation -names, or Logic extraction. Task context applies fixed internal ceilings of 1,000 evidence items, -100,000 examined candidate edges, and 10,000 task-query characters even when broader project -limits are configured. Traversal steps bind the complete project-owned relation vocabulary by hash -rather than copying an unbounded name list into every response. - -The plan and returned context capsule are bound to the effective policy and one immutable index -generation. Every evidence item identifies its indexed source path, content hash, graph path, -additional qualifying relationship reasons, and the provenance facts that the current graph -cannot prove. Required evidence gaps distinguish an undeclared relation category, a completed -bounded search with no selected evidence, and an incomplete proof caused by a work, result, token, -or response limit. Unknown project relations remain present with their raw names and an -`unclassified_relation` limitation; DocForge never infers semantics from spelling outside the -versioned alias map. - -Path relationship direction is relative to the preceding traversal node. Additional -`relationship_reasons` direction is relative to the evidence item itself: `outgoing` when that -evidence node is the stored source and `incoming` when it is the stored target. - -The exact version-1 aliases are: structure (`contains`, `defined_in`, `defines`, `owns`); -implementation (`implemented_by`, `implements`, `inherits`, `inherits_from`); dependency -(`depends_on`, `imports`); execution (`activates`, `calls`, `dispatches_to`, `launches`); data -(`reads`, `writes`); evidence (`documents`, `governs`, `proves`, `tested_by`, `verifies`); and -context (`relates_to`). Every other allowed relation is `unclassified`. - -Task-context continuation partitions the immutable evidence stream without changing its -`request_hash`, `plan_hash`, `collection_hash`, or `capsule_hash`. Its cursor additionally binds -the effective policy and task request. One evidence item that cannot fit advances exactly once as -a hash-identified `response_limit` omission. A changed generation, policy, plan, or collection -fails as `stale_cursor`. - -`docforge_get_generation_diff` accepts only optional `limit` and `cursor` fields. It reads the one -latest version-1 primary-graph transition receipt; it does not accept arbitrary generations, -paths, or history selectors. Exact summary counts and the full item-collection hash cover the -complete transition. Pagination covers only the deterministically ordered retained details and -states separately when the fixed 1,000-item or 1 MiB publication limit permanently omitted -details. - -The receipt binds project, root, adapter, index schema, from/to source identity, node and edge -hashes and counts, the committed index file identity, retained and full collection hashes, and its -own canonical hash. Node changes compare every core `Node` field. Edge identity is the exact -`(source_id, relation, target_id)` triple. Logic is excluded from public diff details. - -Current pages use `page_schema_version = 1`. The nested `receipt_header` contains every stored -receipt field except `items`; its `stored_receipt_hash` is the hash of the complete stored receipt, -not of the header. Page items and hash-identified response-limit omissions are siblings of that -header. The only pagination object is at the top level, and its `next_cursor` is the only cursor -copy. The page hash covers the complete header, page items, omissions, receipt state, and -pagination receipt. - -Receipt states are fail-closed: `current` is proven against cheap source identity and exact index -and receipt inodes; `stale` is a proven generation mismatch; `missing` means no receipt; -`unsafe` means confinement or file-type checks failed; `unverified` covers corrupt, foreign, -oversized, or concurrently changed evidence; and `unknown` means the project cannot provide a -cheap generation identity. Only `current` returns a page. - -This status boundary is non-repairing. It never opens SQLite, loads or extracts an adapter -projection, parses source, synchronizes, builds, or writes a receipt. Cheap source identity and -stable receipt/index file identities can establish `current`; legacy projects without cheap -identity report `unknown`. Invalid or unavailable disposable evidence remains an explicit status -instead of triggering hidden recovery. - -The legacy `docforge_get_context` tool and its custom three-argument provider contract remain -unchanged. A server with a custom context provider does not silently inherit the core task planner; -version 1 exposes no custom task-planner extension point. `docforge_get_task_context` returns -`task_context_unavailable` without synchronizing or loading the custom projection. - Version-1 cursors are canonical JSON encoded as base64url with a domain-separated SHA-256 corruption checksum. They are opaque and fail closed, but are not authenticated authorization tokens. Cursors bind the project, adapter, source generation, operation parameters, collection @@ -345,7 +257,3 @@ and indexed Logic, but it does not inspect arbitrary adapter source to prove whi implementation the adapter uses. DocForge still exposes no filesystem sandbox and cannot prevent an unrelated process with direct repository write access from editing adapter files. Repository permissions and project instructions remain responsible for that broader boundary. - -The legacy `adapter_policy` object remains byte-compatible. It is now a projection of the -versioned `effective_policy`; `--no-ast` restrictively overrides adapter evolution, AST analysis, -and Logic indexing without widening any other capability. diff --git a/docs/MILESTONE_2_BASELINE.md b/docs/MILESTONE_2_BASELINE.md deleted file mode 100644 index 44089f6..0000000 --- a/docs/MILESTONE_2_BASELINE.md +++ /dev/null @@ -1,85 +0,0 @@ -# Milestone 2 baseline - -## Scope and method - -This baseline records the agent-retrieval and client-integration behavior added in Milestone 2. -It was captured on 2026-07-29 from clean candidate commit -`fb0df5e4a1c591c2a84788fd4814d98550f11863`. - -The maintained command was: - -```bash -.venv/bin/python tools/milestone2_benchmark.py \ - --nodes 1000 \ - --samples 10 \ - --output /tmp/docforge-milestone2-final.json -``` - -The fixture contains 1,000 Markdown nodes and 999 edges in a direct fan-in around one focus node. -The configured MCP response limit is 200,000 characters. Durations use -`time.perf_counter_ns()` and nearest-rank p95. Peak memory uses an isolated child process and -`RUSAGE_SELF`. Every warmup and measured invocation is validated. - -Environment: - -- Linux 7.1.3-200.nobara.fc44.x86_64. -- CPython 3.14.6. -- x86_64. -- Ten warm samples after one warmup. -- Isolated memory ceiling: 262,144 KiB. - -The complete machine-readable result is -[`benchmarks/milestone2-2026-07-29.json`](../benchmarks/milestone2-2026-07-29.json). - -## Results - -| Operation | Median | p95 | Limit | Maximum response | -|---|---:|---:|---:|---:| -| Read bootstrap | 9.406 ms | 9.884 ms | 100 ms | 7,495 B | -| No-AST bootstrap | 9.153 ms | 9.379 ms | 100 ms | 8,796 B | -| Task diagnostic page | 73.326 ms | 93.115 ms | 500 ms | 7,190 B | -| Task complete traversal | 703.561 ms | 721.847 ms | 2,500 ms | 151,172 B/page | -| Generation diagnostic page | 40.961 ms | 41.650 ms | 100 ms | 66,516 B | -| Generation maximum page | 55.565 ms | 59.184 ms | 100 ms | 199,566 B | -| Generation complete traversal | 418.607 ms | 425.315 ms | 500 ms | 66,516 B/page | -| Codex configuration preview | 314.365 ms | 364.383 ms | 500 ms | 2,627 B | -| Claude configuration preview | 314.326 ms | 364.365 ms | 500 ms | 2,748 B | -| OpenClaw configuration preview | 314.401 ms | 364.532 ms | 500 ms | 2,869 B | -| Codex doctor | 0.421 ms | 0.556 ms | 100 ms | 3,595 B | -| Claude doctor | 0.364 ms | 0.446 ms | 100 ms | 3,669 B | -| OpenClaw doctor | 0.384 ms | 0.484 ms | 100 ms | 3,602 B | - -Isolated peak RSS was 86,448 KiB. - -Task traversal returned 108 evidence records and 892 explicit omissions across 11 pages. One -individually oversized focus record became a response-limit surrogate bound to the original record -hash. The remaining omissions were token-budget evidence. The benchmark verified every unique -subject, reconstructed the original collection hash, and matched the exact 1,000-node fixture. - -Generation traversal returned all 1,000 changed-node details across 10 pages. It reconstructed the -stored retained-collection hash. The maximum generation page approached the response limit and -proved that optional diagnostics were dropped before the primary result. - -## Structured-work gates - -Configuration preview and doctor performed zero project loads, source parses, adapter projection -loads, adapter extraction, index checks, synchronization, index builds, render preparation, -rendered-byte construction or hashing, and viewer-manager requests. - -Task-context pages performed exactly one index check and two cheap source-generation checks. They -performed none of the hidden work above. Generation-diff pages performed exactly two cheap -source-generation checks and no index check or hidden work. - -## Measured limits and future notes - -- Continuation is stateless and regenerates the task capsule for each page. The complete - 11-page traversal remains within its gate, but later work can avoid repeated planning without - weakening generation binding. -- Configuration preview deliberately spends about 314 ms proving that the exact isolated - interpreter can import the MCP module. Discovery-only checks were rejected as unsafe. -- Claude configuration syntax is supported, but its timeout representation remains unverified. - Doctor therefore reports degraded rather than healthy. -- Doctor is a configuration inspector, not an MCP connection or SQLite integrity test. -- Legacy adapters without cheap source-generation identity report unknown for generation-diff - freshness. -- The results do not justify a storage rewrite. SQLite remains fast after one generation is pinned. diff --git a/docs/MILESTONE_2_CLOSEOUT.md b/docs/MILESTONE_2_CLOSEOUT.md deleted file mode 100644 index 8ff2f8f..0000000 --- a/docs/MILESTONE_2_CLOSEOUT.md +++ /dev/null @@ -1,68 +0,0 @@ -# Milestone 2 closeout - -## Outcome - -Milestone 2 is complete. One project-bound server can expose an explicit effective policy and -return compact, task-shaped, explainable context. Users can generate deterministic client -fragments and inspect their bindings without hidden runtime work. - -Implemented contracts: - -- Version-1 effective policy and capability-aware bootstrap. -- Version-1 retrieval plans and context capsules. -- Bounded task-context pagination with evidence gaps and explicit omissions. -- One disposable latest-generation transition receipt and paged read surface. -- Deterministic Codex, Claude, and OpenClaw standalone configuration fragments. -- Fixed-inventory read-only doctor results. -- Dedicated configuration and doctor JSON schemas. -- Repository-native Milestone 2 contract, smoke, scale, response-size, counter, and memory gates. - -## Candidate evidence - -The frozen implementation candidate is -`fb0df5e4a1c591c2a84788fd4814d98550f11863`. - -The complete repository gate passed: - -- Ruff formatting and lint. -- HTML, rendered-manual HTML, CSS, and JavaScript checks. -- Pyright with zero diagnostics. -- Warning-strict compilation and tests. -- 205 tests and 120 subtests. -- Lock and npm dependency-tree checks. -- Wheel and source-distribution builds. -- Milestone 0, 1, and 2 smoke benchmarks. - -Three independent read-only adversarial audits covered client publication and policy binding, -doctor race and malformed-input behavior, and benchmark/contract evidence. Reproduced descriptor, -parent, target, filesystem, policy, secret-redaction, ambiguity, parser, response-size, and hidden -work defects were fixed and regression-tested before the candidate was frozen. - -The clean ten-sample 1,000-node benchmark passed every threshold. Exact measurements and counter -ranges are recorded in -[`MILESTONE_2_BASELINE.md`](MILESTONE_2_BASELINE.md) and -[`benchmarks/milestone2-2026-07-29.json`](../benchmarks/milestone2-2026-07-29.json). - -## Preserved boundaries - -- The `docforge` package, imports, CLI executable, MCP executable, and existing tool names remain. -- Legacy one-method `load_projection()` adapters remain supported. -- The no-AST shorthand and legacy adapter-policy payload remain compatible. -- Project descriptor schema version 1 remains unchanged. -- No storage replacement was introduced. -- No legacy DocForge MCP or DocForge2 self-hosting was used. -- WorldForge and ScrapeStation were not touched. -- No production MCP integration was repointed. -- The legacy Forgejo repository and `legacy` remote were not changed. -- No tag, release, release announcement, or visibility change was created. - -## Known follow-up work - -The next active milestone may improve projection independence. It must not silently absorb these -separate future ideas: - -- Avoid recomputing a complete task capsule for every continuation page. -- Add authenticated cursors only if a stronger threat model requires them. -- Verify Claude's native timeout representation. -- Add versioned adapter-owned launcher metadata before generating custom-adapter configurations. -- Keep doctor read-only; a live connection test must be an explicit separate operation. diff --git a/docs/USER_MANUAL.md b/docs/USER_MANUAL.md index a92ed1b..f161c56 100644 --- a/docs/USER_MANUAL.md +++ b/docs/USER_MANUAL.md @@ -478,15 +478,8 @@ backlinks NODE_ID [--relation RELATION] [--limit N] dependencies NODE_ID [--depth N] [--limit N] impact NODE_ID [--depth N] [--limit N] context PROFILE [--budget N] [--limit N] [--cursor OPAQUE] -generation-diff [--limit N] [--cursor OPAQUE] ``` -`generation-diff` returns the latest verified primary-graph transition. It is not a history query. -Current results carry a version-1 page, a `receipt_header` bound to the complete stored receipt by -`stored_receipt_hash`, and one top-level pagination cursor. Missing, unsafe, stale, corrupt, or -unprovable disposable evidence is reported as a non-repairing receipt status. The command never -builds or repairs the index. - ### Render and proposal commands ```text @@ -508,67 +501,6 @@ visualization-status visualization-stop ``` -### Client configuration and doctor - -Preview one deterministic standalone client fragment: - -```bash -docforge configure codex --project /absolute/path/MyProject -docforge configure claude --project /absolute/path/MyProject -docforge configure openclaw --project /absolute/path/MyProject -``` - -Preview is the default. Add `--output /absolute/path/fragment` to create a new private fragment in -an existing real directory. Publication is create-only. DocForge accepts an identical existing -private single-link file as unchanged, but it never merges, replaces, broadens permissions, or -follows a symlink. Descriptor, parent, and target identities are revalidated across the -publication commit. - -The generated command uses the exact current Python interpreter with isolated module startup. -Generation first proves that this interpreter can import `docforge.mcp_server`. The result binds -the project root, effective policy, arguments, artifact bytes, and all hashes. It copies no ambient -environment values. - -Select authority explicitly: - -```bash -docforge configure codex \ - --project /absolute/path/MyProject \ - --capability-mode proposal \ - --proposal-writer project-editor - -docforge configure codex \ - --project /absolute/path/MyProject \ - --capability-mode application \ - --proposal-writer project-editor \ - --canonical-applier project-editor -``` - -Read mode is the default. Proposal and application modes fail closed unless the descriptor -declares the named writer, and application requires the same writer/applier identity. Add -`--no-ast` to preserve the no-AST binding. Generic CLI generation refuses project-owned adapters -because it cannot safely reconstruct their composition. - -Inspect one configured client binding: - -```bash -docforge doctor --client codex --project /absolute/path/MyProject -docforge doctor --client codex \ - --project /absolute/path/MyProject \ - --config /absolute/path/config.toml \ - --server-name my-project-docforge -``` - -Doctor returns `healthy`, `degraded`, or `unhealthy` with exit codes 0, 1, or 2. Its fixed -version-1 inventory checks project and descriptor binding, the client driver and entry, executable -and arguments, project root, effective policy, no-AST state, timeouts, environment-key names, -tool-filter representation, and stat-only index presence. - -Doctor is intentionally not a connection test. It never loads canonical sources, opens SQLite, -starts MCP, executes the configured command, synchronizes, builds, renders, starts a viewer, or -writes configuration. Claude timeout representation and client filtering that cannot be proved -locally remain explicit warnings. - ## MCP usage Run one MCP server per project with absolute paths: @@ -581,19 +513,6 @@ docforge-mcp \ Omit `--proposal-writer` when the MCP client should not create or append proposals. -Select the session's declared surface explicitly when useful: - -```bash -docforge-mcp \ - --project-root /absolute/path/MyProject \ - --capability-mode read -``` - -Supported modes are `read`, `proposal`, `application`, and `operator`. Existing startup defaults -remain compatible. Capability mode describes the registered surface; bootstrap separately reports -whether a configured writer or applier actually grants mutation access. Application mode refuses -startup without a canonical applier. Operator mode is reserved and currently adds no tools. - Add `--diagnostics` when profiling a development or benchmark session. Each MCP response then includes bounded stage timings and compiler-work counters. The same flag is available on `docforge`. Diagnostics are disabled by default, record no project content or paths, and never @@ -611,11 +530,6 @@ docforge-mcp \ Without `--canonical-applier`, `docforge_apply_changeset` is not registered. The flag is an identity, not a command. The changeset creator, configured writer, and canonical applier must agree. -Call `docforge_bootstrap` first. Its version-1 `session_contract` contains the fixed binding, -current graph generation, effective policy, actual capabilities, render policies, prohibitions, -and a recommended first operation. Workflow guidance does not recommend registration or -application when those startup capabilities are unavailable. - Example MCP client configuration: ```json @@ -650,13 +564,11 @@ Example MCP client configuration: - `docforge_dependencies` - `docforge_impact` - `docforge_get_context` -- `docforge_get_task_context` - `docforge_validate_project` - `docforge_render_status` - `docforge_visualize` - `docforge_visualization_status` - `docforge_stop_visualization` -- `docforge_get_generation_diff` ### Proposal tools @@ -683,54 +595,6 @@ The application call requires `changeset_id` and `expected_changeset_hash`. Alwa inspect the final diff after the last proposal mutation. Apply that exact hash. A proposal mutation creates a new hash, so an earlier approval cannot silently apply later content. -Use `docforge_get_task_context` when an agent needs one bounded task-shaped intake instead of a -named profile. Choose `task_kind` from `change`, `implementation`, `failure`, `ownership`, `test`, -`operation`, or `release`. Supply `focus_node_id` when the stable node is known. Without it, -DocForge performs a bounded lexical focus search and refuses a tied best match instead of silently -choosing one. - -The returned version-1 capsule includes: - -- The exact project, adapter, source generation, effective policy, request, and retrieval-plan - hashes. -- Ordered focus and related evidence with source paths, content hashes, graph paths, and all - qualifying relationship reasons observed during the bounded traversal. -- Explicit evidence gaps and omissions, including whether a check completed. -- Provenance limitations for facts that the current graph does not carry, such as extractor - identity, observation time, and source provenance for relationships. - -Project descriptors still own the valid relation vocabulary. The planner recognizes a fixed alias -map for structure, implementation, dependency, execution, data, evidence, and context. Any other -valid project relation is returned unchanged as `unclassified`; it is never assigned guessed task -semantics. - -A relationship inside `relationship_path` describes the direction traveled from the preceding -node. A relationship inside `relationship_reasons` describes direction from the evidence item -itself. This keeps stored source and target identity exact while making each evidence explanation -locally readable. - -Task context never exceeds 1,000 evidence items, 100,000 examined candidate edges, or 10,000 task -query characters, even when a project configures broader general limits. An edge-work or -unclassified-relation ceiling appears as an explicit omission rather than an unbounded response. - -Use `docforge_get_generation_diff` after synchronization or a completed implementation slice to -inspect the one latest verified primary-graph transition. The version-1 receipt reports exact -added, removed, and changed node counts plus added and removed edge counts. Retained node details -identify changed fields and before/after hashes and source paths. Edge details retain the exact raw -relation triple. The receipt stores no source text, rendered content, Logic identities, or -historical sequence. - -The first successful publication is an explicit baseline and does not claim every current node was -added. A corrupt, foreign, unsafe, or unavailable predecessor produces an unavailable comparison -rather than fabricated removals. A same-generation reindex preserves the latest meaningful -transition. Each later real transition atomically replaces the single disposable receipt. - -Generation-diff reads use only the bounded receipt, stable file identities, and an adapter's cheap -source-generation proof. They do not open SQLite, load a complete adapter projection, parse source, -synchronize, build, or repair. Legacy adapters without cheap identity report `unknown`. Missing, -corrupt, foreign, oversized, or concurrently changed receipts report an explicit receipt state and -do not trigger hidden recovery. - Recommended release-candidate sequence: 1. Call `docforge_bootstrap`. It synchronizes derived state and reports the exact fixed binding. @@ -793,16 +657,6 @@ inspection pages may use hash summaries. A large diff may return `result_mode = "canonical_json_chunk"`; concatenate the chunks in order and verify `payload_hash` before decoding the reconstructed `operations` and `changes` object. -`docforge_get_task_context` uses the same opaque continuation discipline over capsule evidence -followed by capsule omissions. Keep the semantic task arguments unchanged while paging. Page size -may change. Every page retains the same plan, collection, and capsule hashes. A `stale_cursor` -means that the generation, policy, plan, or collection changed; discard earlier pages and restart. - -`docforge_get_generation_diff` paginates only the details retained in the latest bounded receipt. -Its summary counts and full collection hash still cover permanently truncated details. The cursor -binds the exact receipt, target generation, retained and full collection hashes, receipt state, and -effective policy. A replacement receipt returns `stale_cursor`; restart from its first page. - Canonical application records its terminal receipt immediately after the project-owned serializer verifies the new canonical state. A later index or render refresh failure is reported as degraded derived state with remediation, not as permission to apply the same canonical change again. diff --git a/schemas/client-configuration.schema.json b/schemas/client-configuration.schema.json deleted file mode 100644 index b4cf208..0000000 --- a/schemas/client-configuration.schema.json +++ /dev/null @@ -1,894 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://docforge.local/schema/client-configuration-v1.json", - "title": "DocForge deterministic client configuration plan", - "$defs": { - "sha256": { - "type": "string", - "pattern": "^[0-9a-f]{64}$" - }, - "adapter_policy": { - "oneOf": [ - { - "type": "object", - "required": [ - "mode", - "ast_analysis", - "logic_projection", - "incremental_extraction", - "adapter_rewrite" - ], - "properties": { - "mode": { "const": "standard" }, - "ast_analysis": { "const": "allowed" }, - "logic_projection": { "const": "allowed" }, - "incremental_extraction": { "const": "allowed" }, - "adapter_rewrite": { "const": "not_requested" } - }, - "additionalProperties": false - }, - { - "type": "object", - "required": [ - "mode", - "ast_analysis", - "logic_projection", - "incremental_extraction", - "adapter_rewrite", - "blocked_tools", - "instruction" - ], - "properties": { - "mode": { "const": "preserve-no-ast" }, - "ast_analysis": { "const": "forbidden" }, - "logic_projection": { "const": "forbidden" }, - "incremental_extraction": { "const": "allowed" }, - "adapter_rewrite": { "const": "forbidden" }, - "blocked_tools": { - "const": ["docforge_get_logic"] - }, - "instruction": { - "const": "Preserve the existing adapter extraction strategy. Do not add Python AST, Tree-sitter, compiler-AST, or function-Logic extraction. Non-AST incremental fingerprinting and caching remain allowed." - } - }, - "additionalProperties": false - } - ] - }, - "effective_policy": { - "type": "object", - "required": [ - "schema_version", - "capability_mode", - "capability_source", - "adapter_evolution", - "ast_analysis", - "logic_indexing", - "synchronization", - "integrity", - "manual_render", - "graph_render", - "live_viewer", - "profiling", - "blocked_tools", - "prohibitions", - "precedence" - ], - "properties": { - "schema_version": { "const": 1 }, - "capability_mode": { - "enum": ["read", "proposal", "application"] - }, - "capability_source": { "const": "explicit" }, - "adapter_evolution": { "enum": ["allowed", "preserve"] }, - "ast_analysis": { "enum": ["allowed", "forbidden"] }, - "logic_indexing": { "enum": ["full", "off"] }, - "synchronization": { "const": "automatic" }, - "integrity": { "const": "validated" }, - "manual_render": { "enum": ["auto", "explicit", "disabled"] }, - "graph_render": { "const": "disabled" }, - "live_viewer": { "const": "on-demand" }, - "profiling": { "const": "disabled" }, - "blocked_tools": { - "type": "array", - "maxItems": 1, - "items": { "const": "docforge_get_logic" }, - "uniqueItems": true - }, - "prohibitions": { - "type": "array", - "minItems": 7, - "maxItems": 11, - "items": { - "enum": [ - "arbitrary_file_access", - "arbitrary_renderer_execution", - "shell_execution", - "git_mutation", - "deployment", - "publication", - "project_switching", - "adapter_ast_upgrade", - "tree_sitter_upgrade", - "compiler_ast_upgrade", - "function_logic_extraction" - ] - }, - "uniqueItems": true - }, - "precedence": { - "const": [ - "core_safety", - "explicit_binding", - "no_ast_shorthand", - "resource_availability" - ] - } - }, - "additionalProperties": false - }, - "diagnostics": { - "type": "object", - "required": [ - "schema_version", - "operation", - "outcome", - "elapsed_ns", - "stages", - "counters" - ], - "properties": { - "schema_version": { "const": 1 }, - "operation": { "const": "cli.configure" }, - "outcome": { "const": "ok" }, - "elapsed_ns": { "type": "integer", "minimum": 0 }, - "stages": { - "type": "object", - "maxProperties": 14, - "propertyNames": { - "enum": [ - "source.generation", - "source.parse", - "adapter.projection", - "adapter.extract", - "index.check", - "index.synchronize", - "index.build", - "index.read", - "render.status", - "render.prepare", - "render.output_hash", - "visualization.status", - "viewer.manager", - "mcp.runtime_validation" - ] - }, - "additionalProperties": { - "type": "object", - "required": ["calls", "elapsed_ns"], - "properties": { - "calls": { "type": "integer", "minimum": 1 }, - "elapsed_ns": { "type": "integer", "minimum": 0 } - }, - "additionalProperties": false - } - }, - "counters": { - "type": "object", - "required": [ - "project_loads", - "source_files_parsed", - "source_bytes_parsed", - "adapter_projection_loads", - "adapter_source_extractions", - "source_generation_checks", - "index_checks", - "index_synchronizations", - "index_builds", - "render_prepare_calls", - "render_output_bytes_built", - "render_output_bytes_hashed", - "viewer_manager_requests" - ], - "additionalProperties": { - "type": "integer", - "minimum": 0 - }, - "maxProperties": 13 - } - }, - "additionalProperties": false - } - }, - "type": "object", - "required": [ - "status", - "schema_version", - "operation", - "action", - "client", - "server_name", - "project", - "binding", - "effective_policy", - "artifact", - "configuration_hash", - "warnings" - ], - "properties": { - "status": { "const": "ok" }, - "schema_version": { "const": 1 }, - "operation": { "const": "client.configure" }, - "action": { "enum": ["preview", "write"] }, - "client": { "enum": ["codex", "claude", "openclaw"] }, - "server_name": { - "type": "string", - "pattern": "^[a-z0-9][a-z0-9_-]{0,63}$" - }, - "project": { - "type": "object", - "required": [ - "project_id", - "project_root", - "project_root_fingerprint", - "adapter" - ], - "properties": { - "project_id": { "type": "string", "minLength": 1 }, - "project_root": { "type": "string", "minLength": 1 }, - "project_root_fingerprint": { - "type": "string", - "pattern": "^[0-9a-f]{16}$" - }, - "adapter": { "type": "string", "minLength": 1 } - }, - "additionalProperties": false - }, - "binding": { - "type": "object", - "required": [ - "transport", - "capability_mode", - "adapter_policy", - "render_policy", - "command", - "args", - "environment", - "timeouts" - ], - "properties": { - "transport": { "const": "stdio" }, - "capability_mode": { - "enum": ["read", "proposal", "application"] - }, - "adapter_policy": { "$ref": "#/$defs/adapter_policy" }, - "render_policy": { - "type": "object", - "required": ["manual", "graph", "live_viewer"], - "properties": { - "manual": { "enum": ["auto", "explicit", "disabled"] }, - "graph": { "const": "disabled" }, - "live_viewer": { "const": "on-demand" } - }, - "additionalProperties": false - }, - "command": { "type": "string", "minLength": 1 }, - "args": { - "type": "array", - "minItems": 7, - "maxItems": 64, - "prefixItems": [ - { "const": "-I" }, - { "const": "-m" }, - { "const": "docforge.mcp_server" }, - { "const": "--project-root" }, - { "type": "string", "minLength": 1, "maxLength": 4096 }, - { "const": "--capability-mode" }, - { "enum": ["read", "proposal", "application"] } - ], - "items": { - "type": "string", - "minLength": 1, - "maxLength": 4096 - } - }, - "environment": { - "type": "object", - "maxProperties": 0 - }, - "timeouts": { - "type": "object", - "required": ["startup_seconds", "tool_seconds"], - "properties": { - "startup_seconds": { - "type": "integer", - "minimum": 1, - "maximum": 3600 - }, - "tool_seconds": { - "type": "integer", - "minimum": 1, - "maximum": 86400 - } - }, - "additionalProperties": false - } - }, - "additionalProperties": false - }, - "effective_policy": { "$ref": "#/$defs/effective_policy" }, - "artifact": { - "type": "object", - "required": [ - "format", - "content", - "content_sha256", - "output_path", - "write_state", - "durability" - ], - "properties": { - "format": { - "enum": [ - "codex-toml-fragment-v1", - "claude-json-fragment-v1", - "openclaw-json-fragment-v1" - ] - }, - "content": { - "type": "string", - "minLength": 1, - "maxLength": 65536 - }, - "content_sha256": { "$ref": "#/$defs/sha256" }, - "output_path": { - "type": ["string", "null"], - "minLength": 1 - }, - "write_state": { - "enum": ["not_requested", "created", "unchanged"] - }, - "durability": { - "enum": ["not_applicable", "confirmed", "unconfirmed"] - } - }, - "additionalProperties": false - }, - "configuration_hash": { "$ref": "#/$defs/sha256" }, - "warnings": { - "type": "array", - "maxItems": 8, - "items": { - "type": "object", - "required": ["code"], - "properties": { - "code": { - "enum": [ - "timeout_format_unverified", - "publication_durability_unconfirmed", - "publication_location_unconfirmed", - "publication_binding_unconfirmed" - ] - } - }, - "additionalProperties": false - } - }, - "diagnostics": { "$ref": "#/$defs/diagnostics" } - }, - "allOf": [ - { - "if": { - "properties": { - "warnings": { - "contains": { - "properties": { - "code": { "const": "publication_binding_unconfirmed" } - }, - "required": ["code"] - } - } - }, - "required": ["warnings"] - }, - "then": { - "properties": { - "artifact": { - "properties": { - "output_path": { "type": "null" }, - "write_state": { "const": "created" }, - "durability": { "const": "unconfirmed" } - } - } - } - } - }, - { - "if": { - "properties": { "client": { "const": "codex" } }, - "required": ["client"] - }, - "then": { - "properties": { - "artifact": { - "properties": { - "format": { "const": "codex-toml-fragment-v1" } - } - }, - "warnings": { - "not": { - "contains": { - "properties": { - "code": { "const": "timeout_format_unverified" } - }, - "required": ["code"] - } - } - } - } - } - }, - { - "if": { - "properties": { "client": { "const": "openclaw" } }, - "required": ["client"] - }, - "then": { - "properties": { - "artifact": { - "properties": { - "format": { "const": "openclaw-json-fragment-v1" } - } - }, - "warnings": { - "not": { - "contains": { - "properties": { - "code": { "const": "timeout_format_unverified" } - }, - "required": ["code"] - } - } - } - } - } - }, - { - "if": { - "properties": { "client": { "const": "claude" } }, - "required": ["client"] - }, - "then": { - "properties": { - "artifact": { - "properties": { - "format": { "const": "claude-json-fragment-v1" } - } - }, - "warnings": { - "contains": { - "properties": { - "code": { "const": "timeout_format_unverified" } - }, - "required": ["code"] - } - } - } - } - }, - { - "if": { - "properties": { "action": { "const": "preview" } }, - "required": ["action"] - }, - "then": { - "properties": { - "artifact": { - "properties": { - "output_path": { "type": "null" }, - "write_state": { "const": "not_requested" }, - "durability": { "const": "not_applicable" } - } - } - } - }, - "else": { - "properties": { - "artifact": { - "properties": { - "output_path": { - "type": ["string", "null"], - "minLength": 1 - }, - "write_state": { "enum": ["created", "unchanged"] } - }, - "allOf": [ - { - "if": { - "properties": { "write_state": { "const": "created" } }, - "required": ["write_state"] - }, - "then": { - "properties": { - "durability": { "enum": ["confirmed", "unconfirmed"] } - } - }, - "else": { - "properties": { - "durability": { "const": "not_applicable" } - } - } - } - ] - } - } - } - }, - { - "if": { - "properties": { - "binding": { - "properties": { "capability_mode": { "const": "read" } }, - "required": ["capability_mode"] - } - }, - "required": ["binding"] - }, - "then": { - "properties": { - "binding": { - "properties": { - "args": { - "prefixItems": [{}, {}, {}, {}, {}, {}, { "const": "read" }] - } - } - }, - "effective_policy": { - "properties": { "capability_mode": { "const": "read" } } - } - } - } - }, - { - "if": { - "properties": { - "binding": { - "properties": { "capability_mode": { "const": "proposal" } }, - "required": ["capability_mode"] - } - }, - "required": ["binding"] - }, - "then": { - "properties": { - "binding": { - "properties": { - "args": { - "prefixItems": [{}, {}, {}, {}, {}, {}, { "const": "proposal" }] - } - } - }, - "effective_policy": { - "properties": { "capability_mode": { "const": "proposal" } } - } - } - } - }, - { - "if": { - "properties": { - "binding": { - "properties": { "capability_mode": { "const": "application" } }, - "required": ["capability_mode"] - } - }, - "required": ["binding"] - }, - "then": { - "properties": { - "binding": { - "properties": { - "args": { - "prefixItems": [{}, {}, {}, {}, {}, {}, { "const": "application" }] - } - } - }, - "effective_policy": { - "properties": { "capability_mode": { "const": "application" } } - } - } - } - }, - { - "if": { - "properties": { - "binding": { - "properties": { - "adapter_policy": { - "properties": { "mode": { "const": "preserve-no-ast" } }, - "required": ["mode"] - } - }, - "required": ["adapter_policy"] - } - }, - "required": ["binding"] - }, - "then": { - "properties": { - "binding": { - "properties": { - "args": { - "contains": { "const": "--no-ast" }, - "minContains": 1, - "maxContains": 1 - } - } - }, - "effective_policy": { - "properties": { - "adapter_evolution": { "const": "preserve" }, - "ast_analysis": { "const": "forbidden" }, - "logic_indexing": { "const": "off" }, - "blocked_tools": { "const": ["docforge_get_logic"] }, - "prohibitions": { - "const": [ - "arbitrary_file_access", - "arbitrary_renderer_execution", - "shell_execution", - "git_mutation", - "deployment", - "publication", - "project_switching", - "adapter_ast_upgrade", - "tree_sitter_upgrade", - "compiler_ast_upgrade", - "function_logic_extraction" - ] - } - } - } - } - }, - "else": { - "properties": { - "binding": { - "properties": { - "args": { - "not": { - "contains": { "const": "--no-ast" } - } - } - } - }, - "effective_policy": { - "properties": { - "adapter_evolution": { "const": "allowed" }, - "ast_analysis": { "const": "allowed" }, - "logic_indexing": { "const": "full" }, - "blocked_tools": { "const": [] }, - "prohibitions": { - "const": [ - "arbitrary_file_access", - "arbitrary_renderer_execution", - "shell_execution", - "git_mutation", - "deployment", - "publication", - "project_switching" - ] - } - } - } - } - } - }, - { - "if": { - "properties": { - "binding": { - "properties": { - "render_policy": { - "properties": { "manual": { "const": "auto" } }, - "required": ["manual"] - } - }, - "required": ["render_policy"] - } - }, - "required": ["binding"] - }, - "then": { - "properties": { - "effective_policy": { - "properties": { "manual_render": { "const": "auto" } } - } - } - } - }, - { - "if": { - "properties": { - "binding": { - "properties": { - "render_policy": { - "properties": { "manual": { "const": "explicit" } }, - "required": ["manual"] - } - }, - "required": ["render_policy"] - } - }, - "required": ["binding"] - }, - "then": { - "properties": { - "effective_policy": { - "properties": { "manual_render": { "const": "explicit" } } - } - } - } - }, - { - "if": { - "properties": { - "binding": { - "properties": { - "render_policy": { - "properties": { "manual": { "const": "disabled" } }, - "required": ["manual"] - } - }, - "required": ["render_policy"] - } - }, - "required": ["binding"] - }, - "then": { - "properties": { - "effective_policy": { - "properties": { "manual_render": { "const": "disabled" } } - } - } - } - }, - { - "if": { - "properties": { - "artifact": { - "properties": { - "durability": { "const": "unconfirmed" } - }, - "required": ["durability"] - } - }, - "required": ["artifact"] - }, - "then": { - "properties": { - "artifact": { - "properties": { - "write_state": { "const": "created" } - } - }, - "warnings": { - "anyOf": [ - { - "contains": { - "properties": { - "code": { "const": "publication_durability_unconfirmed" } - }, - "required": ["code"] - } - }, - { - "contains": { - "properties": { - "code": { "const": "publication_location_unconfirmed" } - }, - "required": ["code"] - } - }, - { - "contains": { - "properties": { - "code": { "const": "publication_binding_unconfirmed" } - }, - "required": ["code"] - } - } - ] - } - } - } - }, - { - "if": { - "properties": { - "warnings": { - "contains": { - "properties": { - "code": { "const": "publication_location_unconfirmed" } - }, - "required": ["code"] - } - } - }, - "required": ["warnings"] - }, - "then": { - "properties": { - "artifact": { - "properties": { - "output_path": { "type": "null" }, - "write_state": { "const": "created" }, - "durability": { "const": "unconfirmed" } - } - } - } - } - }, - { - "if": { - "properties": { - "action": { "const": "write" }, - "artifact": { - "properties": { "output_path": { "type": "null" } }, - "required": ["output_path"] - } - }, - "required": ["action", "artifact"] - }, - "then": { - "properties": { - "warnings": { - "anyOf": [ - { - "contains": { - "properties": { - "code": { "const": "publication_location_unconfirmed" } - }, - "required": ["code"] - } - }, - { - "contains": { - "properties": { - "code": { "const": "publication_binding_unconfirmed" } - }, - "required": ["code"] - } - } - ] - } - } - } - }, - { - "if": { - "properties": { - "warnings": { - "contains": { - "properties": { - "code": { "const": "publication_durability_unconfirmed" } - }, - "required": ["code"] - } - } - }, - "required": ["warnings"] - }, - "then": { - "properties": { - "artifact": { - "properties": { - "write_state": { "const": "created" }, - "durability": { "const": "unconfirmed" } - } - } - } - } - } - ], - "additionalProperties": false -} diff --git a/schemas/context-capsule.schema.json b/schemas/context-capsule.schema.json deleted file mode 100644 index 8e9096b..0000000 --- a/schemas/context-capsule.schema.json +++ /dev/null @@ -1,435 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://docforge.local/schema/context-capsule-v1.json", - "title": "DocForge task context capsule", - "$defs": { - "sha256": { - "type": "string", - "pattern": "^[0-9a-f]{64}$" - }, - "pagination": { - "type": "object", - "required": [ - "schema_version", - "kind", - "returned_count", - "limit", - "total_count", - "has_more", - "next_cursor" - ], - "properties": { - "schema_version": { "const": 1 }, - "kind": { "const": "task-context.items" }, - "returned_count": { "type": "integer", "minimum": 0 }, - "limit": { "type": "integer", "minimum": 1 }, - "total_count": { "type": "integer", "minimum": 0 }, - "has_more": { "type": "boolean" }, - "next_cursor": { - "type": ["string", "null"], - "minLength": 1, - "maxLength": 8192 - } - }, - "additionalProperties": false - }, - "category": { - "enum": [ - "structure", - "implementation", - "dependency", - "execution", - "data", - "evidence", - "context", - "unclassified" - ] - }, - "step": { - "type": "object", - "required": [ - "step_id", - "operation", - "relation_scope", - "relation_set_hash", - "direction", - "depth", - "limit", - "required", - "evidence_role" - ], - "properties": { - "step_id": { "type": "string", "minLength": 1, "maxLength": 64 }, - "operation": { - "enum": ["exact", "search", "outgoing", "incoming", "metadata"] - }, - "relation_scope": { "enum": ["none", "project_allowed"] }, - "relation_set_hash": { - "oneOf": [ - { "$ref": "#/$defs/sha256" }, - { "type": "null" } - ] - }, - "direction": { "enum": ["none", "outgoing", "incoming"] }, - "depth": { "type": "integer", "minimum": 0 }, - "limit": { "type": "integer", "minimum": 1 }, - "required": { "type": "boolean" }, - "evidence_role": { "type": "string", "minLength": 1, "maxLength": 64 } - }, - "additionalProperties": false - }, - "requirement": { - "type": "object", - "required": ["requirement_id", "check", "category", "required"], - "properties": { - "requirement_id": { "type": "string", "minLength": 1, "maxLength": 128 }, - "check": { "const": "selected_relation_category" }, - "category": { "$ref": "#/$defs/category" }, - "required": { "const": true } - }, - "additionalProperties": false - }, - "plan": { - "type": "object", - "required": [ - "schema_version", - "planner", - "task_kind", - "request_hash", - "effective_policy_hash", - "focus_node_id", - "limits", - "category_order", - "steps", - "requirements", - "plan_hash" - ], - "properties": { - "schema_version": { "const": 1 }, - "planner": { - "type": "object", - "required": ["id", "version"], - "properties": { - "id": { "const": "docforge.core.task-context" }, - "version": { "const": 1 } - }, - "additionalProperties": false - }, - "task_kind": { - "enum": [ - "change", - "implementation", - "failure", - "ownership", - "test", - "operation", - "release" - ] - }, - "request_hash": { "$ref": "#/$defs/sha256" }, - "effective_policy_hash": { "$ref": "#/$defs/sha256" }, - "focus_node_id": { "type": ["string", "null"], "maxLength": 256 }, - "limits": { - "type": "object", - "required": [ - "max_evidence", - "max_tokens", - "max_depth", - "max_candidate_edges" - ], - "properties": { - "max_evidence": { "type": "integer", "minimum": 1, "maximum": 1000 }, - "max_tokens": { "type": "integer", "minimum": 1 }, - "max_depth": { "type": "integer", "minimum": 0 }, - "max_candidate_edges": { - "type": "integer", - "minimum": 1, - "maximum": 100000 - } - }, - "additionalProperties": false - }, - "category_order": { - "type": "array", - "minItems": 8, - "maxItems": 8, - "items": { "$ref": "#/$defs/category" }, - "uniqueItems": true - }, - "steps": { - "type": "array", - "minItems": 4, - "maxItems": 4, - "items": { "$ref": "#/$defs/step" } - }, - "requirements": { - "type": "array", - "minItems": 1, - "maxItems": 8, - "items": { "$ref": "#/$defs/requirement" } - }, - "plan_hash": { "$ref": "#/$defs/sha256" } - }, - "additionalProperties": false - }, - "relationship": { - "type": "object", - "required": [ - "source_id", - "relation", - "target_id", - "direction", - "category", - "provenance" - ], - "properties": { - "source_id": { "type": "string", "minLength": 1 }, - "relation": { "type": "string", "minLength": 1 }, - "target_id": { "type": "string", "minLength": 1 }, - "direction": { "enum": ["outgoing", "incoming"] }, - "category": { "$ref": "#/$defs/category" }, - "provenance": { - "const": "validated_graph_edge_without_source_provenance" - } - }, - "additionalProperties": false - }, - "evidence": { - "type": "object", - "required": [ - "evidence_hash", - "role", - "reason_code", - "node_id", - "title", - "family", - "authority", - "status", - "tags", - "summary", - "text", - "estimated_tokens", - "source", - "depth", - "relationship_path", - "relationship_reasons", - "provenance_limitations" - ], - "properties": { - "evidence_hash": { "$ref": "#/$defs/sha256" }, - "role": { "enum": ["focus", "related"] }, - "reason_code": { - "enum": ["exact_focus", "lexical_focus", "relationship_path"] - }, - "node_id": { "type": "string", "minLength": 1 }, - "title": { "type": "string" }, - "family": { "type": "string" }, - "authority": { "type": "string" }, - "status": { "type": "string" }, - "tags": { - "type": "array", - "items": { "type": "string" } - }, - "summary": { "type": "string" }, - "text": { "type": "string" }, - "estimated_tokens": { "type": "integer", "minimum": 1 }, - "source": { - "type": "object", - "required": ["path", "anchor", "content_hash"], - "properties": { - "path": { "type": "string", "minLength": 1 }, - "anchor": { "type": ["string", "null"] }, - "content_hash": { "$ref": "#/$defs/sha256" } - }, - "additionalProperties": false - }, - "depth": { "type": "integer", "minimum": 0 }, - "relationship_path": { - "type": "array", - "items": { "$ref": "#/$defs/relationship" } - }, - "relationship_reasons": { - "type": "array", - "items": { "$ref": "#/$defs/relationship" }, - "uniqueItems": true - }, - "provenance_limitations": { - "const": [ - "evidence_type_unavailable", - "extractor_identity_unavailable", - "relationship_provenance_unavailable", - "observation_time_unavailable" - ] - } - }, - "additionalProperties": false - }, - "gap": { - "type": "object", - "required": [ - "code", - "requirement_id", - "category", - "state", - "check_complete", - "detail" - ], - "properties": { - "code": { - "enum": [ - "focus_not_found", - "focus_ambiguous", - "category_not_declared", - "no_selected_evidence", - "evidence_incomplete", - "unclassified_relation" - ] - }, - "requirement_id": { "type": "string", "minLength": 1 }, - "category": { - "oneOf": [ - { "$ref": "#/$defs/category" }, - { "type": "null" } - ] - }, - "state": { "enum": ["missing", "incomplete", "blocked", "limitation"] }, - "check_complete": { "type": "boolean" }, - "detail": { "type": "string", "minLength": 1, "maxLength": 1000 } - }, - "additionalProperties": false - }, - "omission": { - "type": "object", - "required": ["code", "subject", "detail_hash"], - "properties": { - "code": { - "enum": [ - "result_limit", - "token_budget", - "response_limit", - "edge_examination_limit", - "unclassified_relation_limit" - ] - }, - "subject": { "type": "string", "minLength": 1, "maxLength": 256 }, - "detail_hash": { "$ref": "#/$defs/sha256" } - }, - "additionalProperties": false - } - }, - "type": "object", - "required": [ - "schema_version", - "state", - "task_kind", - "generation", - "plan", - "focus", - "evidence", - "gaps", - "omissions", - "summary", - "collection_hash", - "capsule_hash" - ], - "properties": { - "schema_version": { "const": 1 }, - "state": { "enum": ["complete", "incomplete", "blocked"] }, - "task_kind": { - "enum": [ - "change", - "implementation", - "failure", - "ownership", - "test", - "operation", - "release" - ] - }, - "generation": { - "type": "object", - "required": [ - "project_id", - "project_root_fingerprint", - "adapter", - "revision", - "source_hash", - "index_schema_version" - ], - "properties": { - "project_id": { "type": "string", "minLength": 1 }, - "project_root_fingerprint": { - "type": "string", - "pattern": "^[0-9a-f]{16}$" - }, - "adapter": { "type": "string", "minLength": 1 }, - "revision": { "type": "string", "minLength": 1 }, - "source_hash": { "$ref": "#/$defs/sha256" }, - "index_schema_version": { "type": "integer", "minimum": 1 } - }, - "additionalProperties": false - }, - "plan": { "$ref": "#/$defs/plan" }, - "focus": { - "type": "object", - "required": ["state", "node_id", "candidate_count"], - "properties": { - "state": { "enum": ["resolved", "not_found", "ambiguous"] }, - "node_id": { "type": ["string", "null"] }, - "candidate_count": { "type": "integer", "minimum": 0 } - }, - "additionalProperties": false - }, - "evidence": { - "type": "array", - "maxItems": 10000, - "items": { "$ref": "#/$defs/evidence" } - }, - "gaps": { - "type": "array", - "maxItems": 10000, - "items": { "$ref": "#/$defs/gap" } - }, - "omissions": { - "type": "array", - "maxItems": 10000, - "items": { "$ref": "#/$defs/omission" } - }, - "summary": { - "type": "object", - "required": [ - "evidence_count", - "gap_count", - "omission_count", - "selected_count", - "examined_edge_count", - "estimated_tokens", - "unclassified_relations" - ], - "properties": { - "evidence_count": { "type": "integer", "minimum": 0 }, - "gap_count": { "type": "integer", "minimum": 0 }, - "omission_count": { "type": "integer", "minimum": 0 }, - "selected_count": { "type": "integer", "minimum": 0 }, - "examined_edge_count": { "type": "integer", "minimum": 0 }, - "estimated_tokens": { "type": "integer", "minimum": 0 }, - "unclassified_relations": { - "type": "array", - "maxItems": 10000, - "items": { "type": "string", "minLength": 1 }, - "uniqueItems": true - }, - "page_evidence_count": { "type": "integer", "minimum": 0 }, - "page_omission_count": { "type": "integer", "minimum": 0 }, - "page_item_count": { "type": "integer", "minimum": 0 } - }, - "additionalProperties": false - }, - "collection_hash": { "$ref": "#/$defs/sha256" }, - "capsule_hash": { "$ref": "#/$defs/sha256" }, - "page_state": { "enum": ["complete", "incomplete", "blocked"] }, - "page_hash": { "$ref": "#/$defs/sha256" }, - "pagination": { "$ref": "#/$defs/pagination" } - }, - "additionalProperties": false -} diff --git a/schemas/doctor-result.schema.json b/schemas/doctor-result.schema.json deleted file mode 100644 index 6e7eb4a..0000000 --- a/schemas/doctor-result.schema.json +++ /dev/null @@ -1,466 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://docforge.local/schema/doctor-result-v1.json", - "title": "DocForge bounded read-only integration doctor result", - "$defs": { - "diagnostics": { - "type": "object", - "required": [ - "schema_version", - "operation", - "outcome", - "elapsed_ns", - "stages", - "counters" - ], - "properties": { - "schema_version": { "const": 1 }, - "operation": { "const": "cli.doctor" }, - "outcome": { "const": "ok" }, - "elapsed_ns": { "type": "integer", "minimum": 0 }, - "stages": { - "type": "object", - "maxProperties": 14, - "propertyNames": { - "enum": [ - "source.generation", - "source.parse", - "adapter.projection", - "adapter.extract", - "index.check", - "index.synchronize", - "index.build", - "index.read", - "render.status", - "render.prepare", - "render.output_hash", - "visualization.status", - "viewer.manager", - "mcp.runtime_validation" - ] - }, - "additionalProperties": { - "type": "object", - "required": ["calls", "elapsed_ns"], - "properties": { - "calls": { "type": "integer", "minimum": 1 }, - "elapsed_ns": { "type": "integer", "minimum": 0 } - }, - "additionalProperties": false - } - }, - "counters": { - "type": "object", - "required": [ - "project_loads", - "source_files_parsed", - "source_bytes_parsed", - "adapter_projection_loads", - "adapter_source_extractions", - "source_generation_checks", - "index_checks", - "index_synchronizations", - "index_builds", - "render_prepare_calls", - "render_output_bytes_built", - "render_output_bytes_hashed", - "viewer_manager_requests" - ], - "additionalProperties": { - "type": "integer", - "minimum": 0 - }, - "maxProperties": 13 - } - }, - "additionalProperties": false - } - }, - "type": "object", - "required": [ - "status", - "schema_version", - "doctor_state", - "client", - "project", - "config", - "summary", - "guarantees", - "checks" - ], - "properties": { - "status": { "const": "ok" }, - "schema_version": { "const": 1 }, - "doctor_state": { "enum": ["healthy", "degraded", "unhealthy"] }, - "client": { "enum": ["codex", "claude", "openclaw"] }, - "project": { - "type": "object", - "required": [ - "project_id", - "project_root", - "project_root_fingerprint", - "adapter" - ], - "properties": { - "project_id": { "type": "string", "minLength": 1, "maxLength": 128 }, - "project_root": { "type": "string", "minLength": 1, "maxLength": 4096 }, - "project_root_fingerprint": { - "type": "string", - "pattern": "^[0-9a-f]{16}$" - }, - "adapter": { "type": "string", "minLength": 1, "maxLength": 256 } - }, - "additionalProperties": false - }, - "config": { - "type": "object", - "required": ["path", "server_name"], - "properties": { - "path": { "type": "string", "minLength": 1, "maxLength": 4096 }, - "server_name": { - "type": ["string", "null"], - "maxLength": 256 - } - }, - "additionalProperties": false - }, - "summary": { - "type": "object", - "required": ["passed", "warning", "failed", "skipped"], - "properties": { - "passed": { "type": "integer", "minimum": 0, "maximum": 14 }, - "warning": { "type": "integer", "minimum": 0, "maximum": 14 }, - "failed": { "type": "integer", "minimum": 0, "maximum": 14 }, - "skipped": { "type": "integer", "minimum": 0, "maximum": 14 } - }, - "additionalProperties": false - }, - "guarantees": { - "type": "object", - "required": [ - "read_only", - "project_loads", - "adapter_projection_loads", - "adapter_source_extractions", - "sqlite_opens", - "index_checks", - "index_synchronizations", - "index_builds", - "renders", - "viewer_operations", - "client_config_writes", - "configured_command_executions" - ], - "properties": { - "read_only": { "const": true }, - "project_loads": { "const": 0 }, - "adapter_projection_loads": { "const": 0 }, - "adapter_source_extractions": { "const": 0 }, - "sqlite_opens": { "const": 0 }, - "index_checks": { "const": 0 }, - "index_synchronizations": { "const": 0 }, - "index_builds": { "const": 0 }, - "renders": { "const": 0 }, - "viewer_operations": { "const": 0 }, - "client_config_writes": { "const": 0 }, - "configured_command_executions": { "const": 0 } - }, - "additionalProperties": false - }, - "checks": { - "type": "array", - "minItems": 14, - "maxItems": 14, - "items": { - "type": "object", - "required": ["check_id", "state", "code", "message", "details"], - "properties": { - "check_id": { - "enum": [ - "project.binding", - "project.canonical_validation", - "client.driver", - "client.config", - "client.entry", - "server.executable", - "server.arguments", - "server.project_binding", - "policy.effective", - "policy.no_ast", - "client.timeouts", - "client.environment", - "client.tool_filter", - "derived.index" - ] - }, - "state": { - "enum": ["passed", "warning", "failed", "skipped"] - }, - "code": { - "type": "string", - "minLength": 1, - "maxLength": 128 - }, - "message": { - "type": "string", - "minLength": 1, - "maxLength": 512 - }, - "details": { - "type": "object", - "maxProperties": 16, - "propertyNames": { - "type": "string", - "minLength": 1, - "maxLength": 128 - }, - "additionalProperties": { - "oneOf": [ - { "type": "string", "maxLength": 512 }, - { "type": "integer" }, - { "type": "boolean" }, - { "type": "null" }, - { - "type": "array", - "maxItems": 16, - "items": { - "oneOf": [ - { "type": "string", "maxLength": 256 }, - { "type": "integer" }, - { "type": "boolean" }, - { "type": "null" } - ] - } - } - ] - } - } - }, - "additionalProperties": false - } - }, - "diagnostics": { "$ref": "#/$defs/diagnostics" } - }, - "allOf": [ - { - "properties": { - "checks": { - "contains": { - "properties": { "check_id": { "const": "project.binding" } }, - "required": ["check_id"] - }, - "minContains": 1, - "maxContains": 1 - } - } - }, - { - "properties": { - "checks": { - "contains": { - "properties": { - "check_id": { "const": "project.canonical_validation" } - }, - "required": ["check_id"] - }, - "minContains": 1, - "maxContains": 1 - } - } - }, - { - "properties": { - "checks": { - "contains": { - "properties": { "check_id": { "const": "client.driver" } }, - "required": ["check_id"] - }, - "minContains": 1, - "maxContains": 1 - } - } - }, - { - "properties": { - "checks": { - "contains": { - "properties": { "check_id": { "const": "client.config" } }, - "required": ["check_id"] - }, - "minContains": 1, - "maxContains": 1 - } - } - }, - { - "properties": { - "checks": { - "contains": { - "properties": { "check_id": { "const": "client.entry" } }, - "required": ["check_id"] - }, - "minContains": 1, - "maxContains": 1 - } - } - }, - { - "properties": { - "checks": { - "contains": { - "properties": { "check_id": { "const": "server.executable" } }, - "required": ["check_id"] - }, - "minContains": 1, - "maxContains": 1 - } - } - }, - { - "properties": { - "checks": { - "contains": { - "properties": { "check_id": { "const": "server.arguments" } }, - "required": ["check_id"] - }, - "minContains": 1, - "maxContains": 1 - } - } - }, - { - "properties": { - "checks": { - "contains": { - "properties": { - "check_id": { "const": "server.project_binding" } - }, - "required": ["check_id"] - }, - "minContains": 1, - "maxContains": 1 - } - } - }, - { - "properties": { - "checks": { - "contains": { - "properties": { "check_id": { "const": "policy.effective" } }, - "required": ["check_id"] - }, - "minContains": 1, - "maxContains": 1 - } - } - }, - { - "properties": { - "checks": { - "contains": { - "properties": { "check_id": { "const": "policy.no_ast" } }, - "required": ["check_id"] - }, - "minContains": 1, - "maxContains": 1 - } - } - }, - { - "properties": { - "checks": { - "contains": { - "properties": { "check_id": { "const": "client.timeouts" } }, - "required": ["check_id"] - }, - "minContains": 1, - "maxContains": 1 - } - } - }, - { - "properties": { - "checks": { - "contains": { - "properties": { "check_id": { "const": "client.environment" } }, - "required": ["check_id"] - }, - "minContains": 1, - "maxContains": 1 - } - } - }, - { - "properties": { - "checks": { - "contains": { - "properties": { "check_id": { "const": "client.tool_filter" } }, - "required": ["check_id"] - }, - "minContains": 1, - "maxContains": 1 - } - } - }, - { - "properties": { - "checks": { - "contains": { - "properties": { "check_id": { "const": "derived.index" } }, - "required": ["check_id"] - }, - "minContains": 1, - "maxContains": 1 - } - } - }, - { - "if": { - "properties": { "doctor_state": { "const": "healthy" } }, - "required": ["doctor_state"] - }, - "then": { - "properties": { - "summary": { - "properties": { - "warning": { "const": 0 }, - "failed": { "const": 0 } - } - } - } - } - }, - { - "if": { - "properties": { "doctor_state": { "const": "degraded" } }, - "required": ["doctor_state"] - }, - "then": { - "properties": { - "summary": { - "properties": { - "warning": { "minimum": 1 }, - "failed": { "const": 0 } - } - } - } - } - }, - { - "if": { - "properties": { "doctor_state": { "const": "unhealthy" } }, - "required": ["doctor_state"] - }, - "then": { - "properties": { - "summary": { - "properties": { - "failed": { "minimum": 1 } - } - } - } - } - } - ], - "additionalProperties": false -} diff --git a/schemas/generation-diff-page.schema.json b/schemas/generation-diff-page.schema.json deleted file mode 100644 index f2894ec..0000000 --- a/schemas/generation-diff-page.schema.json +++ /dev/null @@ -1,430 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://docforge.local/schema/generation-diff-page-v1.json", - "title": "DocForge bounded latest-generation diff page", - "$defs": { - "sha256": { - "type": "string", - "pattern": "^[0-9a-f]{64}$" - }, - "generation": { - "type": "object", - "required": [ - "revision", - "source_hash", - "node_count", - "node_hash", - "edge_count", - "edge_hash", - "index_schema_version" - ], - "properties": { - "revision": { "type": "string", "minLength": 1 }, - "source_hash": { "$ref": "#/$defs/sha256" }, - "node_count": { "type": "integer", "minimum": 0 }, - "node_hash": { "$ref": "#/$defs/sha256" }, - "edge_count": { "type": "integer", "minimum": 0 }, - "edge_hash": { "$ref": "#/$defs/sha256" }, - "index_schema_version": { "type": "integer", "minimum": 1 } - }, - "additionalProperties": false - }, - "summary": { - "type": "object", - "required": [ - "nodes_added", - "nodes_removed", - "nodes_changed", - "edges_added", - "edges_removed", - "total_changes" - ], - "properties": { - "nodes_added": { "type": "integer", "minimum": 0 }, - "nodes_removed": { "type": "integer", "minimum": 0 }, - "nodes_changed": { "type": "integer", "minimum": 0 }, - "edges_added": { "type": "integer", "minimum": 0 }, - "edges_removed": { "type": "integer", "minimum": 0 }, - "total_changes": { "type": "integer", "minimum": 0 } - }, - "additionalProperties": false - }, - "node_item": { - "type": "object", - "required": [ - "entity", - "change", - "node_id", - "before_content_hash", - "after_content_hash", - "before_source_path", - "after_source_path", - "before_node_hash", - "after_node_hash", - "changed_fields", - "item_hash" - ], - "properties": { - "entity": { "const": "node" }, - "change": { "enum": ["added", "removed", "changed"] }, - "node_id": { "type": "string", "minLength": 1 }, - "before_content_hash": { - "oneOf": [{ "$ref": "#/$defs/sha256" }, { "type": "null" }] - }, - "after_content_hash": { - "oneOf": [{ "$ref": "#/$defs/sha256" }, { "type": "null" }] - }, - "before_source_path": { "type": ["string", "null"] }, - "after_source_path": { "type": ["string", "null"] }, - "before_node_hash": { - "oneOf": [{ "$ref": "#/$defs/sha256" }, { "type": "null" }] - }, - "after_node_hash": { - "oneOf": [{ "$ref": "#/$defs/sha256" }, { "type": "null" }] - }, - "changed_fields": { - "type": "array", - "uniqueItems": true, - "items": { - "enum": [ - "title", - "family", - "authority", - "status", - "tags", - "summary", - "content", - "source_path", - "source_anchor", - "content_hash" - ] - } - }, - "item_hash": { "$ref": "#/$defs/sha256" } - }, - "allOf": [ - { - "if": { - "properties": { "change": { "const": "added" } }, - "required": ["change"] - }, - "then": { - "properties": { - "before_content_hash": { "type": "null" }, - "before_source_path": { "type": "null" }, - "before_node_hash": { "type": "null" }, - "after_content_hash": { "$ref": "#/$defs/sha256" }, - "after_source_path": { "type": "string", "minLength": 1 }, - "after_node_hash": { "$ref": "#/$defs/sha256" }, - "changed_fields": { "maxItems": 0 } - } - } - }, - { - "if": { - "properties": { "change": { "const": "removed" } }, - "required": ["change"] - }, - "then": { - "properties": { - "before_content_hash": { "$ref": "#/$defs/sha256" }, - "before_source_path": { "type": "string", "minLength": 1 }, - "before_node_hash": { "$ref": "#/$defs/sha256" }, - "after_content_hash": { "type": "null" }, - "after_source_path": { "type": "null" }, - "after_node_hash": { "type": "null" }, - "changed_fields": { "maxItems": 0 } - } - } - }, - { - "if": { - "properties": { "change": { "const": "changed" } }, - "required": ["change"] - }, - "then": { - "properties": { - "before_content_hash": { "$ref": "#/$defs/sha256" }, - "before_source_path": { "type": "string", "minLength": 1 }, - "before_node_hash": { "$ref": "#/$defs/sha256" }, - "after_content_hash": { "$ref": "#/$defs/sha256" }, - "after_source_path": { "type": "string", "minLength": 1 }, - "after_node_hash": { "$ref": "#/$defs/sha256" }, - "changed_fields": { "minItems": 1 } - } - } - } - ], - "additionalProperties": false - }, - "edge_item": { - "type": "object", - "required": [ - "entity", - "change", - "source_id", - "relation", - "target_id", - "item_hash" - ], - "properties": { - "entity": { "const": "edge" }, - "change": { "enum": ["added", "removed"] }, - "source_id": { "type": "string", "minLength": 1 }, - "relation": { "type": "string", "minLength": 1 }, - "target_id": { "type": "string", "minLength": 1 }, - "item_hash": { "$ref": "#/$defs/sha256" } - }, - "additionalProperties": false - }, - "receipt_header": { - "type": "object", - "required": [ - "schema_version", - "diff_semantics_version", - "project_id", - "project_root_fingerprint", - "adapter", - "kind", - "reason", - "from_generation", - "to_generation", - "summary", - "full_item_count", - "retained_item_count", - "details_truncated", - "truncation_reason", - "full_collection_hash", - "retained_collection_hash", - "index_signature", - "stored_receipt_hash" - ], - "properties": { - "schema_version": { "const": 1 }, - "diff_semantics_version": { "const": 1 }, - "project_id": { "type": "string", "minLength": 1 }, - "project_root_fingerprint": { - "type": "string", - "pattern": "^[0-9a-f]{16}$" - }, - "adapter": { "type": "string", "minLength": 1 }, - "kind": { "enum": ["baseline", "transition"] }, - "reason": { - "enum": [ - null, - "no_predecessor", - "predecessor_unsafe", - "predecessor_unsupported_schema", - "predecessor_foreign", - "predecessor_policy_incompatible", - "predecessor_corrupt", - "predecessor_unattested", - "predecessor_changed", - "no_meaningful_transition" - ] - }, - "from_generation": { - "oneOf": [{ "$ref": "#/$defs/generation" }, { "type": "null" }] - }, - "to_generation": { "$ref": "#/$defs/generation" }, - "summary": { "$ref": "#/$defs/summary" }, - "full_item_count": { "type": "integer", "minimum": 0 }, - "retained_item_count": { - "type": "integer", - "minimum": 0, - "maximum": 1000 - }, - "details_truncated": { "type": "boolean" }, - "truncation_reason": { - "enum": [null, "receipt_item_limit", "receipt_byte_limit"] - }, - "full_collection_hash": { "$ref": "#/$defs/sha256" }, - "retained_collection_hash": { "$ref": "#/$defs/sha256" }, - "index_signature": { - "type": "object", - "required": ["device", "inode", "size", "mtime_ns", "ctime_ns"], - "properties": { - "device": { "type": "integer", "minimum": 0 }, - "inode": { "type": "integer", "minimum": 0 }, - "size": { "type": "integer", "minimum": 0 }, - "mtime_ns": { "type": "integer", "minimum": 0 }, - "ctime_ns": { "type": "integer", "minimum": 0 } - }, - "additionalProperties": false - }, - "stored_receipt_hash": { "$ref": "#/$defs/sha256" } - }, - "allOf": [ - { - "if": { - "properties": { "kind": { "const": "baseline" } }, - "required": ["kind"] - }, - "then": { - "properties": { - "from_generation": { "type": "null" }, - "reason": { "not": { "type": "null" } }, - "summary": { - "properties": { - "nodes_added": { "const": 0 }, - "nodes_removed": { "const": 0 }, - "nodes_changed": { "const": 0 }, - "edges_added": { "const": 0 }, - "edges_removed": { "const": 0 }, - "total_changes": { "const": 0 } - } - }, - "full_item_count": { "const": 0 }, - "retained_item_count": { "const": 0 }, - "details_truncated": { "const": false }, - "truncation_reason": { "const": null }, - "full_collection_hash": { - "const": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945" - }, - "retained_collection_hash": { - "const": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945" - } - } - }, - "else": { - "properties": { - "from_generation": { "$ref": "#/$defs/generation" }, - "reason": { "type": "null" } - } - } - }, - { - "if": { - "properties": { "truncation_reason": { "const": null } }, - "required": ["truncation_reason"] - }, - "then": { - "properties": { - "details_truncated": { "const": false }, - "full_item_count": { "maximum": 1000 } - } - } - }, - { - "if": { - "properties": { - "truncation_reason": { "const": "receipt_item_limit" } - }, - "required": ["truncation_reason"] - }, - "then": { - "properties": { - "details_truncated": { "const": true }, - "full_item_count": { "minimum": 1001 }, - "retained_item_count": { "const": 1000 } - } - } - }, - { - "if": { - "properties": { - "truncation_reason": { "const": "receipt_byte_limit" } - }, - "required": ["truncation_reason"] - }, - "then": { - "properties": { - "details_truncated": { "const": true } - } - } - } - ], - "additionalProperties": false - }, - "page_body": { - "type": "object", - "required": [ - "page_schema_version", - "receipt_header", - "items", - "omissions", - "page_hash" - ], - "properties": { - "page_schema_version": { "const": 1 }, - "receipt_header": { "$ref": "#/$defs/receipt_header" }, - "items": { - "type": "array", - "maxItems": 1000, - "items": { - "oneOf": [ - { "$ref": "#/$defs/node_item" }, - { "$ref": "#/$defs/edge_item" } - ] - } - }, - "omissions": { - "type": "array", - "maxItems": 1, - "items": { - "type": "object", - "required": ["code", "item_hash"], - "properties": { - "code": { "const": "response_limit" }, - "item_hash": { "$ref": "#/$defs/sha256" } - }, - "additionalProperties": false - } - }, - "page_hash": { "$ref": "#/$defs/sha256" } - }, - "additionalProperties": false - }, - "pagination": { - "type": "object", - "required": [ - "schema_version", - "kind", - "returned_count", - "limit", - "total_count", - "has_more", - "next_cursor" - ], - "properties": { - "schema_version": { "const": 1 }, - "kind": { "const": "generation-diff.items" }, - "returned_count": { "type": "integer", "minimum": 0 }, - "limit": { "type": "integer", "minimum": 1, "maximum": 1000 }, - "total_count": { "type": "integer", "minimum": 0, "maximum": 1000 }, - "has_more": { "type": "boolean" }, - "next_cursor": { - "type": ["string", "null"], - "minLength": 1, - "maxLength": 8192 - } - }, - "allOf": [ - { - "if": { - "properties": { "has_more": { "const": true } }, - "required": ["has_more"] - }, - "then": { - "properties": { - "next_cursor": { "type": "string", "minLength": 1 } - } - }, - "else": { - "properties": { - "next_cursor": { "type": "null" } - } - } - } - ], - "additionalProperties": false - } - }, - "type": "object", - "required": ["generation_diff", "pagination"], - "properties": { - "generation_diff": { "$ref": "#/$defs/page_body" }, - "pagination": { "$ref": "#/$defs/pagination" } - }, - "additionalProperties": false -} diff --git a/schemas/generation-diff.schema.json b/schemas/generation-diff.schema.json deleted file mode 100644 index d4c05fc..0000000 --- a/schemas/generation-diff.schema.json +++ /dev/null @@ -1,363 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://docforge.local/schema/generation-diff-v1.json", - "title": "DocForge latest primary-graph generation diff receipt", - "$defs": { - "sha256": { - "type": "string", - "pattern": "^[0-9a-f]{64}$" - }, - "generation": { - "type": "object", - "required": [ - "revision", - "source_hash", - "node_count", - "node_hash", - "edge_count", - "edge_hash", - "index_schema_version" - ], - "properties": { - "revision": { "type": "string", "minLength": 1 }, - "source_hash": { "$ref": "#/$defs/sha256" }, - "node_count": { "type": "integer", "minimum": 0 }, - "node_hash": { "$ref": "#/$defs/sha256" }, - "edge_count": { "type": "integer", "minimum": 0 }, - "edge_hash": { "$ref": "#/$defs/sha256" }, - "index_schema_version": { "type": "integer", "minimum": 1 } - }, - "additionalProperties": false - }, - "node_item": { - "type": "object", - "required": [ - "entity", - "change", - "node_id", - "before_content_hash", - "after_content_hash", - "before_source_path", - "after_source_path", - "before_node_hash", - "after_node_hash", - "changed_fields", - "item_hash" - ], - "properties": { - "entity": { "const": "node" }, - "change": { "enum": ["added", "removed", "changed"] }, - "node_id": { "type": "string", "minLength": 1 }, - "before_content_hash": { - "oneOf": [ - { "$ref": "#/$defs/sha256" }, - { "type": "null" } - ] - }, - "after_content_hash": { - "oneOf": [ - { "$ref": "#/$defs/sha256" }, - { "type": "null" } - ] - }, - "before_source_path": { "type": ["string", "null"] }, - "after_source_path": { "type": ["string", "null"] }, - "before_node_hash": { - "oneOf": [ - { "$ref": "#/$defs/sha256" }, - { "type": "null" } - ] - }, - "after_node_hash": { - "oneOf": [ - { "$ref": "#/$defs/sha256" }, - { "type": "null" } - ] - }, - "changed_fields": { - "type": "array", - "items": { - "enum": [ - "title", - "family", - "authority", - "status", - "tags", - "summary", - "content", - "source_path", - "source_anchor", - "content_hash" - ] - }, - "uniqueItems": true - }, - "item_hash": { "$ref": "#/$defs/sha256" } - }, - "allOf": [ - { - "if": { - "properties": { "change": { "const": "added" } }, - "required": ["change"] - }, - "then": { - "properties": { - "before_content_hash": { "type": "null" }, - "before_source_path": { "type": "null" }, - "before_node_hash": { "type": "null" }, - "after_content_hash": { "$ref": "#/$defs/sha256" }, - "after_source_path": { "type": "string", "minLength": 1 }, - "after_node_hash": { "$ref": "#/$defs/sha256" }, - "changed_fields": { "maxItems": 0 } - } - } - }, - { - "if": { - "properties": { "change": { "const": "removed" } }, - "required": ["change"] - }, - "then": { - "properties": { - "before_content_hash": { "$ref": "#/$defs/sha256" }, - "before_source_path": { "type": "string", "minLength": 1 }, - "before_node_hash": { "$ref": "#/$defs/sha256" }, - "after_content_hash": { "type": "null" }, - "after_source_path": { "type": "null" }, - "after_node_hash": { "type": "null" }, - "changed_fields": { "maxItems": 0 } - } - } - }, - { - "if": { - "properties": { "change": { "const": "changed" } }, - "required": ["change"] - }, - "then": { - "properties": { - "before_content_hash": { "$ref": "#/$defs/sha256" }, - "before_source_path": { "type": "string", "minLength": 1 }, - "before_node_hash": { "$ref": "#/$defs/sha256" }, - "after_content_hash": { "$ref": "#/$defs/sha256" }, - "after_source_path": { "type": "string", "minLength": 1 }, - "after_node_hash": { "$ref": "#/$defs/sha256" }, - "changed_fields": { "minItems": 1 } - } - } - } - ], - "additionalProperties": false - }, - "edge_item": { - "type": "object", - "required": [ - "entity", - "change", - "source_id", - "relation", - "target_id", - "item_hash" - ], - "properties": { - "entity": { "const": "edge" }, - "change": { "enum": ["added", "removed"] }, - "source_id": { "type": "string", "minLength": 1 }, - "relation": { "type": "string", "minLength": 1 }, - "target_id": { "type": "string", "minLength": 1 }, - "item_hash": { "$ref": "#/$defs/sha256" } - }, - "additionalProperties": false - } - }, - "type": "object", - "required": [ - "schema_version", - "diff_semantics_version", - "project_id", - "project_root_fingerprint", - "adapter", - "kind", - "reason", - "from_generation", - "to_generation", - "summary", - "items", - "full_item_count", - "retained_item_count", - "details_truncated", - "truncation_reason", - "full_collection_hash", - "retained_collection_hash", - "index_signature", - "receipt_hash" - ], - "properties": { - "schema_version": { "const": 1 }, - "diff_semantics_version": { "const": 1 }, - "project_id": { "type": "string", "minLength": 1 }, - "project_root_fingerprint": { - "type": "string", - "pattern": "^[0-9a-f]{16}$" - }, - "adapter": { "type": "string", "minLength": 1 }, - "kind": { "enum": ["baseline", "transition"] }, - "reason": { - "enum": [ - null, - "no_predecessor", - "predecessor_unsafe", - "predecessor_unsupported_schema", - "predecessor_foreign", - "predecessor_policy_incompatible", - "predecessor_corrupt", - "predecessor_unattested", - "predecessor_changed", - "no_meaningful_transition" - ] - }, - "from_generation": { - "oneOf": [ - { "$ref": "#/$defs/generation" }, - { "type": "null" } - ] - }, - "to_generation": { "$ref": "#/$defs/generation" }, - "summary": { - "type": "object", - "required": [ - "nodes_added", - "nodes_removed", - "nodes_changed", - "edges_added", - "edges_removed", - "total_changes" - ], - "properties": { - "nodes_added": { "type": "integer", "minimum": 0 }, - "nodes_removed": { "type": "integer", "minimum": 0 }, - "nodes_changed": { "type": "integer", "minimum": 0 }, - "edges_added": { "type": "integer", "minimum": 0 }, - "edges_removed": { "type": "integer", "minimum": 0 }, - "total_changes": { "type": "integer", "minimum": 0 } - }, - "additionalProperties": false - }, - "items": { - "type": "array", - "maxItems": 1000, - "items": { - "oneOf": [ - { "$ref": "#/$defs/node_item" }, - { "$ref": "#/$defs/edge_item" } - ] - } - }, - "full_item_count": { "type": "integer", "minimum": 0 }, - "retained_item_count": { - "type": "integer", - "minimum": 0, - "maximum": 1000 - }, - "details_truncated": { "type": "boolean" }, - "truncation_reason": { - "enum": [null, "receipt_item_limit", "receipt_byte_limit"] - }, - "full_collection_hash": { "$ref": "#/$defs/sha256" }, - "retained_collection_hash": { "$ref": "#/$defs/sha256" }, - "index_signature": { - "type": "object", - "required": ["device", "inode", "size", "mtime_ns", "ctime_ns"], - "properties": { - "device": { "type": "integer", "minimum": 0 }, - "inode": { "type": "integer", "minimum": 0 }, - "size": { "type": "integer", "minimum": 0 }, - "mtime_ns": { "type": "integer", "minimum": 0 }, - "ctime_ns": { "type": "integer", "minimum": 0 } - }, - "additionalProperties": false - }, - "receipt_hash": { "$ref": "#/$defs/sha256" } - }, - "allOf": [ - { - "if": { - "properties": { "kind": { "const": "baseline" } }, - "required": ["kind"] - }, - "then": { - "properties": { - "from_generation": { "type": "null" }, - "reason": { "not": { "type": "null" } }, - "summary": { - "properties": { - "nodes_added": { "const": 0 }, - "nodes_removed": { "const": 0 }, - "nodes_changed": { "const": 0 }, - "edges_added": { "const": 0 }, - "edges_removed": { "const": 0 }, - "total_changes": { "const": 0 } - } - }, - "full_item_count": { "const": 0 }, - "retained_item_count": { "const": 0 }, - "details_truncated": { "const": false }, - "truncation_reason": { "const": null }, - "full_collection_hash": { - "const": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945" - }, - "retained_collection_hash": { - "const": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945" - } - } - }, - "else": { - "properties": { - "from_generation": { "$ref": "#/$defs/generation" }, - "reason": { "type": "null" } - } - } - }, - { - "if": { - "properties": { "truncation_reason": { "const": null } }, - "required": ["truncation_reason"] - }, - "then": { - "properties": { - "details_truncated": { "const": false }, - "full_item_count": { "maximum": 1000 } - } - } - }, - { - "if": { - "properties": { - "truncation_reason": { "const": "receipt_item_limit" } - }, - "required": ["truncation_reason"] - }, - "then": { - "properties": { - "details_truncated": { "const": true }, - "full_item_count": { "minimum": 1001 }, - "retained_item_count": { "const": 1000 } - } - } - }, - { - "if": { - "properties": { - "truncation_reason": { "const": "receipt_byte_limit" } - }, - "required": ["truncation_reason"] - }, - "then": { - "properties": { - "details_truncated": { "const": true } - } - } - } - ], - "additionalProperties": false -} diff --git a/schemas/policy.schema.json b/schemas/policy.schema.json deleted file mode 100644 index 4260a32..0000000 --- a/schemas/policy.schema.json +++ /dev/null @@ -1,62 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://docforge.local/schema/policy-v1.json", - "title": "DocForge effective process policy", - "type": "object", - "required": [ - "schema_version", - "capability_mode", - "capability_source", - "adapter_evolution", - "ast_analysis", - "logic_indexing", - "synchronization", - "integrity", - "manual_render", - "graph_render", - "live_viewer", - "profiling", - "blocked_tools", - "prohibitions", - "precedence" - ], - "properties": { - "schema_version": { "const": 1 }, - "capability_mode": { - "enum": ["read", "proposal", "application", "operator"] - }, - "capability_source": { - "enum": ["factory_default", "explicit"] - }, - "adapter_evolution": { "enum": ["allowed", "preserve"] }, - "ast_analysis": { "enum": ["allowed", "forbidden"] }, - "logic_indexing": { "enum": ["full", "off"] }, - "synchronization": { "const": "automatic" }, - "integrity": { "const": "validated" }, - "manual_render": { "enum": ["auto", "explicit", "disabled"] }, - "graph_render": { "const": "disabled" }, - "live_viewer": { "const": "on-demand" }, - "profiling": { "enum": ["enabled", "disabled"] }, - "blocked_tools": { - "type": "array", - "maxItems": 64, - "items": { "type": "string", "minLength": 1 }, - "uniqueItems": true - }, - "prohibitions": { - "type": "array", - "maxItems": 64, - "items": { "type": "string", "minLength": 1 }, - "uniqueItems": true - }, - "precedence": { - "const": [ - "core_safety", - "explicit_binding", - "no_ast_shorthand", - "resource_availability" - ] - } - }, - "additionalProperties": false -} diff --git a/schemas/result.schema.json b/schemas/result.schema.json index aac219b..ef298a3 100644 --- a/schemas/result.schema.json +++ b/schemas/result.schema.json @@ -19,7 +19,6 @@ "enum": [ "test", "benchmark.m1", - "benchmark.m2", "mcp.invoke", "mcp.bootstrap", "mcp.sync", @@ -33,8 +32,6 @@ "mcp.dependencies", "mcp.impact", "mcp.context", - "mcp.task_context", - "mcp.generation_diff", "mcp.validate_project", "mcp.render_status", "mcp.visualize", @@ -57,9 +54,6 @@ "cli.dependencies", "cli.impact", "cli.context", - "cli.generation-diff", - "cli.configure", - "cli.doctor", "cli.render", "cli.render-status", "cli.preview", @@ -155,8 +149,6 @@ "kind": { "enum": [ "context.items", - "task-context.items", - "generation-diff.items", "changeset.list", "changeset.inspect", "changeset.validate", @@ -222,18 +214,7 @@ "properties": { "code": { "type": "string", "minLength": 1 }, "message": { "type": "string", "minLength": 1 }, - "details": { "type": "object" }, - "remediation": { - "type": "object", - "required": ["retryable"], - "properties": { - "retryable": { "type": "boolean" }, - "action": { "type": "string", "minLength": 1 }, - "tool": { "type": "string", "minLength": 1 }, - "arguments": { "type": "object" } - }, - "additionalProperties": false - } + "details": { "type": "object" } }, "additionalProperties": false } diff --git a/src/docforge/_fs_safety.py b/src/docforge/_fs_safety.py deleted file mode 100644 index e2c8c94..0000000 --- a/src/docforge/_fs_safety.py +++ /dev/null @@ -1,71 +0,0 @@ -"""Internal directory binding helpers for disposable publication paths.""" - -from __future__ import annotations - -import os -import stat -from pathlib import Path - -from .errors import DocForgeError - - -def open_bound_directory(path: Path) -> int: - """Open one real directory and bind its current inode for later operations.""" - - try: - path_status = path.lstat() - if ( - stat.S_ISLNK(path_status.st_mode) - or not stat.S_ISDIR(path_status.st_mode) - or path.resolve(strict=True) != path - ): - raise DocForgeError( - "path_escape", - "Derived cache root is not a safe real directory", - ) - directory_fd = os.open(path, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW) - except FileNotFoundError as error: - raise DocForgeError( - "missing_index", - "Derived cache root does not exist", - ) from error - except OSError as error: - raise DocForgeError( - "path_escape", - "Derived cache root cannot be opened safely", - ) from error - try: - opened_status = os.fstat(directory_fd) - if opened_status.st_dev != path_status.st_dev or opened_status.st_ino != path_status.st_ino: - raise DocForgeError( - "path_escape", - "Derived cache root changed while opening", - ) - except Exception: - os.close(directory_fd) - raise - return directory_fd - - -def require_bound_directory(path: Path, directory_fd: int) -> None: - """Require a path to still name the exact opened real directory.""" - - try: - path_status = path.lstat() - opened_status = os.fstat(directory_fd) - if ( - stat.S_ISLNK(path_status.st_mode) - or not stat.S_ISDIR(path_status.st_mode) - or path.resolve(strict=True) != path - or opened_status.st_dev != path_status.st_dev - or opened_status.st_ino != path_status.st_ino - ): - raise DocForgeError( - "path_escape", - "Derived cache root changed during publication", - ) - except FileNotFoundError as error: - raise DocForgeError( - "path_escape", - "Derived cache root disappeared during publication", - ) from error diff --git a/src/docforge/cli.py b/src/docforge/cli.py index 35b55d2..ef4d749 100644 --- a/src/docforge/cli.py +++ b/src/docforge/cli.py @@ -9,9 +9,7 @@ import webbrowser from pathlib import Path from .application import CanonicalApplicationService, GenericCanonicalApplier -from .client_config import CLIENT_NAMES, generate_client_configuration from .context import compile_context -from .doctor import run_doctor from .errors import DocForgeError from .index import ProjectIndex from .onboarding import assess_project, scaffold_project @@ -23,33 +21,13 @@ from .viewer_manager import ViewerManagerClient def _parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser(prog="docforge") - parser.add_argument("--project-root", type=Path) + parser.add_argument("--project-root", type=Path, required=True) parser.add_argument( "--diagnostics", action="store_true", help="Attach bounded request-local stage timings and counters", ) commands = parser.add_subparsers(dest="command", required=True) - configure = commands.add_parser("configure") - configure.add_argument("client", choices=CLIENT_NAMES) - configure.add_argument("--project", type=Path, required=True) - configure.add_argument("--name") - configure.add_argument( - "--capability-mode", - choices=("read", "proposal", "application"), - default="read", - ) - configure.add_argument("--proposal-writer") - configure.add_argument("--canonical-applier") - configure.add_argument("--no-ast", action="store_true") - configure.add_argument("--startup-timeout", type=int, default=30) - configure.add_argument("--tool-timeout", type=int, default=300) - configure.add_argument("--output", type=Path) - doctor = commands.add_parser("doctor") - doctor.add_argument("--client", choices=CLIENT_NAMES, required=True) - doctor.add_argument("--project", type=Path) - doctor.add_argument("--config", type=Path) - doctor.add_argument("--server-name") onboard = commands.add_parser("onboard") onboard.add_argument("--language", action="append", default=[]) onboard.add_argument("--scaffold", action="store_true") @@ -87,9 +65,6 @@ def _parser() -> argparse.ArgumentParser: context.add_argument("--budget", type=int) context.add_argument("--limit", type=int) context.add_argument("--cursor") - generation_diff = commands.add_parser("generation-diff") - generation_diff.add_argument("--limit", type=int) - generation_diff.add_argument("--cursor") render = commands.add_parser("render") render.add_argument("view_id") render_status = commands.add_parser("render-status") @@ -114,33 +89,6 @@ def _parser() -> argparse.ArgumentParser: def _run(arguments: argparse.Namespace) -> dict[str, object]: - if arguments.command == "configure": - project = Project.open(arguments.project) - return generate_client_configuration( - project, - arguments.client, - server_name=arguments.name, - capability_mode=arguments.capability_mode, - proposal_writer=arguments.proposal_writer, - canonical_applier=arguments.canonical_applier, - no_ast=arguments.no_ast, - startup_timeout=arguments.startup_timeout, - tool_timeout=arguments.tool_timeout, - output=arguments.output, - ) - if arguments.command == "doctor": - root = arguments.project or arguments.project_root or Path.cwd() - return run_doctor( - Project.open(root), - arguments.client, - config_path=arguments.config, - server_name=arguments.server_name, - ) - if arguments.project_root is None: - raise DocForgeError( - "missing_project_root", - "This command requires --project-root", - ) if arguments.command == "onboard": languages = tuple(arguments.language) if arguments.scaffold: @@ -239,16 +187,6 @@ def _run(arguments: argparse.Namespace) -> dict[str, object]: cursor=arguments.cursor, ) return compile_context(index, arguments.profile, arguments.budget) - if arguments.command == "generation-diff": - from .mcp_server import DocForgeService - - return DocForgeService( - project, - capability_mode_name="read", - ).generation_diff( - limit=arguments.limit, - cursor=arguments.cursor, - ) if arguments.command == "render": return RenderService(project).render(arguments.view_id) if arguments.command == "render-status": @@ -299,14 +237,13 @@ def main(argv: list[str] | None = None) -> int: ) as collector: try: result = _run(arguments) - doctor_state = result.get("doctor_state") - code = 2 if doctor_state == "unhealthy" else (1 if doctor_state == "degraded" else 0) + code = 0 except DocForgeError as error: result = {"status": "error", "error": error.as_dict()} code = 2 if collector is not None: result["diagnostics"] = collector.as_dict( - outcome="ok" if result.get("status") == "ok" else "error", + outcome="ok" if code == 0 else "error", ) print(json.dumps(result, sort_keys=True, indent=2)) return code diff --git a/src/docforge/client_config.py b/src/docforge/client_config.py deleted file mode 100644 index 7b482d5..0000000 --- a/src/docforge/client_config.py +++ /dev/null @@ -1,937 +0,0 @@ -"""Deterministic, explicit client-configuration plans for DocForge MCP.""" - -from __future__ import annotations - -import hashlib -import json -import os -import re -import secrets -import stat -import subprocess -import sys -from collections.abc import Callable -from contextlib import suppress -from dataclasses import dataclass -from pathlib import Path -from typing import Literal, cast - -from .changeset_contract import document_hash -from .errors import DocForgeError -from .models import ProjectService -from .policy import CapabilityMode, compose_effective_policy -from .project import project_root_fingerprint, validate_descriptor_binding - -ClientName = Literal["codex", "claude", "openclaw"] -CLIENT_NAMES: tuple[ClientName, ...] = ("codex", "claude", "openclaw") -MAX_CLIENT_FRAGMENT_BYTES = 1_000_000 -GENERATED_CAPABILITY_MODES: tuple[CapabilityMode, ...] = ( - "read", - "proposal", - "application", -) -_SERVER_NAME = re.compile(r"[a-z0-9][a-z0-9_-]{0,63}") - - -@dataclass(frozen=True) -class _FileIdentity: - device: int - inode: int - mode: int - size: int - mtime_ns: int - ctime_ns: int - uid: int - link_count: int - - -def _file_identity(status: os.stat_result) -> _FileIdentity: - return _FileIdentity( - device=status.st_dev, - inode=status.st_ino, - mode=status.st_mode, - size=status.st_size, - mtime_ns=status.st_mtime_ns, - ctime_ns=status.st_ctime_ns, - uid=status.st_uid, - link_count=status.st_nlink, - ) - - -def _client_name(value: str) -> ClientName: - if value not in CLIENT_NAMES: - raise DocForgeError( - "unsupported_client", - "Client configuration target is unsupported", - client=value, - allowed=list(CLIENT_NAMES), - ) - return value - - -def _capability_mode(value: str) -> CapabilityMode: - if value not in GENERATED_CAPABILITY_MODES: - raise DocForgeError( - "invalid_capability_mode", - "Generated configuration supports read, proposal, or application mode", - capability_mode=value, - allowed=list(GENERATED_CAPABILITY_MODES), - ) - return value - - -def _bounded_seconds(value: int, *, field: str, maximum: int) -> int: - if type(value) is not int or value < 1 or value > maximum: - raise DocForgeError( - "invalid_timeout", - "Client timeout is outside the supported range", - field=field, - minimum=1, - maximum=maximum, - ) - return value - - -def _default_server_name(project_id: str, fingerprint: str) -> str: - prefix = re.sub(r"[^a-z0-9_-]+", "-", project_id.lower()).strip("-_") - prefix = prefix or "project" - suffix = f"-{fingerprint}" - available = 64 - len("docforge-") - len(suffix) - return f"docforge-{prefix[:available]}{suffix}" - - -def _validated_server_name(value: str | None, *, project_id: str, fingerprint: str) -> str: - selected = value or _default_server_name(project_id, fingerprint) - if _SERVER_NAME.fullmatch(selected) is None: - raise DocForgeError( - "invalid_server_name", - "Generated server name must be a stable lowercase client identifier", - pattern=_SERVER_NAME.pattern, - maximum_length=64, - ) - return selected - - -def _toml_string(value: str) -> str: - return json.dumps(value, ensure_ascii=False) - - -def _toml_array(values: list[str]) -> str: - return "[" + ", ".join(_toml_string(value) for value in values) + "]" - - -def _artifact( - client: ClientName, - *, - server_name: str, - command: str, - arguments: list[str], - startup_timeout: int, - tool_timeout: int, -) -> tuple[str, str, str | None]: - if client == "codex": - content = "\n".join( - ( - f'[mcp_servers."{server_name}"]', - f"command = {_toml_string(command)}", - f"args = {_toml_array(arguments)}", - "env = {}", - f"startup_timeout_sec = {startup_timeout}", - f"tool_timeout_sec = {tool_timeout}", - "", - ) - ) - return "codex-toml-fragment-v1", content, None - if client == "openclaw": - content = ( - json.dumps( - { - "mcp": { - "servers": { - server_name: { - "args": arguments, - "command": command, - "connectTimeout": startup_timeout, - "env": {}, - "supportsParallelToolCalls": False, - "timeout": tool_timeout, - } - } - } - }, - ensure_ascii=False, - indent=2, - sort_keys=True, - ) - + "\n" - ) - return "openclaw-json-fragment-v1", content, None - content = ( - json.dumps( - { - "mcpServers": { - server_name: { - "args": arguments, - "command": command, - "env": {}, - } - } - }, - ensure_ascii=False, - indent=2, - sort_keys=True, - ) - + "\n" - ) - return ( - "claude-json-fragment-v1", - content, - "Claude per-server timeout representation is not yet verified.", - ) - - -def _signature( - directory_fd: int, - name: str, -) -> _FileIdentity | None: - try: - status = os.stat(name, dir_fd=directory_fd, follow_symlinks=False) - except FileNotFoundError: - return None - except OSError as error: - raise DocForgeError( - "unsafe_output", - "Configuration output cannot be inspected safely", - ) from error - if stat.S_ISLNK(status.st_mode) or not stat.S_ISREG(status.st_mode): - raise DocForgeError( - "unsafe_output", - "Configuration output must be a regular file and not a symbolic link", - ) - return _file_identity(status) - - -def _parent_binding_current(path: Path, directory_fd: int) -> bool: - try: - before = path.lstat() - resolved = path.resolve(strict=True) - after = path.lstat() - opened = os.fstat(directory_fd) - return ( - not stat.S_ISLNK(before.st_mode) - and stat.S_ISDIR(before.st_mode) - and resolved == path - and (before.st_dev, before.st_ino, before.st_mode) - == (after.st_dev, after.st_ino, after.st_mode) - == (opened.st_dev, opened.st_ino, opened.st_mode) - ) - except OSError: - return False - - -def _require_parent_binding(path: Path, directory_fd: int) -> None: - if not _parent_binding_current(path, directory_fd): - raise DocForgeError( - "output_changed", - "Configuration output parent changed during publication", - ) - - -def _bound_parent(path: Path) -> tuple[Path, int]: - absolute = Path(os.path.abspath(path.expanduser())) - parent = absolute.parent - try: - parent_status = parent.lstat() - resolved = parent.resolve(strict=True) - except OSError as error: - raise DocForgeError( - "invalid_output", - "Configuration output parent does not exist", - ) from error - if ( - stat.S_ISLNK(parent_status.st_mode) - or not stat.S_ISDIR(parent_status.st_mode) - or resolved != parent - or absolute.name in {"", ".", ".."} - ): - raise DocForgeError( - "unsafe_output", - "Configuration output parent must be one real non-symlinked directory", - ) - try: - directory_fd = os.open( - parent, - os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, - ) - except OSError as error: - raise DocForgeError( - "unsafe_output", - "Configuration output parent cannot be opened safely", - ) from error - opened = os.fstat(directory_fd) - if opened.st_dev != parent_status.st_dev or opened.st_ino != parent_status.st_ino: - with suppress(OSError): - os.close(directory_fd) - raise DocForgeError( - "output_changed", - "Configuration output parent changed while it was opened", - ) - return absolute, directory_fd - - -def _read_existing( - directory_fd: int, - name: str, - signature: _FileIdentity, -) -> bytes: - if signature.size > MAX_CLIENT_FRAGMENT_BYTES: - raise DocForgeError( - "output_oversized", - "Existing configuration output exceeds the bounded fragment limit", - maximum_bytes=MAX_CLIENT_FRAGMENT_BYTES, - ) - try: - descriptor = os.open( - name, - os.O_RDONLY | os.O_NOFOLLOW, - dir_fd=directory_fd, - ) - except OSError as error: - raise DocForgeError( - "unsafe_output", - "Configuration output cannot be opened safely", - ) from error - try: - opened = os.fstat(descriptor) - opened_signature = _file_identity(opened) - if opened_signature != signature: - raise DocForgeError( - "output_changed", - "Configuration output changed while it was opened", - ) - remaining = MAX_CLIENT_FRAGMENT_BYTES + 1 - chunks: list[bytes] = [] - while remaining: - chunk = os.read(descriptor, min(65_536, remaining)) - if not chunk: - break - chunks.append(chunk) - remaining -= len(chunk) - raw = b"".join(chunks) - finally: - with suppress(OSError): - os.close(descriptor) - if len(raw) > MAX_CLIENT_FRAGMENT_BYTES or _signature(directory_fd, name) != signature: - raise DocForgeError( - "output_changed", - "Configuration output changed while it was read", - ) - return raw - - -def _private_existing(identity: _FileIdentity) -> bool: - return ( - identity.uid == os.geteuid() - and stat.S_IMODE(identity.mode) & 0o077 == 0 - and identity.link_count == 1 - ) - - -def _rollback_link( - directory_fd: int, - name: str, - expected: _FileIdentity, -) -> bool: - try: - current = _signature(directory_fd, name) - if current is None: - return True - if current.device != expected.device or current.inode != expected.inode: - return True - os.unlink(name, dir_fd=directory_fd) - return True - except (DocForgeError, OSError): - return False - - -def _rollback_and_sync( - directory_fd: int, - name: str, - expected: _FileIdentity, -) -> bool: - if not _rollback_link(directory_fd, name, expected): - return False - try: - os.fsync(directory_fd) - except OSError: - return False - return True - - -def _atomic_write( - path: Path, - content: str, - *, - validate_binding: Callable[[], None], -) -> tuple[str, str, str | None, Path | None]: - target, directory_fd = _bound_parent(path) - encoded = content.encode("utf-8") - if len(encoded) > MAX_CLIENT_FRAGMENT_BYTES: - with suppress(OSError): - os.close(directory_fd) - raise DocForgeError( - "output_oversized", - "Generated configuration fragment exceeds the bounded limit", - maximum_bytes=MAX_CLIENT_FRAGMENT_BYTES, - ) - temporary_name = f".docforge-client-{secrets.token_hex(12)}" - temporary_created = False - committed = False - linked_identity: _FileIdentity | None = None - try: - _require_parent_binding(target.parent, directory_fd) - before = _signature(directory_fd, target.name) - if before is not None: - if not _private_existing(before): - raise DocForgeError( - "unsafe_output", - ( - "Existing configuration fragment must be owned by the current user, " - "private, and singly linked" - ), - ) - existing = _read_existing(directory_fd, target.name, before) - if existing == encoded: - validate_binding() - _require_parent_binding(target.parent, directory_fd) - current = _signature(directory_fd, target.name) - if ( - current != before - or current is None - or not _private_existing(current) - or _read_existing(directory_fd, target.name, current) != encoded - ): - raise DocForgeError( - "output_changed", - "Configuration output changed before unchanged publication was confirmed", - ) - validate_binding() - _require_parent_binding(target.parent, directory_fd) - return "unchanged", "not_applicable", None, target - raise DocForgeError( - "output_conflict", - "Configuration fragment already exists with different content", - existing_sha256=hashlib.sha256(existing).hexdigest(), - generated_sha256=hashlib.sha256(encoded).hexdigest(), - ) - - try: - temporary_fd = os.open( - temporary_name, - os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, - 0o600, - dir_fd=directory_fd, - ) - except OSError as error: - raise DocForgeError( - "output_publication_failed", - "Configuration fragment temporary file could not be created", - ) from error - temporary_created = True - try: - with os.fdopen(temporary_fd, "wb", closefd=True) as handle: - handle.write(encoded) - handle.flush() - os.fsync(handle.fileno()) - except OSError as error: - raise DocForgeError( - "output_publication_failed", - "Configuration fragment temporary file could not be written durably", - ) from error - temporary_identity = _signature(directory_fd, temporary_name) - if temporary_identity is None: - raise DocForgeError( - "output_changed", - "Configuration fragment temporary file disappeared before publication", - ) - if _signature(directory_fd, target.name) is not None: - raise DocForgeError( - "output_changed", - "Configuration output appeared before atomic publication", - ) - _require_parent_binding(target.parent, directory_fd) - validate_binding() - try: - os.link( - temporary_name, - target.name, - src_dir_fd=directory_fd, - dst_dir_fd=directory_fd, - follow_symlinks=False, - ) - except FileExistsError as error: - raise DocForgeError( - "output_changed", - "Configuration output appeared during atomic publication", - ) from error - except OSError as error: - raise DocForgeError( - "output_publication_failed", - "Configuration fragment could not be published atomically", - ) from error - linked_identity = temporary_identity - try: - validate_binding() - except DocForgeError: - if _rollback_link(directory_fd, target.name, temporary_identity): - raise - committed = True - return ( - "created", - "unconfirmed", - "publication_binding_unconfirmed", - None, - ) - linked = _signature(directory_fd, target.name) - if ( - linked is None - or linked.device != temporary_identity.device - or linked.inode != temporary_identity.inode - or _read_existing(directory_fd, target.name, linked) != encoded - or not _parent_binding_current(target.parent, directory_fd) - ): - if _rollback_link(directory_fd, target.name, temporary_identity): - raise DocForgeError( - "output_changed", - "Configuration output changed during atomic publication", - ) - committed = True - return ( - "created", - "unconfirmed", - "publication_location_unconfirmed", - None, - ) - durability = "confirmed" - warning: str | None = None - try: - os.unlink(temporary_name, dir_fd=directory_fd) - temporary_created = False - published = _signature(directory_fd, target.name) - if ( - published is None - or not _private_existing(published) - or published.device != temporary_identity.device - or published.inode != temporary_identity.inode - or _read_existing(directory_fd, target.name, published) != encoded - or not _parent_binding_current(target.parent, directory_fd) - ): - if _rollback_link(directory_fd, target.name, temporary_identity): - raise DocForgeError( - "output_changed", - "Configuration output changed after atomic publication", - ) - committed = True - return ( - "created", - "unconfirmed", - "publication_location_unconfirmed", - None, - ) - try: - validate_binding() - except DocForgeError: - if _rollback_link(directory_fd, target.name, temporary_identity): - raise - committed = True - return ( - "created", - "unconfirmed", - "publication_binding_unconfirmed", - None, - ) - os.fsync(directory_fd) - except OSError: - durability = "unconfirmed" - warning = "publication_durability_unconfirmed" - try: - validate_binding() - except DocForgeError: - if _rollback_and_sync(directory_fd, target.name, temporary_identity): - raise - committed = True - return ( - "created", - "unconfirmed", - "publication_binding_unconfirmed", - None, - ) - try: - published = _signature(directory_fd, target.name) - publication_current = ( - published is not None - and _private_existing(published) - and published.device == temporary_identity.device - and published.inode == temporary_identity.inode - and _read_existing(directory_fd, target.name, published) == encoded - and _parent_binding_current(target.parent, directory_fd) - ) - except DocForgeError: - publication_current = False - if not publication_current: - if _rollback_and_sync(directory_fd, target.name, temporary_identity): - raise DocForgeError( - "output_changed", - "Configuration output changed before publication was finalized", - ) - committed = True - return ( - "created", - "unconfirmed", - "publication_location_unconfirmed", - None, - ) - committed = True - return "created", durability, warning, target - except Exception: - if not committed and linked_identity is not None: - _rollback_link(directory_fd, target.name, linked_identity) - if not committed and temporary_created: - with suppress(OSError): - os.unlink(temporary_name, dir_fd=directory_fd) - raise - finally: - with suppress(OSError): - os.close(directory_fd) - - -def _validate_configuration_result(result: dict[str, object]) -> None: - artifact = cast(dict[str, object], result["artifact"]) - binding = cast(dict[str, object], result["binding"]) - policy = cast(dict[str, object], result["effective_policy"]) - project = cast(dict[str, object], result["project"]) - content = cast(str, artifact["content"]) - if artifact["content_sha256"] != hashlib.sha256(content.encode("utf-8")).hexdigest(): - raise AssertionError("Generated client content hash drifted") - timeouts = cast(dict[str, object], binding["timeouts"]) - expected_format, expected_content, _ = _artifact( - cast(ClientName, result["client"]), - server_name=cast(str, result["server_name"]), - command=cast(str, binding["command"]), - arguments=cast(list[str], binding["args"]), - startup_timeout=cast(int, timeouts["startup_seconds"]), - tool_timeout=cast(int, timeouts["tool_seconds"]), - ) - if artifact["format"] != expected_format or content != expected_content: - raise AssertionError("Generated client artifact drifted from its binding") - adapter_policy = cast(dict[str, object], binding["adapter_policy"]) - render_policy = cast(dict[str, object], binding["render_policy"]) - arguments = cast(list[str], binding["args"]) - prefix = [ - "-I", - "-m", - "docforge.mcp_server", - "--project-root", - cast(str, project["project_root"]), - "--capability-mode", - cast(str, binding["capability_mode"]), - ] - if arguments[:7] != prefix: - raise AssertionError("Generated client arguments drifted from their binding") - remaining = arguments[7:] - no_ast_argument = "--no-ast" in arguments - if no_ast_argument: - if remaining[-1:] != ["--no-ast"] or arguments.count("--no-ast") != 1: - raise AssertionError("Generated no-AST argument layout drifted") - remaining = remaining[:-1] - mode = binding["capability_mode"] - if ( - (mode == "read" and remaining) - or ( - mode == "proposal" - and (len(remaining) != 2 or remaining[0] != "--proposal-writer" or not remaining[1]) - ) - or ( - mode == "application" - and ( - len(remaining) != 4 - or remaining[0] != "--proposal-writer" - or remaining[2] != "--canonical-applier" - or not remaining[1] - or remaining[1] != remaining[3] - ) - ) - ): - raise AssertionError("Generated authority argument layout drifted") - composed_policy = compose_effective_policy( - selected_mode=cast(CapabilityMode, mode), - capability_source="explicit", - no_ast=adapter_policy["mode"] == "preserve-no-ast", - diagnostics=False, - render_configured=render_policy["manual"] != "disabled", - application_enabled=mode == "application", - ) - expected_policy = composed_policy.as_dict() - if ( - policy != expected_policy - or adapter_policy != composed_policy.adapter_policy() - or binding["capability_mode"] != policy["capability_mode"] - or no_ast_argument != (adapter_policy["mode"] == "preserve-no-ast") - or render_policy["manual"] != policy["manual_render"] - or render_policy["graph"] != policy["graph_render"] - or render_policy["live_viewer"] != policy["live_viewer"] - or ( - adapter_policy["mode"] == "preserve-no-ast" - and ( - policy["adapter_evolution"] != "preserve" - or policy["ast_analysis"] != "forbidden" - or policy["logic_indexing"] != "off" - ) - ) - or ( - adapter_policy["mode"] == "standard" - and ( - policy["adapter_evolution"] != "allowed" - or policy["ast_analysis"] != "allowed" - or policy["logic_indexing"] != "full" - ) - ) - ): - raise AssertionError("Generated client policy drifted from its binding") - expected_hash = document_hash( - { - "schema_version": 1, - "client": result["client"], - "server_name": result["server_name"], - "project": project, - "binding": binding, - "effective_policy": policy, - "artifact_format": artifact["format"], - "artifact_content_sha256": artifact["content_sha256"], - } - ) - if result["configuration_hash"] != expected_hash: - raise AssertionError("Generated client configuration hash drifted") - - -def generate_client_configuration( - project: ProjectService, - client: str, - *, - server_name: str | None = None, - capability_mode: str = "read", - proposal_writer: str | None = None, - canonical_applier: str | None = None, - no_ast: bool = False, - startup_timeout: int = 30, - tool_timeout: int = 300, - output: Path | None = None, -) -> dict[str, object]: - """Build one deterministic client fragment and optionally publish it explicitly.""" - - validate_descriptor_binding(project.descriptor) - selected_client = _client_name(client) - selected_mode = _capability_mode(capability_mode) - startup_seconds = _bounded_seconds( - startup_timeout, - field="startup_timeout", - maximum=3_600, - ) - tool_seconds = _bounded_seconds( - tool_timeout, - field="tool_timeout", - maximum=86_400, - ) - descriptor = project.descriptor - if descriptor.adapter != "generic": - raise DocForgeError( - "client_configuration_unavailable", - "Generic CLI configuration cannot reconstruct a project-owned adapter", - adapter=descriptor.adapter, - ) - writer_ids = {writer.writer_id for writer in descriptor.proposal_writers} - if selected_mode == "read": - if proposal_writer is not None or canonical_applier is not None: - raise DocForgeError( - "invalid_capability_binding", - "Read configuration cannot bind proposal or application authority", - ) - elif selected_mode == "proposal": - if proposal_writer is None or proposal_writer not in writer_ids: - raise DocForgeError( - "capability_unavailable", - "Proposal configuration requires a descriptor-declared writer", - required="proposal_writer", - ) - if canonical_applier is not None: - raise DocForgeError( - "invalid_capability_binding", - "Proposal configuration cannot bind a canonical applier", - ) - else: - if ( - proposal_writer is None - or canonical_applier is None - or proposal_writer != canonical_applier - or proposal_writer not in writer_ids - ): - raise DocForgeError( - "capability_unavailable", - "Application configuration requires one declared writer/applier identity", - required="matching_declared_writer_and_applier", - ) - - fingerprint = project_root_fingerprint(descriptor.root) - selected_name = _validated_server_name( - server_name, - project_id=descriptor.project_id, - fingerprint=fingerprint, - ) - executable = Path(os.path.abspath(sys.executable)) - try: - executable_status = executable.stat() - except OSError as error: - raise DocForgeError( - "client_configuration_unavailable", - "Current Python executable cannot be inspected", - ) from error - if not stat.S_ISREG(executable_status.st_mode) or not os.access(executable, os.X_OK): - raise DocForgeError( - "client_configuration_unavailable", - "Current Python executable is not a runnable regular file", - ) - try: - probe = subprocess.run( - [ - str(executable), - "-I", - "-B", - "-c", - "import docforge.mcp_server", - ], - check=False, - stdin=subprocess.DEVNULL, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - timeout=10, - env=os.environ.copy(), - ) - except (OSError, subprocess.SubprocessError) as error: - raise DocForgeError( - "client_configuration_unavailable", - "Current isolated Python executable could not be probed safely", - ) from error - if probe.returncode != 0: - raise DocForgeError( - "client_configuration_unavailable", - "Current isolated Python executable cannot import docforge.mcp_server", - ) - arguments = [ - "-I", - "-m", - "docforge.mcp_server", - "--project-root", - str(descriptor.root), - "--capability-mode", - selected_mode, - ] - if proposal_writer is not None: - arguments.extend(("--proposal-writer", proposal_writer)) - if canonical_applier is not None: - arguments.extend(("--canonical-applier", canonical_applier)) - if no_ast: - arguments.append("--no-ast") - - policy = compose_effective_policy( - selected_mode=selected_mode, - capability_source="explicit", - no_ast=no_ast, - diagnostics=False, - render_configured=descriptor.render is not None, - application_enabled=canonical_applier is not None, - ) - artifact_format, content, warning = _artifact( - selected_client, - server_name=selected_name, - command=str(executable), - arguments=arguments, - startup_timeout=startup_seconds, - tool_timeout=tool_seconds, - ) - if output is None: - validate_descriptor_binding(descriptor) - write_state = "not_requested" - durability = "not_applicable" - publication_warning = None - output_path = None - else: - validate_descriptor_binding(descriptor) - write_state, durability, publication_warning, published_path = _atomic_write( - output, - content, - validate_binding=lambda: validate_descriptor_binding(descriptor), - ) - output_path = str(published_path) if published_path is not None else None - artifact = { - "format": artifact_format, - "content": content, - "content_sha256": hashlib.sha256(content.encode("utf-8")).hexdigest(), - "output_path": output_path, - "write_state": write_state, - "durability": durability, - } - binding = { - "transport": "stdio", - "capability_mode": selected_mode, - "adapter_policy": policy.adapter_policy(), - "render_policy": { - "manual": policy.manual_render, - "graph": policy.graph_render, - "live_viewer": policy.live_viewer, - }, - "command": str(executable), - "args": arguments, - "environment": {}, - "timeouts": { - "startup_seconds": startup_seconds, - "tool_seconds": tool_seconds, - }, - } - project_binding = { - "project_id": descriptor.project_id, - "project_root": str(descriptor.root), - "project_root_fingerprint": fingerprint, - "adapter": descriptor.adapter, - } - policy_payload = policy.as_dict() - plan_hash = document_hash( - { - "schema_version": 1, - "client": selected_client, - "server_name": selected_name, - "project": project_binding, - "binding": binding, - "effective_policy": policy_payload, - "artifact_format": artifact_format, - "artifact_content_sha256": artifact["content_sha256"], - } - ) - result: dict[str, object] = { - "status": "ok", - "schema_version": 1, - "operation": "client.configure", - "action": "write" if output is not None else "preview", - "client": selected_client, - "server_name": selected_name, - "project": project_binding, - "binding": binding, - "effective_policy": policy_payload, - "artifact": artifact, - "configuration_hash": plan_hash, - "warnings": [ - *([] if warning is None else [{"code": "timeout_format_unverified"}]), - *([] if publication_warning is None else [{"code": publication_warning}]), - ], - } - _validate_configuration_result(result) - return result diff --git a/src/docforge/doctor.py b/src/docforge/doctor.py deleted file mode 100644 index e79ed61..0000000 --- a/src/docforge/doctor.py +++ /dev/null @@ -1,1287 +0,0 @@ -"""Bounded, non-mutating checks for one project-bound client integration.""" - -from __future__ import annotations - -import hashlib -import json -import os -import stat -import sys -import tomllib -from contextlib import suppress -from pathlib import Path -from typing import Literal, cast - -from .client_config import CLIENT_NAMES, ClientName -from .errors import DocForgeError -from .models import ProjectService -from .policy import CapabilityMode, compose_effective_policy -from .project import ( - project_root_fingerprint, - validate_descriptor_binding, -) - -MAX_CLIENT_CONFIG_BYTES = 1_000_000 -MAX_CLIENT_SERVERS = 256 -MAX_CLIENT_ARGUMENTS = 64 -MAX_CLIENT_ARGUMENT_CHARS = 4_096 -MAX_CLIENT_ENVIRONMENT_KEYS = 64 -RISKY_ENVIRONMENT_KEYS = frozenset({"LD_PRELOAD", "PYTHONHOME", "PYTHONPATH", "PYTHONSTARTUP"}) - -CheckState = Literal["passed", "warning", "failed", "skipped"] -DOCTOR_RESULT_LIMIT_BYTES = 32_768 -CHECK_IDS = ( - "project.binding", - "project.canonical_validation", - "client.driver", - "client.config", - "client.entry", - "server.executable", - "server.arguments", - "server.project_binding", - "policy.effective", - "policy.no_ast", - "client.timeouts", - "client.environment", - "client.tool_filter", - "derived.index", -) - - -def _check( - check_id: str, - state: CheckState, - code: str, - message: str, - **details: object, -) -> dict[str, object]: - bounded_details: dict[str, object] = {} - for key, value in sorted(details.items())[:16]: - if isinstance(value, str): - bounded_details[key] = value[:512] - elif isinstance(value, (bool, int)) or value is None: - bounded_details[key] = value - elif isinstance(value, list): - items = cast(list[object], value) - bounded_details[key] = [ - item[:256] if isinstance(item, str) else item - for item in items[:16] - if isinstance(item, (str, bool, int)) or item is None - ] - return { - "check_id": check_id, - "state": state, - "code": code, - "message": message, - "details": bounded_details, - } - - -def _replace_check( - checks: list[dict[str, object]], - replacement: dict[str, object], -) -> None: - check_id = replacement["check_id"] - for index, check in enumerate(checks): - if check["check_id"] == check_id: - checks[index] = replacement - return - raise AssertionError(f"Doctor check {check_id!r} was not initialized") - - -def _skipped(check_id: str, code: str, message: str) -> dict[str, object]: - return _check(check_id, "skipped", code, message) - - -def _validate_doctor_result(result: dict[str, object]) -> None: - checks = cast(list[dict[str, object]], result["checks"]) - check_ids = [cast(str, check["check_id"]) for check in checks] - if tuple(check_ids) != CHECK_IDS: - raise AssertionError("Doctor did not emit its exact ordered check inventory") - expected = { - state: sum(check["state"] == state for check in checks) - for state in ("passed", "warning", "failed", "skipped") - } - if result["summary"] != expected: - raise AssertionError("Doctor summary does not match its checks") - expected_state = ( - "unhealthy" if expected["failed"] else ("degraded" if expected["warning"] else "healthy") - ) - if result["doctor_state"] != expected_state: - raise AssertionError("Doctor health does not match its checks") - if len(json.dumps(result, sort_keys=True, separators=(",", ":")).encode("utf-8")) > ( - DOCTOR_RESULT_LIMIT_BYTES - ): - raise AssertionError("Doctor result exceeded its bounded response contract") - - -def _default_config(client: ClientName) -> Path: - if client == "codex": - codex_home = os.environ.get("CODEX_HOME") - if codex_home: - return Path(codex_home).expanduser() / "config.toml" - return Path.home() / ".codex" / "config.toml" - if client == "openclaw": - explicit = os.environ.get("OPENCLAW_CONFIG_PATH") - if explicit: - return Path(explicit).expanduser() - state_root = os.environ.get("OPENCLAW_STATE_DIR") - if state_root: - return Path(state_root).expanduser() / "openclaw.json" - return Path.home() / ".openclaw" / "openclaw.json" - claude_root = os.environ.get("CLAUDE_CONFIG_DIR") - if claude_root: - return Path(claude_root).expanduser() / ".claude.json" - return Path.home() / ".claude.json" - - -def _parent_binding_current(path: Path, directory_fd: int) -> bool: - try: - before = path.lstat() - resolved = path.resolve(strict=True) - after = path.lstat() - opened = os.fstat(directory_fd) - return ( - not stat.S_ISLNK(before.st_mode) - and stat.S_ISDIR(before.st_mode) - and resolved == path - and (before.st_dev, before.st_ino, before.st_mode) - == (after.st_dev, after.st_ino, after.st_mode) - == (opened.st_dev, opened.st_ino, opened.st_mode) - ) - except OSError: - return False - - -def _read_stable_regular( - path: Path, -) -> tuple[bytes, tuple[int, int, int, int, int]]: - absolute = Path(os.path.abspath(path.expanduser())) - try: - parent = absolute.parent - parent_status = parent.lstat() - if ( - stat.S_ISLNK(parent_status.st_mode) - or not stat.S_ISDIR(parent_status.st_mode) - or parent.resolve(strict=True) != parent - ): - raise DocForgeError( - "client_config_unsafe", - "Client configuration parent is not one real directory", - ) - directory_fd = os.open( - parent, - os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, - ) - except FileNotFoundError as error: - raise DocForgeError( - "client_config_missing", - "Client configuration does not exist", - ) from error - except OSError as error: - raise DocForgeError( - "client_config_unsafe", - "Client configuration cannot be inspected safely", - ) from error - opened_parent = os.fstat(directory_fd) - if opened_parent.st_dev != parent_status.st_dev or opened_parent.st_ino != parent_status.st_ino: - with suppress(OSError): - os.close(directory_fd) - raise DocForgeError( - "client_config_changed", - "Client configuration parent changed while it was opened", - ) - try: - if not _parent_binding_current(parent, directory_fd): - raise DocForgeError( - "client_config_changed", - "Client configuration parent changed before it was read", - ) - try: - before = os.stat( - absolute.name, - dir_fd=directory_fd, - follow_symlinks=False, - ) - except FileNotFoundError as error: - raise DocForgeError( - "client_config_missing", - "Client configuration does not exist", - ) from error - except OSError as error: - raise DocForgeError( - "client_config_unsafe", - "Client configuration cannot be inspected safely", - ) from error - if stat.S_ISLNK(before.st_mode) or not stat.S_ISREG(before.st_mode): - raise DocForgeError( - "client_config_unsafe", - "Client configuration must be a regular file and not a symbolic link", - ) - if before.st_size > MAX_CLIENT_CONFIG_BYTES: - raise DocForgeError( - "client_config_oversized", - "Client configuration exceeds the bounded doctor limit", - maximum_bytes=MAX_CLIENT_CONFIG_BYTES, - ) - try: - descriptor = os.open( - absolute.name, - os.O_RDONLY | os.O_NOFOLLOW, - dir_fd=directory_fd, - ) - except OSError as error: - raise DocForgeError( - "client_config_unsafe", - "Client configuration cannot be opened safely", - ) from error - try: - opened = os.fstat(descriptor) - if opened.st_dev != before.st_dev or opened.st_ino != before.st_ino: - raise DocForgeError( - "client_config_changed", - "Client configuration changed while it was opened", - ) - chunks: list[bytes] = [] - remaining = MAX_CLIENT_CONFIG_BYTES + 1 - while remaining: - chunk = os.read(descriptor, min(65_536, remaining)) - if not chunk: - break - chunks.append(chunk) - remaining -= len(chunk) - raw = b"".join(chunks) - finally: - with suppress(OSError): - os.close(descriptor) - if len(raw) > MAX_CLIENT_CONFIG_BYTES: - raise DocForgeError( - "client_config_oversized", - "Client configuration exceeds the bounded doctor limit", - maximum_bytes=MAX_CLIENT_CONFIG_BYTES, - ) - try: - after = os.stat( - absolute.name, - dir_fd=directory_fd, - follow_symlinks=False, - ) - except OSError as error: - raise DocForgeError( - "client_config_changed", - "Client configuration changed while it was read", - ) from error - before_identity = ( - before.st_dev, - before.st_ino, - before.st_size, - before.st_mtime_ns, - before.st_ctime_ns, - ) - after_identity = ( - after.st_dev, - after.st_ino, - after.st_size, - after.st_mtime_ns, - after.st_ctime_ns, - ) - if before_identity != after_identity: - raise DocForgeError( - "client_config_changed", - "Client configuration changed while it was read", - ) - if not _parent_binding_current(parent, directory_fd): - raise DocForgeError( - "client_config_changed", - "Client configuration parent changed while it was read", - ) - return raw, before_identity - finally: - with suppress(OSError): - os.close(directory_fd) - - -def _safe_regular_state(path: Path) -> Literal["missing", "present", "unsafe"]: - absolute = Path(os.path.abspath(path)) - parent = absolute.parent - try: - parent_status = parent.lstat() - if ( - stat.S_ISLNK(parent_status.st_mode) - or not stat.S_ISDIR(parent_status.st_mode) - or parent.resolve(strict=True) != parent - ): - return "unsafe" - directory_fd = os.open( - parent, - os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, - ) - except FileNotFoundError: - return "missing" - except OSError: - return "unsafe" - try: - opened = os.fstat(directory_fd) - if opened.st_dev != parent_status.st_dev or opened.st_ino != parent_status.st_ino: - return "unsafe" - if not _parent_binding_current(parent, directory_fd): - return "unsafe" - try: - before = os.stat( - absolute.name, - dir_fd=directory_fd, - follow_symlinks=False, - ) - except FileNotFoundError: - return "missing" if _parent_binding_current(parent, directory_fd) else "unsafe" - except OSError: - return "unsafe" - state: Literal["present", "unsafe"] = ( - "present" - if stat.S_ISREG(before.st_mode) and not stat.S_ISLNK(before.st_mode) - else "unsafe" - ) - try: - after = os.stat( - absolute.name, - dir_fd=directory_fd, - follow_symlinks=False, - ) - except OSError: - return "unsafe" - if ( - before.st_dev, - before.st_ino, - before.st_mode, - before.st_size, - before.st_mtime_ns, - before.st_ctime_ns, - ) != ( - after.st_dev, - after.st_ino, - after.st_mode, - after.st_size, - after.st_mtime_ns, - after.st_ctime_ns, - ): - return "unsafe" - return state if _parent_binding_current(parent, directory_fd) else "unsafe" - finally: - with suppress(OSError): - os.close(directory_fd) - - -def _safe_directory(path: Path) -> bool: - if not path.is_absolute(): - return False - try: - before = path.lstat() - if stat.S_ISLNK(before.st_mode) or not stat.S_ISDIR(before.st_mode): - return False - directory_fd = os.open( - path, - os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, - ) - except OSError: - return False - try: - resolved = path.resolve(strict=True) - after = path.lstat() - opened = os.fstat(directory_fd) - return resolved == path and (before.st_dev, before.st_ino, before.st_mode) == ( - after.st_dev, - after.st_ino, - after.st_mode, - ) == (opened.st_dev, opened.st_ino, opened.st_mode) - except OSError: - return False - finally: - with suppress(OSError): - os.close(directory_fd) - - -def _server_documents(client: ClientName, raw: bytes) -> dict[str, object]: - try: - text = raw.decode("utf-8") - except UnicodeDecodeError as error: - raise DocForgeError( - "client_config_invalid", - "Client configuration is not UTF-8", - ) from error - try: - if client == "codex": - document = cast(dict[str, object], tomllib.loads(text)) - servers = document.get("mcp_servers", {}) - else: - loaded = cast(object, json.loads(text)) - if not isinstance(loaded, dict): - raise TypeError - document = cast(dict[str, object], loaded) - if client == "openclaw": - mcp_value = document.get("mcp", {}) - if not isinstance(mcp_value, dict): - raise TypeError - mcp = cast(dict[str, object], mcp_value) - servers = mcp.get("servers", {}) - else: - servers = document.get("mcpServers", {}) - except ( - json.JSONDecodeError, - tomllib.TOMLDecodeError, - RecursionError, - TypeError, - ValueError, - ) as error: - raise DocForgeError( - "client_config_invalid", - "Client configuration has invalid syntax or structure", - ) from error - if not isinstance(servers, dict): - raise DocForgeError( - "client_config_invalid", - "Client MCP server collection must be an object", - ) - server_map = cast(dict[object, object], servers) - if len(server_map) > MAX_CLIENT_SERVERS: - raise DocForgeError( - "client_config_oversized", - "Client configuration declares too many MCP servers", - maximum_servers=MAX_CLIENT_SERVERS, - ) - if any(not isinstance(name, str) or len(name) > 256 for name in server_map): - raise DocForgeError( - "client_config_invalid", - "Client MCP server names are invalid", - ) - return cast(dict[str, object], server_map) - - -def _normalize_entry(name: str, value: object, *, client: ClientName) -> dict[str, object]: - if not isinstance(value, dict): - raise DocForgeError( - "client_entry_invalid", - "Client MCP server entry must be an object", - server_name=name, - ) - entry = cast(dict[str, object], value) - allowed_fields = { - "codex": { - "command", - "args", - "env", - "startup_timeout_sec", - "tool_timeout_sec", - "tools", - }, - "openclaw": { - "command", - "args", - "env", - "connectTimeout", - "timeout", - "supportsParallelToolCalls", - "cwd", - "toolFilter", - }, - "claude": {"command", "args", "env"}, - }[client] - unknown_fields = sorted(set(entry) - allowed_fields) - if unknown_fields: - raise DocForgeError( - "client_entry_invalid", - "Client MCP server entry contains unsupported fields", - server_name=name, - field_count=len(unknown_fields), - fields_sha256=hashlib.sha256("\0".join(unknown_fields).encode("utf-8")).hexdigest(), - ) - command = entry.get("command") - arguments_value = entry.get("args", []) - environment_value = entry.get("env", {}) - if ( - not isinstance(command, str) - or not command - or len(command) > MAX_CLIENT_ARGUMENT_CHARS - or "\0" in command - or not isinstance(arguments_value, list) - or not isinstance(environment_value, dict) - ): - raise DocForgeError( - "client_entry_invalid", - "Client MCP server command, arguments, or environment is invalid", - server_name=name, - ) - arguments = cast(list[object], arguments_value) - if len(arguments) > MAX_CLIENT_ARGUMENTS or any( - not isinstance(argument, str) - or len(argument) > MAX_CLIENT_ARGUMENT_CHARS - or "\0" in argument - for argument in arguments - ): - raise DocForgeError( - "client_entry_invalid", - "Client MCP server arguments exceed the bounded contract", - server_name=name, - ) - environment = cast(dict[object, object], environment_value) - if len(environment) > MAX_CLIENT_ENVIRONMENT_KEYS or any( - not isinstance(value, str) or len(value) > MAX_CLIENT_ARGUMENT_CHARS or "\0" in value - for value in environment.values() - ): - raise DocForgeError( - "client_entry_invalid", - "Client MCP server environment values exceed the bounded contract", - server_name=name, - ) - raw_environment_keys = list(environment) - if any(not isinstance(key, str) or len(key) > 256 for key in raw_environment_keys): - raise DocForgeError( - "client_entry_invalid", - "Client MCP server environment keys are invalid", - server_name=name, - ) - environment_keys = sorted(cast(list[str], raw_environment_keys)) - startup: object = None - tool: object = None - if client == "codex": - startup = entry.get("startup_timeout_sec") - tool = entry.get("tool_timeout_sec") - elif client == "openclaw": - startup = entry.get("connectTimeout") - tool = entry.get("timeout") - cwd = entry.get("cwd") - if cwd is not None and ( - not isinstance(cwd, str) or not cwd or len(cwd) > MAX_CLIENT_ARGUMENT_CHARS or "\0" in cwd - ): - raise DocForgeError( - "client_entry_invalid", - "Client MCP working directory is invalid", - server_name=name, - ) - parallel_calls = entry.get("supportsParallelToolCalls") - if parallel_calls is not None and type(parallel_calls) is not bool: - raise DocForgeError( - "client_entry_invalid", - "Client parallel-call setting must be Boolean", - server_name=name, - ) - filter_value = entry.get("toolFilter", entry.get("tools")) - filter_valid = True - if filter_value is not None: - if not isinstance(filter_value, dict): - filter_valid = False - elif client == "openclaw": - filter_document = cast(dict[object, object], filter_value) - filter_valid = set(filter_document).issubset({"include", "exclude"}) and all( - isinstance(values, list) - and len(cast(list[object], values)) <= 256 - and all( - isinstance(item, str) and 0 < len(item) <= 256 and "\0" not in item - for item in cast(list[object], values) - ) - for values in filter_document.values() - ) - if not filter_valid: - raise DocForgeError( - "client_entry_invalid", - "Client tool-filter setting is malformed", - server_name=name, - ) - return { - "server_name": name, - "command": command, - "args": cast(list[str], arguments), - "environment_keys": environment_keys, - "startup_timeout": startup, - "tool_timeout": tool, - "cwd": cwd, - "tool_filter_present": filter_value is not None, - "parallel_calls": parallel_calls, - } - - -def _parse_binding(arguments: list[str]) -> dict[str, object]: - remaining = list(arguments) - if remaining[:3] != ["-I", "-m", "docforge.mcp_server"]: - raise DocForgeError( - "server_arguments_invalid", - "Python launch arguments must begin with the isolated DocForge module prefix", - ) - remaining = remaining[3:] - values: dict[str, str] = {} - flags: set[str] = set() - value_options = { - "--project-root", - "--proposal-writer", - "--canonical-applier", - "--capability-mode", - } - flag_options = {"--no-ast", "--diagnostics"} - position = 0 - while position < len(remaining): - option = remaining[position] - if option in value_options: - if option in values or position + 1 >= len(remaining): - raise DocForgeError( - "server_arguments_invalid", - "Server arguments contain a duplicate or missing option value", - option=option, - ) - values[option] = remaining[position + 1] - position += 2 - continue - if option in flag_options: - if option in flags: - raise DocForgeError( - "server_arguments_invalid", - "Server arguments contain a duplicate flag", - option=option, - ) - flags.add(option) - position += 1 - continue - raise DocForgeError( - "server_arguments_invalid", - "Server arguments contain an unsupported option", - argument_index=position + 3, - argument_sha256=hashlib.sha256(option.encode("utf-8")).hexdigest(), - ) - selected_mode = values.get("--capability-mode") - implicit = selected_mode is None - if selected_mode is None: - selected_mode = "application" if "--canonical-applier" in values else "proposal" - if selected_mode not in {"read", "proposal", "application", "operator"}: - raise DocForgeError( - "server_arguments_invalid", - "Configured capability mode is unsupported", - ) - return { - "project_root": values.get("--project-root"), - "proposal_writer": values.get("--proposal-writer"), - "canonical_applier": values.get("--canonical-applier"), - "capability_mode": selected_mode, - "capability_mode_implicit": implicit, - "no_ast": "--no-ast" in flags, - "diagnostics": "--diagnostics" in flags, - } - - -def _partial_project_roots(value: object) -> tuple[str, ...]: - if not isinstance(value, dict): - return () - entry = cast(dict[str, object], value) - arguments_value = entry.get("args") - if not isinstance(arguments_value, list): - return () - arguments = cast(list[object], arguments_value) - roots: list[str] = [] - for position, argument in enumerate(arguments[:-1]): - if argument == "--project-root" and isinstance(arguments[position + 1], str): - roots.append(cast(str, arguments[position + 1])) - return tuple(roots) - - -def _entry_for_project( - servers: dict[str, object], - *, - client: ClientName, - project_root: Path, - server_name: str | None, -) -> dict[str, object] | None: - if server_name is not None: - value = servers.get(server_name) - return None if value is None else _normalize_entry(server_name, value, client=client) - candidates: list[tuple[str, object]] = [] - for name, value in sorted(servers.items()): - for configured_root in _partial_project_roots(value): - try: - configured_path = Path(os.path.normpath(configured_root)) - matches = configured_path.is_absolute() and configured_path == project_root - except ValueError: - matches = False - if matches: - candidates.append((name, value)) - break - if len(candidates) > 1: - raise DocForgeError( - "client_entry_ambiguous", - "More than one client entry binds the project; select --server-name", - count=len(candidates), - ) - if not candidates: - return None - name, value = candidates[0] - return _normalize_entry(name, value, client=client) - - -def _runtime_policy_check( - project: ProjectService, - binding: dict[str, object], -) -> tuple[CheckState, str, str]: - mode = cast(str, binding["capability_mode"]) - selected_mode = cast(CapabilityMode, mode) - proposal_writer = cast(str | None, binding["proposal_writer"]) - canonical_applier = cast(str | None, binding["canonical_applier"]) - writer_ids = {writer.writer_id for writer in project.descriptor.proposal_writers} - if proposal_writer is not None and proposal_writer not in writer_ids: - raise DocForgeError( - "effective_policy_invalid", - "Configured proposal writer is not declared by the project", - ) - if canonical_applier is not None and canonical_applier not in writer_ids: - raise DocForgeError( - "effective_policy_invalid", - "Configured canonical applier is not declared by the project", - ) - if selected_mode == "application" and canonical_applier is None: - raise DocForgeError( - "effective_policy_invalid", - "Application capability requires a canonical applier", - ) - compose_effective_policy( - selected_mode=selected_mode, - capability_source=( - "factory_default" if cast(bool, binding["capability_mode_implicit"]) else "explicit" - ), - no_ast=cast(bool, binding["no_ast"]), - diagnostics=cast(bool, binding["diagnostics"]), - render_configured=project.descriptor.render is not None, - application_enabled=( - canonical_applier is not None and selected_mode in {"application", "operator"} - ), - ) - if cast(bool, binding["capability_mode_implicit"]): - return ( - "warning", - "capability_mode_implicit", - "Legacy configuration infers capability mode.", - ) - if selected_mode == "read" and (proposal_writer is not None or canonical_applier is not None): - return ( - "warning", - "read_authority_shadowed", - "Read mode shadows configured proposal or application authority.", - ) - if selected_mode == "proposal" and proposal_writer is None: - return ( - "warning", - "proposal_access_disabled", - "Proposal surface is present, but mutation access has no writer.", - ) - if selected_mode == "operator": - return ( - "warning", - "operator_mode_reserved", - "Operator mode is valid but currently adds no tools.", - ) - return ( - "passed", - "effective_policy_valid", - "Configured capability and authority are valid.", - ) - - -def run_doctor( - project: ProjectService, - client: str, - *, - config_path: Path | None = None, - server_name: str | None = None, -) -> dict[str, object]: - """Inspect one client binding without loading project sources or mutating state.""" - - if client not in CLIENT_NAMES: - raise DocForgeError( - "unsupported_client", - "Doctor client is unsupported", - client=client, - allowed=list(CLIENT_NAMES), - ) - selected_client = client - descriptor = project.descriptor - validate_descriptor_binding(descriptor) - checks: list[dict[str, object]] = [ - _check( - "project.binding", - "passed", - "project_binding_valid", - "Project descriptor and root binding are valid.", - ), - _check( - "project.canonical_validation", - "skipped", - "canonical_validation_not_run", - "Doctor does not parse canonical project sources.", - ), - _check( - "client.driver", - "passed" if selected_client != "claude" else "warning", - ( - "client_driver_valid" - if selected_client != "claude" - else "client_driver_format_partially_verified" - ), - ( - "Client configuration format is supported." - if selected_client != "claude" - else "Claude fragment syntax is supported, but timeout fields are unverified." - ), - ), - ] - candidate_path = str((config_path or _default_config(selected_client)).expanduser()) - path_error: DocForgeError | None = None - if "\0" in candidate_path or len(candidate_path) > 4_096: - path_hash = hashlib.sha256(candidate_path.encode("utf-8", errors="replace")).hexdigest() - selected_path = Path.cwd() / ".invalid-docforge-client-config" - displayed_path = f"" - path_error = DocForgeError( - "client_config_invalid", - "Client configuration path is empty, oversized, or contains NUL", - path_sha256=path_hash, - ) - else: - selected_path = Path(os.path.abspath(candidate_path)) - displayed_path = str(selected_path) - selected_server_name = server_name - server_name_error: DocForgeError | None = None - if selected_server_name is not None and ( - not selected_server_name or len(selected_server_name) > 256 or "\0" in selected_server_name - ): - name_hash = hashlib.sha256( - selected_server_name.encode("utf-8", errors="replace") - ).hexdigest() - selected_server_name = None - server_name_error = DocForgeError( - "client_entry_invalid", - "Explicit server name is empty, oversized, or contains NUL", - server_name_sha256=name_hash, - ) - entry: dict[str, object] | None = None - binding: dict[str, object] | None = None - servers: dict[str, object] | None = None - initial_config: bytes | None = None - initial_config_identity: tuple[int, int, int, int, int] | None = None - try: - if path_error is not None: - raise path_error - initial_config, initial_config_identity = _read_stable_regular(selected_path) - servers = _server_documents(selected_client, initial_config) - checks.append( - _check( - "client.config", - "passed", - "client_config_valid", - "Client configuration is bounded, stable, and parseable.", - server_count=len(servers), - ) - ) - except DocForgeError as error: - checks.append( - _check( - "client.config", - "failed", - error.code, - error.message, - **error.details, - ) - ) - if servers is None: - checks.append( - _skipped( - "client.entry", - "client_config_unavailable", - "Client entry selection requires one valid configuration.", - ) - ) - elif server_name_error is not None: - checks.append( - _check( - "client.entry", - "failed", - server_name_error.code, - server_name_error.message, - **server_name_error.details, - ) - ) - else: - try: - entry = _entry_for_project( - servers, - client=selected_client, - project_root=descriptor.root, - server_name=selected_server_name, - ) - checks.append( - _check( - "client.entry", - "passed" if entry is not None else "failed", - "client_entry_valid" if entry is not None else "client_entry_missing", - ( - "One client entry uniquely binds this project." - if entry is not None - else "No client entry binds this project." - ), - **({"server_name": entry["server_name"]} if entry is not None else {}), - ) - ) - except DocForgeError as error: - checks.append( - _check( - "client.entry", - "failed", - error.code, - error.message, - **error.details, - ) - ) - - if entry is None: - checks.extend( - ( - _skipped( - "server.executable", - "client_entry_unavailable", - "Executable validation requires one selected client entry.", - ), - _skipped( - "server.arguments", - "client_entry_unavailable", - "Argument validation requires one selected client entry.", - ), - ) - ) - else: - command = cast(str, entry["command"]) - expected = os.path.abspath(sys.executable) - executable_valid = False - if command == expected: - try: - command_path = Path(expected) - command_status = command_path.stat() - executable_valid = ( - command_path.is_absolute() - and stat.S_ISREG(command_status.st_mode) - and os.access(command_path, os.X_OK) - ) - except (OSError, ValueError): - pass - checks.append( - _check( - "server.executable", - "passed" if executable_valid and command == expected else "failed", - ( - "server_executable_valid" - if executable_valid and command == expected - else "server_executable_unexpected" - ), - ( - "Configured server uses the current absolute Python executable." - if executable_valid and command == expected - else "Configured server executable is missing, unsafe, or unexpected." - ), - ) - ) - try: - binding = _parse_binding(cast(list[str], entry["args"])) - checks.append( - _check( - "server.arguments", - "passed", - "server_arguments_valid", - "Server arguments use the closed DocForge option set.", - ) - ) - except DocForgeError as error: - checks.append( - _check( - "server.arguments", - "failed", - error.code, - error.message, - **error.details, - ) - ) - - if entry is None or binding is None: - prerequisite = ( - "client_entry_unavailable" if entry is None else "server_arguments_unavailable" - ) - checks.extend( - _skipped( - check_id, - prerequisite, - "Check requires one selected entry with valid server arguments.", - ) - for check_id in ( - "server.project_binding", - "policy.effective", - "policy.no_ast", - "client.timeouts", - "client.environment", - "client.tool_filter", - ) - ) - else: - configured_root = binding["project_root"] - root_matches = False - if isinstance(configured_root, str): - try: - configured_path = Path(os.path.normpath(configured_root)) - root_matches = configured_path.is_absolute() and configured_path == descriptor.root - except ValueError: - root_matches = False - checks.append( - _check( - "server.project_binding", - "passed" if root_matches else "failed", - "project_binding_matches" if root_matches else "project_binding_mismatch", - ( - "Configured project root matches the inspected project." - if root_matches - else "Configured project root does not match the inspected project." - ), - ) - ) - try: - policy_state, policy_code, policy_message = _runtime_policy_check( - project, - binding, - ) - checks.append( - _check( - "policy.effective", - policy_state, - policy_code, - policy_message, - ) - ) - except DocForgeError as error: - checks.append( - _check( - "policy.effective", - "failed", - error.code, - error.message, - **error.details, - ) - ) - no_ast = cast(bool, binding["no_ast"]) - checks.append( - _check( - "policy.no_ast", - "passed", - "no_ast_policy_valid" if no_ast else "no_ast_disabled", - ( - "No-AST binding blocks Logic publication and retrieval surfaces." - if no_ast - else "No-AST compatibility shorthand is not enabled." - ), - adapter_internals="unverifiable" if no_ast else "not_applicable", - ) - ) - startup = entry["startup_timeout"] - tool = entry["tool_timeout"] - timeouts_valid = ( - type(startup) is int - and 1 <= startup <= 3_600 - and type(tool) is int - and 1 <= tool <= 86_400 - ) - timeouts_invalid = ( - startup is not None and not (type(startup) is int and 1 <= startup <= 3_600) - ) or (tool is not None and not (type(tool) is int and 1 <= tool <= 86_400)) - checks.append( - _check( - "client.timeouts", - ("passed" if timeouts_valid else ("failed" if timeouts_invalid else "warning")), - ( - "client_timeouts_valid" - if timeouts_valid - else ( - "client_timeouts_invalid" - if timeouts_invalid - else "client_timeouts_unverified" - ) - ), - ( - "Client timeouts are explicit and bounded." - if timeouts_valid - else ( - "Client timeout representation is invalid." - if timeouts_invalid - else "Client timeout representation is missing or unverified." - ) - ), - ) - ) - environment_keys = cast(list[str], entry["environment_keys"]) - risky_environment = sorted(set(environment_keys).intersection(RISKY_ENVIRONMENT_KEYS)) - checks.append( - _check( - "client.environment", - "passed" if not environment_keys else "warning", - ( - "client_environment_valid" - if not environment_keys - else ( - "client_environment_risky" - if risky_environment - else "client_environment_present" - ) - ), - ( - "Client entry inherits no secret-bearing environment values." - if not environment_keys - else ( - "Client entry declares environment keys; " - "values were not returned or logged." - ) - ), - environment_keys=environment_keys, - risky_environment_keys=risky_environment, - ) - ) - tool_filter_present = cast(bool, entry["tool_filter_present"]) - parallel_calls = entry["parallel_calls"] - cwd = entry["cwd"] - cwd_valid = True - if cwd is not None: - try: - cwd_valid = _safe_directory(Path(cast(str, cwd))) - except ValueError: - cwd_valid = False - runtime_controls_present = tool_filter_present or parallel_calls is True - checks.append( - _check( - "client.tool_filter", - ( - "failed" - if not cwd_valid - else ("warning" if runtime_controls_present else "passed") - ), - ( - "client_cwd_invalid" - if not cwd_valid - else ( - "client_tool_filter_unverified" - if tool_filter_present - else ( - "client_parallel_calls_unverified" - if parallel_calls is True - else "client_tool_filter_not_configured" - ) - ) - ), - ( - "Client working directory is missing, unsafe, or not absolute." - if not cwd_valid - else ( - "Client runtime filtering or parallel-call controls are not interpreted." - if runtime_controls_present - else "Server-side capability registration is authoritative." - ) - ), - parallel_calls=parallel_calls, - tool_filter_present=tool_filter_present, - ) - ) - - index_state = _safe_regular_state(descriptor.index_path) - checks.append( - _check( - "derived.index", - "warning" if index_state != "present" else "passed", - { - "present": "index_present_unverified", - "missing": "index_missing", - "unsafe": "index_unsafe", - }[index_state], - ( - "Index exists but was not opened or validated." - if index_state == "present" - else ( - "Index is absent and may be built by an explicit bootstrap." - if index_state == "missing" - else "Index path is not a safe regular file." - ) - ), - ) - ) - if initial_config is not None and initial_config_identity is not None: - try: - final_config, final_config_identity = _read_stable_regular(selected_path) - if final_config != initial_config or final_config_identity != initial_config_identity: - raise DocForgeError( - "client_config_changed", - "Client configuration changed during doctor inspection", - ) - except DocForgeError as error: - _replace_check( - checks, - _check( - "client.config", - "failed", - error.code, - error.message, - **error.details, - ), - ) - counts = { - state: sum(check["state"] == state for check in checks) - for state in ("passed", "warning", "failed", "skipped") - } - doctor_state = ( - "unhealthy" if counts["failed"] else ("degraded" if counts["warning"] else "healthy") - ) - validate_descriptor_binding(descriptor) - result: dict[str, object] = { - "status": "ok", - "schema_version": 1, - "doctor_state": doctor_state, - "client": selected_client, - "project": { - "project_id": descriptor.project_id, - "project_root": str(descriptor.root), - "project_root_fingerprint": project_root_fingerprint(descriptor.root), - "adapter": descriptor.adapter, - }, - "config": { - "path": displayed_path, - "server_name": ( - entry["server_name"] - if entry is not None - else ( - selected_server_name - if selected_server_name is not None - else ( - None - if server_name is None - else ( - "" - ) - ) - ) - ), - }, - "summary": counts, - "guarantees": { - "read_only": True, - "project_loads": 0, - "adapter_projection_loads": 0, - "adapter_source_extractions": 0, - "sqlite_opens": 0, - "index_checks": 0, - "index_synchronizations": 0, - "index_builds": 0, - "renders": 0, - "viewer_operations": 0, - "client_config_writes": 0, - "configured_command_executions": 0, - }, - "checks": checks, - } - _validate_doctor_result(result) - return result diff --git a/src/docforge/generation_diff.py b/src/docforge/generation_diff.py deleted file mode 100644 index 4a76a8d..0000000 --- a/src/docforge/generation_diff.py +++ /dev/null @@ -1,911 +0,0 @@ -"""Bounded latest-generation transition receipts for disposable graph indexes.""" - -from __future__ import annotations - -import json -import os -import secrets -import stat -from collections.abc import Mapping -from contextlib import suppress -from dataclasses import dataclass -from pathlib import Path -from typing import Literal, cast - -from ._fs_safety import open_bound_directory, require_bound_directory -from .errors import DocForgeError -from .models import Edge, Node, ProjectDescriptor, ProjectSnapshot -from .pagination import canonical_hash - -GENERATION_DIFF_SCHEMA_VERSION = 1 -GENERATION_DIFF_SEMANTICS_VERSION = 1 -MAX_GENERATION_DIFF_ITEMS = 1_000 -MAX_GENERATION_DIFF_BYTES = 1_048_576 -GENERATION_DIFF_FILENAME = "generation-diff.json" - -IndexSignature = tuple[int, int, int, int, int] -PredecessorReason = Literal[ - "no_predecessor", - "predecessor_unsafe", - "predecessor_unsupported_schema", - "predecessor_foreign", - "predecessor_policy_incompatible", - "predecessor_corrupt", - "predecessor_unattested", - "predecessor_changed", - "no_meaningful_transition", -] - -_GENERATION_KEYS = frozenset( - { - "revision", - "source_hash", - "node_count", - "node_hash", - "edge_count", - "edge_hash", - "index_schema_version", - } -) -_SUMMARY_KEYS = frozenset( - { - "nodes_added", - "nodes_removed", - "nodes_changed", - "edges_added", - "edges_removed", - "total_changes", - } -) -_SIGNATURE_KEYS = frozenset({"device", "inode", "size", "mtime_ns", "ctime_ns"}) -_RECEIPT_KEYS = frozenset( - { - "schema_version", - "diff_semantics_version", - "project_id", - "project_root_fingerprint", - "adapter", - "kind", - "reason", - "from_generation", - "to_generation", - "summary", - "items", - "full_item_count", - "retained_item_count", - "details_truncated", - "truncation_reason", - "full_collection_hash", - "retained_collection_hash", - "index_signature", - "receipt_hash", - } -) -_BASELINE_REASONS = frozenset( - { - "no_predecessor", - "predecessor_unsafe", - "predecessor_unsupported_schema", - "predecessor_foreign", - "predecessor_policy_incompatible", - "predecessor_corrupt", - "predecessor_unattested", - "predecessor_changed", - "no_meaningful_transition", - } -) -_NODE_ITEM_KEYS = frozenset( - { - "entity", - "change", - "node_id", - "before_content_hash", - "after_content_hash", - "before_source_path", - "after_source_path", - "before_node_hash", - "after_node_hash", - "changed_fields", - "item_hash", - } -) -_NODE_CHANGED_FIELDS = frozenset( - { - "title", - "family", - "authority", - "status", - "tags", - "summary", - "content", - "source_path", - "source_anchor", - "content_hash", - } -) -_EDGE_ITEM_KEYS = frozenset( - { - "entity", - "change", - "source_id", - "relation", - "target_id", - "item_hash", - } -) - - -@dataclass(frozen=True) -class PublishedGraph: - """One completely verified predecessor publication.""" - - generation: dict[str, object] - nodes: tuple[Node, ...] - edges: tuple[Edge, ...] - logic_hash: str - signature: IndexSignature - - -@dataclass(frozen=True) -class GenerationDiffDraft: - """A receipt body awaiting the committed index file identity.""" - - fields: dict[str, object] - items: tuple[dict[str, object], ...] - - -def generation_diff_path(descriptor: ProjectDescriptor) -> Path: - """Return the fixed project-confined latest-transition receipt path.""" - - return descriptor.cache_root / GENERATION_DIFF_FILENAME - - -def index_signature(path: Path) -> IndexSignature: - """Return the exact identity of one safe regular index publication.""" - - try: - status = path.lstat() - except OSError as error: - raise DocForgeError("missing_index", "Derived index does not exist") from error - if stat.S_ISLNK(status.st_mode) or not stat.S_ISREG(status.st_mode): - raise DocForgeError("path_escape", "Derived index path is not a safe regular file") - return ( - status.st_dev, - status.st_ino, - status.st_size, - status.st_mtime_ns, - status.st_ctime_ns, - ) - - -def signature_payload(signature: IndexSignature) -> dict[str, int]: - """Convert one stat identity to its versioned JSON representation.""" - - return { - "device": signature[0], - "inode": signature[1], - "size": signature[2], - "mtime_ns": signature[3], - "ctime_ns": signature[4], - } - - -def generation_identity(status: Mapping[str, object]) -> dict[str, object]: - """Select the primary-graph identity stored in a public diff receipt.""" - - return { - "revision": status["revision"], - "source_hash": status["source_hash"], - "node_count": status["node_count"], - "node_hash": status["node_hash"], - "edge_count": status["edge_count"], - "edge_hash": status["edge_hash"], - "index_schema_version": status["index_schema_version"], - } - - -def prepare_generation_diff( - descriptor: ProjectDescriptor, - *, - predecessor: PublishedGraph | None, - predecessor_reason: PredecessorReason | None, - current_snapshot: ProjectSnapshot, - current_status: Mapping[str, object], - preserved_receipt: Mapping[str, object] | None = None, -) -> GenerationDiffDraft: - """Prepare one deterministic latest transition without publishing it.""" - - current_generation = generation_identity(current_status) - if predecessor is None: - return _baseline_draft( - descriptor, - current_generation, - predecessor_reason or "no_predecessor", - ) - - if predecessor.generation == current_generation: - if preserved_receipt is not None and _receipt_targets( - preserved_receipt, - descriptor, - current_generation, - predecessor.signature, - ): - fields = { - key: value - for key, value in preserved_receipt.items() - if key not in {"index_signature", "receipt_hash", "items"} - } - preserved_items = preserved_receipt.get("items") - if not isinstance(preserved_items, list): - raise DocForgeError( - "invalid_generation_diff", - "Preserved generation diff has no item collection", - ) - items = tuple( - cast(dict[str, object], item) - for item in cast(list[object], preserved_items) - if isinstance(item, dict) - ) - return GenerationDiffDraft(fields=fields, items=items) - return _baseline_draft( - descriptor, - current_generation, - "no_meaningful_transition", - ) - - items = _change_items( - predecessor.nodes, - predecessor.edges, - current_snapshot.nodes, - current_snapshot.edges, - ) - summary = _summary(items) - return GenerationDiffDraft( - fields={ - "schema_version": GENERATION_DIFF_SCHEMA_VERSION, - "diff_semantics_version": GENERATION_DIFF_SEMANTICS_VERSION, - "project_id": descriptor.project_id, - "project_root_fingerprint": _root_fingerprint(descriptor), - "adapter": descriptor.adapter, - "kind": "transition", - "reason": None, - "from_generation": predecessor.generation, - "to_generation": current_generation, - "summary": summary, - "full_item_count": len(items), - "full_collection_hash": canonical_hash([item["item_hash"] for item in items]), - }, - items=items, - ) - - -def finalize_generation_diff( - draft: GenerationDiffDraft, - *, - signature: IndexSignature, -) -> dict[str, object]: - """Bind a draft to the committed index and enforce fixed receipt ceilings.""" - - full_count = cast(int, draft.fields["full_item_count"]) - retained = list(draft.items[:MAX_GENERATION_DIFF_ITEMS]) - item_limited = full_count > len(retained) - byte_limited = draft.fields.get("truncation_reason") == "receipt_byte_limit" - while True: - receipt = _final_receipt( - draft.fields, - retained, - signature=signature, - item_limited=item_limited, - byte_limited=byte_limited, - ) - if len(_receipt_bytes(receipt)) <= MAX_GENERATION_DIFF_BYTES: - return receipt - if not retained: - raise DocForgeError( - "generation_diff_failure", - "Generation diff identity exceeds the fixed receipt byte limit", - ) - retained.pop() - byte_limited = True - - -def publish_generation_diff( - descriptor: ProjectDescriptor, - receipt: Mapping[str, object], -) -> None: - """Atomically replace the single latest-generation receipt.""" - - root = descriptor.cache_root - if generation_diff_path(descriptor).parent != root: - raise DocForgeError("path_escape", "Generation diff path is not confined") - raw = _receipt_bytes(receipt) - if len(raw) > MAX_GENERATION_DIFF_BYTES: - raise DocForgeError( - "generation_diff_failure", - "Generation diff receipt exceeds the fixed byte limit", - ) - root_fd = open_bound_directory(root) - temporary_name = f".generation-diff-{secrets.token_hex(12)}" - temporary_created = False - try: - try: - status = os.stat( - GENERATION_DIFF_FILENAME, - dir_fd=root_fd, - follow_symlinks=False, - ) - except FileNotFoundError: - status = None - if status is not None and ( - stat.S_ISLNK(status.st_mode) or not stat.S_ISREG(status.st_mode) - ): - raise DocForgeError( - "path_escape", - "Generation diff receipt path is not a safe regular file", - ) - descriptor_fd = os.open( - temporary_name, - os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, - 0o600, - dir_fd=root_fd, - ) - temporary_created = True - with os.fdopen(descriptor_fd, "wb") as handle: - handle.write(raw) - handle.flush() - os.fsync(handle.fileno()) - require_bound_directory(root, root_fd) - os.replace( - temporary_name, - GENERATION_DIFF_FILENAME, - src_dir_fd=root_fd, - dst_dir_fd=root_fd, - ) - temporary_created = False - os.fsync(root_fd) - require_bound_directory(root, root_fd) - except Exception: - if temporary_created: - with suppress(OSError): - os.unlink(temporary_name, dir_fd=root_fd) - raise - finally: - os.close(root_fd) - - -def load_generation_diff( - descriptor: ProjectDescriptor, -) -> tuple[dict[str, object] | None, str | None, IndexSignature | None]: - """Read and validate one receipt without repairing any derived state.""" - - try: - root_fd = open_bound_directory(descriptor.cache_root) - except DocForgeError as error: - reason = "missing_receipt" if error.code == "missing_index" else "unsafe_receipt" - return None, reason, None - try: - try: - descriptor_fd = os.open( - GENERATION_DIFF_FILENAME, - os.O_RDONLY | os.O_NOFOLLOW, - dir_fd=root_fd, - ) - except FileNotFoundError: - return None, "missing_receipt", None - except OSError: - return None, "unsafe_receipt", None - with os.fdopen(descriptor_fd, "rb") as handle: - status_before = os.fstat(handle.fileno()) - if stat.S_ISLNK(status_before.st_mode) or not stat.S_ISREG(status_before.st_mode): - return None, "unsafe_receipt", None - before: IndexSignature = ( - status_before.st_dev, - status_before.st_ino, - status_before.st_size, - status_before.st_mtime_ns, - status_before.st_ctime_ns, - ) - if before[2] > MAX_GENERATION_DIFF_BYTES: - return None, "oversized_receipt", before - raw = handle.read(MAX_GENERATION_DIFF_BYTES + 1) - status_after = os.fstat(handle.fileno()) - after: IndexSignature = ( - status_after.st_dev, - status_after.st_ino, - status_after.st_size, - status_after.st_mtime_ns, - status_after.st_ctime_ns, - ) - if len(raw) > MAX_GENERATION_DIFF_BYTES: - return None, "oversized_receipt", before - try: - path_status = os.stat( - GENERATION_DIFF_FILENAME, - dir_fd=root_fd, - follow_symlinks=False, - ) - except OSError: - return None, "receipt_changed", before - path_signature: IndexSignature = ( - path_status.st_dev, - path_status.st_ino, - path_status.st_size, - path_status.st_mtime_ns, - path_status.st_ctime_ns, - ) - try: - require_bound_directory(descriptor.cache_root, root_fd) - except DocForgeError: - return None, "unsafe_receipt", before - finally: - os.close(root_fd) - if before != after or before != path_signature or len(raw) != before[2]: - return None, "receipt_changed", before - try: - parsed: object = json.loads(raw.decode("utf-8")) - except (UnicodeDecodeError, json.JSONDecodeError): - return None, "corrupt_receipt", before - if not isinstance(parsed, dict): - return None, "corrupt_receipt", before - receipt = cast(dict[str, object], parsed) - if not validate_generation_diff_receipt(receipt): - return None, "corrupt_receipt", before - if ( - receipt.get("project_id") != descriptor.project_id - or receipt.get("project_root_fingerprint") != _root_fingerprint(descriptor) - or receipt.get("adapter") != descriptor.adapter - ): - return None, "foreign_receipt", before - return receipt, None, before - - -def valid_generation_identity(value: object) -> bool: - """Return whether a primary-graph generation has the exact version-1 shape.""" - - return _valid_generation(value) - - -def validate_generation_diff_receipt( - receipt: Mapping[str, object], - *, - descriptor: ProjectDescriptor | None = None, -) -> bool: - """Strictly validate the complete version-1 receipt and its hashes.""" - - if frozenset(receipt) != _RECEIPT_KEYS: - return False - if ( - receipt.get("schema_version") != GENERATION_DIFF_SCHEMA_VERSION - or receipt.get("diff_semantics_version") != GENERATION_DIFF_SEMANTICS_VERSION - or not _nonempty(receipt.get("project_id")) - or not _fingerprint(receipt.get("project_root_fingerprint")) - or not _nonempty(receipt.get("adapter")) - ): - return False - if descriptor is not None and ( - receipt["project_id"] != descriptor.project_id - or receipt["project_root_fingerprint"] != _root_fingerprint(descriptor) - or receipt["adapter"] != descriptor.adapter - ): - return False - kind = receipt.get("kind") - reason = receipt.get("reason") - from_generation = receipt.get("from_generation") - if kind == "baseline": - if reason not in _BASELINE_REASONS or from_generation is not None: - return False - elif kind == "transition": - if ( - reason is not None - or not _valid_generation(from_generation) - or from_generation == receipt.get("to_generation") - ): - return False - else: - return False - if not _valid_generation(receipt.get("to_generation")): - return False - summary_value = receipt.get("summary") - items_value = receipt.get("items") - if not isinstance(summary_value, dict) or not isinstance(items_value, list): - return False - summary = cast(dict[str, object], summary_value) - items = cast(list[object], items_value) - if frozenset(summary) != _SUMMARY_KEYS or any( - type(value) is not int or value < 0 for value in summary.values() - ): - return False - total = sum( - cast(int, summary[key]) - for key in ( - "nodes_added", - "nodes_removed", - "nodes_changed", - "edges_added", - "edges_removed", - ) - ) - if summary.get("total_changes") != total: - return False - if kind == "baseline" and (total != 0 or items): - return False - if len(items) > MAX_GENERATION_DIFF_ITEMS or not all(_valid_item(item) for item in items): - return False - typed_items = tuple(cast(dict[str, object], item) for item in items) - if list(typed_items) != sorted(typed_items, key=_item_sort_key): - return False - identities = tuple(_item_identity(item) for item in typed_items) - if len(identities) != len(set(identities)): - return False - retained_summary = _summary(typed_items) - if any( - retained_summary[key] > cast(int, summary[key]) - for key in ( - "nodes_added", - "nodes_removed", - "nodes_changed", - "edges_added", - "edges_removed", - ) - ): - return False - full_count = receipt.get("full_item_count") - retained_count = receipt.get("retained_item_count") - truncated = receipt.get("details_truncated") - truncation_reason = receipt.get("truncation_reason") - if ( - type(full_count) is not int - or type(retained_count) is not int - or full_count != total - or retained_count != len(items) - or retained_count > full_count - or type(truncated) is not bool - or truncated != (retained_count < full_count) - ): - return False - if truncation_reason is None: - if truncated or full_count > MAX_GENERATION_DIFF_ITEMS: - return False - elif truncation_reason == "receipt_item_limit": - if ( - not truncated - or full_count <= MAX_GENERATION_DIFF_ITEMS - or retained_count != MAX_GENERATION_DIFF_ITEMS - ): - return False - elif truncation_reason == "receipt_byte_limit": - if not truncated or retained_count >= min(full_count, MAX_GENERATION_DIFF_ITEMS): - return False - else: - return False - item_hashes = [cast(dict[str, object], item)["item_hash"] for item in items] - retained_hash = canonical_hash(item_hashes) - if receipt.get("retained_collection_hash") != retained_hash: - return False - full_hash = receipt.get("full_collection_hash") - if not _sha256(full_hash): - return False - if not truncated and full_hash != retained_hash: - return False - signature_value = receipt.get("index_signature") - if not isinstance(signature_value, dict): - return False - signature = cast(dict[str, object], signature_value) - if frozenset(signature) != _SIGNATURE_KEYS: - return False - if any(type(value) is not int or value < 0 for value in signature.values()): - return False - receipt_hash = receipt.get("receipt_hash") - if not _sha256(receipt_hash): - return False - unhashed = {key: value for key, value in receipt.items() if key != "receipt_hash"} - return receipt_hash == canonical_hash(unhashed) - - -def _baseline_draft( - descriptor: ProjectDescriptor, - current_generation: Mapping[str, object], - reason: PredecessorReason, -) -> GenerationDiffDraft: - empty_hash = canonical_hash([]) - return GenerationDiffDraft( - fields={ - "schema_version": GENERATION_DIFF_SCHEMA_VERSION, - "diff_semantics_version": GENERATION_DIFF_SEMANTICS_VERSION, - "project_id": descriptor.project_id, - "project_root_fingerprint": _root_fingerprint(descriptor), - "adapter": descriptor.adapter, - "kind": "baseline", - "reason": reason, - "from_generation": None, - "to_generation": dict(current_generation), - "summary": { - "nodes_added": 0, - "nodes_removed": 0, - "nodes_changed": 0, - "edges_added": 0, - "edges_removed": 0, - "total_changes": 0, - }, - "full_item_count": 0, - "full_collection_hash": empty_hash, - }, - items=(), - ) - - -def _change_items( - before_nodes: tuple[Node, ...], - before_edges: tuple[Edge, ...], - after_nodes: tuple[Node, ...], - after_edges: tuple[Edge, ...], -) -> tuple[dict[str, object], ...]: - before_by_id = {node.node_id: node for node in before_nodes} - after_by_id = {node.node_id: node for node in after_nodes} - items: list[dict[str, object]] = [] - for node_id in sorted(before_by_id.keys() | after_by_id.keys()): - before = before_by_id.get(node_id) - after = after_by_id.get(node_id) - if before == after: - continue - if before is None: - change = "added" - elif after is None: - change = "removed" - else: - change = "changed" - before_payload = before.as_dict() if before is not None else None - after_payload = after.as_dict() if after is not None else None - changed_fields = ( - [] - if before_payload is None or after_payload is None - else sorted(key for key in before_payload if before_payload[key] != after_payload[key]) - ) - payload: dict[str, object] = { - "entity": "node", - "change": change, - "node_id": node_id, - "before_content_hash": None if before is None else before.content_hash, - "after_content_hash": None if after is None else after.content_hash, - "before_source_path": None if before is None else before.source_path, - "after_source_path": None if after is None else after.source_path, - "before_node_hash": ( - None if before_payload is None else canonical_hash(before_payload) - ), - "after_node_hash": (None if after_payload is None else canonical_hash(after_payload)), - "changed_fields": changed_fields, - } - payload["item_hash"] = canonical_hash(payload) - items.append(payload) - before_edge_set = {(edge.source_id, edge.relation, edge.target_id) for edge in before_edges} - after_edge_set = {(edge.source_id, edge.relation, edge.target_id) for edge in after_edges} - for change, values in ( - ("removed", sorted(before_edge_set - after_edge_set)), - ("added", sorted(after_edge_set - before_edge_set)), - ): - for source_id, relation, target_id in values: - payload = { - "entity": "edge", - "change": change, - "source_id": source_id, - "relation": relation, - "target_id": target_id, - } - payload["item_hash"] = canonical_hash(payload) - items.append(payload) - return tuple(sorted(items, key=_item_sort_key)) - - -def _item_sort_key(item: Mapping[str, object]) -> tuple[str, str, str, str, str]: - return ( - cast(str, item["entity"]), - cast(str, item.get("node_id", item.get("source_id", ""))), - cast(str, item.get("relation", "")), - cast(str, item.get("target_id", "")), - cast(str, item["change"]), - ) - - -def _item_identity(item: Mapping[str, object]) -> tuple[str, ...]: - if item["entity"] == "node": - return ("node", cast(str, item["node_id"])) - return ( - "edge", - cast(str, item["source_id"]), - cast(str, item["relation"]), - cast(str, item["target_id"]), - ) - - -def _summary(items: tuple[dict[str, object], ...]) -> dict[str, int]: - result = { - "nodes_added": 0, - "nodes_removed": 0, - "nodes_changed": 0, - "edges_added": 0, - "edges_removed": 0, - "total_changes": len(items), - } - for item in items: - entity = cast(str, item["entity"]) - change = cast(str, item["change"]) - key = f"{entity}s_{change}" - result[key] += 1 - return result - - -def _final_receipt( - fields: Mapping[str, object], - retained: list[dict[str, object]], - *, - signature: IndexSignature, - item_limited: bool, - byte_limited: bool, -) -> dict[str, object]: - full_count = cast(int, fields["full_item_count"]) - truncated = len(retained) < full_count - if byte_limited: - reason: str | None = "receipt_byte_limit" - elif item_limited: - reason = "receipt_item_limit" - else: - reason = None - receipt = { - **fields, - "items": retained, - "retained_item_count": len(retained), - "details_truncated": truncated, - "truncation_reason": reason, - "retained_collection_hash": canonical_hash([item["item_hash"] for item in retained]), - "index_signature": signature_payload(signature), - } - receipt["receipt_hash"] = canonical_hash(receipt) - return receipt - - -def _receipt_targets( - receipt: Mapping[str, object], - descriptor: ProjectDescriptor, - generation: Mapping[str, object], - signature: IndexSignature, -) -> bool: - return ( - validate_generation_diff_receipt(receipt, descriptor=descriptor) - and receipt.get("to_generation") == dict(generation) - and receipt.get("index_signature") == signature_payload(signature) - ) - - -def _valid_generation(value: object) -> bool: - if not isinstance(value, dict): - return False - generation = cast(dict[str, object], value) - if frozenset(generation) != _GENERATION_KEYS: - return False - source_hash = generation.get("source_hash") - node_hash = generation.get("node_hash") - edge_hash = generation.get("edge_hash") - return ( - _nonempty(generation.get("revision")) - and _sha256(source_hash) - and _sha256(node_hash) - and _sha256(edge_hash) - and type(generation.get("node_count")) is int - and cast(int, generation["node_count"]) >= 0 - and type(generation.get("edge_count")) is int - and cast(int, generation["edge_count"]) >= 0 - and type(generation.get("index_schema_version")) is int - and cast(int, generation["index_schema_version"]) >= 1 - ) - - -def _valid_item(value: object) -> bool: - if not isinstance(value, dict): - return False - payload = cast(dict[str, object], value) - entity = payload.get("entity") - if entity == "node": - if frozenset(payload) != _NODE_ITEM_KEYS: - return False - changed_fields_value = payload.get("changed_fields") - if not isinstance(changed_fields_value, list): - return False - changed_fields = cast(list[object], changed_fields_value) - if ( - payload.get("change") not in {"added", "removed", "changed"} - or not _nonempty(payload.get("node_id")) - or any( - not _nonempty(field) or field not in _NODE_CHANGED_FIELDS - for field in changed_fields - ) - or changed_fields != sorted(set(cast(list[str], changed_fields))) - ): - return False - for key in ( - "before_content_hash", - "after_content_hash", - "before_node_hash", - "after_node_hash", - ): - candidate = payload.get(key) - if candidate is not None and not _sha256(candidate): - return False - for key in ("before_source_path", "after_source_path"): - candidate = payload.get(key) - if candidate is not None and not _nonempty(candidate): - return False - change = payload["change"] - before_values = ( - payload["before_content_hash"], - payload["before_source_path"], - payload["before_node_hash"], - ) - after_values = ( - payload["after_content_hash"], - payload["after_source_path"], - payload["after_node_hash"], - ) - if change == "added": - if ( - any(value is not None for value in before_values) - or not all(value is not None for value in after_values) - or changed_fields - ): - return False - elif change == "removed": - if ( - not all(value is not None for value in before_values) - or any(value is not None for value in after_values) - or changed_fields - ): - return False - elif ( - not all(value is not None for value in (*before_values, *after_values)) - or not changed_fields - or payload["before_node_hash"] == payload["after_node_hash"] - ): - return False - elif entity == "edge": - if ( - frozenset(payload) != _EDGE_ITEM_KEYS - or payload.get("change") not in {"added", "removed"} - or not _nonempty(payload.get("source_id")) - or not _nonempty(payload.get("relation")) - or not _nonempty(payload.get("target_id")) - ): - return False - else: - return False - item_hash = payload.get("item_hash") - unhashed = {key: item for key, item in payload.items() if key != "item_hash"} - return _sha256(item_hash) and item_hash == canonical_hash(unhashed) - - -def _receipt_bytes(receipt: Mapping[str, object]) -> bytes: - return json.dumps(receipt, sort_keys=True, indent=2, ensure_ascii=True).encode("utf-8") + b"\n" - - -def _root_fingerprint(descriptor: ProjectDescriptor) -> str: - from .project import project_root_fingerprint - - return project_root_fingerprint(descriptor.root) - - -def _nonempty(value: object) -> bool: - return isinstance(value, str) and bool(value) - - -def _sha256(value: object) -> bool: - return ( - isinstance(value, str) - and len(value) == 64 - and all(character in "0123456789abcdef" for character in value) - ) - - -def _fingerprint(value: object) -> bool: - return ( - isinstance(value, str) - and len(value) == 16 - and all(character in "0123456789abcdef" for character in value) - ) diff --git a/src/docforge/index.py b/src/docforge/index.py index 60647df..f0cc06c 100644 --- a/src/docforge/index.py +++ b/src/docforge/index.py @@ -6,33 +6,17 @@ import fcntl import hashlib import json import os -import secrets import sqlite3 -import stat +import tempfile import time from collections import deque from collections.abc import Callable, Generator -from contextlib import contextmanager, suppress +from contextlib import contextmanager from dataclasses import dataclass from pathlib import Path -from typing import Literal, cast +from typing import cast -from ._fs_safety import open_bound_directory, require_bound_directory from .errors import DocForgeError -from .generation_diff import ( - GenerationDiffDraft, - PredecessorReason, - PublishedGraph, - finalize_generation_diff, - generation_diff_path, - generation_identity, - index_signature, - load_generation_diff, - prepare_generation_diff, - publish_generation_diff, - valid_generation_identity, - validate_generation_diff_receipt, -) from .models import ( BuildReportingProject, Edge, @@ -48,25 +32,11 @@ from .models import ( ProjectSnapshot, ProjectState, ) -from .pagination import canonical_hash from .project import project_root_fingerprint -from .retrieval import ( - CapsuleEvidenceV1, - CapsuleOmissionV1, - CapsuleRelationshipV1, - ContextCapsuleV1, - EvidenceGapV1, - RetrievalPlanV1, - capsule_evidence, - finalize_capsule, - relation_category, - validate_retrieval_plan, -) from .telemetry import increment, stage INDEX_SCHEMA_VERSION = 3 APPLICATION_ID = 1_146_683_778 -_SQLITE_PARAMETER_CHUNK = 500 def _node_hash(nodes: tuple[Node, ...]) -> str: @@ -92,27 +62,19 @@ def _logic_hash(projections: tuple[LogicProjection, ...]) -> str: return hashlib.sha256(payload).hexdigest() -def _connect_read_only(path: Path, *, immutable: bool = False) -> sqlite3.Connection: +def _connect_read_only(path: Path) -> sqlite3.Connection: if not path.is_file(): raise DocForgeError("missing_index", "Derived index does not exist; run build first") - immutable_parameter = "&immutable=1" if immutable else "" - connection = sqlite3.connect( - f"file:{path}?mode=ro{immutable_parameter}", - uri=True, - ) + connection = sqlite3.connect(f"file:{path}?mode=ro", uri=True) connection.row_factory = sqlite3.Row return connection @contextmanager -def _read_connection( - path: Path, - *, - immutable: bool = False, -) -> Generator[sqlite3.Connection, None, None]: +def _read_connection(path: Path) -> Generator[sqlite3.Connection, None, None]: connection: sqlite3.Connection | None = None try: - connection = _connect_read_only(path, immutable=immutable) + connection = _connect_read_only(path) yield connection except DocForgeError: raise @@ -185,17 +147,6 @@ class _IndexReadSnapshot: } -@dataclass(frozen=True) -class _TaskSelection: - node_id: str - role: str - reason_code: str - depth: int - category_rank: int - first_edge: tuple[str, str, str] - relationship_path: tuple[CapsuleRelationshipV1, ...] - - class ProjectIndex: """A disposable index that always checks current canonical source before queries.""" @@ -283,24 +234,9 @@ class ProjectIndex: return self._build_locked_core() def _build_locked_core(self) -> dict[str, object]: - predecessor, predecessor_reason, predecessor_signature = self._capture_predecessor() - if predecessor_reason == "predecessor_unsafe": - raise DocForgeError( - "path_escape", - "Derived predecessor is not a safe regular file", - ) - preserved_receipt, _, _ = load_generation_diff(self.project.descriptor) snapshot = self.project.load() logic = self._logic_projections() status = _status(snapshot, logic) - draft = prepare_generation_diff( - snapshot.descriptor, - predecessor=predecessor, - predecessor_reason=predecessor_reason, - current_snapshot=snapshot, - current_status=status, - preserved_receipt=preserved_receipt, - ) build_report = ( self.project.build_report() if isinstance(self.project, BuildReportingProject) else None ) @@ -308,18 +244,10 @@ class ProjectIndex: build_report = None cache_root = snapshot.descriptor.cache_root cache_root.mkdir(parents=True, exist_ok=True) - if self.path.parent != cache_root: - raise DocForgeError("path_escape", "Derived index path is not confined") - cache_root_fd = open_bound_directory(cache_root) - temporary_name = f"index-{secrets.token_hex(12)}.sqlite3" - temporary_descriptor = os.open( - temporary_name, - os.O_RDWR | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, - 0o600, - dir_fd=cache_root_fd, - ) - os.close(temporary_descriptor) - temporary = Path(f"/proc/self/fd/{cache_root_fd}/{temporary_name}") + with tempfile.NamedTemporaryFile( + prefix="index-", suffix=".sqlite3", dir=cache_root, delete=False + ) as descriptor: + temporary = Path(descriptor.name) try: connection = sqlite3.connect(temporary) try: @@ -467,367 +395,25 @@ class ProjectIndex: if ( current.source_hash != snapshot.source_hash or current.revision != snapshot.revision - or current.nodes != snapshot.nodes - or current.edges != snapshot.edges or current_logic != logic ): raise DocForgeError("source_changed", "Canonical source changed during index build") - if ( - predecessor is not None - and predecessor.generation["source_hash"] == status["source_hash"] - and predecessor.generation["revision"] == status["revision"] - and ( - predecessor.generation != generation_identity(status) - or predecessor.logic_hash != status["logic_hash"] - ) - ): - raise DocForgeError( - "generation_collision", - "One canonical generation produced different graph content", - ) - if predecessor is None: - if predecessor_signature is None: - if self.path.exists() or self.path.is_symlink(): - raise DocForgeError( - "source_changed", - "Derived index appeared during index build", - ) - elif index_signature(self.path) != predecessor_signature: - raise DocForgeError( - "source_changed", - "Derived predecessor changed during index build", - ) - elif index_signature(self.path) != predecessor.signature: - raise DocForgeError( - "source_changed", - "Derived predecessor changed during index build", - ) - self._require_no_index_sidecars() - require_bound_directory(cache_root, cache_root_fd) - try: - os.replace( - temporary_name, - self.path.name, - src_dir_fd=cache_root_fd, - dst_dir_fd=cache_root_fd, - ) - except OSError as error: - raise DocForgeError( - "index_failure", - "Could not publish the derived index", - ) from error - publication_errors: list[dict[str, object]] = [] - try: - self._fsync_cache_directory(cache_root_fd) - except Exception as error: - publication_errors.append( - { - "stage": "index_directory_sync", - "error_type": type(error).__name__, - } - ) - try: - published_signature = self._published_index_signature(cache_root_fd) - except Exception as error: - publication_errors.append( - { - "stage": "index_identity", - "error_type": type(error).__name__, - } - ) - self._verified_index_signature = None - publication = self._unavailable_post_commit_publication(publication_errors) - else: - self._verified_index_signature = published_signature - try: - publication = self._publish_post_commit_receipts( - current, - draft, - signature=published_signature, - initial_errors=publication_errors, - ) - except Exception as error: - publication_errors.append( - { - "stage": "receipt_pipeline", - "error_type": type(error).__name__, - } - ) - publication = self._unavailable_post_commit_publication(publication_errors) + os.replace(temporary, self.path) + self._verified_index_signature = self._index_signature() + self._write_attestation() + if isinstance(self.project, GenerationRecordingProject): + self.project.record_generation(current) except sqlite3.Error as error: - with suppress(OSError): - os.unlink(temporary_name, dir_fd=cache_root_fd) + temporary.unlink(missing_ok=True) raise DocForgeError("index_failure", "Could not build the derived index") from error except Exception: - with suppress(OSError): - os.unlink(temporary_name, dir_fd=cache_root_fd) + temporary.unlink(missing_ok=True) raise - finally: - os.close(cache_root_fd) result: dict[str, object] = {**status, "database": str(self.path)} if build_report is not None: result["build"] = build_report - if publication["state"] == "degraded": - result["publication"] = publication return result - def _capture_predecessor( - self, - ) -> tuple[PublishedGraph | None, PredecessorReason, tuple[int, int, int, int, int] | None]: - """Read one predecessor completely without repairing any derived receipt.""" - - if not self.path.exists() and not self.path.is_symlink(): - return None, "no_predecessor", None - self._require_no_index_sidecars() - try: - signature = index_signature(self.path) - except DocForgeError: - return None, "predecessor_unsafe", None - if not self._attestation_matches(expected_signature=signature): - return None, "predecessor_unattested", signature - try: - with _read_connection(self.path, immutable=True) as connection: - application_id = connection.execute("PRAGMA application_id").fetchone()[0] - schema_version = connection.execute("PRAGMA user_version").fetchone()[0] - if application_id != APPLICATION_ID or schema_version != INDEX_SCHEMA_VERSION: - return None, "predecessor_unsupported_schema", signature - metadata = dict(connection.execute("SELECT key, value FROM metadata")) - descriptor = self.project.descriptor - identity = { - "project_id": descriptor.project_id, - "project_root_fingerprint": project_root_fingerprint(descriptor.root), - "adapter": descriptor.adapter, - "index_schema_version": str(INDEX_SCHEMA_VERSION), - } - if ( - any(metadata.get(key) != value for key, value in identity.items()) - or metadata.get("status") != "ok" - ): - return None, "predecessor_foreign", signature - integrity = connection.execute("PRAGMA integrity_check").fetchone() - if integrity is None or integrity[0] != "ok": - return None, "predecessor_corrupt", signature - nodes = tuple( - _row_to_node(row) - for row in connection.execute("SELECT * FROM nodes ORDER BY node_id") - ) - edges = tuple( - Edge(*row) - for row in connection.execute( - "SELECT source_id, relation, target_id FROM edges " - "ORDER BY source_id, relation, target_id" - ) - ) - logic = _logic_from_connection(connection) - fts_count = connection.execute("SELECT COUNT(*) FROM node_fts").fetchone()[0] - node_hash = _node_hash(nodes) - edge_hash = _edge_hash(edges) - logic_hash = _logic_hash(logic) - logic_node_count = sum(len(projection.nodes) for projection in logic) - logic_edge_count = sum(len(projection.edges) for projection in logic) - if ( - metadata.get("node_hash") != node_hash - or metadata.get("node_count") != str(len(nodes)) - or metadata.get("edge_hash") != edge_hash - or metadata.get("edge_count") != str(len(edges)) - or metadata.get("logic_hash") != logic_hash - or metadata.get("logic_projection_count") != str(len(logic)) - or metadata.get("logic_node_count") != str(logic_node_count) - or metadata.get("logic_edge_count") != str(logic_edge_count) - or fts_count != len(nodes) - ): - return None, "predecessor_corrupt", signature - if logic and not self.allow_logic: - return None, "predecessor_policy_incompatible", signature - generation: dict[str, object] = { - "revision": metadata["revision"], - "source_hash": metadata["source_hash"], - "node_count": len(nodes), - "node_hash": node_hash, - "edge_count": len(edges), - "edge_hash": edge_hash, - "index_schema_version": INDEX_SCHEMA_VERSION, - } - if not valid_generation_identity(generation): - return None, "predecessor_corrupt", signature - except ( - DocForgeError, - sqlite3.Error, - json.JSONDecodeError, - KeyError, - TypeError, - ValueError, - ): - return None, "predecessor_corrupt", signature - self._require_no_index_sidecars() - try: - if index_signature(self.path) != signature: - return None, "predecessor_changed", signature - except DocForgeError: - return None, "predecessor_changed", signature - return ( - PublishedGraph( - generation=generation, - nodes=nodes, - edges=edges, - logic_hash=logic_hash, - signature=signature, - ), - "no_predecessor", - signature, - ) - - def _publish_post_commit_receipts( - self, - snapshot: ProjectSnapshot, - draft: GenerationDiffDraft, - *, - signature: tuple[int, int, int, int, int], - initial_errors: list[dict[str, object]] | None = None, - ) -> dict[str, object]: - """Publish independent evidence without misreporting a committed index.""" - - receipts: dict[str, dict[str, object]] = {} - errors = list(initial_errors or ()) - - def attempt(name: str, operation: Callable[[], None]) -> None: - try: - if self._index_signature() != signature: - raise DocForgeError( - "invalid_index", - "Committed index changed before receipt publication", - ) - operation() - if self._index_signature() != signature: - raise DocForgeError( - "invalid_index", - "Committed index changed during receipt publication", - ) - except Exception as error: - receipts[name] = {"state": "unavailable", "reason": "publication_failed"} - errors.append( - { - "stage": name, - "error_type": type(error).__name__, - } - ) - else: - receipts[name] = {"state": "published"} - - attempt("attestation", lambda: self._write_attestation(expected_signature=signature)) - if isinstance(self.project, GenerationRecordingProject): - record_generation = self.project.record_generation - attempt("source_generation", lambda: record_generation(snapshot)) - else: - receipts["source_generation"] = { - "state": "unavailable", - "reason": "project_does_not_record_generation", - } - - finalized: dict[str, object] | None = None - - def publish_diff() -> None: - nonlocal finalized - finalized = finalize_generation_diff(draft, signature=signature) - if not validate_generation_diff_receipt( - finalized, - descriptor=self.project.descriptor, - ): - raise DocForgeError( - "invalid_generation_diff", - "Finalized generation diff failed its publication contract", - ) - publish_generation_diff(self.project.descriptor, finalized) - - attempt("generation_diff", publish_diff) - if finalized is not None and receipts["generation_diff"]["state"] == "published": - receipts["generation_diff"].update( - { - "kind": finalized["kind"], - "receipt_hash": finalized["receipt_hash"], - "full_item_count": finalized["full_item_count"], - "retained_item_count": finalized["retained_item_count"], - "details_truncated": finalized["details_truncated"], - } - ) - return { - "state": "degraded" if errors else "published", - "index": "published", - "receipts": receipts, - "errors": errors, - } - - @staticmethod - def _unavailable_post_commit_publication( - errors: list[dict[str, object]], - ) -> dict[str, object]: - return { - "state": "degraded", - "index": "published", - "receipts": { - name: { - "state": "unavailable", - "reason": "index_identity_unavailable", - } - for name in ("attestation", "source_generation", "generation_diff") - }, - "errors": errors, - } - - def _require_no_index_sidecars(self) -> None: - """Refuse SQLite state that is not contained in the main index inode.""" - - for suffix in ("-wal", "-journal", "-shm"): - sidecar = Path(f"{self.path}{suffix}") - try: - sidecar.lstat() - except FileNotFoundError: - continue - raise DocForgeError( - "index_busy", - "Derived index has active or unproven SQLite sidecar state", - sidecar=suffix, - ) - - def _published_index_signature( - self, - cache_root_fd: int, - ) -> tuple[int, int, int, int, int]: - require_bound_directory(self.project.descriptor.cache_root, cache_root_fd) - try: - status = os.stat( - self.path.name, - dir_fd=cache_root_fd, - follow_symlinks=False, - ) - except OSError as error: - raise DocForgeError( - "missing_index", - "Committed index identity is unavailable", - ) from error - if not stat.S_ISREG(status.st_mode) or stat.S_ISLNK(status.st_mode): - raise DocForgeError( - "path_escape", - "Committed index is not a safe regular file", - ) - signature = ( - status.st_dev, - status.st_ino, - status.st_size, - status.st_mtime_ns, - status.st_ctime_ns, - ) - if index_signature(self.path) != signature: - raise DocForgeError( - "invalid_index", - "Committed index path is not bound to its publication inode", - ) - return signature - - @staticmethod - def _fsync_cache_directory(cache_root_fd: int) -> None: - os.fsync(cache_root_fd) - @contextmanager def _build_lock(self) -> Generator[None, None, None]: cache_root = self.project.descriptor.cache_root @@ -853,7 +439,6 @@ class ProjectIndex: candidates = ( *self.project.descriptor.cache_root.glob("index-*.sqlite3"), *self.project.descriptor.cache_root.glob(".index-attestation-*"), - *self.project.descriptor.cache_root.glob(".generation-diff-*"), ) for path in sorted(candidates): if path == self.path or path.is_symlink() or not path.is_file(): @@ -1130,49 +715,15 @@ class ProjectIndex: "database": str(self.path), } - def _attestation_matches( - self, - *, - expected_signature: tuple[int, int, int, int, int] | None = None, - ) -> bool: + def _attestation_matches(self) -> bool: """Verify a persisted whole-file digest before trusting a warm derived index.""" path = self.attestation_path - try: - before_path = index_signature(path) - descriptor = os.open(path, os.O_RDONLY | os.O_NOFOLLOW) - with os.fdopen(descriptor, "rb") as handle: - before = os.fstat(handle.fileno()) - raw = handle.read(4_097) - after = os.fstat(handle.fileno()) - after_path = index_signature(path) - except (DocForgeError, OSError): - return False - before_signature = ( - before.st_dev, - before.st_ino, - before.st_size, - before.st_mtime_ns, - before.st_ctime_ns, - ) - after_signature = ( - after.st_dev, - after.st_ino, - after.st_size, - after.st_mtime_ns, - after.st_ctime_ns, - ) - if ( - before_path != before_signature - or after_path != before_signature - or after_signature != before_signature - or len(raw) > 4_096 - or len(raw) != before.st_size - ): + if not path.is_file() or path.is_symlink(): return False try: - parsed: object = json.loads(raw.decode("utf-8")) - except (UnicodeDecodeError, json.JSONDecodeError): + parsed: object = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError): return False if not isinstance(parsed, dict): return False @@ -1180,660 +731,71 @@ class ProjectIndex: expected_size = payload.get("index_size") expected_hash = payload.get("index_sha256") if ( - set(payload) != {"schema_version", "index_size", "index_sha256"} - or payload.get("schema_version") != 1 + payload.get("schema_version") != 1 or type(expected_size) is not int or not isinstance(expected_hash, str) or len(expected_hash) != 64 - or any(character not in "0123456789abcdef" for character in expected_hash) ): return False try: - size, actual_hash, _ = self._index_digest(expected_signature=expected_signature) - except DocForgeError: + if self.path.stat().st_size != expected_size: + return False + with self.path.open("rb") as handle: + actual_hash = hashlib.file_digest(handle, "sha256").hexdigest() + except OSError: return False - return size == expected_size and actual_hash == expected_hash + return actual_hash == expected_hash - def _index_digest( - self, - *, - expected_signature: tuple[int, int, int, int, int] | None = None, - ) -> tuple[int, str, tuple[int, int, int, int, int]]: - """Hash one exact safe index inode and prove it stayed path-bound.""" - - try: - descriptor = os.open(self.path, os.O_RDONLY | os.O_NOFOLLOW) - with os.fdopen(descriptor, "rb") as handle: - before = os.fstat(handle.fileno()) - signature = ( - before.st_dev, - before.st_ino, - before.st_size, - before.st_mtime_ns, - before.st_ctime_ns, - ) - if expected_signature is not None and signature != expected_signature: - raise DocForgeError( - "invalid_index", - "Derived index changed before hashing", - ) - index_hash = hashlib.file_digest(handle, "sha256").hexdigest() - after = os.fstat(handle.fileno()) - after_signature = ( - after.st_dev, - after.st_ino, - after.st_size, - after.st_mtime_ns, - after.st_ctime_ns, - ) - if after_signature != signature or index_signature(self.path) != signature: - raise DocForgeError( - "invalid_index", - "Derived index changed during hashing", - ) - except OSError as error: - raise DocForgeError("missing_index", "Derived index cannot be hashed") from error - return before.st_size, index_hash, signature - - def _write_attestation( - self, - *, - expected_signature: tuple[int, int, int, int, int] | None = None, - ) -> None: + def _write_attestation(self) -> None: """Atomically persist the digest of an index that passed complete verification.""" root = self.project.descriptor.cache_root path = self.attestation_path if path.parent != root or path.is_symlink(): raise DocForgeError("path_escape", "Index attestation path is not safe") - size, index_hash, before = self._index_digest( - expected_signature=expected_signature, - ) + try: + size = self.path.stat().st_size + with self.path.open("rb") as handle: + index_hash = hashlib.file_digest(handle, "sha256").hexdigest() + except OSError as error: + raise DocForgeError("missing_index", "Derived index cannot be attested") from error payload = { "schema_version": 1, "index_size": size, "index_sha256": index_hash, } raw = json.dumps(payload, sort_keys=True, indent=2).encode("utf-8") + b"\n" - root_fd = open_bound_directory(root) - temporary_name = f".index-attestation-{secrets.token_hex(12)}" - temporary_created = False + descriptor, temporary_name = tempfile.mkstemp(prefix=".index-attestation-", dir=root) + temporary = Path(temporary_name) try: - try: - existing = os.stat( - path.name, - dir_fd=root_fd, - follow_symlinks=False, - ) - except FileNotFoundError: - existing = None - if existing is not None and ( - stat.S_ISLNK(existing.st_mode) or not stat.S_ISREG(existing.st_mode) - ): - raise DocForgeError("path_escape", "Index attestation path is not safe") - descriptor = os.open( - temporary_name, - os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, - 0o600, - dir_fd=root_fd, - ) - temporary_created = True with os.fdopen(descriptor, "wb") as handle: handle.write(raw) handle.flush() os.fsync(handle.fileno()) - require_bound_directory(root, root_fd) - if self._index_signature() != before: - raise DocForgeError( - "invalid_index", - "Derived index changed before attestation publication", - ) - os.replace( - temporary_name, - path.name, - src_dir_fd=root_fd, - dst_dir_fd=root_fd, - ) - temporary_created = False - os.fsync(root_fd) - require_bound_directory(root, root_fd) + os.replace(temporary, path) + directory_descriptor = os.open(root, os.O_RDONLY) + try: + os.fsync(directory_descriptor) + finally: + os.close(directory_descriptor) except Exception: - if temporary_created: - with suppress(OSError): - os.unlink(temporary_name, dir_fd=root_fd) + temporary.unlink(missing_ok=True) raise - finally: - os.close(root_fd) def _index_signature(self) -> tuple[int, int, int, int, int]: - return index_signature(self.path) - - def task_context(self, plan: RetrievalPlanV1) -> dict[str, object]: - """Execute one fixed task plan inside one immutable index generation.""" - - validate_retrieval_plan(plan, self.project.descriptor) - with self._read_snapshot() as snapshot: - capsule = self._task_context_capsule(snapshot, plan) - return snapshot.result(capsule=capsule.as_dict()) - - def generation_diff(self) -> dict[str, object]: - """Return the latest bounded transition without loading or repairing source.""" - - descriptor = self.project.descriptor - identity = { - "status": "ok", - "project_id": descriptor.project_id, - "project_root_fingerprint": project_root_fingerprint(descriptor.root), - "adapter": descriptor.adapter, - } - if not isinstance(self.project, IncrementalStateProject): - return { - **identity, - "revision": "unknown", - "source_hash": None, - "receipt_state": "unknown", - "receipt_reason": "source_identity_unavailable", - "generation_diff": None, - "staleness": "unknown", - } - before_state = self.project.incremental_state() - if before_state is None: - return { - **identity, - "revision": "unknown", - "source_hash": None, - "receipt_state": "unknown", - "receipt_reason": "source_identity_unavailable", - "generation_diff": None, - "staleness": "unknown", - } - receipt, reason, receipt_signature = load_generation_diff(descriptor) - if receipt is None: - receipt_state = ( - "unsafe" - if reason == "unsafe_receipt" - else ("missing" if reason == "missing_receipt" else "unverified") - ) - return { - **identity, - "revision": before_state.revision, - "source_hash": before_state.source_hash, - "receipt_state": receipt_state, - "receipt_reason": reason, - "generation_diff": None, - "staleness": "unknown", - } try: - current_index_signature = index_signature(self.path) - except DocForgeError as error: - return { - **identity, - "revision": before_state.revision, - "source_hash": before_state.source_hash, - "receipt_state": "unsafe" if error.code == "path_escape" else "stale", - "receipt_reason": ( - "unsafe_index" if error.code == "path_escape" else "missing_index" - ), - "generation_diff": None, - "staleness": "stale", - } - target = cast(dict[str, object], receipt["to_generation"]) - expected_index_signature = cast(dict[str, object], receipt["index_signature"]) - observed_index_signature = { - "device": current_index_signature[0], - "inode": current_index_signature[1], - "size": current_index_signature[2], - "mtime_ns": current_index_signature[3], - "ctime_ns": current_index_signature[4], - } - if ( - target["revision"] != before_state.revision - or target["source_hash"] != before_state.source_hash - or expected_index_signature != observed_index_signature - ): - return { - **identity, - "revision": before_state.revision, - "source_hash": before_state.source_hash, - "receipt_state": "stale", - "receipt_reason": "generation_mismatch", - "generation_diff": None, - "staleness": "stale", - } - after_state = self.project.incremental_state() - try: - after_index_signature = index_signature(self.path) - after_receipt_signature = index_signature(generation_diff_path(descriptor)) - except DocForgeError: - after_state = None - after_index_signature = () - after_receipt_signature = () - if ( - after_state != before_state - or after_index_signature != current_index_signature - or after_receipt_signature != receipt_signature - ): - return { - **identity, - "revision": before_state.revision, - "source_hash": before_state.source_hash, - "receipt_state": "unverified", - "receipt_reason": "concurrent_change", - "generation_diff": None, - "staleness": "unknown", - } - return { - **identity, - "revision": before_state.revision, - "source_hash": before_state.source_hash, - "receipt_state": "current", - "receipt_reason": None, - "generation_diff": receipt, - "staleness": "current", - } - - def _task_context_capsule( - self, - snapshot: _IndexReadSnapshot, - plan: RetrievalPlanV1, - ) -> ContextCapsuleV1: - generation = { - "project_id": snapshot.checked["project_id"], - "project_root_fingerprint": snapshot.checked["project_root_fingerprint"], - "adapter": snapshot.checked["adapter"], - "revision": snapshot.checked["revision"], - "source_hash": snapshot.checked["source_hash"], - "index_schema_version": INDEX_SCHEMA_VERSION, - } - focus_rows: list[sqlite3.Row] - focus_reason = "exact_focus" - if plan.focus_node_id is not None: - row = snapshot.connection.execute( - "SELECT * FROM nodes WHERE node_id = ?", - (plan.focus_node_id,), - ).fetchone() - if row is None: - raise DocForgeError( - "missing_node", - "No node has the requested stable ID", - node_id=plan.focus_node_id, - ) - focus_rows = [row] - else: - terms = re_tokenize(plan.task_query) - if not terms: - raise DocForgeError( - "invalid_task_focus", - "Task description contains no searchable text", - ) - expression = " AND ".join(f'"{term.replace(chr(34), chr(34) * 2)}"' for term in terms) - focus_rows = snapshot.connection.execute( - """ - SELECT nodes.*, bm25(node_fts) AS rank - FROM node_fts JOIN nodes USING(node_id) - WHERE node_fts MATCH ? - ORDER BY rank, nodes.node_id - LIMIT ? - """, - (expression, plan.max_evidence + 1), - ).fetchall() - focus_reason = "lexical_focus" - if not focus_rows: - return finalize_capsule( - state="blocked", - task_kind=plan.task_kind, - generation=generation, - plan=plan, - focus_state="not_found", - focus_node_id=None, - focus_candidate_count=0, - evidence=(), - gaps=( - EvidenceGapV1( - code="focus_not_found", - requirement_id="focus", - category=None, - state="blocked", - check_complete=True, - detail="No indexed node matched the bounded lexical focus.", - ), - ), - omissions=(), - selected_count=0, - examined_edge_count=0, - estimated_tokens=0, - unclassified_relations=(), - ) - if len(focus_rows) > 1 and focus_rows[0]["rank"] == focus_rows[1]["rank"]: - return finalize_capsule( - state="blocked", - task_kind=plan.task_kind, - generation=generation, - plan=plan, - focus_state="ambiguous", - focus_node_id=None, - focus_candidate_count=len(focus_rows), - evidence=(), - gaps=( - EvidenceGapV1( - code="focus_ambiguous", - requirement_id="focus", - category=None, - state="blocked", - check_complete=True, - detail=( - "The highest-ranked lexical focus is tied; provide focus_node_id." - ), - ), - ), - omissions=(), - selected_count=0, - examined_edge_count=0, - estimated_tokens=0, - unclassified_relations=(), - ) - focus_rows = focus_rows[:1] - - focus_node_id = cast(str, focus_rows[0]["node_id"]) - selections: dict[str, _TaskSelection] = { - focus_node_id: _TaskSelection( - node_id=focus_node_id, - role="focus", - reason_code=focus_reason, - depth=0, - category_rank=-1, - first_edge=("", "", ""), - relationship_path=(), - ) - } - relationship_reasons: dict[str, set[CapsuleRelationshipV1]] = {focus_node_id: set()} - queue: deque[str] = deque((focus_node_id,)) - examined_edge_count = 0 - traversal_incomplete = False - edge_examination_limit_reached = False - first_omitted_node: str | None = None - unclassified_relations: set[str] = set() - while queue and not traversal_incomplete: - current = queue.popleft() - current_selection = selections[current] - if current_selection.depth >= plan.max_depth: - continue - remaining = plan.max_candidate_edges - examined_edge_count - if remaining <= 0: - traversal_incomplete = True - edge_examination_limit_reached = True - break - outgoing = snapshot.connection.execute( - "SELECT source_id, relation, target_id FROM edges " - "WHERE source_id = ? ORDER BY source_id, relation, target_id LIMIT ?", - (current, remaining + 1), - ).fetchall() - incoming = snapshot.connection.execute( - "SELECT source_id, relation, target_id FROM edges " - "WHERE target_id = ? ORDER BY source_id, relation, target_id LIMIT ?", - (current, remaining + 1), - ).fetchall() - candidates = { - ( - cast(str, row["source_id"]), - cast(str, row["relation"]), - cast(str, row["target_id"]), - "outgoing" if row["source_id"] == current else "incoming", - ) - for row in (*outgoing, *incoming) - } - ordered = sorted( - candidates, - key=lambda item: ( - plan.category_order.index(relation_category(item[1])), - item[0], - item[1], - item[2], - item[3], - ), - ) - if len(ordered) > remaining: - traversal_incomplete = True - edge_examination_limit_reached = True - ordered = ordered[:remaining] - for source_id, relation, target_id, direction in ordered: - examined_edge_count += 1 - category = relation_category(relation) - if category == "unclassified": - unclassified_relations.add(relation) - neighbor = target_id if source_id == current else source_id - relationship = CapsuleRelationshipV1( - source_id=source_id, - relation=relation, - target_id=target_id, - direction=cast(Literal["outgoing", "incoming"], direction), - category=category, - ) - evidence_reason = CapsuleRelationshipV1( - source_id=source_id, - relation=relation, - target_id=target_id, - direction="outgoing" if source_id == neighbor else "incoming", - category=category, - ) - if neighbor in selections: - relationship_reasons.setdefault(neighbor, set()).add(evidence_reason) - continue - if len(selections) >= plan.max_evidence: - traversal_incomplete = True - first_omitted_node = neighbor - break - selections[neighbor] = _TaskSelection( - node_id=neighbor, - role="related", - reason_code="relationship_path", - depth=current_selection.depth + 1, - category_rank=plan.category_order.index(category), - first_edge=(source_id, relation, target_id), - relationship_path=( - *current_selection.relationship_path, - relationship, - ), - ) - relationship_reasons[neighbor] = {evidence_reason} - queue.append(neighbor) - - ordered_selections = sorted( - selections.values(), - key=lambda selection: ( - 0 if selection.role == "focus" else 1, - selection.category_rank, - selection.depth, - selection.first_edge, - selection.node_id, - ), - ) - selected_node_ids = tuple(selection.node_id for selection in ordered_selections) - rows: list[sqlite3.Row] = [] - for position in range(0, len(selected_node_ids), _SQLITE_PARAMETER_CHUNK): - node_id_chunk = selected_node_ids[position : position + _SQLITE_PARAMETER_CHUNK] - placeholders = ",".join("?" for _ in node_id_chunk) - rows.extend( - snapshot.connection.execute( - f"SELECT * FROM nodes WHERE node_id IN ({placeholders}) ORDER BY node_id", - node_id_chunk, - ).fetchall() - ) - nodes = {cast(str, row["node_id"]): _row_to_node(row) for row in rows} - evidence: list[CapsuleEvidenceV1] = [] - omissions: list[CapsuleOmissionV1] = [] - used_tokens = 0 - returned_categories: set[str] = set() - for selection in ordered_selections: - node = nodes[selection.node_id] - item = capsule_evidence( - role=cast(Literal["focus", "related"], selection.role), - reason_code=cast( - Literal["exact_focus", "lexical_focus", "relationship_path"], - selection.reason_code, - ), - node=node, - depth=selection.depth, - relationship_path=selection.relationship_path, - relationship_reasons=tuple( - sorted( - relationship_reasons.get(selection.node_id, set()), - key=lambda relationship: ( - relationship.source_id, - relationship.relation, - relationship.target_id, - relationship.direction, - ), - ) - ), - ) - if used_tokens + item.estimated_tokens > plan.max_tokens: - omissions.append( - CapsuleOmissionV1( - code="token_budget", - subject=node.node_id, - detail_hash=canonical_hash( - { - "node_id": node.node_id, - "content_hash": node.content_hash, - "estimated_tokens": item.estimated_tokens, - } - ), - ) - ) - continue - evidence.append(item) - used_tokens += item.estimated_tokens - returned_categories.update( - relationship.category for relationship in item.relationship_reasons - ) - if first_omitted_node is not None: - omissions.append( - CapsuleOmissionV1( - code="result_limit", - subject=first_omitted_node, - detail_hash=canonical_hash( - { - "node_id": first_omitted_node, - "max_evidence": plan.max_evidence, - } - ), - ) - ) - if edge_examination_limit_reached: - omissions.append( - CapsuleOmissionV1( - code="edge_examination_limit", - subject=focus_node_id, - detail_hash=canonical_hash( - { - "focus_node_id": focus_node_id, - "examined_edge_count": examined_edge_count, - "max_candidate_edges": plan.max_candidate_edges, - } - ), - ) - ) - - ordered_unclassified_relations = tuple(sorted(unclassified_relations)) - bounded_unclassified_relations = ordered_unclassified_relations[: plan.max_evidence] - if len(ordered_unclassified_relations) > len(bounded_unclassified_relations): - omissions.append( - CapsuleOmissionV1( - code="unclassified_relation_limit", - subject=focus_node_id, - detail_hash=canonical_hash( - { - "unclassified_relation_count": len(ordered_unclassified_relations), - "returned_count": len(bounded_unclassified_relations), - } - ), - ) - ) - - declared_categories = { - relation_category(relation) for relation in self.project.descriptor.allowed_relations - } - gaps: list[EvidenceGapV1] = [] - incomplete = traversal_incomplete or bool(omissions) - for requirement in plan.requirements: - category = requirement.category - if category not in declared_categories: - gaps.append( - EvidenceGapV1( - code="category_not_declared", - requirement_id=requirement.requirement_id, - category=category, - state="missing", - check_complete=True, - detail=( - "No declared project relation maps to this required plan category." - ), - ) - ) - elif category in returned_categories: - continue - elif incomplete: - gaps.append( - EvidenceGapV1( - code="evidence_incomplete", - requirement_id=requirement.requirement_id, - category=category, - state="incomplete", - check_complete=False, - detail=( - "A bounded work, result, or token limit prevented a complete " - "returned-evidence proof." - ), - ) - ) - else: - gaps.append( - EvidenceGapV1( - code="no_selected_evidence", - requirement_id=requirement.requirement_id, - category=category, - state="missing", - check_complete=True, - detail=( - "The complete bounded check found no selected graph evidence in " - "this category." - ), - ) - ) - for relation in bounded_unclassified_relations: - gaps.append( - EvidenceGapV1( - code="unclassified_relation", - requirement_id=f"relation.{relation}", - category="unclassified", - state="limitation", - check_complete=True, - detail="The project relation is preserved without inferred task semantics.", - ) - ) - gaps.sort( - key=lambda gap: ( - gap.requirement_id, - gap.code, - "" if gap.category is None else gap.category, - ) - ) - return finalize_capsule( - state="incomplete" if incomplete else "complete", - task_kind=plan.task_kind, - generation=generation, - plan=plan, - focus_state="resolved", - focus_node_id=focus_node_id, - focus_candidate_count=len(focus_rows), - evidence=tuple(evidence), - gaps=tuple(gaps), - omissions=tuple(omissions), - selected_count=len(selections), - examined_edge_count=examined_edge_count, - estimated_tokens=used_tokens, - unclassified_relations=bounded_unclassified_relations, + status = self.path.stat() + except OSError as error: + raise DocForgeError( + "missing_index", + "Derived index does not exist; run build first", + ) from error + return ( + status.st_dev, + status.st_ino, + status.st_size, + status.st_mtime_ns, + status.st_ctime_ns, ) def get_node(self, node_id: str) -> dict[str, object]: diff --git a/src/docforge/mcp_server.py b/src/docforge/mcp_server.py index d90e7b2..e4c721a 100644 --- a/src/docforge/mcp_server.py +++ b/src/docforge/mcp_server.py @@ -18,10 +18,8 @@ from .errors import DocForgeError from .index import ProjectIndex from .models import IncrementalStateProject, ProjectService, RuntimeValidatedProject from .pagination import canonical_hash, decode_cursor, page_limit, page_receipt -from .policy import CapabilityMode, capability_mode, compose_effective_policy from .project import Project, project_root_fingerprint from .rendering import RenderService -from .retrieval import MAX_TASK_EVIDENCE, TaskKind, build_retrieval_plan from .telemetry import request, stage from .viewer_manager import ViewerManagerClient @@ -44,13 +42,11 @@ READ_TOOLS = ( "docforge_dependencies", "docforge_impact", "docforge_get_context", - "docforge_get_task_context", "docforge_validate_project", "docforge_render_status", "docforge_visualize", "docforge_stop_visualization", "docforge_visualization_status", - "docforge_get_generation_diff", ) PROPOSAL_TOOLS = ( "docforge_create_changeset", @@ -133,96 +129,50 @@ class DocForgeService: binding_metadata: Mapping[str, object] | None = None, no_ast: bool = False, diagnostics: bool = False, - capability_mode_name: str | None = None, ) -> None: self.project = project - default_mode: CapabilityMode = ( - "application" if canonical_applier is not None else "proposal" - ) - selected_mode = capability_mode(capability_mode_name, default=default_mode) - application_enabled = ( - canonical_applier_id is not None - and canonical_applier is not None - and selected_mode in {"application", "operator"} - ) - self.policy = compose_effective_policy( - selected_mode=selected_mode, - capability_source=("factory_default" if capability_mode_name is None else "explicit"), - no_ast=no_ast, - diagnostics=diagnostics, - render_configured=project.descriptor.render is not None, - application_enabled=application_enabled, - ) - self.index = ProjectIndex(self.project, allow_logic=not self.policy.no_ast) - self.changesets = ChangesetStore( - self.project, - proposal_writer if selected_mode != "read" else None, - ) + self.index = ProjectIndex(self.project, allow_logic=not no_ast) + self.changesets = ChangesetStore(self.project, proposal_writer) self.rendering = RenderService(self.project, self.changesets) self.application = CanonicalApplicationService( self.project, - applier_id=canonical_applier_id if application_enabled else None, - applier=canonical_applier if application_enabled else None, + applier_id=canonical_applier_id, + applier=canonical_applier, index=self.index, ) self.visualization = ViewerManagerClient(self.index) self.context_provider = context_provider - self.task_context_available = context_provider is compile_context self.binding_metadata = dict(binding_metadata or {}) - self.no_ast = self.policy.no_ast + self.no_ast = no_ast self.diagnostics = diagnostics - default_surface = ( - READ_TOOLS - if selected_mode == "read" - else ( - *ALL_TOOLS, - *( - APPLICATION_TOOLS - if self.application.enabled and selected_mode in {"application", "operator"} - else () - ), - ) + self.tool_surface = tool_surface or ( + *ALL_TOOLS, + *(APPLICATION_TOOLS if self.application.enabled else ()), ) - self.tool_surface = tool_surface or default_surface def adapter_policy(self) -> dict[str, object]: """Return the immutable adapter-evolution policy for this MCP binding.""" - return self.policy.adapter_policy() - - def capabilities(self) -> dict[str, object]: - """Return the registered surfaces separately from startup-bound authority.""" - - proposal_access = self.changesets.access() - application_access = self.application.access() + if not self.no_ast: + return { + "mode": "standard", + "ast_analysis": "allowed", + "logic_projection": "allowed", + "incremental_extraction": "allowed", + "adapter_rewrite": "not_requested", + } return { - "schema_version": 1, - "mode": self.policy.capability_mode, - "registered_tools": list(self.tool_surface), - "read": { - "enabled": True, - "tools": [tool for tool in READ_TOOLS if tool in self.tool_surface], - }, - "task_context": { - "enabled": self.task_context_available, - "reason": ( - None - if self.task_context_available - else "custom_context_policy_not_supported_by_task_context_v1" - ), - }, - "proposal": { - "surface_enabled": any(tool in self.tool_surface for tool in PROPOSAL_TOOLS), - "mutation_access": proposal_access, - }, - "application": { - "surface_enabled": any(tool in self.tool_surface for tool in APPLICATION_TOOLS), - "mutation_access": application_access, - }, - "operator": { - "enabled": self.policy.capability_mode == "operator", - "tools": [], - }, + "mode": "preserve-no-ast", + "ast_analysis": "forbidden", + "logic_projection": "forbidden", + "incremental_extraction": "allowed", + "adapter_rewrite": "forbidden", + "blocked_tools": ["docforge_get_logic"], + "instruction": ( + "Preserve the existing adapter extraction strategy. Do not add Python AST, " + "Tree-sitter, compiler-AST, or function-Logic extraction. Non-AST incremental " + "fingerprinting and caching remain allowed." + ), } def invoke( @@ -557,39 +507,27 @@ class DocForgeService: def bootstrap(self) -> dict[str, object]: def operation() -> dict[str, object]: synchronized = self.index.synchronize() - descriptor = self.project.descriptor - root = descriptor.root + snapshot = self.project.load() + root = snapshot.descriptor.root binding = { "project_root": str(root), - "descriptor_path": str(descriptor.descriptor_path), - "adapter": descriptor.adapter, - "cache_root": str(descriptor.cache_root), - "index_path": str(descriptor.index_path), - "changeset_root": str(descriptor.changeset_root), + "descriptor_path": str(snapshot.descriptor.descriptor_path), + "adapter": snapshot.descriptor.adapter, + "cache_root": str(snapshot.descriptor.cache_root), + "index_path": str(snapshot.descriptor.index_path), + "changeset_root": str(snapshot.descriptor.changeset_root), **self.binding_metadata, "adapter_policy": self.adapter_policy(), } recommended_workflow = [ - ( - "docforge_get_task_context, docforge_get_context, or targeted read tools" - if self.task_context_available - else "docforge_get_context or targeted read tools" - ), + "docforge_get_context or targeted read tools", "make and verify one coherent implementation slice", "docforge_sync", + "docforge_register_changes", + "docforge_get_changeset_diff", + "docforge_apply_changeset", + "docforge_bootstrap", ] - proposal_access = self.changesets.access() - application_access = self.application.access() - if proposal_access["enabled"] and "docforge_register_changes" in self.tool_surface: - recommended_workflow.extend( - ( - "docforge_register_changes", - "docforge_get_changeset_diff", - ) - ) - if application_access["enabled"] and "docforge_apply_changeset" in self.tool_surface: - recommended_workflow.append("docforge_apply_changeset") - recommended_workflow.append("docforge_bootstrap") if self.no_ast: recommended_workflow.insert( 1, @@ -598,64 +536,19 @@ class DocForgeService: "compiler-AST, or function-Logic extraction" ), ) - if self.task_context_available: - recommended_first_operation: dict[str, object] = { - "tool": "docforge_get_task_context", - "arguments": { - "task_kind": "implementation", - "task": "", - }, - "reason": "Begin with one bounded task-shaped context capsule.", - } - elif descriptor.profiles: - recommended_first_operation = { - "tool": "docforge_get_context", - "arguments": {"profile": descriptor.profiles[0].profile_id}, - "reason": "Begin with one configured bounded context profile.", - } - else: - recommended_first_operation = { - "tool": "docforge_project_info", - "arguments": dict[str, object](), - "reason": "Confirm the fixed binding before targeted retrieval.", - } - capabilities = self.capabilities() - effective_policy = self.policy.as_dict() - session_contract: dict[str, object] = { - "schema_version": 1, - "binding": binding, - "generation": { - "revision": synchronized["revision"], - "source_hash": synchronized["source_hash"], - "freshness": "current", - }, - "effective_policy": effective_policy, - "capabilities": capabilities, - "render_policies": { - "manual": effective_policy["manual_render"], - "graph": effective_policy["graph_render"], - "live_viewer": effective_policy["live_viewer"], - }, - "recommended_first_operation": recommended_first_operation, - "recommended_workflow": recommended_workflow, - "prohibitions": effective_policy["prohibitions"], - } return { "status": "ok", - "project_id": descriptor.project_id, + "project_id": snapshot.descriptor.project_id, "project_root_fingerprint": project_root_fingerprint(root), - "title": descriptor.title, - "adapter": descriptor.adapter, - "revision": synchronized["revision"], - "source_hash": synchronized["source_hash"], + "title": snapshot.descriptor.title, + "adapter": snapshot.descriptor.adapter, + "revision": snapshot.revision, + "source_hash": snapshot.source_hash, "binding": binding, - "canonical_paths": [str(path) for path in descriptor.content_roots], + "canonical_paths": [str(path) for path in snapshot.descriptor.content_roots], "adapter_policy": self.adapter_policy(), - "effective_policy": effective_policy, - "capabilities": capabilities, - "session_contract": session_contract, - "proposal_access": proposal_access, - "canonical_application_access": application_access, + "proposal_access": self.changesets.access(), + "canonical_application_access": self.application.access(), "synchronization": synchronized["synchronization"], "recommended_workflow": recommended_workflow, } @@ -713,8 +606,6 @@ class DocForgeService: "Canonical project files own facts; DocForge results are derived." ), "adapter_policy": self.adapter_policy(), - "effective_policy": self.policy.as_dict(), - "capabilities": self.capabilities(), "canonical_paths": [ *(relative(path) for path in snapshot.descriptor.content_roots), *(relative(path) for path in snapshot.descriptor.authority_files), @@ -747,20 +638,14 @@ class DocForgeService: ], "allowed_tools": list(self.tool_surface), "excluded_operations": list( - dict.fromkeys( - EXCLUDED_OPERATIONS - + ( - ("canonical_writes", "canonical_changeset_application") - if not self.application.enabled - else () - ) - + ( - READ_ONLY_EXCLUDED_OPERATIONS - if self.policy.capability_mode == "read" - else () - ) - + tuple(self.policy.prohibitions) + EXCLUDED_OPERATIONS + + ( + ("canonical_writes", "canonical_changeset_application") + if not self.application.enabled + else () ) + + (READ_ONLY_EXCLUDED_OPERATIONS if self.tool_surface == READ_TOOLS else ()) + + (("adapter_ast_upgrade", "function_logic_extraction") if self.no_ast else ()) ), "proposal_access": self.changesets.access(), "canonical_application_access": self.application.access(), @@ -856,370 +741,6 @@ class DocForgeService: operation_name="mcp.context", ) - def task_context( - self, - task_kind: TaskKind, - task: str, - *, - focus_node_id: str | None = None, - budget: int | None = None, - limit: int | None = None, - cursor: str | None = None, - ) -> dict[str, Any]: - """Return one task-shaped capsule from a single immutable graph generation.""" - - if not self.task_context_available: - - def unavailable() -> dict[str, object]: - raise DocForgeError( - "task_context_unavailable", - ( - "This binding uses a custom context provider; " - "core task planning is unavailable" - ), - ) - - return self.invoke( - unavailable, - synchronize=False, - load_error_identity=False, - operation_name="mcp.task_context", - ) - - def operation() -> dict[str, object]: - maximum_evidence = min( - self.project.descriptor.limits.max_results, - MAX_TASK_EVIDENCE, - ) - selected_limit = page_limit( - limit, - default=min(20, maximum_evidence), - maximum=maximum_evidence, - ) - plan = build_retrieval_plan( - self.project.descriptor, - task_kind=task_kind, - task=task, - focus_node_id=focus_node_id, - budget=budget, - limit=maximum_evidence, - effective_policy=self.policy.as_dict(), - ) - result = self.index.task_context(plan) - return self._page_task_context_result( - result, - selected_limit=selected_limit, - cursor=cursor, - ) - - return self.invoke(operation, operation_name="mcp.task_context") - - def generation_diff( - self, - *, - limit: int | None = None, - cursor: str | None = None, - ) -> dict[str, Any]: - """Return one bounded page from the latest verified graph transition.""" - - def operation() -> dict[str, object]: - maximum_items = min( - self.project.descriptor.limits.max_results, - 1_000, - ) - selected_limit = page_limit( - limit, - default=min(20, maximum_items), - maximum=maximum_items, - ) - result = self.index.generation_diff() - receipt_value = result.get("generation_diff") - if receipt_value is None: - if cursor is not None: - decode_cursor( - cursor, - kind="generation-diff.items", - binding={ - "project_id": result.get("project_id"), - "receipt_state": result.get("receipt_state"), - "effective_policy_hash": canonical_hash(self.policy.as_dict()), - }, - total_count=0, - ) - return result - if not isinstance(receipt_value, Mapping): - raise DocForgeError( - "invalid_generation_diff", - "Generation diff receipt is malformed", - ) - receipt = dict(cast(Mapping[str, object], receipt_value)) - items_value = receipt.pop("items", None) - if not isinstance(items_value, list): - raise DocForgeError( - "invalid_generation_diff", - "Generation diff receipt has no bounded item collection", - ) - items = cast(list[object], items_value) - stored_receipt_hash = receipt.pop("receipt_hash", None) - if not isinstance(stored_receipt_hash, str): - raise DocForgeError( - "invalid_generation_diff", - "Generation diff receipt has no stable identity", - ) - receipt_header = { - **receipt, - "stored_receipt_hash": stored_receipt_hash, - } - binding = { - "stored_receipt_hash": stored_receipt_hash, - "effective_policy_hash": canonical_hash(self.policy.as_dict()), - } - position = decode_cursor( - cursor, - kind="generation-diff.items", - binding=binding, - total_count=len(items), - ) - page_items: list[object] = [] - page_omissions: list[dict[str, object]] = [] - consumed = 0 - maximum_chars = self.project.descriptor.limits.max_tool_output_chars - - def page_result() -> dict[str, object]: - pagination = page_receipt( - kind="generation-diff.items", - binding=binding, - position=position, - count=consumed, - limit=selected_limit, - total_count=len(items), - ) - page_hash = canonical_hash( - { - "page_schema_version": 1, - "receipt_state": result["receipt_state"], - "receipt_header": receipt_header, - "pagination": pagination, - "items": page_items, - "omissions": page_omissions, - } - ) - return { - **result, - "generation_diff": { - "page_schema_version": 1, - "receipt_header": receipt_header, - "items": page_items, - "omissions": page_omissions, - "page_hash": page_hash, - }, - "pagination": pagination, - } - - candidates = items[position : position + selected_limit] - - def fits(candidate_count: int) -> bool: - nonlocal consumed - page_items[:] = candidates[:candidate_count] - consumed = candidate_count - decorated = { - **page_result(), - "server_version": SERVER_VERSION, - "content_warning": CONTENT_WARNING, - } - return self._encoded_length(decorated) <= maximum_chars - - lower = 0 - upper = len(candidates) - while lower < upper: - midpoint = (lower + upper + 1) // 2 - if fits(midpoint): - lower = midpoint - else: - upper = midpoint - 1 - fits(lower) - if lower == 0 and candidates: - item = candidates[0] - item_payload: Mapping[str, object] = ( - cast(Mapping[str, object], item) if isinstance(item, Mapping) else {} - ) - page_omissions.append( - { - "code": "response_limit", - "item_hash": item_payload.get("item_hash"), - } - ) - consumed = 1 - return page_result() - - return self.invoke( - operation, - synchronize=False, - load_error_identity=False, - operation_name="mcp.generation_diff", - ) - - def _page_task_context_result( - self, - result: dict[str, object], - *, - selected_limit: int, - cursor: str | None, - ) -> dict[str, object]: - capsule_value = result.get("capsule") - if not isinstance(capsule_value, Mapping): - raise DocForgeError( - "invalid_task_context_result", - "Task context did not return a versioned capsule", - ) - capsule = dict(cast(Mapping[str, object], capsule_value)) - plan_value = capsule.get("plan") - generation_value = capsule.get("generation") - evidence_value = capsule.get("evidence") - omissions_value = capsule.get("omissions") - gaps_value = capsule.get("gaps") - if ( - capsule.get("schema_version") != 1 - or not isinstance(plan_value, Mapping) - or not isinstance(generation_value, Mapping) - or not isinstance(evidence_value, list) - or not isinstance(omissions_value, list) - or not isinstance(gaps_value, list) - or not isinstance(capsule.get("collection_hash"), str) - or not isinstance(capsule.get("capsule_hash"), str) - ): - raise DocForgeError( - "invalid_task_context_result", - "Task context capsule is malformed", - ) - plan_payload = cast(Mapping[str, object], plan_value) - generation = cast(Mapping[str, object], generation_value) - evidence = cast(list[object], evidence_value) - omissions = cast(list[object], omissions_value) - gaps = cast(list[object], gaps_value) - binding = { - "project_id": result.get("project_id"), - "project_root_fingerprint": result.get("project_root_fingerprint"), - "adapter": result.get("adapter"), - "revision": result.get("revision"), - "source_hash": result.get("source_hash"), - "index_schema_version": generation.get("index_schema_version"), - "effective_policy_hash": plan_payload.get("effective_policy_hash"), - "request_hash": plan_payload.get("request_hash"), - "plan_hash": plan_payload.get("plan_hash"), - "collection_hash": capsule["collection_hash"], - "capsule_hash": capsule["capsule_hash"], - } - items = [ - *(("evidence", item) for item in evidence), - *(("omission", item) for item in omissions), - ] - position = decode_cursor( - cursor, - kind="task-context.items", - binding=binding, - total_count=len(items), - ) - page_evidence: list[object] = [] - page_omissions: list[object] = [] - consumed = 0 - response_limited = False - maximum = self.project.descriptor.limits.max_tool_output_chars - - def page_result() -> dict[str, object]: - pagination = page_receipt( - kind="task-context.items", - binding=binding, - position=position, - count=consumed, - limit=selected_limit, - total_count=len(items), - ) - page_state = "incomplete" if response_limited else capsule.get("state") - page_summary = { - **cast(dict[str, object], capsule.get("summary", {})), - "page_evidence_count": len(page_evidence), - "page_omission_count": len(page_omissions), - "page_item_count": consumed, - } - page_hash = canonical_hash( - { - "capsule_hash": capsule["capsule_hash"], - "position": position, - "page_state": page_state, - "pagination": pagination, - "summary": page_summary, - "evidence": page_evidence, - "gaps": gaps, - "omissions": page_omissions, - } - ) - page_capsule = { - **capsule, - "evidence": page_evidence, - "omissions": page_omissions, - "page_state": page_state, - "page_hash": page_hash, - "pagination": pagination, - "summary": page_summary, - } - return { - **result, - "capsule": page_capsule, - "next_cursor": pagination["next_cursor"], - "pagination": pagination, - } - - candidates = items[position : position + selected_limit] - - def populate(candidate_count: int) -> None: - nonlocal consumed, response_limited - page_evidence.clear() - page_omissions.clear() - for kind, item in candidates[:candidate_count]: - destination = page_evidence if kind == "evidence" else page_omissions - destination.append(item) - consumed = candidate_count - response_limited = candidate_count < len(candidates) - - def fits(candidate_count: int) -> bool: - populate(candidate_count) - decorated = { - **page_result(), - "server_version": SERVER_VERSION, - "content_warning": CONTENT_WARNING, - "staleness": "current", - } - return self._encoded_length(decorated) <= maximum - - lower = 0 - upper = len(candidates) - while lower < upper: - midpoint = (lower + upper + 1) // 2 - if fits(midpoint): - lower = midpoint - else: - upper = midpoint - 1 - populate(lower) - if lower == 0 and candidates: - _, item = candidates[0] - subject = "unknown" - if isinstance(item, Mapping): - item_payload = cast(Mapping[str, object], item) - candidate = item_payload.get("node_id") or item_payload.get("subject") - if isinstance(candidate, str) and candidate: - subject = candidate[:256] - page_omissions.append( - { - "code": "response_limit", - "subject": subject, - "detail_hash": canonical_hash(cast(object, item)), - } - ) - consumed = 1 - response_limited = True - return page_result() - def _page_context_result( self, result: dict[str, object], @@ -1386,20 +907,19 @@ class DocForgeService: def _create_bound_server(service: DocForgeService, *, read_only: bool) -> FastMCP: - capability = { - "read": "Read validated documentation for exactly one configured project.", - "proposal": ( - "Read validated documentation and use startup-gated isolated proposal changesets and " - "previews for exactly one configured project." - ), - "application": ( - "Read validated documentation, use startup-gated isolated proposals, and apply one " - "exact validated changeset hash for exactly one configured project." - ), - "operator": ( - "Operate the fixed validated documentation binding for exactly one configured project." - ), - }[service.policy.capability_mode] + capability = ( + "Read validated documentation for exactly one configured project." + if read_only + else ( + "Read validated documentation and write isolated proposal changesets and previews for " + "exactly one configured project" + + ( + ", with hash-bound canonical application enabled." + if service.application.enabled + else "." + ) + ) + ) server = FastMCP( "DocForge", instructions=( @@ -1540,26 +1060,6 @@ def _create_bound_server(service: DocForgeService, *, read_only: bool) -> FastMC return service.context(profile, budget, limit=limit, cursor=cursor) - @server.tool(name="docforge_get_task_context") - def get_task_context( - task_kind: TaskKind, - task: str, - focus_node_id: str | None = None, - budget: int | None = None, - limit: int | None = None, - cursor: str | None = None, - ) -> dict[str, Any]: - """Return one bounded task-shaped context capsule with explicit evidence gaps.""" - - return service.task_context( - task_kind, - task, - focus_node_id=focus_node_id, - budget=budget, - limit=limit, - cursor=cursor, - ) - @server.tool(name="docforge_validate_project") def validate_project() -> dict[str, Any]: """Validate current canonical sources and graph without writing any project file.""" @@ -1597,15 +1097,6 @@ def _create_bound_server(service: DocForgeService, *, read_only: bool) -> FastMC return service.visualization_status() - @server.tool(name="docforge_get_generation_diff") - def get_generation_diff( - limit: int | None = None, - cursor: str | None = None, - ) -> dict[str, Any]: - """Return the latest bounded primary-graph generation transition.""" - - return service.generation_diff(limit=limit, cursor=cursor) - _registered_read_tools = ( bootstrap, synchronize, @@ -1619,13 +1110,11 @@ def _create_bound_server(service: DocForgeService, *, read_only: bool) -> FastMC dependencies, impact, get_context, - get_task_context, validate_project, render_status, visualize, - stop_visualization, visualization_status, - get_generation_diff, + stop_visualization, ) if read_only: return server @@ -2003,7 +1492,6 @@ def create_server( canonical_applier_id: str | None = None, no_ast: bool = False, diagnostics: bool = False, - capability_mode: str | None = None, ) -> FastMCP: project = Project.open(project_root) return create_project_server( @@ -2019,7 +1507,6 @@ def create_server( }, no_ast=no_ast, diagnostics=diagnostics, - capability_mode=capability_mode, ) @@ -2033,7 +1520,6 @@ def create_project_server( binding_metadata: Mapping[str, object] | None = None, no_ast: bool = False, diagnostics: bool = False, - capability_mode: str | None = None, ) -> FastMCP: """Create the full fixed MCP surface for one explicitly configured project service.""" @@ -2046,12 +1532,8 @@ def create_project_server( binding_metadata=binding_metadata, no_ast=no_ast, diagnostics=diagnostics, - capability_mode_name=capability_mode, - ) - return _create_bound_server( - service, - read_only=service.policy.capability_mode == "read", ) + return _create_bound_server(service, read_only=False) def create_read_only_server( @@ -2061,7 +1543,6 @@ def create_read_only_server( binding_metadata: Mapping[str, object] | None = None, no_ast: bool = False, diagnostics: bool = False, - capability_mode: str | None = None, ) -> FastMCP: """Create an adapter-capable MCP server exposing only the fixed read tool surface.""" @@ -2072,14 +1553,7 @@ def create_read_only_server( binding_metadata=binding_metadata, no_ast=no_ast, diagnostics=diagnostics, - capability_mode_name="read" if capability_mode is None else capability_mode, ) - if service.policy.capability_mode != "read": - raise DocForgeError( - "invalid_capability_mode", - "Read-only server factory accepts only read capability mode", - capability_mode=service.policy.capability_mode, - ) return _create_bound_server(service, read_only=True) @@ -2101,11 +1575,6 @@ def main() -> None: action="store_true", help="Attach bounded request-local stage timings and counters", ) - parser.add_argument( - "--capability-mode", - choices=("read", "proposal", "application", "operator"), - help="Expose the versioned project-bound capability surface", - ) arguments = parser.parse_args() create_server( arguments.project_root, @@ -2113,7 +1582,6 @@ def main() -> None: canonical_applier_id=arguments.canonical_applier, no_ast=arguments.no_ast, diagnostics=arguments.diagnostics, - capability_mode=arguments.capability_mode, ).run(transport="stdio") diff --git a/src/docforge/pagination.py b/src/docforge/pagination.py index 12e4a68..a90916c 100644 --- a/src/docforge/pagination.py +++ b/src/docforge/pagination.py @@ -103,6 +103,7 @@ def decode_cursor( or not isinstance(stored_binding, dict) or type(position) is not int or position < 0 + or position >= total_count or not isinstance(checksum, str) or len(checksum) != 64 ): @@ -116,8 +117,6 @@ def decode_cursor( "stale_cursor", "Pagination cursor does not match the current result generation", ) - if position >= total_count: - raise _invalid_cursor() return position diff --git a/src/docforge/policy.py b/src/docforge/policy.py deleted file mode 100644 index c8b0009..0000000 --- a/src/docforge/policy.py +++ /dev/null @@ -1,166 +0,0 @@ -"""Versioned immutable policy composition for one project-bound server.""" - -from __future__ import annotations - -from dataclasses import dataclass -from typing import Literal - -from .errors import DocForgeError - -CapabilityMode = Literal["read", "proposal", "application", "operator"] -CAPABILITY_MODES: tuple[CapabilityMode, ...] = ( - "read", - "proposal", - "application", - "operator", -) - -POLICY_PRECEDENCE = ( - "core_safety", - "explicit_binding", - "no_ast_shorthand", - "resource_availability", -) - - -def capability_mode(value: str | None, *, default: CapabilityMode) -> CapabilityMode: - """Validate one additive capability-mode selection.""" - - selected = default if value is None else value - if selected not in CAPABILITY_MODES: - raise DocForgeError( - "invalid_capability_mode", - "Capability mode is unsupported", - capability_mode=selected, - allowed=list(CAPABILITY_MODES), - ) - return selected # type: ignore[return-value] - - -@dataclass(frozen=True) -class EffectivePolicyV1: - """One fully composed process policy shared by every public projection.""" - - capability_mode: CapabilityMode - capability_source: Literal["factory_default", "explicit"] - adapter_evolution: Literal["allowed", "preserve"] - ast_analysis: Literal["allowed", "forbidden"] - logic_indexing: Literal["full", "off"] - synchronization: Literal["automatic"] - integrity: Literal["validated"] - manual_render: Literal["auto", "explicit", "disabled"] - graph_render: Literal["disabled"] - live_viewer: Literal["on-demand"] - profiling: Literal["enabled", "disabled"] - blocked_tools: tuple[str, ...] - prohibitions: tuple[str, ...] - - @property - def no_ast(self) -> bool: - return self.ast_analysis == "forbidden" - - def as_dict(self) -> dict[str, object]: - return { - "schema_version": 1, - "capability_mode": self.capability_mode, - "capability_source": self.capability_source, - "adapter_evolution": self.adapter_evolution, - "ast_analysis": self.ast_analysis, - "logic_indexing": self.logic_indexing, - "synchronization": self.synchronization, - "integrity": self.integrity, - "manual_render": self.manual_render, - "graph_render": self.graph_render, - "live_viewer": self.live_viewer, - "profiling": self.profiling, - "blocked_tools": list(self.blocked_tools), - "prohibitions": list(self.prohibitions), - "precedence": list(POLICY_PRECEDENCE), - } - - def adapter_policy(self) -> dict[str, object]: - """Preserve the exact legacy adapter-policy projection.""" - - if not self.no_ast: - return { - "mode": "standard", - "ast_analysis": "allowed", - "logic_projection": "allowed", - "incremental_extraction": "allowed", - "adapter_rewrite": "not_requested", - } - return { - "mode": "preserve-no-ast", - "ast_analysis": "forbidden", - "logic_projection": "forbidden", - "incremental_extraction": "allowed", - "adapter_rewrite": "forbidden", - "blocked_tools": ["docforge_get_logic"], - "instruction": ( - "Preserve the existing adapter extraction strategy. Do not add Python AST, " - "Tree-sitter, compiler-AST, or function-Logic extraction. Non-AST incremental " - "fingerprinting and caching remain allowed." - ), - } - - -def compose_effective_policy( - *, - selected_mode: CapabilityMode, - capability_source: Literal["factory_default", "explicit"], - no_ast: bool, - diagnostics: bool, - render_configured: bool, - application_enabled: bool, -) -> EffectivePolicyV1: - """Compose fixed defaults with restrictive compatibility shorthands.""" - - if selected_mode == "application" and not application_enabled: - raise DocForgeError( - "capability_unavailable", - "Application capability requires a startup-bound canonical applier", - capability_mode=selected_mode, - required="canonical_applier", - ) - prohibitions = [ - "arbitrary_file_access", - "arbitrary_renderer_execution", - "shell_execution", - "git_mutation", - "deployment", - "publication", - "project_switching", - ] - blocked_tools: tuple[str, ...] = () - if no_ast: - prohibitions.extend( - ( - "adapter_ast_upgrade", - "tree_sitter_upgrade", - "compiler_ast_upgrade", - "function_logic_extraction", - ) - ) - blocked_tools = ("docforge_get_logic",) - manual_render: Literal["auto", "explicit", "disabled"] - if not render_configured: - manual_render = "disabled" - elif application_enabled and selected_mode in {"application", "operator"}: - manual_render = "auto" - else: - manual_render = "explicit" - return EffectivePolicyV1( - capability_mode=selected_mode, - capability_source=capability_source, - adapter_evolution="preserve" if no_ast else "allowed", - ast_analysis="forbidden" if no_ast else "allowed", - logic_indexing="off" if no_ast else "full", - synchronization="automatic", - integrity="validated", - manual_render=manual_render, - graph_render="disabled", - live_viewer="on-demand", - profiling="enabled" if diagnostics else "disabled", - blocked_tools=blocked_tools, - prohibitions=tuple(prohibitions), - ) diff --git a/src/docforge/project.py b/src/docforge/project.py index 3d689bd..29995cc 100644 --- a/src/docforge/project.py +++ b/src/docforge/project.py @@ -11,7 +11,6 @@ import tempfile import tomllib from collections import Counter from collections.abc import Mapping -from contextlib import suppress from dataclasses import dataclass, replace from pathlib import Path, PurePosixPath from typing import Any, cast @@ -40,7 +39,6 @@ from .telemetry import increment, stage SOURCE_GENERATION_SCHEMA_VERSION = 1 GENERIC_SOURCE_CONTRACT = "docforge-core:0.7.1:index:1" -MAX_PROJECT_DESCRIPTOR_BYTES = 1_000_000 _CORE_METADATA = frozenset( { @@ -212,167 +210,12 @@ def _receipt_signature(path: Path) -> tuple[int, int, int, int, int] | None: ) -def _read_descriptor(descriptor_path: Path) -> bytes: - try: - parent = descriptor_path.parent - parent_status = parent.lstat() - if ( - stat.S_ISLNK(parent_status.st_mode) - or not stat.S_ISDIR(parent_status.st_mode) - or parent.resolve(strict=True) != parent - ): - raise DocForgeError( - "project_descriptor_unsafe", - "Project descriptor parent must be one real confined directory", - ) - directory_fd = os.open( - parent, - os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, - ) - except FileNotFoundError as error: - raise DocForgeError("missing_config", "Missing .docforge/project.toml") from error - except DocForgeError: - raise - except OSError as error: - raise DocForgeError( - "project_descriptor_unsafe", - "Project descriptor cannot be inspected safely", - ) from error - opened_parent = os.fstat(directory_fd) - if opened_parent.st_dev != parent_status.st_dev or opened_parent.st_ino != parent_status.st_ino: - with suppress(OSError): - os.close(directory_fd) - raise DocForgeError( - "project_descriptor_changed", - "Project descriptor parent changed while it was opened", - ) - try: - - def parent_current() -> bool: - try: - before = parent.lstat() - resolved = parent.resolve(strict=True) - after = parent.lstat() - opened = os.fstat(directory_fd) - return ( - not stat.S_ISLNK(before.st_mode) - and stat.S_ISDIR(before.st_mode) - and resolved == parent - and (before.st_dev, before.st_ino, before.st_mode) - == (after.st_dev, after.st_ino, after.st_mode) - == (opened.st_dev, opened.st_ino, opened.st_mode) - ) - except OSError: - return False - - if not parent_current(): - raise DocForgeError( - "project_descriptor_changed", - "Project descriptor parent changed before it was read", - ) - try: - before = os.stat( - descriptor_path.name, - dir_fd=directory_fd, - follow_symlinks=False, - ) - except FileNotFoundError as error: - raise DocForgeError("missing_config", "Missing .docforge/project.toml") from error - except OSError as error: - raise DocForgeError( - "project_descriptor_unsafe", - "Project descriptor cannot be inspected safely", - ) from error - if stat.S_ISLNK(before.st_mode) or not stat.S_ISREG(before.st_mode): - raise DocForgeError( - "project_descriptor_unsafe", - "Project descriptor must be a regular file and not a symbolic link", - ) - if before.st_size > MAX_PROJECT_DESCRIPTOR_BYTES: - raise DocForgeError( - "project_descriptor_oversized", - "Project descriptor exceeds the bounded configuration limit", - maximum_bytes=MAX_PROJECT_DESCRIPTOR_BYTES, - ) - try: - descriptor = os.open( - descriptor_path.name, - os.O_RDONLY | os.O_NOFOLLOW, - dir_fd=directory_fd, - ) - except OSError as error: - raise DocForgeError( - "project_descriptor_unsafe", - "Project descriptor cannot be opened safely", - ) from error - try: - opened = os.fstat(descriptor) - if opened.st_dev != before.st_dev or opened.st_ino != before.st_ino: - raise DocForgeError( - "project_descriptor_changed", - "Project descriptor changed while it was opened", - ) - chunks: list[bytes] = [] - remaining = MAX_PROJECT_DESCRIPTOR_BYTES + 1 - while remaining: - chunk = os.read(descriptor, min(65_536, remaining)) - if not chunk: - break - chunks.append(chunk) - remaining -= len(chunk) - raw = b"".join(chunks) - finally: - with suppress(OSError): - os.close(descriptor) - if len(raw) > MAX_PROJECT_DESCRIPTOR_BYTES: - raise DocForgeError( - "project_descriptor_oversized", - "Project descriptor exceeds the bounded configuration limit", - maximum_bytes=MAX_PROJECT_DESCRIPTOR_BYTES, - ) - try: - after = os.stat( - descriptor_path.name, - dir_fd=directory_fd, - follow_symlinks=False, - ) - except OSError as error: - raise DocForgeError( - "project_descriptor_changed", - "Project descriptor changed while it was read", - ) from error - if ( - before.st_dev, - before.st_ino, - before.st_size, - before.st_mtime_ns, - before.st_ctime_ns, - ) != ( - after.st_dev, - after.st_ino, - after.st_size, - after.st_mtime_ns, - after.st_ctime_ns, - ): - raise DocForgeError( - "project_descriptor_changed", - "Project descriptor changed while it was read", - ) - if not parent_current(): - raise DocForgeError( - "project_descriptor_changed", - "Project descriptor parent changed while it was read", - ) - return raw - finally: - with suppress(OSError): - os.close(directory_fd) - - def _load_descriptor(root: Path) -> ProjectDescriptor: descriptor_path = root / ".docforge" / "project.toml" - descriptor_bytes = _read_descriptor(descriptor_path) + if not descriptor_path.is_file(): + raise DocForgeError("missing_config", "Missing .docforge/project.toml") try: + descriptor_bytes = descriptor_path.read_bytes() document = cast(dict[str, object], tomllib.loads(descriptor_bytes.decode("utf-8"))) except UnicodeDecodeError as error: raise DocForgeError("invalid_config", "Project descriptor is not UTF-8") from error @@ -629,17 +472,6 @@ def _load_descriptor(root: Path) -> ProjectDescriptor: ) -def validate_descriptor_binding(descriptor: ProjectDescriptor) -> None: - """Require the bounded descriptor bytes to match one opened project binding.""" - - descriptor_bytes = _read_descriptor(descriptor.descriptor_path) - if hashlib.sha256(descriptor_bytes).hexdigest() != descriptor.descriptor_hash: - raise DocForgeError( - "source_changed", - "Project descriptor changed after the project was opened", - ) - - def _markdown_record(path: Path, text: str) -> tuple[dict[str, Any], str]: lines = text.splitlines() if not lines or lines[0] != "+++": @@ -907,7 +739,11 @@ class Project: def load(self) -> ProjectSnapshot: increment("project_loads") - validate_descriptor_binding(self.descriptor) + descriptor_bytes = self.descriptor.descriptor_path.read_bytes() + if hashlib.sha256(descriptor_bytes).hexdigest() != self.descriptor.descriptor_hash: + raise DocForgeError( + "source_changed", "Project descriptor changed after the project was opened" + ) ordered_sources, ordered_directories = self._canonical_inventory() generation_paths = ( self.descriptor.descriptor_path, diff --git a/src/docforge/retrieval.py b/src/docforge/retrieval.py deleted file mode 100644 index 8302abc..0000000 --- a/src/docforge/retrieval.py +++ /dev/null @@ -1,671 +0,0 @@ -"""Versioned task-shaped retrieval plans and immutable context capsules.""" - -from __future__ import annotations - -from dataclasses import dataclass, replace -from typing import Literal, cast - -from .errors import DocForgeError -from .models import Edge, Node, ProjectDescriptor -from .pagination import canonical_hash - -TaskKind = Literal[ - "change", - "implementation", - "failure", - "ownership", - "test", - "operation", - "release", -] -TASK_KINDS: tuple[TaskKind, ...] = ( - "change", - "implementation", - "failure", - "ownership", - "test", - "operation", - "release", -) - -RelationCategory = Literal[ - "structure", - "implementation", - "dependency", - "execution", - "data", - "evidence", - "context", - "unclassified", -] -BASE_RELATION_CATEGORIES: tuple[RelationCategory, ...] = ( - "structure", - "implementation", - "dependency", - "execution", - "data", - "evidence", - "context", -) - -RELATION_CATEGORIES: dict[RelationCategory, tuple[str, ...]] = { - "structure": ("contains", "defined_in", "defines", "owns"), - "implementation": ( - "implemented_by", - "implements", - "inherits", - "inherits_from", - ), - "dependency": ("depends_on", "imports"), - "execution": ("activates", "calls", "dispatches_to", "launches"), - "data": ("reads", "writes"), - "evidence": ("documents", "governs", "proves", "tested_by", "verifies"), - "context": ("relates_to",), - "unclassified": (), -} - -TASK_REQUIREMENTS: dict[TaskKind, tuple[RelationCategory, ...]] = { - "change": ("dependency",), - "implementation": ("implementation",), - "failure": ("execution",), - "ownership": ("structure",), - "test": ("evidence",), - "operation": ("execution",), - "release": ("evidence",), -} - -PLANNER_ID = "docforge.core.task-context" -PLANNER_VERSION = 1 -MAX_TASK_EVIDENCE = 1_000 -MAX_TASK_CANDIDATE_EDGES = 100_000 -MAX_TASK_QUERY_CHARS = 10_000 -PROVENANCE_LIMITATIONS = ( - "evidence_type_unavailable", - "extractor_identity_unavailable", - "relationship_provenance_unavailable", - "observation_time_unavailable", -) - -_RELATION_TO_CATEGORY = { - relation: category - for category, relations in RELATION_CATEGORIES.items() - for relation in relations -} - - -def relation_category(relation: str) -> RelationCategory: - """Classify only versioned known aliases; preserve every other relation.""" - - return cast(RelationCategory, _RELATION_TO_CATEGORY.get(relation, "unclassified")) - - -@dataclass(frozen=True) -class RetrievalStepV1: - step_id: str - operation: Literal["exact", "search", "outgoing", "incoming", "metadata"] - relation_scope: Literal["none", "project_allowed"] - relation_set_hash: str | None - direction: Literal["none", "outgoing", "incoming"] - depth: int - limit: int - required: bool - evidence_role: str - - def as_dict(self) -> dict[str, object]: - return { - "step_id": self.step_id, - "operation": self.operation, - "relation_scope": self.relation_scope, - "relation_set_hash": self.relation_set_hash, - "direction": self.direction, - "depth": self.depth, - "limit": self.limit, - "required": self.required, - "evidence_role": self.evidence_role, - } - - -@dataclass(frozen=True) -class RetrievalRequirementV1: - requirement_id: str - category: RelationCategory - - def as_dict(self) -> dict[str, object]: - return { - "requirement_id": self.requirement_id, - "check": "selected_relation_category", - "category": self.category, - "required": True, - } - - -@dataclass(frozen=True) -class RetrievalPlanV1: - schema_version: Literal[1] - planner_id: str - planner_version: int - task_kind: TaskKind - task_query: str - focus_node_id: str | None - request_hash: str - effective_policy_hash: str - max_evidence: int - max_tokens: int - max_depth: int - max_candidate_edges: int - category_order: tuple[RelationCategory, ...] - steps: tuple[RetrievalStepV1, ...] - requirements: tuple[RetrievalRequirementV1, ...] - plan_hash: str - - def as_dict(self) -> dict[str, object]: - return self.payload(include_hash=True) - - def payload(self, *, include_hash: bool) -> dict[str, object]: - result: dict[str, object] = { - "schema_version": self.schema_version, - "planner": { - "id": self.planner_id, - "version": self.planner_version, - }, - "task_kind": self.task_kind, - "request_hash": self.request_hash, - "effective_policy_hash": self.effective_policy_hash, - "focus_node_id": self.focus_node_id, - "limits": { - "max_evidence": self.max_evidence, - "max_tokens": self.max_tokens, - "max_depth": self.max_depth, - "max_candidate_edges": self.max_candidate_edges, - }, - "category_order": list(self.category_order), - "steps": [step.as_dict() for step in self.steps], - "requirements": [requirement.as_dict() for requirement in self.requirements], - } - if include_hash: - result["plan_hash"] = self.plan_hash - return result - - -def build_retrieval_plan( - descriptor: ProjectDescriptor, - *, - task_kind: str, - task: str, - focus_node_id: str | None, - budget: int | None, - limit: int | None, - effective_policy: dict[str, object], -) -> RetrievalPlanV1: - """Derive one fixed plan from bounded inputs rather than accepting caller operations.""" - - return _build_retrieval_plan( - descriptor, - task_kind=task_kind, - task=task, - focus_node_id=focus_node_id, - budget=budget, - limit=limit, - effective_policy_hash=canonical_hash(effective_policy), - ) - - -def validate_retrieval_plan( - plan: RetrievalPlanV1, - descriptor: ProjectDescriptor, -) -> RetrievalPlanV1: - """Reject forged, stale-shape, or internally inconsistent public plan objects.""" - - try: - expected = _build_retrieval_plan( - descriptor, - task_kind=plan.task_kind, - task=plan.task_query, - focus_node_id=plan.focus_node_id, - budget=plan.max_tokens, - limit=plan.max_evidence, - effective_policy_hash=plan.effective_policy_hash, - ) - except (AttributeError, TypeError, DocForgeError) as error: - raise DocForgeError( - "invalid_retrieval_plan", - "Task retrieval plan is malformed or outside the fixed version-1 contract", - ) from error - if plan != expected: - raise DocForgeError( - "invalid_retrieval_plan", - "Task retrieval plan does not match its fixed version-1 derivation", - ) - return plan - - -def _build_retrieval_plan( - descriptor: ProjectDescriptor, - *, - task_kind: str, - task: str, - focus_node_id: str | None, - budget: int | None, - limit: int | None, - effective_policy_hash: str, -) -> RetrievalPlanV1: - if task_kind not in TASK_KINDS: - raise DocForgeError( - "invalid_task_kind", - "Task context kind is unsupported", - task_kind=task_kind, - allowed=list(TASK_KINDS), - ) - selected_kind: TaskKind = task_kind # type: ignore[assignment] - normalized_task = task.strip() - if not normalized_task or len(normalized_task) > min( - descriptor.limits.max_query_chars, - MAX_TASK_QUERY_CHARS, - ): - raise DocForgeError( - "invalid_task_focus", - "Task description is empty or exceeds the configured query limit", - ) - if focus_node_id is not None and (not focus_node_id or len(focus_node_id) > 256): - raise DocForgeError("invalid_task_focus", "Task focus node ID is invalid") - selected_budget = _bounded_value( - budget, - default=min(8_000, descriptor.limits.max_context_tokens), - maximum=descriptor.limits.max_context_tokens, - code="invalid_budget", - ) - selected_limit = _bounded_value( - limit, - default=min(20, descriptor.limits.max_results), - maximum=min(descriptor.limits.max_results, MAX_TASK_EVIDENCE), - code="invalid_limit", - ) - if not _is_sha256(effective_policy_hash): - raise DocForgeError( - "invalid_retrieval_plan", - "Effective policy identity is not a SHA-256 value", - ) - selected_depth = min(2, descriptor.limits.max_traversal_depth) - requirements = tuple( - RetrievalRequirementV1( - requirement_id=f"{selected_kind}.{category}", - category=category, - ) - for category in TASK_REQUIREMENTS[selected_kind] - ) - category_order: tuple[RelationCategory, ...] = ( - *TASK_REQUIREMENTS[selected_kind], - *( - category - for category in BASE_RELATION_CATEGORIES - if category not in TASK_REQUIREMENTS[selected_kind] - ), - "unclassified", - ) - relation_set_hash = canonical_hash(sorted(descriptor.allowed_relations)) - focus_operation: Literal["exact", "search"] = "exact" if focus_node_id else "search" - steps = ( - RetrievalStepV1( - step_id="focus", - operation=focus_operation, - relation_scope="none", - relation_set_hash=None, - direction="none", - depth=0, - limit=1, - required=True, - evidence_role="focus", - ), - RetrievalStepV1( - step_id="outgoing", - operation="outgoing", - relation_scope="project_allowed", - relation_set_hash=relation_set_hash, - direction="outgoing", - depth=selected_depth, - limit=selected_limit, - required=False, - evidence_role="related", - ), - RetrievalStepV1( - step_id="incoming", - operation="incoming", - relation_scope="project_allowed", - relation_set_hash=relation_set_hash, - direction="incoming", - depth=selected_depth, - limit=selected_limit, - required=False, - evidence_role="related", - ), - RetrievalStepV1( - step_id="metadata", - operation="metadata", - relation_scope="none", - relation_set_hash=None, - direction="none", - depth=0, - limit=selected_limit, - required=True, - evidence_role="provenance", - ), - ) - request_hash = canonical_hash( - { - "task_kind": selected_kind, - "task": normalized_task, - "focus_node_id": focus_node_id, - "budget": selected_budget, - "limit": selected_limit, - } - ) - placeholder = RetrievalPlanV1( - schema_version=1, - planner_id=PLANNER_ID, - planner_version=PLANNER_VERSION, - task_kind=selected_kind, - task_query=normalized_task, - focus_node_id=focus_node_id, - request_hash=request_hash, - effective_policy_hash=effective_policy_hash, - max_evidence=selected_limit, - max_tokens=selected_budget, - max_depth=selected_depth, - max_candidate_edges=min( - (selected_limit + 1) ** 2, - MAX_TASK_CANDIDATE_EDGES, - ), - category_order=category_order, - steps=steps, - requirements=requirements, - plan_hash="", - ) - return replace( - placeholder, - plan_hash=canonical_hash(placeholder.payload(include_hash=False)), - ) - - -@dataclass(frozen=True) -class CapsuleRelationshipV1: - source_id: str - relation: str - target_id: str - direction: Literal["outgoing", "incoming"] - category: RelationCategory - - def as_dict(self) -> dict[str, object]: - return { - "source_id": self.source_id, - "relation": self.relation, - "target_id": self.target_id, - "direction": self.direction, - "category": self.category, - "provenance": "validated_graph_edge_without_source_provenance", - } - - -@dataclass(frozen=True) -class CapsuleEvidenceV1: - evidence_hash: str - role: Literal["focus", "related"] - reason_code: Literal["exact_focus", "lexical_focus", "relationship_path"] - node: Node - depth: int - relationship_path: tuple[CapsuleRelationshipV1, ...] - relationship_reasons: tuple[CapsuleRelationshipV1, ...] - estimated_tokens: int - - def as_dict(self) -> dict[str, object]: - result = self.payload() - return {"evidence_hash": self.evidence_hash, **result} - - def payload(self) -> dict[str, object]: - result: dict[str, object] = { - "role": self.role, - "reason_code": self.reason_code, - "node_id": self.node.node_id, - "title": self.node.title, - "family": self.node.family, - "authority": self.node.authority, - "status": self.node.status, - "tags": list(self.node.tags), - "summary": self.node.summary, - "text": _node_text(self.node), - "estimated_tokens": self.estimated_tokens, - "source": { - "path": self.node.source_path, - "anchor": self.node.source_anchor, - "content_hash": self.node.content_hash, - }, - "depth": self.depth, - "relationship_path": [ - relationship.as_dict() for relationship in self.relationship_path - ], - "relationship_reasons": [ - relationship.as_dict() for relationship in self.relationship_reasons - ], - "provenance_limitations": list(PROVENANCE_LIMITATIONS), - } - return result - - -def capsule_evidence( - *, - role: Literal["focus", "related"], - reason_code: Literal["exact_focus", "lexical_focus", "relationship_path"], - node: Node, - depth: int, - relationship_path: tuple[CapsuleRelationshipV1, ...], - relationship_reasons: tuple[CapsuleRelationshipV1, ...], -) -> CapsuleEvidenceV1: - tokens = estimate_tokens(_node_text(node)) - placeholder = CapsuleEvidenceV1( - evidence_hash="", - role=role, - reason_code=reason_code, - node=node, - depth=depth, - relationship_path=relationship_path, - relationship_reasons=relationship_reasons, - estimated_tokens=tokens, - ) - return replace( - placeholder, - evidence_hash=canonical_hash(placeholder.payload()), - ) - - -@dataclass(frozen=True) -class EvidenceGapV1: - code: Literal[ - "focus_not_found", - "focus_ambiguous", - "category_not_declared", - "no_selected_evidence", - "evidence_incomplete", - "unclassified_relation", - ] - requirement_id: str - category: RelationCategory | None - state: Literal["missing", "incomplete", "blocked", "limitation"] - check_complete: bool - detail: str - - def as_dict(self) -> dict[str, object]: - return { - "code": self.code, - "requirement_id": self.requirement_id, - "category": self.category, - "state": self.state, - "check_complete": self.check_complete, - "detail": self.detail, - } - - -@dataclass(frozen=True) -class CapsuleOmissionV1: - code: Literal[ - "result_limit", - "token_budget", - "response_limit", - "edge_examination_limit", - "unclassified_relation_limit", - ] - subject: str - detail_hash: str - - def as_dict(self) -> dict[str, str]: - return { - "code": self.code, - "subject": self.subject, - "detail_hash": self.detail_hash, - } - - -@dataclass(frozen=True) -class ContextCapsuleV1: - schema_version: Literal[1] - state: Literal["complete", "incomplete", "blocked"] - task_kind: TaskKind - generation: tuple[tuple[str, object], ...] - plan: RetrievalPlanV1 - focus_state: Literal["resolved", "not_found", "ambiguous"] - focus_node_id: str | None - focus_candidate_count: int - evidence: tuple[CapsuleEvidenceV1, ...] - gaps: tuple[EvidenceGapV1, ...] - omissions: tuple[CapsuleOmissionV1, ...] - selected_count: int - examined_edge_count: int - estimated_tokens: int - unclassified_relations: tuple[str, ...] - collection_hash: str - capsule_hash: str - - def as_dict(self) -> dict[str, object]: - return self.payload(include_hashes=True) - - def payload(self, *, include_hashes: bool) -> dict[str, object]: - evidence = [item.as_dict() for item in self.evidence] - gaps = [gap.as_dict() for gap in self.gaps] - omissions = [omission.as_dict() for omission in self.omissions] - result: dict[str, object] = { - "schema_version": self.schema_version, - "state": self.state, - "task_kind": self.task_kind, - "generation": dict(self.generation), - "plan": self.plan.as_dict(), - "focus": { - "state": self.focus_state, - "node_id": self.focus_node_id, - "candidate_count": self.focus_candidate_count, - }, - "evidence": evidence, - "gaps": gaps, - "omissions": omissions, - "summary": { - "evidence_count": len(evidence), - "gap_count": len(gaps), - "omission_count": len(omissions), - "selected_count": self.selected_count, - "examined_edge_count": self.examined_edge_count, - "estimated_tokens": self.estimated_tokens, - "unclassified_relations": list(self.unclassified_relations), - }, - } - if include_hashes: - result["collection_hash"] = self.collection_hash - result["capsule_hash"] = self.capsule_hash - return result - - -def finalize_capsule( - *, - state: Literal["complete", "incomplete", "blocked"], - task_kind: TaskKind, - generation: dict[str, object], - plan: RetrievalPlanV1, - focus_state: Literal["resolved", "not_found", "ambiguous"], - focus_node_id: str | None, - focus_candidate_count: int, - evidence: tuple[CapsuleEvidenceV1, ...], - gaps: tuple[EvidenceGapV1, ...], - omissions: tuple[CapsuleOmissionV1, ...], - selected_count: int, - examined_edge_count: int, - estimated_tokens: int, - unclassified_relations: tuple[str, ...], -) -> ContextCapsuleV1: - collection_hash = canonical_hash( - { - "generation": generation, - "plan_hash": plan.plan_hash, - "evidence": [item.evidence_hash for item in evidence], - "gaps": [gap.as_dict() for gap in gaps], - "omissions": [omission.as_dict() for omission in omissions], - } - ) - placeholder = ContextCapsuleV1( - schema_version=1, - state=state, - task_kind=task_kind, - generation=tuple(generation.items()), - plan=plan, - focus_state=focus_state, - focus_node_id=focus_node_id, - focus_candidate_count=focus_candidate_count, - evidence=evidence, - gaps=gaps, - omissions=omissions, - selected_count=selected_count, - examined_edge_count=examined_edge_count, - estimated_tokens=estimated_tokens, - unclassified_relations=unclassified_relations, - collection_hash=collection_hash, - capsule_hash="", - ) - capsule_hash = canonical_hash( - { - **placeholder.payload(include_hashes=False), - "collection_hash": collection_hash, - } - ) - return replace(placeholder, capsule_hash=capsule_hash) - - -def estimate_tokens(text: str) -> int: - return max(1, (len(text) + 3) // 4) - - -def edge_tuple(edge: Edge) -> tuple[str, str, str]: - return edge.source_id, edge.relation, edge.target_id - - -def _node_text(node: Node) -> str: - return ( - f"ID: {node.node_id}\nTitle: {node.title}\nFamily: {node.family}\n" - f"Authority: {node.authority}\nStatus: {node.status}\nSource: {node.source_path}\n" - f"Summary: {node.summary}\n\n{node.content}" - ) - - -def _bounded_value( - value: int | None, - *, - default: int, - maximum: int, - code: str, -) -> int: - selected = default if value is None else value - if type(selected) is not int or selected < 1 or selected > maximum: - raise DocForgeError(code, "Task context limit is outside the configured range") - return selected - - -def _is_sha256(value: object) -> bool: - return ( - isinstance(value, str) - and len(value) == 64 - and all(character in "0123456789abcdef" for character in value) - ) diff --git a/src/docforge/telemetry.py b/src/docforge/telemetry.py index 4c9c6d8..c7e1f8c 100644 --- a/src/docforge/telemetry.py +++ b/src/docforge/telemetry.py @@ -78,7 +78,6 @@ OPERATION_NAMES = frozenset( { "test", "benchmark.m1", - "benchmark.m2", "mcp.invoke", "mcp.bootstrap", "mcp.sync", @@ -92,8 +91,6 @@ OPERATION_NAMES = frozenset( "mcp.dependencies", "mcp.impact", "mcp.context", - "mcp.task_context", - "mcp.generation_diff", "mcp.validate_project", "mcp.render_status", "mcp.visualize", @@ -116,9 +113,6 @@ OPERATION_NAMES = frozenset( "cli.dependencies", "cli.impact", "cli.context", - "cli.generation-diff", - "cli.configure", - "cli.doctor", "cli.render", "cli.render-status", "cli.preview", diff --git a/tests/test_adapter_contract.py b/tests/test_adapter_contract.py index e4bc3f5..cf88e67 100644 --- a/tests/test_adapter_contract.py +++ b/tests/test_adapter_contract.py @@ -2,7 +2,6 @@ from __future__ import annotations import hashlib import importlib -import json import sqlite3 import sys import tempfile @@ -56,10 +55,8 @@ from docforge.visualization import VisualizationIndexSnapshot class Loader: def __init__(self, projection: AdapterProjection) -> None: self.projection = projection - self.load_calls = 0 def load_projection(self) -> AdapterProjection: - self.load_calls += 1 return self.projection @@ -646,17 +643,12 @@ class AdapterContractTests(unittest.TestCase): def test_no_ast_index_accepts_legacy_and_non_logic_incremental_adapters(self) -> None: with tempfile.TemporaryDirectory() as directory: root = Path(directory).resolve() - legacy_loader = Loader(self.projection(root)) legacy = AdapterProject( - legacy_loader, + Loader(self.projection(root)), cache_root=root / ".cache" / "legacy-no-ast", ) legacy_index = ProjectIndex(legacy, allow_logic=False) self.assertEqual(2, legacy_index.build()["node_count"]) - calls_after_build = legacy_loader.load_calls - legacy_diff = legacy_index.generation_diff() - self.assertEqual("unknown", legacy_diff["receipt_state"]) - self.assertEqual(calls_after_build, legacy_loader.load_calls) self.assertEqual( "guide.workflow", legacy_index.get_node("guide.workflow")["node"]["node_id"], @@ -680,12 +672,6 @@ class AdapterContractTests(unittest.TestCase): self.assertEqual(2, second["build"]["cache_hits"]) self.assertEqual(0, second["build"]["reparsed_sources"]) self.assertEqual(0, second["logic_projection_count"]) - generation_diff = index.generation_diff() - self.assertEqual("current", generation_diff["receipt_state"]) - self.assertNotIn( - "logic", - json.dumps(generation_diff["generation_diff"]).casefold(), - ) def test_no_ast_rejects_preexisting_logic_index_and_viewer_snapshot(self) -> None: with tempfile.TemporaryDirectory() as directory: @@ -880,9 +866,8 @@ class AdapterReadOnlyMcpTests(unittest.IsolatedAsyncioTestCase): with tempfile.TemporaryDirectory() as directory: root = Path(directory).resolve() fixture = AdapterContractTests() - loader = Loader(fixture.projection(root)) project = AdapterProject( - loader, + Loader(fixture.projection(root)), cache_root=root / ".cache" / "adapter-read-only", ) index = ProjectIndex(project) @@ -918,14 +903,6 @@ class AdapterReadOnlyMcpTests(unittest.IsolatedAsyncioTestCase): context = await session.call_tool( "docforge_get_context", {"profile": "fixture", "budget": 321} ) - load_calls = loader.load_calls - task_context = await session.call_tool( - "docforge_get_task_context", - { - "task_kind": "change", - "task": "Do not load the legacy projection for this capability error", - }, - ) self.assertEqual(READ_TOOLS, tuple(tool.name for tool in tools.tools)) self.assertEqual("adapter-fixture", info.structuredContent["project_id"]) @@ -938,12 +915,6 @@ class AdapterReadOnlyMcpTests(unittest.IsolatedAsyncioTestCase): self.assertFalse(contract.structuredContent["isolated_changeset_writes_allowed"]) self.assertEqual("fixture", context.structuredContent["profile"]) self.assertEqual([("fixture", 321)], calls) - self.assertEqual("error", task_context.structuredContent["status"]) - self.assertEqual( - "task_context_unavailable", - task_context.structuredContent["error"]["code"], - ) - self.assertEqual(load_calls, loader.load_calls) self.assertFalse(project.descriptor.changeset_root.exists()) async def test_adapter_project_proposals_require_explicit_policy_and_stay_isolated( diff --git a/tests/test_client_integration.py b/tests/test_client_integration.py deleted file mode 100644 index 778b96a..0000000 --- a/tests/test_client_integration.py +++ /dev/null @@ -1,1264 +0,0 @@ -from __future__ import annotations - -import contextlib -import io -import json -import os -import shutil -import stat -import tempfile -import tomllib -import unittest -from collections.abc import Callable -from dataclasses import replace -from pathlib import Path -from unittest import mock - -from jsonschema import Draft202012Validator -from mcp import ClientSession, StdioServerParameters -from mcp.client.stdio import stdio_client - -from docforge.cli import _parser, _run, main -from docforge.client_config import ( - _read_existing, - _validate_configuration_result, - generate_client_configuration, -) -from docforge.doctor import run_doctor -from docforge.errors import DocForgeError -from docforge.index import ProjectIndex -from docforge.mcp_server import READ_TOOLS -from docforge.project import MAX_PROJECT_DESCRIPTOR_BYTES, Project - -ROOT = Path(__file__).resolve().parents[1] -FIXTURES = ROOT / "tests" / "fixtures" -SCHEMAS = ROOT / "schemas" -CONFIGURATION_SCHEMA = json.loads( - (SCHEMAS / "client-configuration.schema.json").read_text(encoding="utf-8") -) -DOCTOR_SCHEMA = json.loads((SCHEMAS / "doctor-result.schema.json").read_text(encoding="utf-8")) -POLICY_SCHEMA = json.loads((SCHEMAS / "policy.schema.json").read_text(encoding="utf-8")) - - -class ClientIntegrationTests(unittest.TestCase): - def copy_fixture(self, destination: Path) -> Path: - root = destination / "alpha" - shutil.copytree(FIXTURES / "alpha", root) - return root - - @staticmethod - def tree_snapshot(root: Path) -> dict[str, tuple[int, int, str]]: - return { - path.relative_to(root).as_posix(): ( - path.stat().st_mode, - path.stat().st_size, - path.read_bytes().hex(), - ) - for path in sorted(root.rglob("*")) - if path.is_file() - } - - def test_exact_outline_cli_forms_and_legacy_project_root_forms_parse(self) -> None: - parser = _parser() - for client in ("codex", "claude", "openclaw"): - configured = parser.parse_args(["configure", client, "--project", "/tmp/project"]) - self.assertEqual("configure", configured.command) - self.assertEqual(client, configured.client) - self.assertEqual(Path("/tmp/project"), configured.project) - doctor = parser.parse_args(["doctor", "--client", "codex"]) - self.assertEqual("doctor", doctor.command) - self.assertIsNone(doctor.project) - legacy = parser.parse_args(["--project-root", "/tmp/project", "info"]) - self.assertEqual(Path("/tmp/project"), legacy.project_root) - with self.assertRaises(DocForgeError) as missing: - _run(parser.parse_args(["info"])) - self.assertEqual("missing_project_root", missing.exception.code) - - def test_configuration_preview_is_deterministic_parseable_and_secret_free(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - before = self.tree_snapshot(root) - previous = os.environ.get("DOCFORGE_TEST_SECRET") - os.environ["DOCFORGE_TEST_SECRET"] = "must-not-appear" - try: - results: dict[str, dict[str, object]] = {} - for client in ("codex", "claude", "openclaw"): - first = generate_client_configuration(project, client, no_ast=True) - second = generate_client_configuration(project, client, no_ast=True) - self.assertEqual(first, second) - Draft202012Validator(CONFIGURATION_SCHEMA).validate(first) - Draft202012Validator(POLICY_SCHEMA).validate(first["effective_policy"]) - self.assertEqual("read", first["binding"]["capability_mode"]) - self.assertEqual( - "preserve-no-ast", - first["binding"]["adapter_policy"]["mode"], - ) - self.assertEqual({}, first["binding"]["environment"]) - self.assertNotIn( - "must-not-appear", - json.dumps(first, sort_keys=True), - ) - results[client] = first - finally: - if previous is None: - os.environ.pop("DOCFORGE_TEST_SECRET", None) - else: - os.environ["DOCFORGE_TEST_SECRET"] = previous - - codex_content = results["codex"]["artifact"]["content"] - self.assertIn("mcp_servers", tomllib.loads(codex_content)) - openclaw_content = results["openclaw"]["artifact"]["content"] - self.assertIn("mcp", json.loads(openclaw_content)) - claude_content = results["claude"]["artifact"]["content"] - self.assertIn("mcpServers", json.loads(claude_content)) - self.assertEqual(before, self.tree_snapshot(root)) - - def test_capability_bindings_fail_closed_and_render_policy_is_derived(self) -> None: - with tempfile.TemporaryDirectory() as directory: - project = Project.open(self.copy_fixture(Path(directory))) - with self.assertRaises(DocForgeError) as missing_writer: - generate_client_configuration( - project, - "codex", - capability_mode="proposal", - ) - self.assertEqual("capability_unavailable", missing_writer.exception.code) - proposal = generate_client_configuration( - project, - "codex", - capability_mode="proposal", - proposal_writer="alpha-editor", - ) - self.assertEqual("explicit", proposal["binding"]["render_policy"]["manual"]) - application = generate_client_configuration( - project, - "openclaw", - capability_mode="application", - proposal_writer="alpha-editor", - canonical_applier="alpha-editor", - ) - self.assertEqual("auto", application["binding"]["render_policy"]["manual"]) - with self.assertRaises(DocForgeError) as mismatch: - generate_client_configuration( - project, - "openclaw", - capability_mode="application", - proposal_writer="alpha-editor", - canonical_applier="other", - ) - self.assertEqual("capability_unavailable", mismatch.exception.code) - with self.assertRaises(DocForgeError) as escalated: - generate_client_configuration( - project, - "codex", - proposal_writer="alpha-editor", - ) - self.assertEqual("invalid_capability_binding", escalated.exception.code) - - def test_explicit_fragment_write_is_atomic_conflict_aware_and_private(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - project = Project.open(self.copy_fixture(parent)) - output = parent / "client" / "docforge.toml" - output.parent.mkdir() - created = generate_client_configuration( - project, - "codex", - output=output, - ) - self.assertEqual("created", created["artifact"]["write_state"]) - self.assertEqual( - stat.S_IMODE(output.stat().st_mode), - 0o600, - ) - unchanged = generate_client_configuration( - project, - "codex", - output=output, - ) - self.assertEqual("unchanged", unchanged["artifact"]["write_state"]) - - output.write_text("different\n", encoding="utf-8") - with self.assertRaises(DocForgeError) as conflict: - generate_client_configuration( - project, - "codex", - output=output, - ) - self.assertEqual("output_conflict", conflict.exception.code) - self.assertEqual("different\n", output.read_text(encoding="utf-8")) - output.unlink() - output.symlink_to(parent / "outside") - with self.assertRaises(DocForgeError) as unsafe: - generate_client_configuration( - project, - "codex", - output=output, - ) - self.assertEqual("unsafe_output", unsafe.exception.code) - self.assertEqual([], list(output.parent.glob(".docforge-client-*"))) - - linked_parent = parent / "linked-client" - outside = parent / "outside-client" - outside.mkdir() - linked_parent.symlink_to(outside, target_is_directory=True) - with self.assertRaises(DocForgeError) as escaped: - generate_client_configuration( - project, - "codex", - output=linked_parent / "fragment.toml", - ) - self.assertEqual("unsafe_output", escaped.exception.code) - self.assertFalse((outside / "fragment.toml").exists()) - - def test_committed_fragment_reports_unconfirmed_directory_durability(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - project = Project.open(self.copy_fixture(parent)) - output = parent / "client" / "docforge.toml" - output.parent.mkdir() - with mock.patch( - "docforge.client_config.os.fsync", - side_effect=[None, OSError("directory fsync unavailable")], - ): - created = generate_client_configuration( - project, - "codex", - output=output, - ) - Draft202012Validator(CONFIGURATION_SCHEMA).validate(created) - self.assertEqual("created", created["artifact"]["write_state"]) - self.assertEqual("unconfirmed", created["artifact"]["durability"]) - self.assertEqual( - [{"code": "publication_durability_unconfirmed"}], - created["warnings"], - ) - self.assertTrue(output.is_file()) - self.assertEqual( - created["artifact"]["content"], - output.read_text(encoding="utf-8"), - ) - unchanged = generate_client_configuration( - project, - "codex", - output=output, - ) - self.assertEqual("unchanged", unchanged["artifact"]["write_state"]) - - def test_configuration_fails_closed_on_stale_or_unimportable_runtime(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - descriptor = root / ".docforge" / "project.toml" - descriptor.write_text( - descriptor.read_text(encoding="utf-8") + "\n", - encoding="utf-8", - ) - with self.assertRaises(DocForgeError) as stale: - generate_client_configuration(project, "codex") - self.assertEqual("source_changed", stale.exception.code) - - current = Project.open(root) - failed_probe = mock.Mock(returncode=1) - with ( - mock.patch( - "docforge.client_config.subprocess.run", - return_value=failed_probe, - ), - self.assertRaises(DocForgeError) as unavailable, - ): - generate_client_configuration(current, "codex") - self.assertEqual( - "client_configuration_unavailable", - unavailable.exception.code, - ) - - def test_fragment_publication_rejects_parent_replacement_and_os_errors(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - project = Project.open(self.copy_fixture(parent)) - output_parent = parent / "client" - output_parent.mkdir() - output = output_parent / "docforge.toml" - moved = parent / "moved-client" - original_link = os.link - swapped = False - - def swap_parent(*args: object, **kwargs: object) -> None: - nonlocal swapped - if not swapped: - output_parent.rename(moved) - output_parent.mkdir() - swapped = True - original_link(*args, **kwargs) - - with ( - mock.patch( - "docforge.client_config.os.link", - side_effect=swap_parent, - ), - self.assertRaises(DocForgeError) as changed, - ): - generate_client_configuration( - project, - "codex", - output=output, - ) - self.assertEqual("output_changed", changed.exception.code) - self.assertFalse(output.exists()) - self.assertEqual([], list(moved.glob("docforge.toml"))) - self.assertEqual([], list(moved.glob(".docforge-client-*"))) - - with ( - mock.patch( - "docforge.client_config.os.link", - side_effect=PermissionError("denied"), - ), - self.assertRaises(DocForgeError) as denied, - ): - generate_client_configuration( - project, - "codex", - output=output, - ) - self.assertEqual("output_publication_failed", denied.exception.code) - self.assertEqual([], list(output_parent.glob(".docforge-client-*"))) - - def test_identical_existing_fragment_must_already_be_private_and_singly_linked( - self, - ) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - project = Project.open(self.copy_fixture(parent)) - output = parent / "docforge.toml" - preview = generate_client_configuration(project, "codex") - output.write_text(preview["artifact"]["content"], encoding="utf-8") - output.chmod(0o644) - with self.assertRaises(DocForgeError) as public: - generate_client_configuration(project, "codex", output=output) - self.assertEqual("unsafe_output", public.exception.code) - - def test_unchanged_fragment_revalidates_binding_after_read(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - project = Project.open(root) - output = parent / "docforge.toml" - generate_client_configuration(project, "codex", output=output) - descriptor = root / ".docforge" / "project.toml" - - def mutate_after_read(*args: object, **kwargs: object) -> bytes: - content = _read_existing(*args, **kwargs) # type: ignore[arg-type] - descriptor.write_text( - descriptor.read_text(encoding="utf-8") + "\n", - encoding="utf-8", - ) - return content - - with ( - mock.patch( - "docforge.client_config._read_existing", - side_effect=mutate_after_read, - ), - self.assertRaises(DocForgeError) as stale, - ): - generate_client_configuration(project, "codex", output=output) - self.assertEqual("source_changed", stale.exception.code) - - def test_fragment_publication_revalidates_descriptor_after_link(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - project = Project.open(root) - output_parent = parent / "client" - output_parent.mkdir() - output = output_parent / "docforge.toml" - descriptor = root / ".docforge" / "project.toml" - original_link = os.link - - def mutate_descriptor(*args: object, **kwargs: object) -> None: - original_link(*args, **kwargs) - descriptor.write_text( - descriptor.read_text(encoding="utf-8") + "\n", - encoding="utf-8", - ) - - with ( - mock.patch( - "docforge.client_config.os.link", - side_effect=mutate_descriptor, - ), - self.assertRaises(DocForgeError) as stale, - ): - generate_client_configuration( - project, - "codex", - output=output, - ) - self.assertEqual("source_changed", stale.exception.code) - self.assertFalse(output.exists()) - self.assertEqual([], list(output_parent.glob(".docforge-client-*"))) - - def test_unprovable_postlink_binding_returns_schema_valid_degraded_evidence( - self, - ) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - project = Project.open(root) - output_parent = parent / "client" - output_parent.mkdir() - output = output_parent / "docforge.toml" - descriptor = root / ".docforge" / "project.toml" - original_link = os.link - - def mutate_descriptor(*args: object, **kwargs: object) -> None: - original_link(*args, **kwargs) - descriptor.write_text( - descriptor.read_text(encoding="utf-8") + "\n", - encoding="utf-8", - ) - - with ( - mock.patch( - "docforge.client_config.os.link", - side_effect=mutate_descriptor, - ), - mock.patch( - "docforge.client_config._rollback_link", - return_value=False, - ), - ): - result = generate_client_configuration( - project, - "codex", - output=output, - ) - Draft202012Validator(CONFIGURATION_SCHEMA).validate(result) - self.assertEqual("created", result["artifact"]["write_state"]) - self.assertEqual("unconfirmed", result["artifact"]["durability"]) - self.assertIsNone(result["artifact"]["output_path"]) - self.assertIn( - {"code": "publication_binding_unconfirmed"}, - result["warnings"], - ) - - def test_fragment_publication_revalidates_after_directory_fsync(self) -> None: - for scenario in ("descriptor", "target", "parent"): - with self.subTest(scenario=scenario), tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - project = Project.open(root) - output_parent = parent / "client" - output_parent.mkdir() - output = output_parent / "docforge.toml" - descriptor = root / ".docforge" / "project.toml" - moved = parent / "moved-client" - original_fsync = os.fsync - calls = 0 - - def mutate_at_directory_fsync( - file_descriptor: int, - selected_scenario: str = scenario, - selected_descriptor: Path = descriptor, - selected_output: Path = output, - selected_parent: Path = output_parent, - selected_moved: Path = moved, - selected_fsync: Callable[[int], None] = original_fsync, - ) -> None: - nonlocal calls - calls += 1 - selected_fsync(file_descriptor) - if calls != 2: - return - if selected_scenario == "descriptor": - selected_descriptor.write_text( - selected_descriptor.read_text(encoding="utf-8") + "\n", - encoding="utf-8", - ) - elif selected_scenario == "target": - selected_output.write_text("tampered\n", encoding="utf-8") - else: - selected_parent.rename(selected_moved) - selected_parent.mkdir() - - with ( - mock.patch( - "docforge.client_config.os.fsync", - side_effect=mutate_at_directory_fsync, - ), - self.assertRaises(DocForgeError) as changed, - ): - generate_client_configuration( - project, - "codex", - output=output, - ) - self.assertIn(changed.exception.code, {"source_changed", "output_changed"}) - self.assertFalse(output.exists()) - self.assertFalse((moved / "docforge.toml").exists()) - - def test_doctor_round_trip_is_schema_valid_and_performs_no_hidden_work(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - project = Project.open(root) - ProjectIndex(project).build() - config = parent / "codex.toml" - generated = generate_client_configuration( - project, - "codex", - no_ast=True, - output=config, - ) - before_project = self.tree_snapshot(root) - before_config = config.read_bytes() - with ( - mock.patch.object( - Project, - "load", - side_effect=AssertionError("doctor must not load project sources"), - ), - mock.patch( - "docforge.index.ProjectIndex.check", - side_effect=AssertionError("doctor must not check SQLite"), - ), - mock.patch( - "docforge.index.ProjectIndex.synchronize", - side_effect=AssertionError("doctor must not synchronize"), - ), - mock.patch( - "docforge.index.ProjectIndex.build", - side_effect=AssertionError("doctor must not build"), - ), - mock.patch( - "subprocess.run", - side_effect=AssertionError("doctor must not execute configured commands"), - ), - ): - result = run_doctor( - project, - "codex", - config_path=config, - server_name=generated["server_name"], - ) - Draft202012Validator(DOCTOR_SCHEMA).validate(result) - self.assertEqual("healthy", result["doctor_state"]) - self.assertEqual(0, result["summary"]["warning"]) - self.assertEqual(0, result["summary"]["failed"]) - self.assertTrue(result["guarantees"]["read_only"]) - no_ast = next( - check for check in result["checks"] if check["check_id"] == "policy.no_ast" - ) - self.assertEqual("no_ast_policy_valid", no_ast["code"]) - self.assertEqual("unverifiable", no_ast["details"]["adapter_internals"]) - self.assertEqual(before_project, self.tree_snapshot(root)) - self.assertEqual(before_config, config.read_bytes()) - - def test_doctor_reports_missing_malformed_and_secret_environment_without_echoing(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - project = Project.open(root) - missing = run_doctor( - project, - "codex", - config_path=parent / "missing.toml", - ) - Draft202012Validator(DOCTOR_SCHEMA).validate(missing) - self.assertEqual("unhealthy", missing["doctor_state"]) - malformed_path = parent / "malformed.toml" - malformed_path.write_text("[invalid", encoding="utf-8") - malformed = run_doctor( - project, - "codex", - config_path=malformed_path, - ) - Draft202012Validator(DOCTOR_SCHEMA).validate(malformed) - self.assertEqual("unhealthy", malformed["doctor_state"]) - - generated = generate_client_configuration(project, "openclaw") - document = json.loads(generated["artifact"]["content"]) - entry = next(iter(document["mcp"]["servers"].values())) - entry["env"] = {"TOKEN": "top-secret-value"} - secret_path = parent / "openclaw.json" - secret_path.write_text( - json.dumps(document, sort_keys=True), - encoding="utf-8", - ) - secret = run_doctor( - project, - "openclaw", - config_path=secret_path, - server_name=generated["server_name"], - ) - Draft202012Validator(DOCTOR_SCHEMA).validate(secret) - self.assertEqual("degraded", secret["doctor_state"]) - encoded = json.dumps(secret, sort_keys=True) - self.assertIn("TOKEN", encoded) - self.assertNotIn("top-secret-value", encoded) - - def test_cli_doctor_exit_codes_are_stable(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - project = Project.open(root) - ProjectIndex(project).build() - config = parent / "codex.toml" - generated = generate_client_configuration(project, "codex", output=config) - output = io.StringIO() - with contextlib.redirect_stdout(output): - healthy = main( - [ - "doctor", - "--client", - "codex", - "--project", - str(root), - "--config", - str(config), - "--server-name", - str(generated["server_name"]), - ] - ) - self.assertEqual(0, healthy) - self.assertEqual("healthy", json.loads(output.getvalue())["doctor_state"]) - - output = io.StringIO() - with contextlib.redirect_stdout(output): - degraded = main( - [ - "doctor", - "--client", - "codex", - "--project", - str(root), - "--config", - str(parent / "missing.toml"), - ] - ) - self.assertEqual(2, degraded) - self.assertEqual("unhealthy", json.loads(output.getvalue())["doctor_state"]) - - malformed = parent / "malformed.toml" - malformed.write_text("[invalid", encoding="utf-8") - output = io.StringIO() - with contextlib.redirect_stdout(output): - unhealthy = main( - [ - "doctor", - "--client", - "codex", - "--project", - str(root), - "--config", - str(malformed), - ] - ) - self.assertEqual(2, unhealthy) - self.assertEqual("unhealthy", json.loads(output.getvalue())["doctor_state"]) - - def test_project_descriptor_read_is_bounded_stable_and_symlink_safe(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - descriptor = root / ".docforge" / "project.toml" - original = descriptor.read_bytes() - descriptor.unlink() - external = parent / "external.toml" - external.write_bytes(original) - descriptor.symlink_to(external) - with self.assertRaises(DocForgeError) as symlinked: - Project.open(root) - self.assertEqual("project_descriptor_unsafe", symlinked.exception.code) - - descriptor.unlink() - descriptor.write_bytes(b"x" * (MAX_PROJECT_DESCRIPTOR_BYTES + 1)) - with self.assertRaises(DocForgeError) as oversized: - Project.open(root) - self.assertEqual("project_descriptor_oversized", oversized.exception.code) - - shutil.rmtree(root / ".docforge") - external_directory = parent / "external-docforge" - external_directory.mkdir() - (external_directory / "project.toml").write_bytes(original) - (root / ".docforge").symlink_to(external_directory, target_is_directory=True) - with self.assertRaises(DocForgeError) as escaped: - Project.open(root) - self.assertEqual("project_descriptor_unsafe", escaped.exception.code) - - def test_project_load_revalidates_the_bounded_descriptor(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - descriptor = root / ".docforge" / "project.toml" - descriptor.write_bytes(b"x" * (MAX_PROJECT_DESCRIPTOR_BYTES + 1)) - with self.assertRaises(DocForgeError) as oversized: - project.load() - self.assertEqual("project_descriptor_oversized", oversized.exception.code) - - def test_doctor_ignores_unrelated_entries_and_honors_codex_home(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - project = Project.open(root) - ProjectIndex(project).build() - codex_home = parent / "codex-home" - codex_home.mkdir() - config = codex_home / "config.toml" - generated = generate_client_configuration(project, "codex", output=config) - config.write_text( - config.read_text(encoding="utf-8") - + '\n[mcp_servers."remote"]\nurl = "https://example.invalid/mcp"\n', - encoding="utf-8", - ) - with mock.patch.dict(os.environ, {"CODEX_HOME": str(codex_home)}): - result = run_doctor(project, "codex") - Draft202012Validator(DOCTOR_SCHEMA).validate(result) - self.assertEqual("healthy", result["doctor_state"]) - self.assertEqual(generated["server_name"], result["config"]["server_name"]) - - openclaw_config = parent / "openclaw.json" - openclaw = generate_client_configuration( - project, - "openclaw", - output=openclaw_config, - ) - document = json.loads(openclaw_config.read_text(encoding="utf-8")) - document["mcp"]["servers"]["remote"] = { - "type": "http", - "url": "https://example.invalid/mcp", - "enabled": True, - } - openclaw_config.write_text( - json.dumps(document, sort_keys=True), - encoding="utf-8", - ) - openclaw_result = run_doctor( - project, - "openclaw", - config_path=openclaw_config, - ) - Draft202012Validator(DOCTOR_SCHEMA).validate(openclaw_result) - self.assertEqual("healthy", openclaw_result["doctor_state"]) - self.assertEqual( - openclaw["server_name"], - openclaw_result["config"]["server_name"], - ) - - selected = next(iter(document["mcp"]["servers"].values())) - selected["cwd"] = str(root) - selected["toolFilter"] = {"include": ["docforge_bootstrap"]} - openclaw_config.write_text( - json.dumps(document, sort_keys=True), - encoding="utf-8", - ) - filtered = run_doctor( - project, - "openclaw", - config_path=openclaw_config, - ) - Draft202012Validator(DOCTOR_SCHEMA).validate(filtered) - self.assertEqual("degraded", filtered["doctor_state"]) - self.assertIn( - "client_tool_filter_unverified", - [check["code"] for check in filtered["checks"]], - ) - - def test_doctor_rejects_nonlaunching_or_malformed_matching_arguments(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - project = Project.open(root) - ProjectIndex(project).build() - generated = generate_client_configuration(project, "openclaw") - document = json.loads(generated["artifact"]["content"]) - entry = next(iter(document["mcp"]["servers"].values())) - entry["args"] = entry["args"][3:] - config = parent / "missing-prefix.json" - config.write_text(json.dumps(document), encoding="utf-8") - missing_prefix = run_doctor( - project, - "openclaw", - config_path=config, - ) - self.assertEqual("unhealthy", missing_prefix["doctor_state"]) - self.assertIn( - "server_arguments_invalid", - [check["code"] for check in missing_prefix["checks"]], - ) - - entry["args"] = [ - "-I", - "-m", - "docforge.mcp_server", - "--project-root", - str(root), - "--bogus", - ] - config.write_text(json.dumps(document), encoding="utf-8") - malformed = run_doctor(project, "openclaw", config_path=config) - self.assertEqual("unhealthy", malformed["doctor_state"]) - self.assertIn( - "server_arguments_invalid", - [check["code"] for check in malformed["checks"]], - ) - - secret = "super-secret-argument" - entry["args"][-1] = secret - config.write_text(json.dumps(document), encoding="utf-8") - redacted = run_doctor(project, "openclaw", config_path=config) - self.assertEqual("unhealthy", redacted["doctor_state"]) - self.assertNotIn(secret, json.dumps(redacted, sort_keys=True)) - - def test_doctor_rejects_ambiguous_duplicate_roots_and_malformed_controls( - self, - ) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - project = Project.open(root) - ProjectIndex(project).build() - generated = generate_client_configuration(project, "openclaw") - document = json.loads(generated["artifact"]["content"]) - name, first = next(iter(document["mcp"]["servers"].items())) - duplicate = json.loads(json.dumps(first)) - root_value = duplicate["args"].index("--project-root") + 1 - duplicate["args"][root_value] = "/first/does/not/match" - for index in range(64): - duplicate["args"].extend(("--project-root", f"/duplicate/does/not/match/{index}")) - duplicate["args"].extend(("--project-root", str(root))) - document["mcp"]["servers"][f"{name}-duplicate"] = duplicate - config = parent / "openclaw.json" - config.write_text(json.dumps(document), encoding="utf-8") - ambiguous = run_doctor(project, "openclaw", config_path=config) - self.assertEqual("unhealthy", ambiguous["doctor_state"]) - self.assertIn( - "client_entry_ambiguous", - [check["code"] for check in ambiguous["checks"]], - ) - - document["mcp"]["servers"].pop(f"{name}-duplicate") - first["supportsParallelToolCalls"] = "not-a-bool" - config.write_text(json.dumps(document), encoding="utf-8") - malformed_parallel = run_doctor( - project, - "openclaw", - config_path=config, - ) - self.assertEqual("unhealthy", malformed_parallel["doctor_state"]) - - first["supportsParallelToolCalls"] = False - first["cwd"] = str(parent / "missing-directory") - config.write_text(json.dumps(document), encoding="utf-8") - missing_cwd = run_doctor(project, "openclaw", config_path=config) - self.assertEqual("unhealthy", missing_cwd["doctor_state"]) - self.assertIn( - "client_cwd_invalid", - [check["code"] for check in missing_cwd["checks"]], - ) - - for malformed_filter in ( - ["not-an-object"], - {"include": "not-a-list"}, - {"include": [1, 2]}, - ): - first.pop("cwd", None) - first["toolFilter"] = malformed_filter - config.write_text(json.dumps(document), encoding="utf-8") - malformed = run_doctor( - project, - "openclaw", - config_path=config, - ) - self.assertEqual("unhealthy", malformed["doctor_state"]) - self.assertIn( - "client_entry_invalid", - [check["code"] for check in malformed["checks"]], - ) - - def test_doctor_bounds_parser_failures_inputs_and_check_inventory(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - project = Project.open(self.copy_fixture(parent)) - deeply_nested = parent / "deep.json" - deeply_nested.write_text("[" * 2_000 + "]" * 2_000, encoding="utf-8") - deep = run_doctor( - project, - "openclaw", - config_path=deeply_nested, - ) - self.assertEqual("unhealthy", deep["doctor_state"]) - - huge_integer = parent / "integer.json" - huge_integer.write_text( - '{"mcp":{"servers":{}},"value":' + "9" * 100_000 + "}", - encoding="utf-8", - ) - integer = run_doctor( - project, - "openclaw", - config_path=huge_integer, - ) - self.assertEqual("unhealthy", integer["doctor_state"]) - - malformed_openclaw = parent / "malformed-openclaw.json" - malformed_openclaw.write_text('{"mcp":"wrong"}', encoding="utf-8") - malformed_driver = run_doctor( - project, - "openclaw", - config_path=malformed_openclaw, - ) - self.assertEqual("unhealthy", malformed_driver["doctor_state"]) - self.assertIn( - "client_config_invalid", - [check["code"] for check in malformed_driver["checks"]], - ) - - oversized_name = run_doctor( - project, - "openclaw", - config_path=parent / "missing.json", - server_name="x" * 5_000, - ) - Draft202012Validator(DOCTOR_SCHEMA).validate(oversized_name) - self.assertEqual("unhealthy", oversized_name["doctor_state"]) - self.assertEqual( - 14, - len(oversized_name["checks"]), - ) - self.assertEqual( - len(oversized_name["checks"]), - len({check["check_id"] for check in oversized_name["checks"]}), - ) - oversized_path = run_doctor( - project, - "openclaw", - config_path=Path("/" + "x" * 40_000), - ) - Draft202012Validator(DOCTOR_SCHEMA).validate(oversized_path) - self.assertEqual("unhealthy", oversized_path["doctor_state"]) - self.assertLessEqual( - len(json.dumps(oversized_path, sort_keys=True).encode("utf-8")), - 32_768, - ) - - def test_doctor_rejects_descriptor_config_and_index_parent_swaps(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - project = Project.open(root) - descriptor_parent = root / ".docforge" - moved_descriptor = root / ".docforge-old" - original_project_open = os.open - descriptor_swapped = False - - def swap_descriptor_parent( - path: object, - flags: int, - *args: object, - **kwargs: object, - ) -> int: - nonlocal descriptor_swapped - descriptor = original_project_open(path, flags, *args, **kwargs) - if ( - not descriptor_swapped - and Path(path) == descriptor_parent - and flags & os.O_DIRECTORY - ): - descriptor_parent.rename(moved_descriptor) - descriptor_parent.mkdir() - shutil.copy2( - moved_descriptor / "project.toml", - descriptor_parent / "project.toml", - ) - descriptor_swapped = True - return descriptor - - with ( - mock.patch( - "docforge.project.os.open", - side_effect=swap_descriptor_parent, - ), - self.assertRaises(DocForgeError) as changed, - ): - run_doctor( - project, - "codex", - config_path=parent / "missing.toml", - ) - self.assertEqual("project_descriptor_changed", changed.exception.code) - - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - project = Project.open(root) - ProjectIndex(project).build() - config_parent = parent / "client" - config_parent.mkdir() - config = config_parent / "config.toml" - generated = generate_client_configuration(project, "codex", output=config) - moved_config = parent / "client-old" - original_doctor_open = os.open - config_swapped = False - - def swap_config_parent( - path: object, - flags: int, - *args: object, - **kwargs: object, - ) -> int: - nonlocal config_swapped - descriptor = original_doctor_open(path, flags, *args, **kwargs) - if not config_swapped and Path(path) == config_parent and flags & os.O_DIRECTORY: - config_parent.rename(moved_config) - config_parent.mkdir() - (config_parent / "config.toml").write_text("", encoding="utf-8") - config_swapped = True - return descriptor - - with mock.patch( - "docforge.doctor.os.open", - side_effect=swap_config_parent, - ): - changed_config = run_doctor( - project, - "codex", - config_path=config, - server_name=generated["server_name"], - ) - self.assertEqual("unhealthy", changed_config["doctor_state"]) - self.assertIn( - "client_config_changed", - [check["code"] for check in changed_config["checks"]], - ) - - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - project = Project.open(root) - ProjectIndex(project).build() - config = parent / "config.toml" - generated = generate_client_configuration(project, "codex", output=config) - index_parent = project.descriptor.index_path.parent - moved_index_parent = parent / "cache-old" - original_doctor_open = os.open - index_swapped = False - - def swap_index_parent( - path: object, - flags: int, - *args: object, - **kwargs: object, - ) -> int: - nonlocal index_swapped - descriptor = original_doctor_open(path, flags, *args, **kwargs) - if not index_swapped and Path(path) == index_parent and flags & os.O_DIRECTORY: - index_parent.rename(moved_index_parent) - index_parent.mkdir() - index_swapped = True - return descriptor - - with mock.patch( - "docforge.doctor.os.open", - side_effect=swap_index_parent, - ): - unsafe_index = run_doctor( - project, - "codex", - config_path=config, - server_name=generated["server_name"], - ) - self.assertEqual("degraded", unsafe_index["doctor_state"]) - self.assertIn( - "index_unsafe", - [check["code"] for check in unsafe_index["checks"]], - ) - - def test_doctor_rechecks_the_exact_client_file_before_return(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - project = Project.open(root) - ProjectIndex(project).build() - config = parent / "config.toml" - generated = generate_client_configuration(project, "codex", output=config) - raw = config.read_bytes() - with mock.patch( - "docforge.doctor._read_stable_regular", - side_effect=[ - (raw, (1, 1, len(raw), 1, 1)), - (raw, (1, 2, len(raw), 1, 1)), - ], - ): - result = run_doctor( - project, - "codex", - config_path=config, - server_name=generated["server_name"], - ) - self.assertEqual("unhealthy", result["doctor_state"]) - self.assertIn( - "client_config_changed", - [check["code"] for check in result["checks"]], - ) - - def test_descriptor_currency_and_custom_adapter_generation_fail_closed(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - descriptor = root / ".docforge" / "project.toml" - descriptor.write_text( - descriptor.read_text(encoding="utf-8") + "\n", - encoding="utf-8", - ) - with self.assertRaises(DocForgeError) as changed: - run_doctor(project, "codex", config_path=root / "missing.toml") - self.assertEqual("source_changed", changed.exception.code) - - current = Project.open(root) - current.descriptor = replace(current.descriptor, adapter="custom-adapter") - with self.assertRaises(DocForgeError) as custom: - generate_client_configuration(current, "codex") - self.assertEqual( - "client_configuration_unavailable", - custom.exception.code, - ) - - def test_diagnostic_cli_results_validate_their_dedicated_schemas(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - root = self.copy_fixture(parent) - preview_output = io.StringIO() - with contextlib.redirect_stdout(preview_output): - preview_code = main( - [ - "--diagnostics", - "configure", - "codex", - "--project", - str(root), - ] - ) - self.assertEqual(0, preview_code) - preview = json.loads(preview_output.getvalue()) - Draft202012Validator(CONFIGURATION_SCHEMA).validate(preview) - self.assertEqual("cli.configure", preview["diagnostics"]["operation"]) - - unhealthy_output = io.StringIO() - with contextlib.redirect_stdout(unhealthy_output): - unhealthy_code = main( - [ - "--diagnostics", - "doctor", - "--client", - "codex", - "--project", - str(root), - "--config", - str(parent / "missing.toml"), - ] - ) - self.assertEqual(2, unhealthy_code) - unhealthy = json.loads(unhealthy_output.getvalue()) - Draft202012Validator(DOCTOR_SCHEMA).validate(unhealthy) - self.assertEqual("cli.doctor", unhealthy["diagnostics"]["operation"]) - - def test_configuration_schema_rejects_cross_field_drift(self) -> None: - with tempfile.TemporaryDirectory() as directory: - project = Project.open(self.copy_fixture(Path(directory))) - result = generate_client_configuration(project, "codex") - validator = Draft202012Validator(CONFIGURATION_SCHEMA) - drifted = json.loads(json.dumps(result)) - drifted["artifact"]["format"] = "openclaw-json-fragment-v1" - self.assertTrue(list(validator.iter_errors(drifted))) - drifted = json.loads(json.dumps(result)) - drifted["effective_policy"] = {} - self.assertTrue(list(validator.iter_errors(drifted))) - drifted = json.loads(json.dumps(result)) - drifted["action"] = "write" - self.assertTrue(list(validator.iter_errors(drifted))) - drifted = json.loads(json.dumps(result)) - drifted["artifact"]["durability"] = "unconfirmed" - self.assertTrue(list(validator.iter_errors(drifted))) - drifted = json.loads(json.dumps(result)) - drifted["binding"]["capability_mode"] = "proposal" - self.assertTrue(list(validator.iter_errors(drifted))) - drifted = json.loads(json.dumps(result)) - drifted["binding"]["adapter_policy"] = { - "schema_version": 1, - "mode": "preserve-no-ast", - "adapter_evolution": "preserve", - "ast_forbidden": True, - "logic_indexing": "off", - "blocked_tools": ["docforge_get_logic"], - "instruction": "Preserve adapter behavior without AST or Logic publication.", - } - self.assertTrue(list(validator.iter_errors(drifted))) - drifted = json.loads(json.dumps(result)) - drifted["binding"]["render_policy"]["manual"] = "disabled" - self.assertTrue(list(validator.iter_errors(drifted))) - drifted = json.loads(json.dumps(result)) - drifted["binding"]["args"][6] = "proposal" - self.assertTrue(list(validator.iter_errors(drifted))) - no_ast = generate_client_configuration(project, "codex", no_ast=True) - drifted = json.loads(json.dumps(no_ast)) - drifted["binding"]["args"].remove("--no-ast") - self.assertTrue(list(validator.iter_errors(drifted))) - drifted = json.loads(json.dumps(result)) - drifted["effective_policy"]["prohibitions"][2] = "made_up_permission" - self.assertTrue(list(validator.iter_errors(drifted))) - with self.assertRaises(AssertionError): - _validate_configuration_result(drifted) - drifted = json.loads(json.dumps(no_ast)) - drifted["binding"]["adapter_policy"]["instruction"] = "AST use is allowed." - self.assertTrue(list(validator.iter_errors(drifted))) - with self.assertRaises(AssertionError): - _validate_configuration_result(drifted) - - def test_doctor_schema_rejects_duplicate_inventory_and_inflated_summary(self) -> None: - with tempfile.TemporaryDirectory() as directory: - parent = Path(directory) - project = Project.open(self.copy_fixture(parent)) - result = run_doctor( - project, - "codex", - config_path=parent / "missing.toml", - ) - validator = Draft202012Validator(DOCTOR_SCHEMA) - duplicate = json.loads(json.dumps(result)) - duplicate["checks"][1]["check_id"] = duplicate["checks"][0]["check_id"] - self.assertTrue(list(validator.iter_errors(duplicate))) - inflated = json.loads(json.dumps(result)) - inflated["summary"]["failed"] = 99 - self.assertTrue(list(validator.iter_errors(inflated))) - - -class GeneratedClientLaunchTests(unittest.IsolatedAsyncioTestCase): - async def test_generated_command_starts_real_project_bound_stdio_server(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = Path(directory) / "alpha" - shutil.copytree(FIXTURES / "alpha", root) - generated = generate_client_configuration(Project.open(root), "codex") - binding = generated["binding"] - parameters = StdioServerParameters( - command=binding["command"], - args=binding["args"], - ) - async with ( - stdio_client(parameters) as (read_stream, write_stream), - ClientSession(read_stream, write_stream) as session, - ): - await session.initialize() - tools = await session.list_tools() - bootstrap = await session.call_tool("docforge_bootstrap", {}) - self.assertEqual(list(READ_TOOLS), [tool.name for tool in tools.tools]) - self.assertEqual("ok", bootstrap.structuredContent["status"]) - self.assertEqual( - "read", bootstrap.structuredContent["effective_policy"]["capability_mode"] - ) - - -if __name__ == "__main__": - unittest.main() diff --git a/tests/test_generation_diff.py b/tests/test_generation_diff.py deleted file mode 100644 index bb67129..0000000 --- a/tests/test_generation_diff.py +++ /dev/null @@ -1,993 +0,0 @@ -from __future__ import annotations - -import contextlib -import io -import json -import shutil -import sqlite3 -import tempfile -import unittest -from collections.abc import Callable -from dataclasses import replace -from pathlib import Path -from unittest import mock - -from jsonschema import Draft202012Validator -from mcp.shared.memory import create_connected_server_and_client_session - -import docforge.generation_diff as generation_diff_module -from docforge.cli import main -from docforge.errors import DocForgeError -from docforge.generation_diff import ( - GenerationDiffDraft, - finalize_generation_diff, - generation_diff_path, - publish_generation_diff, - validate_generation_diff_receipt, -) -from docforge.index import ProjectIndex -from docforge.mcp_server import DocForgeService, create_server -from docforge.models import ProjectDescriptor, ProjectSnapshot -from docforge.pagination import canonical_hash -from docforge.project import Project, project_root_fingerprint -from docforge.telemetry import request - -ROOT = Path(__file__).resolve().parents[1] -FIXTURES = ROOT / "tests" / "fixtures" -GENERATION_DIFF_SCHEMA = json.loads( - (ROOT / "schemas" / "generation-diff.schema.json").read_text(encoding="utf-8") -) -GENERATION_DIFF_PAGE_SCHEMA = json.loads( - (ROOT / "schemas" / "generation-diff-page.schema.json").read_text(encoding="utf-8") -) -RESULT_SCHEMA = json.loads((ROOT / "schemas" / "result.schema.json").read_text(encoding="utf-8")) - - -class StaticProject: - """Small legacy one-method project used to prove load-free status behavior.""" - - def __init__(self, snapshot: ProjectSnapshot) -> None: - self.descriptor: ProjectDescriptor = snapshot.descriptor - self.snapshot = snapshot - self.load_calls = 0 - - def load(self) -> ProjectSnapshot: - self.load_calls += 1 - return self.snapshot - - def canonical_source_paths(self) -> tuple[Path, ...]: - return () - - def validate_proposal( - self, - base: ProjectSnapshot, - projected: ProjectSnapshot, - operations: tuple[object, ...], - ) -> None: - del base, projected, operations - - -class GenerationDiffTests(unittest.TestCase): - def copy_fixture(self, destination: Path) -> Path: - root = destination / "alpha" - shutil.copytree(FIXTURES / "alpha", root) - shutil.rmtree(root / ".docforge" / "cache", ignore_errors=True) - return root - - @staticmethod - def change_graph(root: Path, suffix: str = "changed") -> None: - foundation = root / "docs" / "content" / "foundation.md" - foundation.write_text( - foundation.read_text(encoding="utf-8").replace( - "Defines which Alpha files own documentation facts.", - f"Defines which Alpha files own documentation facts. {suffix}", - ), - encoding="utf-8", - ) - proof = root / "docs" / "content" / "proof.toml" - proof.write_text( - proof.read_text(encoding="utf-8").replace( - 'proves = ["guide.workflow"]', - 'proves = ["guide.foundation"]', - ), - encoding="utf-8", - ) - - @staticmethod - def rehash_receipt(receipt: dict[str, object]) -> dict[str, object]: - items = receipt["items"] - if not isinstance(items, list): - raise AssertionError("receipt items are not a list") - for item in items: - if not isinstance(item, dict): - raise AssertionError("receipt item is not an object") - item["item_hash"] = canonical_hash( - {key: value for key, value in item.items() if key != "item_hash"} - ) - retained_hash = canonical_hash([item["item_hash"] for item in items]) - receipt["retained_collection_hash"] = retained_hash - if not receipt["details_truncated"]: - receipt["full_collection_hash"] = retained_hash - receipt["receipt_hash"] = canonical_hash( - {key: value for key, value in receipt.items() if key != "receipt_hash"} - ) - return receipt - - def transition_receipt(self, root: Path) -> tuple[Project, ProjectIndex, dict[str, object]]: - project = Project.open(root) - index = ProjectIndex(project) - index.build() - self.change_graph(root) - index.build() - receipt = json.loads(generation_diff_path(project.descriptor).read_text(encoding="utf-8")) - return project, index, receipt - - def test_first_build_and_exact_transition_are_schema_valid(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - index = ProjectIndex(project) - - first = index.build() - first_receipt = json.loads( - generation_diff_path(project.descriptor).read_text(encoding="utf-8") - ) - Draft202012Validator(GENERATION_DIFF_SCHEMA).validate(first_receipt) - self.assertTrue( - validate_generation_diff_receipt( - first_receipt, - descriptor=project.descriptor, - ) - ) - self.assertEqual("baseline", first_receipt["kind"]) - self.assertEqual("no_predecessor", first_receipt["reason"]) - self.assertEqual(0, first_receipt["full_item_count"]) - self.assertEqual("ok", first["status"]) - - self.change_graph(root) - second = index.build() - receipt = json.loads( - generation_diff_path(project.descriptor).read_text(encoding="utf-8") - ) - Draft202012Validator(GENERATION_DIFF_SCHEMA).validate(receipt) - self.assertEqual("transition", receipt["kind"]) - self.assertEqual( - { - "nodes_added": 0, - "nodes_removed": 0, - "nodes_changed": 2, - "edges_added": 1, - "edges_removed": 1, - "total_changes": 4, - }, - receipt["summary"], - ) - self.assertEqual( - [ - ("edge", "added"), - ("edge", "removed"), - ("node", "changed"), - ("node", "changed"), - ], - [(item["entity"], item["change"]) for item in receipt["items"]], - ) - self.assertNotIn("logic_hash", json.dumps(receipt, sort_keys=True).casefold()) - self.assertEqual("ok", second["status"]) - - status = ProjectIndex(project).generation_diff() - self.assertEqual("current", status["receipt_state"]) - self.assertEqual(receipt["receipt_hash"], status["generation_diff"]["receipt_hash"]) - - def test_same_generation_reindex_preserves_latest_meaningful_transition(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - self.change_graph(root) - index.build() - before = json.loads( - generation_diff_path(project.descriptor).read_text(encoding="utf-8") - ) - - index.build() - after = json.loads(generation_diff_path(project.descriptor).read_text(encoding="utf-8")) - - self.assertEqual("transition", after["kind"]) - self.assertEqual(before["from_generation"], after["from_generation"]) - self.assertEqual(before["to_generation"], after["to_generation"]) - self.assertEqual(before["summary"], after["summary"]) - self.assertEqual(before["full_collection_hash"], after["full_collection_hash"]) - self.assertNotEqual(before["index_signature"], after["index_signature"]) - self.assertEqual( - ["generation-diff.json"], - [path.name for path in project.descriptor.cache_root.glob("*generation-diff*")], - ) - - def test_receipt_item_and_byte_limits_preserve_exact_summary_hashes(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - descriptor = Project.open(root).descriptor - generation = { - "revision": "unversioned", - "source_hash": "1" * 64, - "node_count": 0, - "node_hash": "2" * 64, - "edge_count": 0, - "edge_hash": "3" * 64, - "index_schema_version": 3, - } - items: list[dict[str, object]] = [] - for index in range(1_002): - item: dict[str, object] = { - "entity": "edge", - "change": "added", - "source_id": f"source-{index:04d}", - "relation": "relates_to", - "target_id": f"target-{index:04d}", - } - item["item_hash"] = canonical_hash(item) - items.append(item) - fields: dict[str, object] = { - "schema_version": 1, - "diff_semantics_version": 1, - "project_id": descriptor.project_id, - "project_root_fingerprint": project_root_fingerprint(descriptor.root), - "adapter": descriptor.adapter, - "kind": "transition", - "reason": None, - "from_generation": generation, - "to_generation": {**generation, "source_hash": "4" * 64}, - "summary": { - "nodes_added": 0, - "nodes_removed": 0, - "nodes_changed": 0, - "edges_added": len(items), - "edges_removed": 0, - "total_changes": len(items), - }, - "full_item_count": len(items), - "full_collection_hash": canonical_hash([item["item_hash"] for item in items]), - } - receipt = finalize_generation_diff( - GenerationDiffDraft(fields=fields, items=tuple(items)), - signature=(1, 2, 3, 4, 5), - ) - Draft202012Validator(GENERATION_DIFF_SCHEMA).validate(receipt) - self.assertEqual(1_002, receipt["full_item_count"]) - self.assertEqual(1_000, receipt["retained_item_count"]) - self.assertEqual("receipt_item_limit", receipt["truncation_reason"]) - self.assertNotEqual( - receipt["full_collection_hash"], - receipt["retained_collection_hash"], - ) - - huge = dict(items[0]) - huge["source_id"] = "source-" + ("x" * 1_100_000) - huge["item_hash"] = canonical_hash( - {key: value for key, value in huge.items() if key != "item_hash"} - ) - huge_fields = { - **fields, - "summary": { - **fields["summary"], - "edges_added": 1, - "total_changes": 1, - }, - "full_item_count": 1, - "full_collection_hash": canonical_hash([huge["item_hash"]]), - } - byte_limited = finalize_generation_diff( - GenerationDiffDraft(fields=huge_fields, items=(huge,)), - signature=(1, 2, 3, 4, 5), - ) - self.assertEqual(0, byte_limited["retained_item_count"]) - self.assertEqual("receipt_byte_limit", byte_limited["truncation_reason"]) - - def test_receipt_runtime_and_schema_reject_malformed_semantics(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project, _, receipt = self.transition_receipt(root) - validator = Draft202012Validator(GENERATION_DIFF_SCHEMA) - - malformed_fields = json.loads(json.dumps(receipt)) - changed = next( - item - for item in malformed_fields["items"] - if item["entity"] == "node" and item["change"] == "changed" - ) - changed["changed_fields"] = 1 - self.rehash_receipt(malformed_fields) - self.assertFalse( - validate_generation_diff_receipt( - malformed_fields, - descriptor=project.descriptor, - ) - ) - self.assertFalse(validator.is_valid(malformed_fields)) - - duplicate_fields = json.loads(json.dumps(receipt)) - changed = next( - item - for item in duplicate_fields["items"] - if item["entity"] == "node" and item["change"] == "changed" - ) - changed["changed_fields"] = ["content", "content"] - self.rehash_receipt(duplicate_fields) - self.assertFalse(validate_generation_diff_receipt(duplicate_fields)) - self.assertFalse(validator.is_valid(duplicate_fields)) - - impossible_added = json.loads(json.dumps(receipt)) - changed = next( - item - for item in impossible_added["items"] - if item["entity"] == "node" and item["change"] == "changed" - ) - changed["change"] = "added" - changed["changed_fields"] = [] - self.rehash_receipt(impossible_added) - self.assertFalse(validate_generation_diff_receipt(impossible_added)) - self.assertFalse(validator.is_valid(impossible_added)) - - reordered = json.loads(json.dumps(receipt)) - reordered["items"].reverse() - self.rehash_receipt(reordered) - self.assertFalse(validate_generation_diff_receipt(reordered)) - - same_generation = json.loads(json.dumps(receipt)) - same_generation["from_generation"] = same_generation["to_generation"] - self.rehash_receipt(same_generation) - self.assertFalse(validate_generation_diff_receipt(same_generation)) - - impossible_truncation = json.loads(json.dumps(receipt)) - impossible_truncation["items"].pop() - impossible_truncation["retained_item_count"] = len(impossible_truncation["items"]) - impossible_truncation["details_truncated"] = True - impossible_truncation["truncation_reason"] = "receipt_item_limit" - self.rehash_receipt(impossible_truncation) - self.assertFalse(validate_generation_diff_receipt(impossible_truncation)) - self.assertFalse(validator.is_valid(impossible_truncation)) - - baseline_with_changes = json.loads(json.dumps(receipt)) - baseline_with_changes["kind"] = "baseline" - baseline_with_changes["reason"] = "no_meaningful_transition" - baseline_with_changes["from_generation"] = None - self.rehash_receipt(baseline_with_changes) - self.assertFalse(validate_generation_diff_receipt(baseline_with_changes)) - self.assertFalse(validator.is_valid(baseline_with_changes)) - - def test_generation_collision_and_nondeterministic_build_fail_precommit(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - base = Project.open(root).load() - static = StaticProject(base) - index = ProjectIndex(static) - index.build() - original_index = index.path.read_bytes() - original_receipt = generation_diff_path(static.descriptor).read_bytes() - - changed_node = replace( - base.nodes[0], - title="Different graph under the same generation", - ) - static.snapshot = replace( - base, - nodes=(changed_node, *base.nodes[1:]), - ) - with self.assertRaises(DocForgeError) as collision: - index.build() - self.assertEqual("generation_collision", collision.exception.code) - self.assertEqual(original_index, index.path.read_bytes()) - self.assertEqual( - original_receipt, - generation_diff_path(static.descriptor).read_bytes(), - ) - - calls = 0 - - def unstable() -> ProjectSnapshot: - nonlocal calls - calls += 1 - return ( - base - if calls == 1 - else replace( - base, - nodes=(changed_node, *base.nodes[1:]), - ) - ) - - static.snapshot = base - with ( - mock.patch.object(static, "load", side_effect=unstable), - self.assertRaises(DocForgeError) as changed, - ): - ProjectIndex(static).build() - self.assertEqual("source_changed", changed.exception.code) - - def test_post_commit_receipt_failures_report_degraded_success(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - self.change_graph(root) - - with ( - mock.patch.object( - ProjectIndex, - "_write_attestation", - side_effect=OSError("attestation failed"), - ), - mock.patch.object( - Project, - "record_generation", - side_effect=OSError("generation failed"), - ), - mock.patch( - "docforge.index.publish_generation_diff", - side_effect=OSError("diff failed"), - ), - ): - result = index.build() - - self.assertEqual("ok", result["status"]) - self.assertEqual("degraded", result["publication"]["state"]) - self.assertEqual("published", result["publication"]["index"]) - self.assertEqual( - {"attestation", "source_generation", "generation_diff"}, - {error["stage"] for error in result["publication"]["errors"]}, - ) - with contextlib.closing(__import__("sqlite3").connect(index.path)) as connection: - metadata = dict(connection.execute("SELECT key, value FROM metadata")) - self.assertEqual(project.load().source_hash, metadata["source_hash"]) - - def test_predecessor_requires_attestation_and_valid_generation_identity(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - index.attestation_path.unlink() - self.change_graph(root) - result = index.build() - receipt = json.loads( - generation_diff_path(project.descriptor).read_text(encoding="utf-8") - ) - self.assertEqual("ok", result["status"]) - self.assertEqual("baseline", receipt["kind"]) - self.assertEqual("predecessor_unattested", receipt["reason"]) - self.assertTrue(validate_generation_diff_receipt(receipt)) - - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - with contextlib.closing(sqlite3.connect(index.path)) as connection: - connection.execute("UPDATE metadata SET value = 'bad' WHERE key = 'source_hash'") - connection.commit() - index._write_attestation() - self.change_graph(root) - index.build() - receipt = json.loads( - generation_diff_path(project.descriptor).read_text(encoding="utf-8") - ) - self.assertEqual("baseline", receipt["kind"]) - self.assertEqual("predecessor_corrupt", receipt["reason"]) - self.assertTrue(validate_generation_diff_receipt(receipt)) - - def test_sqlite_sidecars_refuse_precommit_publication(self) -> None: - for suffix in ("-wal", "-journal", "-shm"): - with self.subTest(suffix=suffix), tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - original_index = index.path.read_bytes() - original_receipt = generation_diff_path(project.descriptor).read_bytes() - Path(f"{index.path}{suffix}").write_bytes(b"unproven-sidecar") - self.change_graph(root) - with self.assertRaises(DocForgeError) as blocked: - index.build() - self.assertEqual("index_busy", blocked.exception.code) - self.assertEqual(original_index, index.path.read_bytes()) - self.assertEqual( - original_receipt, - generation_diff_path(project.descriptor).read_bytes(), - ) - - def test_live_wal_state_cannot_bypass_main_index_identity(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - original_receipt = generation_diff_path(project.descriptor).read_bytes() - connection = sqlite3.connect(index.path) - try: - self.assertEqual( - "wal", - connection.execute("PRAGMA journal_mode=WAL").fetchone()[0], - ) - connection.execute( - "UPDATE metadata SET value = ? WHERE key = 'source_hash'", - ("f" * 64,), - ) - connection.commit() - self.assertTrue(Path(f"{index.path}-wal").exists()) - main_file_after_wal = index.path.read_bytes() - with self.assertRaises(DocForgeError) as blocked: - index.build() - self.assertEqual("index_busy", blocked.exception.code) - self.assertEqual(main_file_after_wal, index.path.read_bytes()) - self.assertEqual( - original_receipt, - generation_diff_path(project.descriptor).read_bytes(), - ) - finally: - connection.close() - - def test_sidecar_or_source_change_during_diff_preparation_aborts_precommit(self) -> None: - for mutation in ("source", "sidecar"): - with self.subTest(mutation=mutation), tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - original_index = index.path.read_bytes() - original_receipt = generation_diff_path(project.descriptor).read_bytes() - self.change_graph(root) - from docforge import index as index_module - - real_prepare = index_module.prepare_generation_diff - - def mutate_after_prepare( - *args: object, - _prepare: Callable[..., object] = real_prepare, - _mutation: str = mutation, - _root: Path = root, - _index: ProjectIndex = index, - **kwargs: object, - ) -> object: - draft = _prepare(*args, **kwargs) - if _mutation == "source": - source = _root / "docs" / "content" / "workflow.md" - source.write_text( - source.read_text(encoding="utf-8") + "\nConcurrent change.\n", - encoding="utf-8", - ) - else: - Path(f"{_index.path}-wal").write_bytes(b"appeared") - return draft - - with ( - mock.patch( - "docforge.index.prepare_generation_diff", - side_effect=mutate_after_prepare, - ), - self.assertRaises(DocForgeError) as blocked, - ): - index.build() - self.assertEqual( - "source_changed" if mutation == "source" else "index_busy", - blocked.exception.code, - ) - self.assertEqual(original_index, index.path.read_bytes()) - self.assertEqual( - original_receipt, - generation_diff_path(project.descriptor).read_bytes(), - ) - - def test_cache_root_symlink_cannot_redirect_receipt_publication(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - receipt = json.loads( - generation_diff_path(project.descriptor).read_text(encoding="utf-8") - ) - cache_root = project.descriptor.cache_root - preserved = cache_root.with_name("preserved-cache") - outside = root / "outside-cache" - outside.mkdir() - cache_root.rename(preserved) - cache_root.symlink_to(outside, target_is_directory=True) - try: - with self.assertRaises(DocForgeError) as blocked: - publish_generation_diff(project.descriptor, receipt) - self.assertEqual("path_escape", blocked.exception.code) - self.assertFalse((outside / "generation-diff.json").exists()) - finally: - cache_root.unlink() - preserved.rename(cache_root) - - def test_post_commit_identity_and_durability_failures_are_degraded(self) -> None: - cases = ( - ("_fsync_cache_directory", OSError("fsync failed"), "index_directory_sync"), - ( - "_published_index_signature", - OSError("signature failed"), - "index_identity", - ), - ) - for method, failure, stage in cases: - with self.subTest(method=method), tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - self.change_graph(root) - with mock.patch.object(ProjectIndex, method, side_effect=failure): - result = index.build() - self.assertEqual("ok", result["status"]) - self.assertEqual("degraded", result["publication"]["state"]) - self.assertIn( - stage, - {error["stage"] for error in result["publication"]["errors"]}, - ) - with contextlib.closing(sqlite3.connect(index.path)) as connection: - metadata = dict(connection.execute("SELECT key, value FROM metadata")) - self.assertEqual(project.load().source_hash, metadata["source_hash"]) - - def test_post_commit_receipts_fail_independently(self) -> None: - cases = ( - ("attestation", "docforge.index.ProjectIndex._write_attestation"), - ("source_generation", "docforge.project.Project.record_generation"), - ("generation_diff", "docforge.index.publish_generation_diff"), - ) - for failed_receipt, target in cases: - with self.subTest(receipt=failed_receipt), tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - self.change_graph(root) - with mock.patch(target, side_effect=OSError("failed independently")): - result = index.build() - receipts = result["publication"]["receipts"] - self.assertEqual("unavailable", receipts[failed_receipt]["state"]) - for name in {"attestation", "source_generation", "generation_diff"} - { - failed_receipt - }: - self.assertEqual("published", receipts[name]["state"]) - - def test_read_is_bounded_read_only_and_legacy_projects_do_not_load(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - receipt_path = generation_diff_path(project.descriptor) - original_receipt = json.loads(receipt_path.read_text(encoding="utf-8")) - before = { - path: path.stat().st_mtime_ns - for path in project.descriptor.cache_root.iterdir() - if path.is_file() - } - with ( - request("test", enabled=True) as collector, - mock.patch.object(project, "load", side_effect=AssertionError("loaded")), - mock.patch.object(index, "check", side_effect=AssertionError("checked")), - mock.patch.object(index, "build", side_effect=AssertionError("built")), - mock.patch.object(index, "synchronize", side_effect=AssertionError("synced")), - ): - result = index.generation_diff() - self.assertEqual("current", result["receipt_state"]) - self.assertIsNotNone(collector) - diagnostics = collector.as_dict(outcome="ok") - for counter in ( - "project_loads", - "source_files_parsed", - "adapter_projection_loads", - "adapter_source_extractions", - "index_checks", - "index_synchronizations", - "index_builds", - ): - self.assertEqual(0, diagnostics["counters"][counter]) - self.assertGreaterEqual(diagnostics["counters"]["source_generation_checks"], 2) - self.assertEqual( - before, - { - path: path.stat().st_mtime_ns - for path in project.descriptor.cache_root.iterdir() - if path.is_file() - }, - ) - with mock.patch.object( - generation_diff_module, - "validate_generation_diff_receipt", - wraps=generation_diff_module.validate_generation_diff_receipt, - ) as validated: - self.assertEqual("current", index.generation_diff()["receipt_state"]) - self.assertEqual(1, validated.call_count) - - receipt_path.write_text("{broken", encoding="utf-8") - corrupt_before = receipt_path.read_bytes() - corrupt = index.generation_diff() - self.assertEqual("unverified", corrupt["receipt_state"]) - self.assertEqual("corrupt_receipt", corrupt["receipt_reason"]) - self.assertEqual(corrupt_before, receipt_path.read_bytes()) - - foreign = {**original_receipt, "project_id": "foreign-project"} - foreign["receipt_hash"] = canonical_hash( - {key: value for key, value in foreign.items() if key != "receipt_hash"} - ) - receipt_path.write_text( - json.dumps(foreign, sort_keys=True, indent=2) + "\n", - encoding="utf-8", - ) - foreign_result = index.generation_diff() - self.assertEqual("unverified", foreign_result["receipt_state"]) - self.assertEqual("foreign_receipt", foreign_result["receipt_reason"]) - - outside = root / "foreign-generation-diff.json" - outside.write_text( - json.dumps(original_receipt, sort_keys=True, indent=2) + "\n", - encoding="utf-8", - ) - receipt_path.unlink() - receipt_path.symlink_to(outside) - unsafe = index.generation_diff() - self.assertEqual("unsafe", unsafe["receipt_state"]) - self.assertEqual("unsafe_receipt", unsafe["receipt_reason"]) - - snapshot = project.load() - legacy = StaticProject(snapshot) - calls = legacy.load_calls - unknown = ProjectIndex(legacy).generation_diff() - self.assertEqual("unknown", unknown["receipt_state"]) - self.assertEqual(calls, legacy.load_calls) - - def test_cli_generation_diff_is_additive_and_paged(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - self.change_graph(root) - index.build() - output = io.StringIO() - with contextlib.redirect_stdout(output): - code = main( - [ - "--project-root", - str(root), - "generation-diff", - "--limit", - "1", - ] - ) - result = json.loads(output.getvalue()) - self.assertEqual(0, code) - Draft202012Validator(RESULT_SCHEMA).validate(result) - Draft202012Validator(GENERATION_DIFF_PAGE_SCHEMA).validate( - { - "generation_diff": result["generation_diff"], - "pagination": result["pagination"], - } - ) - self.assertEqual("generation-diff.items", result["pagination"]["kind"]) - self.assertEqual(1, result["pagination"]["returned_count"]) - self.assertTrue(result["pagination"]["has_more"]) - self.assertNotIn("next_cursor", result) - self.assertNotIn("pagination", result["generation_diff"]) - self.assertEqual( - { - "page_schema_version", - "receipt_header", - "items", - "omissions", - "page_hash", - }, - set(result["generation_diff"]), - ) - page = result["generation_diff"] - self.assertIn("stored_receipt_hash", page["receipt_header"]) - self.assertNotIn("receipt_hash", page["receipt_header"]) - self.assertEqual( - page["page_hash"], - canonical_hash( - { - "page_schema_version": 1, - "receipt_state": result["receipt_state"], - "receipt_header": page["receipt_header"], - "pagination": result["pagination"], - "items": page["items"], - "omissions": page["omissions"], - } - ), - ) - self.assertEqual( - result["pagination"]["returned_count"], - len(page["items"]) + len(page["omissions"]), - ) - self.assertEqual( - result["pagination"]["total_count"], - page["receipt_header"]["retained_item_count"], - ) - - malformed_page = json.loads(json.dumps(result)) - stored_receipt = json.loads( - generation_diff_path(project.descriptor).read_text(encoding="utf-8") - ) - node = next(item for item in stored_receipt["items"] if item["entity"] == "node") - node["change"] = "added" - malformed_page["generation_diff"]["items"] = [node] - self.assertFalse( - Draft202012Validator(GENERATION_DIFF_PAGE_SCHEMA).is_valid( - { - "generation_diff": malformed_page["generation_diff"], - "pagination": malformed_page["pagination"], - } - ) - ) - - malformed_header = json.loads(json.dumps(result)) - header = malformed_header["generation_diff"]["receipt_header"] - header["kind"] = "baseline" - header["reason"] = "no_meaningful_transition" - header["from_generation"] = None - self.assertFalse( - Draft202012Validator(GENERATION_DIFF_PAGE_SCHEMA).is_valid( - { - "generation_diff": malformed_header["generation_diff"], - "pagination": malformed_header["pagination"], - } - ) - ) - - contradictory_pagination = json.loads(json.dumps(result)) - contradictory_pagination["pagination"]["has_more"] = False - self.assertFalse( - Draft202012Validator(GENERATION_DIFF_PAGE_SCHEMA).is_valid( - { - "generation_diff": contradictory_pagination["generation_diff"], - "pagination": contradictory_pagination["pagination"], - } - ) - ) - missing_cursor = json.loads(json.dumps(result)) - missing_cursor["pagination"]["next_cursor"] = None - self.assertFalse( - Draft202012Validator(GENERATION_DIFF_PAGE_SCHEMA).is_valid( - { - "generation_diff": missing_cursor["generation_diff"], - "pagination": missing_cursor["pagination"], - } - ) - ) - - -class GenerationDiffMcpTests(unittest.IsolatedAsyncioTestCase): - def copy_fixture(self, destination: Path) -> Path: - root = destination / "alpha" - shutil.copytree(FIXTURES / "alpha", root) - shutil.rmtree(root / ".docforge" / "cache", ignore_errors=True) - return root - - async def test_mcp_surface_paginates_and_cursors_bind_the_receipt(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - GenerationDiffTests.change_graph(root) - index.build() - - async with create_connected_server_and_client_session( - create_server(root), - raise_exceptions=True, - ) as session: - tools = {tool.name: tool for tool in (await session.list_tools()).tools} - schema = tools["docforge_get_generation_diff"].inputSchema - self.assertEqual({"limit", "cursor"}, set(schema["properties"])) - self.assertEqual([], schema.get("required", [])) - - first = await session.call_tool( - "docforge_get_generation_diff", - {"limit": 1}, - ) - first_result = first.structuredContent - Draft202012Validator(RESULT_SCHEMA).validate(first_result) - Draft202012Validator(GENERATION_DIFF_PAGE_SCHEMA).validate( - { - "generation_diff": first_result["generation_diff"], - "pagination": first_result["pagination"], - } - ) - self.assertEqual("current", first_result["receipt_state"]) - self.assertEqual(1, first_result["pagination"]["returned_count"]) - cursor = first_result["pagination"]["next_cursor"] - self.assertIsInstance(cursor, str) - self.assertLess(len(cursor), 1_000) - self.assertNotIn("next_cursor", first_result) - - second = await session.call_tool( - "docforge_get_generation_diff", - {"limit": 2, "cursor": cursor}, - ) - self.assertEqual(2, second.structuredContent["pagination"]["returned_count"]) - - foundation = root / "docs" / "content" / "foundation.md" - foundation.write_text( - foundation.read_text(encoding="utf-8") + "\nAnother transition.\n", - encoding="utf-8", - ) - ProjectIndex(Project.open(root)).build() - stale = await session.call_tool( - "docforge_get_generation_diff", - {"limit": 1, "cursor": cursor}, - ) - self.assertEqual("stale_cursor", stale.structuredContent["error"]["code"]) - Draft202012Validator(RESULT_SCHEMA).validate(stale.structuredContent) - - def test_service_diagnostics_drop_before_primary_page(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - descriptor_path = root / ".docforge" / "project.toml" - descriptor_path.write_text( - descriptor_path.read_text(encoding="utf-8").replace( - "max_results = 20", - "max_results = 20\nmax_tool_output_chars = 2100", - ), - encoding="utf-8", - ) - project = Project.open(root) - ProjectIndex(project).build() - service = DocForgeService( - project, - diagnostics=True, - capability_mode_name="read", - ) - result = service.generation_diff() - self.assertEqual("ok", result["status"]) - self.assertNotIn("diagnostics", result) - self.assertLessEqual(service._encoded_length(result), 2_100) - - def test_page_sizing_uses_logarithmic_response_encodes(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - descriptor_path = root / ".docforge" / "project.toml" - descriptor_path.write_text( - descriptor_path.read_text(encoding="utf-8").replace( - "max_results = 20", - "max_results = 1000", - ), - encoding="utf-8", - ) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - stored = index.generation_diff() - receipt = dict(stored["generation_diff"]) - receipt["items"] = [ - { - "item_hash": canonical_hash({"ordinal": ordinal}), - "payload": "x" * 500, - } - for ordinal in range(1_000) - ] - service = DocForgeService( - project, - capability_mode_name="read", - ) - with ( - mock.patch.object( - service.index, - "generation_diff", - return_value={**stored, "generation_diff": receipt}, - ), - mock.patch.object( - service, - "_encoded_length", - wraps=service._encoded_length, - ) as encoded, - ): - result = service.generation_diff(limit=1_000) - self.assertEqual("ok", result["status"]) - self.assertGreater(result["pagination"]["returned_count"], 0) - self.assertLess(result["pagination"]["returned_count"], 1_000) - self.assertLessEqual(encoded.call_count, 15) diff --git a/tests/test_mcp_server.py b/tests/test_mcp_server.py index 8c65dd2..b945b29 100644 --- a/tests/test_mcp_server.py +++ b/tests/test_mcp_server.py @@ -12,33 +12,26 @@ from contextlib import contextmanager from pathlib import Path from unittest import mock -from jsonschema import Draft202012Validator from mcp import ClientSession, StdioServerParameters from mcp.client.stdio import stdio_client from mcp.shared.memory import create_connected_server_and_client_session from docforge.changesets import ChangesetStore -from docforge.errors import DocForgeError from docforge.index import ProjectIndex from docforge.mcp_server import ( ALL_TOOLS, APPLICATION_TOOLS, CONTENT_WARNING, PROPOSAL_TOOLS, - READ_TOOLS, - SERVER_VERSION, DocForgeService, _create_bound_server, create_server, ) -from docforge.project import Project, project_root_fingerprint +from docforge.project import Project from docforge.viewer_manager import ViewerManager ROOT = Path(__file__).resolve().parents[1] FIXTURES = ROOT / "tests" / "fixtures" -CAPSULE_SCHEMA = json.loads( - (ROOT / "schemas" / "context-capsule.schema.json").read_text(encoding="utf-8") -) class DocForgeMcpTests(unittest.IsolatedAsyncioTestCase): @@ -90,8 +83,6 @@ class DocForgeMcpTests(unittest.IsolatedAsyncioTestCase): self.assertNotIn("limit", tools[name].inputSchema.get("required", [])) for name in ( "docforge_get_context", - "docforge_get_task_context", - "docforge_get_generation_diff", "docforge_list_changesets", "docforge_get_changeset", "docforge_validate_changeset", @@ -100,22 +91,6 @@ class DocForgeMcpTests(unittest.IsolatedAsyncioTestCase): for field in ("limit", "cursor"): self.assertIn(field, tools[name].inputSchema["properties"]) self.assertNotIn(field, tools[name].inputSchema.get("required", [])) - self.assertEqual( - {"task_kind", "task"}, - set(tools["docforge_get_task_context"].inputSchema["required"]), - ) - self.assertEqual( - [ - "change", - "implementation", - "failure", - "ownership", - "test", - "operation", - "release", - ], - tools["docforge_get_task_context"].inputSchema["properties"]["task_kind"]["enum"], - ) self.assertIn( "deep", tools["docforge_render_status"].inputSchema["properties"], @@ -128,43 +103,6 @@ class DocForgeMcpTests(unittest.IsolatedAsyncioTestCase): ) ) - async def test_explicit_capability_modes_preserve_surfaces_and_fail_closed(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture("alpha", Path(directory)) - ProjectIndex(Project.open(root)).build() - async with create_connected_server_and_client_session( - create_server(root, capability_mode="read"), - raise_exceptions=True, - ) as session: - read_names = tuple(tool.name for tool in (await session.list_tools()).tools) - read_bootstrap = await session.call_tool("docforge_bootstrap", {}) - - self.assertEqual(READ_TOOLS, read_names) - self.assertEqual( - "read", - read_bootstrap.structuredContent["effective_policy"]["capability_mode"], - ) - self.assertNotIn( - "docforge_register_changes", - read_bootstrap.structuredContent["recommended_workflow"], - ) - - async with create_connected_server_and_client_session( - create_server( - root, - "alpha-editor", - canonical_applier_id="alpha-editor", - capability_mode="application", - ), - raise_exceptions=True, - ) as session: - application_names = tuple(tool.name for tool in (await session.list_tools()).tools) - self.assertEqual((*ALL_TOOLS, *APPLICATION_TOOLS), application_names) - - with self.assertRaises(DocForgeError) as unavailable: - create_server(root, capability_mode="application") - self.assertEqual("capability_unavailable", unavailable.exception.code) - async def test_factory_diagnostics_are_additive_through_real_mcp(self) -> None: with tempfile.TemporaryDirectory() as directory: root = self.copy_fixture("alpha", Path(directory)) @@ -183,403 +121,6 @@ class DocForgeMcpTests(unittest.IsolatedAsyncioTestCase): self.assertEqual(0, diagnostics["counters"]["project_loads"]) self.assertEqual(0, diagnostics["counters"]["source_files_parsed"]) - async def test_task_context_is_hash_stable_paged_and_generation_bound(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture("alpha", Path(directory)) - ProjectIndex(Project.open(root)).build() - async with create_connected_server_and_client_session( - create_server(root, capability_mode="read", diagnostics=True), - raise_exceptions=True, - ) as session: - first = await session.call_tool( - "docforge_get_task_context", - { - "task_kind": "change", - "task": "Change the editing workflow", - "focus_node_id": "guide.workflow", - "limit": 1, - }, - ) - first_capsule = first.structuredContent["capsule"] - Draft202012Validator(CAPSULE_SCHEMA).validate(first_capsule) - cursor = first_capsule["pagination"]["next_cursor"] - hashes = { - first_capsule["capsule_hash"], - first_capsule["collection_hash"], - first_capsule["plan"]["plan_hash"], - } - evidence_ids = [item["node_id"] for item in first_capsule["evidence"]] - while cursor is not None: - page = await session.call_tool( - "docforge_get_task_context", - { - "task_kind": "change", - "task": "Change the editing workflow", - "focus_node_id": "guide.workflow", - "limit": 2, - "cursor": cursor, - }, - ) - capsule = page.structuredContent["capsule"] - self.assertEqual(first_capsule["capsule_hash"], capsule["capsule_hash"]) - self.assertEqual( - first_capsule["collection_hash"], - capsule["collection_hash"], - ) - self.assertEqual( - first_capsule["plan"]["plan_hash"], - capsule["plan"]["plan_hash"], - ) - evidence_ids.extend(item["node_id"] for item in capsule["evidence"]) - cursor = capsule["pagination"]["next_cursor"] - - self.assertEqual( - ["guide.workflow", "guide.foundation", "proof.validation"], - evidence_ids, - ) - self.assertEqual(3, len(hashes)) - self.assertEqual( - "mcp.task_context", - first.structuredContent["diagnostics"]["operation"], - ) - counters = first.structuredContent["diagnostics"]["counters"] - self.assertEqual(0, counters["project_loads"]) - self.assertEqual(0, counters["source_files_parsed"]) - self.assertEqual(0, counters["adapter_projection_loads"]) - self.assertEqual(0, counters["adapter_source_extractions"]) - self.assertEqual(0, counters["index_builds"]) - self.assertLessEqual( - len(json.dumps(first.structuredContent, separators=(",", ":"))), - Project.open(root).descriptor.limits.max_tool_output_chars, - ) - - for changed_arguments in ( - {"task": "A different task"}, - {"task_kind": "failure"}, - {"focus_node_id": "guide.foundation"}, - {"budget": 100}, - ): - arguments = { - "task_kind": "change", - "task": "Change the editing workflow", - "focus_node_id": "guide.workflow", - "limit": 1, - "cursor": first_capsule["pagination"]["next_cursor"], - **changed_arguments, - } - changed_cursor = await session.call_tool( - "docforge_get_task_context", - arguments, - ) - self.assertEqual( - "stale_cursor", - changed_cursor.structuredContent["error"]["code"], - ) - - different_policy = DocForgeService( - Project.open(root), - capability_mode_name="proposal", - ).task_context( - "change", - "Change the editing workflow", - focus_node_id="guide.workflow", - limit=1, - cursor=first_capsule["pagination"]["next_cursor"], - ) - self.assertEqual("stale_cursor", different_policy["error"]["code"]) - - changed = root / "docs/content/foundation.md" - changed.write_text( - changed.read_text(encoding="utf-8") + "\nNew generation.\n", - encoding="utf-8", - ) - stale = await session.call_tool( - "docforge_get_task_context", - { - "task_kind": "change", - "task": "Change the editing workflow", - "focus_node_id": "guide.workflow", - "limit": 1, - "cursor": first_capsule["pagination"]["next_cursor"], - }, - ) - self.assertEqual("error", stale.structuredContent["status"]) - self.assertEqual("stale_cursor", stale.structuredContent["error"]["code"]) - - async def test_custom_context_policy_does_not_silently_gain_task_planning(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture("alpha", Path(directory)) - project = Project.open(root) - ProjectIndex(project).build() - service = DocForgeService( - project, - context_provider=lambda index, profile, budget: { - "profile": profile, - "budget": budget, - "entries": [], - "omissions": [], - }, - capability_mode_name="read", - ) - async with create_connected_server_and_client_session( - _create_bound_server(service, read_only=True), - raise_exceptions=True, - ) as session: - bootstrap = await session.call_tool("docforge_bootstrap", {}) - with ( - mock.patch.object( - service.project, - "load", - side_effect=AssertionError("capability errors must not load"), - ), - mock.patch.object( - service.index, - "check", - side_effect=AssertionError("capability errors must not check"), - ), - mock.patch.object( - service.index, - "build", - side_effect=AssertionError("capability errors must not build"), - ), - mock.patch.object( - service.index, - "synchronize", - side_effect=AssertionError("capability errors must not synchronize"), - ), - ): - result = await session.call_tool( - "docforge_get_task_context", - { - "task_kind": "change", - "task": "Do not widen the adapter context policy", - }, - ) - self.assertEqual( - "docforge_get_context", - bootstrap.structuredContent["session_contract"]["recommended_first_operation"][ - "tool" - ], - ) - self.assertFalse(bootstrap.structuredContent["capabilities"]["task_context"]["enabled"]) - self.assertEqual("error", result.structuredContent["status"]) - self.assertEqual( - "task_context_unavailable", - result.structuredContent["error"]["code"], - ) - - def test_task_context_page_hash_binds_final_page_envelope(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture("alpha", Path(directory)) - descriptor = root / ".docforge" / "project.toml" - descriptor.write_text( - descriptor.read_text(encoding="utf-8").replace( - "max_context_tokens = 2000", - "max_context_tokens = 2000\nmax_tool_output_chars = 8000", - ), - encoding="utf-8", - ) - project = Project.open(root) - ProjectIndex(project).build() - service = DocForgeService(project, capability_mode_name="read") - one = service.task_context( - "change", - "Change the editing workflow", - focus_node_id="guide.workflow", - limit=1, - ) - two = service.task_context( - "change", - "Change the editing workflow", - focus_node_id="guide.workflow", - limit=2, - ) - one_capsule = one["capsule"] - two_capsule = two["capsule"] - self.assertEqual( - ["guide.workflow"], - [item["node_id"] for item in one_capsule["evidence"]], - ) - self.assertEqual( - ["guide.workflow"], - [item["node_id"] for item in two_capsule["evidence"]], - ) - self.assertEqual("complete", one_capsule["page_state"]) - self.assertEqual("incomplete", two_capsule["page_state"]) - self.assertNotEqual(one_capsule["page_hash"], two_capsule["page_hash"]) - self.assertNotEqual(one_capsule["pagination"], two_capsule["pagination"]) - self.assertLessEqual( - len(json.dumps(one, sort_keys=True, separators=(",", ":"))), - 8_000, - ) - self.assertLessEqual( - len(json.dumps(two, sort_keys=True, separators=(",", ":"))), - 8_000, - ) - - def test_dense_task_context_page_packing_is_logarithmic(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture("alpha", Path(directory)) - descriptor = root / ".docforge" / "project.toml" - descriptor.write_text( - descriptor.read_text(encoding="utf-8") - .replace("max_results = 20", "max_results = 1000") - .replace( - "max_context_tokens = 2000", - "max_context_tokens = 2000\nmax_tool_output_chars = 20000", - ), - encoding="utf-8", - ) - project = Project.open(root) - service = DocForgeService(project, capability_mode_name="read") - capsule = { - "schema_version": 1, - "plan": { - "effective_policy_hash": "1" * 64, - "request_hash": "2" * 64, - "plan_hash": "3" * 64, - }, - "generation": {"index_schema_version": 3}, - "evidence": [ - { - "node_id": f"node.{index:04d}", - "content": "bounded evidence " * 40, - } - for index in range(1_000) - ], - "gaps": [], - "omissions": [], - "collection_hash": "4" * 64, - "capsule_hash": "5" * 64, - "state": "complete", - "summary": {"evidence_count": 1_000}, - } - result = { - "status": "ok", - "project_id": project.descriptor.project_id, - "project_root_fingerprint": project_root_fingerprint(project.descriptor.root), - "adapter": project.descriptor.adapter, - "revision": "test-revision", - "source_hash": "6" * 64, - "capsule": capsule, - } - - with mock.patch.object( - service, - "_encoded_length", - wraps=service._encoded_length, - ) as encoded_length: - page = service._page_task_context_result( - result, - selected_limit=1_000, - cursor=None, - ) - - pagination = page["pagination"] - self.assertGreater(pagination["returned_count"], 0) - self.assertLess(pagination["returned_count"], 1_000) - returned_count = pagination["returned_count"] - self.assertEqual( - capsule["evidence"][:returned_count], - page["capsule"]["evidence"], - ) - self.assertEqual([], page["capsule"]["omissions"]) - self.assertEqual( - pagination["next_cursor"], - page["capsule"]["pagination"]["next_cursor"], - ) - self.assertEqual( - returned_count, - page["capsule"]["summary"]["page_item_count"], - ) - self.assertLessEqual(encoded_length.call_count, 11) - decorated = { - **page, - "server_version": SERVER_VERSION, - "content_warning": CONTENT_WARNING, - "staleness": "current", - } - self.assertLessEqual( - len(json.dumps(decorated, sort_keys=True, separators=(",", ":"))), - 20_000, - ) - - def test_task_context_default_page_clamps_to_small_project_limit(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture("alpha", Path(directory)) - descriptor = root / ".docforge" / "project.toml" - descriptor.write_text( - descriptor.read_text(encoding="utf-8").replace( - "max_results = 20", - "max_results = 2", - ), - encoding="utf-8", - ) - project = Project.open(root) - ProjectIndex(project).build() - result = DocForgeService( - project, - capability_mode_name="read", - ).task_context( - "change", - "Change the editing workflow", - focus_node_id="guide.workflow", - ) - self.assertEqual(2, result["pagination"]["limit"]) - self.assertLessEqual(result["pagination"]["returned_count"], 2) - - def test_oversized_task_evidence_advances_once_as_an_omission(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture("alpha", Path(directory)) - descriptor = root / ".docforge" / "project.toml" - descriptor.write_text( - descriptor.read_text(encoding="utf-8").replace( - "max_context_tokens = 2000", - "max_context_tokens = 50000\nmax_tool_output_chars = 8000", - ), - encoding="utf-8", - ) - workflow = root / "docs" / "content" / "workflow.md" - workflow.write_text( - workflow.read_text(encoding="utf-8") + "\n" + ("large evidence " * 2_000), - encoding="utf-8", - ) - project = Project.open(root) - ProjectIndex(project).build() - service = DocForgeService(project, capability_mode_name="read") - first = service.task_context( - "change", - "Change the editing workflow", - focus_node_id="guide.workflow", - budget=50_000, - limit=1, - ) - first_capsule = first["capsule"] - self.assertEqual([], first_capsule["evidence"]) - self.assertEqual("response_limit", first_capsule["omissions"][0]["code"]) - self.assertEqual("guide.workflow", first_capsule["omissions"][0]["subject"]) - self.assertEqual(1, first_capsule["pagination"]["returned_count"]) - self.assertTrue(first_capsule["pagination"]["has_more"]) - second = service.task_context( - "change", - "Change the editing workflow", - focus_node_id="guide.workflow", - budget=50_000, - limit=1, - cursor=first_capsule["pagination"]["next_cursor"], - ) - self.assertNotEqual(first_capsule["page_hash"], second["capsule"]["page_hash"]) - self.assertNotIn( - "guide.workflow", - [ - item.get("node_id", item.get("subject")) - for item in ( - *second["capsule"]["evidence"], - *second["capsule"]["omissions"], - ) - ], - ) - async def test_context_pagination_is_complete_and_stale_cursors_fail_closed(self) -> None: with tempfile.TemporaryDirectory() as directory: root = self.copy_fixture("alpha", Path(directory)) @@ -724,14 +265,6 @@ class DocForgeMcpTests(unittest.IsolatedAsyncioTestCase): logic = await session.call_tool( "docforge_get_logic", {"owner_node_id": "guide.workflow"} ) - task_context = await session.call_tool( - "docforge_get_task_context", - { - "task_kind": "change", - "task": "Change the editing workflow without AST analysis", - "focus_node_id": "guide.workflow", - }, - ) policy = bootstrap.structuredContent["adapter_policy"] self.assertEqual("preserve-no-ast", policy["mode"]) @@ -757,14 +290,6 @@ class DocForgeMcpTests(unittest.IsolatedAsyncioTestCase): "adapter_policy_forbids_logic", logic.structuredContent["error"]["code"], ) - self.assertEqual("ok", task_context.structuredContent["status"]) - self.assertNotIn( - "logic", - { - step["operation"] - for step in task_context.structuredContent["capsule"]["plan"]["steps"] - }, - ) async def test_every_read_tool_returns_scoped_structured_results(self) -> None: with tempfile.TemporaryDirectory() as directory: @@ -794,14 +319,6 @@ class DocForgeMcpTests(unittest.IsolatedAsyncioTestCase): ("docforge_visualization_status", {}), ("docforge_bootstrap", {}), ("docforge_sync", {}), - ( - "docforge_get_task_context", - { - "task_kind": "change", - "task": "Change the editing workflow", - "focus_node_id": "guide.workflow", - }, - ), ) with self.running_manager(Path(directory) / "viewer-manager.json"): service = DocForgeService(Project.open(root)) @@ -857,7 +374,6 @@ class DocForgeMcpTests(unittest.IsolatedAsyncioTestCase): context = results[9].structuredContent self.assertLessEqual(context["estimated_tokens"], 180) self.assertTrue(context["omissions"]) - self.assertEqual("complete", results[17].structuredContent["capsule"]["state"]) async def test_invalid_traversal_limit_is_a_structured_domain_error(self) -> None: with tempfile.TemporaryDirectory() as directory: diff --git a/tests/test_observability.py b/tests/test_observability.py index 7b4d143..c4cf2c6 100644 --- a/tests/test_observability.py +++ b/tests/test_observability.py @@ -34,10 +34,6 @@ from docforge.viewer_manager import ViewerManagerClient ROOT = Path(__file__).resolve().parents[1] FIXTURES = ROOT / "tests" / "fixtures" RESULT_SCHEMA = json.loads((ROOT / "schemas" / "result.schema.json").read_text()) -CLIENT_CONFIGURATION_SCHEMA = json.loads( - (ROOT / "schemas" / "client-configuration.schema.json").read_text() -) -DOCTOR_RESULT_SCHEMA = json.loads((ROOT / "schemas" / "doctor-result.schema.json").read_text()) ZERO_WORK_COUNTERS = ( "project_loads", "source_files_parsed", @@ -138,16 +134,6 @@ class TelemetryContractTests(unittest.TestCase): set(COUNTER_NAMES), set(properties["counters"]["properties"]), ) - for schema in (CLIENT_CONFIGURATION_SCHEMA, DOCTOR_RESULT_SCHEMA): - dedicated = schema["$defs"]["diagnostics"]["properties"] - self.assertEqual( - set(STAGE_NAMES), - set(dedicated["stages"]["propertyNames"]["enum"]), - ) - self.assertEqual( - set(COUNTER_NAMES), - set(dedicated["counters"]["required"]), - ) def test_thread_and_async_request_contexts_are_isolated(self) -> None: barrier = threading.Barrier(2) diff --git a/tests/test_policy.py b/tests/test_policy.py deleted file mode 100644 index 47bd1eb..0000000 --- a/tests/test_policy.py +++ /dev/null @@ -1,161 +0,0 @@ -from __future__ import annotations - -import json -import shutil -import tempfile -import unittest -from pathlib import Path -from unittest import mock - -from jsonschema import Draft202012Validator - -from docforge.application import GenericCanonicalApplier -from docforge.errors import DocForgeError -from docforge.index import ProjectIndex -from docforge.mcp_server import ( - ALL_TOOLS, - APPLICATION_TOOLS, - READ_TOOLS, - DocForgeService, -) -from docforge.policy import ( - POLICY_PRECEDENCE, - capability_mode, - compose_effective_policy, -) -from docforge.project import Project - -ROOT = Path(__file__).resolve().parents[1] -FIXTURES = ROOT / "tests" / "fixtures" -POLICY_SCHEMA = json.loads((ROOT / "schemas" / "policy.schema.json").read_text(encoding="utf-8")) - - -class EffectivePolicyTests(unittest.TestCase): - def copy_fixture(self, destination: Path) -> Path: - root = destination / "alpha" - shutil.copytree(FIXTURES / "alpha", root) - return root - - def test_policy_schema_and_legacy_adapter_projection_are_exact(self) -> None: - standard = compose_effective_policy( - selected_mode="proposal", - capability_source="factory_default", - no_ast=False, - diagnostics=False, - render_configured=True, - application_enabled=False, - ) - preserve = compose_effective_policy( - selected_mode="read", - capability_source="explicit", - no_ast=True, - diagnostics=True, - render_configured=False, - application_enabled=False, - ) - validator = Draft202012Validator(POLICY_SCHEMA) - validator.validate(standard.as_dict()) - validator.validate(preserve.as_dict()) - self.assertEqual(list(POLICY_PRECEDENCE), preserve.as_dict()["precedence"]) - self.assertEqual( - { - "mode": "standard", - "ast_analysis": "allowed", - "logic_projection": "allowed", - "incremental_extraction": "allowed", - "adapter_rewrite": "not_requested", - }, - standard.adapter_policy(), - ) - self.assertEqual("preserve-no-ast", preserve.adapter_policy()["mode"]) - self.assertEqual(["docforge_get_logic"], preserve.adapter_policy()["blocked_tools"]) - self.assertEqual("off", preserve.as_dict()["logic_indexing"]) - self.assertEqual("enabled", preserve.as_dict()["profiling"]) - - def test_invalid_or_unavailable_capability_fails_closed(self) -> None: - with self.assertRaises(DocForgeError) as invalid: - capability_mode("admin", default="read") - self.assertEqual("invalid_capability_mode", invalid.exception.code) - with self.assertRaises(DocForgeError) as unavailable: - compose_effective_policy( - selected_mode="application", - capability_source="explicit", - no_ast=False, - diagnostics=False, - render_configured=True, - application_enabled=False, - ) - self.assertEqual("capability_unavailable", unavailable.exception.code) - - def test_modes_preserve_default_surfaces_and_narrow_authority(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - - read = DocForgeService( - project, - proposal_writer="alpha-editor", - capability_mode_name="read", - ) - proposal = DocForgeService( - project, - proposal_writer="alpha-editor", - capability_mode_name="proposal", - ) - application = DocForgeService( - project, - proposal_writer="alpha-editor", - canonical_applier_id="alpha-editor", - canonical_applier=GenericCanonicalApplier(project), - capability_mode_name="application", - ) - operator = DocForgeService( - project, - proposal_writer="alpha-editor", - capability_mode_name="operator", - ) - - self.assertEqual(READ_TOOLS, read.tool_surface) - self.assertFalse(read.changesets.access()["enabled"]) - self.assertEqual(ALL_TOOLS, proposal.tool_surface) - self.assertTrue(proposal.changesets.access()["enabled"]) - self.assertFalse(proposal.application.enabled) - self.assertEqual((*ALL_TOOLS, *APPLICATION_TOOLS), application.tool_surface) - self.assertTrue(application.application.enabled) - self.assertEqual(ALL_TOOLS, operator.tool_surface) - self.assertTrue(operator.capabilities()["operator"]["enabled"]) - - def test_bootstrap_reuses_synchronized_identity_and_filters_workflow(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture(Path(directory)) - project = Project.open(root) - ProjectIndex(project).build() - read = DocForgeService(project, capability_mode_name="read") - with mock.patch.object( - project, - "load", - side_effect=AssertionError("bootstrap must not reload after synchronization"), - ): - result = read.bootstrap() - - self.assertEqual("ok", result["status"]) - self.assertEqual(1, result["session_contract"]["schema_version"]) - self.assertEqual("read", result["effective_policy"]["capability_mode"]) - self.assertEqual( - result["effective_policy"], result["session_contract"]["effective_policy"] - ) - self.assertNotIn("docforge_register_changes", result["recommended_workflow"]) - self.assertNotIn("docforge_apply_changeset", result["recommended_workflow"]) - self.assertEqual( - "docforge_get_task_context", - result["session_contract"]["recommended_first_operation"]["tool"], - ) - - proposal = DocForgeService( - project, - proposal_writer="alpha-editor", - capability_mode_name="proposal", - ).bootstrap() - self.assertIn("docforge_register_changes", proposal["recommended_workflow"]) - self.assertIn("docforge_get_changeset_diff", proposal["recommended_workflow"]) - self.assertNotIn("docforge_apply_changeset", proposal["recommended_workflow"]) diff --git a/tests/test_public_contract.py b/tests/test_public_contract.py index e153b2c..6524824 100644 --- a/tests/test_public_contract.py +++ b/tests/test_public_contract.py @@ -61,8 +61,6 @@ PUBLIC_IMPORTS = { "CanonicalApplicationService", "GenericCanonicalApplier", ), - "docforge.client_config": ("generate_client_configuration",), - "docforge.doctor": ("run_doctor",), "docforge.index": ("ProjectIndex",), "docforge.mcp_server": ( "create_project_server", @@ -79,18 +77,6 @@ PUBLIC_IMPORTS = { "ProjectService", "ProjectSnapshot", ), - "docforge.policy": ( - "EffectivePolicyV1", - "capability_mode", - "compose_effective_policy", - ), - "docforge.retrieval": ( - "ContextCapsuleV1", - "RetrievalPlanV1", - "build_retrieval_plan", - "relation_category", - "validate_retrieval_plan", - ), "docforge.render_contract": ( "GenericHtmlRenderer", "PreparedRender", @@ -110,12 +96,9 @@ EXPECTED_CLI_COMMANDS = { "backlinks", "build", "check", - "configure", "context", "dependencies", - "doctor", "filter", - "generation-diff", "impact", "info", "onboard", @@ -144,8 +127,6 @@ EXPECTED_MCP_TOOLS = { "docforge_get_changeset", "docforge_get_changeset_diff", "docforge_get_context", - "docforge_get_generation_diff", - "docforge_get_task_context", "docforge_get_contract", "docforge_get_logic", "docforge_get_node", @@ -269,14 +250,6 @@ class PublicContractTests(unittest.TestCase): result_validator = Draft202012Validator(self.schema("result.schema.json")) result_validator.validate(success) result_validator.validate(error) - generation_diff = service.generation_diff(limit=1) - result_validator.validate(generation_diff) - Draft202012Validator(self.schema("generation-diff-page.schema.json")).validate( - { - "generation_diff": generation_diff["generation_diff"], - "pagination": generation_diff["pagination"], - } - ) def test_changeset_hash_is_exact_canonical_json_sha256(self) -> None: document = { diff --git a/tests/test_retrieval.py b/tests/test_retrieval.py deleted file mode 100644 index 390d762..0000000 --- a/tests/test_retrieval.py +++ /dev/null @@ -1,464 +0,0 @@ -from __future__ import annotations - -import json -import shutil -import tempfile -import unittest -from dataclasses import replace -from pathlib import Path -from unittest import mock - -from jsonschema import Draft202012Validator - -from docforge.errors import DocForgeError -from docforge.index import ProjectIndex -from docforge.models import ProjectState -from docforge.policy import compose_effective_policy -from docforge.project import Project -from docforge.retrieval import ( - BASE_RELATION_CATEGORIES, - MAX_TASK_CANDIDATE_EDGES, - MAX_TASK_EVIDENCE, - RELATION_CATEGORIES, - TASK_KINDS, - TASK_REQUIREMENTS, - build_retrieval_plan, - relation_category, -) - -ROOT = Path(__file__).resolve().parents[1] -FIXTURES = ROOT / "tests" / "fixtures" -CAPSULE_SCHEMA = json.loads( - (ROOT / "schemas" / "context-capsule.schema.json").read_text(encoding="utf-8") -) - - -class TaskRetrievalTests(unittest.TestCase): - def copy_fixture(self, name: str, destination: Path) -> Path: - root = destination / name - shutil.copytree(FIXTURES / name, root) - return root - - @staticmethod - def effective_policy() -> dict[str, object]: - return compose_effective_policy( - selected_mode="read", - capability_source="explicit", - no_ast=False, - diagnostics=False, - render_configured=True, - application_enabled=False, - ).as_dict() - - def plan( - self, - project: Project, - *, - task_kind: str = "change", - task: str = "Change the editing workflow", - focus_node_id: str | None = "guide.workflow", - budget: int | None = None, - limit: int | None = None, - ): - return build_retrieval_plan( - project.descriptor, - task_kind=task_kind, - task=task, - focus_node_id=focus_node_id, - budget=budget, - limit=limit, - effective_policy=self.effective_policy(), - ) - - def test_every_task_plan_is_closed_deterministic_and_hash_bound(self) -> None: - with tempfile.TemporaryDirectory() as directory: - project = Project.open(self.copy_fixture("alpha", Path(directory))) - hashes = {} - relation_hashes = set() - for task_kind in TASK_KINDS: - first = self.plan(project, task_kind=task_kind) - second = self.plan(project, task_kind=task_kind) - self.assertEqual(first, second) - self.assertEqual(TASK_REQUIREMENTS[task_kind], first.category_order[:1]) - self.assertEqual( - { - *BASE_RELATION_CATEGORIES, - "unclassified", - }, - set(first.category_order), - ) - self.assertEqual(64, len(first.plan_hash)) - relation_hashes.update( - step.relation_set_hash - for step in first.steps - if step.relation_scope == "project_allowed" - ) - hashes[task_kind] = first.plan_hash - self.assertEqual(len(TASK_KINDS), len(set(hashes.values()))) - self.assertEqual(1, len(relation_hashes)) - self.assertEqual( - { - "change": ("dependency",), - "implementation": ("implementation",), - "failure": ("execution",), - "ownership": ("structure",), - "test": ("evidence",), - "operation": ("execution",), - "release": ("evidence",), - }, - TASK_REQUIREMENTS, - ) - - mapped = [ - relation for relations in RELATION_CATEGORIES.values() for relation in relations - ] - self.assertEqual(len(mapped), len(set(mapped))) - self.assertEqual( - { - "structure": ("contains", "defined_in", "defines", "owns"), - "implementation": ( - "implemented_by", - "implements", - "inherits", - "inherits_from", - ), - "dependency": ("depends_on", "imports"), - "execution": ( - "activates", - "calls", - "dispatches_to", - "launches", - ), - "data": ("reads", "writes"), - "evidence": ( - "documents", - "governs", - "proves", - "tested_by", - "verifies", - ), - "context": ("relates_to",), - "unclassified": (), - }, - RELATION_CATEGORIES, - ) - self.assertEqual("dependency", relation_category("depends_on")) - self.assertEqual("evidence", relation_category("proves")) - self.assertEqual("unclassified", relation_category("owns_database")) - self.assertEqual("unclassified", relation_category("when_true")) - - def test_exact_capsule_is_schema_valid_stable_and_explainable(self) -> None: - with tempfile.TemporaryDirectory() as directory: - project = Project.open(self.copy_fixture("alpha", Path(directory))) - index = ProjectIndex(project) - index.build() - plan = self.plan(project) - first = index.task_context(plan) - second = index.task_context(plan) - - self.assertEqual(first, second) - capsule = first["capsule"] - Draft202012Validator(CAPSULE_SCHEMA).validate(capsule) - self.assertEqual("complete", capsule["state"]) - self.assertEqual("resolved", capsule["focus"]["state"]) - self.assertEqual("guide.workflow", capsule["focus"]["node_id"]) - self.assertEqual( - ["guide.workflow", "guide.foundation", "proof.validation"], - [item["node_id"] for item in capsule["evidence"]], - ) - dependency = capsule["evidence"][1]["relationship_path"][0] - self.assertEqual("depends_on", dependency["relation"]) - self.assertEqual("dependency", dependency["category"]) - self.assertEqual("outgoing", dependency["direction"]) - workflow_reasons = capsule["evidence"][0]["relationship_reasons"] - self.assertIn( - ("depends_on", "outgoing"), - { - (relationship["relation"], relationship["direction"]) - for relationship in workflow_reasons - }, - ) - self.assertIn( - ("proves", "incoming"), - { - (relationship["relation"], relationship["direction"]) - for relationship in workflow_reasons - }, - ) - foundation_reason = capsule["evidence"][1]["relationship_reasons"][0] - self.assertEqual("depends_on", foundation_reason["relation"]) - self.assertEqual("incoming", foundation_reason["direction"]) - self.assertNotIn( - "no_selected_evidence", - [gap["code"] for gap in capsule["gaps"]], - ) - self.assertEqual(64, len(capsule["collection_hash"])) - self.assertEqual(64, len(capsule["capsule_hash"])) - - def test_gaps_distinguish_undeclared_complete_and_incomplete_checks(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture("alpha", Path(directory)) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - - undeclared = index.task_context(self.plan(project, task_kind="implementation"))[ - "capsule" - ] - self.assertIn( - "category_not_declared", - [gap["code"] for gap in undeclared["gaps"]], - ) - - descriptor = root / ".docforge" / "project.toml" - descriptor.write_text( - descriptor.read_text(encoding="utf-8").replace( - '"returns_to"]', - '"returns_to", "calls"]', - ), - encoding="utf-8", - ) - declared_project = Project.open(root) - declared_index = ProjectIndex(declared_project) - declared_index.build() - missing = declared_index.task_context( - self.plan( - declared_project, - task_kind="failure", - focus_node_id="guide.foundation", - ) - )["capsule"] - self.assertIn( - "no_selected_evidence", - [gap["code"] for gap in missing["gaps"]], - ) - - incomplete = declared_index.task_context(self.plan(declared_project, budget=1))[ - "capsule" - ] - self.assertEqual("incomplete", incomplete["state"]) - self.assertTrue(incomplete["omissions"]) - self.assertIn( - "evidence_incomplete", - [gap["code"] for gap in incomplete["gaps"]], - ) - self.assertNotIn( - "no_selected_evidence", - [gap["code"] for gap in incomplete["gaps"]], - ) - - def test_unclassified_relation_is_preserved_without_guessed_semantics(self) -> None: - with tempfile.TemporaryDirectory() as directory: - project = Project.open(self.copy_fixture("awesome-ski-game", Path(directory))) - index = ProjectIndex(project) - index.build() - plan = build_retrieval_plan( - project.descriptor, - task_kind="change", - task="Change the first descent session", - focus_node_id="session.first-descent", - budget=None, - limit=None, - effective_policy=self.effective_policy(), - ) - capsule = index.task_context(plan)["capsule"] - self.assertIn("informs", capsule["summary"]["unclassified_relations"]) - relation = next( - relationship - for item in capsule["evidence"] - for relationship in item["relationship_reasons"] - if relationship["relation"] == "informs" - ) - self.assertEqual("unclassified", relation["category"]) - self.assertIn( - "unclassified_relation", - [gap["code"] for gap in capsule["gaps"]], - ) - - def test_plan_is_compact_at_large_valid_relation_scale_and_limits_are_fixed(self) -> None: - with tempfile.TemporaryDirectory() as directory: - project = Project.open(self.copy_fixture("alpha", Path(directory))) - relation_names = tuple(f"relation-{position:05d}" for position in range(33_005)) - descriptor = replace( - project.descriptor, - allowed_relations=relation_names, - limits=replace( - project.descriptor.limits, - max_results=2**63, - ), - ) - plan = build_retrieval_plan( - descriptor, - task_kind="change", - task="Exercise a very large valid relation policy", - focus_node_id="guide.workflow", - budget=None, - limit=None, - effective_policy=self.effective_policy(), - ) - self.assertLessEqual(plan.max_evidence, MAX_TASK_EVIDENCE) - self.assertLessEqual(plan.max_candidate_edges, MAX_TASK_CANDIDATE_EDGES) - traversal = [step for step in plan.steps if step.operation in {"outgoing", "incoming"}] - self.assertEqual(2, len(traversal)) - self.assertTrue(all(step.relation_scope == "project_allowed" for step in traversal)) - self.assertTrue(all(len(step.relation_set_hash or "") == 64 for step in traversal)) - self.assertLess(len(json.dumps(plan.as_dict())), 5_000) - - def test_executor_rejects_tampered_public_plan_objects(self) -> None: - with tempfile.TemporaryDirectory() as directory: - project = Project.open(self.copy_fixture("alpha", Path(directory))) - index = ProjectIndex(project) - index.build() - plan = self.plan(project) - tampered = ( - replace(plan, steps=plan.steps[:-1]), - replace(plan, requirements=()), - replace(plan, task_query="Different task"), - replace(plan, category_order=tuple(reversed(plan.category_order))), - replace(plan, max_evidence=plan.max_evidence + 1), - replace(plan, effective_policy_hash="0" * 64), - replace(plan, plan_hash="0" * 64), - ) - for candidate in tampered: - with self.subTest(candidate=candidate): - with self.assertRaises(DocForgeError) as invalid: - index.task_context(candidate) - self.assertEqual("invalid_retrieval_plan", invalid.exception.code) - - def test_work_and_unclassified_limits_are_explicit_and_bounded(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture("alpha", Path(directory)) - relation_names = [f"relation-{position:03d}" for position in range(450)] - descriptor = root / ".docforge" / "project.toml" - raw_descriptor = descriptor.read_text(encoding="utf-8") - raw_relations = ", ".join( - json.dumps(relation) - for relation in ( - "depends_on", - "proves", - "supersedes", - "relates_to", - "returns_to", - *relation_names, - ) - ) - descriptor.write_text( - raw_descriptor.replace( - '"depends_on", "proves", "supersedes", "relates_to", "returns_to"', - raw_relations, - ), - encoding="utf-8", - ) - workflow = root / "docs" / "content" / "workflow.md" - raw_workflow = workflow.read_text(encoding="utf-8") - relationships = "".join( - f'{relation} = ["guide.foundation"]\n' for relation in relation_names - ) - workflow.write_text( - raw_workflow.replace("+++\n\nEditors", f"{relationships}+++\n\nEditors", 1), - encoding="utf-8", - ) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - capsule = index.task_context(self.plan(project))["capsule"] - Draft202012Validator(CAPSULE_SCHEMA).validate(capsule) - omission_codes = {omission["code"] for omission in capsule["omissions"]} - self.assertIn("edge_examination_limit", omission_codes) - self.assertIn("unclassified_relation_limit", omission_codes) - self.assertEqual( - project.descriptor.limits.max_results, - len(capsule["summary"]["unclassified_relations"]), - ) - self.assertLessEqual( - capsule["summary"]["examined_edge_count"], - MAX_TASK_CANDIDATE_EDGES, - ) - - def test_lexical_focus_blocks_missing_and_ambiguous_selection(self) -> None: - with tempfile.TemporaryDirectory() as directory: - root = self.copy_fixture("alpha", Path(directory)) - project = Project.open(root) - index = ProjectIndex(project) - index.build() - missing = index.task_context( - self.plan( - project, - task="Words absent from every indexed node", - focus_node_id=None, - ) - )["capsule"] - self.assertEqual("blocked", missing["state"]) - self.assertEqual("not_found", missing["focus"]["state"]) - self.assertEqual("focus_not_found", missing["gaps"][0]["code"]) - - duplicate = root / "docs" / "content" / "workflow-copy.md" - duplicate.write_text( - (root / "docs" / "content" / "workflow.md") - .read_text(encoding="utf-8") - .replace('id = "guide.workflow"', 'id = "guide.workflow-copy"'), - encoding="utf-8", - ) - duplicate_project = Project.open(root) - duplicate_index = ProjectIndex(duplicate_project) - duplicate_index.build() - ambiguous = duplicate_index.task_context( - self.plan( - duplicate_project, - task="Editing workflow", - focus_node_id=None, - ) - )["capsule"] - self.assertEqual("blocked", ambiguous["state"]) - self.assertEqual("ambiguous", ambiguous["focus"]["state"]) - self.assertEqual("focus_ambiguous", ambiguous["gaps"][0]["code"]) - - def test_plan_rejects_unbounded_or_untyped_inputs(self) -> None: - with tempfile.TemporaryDirectory() as directory: - project = Project.open(self.copy_fixture("alpha", Path(directory))) - cases = ( - ({"task_kind": "unknown"}, "invalid_task_kind"), - ({"task": " "}, "invalid_task_focus"), - ({"focus_node_id": ""}, "invalid_task_focus"), - ({"budget": True}, "invalid_budget"), - ({"budget": 0}, "invalid_budget"), - ({"limit": True}, "invalid_limit"), - ({"limit": project.descriptor.limits.max_results + 1}, "invalid_limit"), - ) - for overrides, code in cases: - arguments = { - "task_kind": "change", - "task": "Change the editing workflow", - "focus_node_id": "guide.workflow", - "budget": None, - "limit": None, - **overrides, - } - with self.subTest(arguments=overrides): - with self.assertRaises(DocForgeError) as invalid: - self.plan(project, **arguments) - self.assertEqual(code, invalid.exception.code) - - def test_final_generation_change_rejects_the_whole_capsule(self) -> None: - with tempfile.TemporaryDirectory() as directory: - project = Project.open(self.copy_fixture("alpha", Path(directory))) - index = ProjectIndex(project) - built = index.build() - current = ProjectState( - revision=built["revision"], - source_hash=built["source_hash"], - ) - changed = ProjectState( - revision=current.revision, - source_hash="0" * 64, - ) - with ( - mock.patch.object( - project, - "incremental_state", - side_effect=(current, changed), - ), - self.assertRaises(DocForgeError) as stale, - ): - index.task_context(self.plan(project)) - self.assertEqual("source_changed", stale.exception.code) diff --git a/tools/milestone2_benchmark.py b/tools/milestone2_benchmark.py deleted file mode 100644 index 25ecd64..0000000 --- a/tools/milestone2_benchmark.py +++ /dev/null @@ -1,1014 +0,0 @@ -"""Milestone 2 agent-retrieval and client-integration benchmark gates.""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import platform -import resource -import subprocess -import sys -import tempfile -import time -from collections.abc import Callable, Mapping -from pathlib import Path -from typing import cast - -from milestone0_baseline import ( - measure_operation, - synthetic_node_id, - write_synthetic_project, -) - -from docforge.client_config import generate_client_configuration -from docforge.doctor import run_doctor -from docforge.index import ProjectIndex -from docforge.mcp_server import DocForgeService -from docforge.pagination import canonical_hash -from docforge.project import Project -from docforge.retrieval import MAX_TASK_EVIDENCE, build_retrieval_plan -from docforge.telemetry import COUNTER_NAMES, request - -ROOT = Path(__file__).resolve().parents[1] -ZERO_WORK_COUNTERS = ( - "project_loads", - "source_files_parsed", - "source_bytes_parsed", - "adapter_projection_loads", - "adapter_source_extractions", - "index_synchronizations", - "index_builds", - "render_prepare_calls", - "render_output_bytes_built", - "render_output_bytes_hashed", - "viewer_manager_requests", -) - - -def _parser() -> argparse.ArgumentParser: - parser = argparse.ArgumentParser( - description="Gate DocForge2 Milestone 2 on disposable agent workflows." - ) - parser.add_argument("--nodes", type=int, default=1000) - parser.add_argument("--samples", type=int, default=10) - parser.add_argument("--output", type=Path) - parser.add_argument("--memory-probe", action="store_true", help=argparse.SUPPRESS) - return parser - - -def _git(command: list[str]) -> str: - return subprocess.run( - ["git", *command], - cwd=ROOT, - check=True, - capture_output=True, - text=True, - ).stdout.strip() - - -def _compact_size(value: object) -> int: - return len( - json.dumps(value, sort_keys=True, separators=(",", ":"), default=str).encode("utf-8") - ) - - -def _prepare_fixture(root: Path, node_count: int) -> None: - write_synthetic_project(root, node_count) - descriptor = root / ".docforge" / "project.toml" - descriptor.write_text( - descriptor.read_text(encoding="utf-8") - .replace("max_results = 100", "max_results = 1000") - .replace("max_tool_output_chars = 5000000", "max_tool_output_chars = 200000"), - encoding="utf-8", - ) - focus = synthetic_node_id(0) - for index in range(1, node_count): - path = root / "docs" / "content" / f"node-{index:04d}.md" - raw = path.read_text(encoding="utf-8") - previous = synthetic_node_id(index - 1) - path.write_text( - raw.replace( - f'depends_on = ["{previous}"]', - f'depends_on = ["{focus}"]', - ), - encoding="utf-8", - ) - - -def _tree_hash(root: Path) -> str: - digest = hashlib.sha256() - for path in sorted(root.rglob("*")): - if not path.is_file(): - continue - digest.update(path.relative_to(root).as_posix().encode("utf-8")) - digest.update(b"\0") - digest.update(path.read_bytes()) - digest.update(b"\0") - return digest.hexdigest() - - -def _assert_zero_work(diagnostics: Mapping[str, object]) -> None: - counters_value = diagnostics.get("counters") - if not isinstance(counters_value, Mapping): - raise RuntimeError("Measured operation did not expose telemetry counters") - counters = cast(Mapping[str, object], counters_value) - for counter in ZERO_WORK_COUNTERS: - if counters.get(counter) != 0: - raise RuntimeError(f"Milestone 2 operation performed forbidden work: {counter}") - - -def _result_diagnostics( - result: Mapping[str, object], - *, - expected_counters: Mapping[str, int], -) -> Mapping[str, object]: - diagnostics_value = result.get("diagnostics") - if not isinstance(diagnostics_value, Mapping): - raise RuntimeError("Measured MCP result did not include diagnostics") - diagnostics = cast(Mapping[str, object], diagnostics_value) - counters = cast(Mapping[str, object], diagnostics["counters"]) - for counter in ZERO_WORK_COUNTERS: - expected = expected_counters.get(counter, 0) - if counters.get(counter) != expected: - raise RuntimeError( - f"Measured MCP result expected {counter}={expected}, " - f"received {counters.get(counter)!r}" - ) - for counter, expected in expected_counters.items(): - if counters.get(counter) != expected: - raise RuntimeError( - f"Measured MCP result expected {counter}={expected}, " - f"received {counters.get(counter)!r}" - ) - return diagnostics - - -def _maximum_page_validator( - result: Mapping[str, object], - *, - expected_counters: Mapping[str, int], - diagnostics_dropped: list[bool], -) -> None: - if isinstance(result.get("diagnostics"), Mapping): - _result_diagnostics(result, expected_counters=expected_counters) - diagnostics_dropped.append(False) - return - if _compact_size(result) <= 190_000: - raise RuntimeError( - "Maximum-page diagnostics were absent before the primary result approached " - "the response budget" - ) - diagnostics_dropped.append(True) - - -def _measure( - operation: Callable[[], dict[str, object]], - *, - samples: int, - p95_limit_ms: float, - response_limit_bytes: int, - validator: Callable[[dict[str, object]], object] | None = None, -) -> tuple[dict[str, object], dict[str, object]]: - results: list[dict[str, object]] = [] - diagnostics_records: list[Mapping[str, object]] = [] - - def validated_operation() -> dict[str, object]: - result = operation() - if result.get("status") != "ok": - raise RuntimeError("Measured operation did not succeed") - response_bytes = _compact_size(result) - if response_bytes > response_limit_bytes: - raise RuntimeError( - f"Milestone 2 response {response_bytes} exceeds {response_limit_bytes} bytes" - ) - if validator is not None: - validator(result) - diagnostics_value = result.get("diagnostics", result.get("_benchmark_diagnostics")) - if isinstance(diagnostics_value, Mapping): - diagnostics_records.append(cast(Mapping[str, object], diagnostics_value)) - results.append(result) - return result - - measurement, last_value = measure_operation(validated_operation, samples=samples) - if not isinstance(last_value, Mapping): - raise RuntimeError("Measured operation returned a non-object result") - last = dict(cast(Mapping[str, object], last_value)) - p95 = cast(float, measurement["p95_ms"]) - if p95 > p95_limit_ms: - raise RuntimeError(f"Milestone 2 operation p95 {p95:.3f} ms exceeds {p95_limit_ms:.3f} ms") - counter_ranges: dict[str, dict[str, int]] = {} - if diagnostics_records: - for counter in COUNTER_NAMES: - values = [ - cast( - int, - cast(Mapping[str, object], record["counters"])[counter], - ) - for record in diagnostics_records - ] - counter_ranges[counter] = { - "minimum": min(values), - "maximum": max(values), - } - return ( - { - **measurement, - "p95_limit_ms": p95_limit_ms, - "response_limit_bytes": response_limit_bytes, - "validated_invocations": len(results), - "maximum_response_bytes": max(_compact_size(result) for result in results), - **({"counter_ranges": counter_ranges} if counter_ranges else {}), - }, - last, - ) - - -def _profiled( - operation: Callable[[], dict[str, object]], -) -> dict[str, object]: - with request("benchmark.m2", enabled=True) as collector: - result = operation() - if collector is None: - raise RuntimeError("Milestone 2 profiling collector was not created") - diagnostics = collector.as_dict(outcome="ok") - _assert_zero_work(diagnostics) - result["_benchmark_diagnostics"] = diagnostics - return result - - -def _page_summary( - pages: list[dict[str, object]], - *, - started_ns: int, -) -> dict[str, object]: - sizes = [_compact_size(page) for page in pages] - paginations = [cast(dict[str, object], page["pagination"]) for page in pages] - result: dict[str, object] = { - "status": "ok", - "page_count": len(pages), - "maximum_page_bytes": max(sizes), - "aggregate_page_bytes": sum(sizes), - "elapsed_ms": round((time.perf_counter_ns() - started_ns) / 1_000_000, 3), - "maximum_cursor_bytes": max( - ( - len(cast(str, page["next_cursor"]).encode("utf-8")) - for page in paginations - if page["next_cursor"] is not None - ), - default=0, - ), - } - if cast(int, result["maximum_cursor_bytes"]) > 4_096: - raise RuntimeError("Milestone 2 cursor exceeded 4,096 bytes") - return result - - -def _task_oracle( - service: DocForgeService, - focus: str, -) -> dict[str, object]: - plan = build_retrieval_plan( - service.project.descriptor, - task_kind="change", - task="Change the central synthetic workflow", - focus_node_id=focus, - budget=None, - limit=min( - service.project.descriptor.limits.max_results, - MAX_TASK_EVIDENCE, - ), - effective_policy=service.policy.as_dict(), - ) - result = service.index.task_context(plan) - capsule = cast(dict[str, object], result["capsule"]) - generation = cast(Mapping[str, object], capsule["generation"]) - plan_payload = cast(Mapping[str, object], capsule["plan"]) - evidence = cast(list[dict[str, object]], capsule["evidence"]) - gaps = cast(list[dict[str, object]], capsule["gaps"]) - omissions = cast(list[dict[str, object]], capsule["omissions"]) - evidence_hashes = [cast(str, item["evidence_hash"]) for item in evidence] - collection_hash = canonical_hash( - { - "generation": dict(generation), - "plan_hash": plan_payload["plan_hash"], - "evidence": evidence_hashes, - "gaps": gaps, - "omissions": omissions, - } - ) - if capsule["collection_hash"] != collection_hash: - raise RuntimeError("Task-context full oracle has an invalid collection hash") - return { - "generation": dict(generation), - "plan_hash": plan_payload["plan_hash"], - "evidence": evidence, - "evidence_hashes": evidence_hashes, - "gaps": gaps, - "omissions": omissions, - "collection_hash": collection_hash, - } - - -def _task_traversal( - service: DocForgeService, - focus: str, - *, - node_count: int, - oracle: Mapping[str, object], -) -> dict[str, object]: - started = time.perf_counter_ns() - oracle_generation = cast(Mapping[str, object], oracle["generation"]) - oracle_plan_hash = cast(str, oracle["plan_hash"]) - oracle_evidence = cast(list[dict[str, object]], oracle["evidence"]) - oracle_evidence_hashes = cast(list[str], oracle["evidence_hashes"]) - oracle_gaps = cast(list[dict[str, object]], oracle["gaps"]) - oracle_omissions = cast(list[dict[str, object]], oracle["omissions"]) - oracle_collection_hash = cast(str, oracle["collection_hash"]) - oracle_evidence_by_node = {cast(str, item["node_id"]): item for item in oracle_evidence} - pages: list[dict[str, object]] = [] - cursor: str | None = None - capsule_hash: str | None = None - collection_hash: str | None = None - plan_hash: str | None = None - returned = 0 - seen_cursors: set[str] = set() - evidence_hashes: list[str] = [] - evidence_nodes: list[str] = [] - page_omissions: list[dict[str, object]] = [] - omitted_nodes: list[str] = [] - reconstructed_evidence_hashes: list[str] = [] - reconstructed_omissions: list[dict[str, object]] = [] - diagnostics_records: list[Mapping[str, object]] = [] - first_generation: Mapping[str, object] | None = None - first_gaps: list[object] | None = None - while True: - page = service.task_context( - "change", - "Change the central synthetic workflow", - focus_node_id=focus, - limit=100, - cursor=cursor, - ) - if page.get("status") != "ok": - raise RuntimeError("Task-context page failed") - diagnostics_records.append( - _result_diagnostics( - page, - expected_counters={ - "index_checks": 1, - "index_synchronizations": 0, - "source_generation_checks": 2, - }, - ) - ) - if _compact_size(page) > 200_000: - raise RuntimeError("Task-context page exceeded the configured response budget") - capsule = cast(dict[str, object], page["capsule"]) - plan = cast(dict[str, object], capsule["plan"]) - current_capsule_hash = cast(str, capsule["capsule_hash"]) - current_collection_hash = cast(str, capsule["collection_hash"]) - current_plan_hash = cast(str, plan["plan_hash"]) - generation = cast(Mapping[str, object], capsule["generation"]) - gaps = cast(list[object], capsule["gaps"]) - if first_generation is None: - first_generation = generation - first_gaps = gaps - elif generation != first_generation or gaps != first_gaps: - raise RuntimeError("Task-context invariant payload changed during traversal") - if capsule_hash is None: - capsule_hash = current_capsule_hash - collection_hash = current_collection_hash - plan_hash = current_plan_hash - elif ( - current_capsule_hash != capsule_hash - or current_collection_hash != collection_hash - or current_plan_hash != plan_hash - ): - raise RuntimeError("Task-context page binding changed during traversal") - pagination = cast(dict[str, object], page["pagination"]) - current_evidence = cast(list[dict[str, object]], capsule["evidence"]) - current_omissions = cast(list[dict[str, object]], capsule["omissions"]) - if pagination["returned_count"] != len(current_evidence) + len(current_omissions): - raise RuntimeError("Task-context page count does not match its items") - for item in current_evidence: - node_id = cast(str, item["node_id"]) - if oracle_evidence_by_node.get(node_id) != item: - raise RuntimeError("Task-context page evidence drifted from the full oracle") - evidence_hash = cast(str, item["evidence_hash"]) - evidence_hashes.append(evidence_hash) - reconstructed_evidence_hashes.append(evidence_hash) - evidence_nodes.append(node_id) - for omission in current_omissions: - subject = omission.get("subject") - if not isinstance(subject, str) or omission.get("code") not in { - "response_limit", - "token_budget", - }: - raise RuntimeError("Task-context omission has unexpected semantics") - if omission["code"] == "response_limit": - replaced = oracle_evidence_by_node.get(subject) - if replaced is None or omission.get("detail_hash") != canonical_hash(replaced): - raise RuntimeError( - "Task-context response-limit omission does not attest its oracle item" - ) - reconstructed_evidence_hashes.append(cast(str, replaced["evidence_hash"])) - else: - reconstructed_omissions.append(omission) - omitted_nodes.append(subject) - page_omissions.extend(current_omissions) - returned += cast(int, pagination["returned_count"]) - pages.append(page) - next_cursor = pagination["next_cursor"] - if next_cursor is None: - if returned != pagination["total_count"]: - raise RuntimeError("Task-context traversal did not reconstruct every item") - break - cursor = cast(str, next_cursor) - if cursor in seen_cursors: - raise RuntimeError("Task-context pagination repeated a cursor") - seen_cursors.add(cursor) - if len(evidence_hashes) != len(set(evidence_hashes)): - raise RuntimeError("Task-context traversal repeated evidence") - if len(evidence_nodes) != len(set(evidence_nodes)): - raise RuntimeError("Task-context traversal repeated node evidence") - if len(omitted_nodes) != len(set(omitted_nodes)): - raise RuntimeError("Task-context traversal repeated an omitted subject") - expected_nodes = {synthetic_node_id(index) for index in range(node_count)} - reconstructed_nodes = evidence_nodes + omitted_nodes - if ( - returned != node_count - or len(reconstructed_nodes) != node_count - or len(reconstructed_nodes) != len(set(reconstructed_nodes)) - or set(reconstructed_nodes) != expected_nodes - ): - raise RuntimeError("Task-context traversal did not reconstruct every synthetic candidate") - assert first_generation is not None - assert first_gaps is not None - if ( - collection_hash != oracle_collection_hash - or dict(first_generation) != dict(oracle_generation) - or plan_hash != oracle_plan_hash - or first_gaps != oracle_gaps - or reconstructed_evidence_hashes != oracle_evidence_hashes - or reconstructed_omissions != oracle_omissions - or collection_hash - != canonical_hash( - { - "generation": dict(first_generation), - "plan_hash": plan_hash, - "evidence": reconstructed_evidence_hashes, - "gaps": first_gaps, - "omissions": reconstructed_omissions, - } - ) - ): - raise RuntimeError("Task-context traversal did not reconstruct its full-oracle binding") - summary = _page_summary(pages, started_ns=started) - if cast(float, summary["elapsed_ms"]) > 2_500: - raise RuntimeError("Complete task-context traversal exceeded 2,500 ms") - summary.update( - { - "capsule_hash": capsule_hash, - "collection_hash": collection_hash, - "plan_hash": plan_hash, - "item_count": returned, - "evidence_count": len(evidence_nodes), - "omission_count": len(omitted_nodes), - "ordered_evidence_hash": canonical_hash(reconstructed_evidence_hashes), - "ordered_candidate_hash": canonical_hash(reconstructed_nodes), - "collection_hash_reconstructed": True, - "counter_ranges": { - counter: { - "minimum": min( - cast(int, cast(Mapping[str, object], item["counters"])[counter]) - for item in diagnostics_records - ), - "maximum": max( - cast(int, cast(Mapping[str, object], item["counters"])[counter]) - for item in diagnostics_records - ), - } - for counter in COUNTER_NAMES - }, - } - ) - return summary - - -def _generation_traversal( - service: DocForgeService, - *, - node_count: int, -) -> dict[str, object]: - started = time.perf_counter_ns() - pages: list[dict[str, object]] = [] - cursor: str | None = None - receipt_hash: str | None = None - returned = 0 - seen_cursors: set[str] = set() - item_hashes: list[str] = [] - node_ids: list[str] = [] - diagnostics_records: list[Mapping[str, object]] = [] - retained_collection_hash: str | None = None - while True: - page = service.generation_diff(limit=100, cursor=cursor) - if page.get("status") != "ok": - raise RuntimeError("Generation-diff page failed") - diagnostics_records.append( - _result_diagnostics( - page, - expected_counters={ - "index_checks": 0, - "index_synchronizations": 0, - "source_generation_checks": 2, - }, - ) - ) - if _compact_size(page) > 200_000: - raise RuntimeError("Generation-diff page exceeded the configured response budget") - generation_diff = cast(dict[str, object], page["generation_diff"]) - header = cast(dict[str, object], generation_diff["receipt_header"]) - current_receipt_hash = cast(str, header["stored_receipt_hash"]) - current_retained_hash = cast(str, header["retained_collection_hash"]) - if receipt_hash is None: - receipt_hash = current_receipt_hash - retained_collection_hash = current_retained_hash - elif current_receipt_hash != receipt_hash: - raise RuntimeError("Generation-diff receipt changed during traversal") - elif current_retained_hash != retained_collection_hash: - raise RuntimeError("Generation-diff collection changed during traversal") - pagination = cast(dict[str, object], page["pagination"]) - items = cast(list[dict[str, object]], generation_diff["items"]) - omissions = cast(list[dict[str, object]], generation_diff["omissions"]) - if pagination["returned_count"] != len(items) + len(omissions): - raise RuntimeError("Generation-diff page count does not match its items") - if omissions: - raise RuntimeError("Generation-diff traversal omitted a retained item") - for item in items: - item_hashes.append(cast(str, item["item_hash"])) - node_id = item.get("node_id") - if ( - item.get("entity") != "node" - or item.get("change") != "changed" - or not isinstance(node_id, str) - ): - raise RuntimeError("Generation-diff synthetic item has unexpected semantics") - node_ids.append(node_id) - returned += cast(int, pagination["returned_count"]) - pages.append(page) - next_cursor = pagination["next_cursor"] - if next_cursor is None: - if returned != pagination["total_count"]: - raise RuntimeError("Generation-diff traversal did not reconstruct every item") - break - cursor = cast(str, next_cursor) - if cursor in seen_cursors: - raise RuntimeError("Generation-diff pagination repeated a cursor") - seen_cursors.add(cursor) - if len(item_hashes) != len(set(item_hashes)): - raise RuntimeError("Generation-diff traversal repeated a retained item") - if ( - returned != node_count - or len(item_hashes) != node_count - or len(node_ids) != node_count - or set(node_ids) != {synthetic_node_id(index) for index in range(node_count)} - ): - raise RuntimeError("Generation-diff traversal did not reconstruct every changed node") - if retained_collection_hash != canonical_hash(item_hashes): - raise RuntimeError("Generation-diff traversal did not reconstruct its collection hash") - summary = _page_summary(pages, started_ns=started) - if cast(float, summary["elapsed_ms"]) > 500: - raise RuntimeError("Complete generation-diff traversal exceeded 500 ms") - summary.update( - { - "receipt_hash": receipt_hash, - "item_count": returned, - "ordered_item_hash": canonical_hash(item_hashes), - "counter_ranges": { - counter: { - "minimum": min( - cast(int, cast(Mapping[str, object], item["counters"])[counter]) - for item in diagnostics_records - ), - "maximum": max( - cast(int, cast(Mapping[str, object], item["counters"])[counter]) - for item in diagnostics_records - ), - } - for counter in COUNTER_NAMES - }, - } - ) - return summary - - -def _benchmark(root: Path, node_count: int, samples: int) -> dict[str, object]: - project = Project.open(root) - index = ProjectIndex(project) - initial_snapshot = project.load() - focus = synthetic_node_id(0) - if len(initial_snapshot.edges) != node_count - 1 or any( - edge.target_id != focus or edge.source_id == focus for edge in initial_snapshot.edges - ): - raise RuntimeError("Milestone 2 fixture is not the expected focus fan-in graph") - index.build() - service = DocForgeService(project, capability_mode_name="read", diagnostics=True) - no_ast_service = DocForgeService( - project, - capability_mode_name="read", - no_ast=True, - diagnostics=True, - ) - operations: dict[str, object] = {} - - operations["bootstrap_read"], bootstrap = _measure( - service.bootstrap, - samples=samples, - p95_limit_ms=100, - response_limit_bytes=32_768, - validator=lambda result: _result_diagnostics( - result, - expected_counters={ - "index_checks": 1, - "index_synchronizations": 1, - "source_generation_checks": 1, - }, - ), - ) - bootstrap_policy = cast(dict[str, object], bootstrap["effective_policy"]) - if bootstrap_policy["capability_mode"] != "read": - raise RuntimeError("Read bootstrap did not preserve the explicit capability mode") - operations["bootstrap_no_ast"], no_ast_bootstrap = _measure( - no_ast_service.bootstrap, - samples=samples, - p95_limit_ms=100, - response_limit_bytes=32_768, - validator=lambda result: _result_diagnostics( - result, - expected_counters={ - "index_checks": 1, - "index_synchronizations": 1, - "source_generation_checks": 1, - }, - ), - ) - no_ast_policy = cast(dict[str, object], no_ast_bootstrap["adapter_policy"]) - if no_ast_policy["mode"] != "preserve-no-ast": - raise RuntimeError("No-AST bootstrap did not preserve the adapter policy") - - task_oracle = _task_oracle(service, focus) - task_probe = service.task_context( - "change", - "Change the central synthetic workflow", - focus_node_id=focus, - limit=1, - ) - _result_diagnostics( - task_probe, - expected_counters={ - "index_checks": 1, - "index_synchronizations": 0, - "source_generation_checks": 2, - }, - ) - gap = service.task_context( - "implementation", - "Implement the central synthetic workflow", - focus_node_id=focus, - limit=1, - ) - gap_codes = { - cast(str, item["code"]) for item in cast(list[dict[str, object]], gap["capsule"]["gaps"]) - } - if "category_not_declared" not in gap_codes: - raise RuntimeError("Task-context evidence-gap diagnostic was not preserved") - operations["task_context_diagnostic_page"], _ = _measure( - lambda: service.task_context( - "change", - "Change the central synthetic workflow", - focus_node_id=focus, - limit=100, - ), - samples=samples, - p95_limit_ms=500, - response_limit_bytes=200_000, - validator=lambda result: _result_diagnostics( - result, - expected_counters={ - "index_checks": 1, - "index_synchronizations": 0, - "source_generation_checks": 2, - }, - ), - ) - task_diagnostics_dropped: list[bool] = [] - operations["task_context_maximum_page"], _ = _measure( - lambda: service.task_context( - "change", - "Change the central synthetic workflow", - focus_node_id=focus, - limit=1_000, - ), - samples=samples, - p95_limit_ms=500, - response_limit_bytes=200_000, - validator=lambda result: _maximum_page_validator( - result, - expected_counters={ - "index_checks": 1, - "index_synchronizations": 0, - "source_generation_checks": 2, - }, - diagnostics_dropped=task_diagnostics_dropped, - ), - ) - operations["task_context_maximum_page"]["diagnostics_dropped_for_budget"] = any( - task_diagnostics_dropped - ) - task_order_hash: str | None = None - - def validate_task_summary(result: dict[str, object]) -> None: - nonlocal task_order_hash - current = cast(str, result["ordered_evidence_hash"]) - if task_order_hash is None: - task_order_hash = current - elif current != task_order_hash: - raise RuntimeError("Task-context traversal order changed across samples") - - task_measurement, task_summary = _measure( - lambda: _task_traversal( - service, - focus, - node_count=node_count, - oracle=task_oracle, - ), - samples=samples, - p95_limit_ms=2_500, - response_limit_bytes=32_768, - validator=validate_task_summary, - ) - operations["task_context_complete"] = { - **task_measurement, - "result_summary": task_summary, - } - - baseline = service.generation_diff(limit=1) - if baseline.get("receipt_state") != "current": - raise RuntimeError("Initial generation-diff receipt is not current") - for path in sorted((root / "docs" / "content").glob("node-*.md")): - path.write_text( - path.read_text(encoding="utf-8") + "\nMilestone 2 transition generation.\n", - encoding="utf-8", - ) - index.build() - generation_probe = service.generation_diff(limit=1) - _result_diagnostics( - generation_probe, - expected_counters={ - "index_checks": 0, - "index_synchronizations": 0, - "source_generation_checks": 2, - }, - ) - operations["generation_diff_diagnostic_page"], _ = _measure( - lambda: service.generation_diff(limit=100), - samples=samples, - p95_limit_ms=100, - response_limit_bytes=200_000, - validator=lambda result: _result_diagnostics( - result, - expected_counters={ - "index_checks": 0, - "index_synchronizations": 0, - "source_generation_checks": 2, - }, - ), - ) - generation_diagnostics_dropped: list[bool] = [] - operations["generation_diff_maximum_page"], _ = _measure( - lambda: service.generation_diff(limit=1_000), - samples=samples, - p95_limit_ms=100, - response_limit_bytes=200_000, - validator=lambda result: _maximum_page_validator( - result, - expected_counters={ - "index_checks": 0, - "index_synchronizations": 0, - "source_generation_checks": 2, - }, - diagnostics_dropped=generation_diagnostics_dropped, - ), - ) - operations["generation_diff_maximum_page"]["diagnostics_dropped_for_budget"] = any( - generation_diagnostics_dropped - ) - generation_order_hash: str | None = None - - def validate_generation_summary(result: dict[str, object]) -> None: - nonlocal generation_order_hash - current = cast(str, result["ordered_item_hash"]) - if generation_order_hash is None: - generation_order_hash = current - elif current != generation_order_hash: - raise RuntimeError("Generation-diff traversal order changed across samples") - - generation_measurement, generation_summary = _measure( - lambda: _generation_traversal(service, node_count=node_count), - samples=samples, - p95_limit_ms=500, - response_limit_bytes=32_768, - validator=validate_generation_summary, - ) - operations["generation_diff_complete"] = { - **generation_measurement, - "result_summary": generation_summary, - } - - project_tree_before = _tree_hash(root) - configurations: dict[str, dict[str, object]] = {} - for client in ("codex", "claude", "openclaw"): - configuration_hash: str | None = None - - def validate_configuration( - result: dict[str, object], - ) -> None: - nonlocal configuration_hash - diagnostics = cast( - Mapping[str, object], - result["_benchmark_diagnostics"], - ) - _assert_zero_work(diagnostics) - counters = cast(Mapping[str, object], diagnostics["counters"]) - if counters["index_checks"] != 0 or counters["source_generation_checks"] != 0: - raise RuntimeError("Configuration preview performed hidden project work") - current = cast(str, result["configuration_hash"]) - if configuration_hash is None: - configuration_hash = current - elif current != configuration_hash: - raise RuntimeError("Configuration preview is not deterministic") - - measurement, result = _measure( - lambda selected=client: _profiled( - lambda: generate_client_configuration(project, selected) - ), - samples=samples, - p95_limit_ms=500, - response_limit_bytes=32_768, - validator=validate_configuration, - ) - artifact = cast(dict[str, object], result["artifact"]) - configurations[client] = { - **measurement, - "configuration_hash": result["configuration_hash"], - "artifact_format": artifact["format"], - } - operations["configuration_preview"] = configurations - if _tree_hash(root) != project_tree_before: - raise RuntimeError("Configuration preview changed the project tree") - - doctors: dict[str, dict[str, object]] = {} - for client in ("codex", "claude", "openclaw"): - config_path = root.parent / f"doctor-{client}.config" - generated = generate_client_configuration( - project, - client, - output=config_path, - ) - generated_name = cast(str, generated["server_name"]) - config_before = config_path.read_bytes() - expected_state = "degraded" if client == "claude" else "healthy" - - def validate_doctor( - result: dict[str, object], - *, - selected_client: str = client, - selected_state: str = expected_state, - ) -> None: - diagnostics = cast( - Mapping[str, object], - result["_benchmark_diagnostics"], - ) - _assert_zero_work(diagnostics) - counters = cast(Mapping[str, object], diagnostics["counters"]) - if counters["index_checks"] != 0 or counters["source_generation_checks"] != 0: - raise RuntimeError("Doctor performed hidden project work") - if result["doctor_state"] != selected_state: - raise RuntimeError(f"Generated {selected_client} configuration did not pass doctor") - - doctor_measurement, doctor_result = _measure( - lambda selected=client, path=config_path, name=generated_name: _profiled( - lambda: run_doctor( - project, - selected, - config_path=path, - server_name=name, - ) - ), - samples=samples, - p95_limit_ms=100, - response_limit_bytes=32_768, - validator=validate_doctor, - ) - if _tree_hash(root) != project_tree_before or config_path.read_bytes() != config_before: - raise RuntimeError("Doctor changed project or client configuration state") - doctors[client] = { - **doctor_measurement, - "doctor_state": doctor_result["doctor_state"], - "summary": doctor_result["summary"], - } - operations["doctor"] = doctors - return { - "fixture": { - "kind": "synthetic_generic_focus_fan_in", - "node_count": node_count, - "edge_count": node_count - 1, - "source_file_count": node_count, - "max_tool_output_chars": 200_000, - }, - "operations": operations, - "process_peak_rss_kib": int(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss), - } - - -def _isolated_memory(nodes: int, samples: int) -> int: - completed = subprocess.run( - [ - sys.executable, - str(Path(__file__).resolve()), - "--nodes", - str(nodes), - "--samples", - str(samples), - "--memory-probe", - ], - cwd=ROOT, - check=True, - capture_output=True, - text=True, - timeout=180, - ) - payload = cast(dict[str, object], json.loads(completed.stdout)) - return cast(int, payload["peak_rss_kib"]) - - -def main() -> int: - arguments = _parser().parse_args() - if arguments.nodes < 2: - raise SystemExit("--nodes must be at least 2") - if arguments.samples < 1: - raise SystemExit("--samples must be positive") - with tempfile.TemporaryDirectory(prefix="docforge-milestone2-") as directory: - root = (Path(directory) / "project").resolve() - _prepare_fixture(root, arguments.nodes) - measurement = _benchmark(root, arguments.nodes, arguments.samples) - if arguments.memory_probe: - sys.stdout.write( - json.dumps( - {"peak_rss_kib": measurement["process_peak_rss_kib"]}, - sort_keys=True, - ) - ) - return 0 - isolated_peak = _isolated_memory(arguments.nodes, arguments.samples) - if isolated_peak > 262_144: - raise RuntimeError("Milestone 2 isolated process exceeded 256 MiB peak RSS") - result = { - "schema_version": 1, - "benchmark": "docforge2_milestone2", - "source": { - "revision": _git(["rev-parse", "HEAD"]), - "dirty": bool(_git(["status", "--porcelain"])), - }, - "environment": { - "platform": platform.platform(), - "machine": platform.machine(), - "python": platform.python_version(), - "implementation": platform.python_implementation(), - }, - "method": { - "clock": "time.perf_counter_ns", - "response_size": "UTF-8 bytes of compact sorted JSON", - "samples": arguments.samples, - "memory_probe_samples": arguments.samples, - "warmups": 1, - "percentile": "nearest-rank", - "memory": "isolated child-process resource.getrusage(RUSAGE_SELF).ru_maxrss", - "memory_limit_kib": 262_144, - "zero_work_counters": list(ZERO_WORK_COUNTERS), - }, - **measurement, - "isolated_process_peak_rss_kib": isolated_peak, - } - encoded = json.dumps(result, sort_keys=True, indent=2) + "\n" - if arguments.output is not None: - output = arguments.output.resolve() - output.parent.mkdir(parents=True, exist_ok=True) - output.write_text(encoded, encoding="utf-8") - sys.stdout.write(encoded) - return 0 - - -if __name__ == "__main__": - sys.exit(main())