525 lines
32 KiB
Markdown
525 lines
32 KiB
Markdown
# DocForge2 development notes
|
||
|
||
This is the running implementation record for DocForge2. It records what is active, what was
|
||
measured, what changed, what failed, why architectural decisions were made, and which ideas were
|
||
deferred. Stable user and compatibility contracts still belong in dedicated documentation.
|
||
|
||
## Working rules
|
||
|
||
- Only one milestone is active at a time.
|
||
- `main` remains the last fully verified milestone.
|
||
- Active implementation occurs on `dev`.
|
||
- Every milestone begins from direct repository evidence and ends with focused tests, the complete
|
||
repository gate, updated measurements, documentation closeout, and a clean pushed state.
|
||
- WorldForge, ScrapeStation, legacy DocForge, and production MCP bindings remain out of scope.
|
||
- DocForge2 does not self-host during this program.
|
||
- Release tags and Forgejo releases require Rob's explicit approval.
|
||
|
||
## Milestone 0 — complete
|
||
|
||
Milestone 0 established the public successor, preserved the complete lineage and v1 tag, integrated
|
||
the no-AST and adapter-lifecycle work, froze compatibility guarantees, added repository-native
|
||
quality and contract gates, and recorded cold/warm performance, memory, rendering, and response
|
||
sizes.
|
||
|
||
The central measurement was decisive: a 1,000-node warm exact lookup took about 286 ms while the
|
||
generation-pinned SQLite query path took about 0.4–1.4 ms. Repeated whole-source loading and
|
||
validation, not SQLite, is the first optimization target.
|
||
|
||
## Milestone 1 — complete: fast, observable core
|
||
|
||
### Outcome
|
||
|
||
Warm retrieval should disappear into normal tool overhead. Routine reads must not parse project
|
||
sources. Status must not render or rebuild hidden work. Results must remain bounded independently
|
||
of project size.
|
||
|
||
### Starting evidence
|
||
|
||
- Generic `Project.load()` walks, captures, parses, rereads, validates, hashes, and checks Git for
|
||
the complete source set.
|
||
- Exact retrieval validates twice around one bounded SQLite query.
|
||
- Context compilation performs three full project loads.
|
||
- Render status recompiles the complete manual.
|
||
- Incremental adapters already prove that manifest attestation can make no-change synchronization
|
||
and exact retrieval sub-millisecond on a tiny fixture.
|
||
- Pinned viewer queries prove the current SQLite schema can serve bounded reads quickly.
|
||
|
||
### Final outcome
|
||
|
||
Routine generic reads now parse zero canonical source files, use one generation-pinned SQLite
|
||
snapshot, and return bounded results. Status checks perform no hidden rendering or rebuilding.
|
||
Structured counters prove those invariants independently of machine timing. Complete loading and
|
||
deep validation remain recovery and equivalence oracles.
|
||
|
||
### Work log
|
||
|
||
#### Linear dependency validation
|
||
|
||
The inherited dependency-cycle preparation scanned every edge once for every node. The graph
|
||
validator now constructs dependency adjacency in one edge pass and sorts each adjacency list before
|
||
an iterative deterministic depth-first cycle check. The iterative stack also removes recursion
|
||
depth as a failure mode on large valid graphs. The same edge pass now rejects missing sources as
|
||
well as missing targets.
|
||
|
||
A 10,000-node regression test counts complete edge-collection iteration passes and caps them at
|
||
four. The focused correctness and bounded-pass tests pass, and the configured strict source type
|
||
gate is clean.
|
||
|
||
One validation command initially included `tests/test_core.py` in a direct Pyright invocation.
|
||
Repository Pyright intentionally covers `src` and `tools`, so that command reported existing
|
||
untyped test-result indexing rather than a source defect. Rerunning the repository-configured type
|
||
gate produced zero diagnostics.
|
||
|
||
#### Persistent generic source generations
|
||
|
||
Generic projects now persist a version-1 source-generation receipt only after complete source
|
||
loading and fully verified index publication. The receipt binds the explicit generic source
|
||
contract, project and root identity, adapter, revision, source hash, every canonical/authority/
|
||
descriptor regular-file identity, and every source-membership directory identity.
|
||
|
||
A normal warm check reads no canonical source bytes. It validates the known directories and files
|
||
directly using device, inode, mode, size, nanosecond modification time, and nanosecond change time.
|
||
Directory identities detect add, delete, and rename operations without an `rglob`. Any missing,
|
||
malformed, incompatible, foreign, or dirty receipt becomes a cache miss and falls back to the full
|
||
canonical load and row-verification oracle. Successful fallback verification repairs the disposable
|
||
receipt.
|
||
|
||
The final 1,000-node evidence run initially exposed a repeatable 52 ms exact-read maximum against
|
||
the 50 ms target. Profiling showed no source parsing or SQLite cost; each request rebuilt and
|
||
revalidated 1,000 `Path` objects from the unchanged JSON receipt twice. The project binding now
|
||
caches only the strictly validated receipt structure behind its device, inode, size, modification
|
||
time, and change-time signature. Canonical file and directory identities are still recaptured
|
||
before and after every query. Receipt replacement or mutation invalidates the cache and fails
|
||
closed. The same ordered exact-read profile fell from about 40–52 ms to about 18 ms without
|
||
weakening stale-read refusal.
|
||
|
||
The receipt is deliberately generic-project behavior. Incremental adapter manifests retain
|
||
authority over generated or specialist source identities. A one-method legacy adapter continues to
|
||
work even when it cannot provide a cheap generation.
|
||
|
||
#### Request-scoped immutable reads
|
||
|
||
Index reads now use one read-only SQLite transaction pinned to one verified file signature and one
|
||
source identity. Existence checks and queries share that connection. Before returning, the request
|
||
rechecks the index signature and current cheap source generation. A concurrent source or index
|
||
change fails closed.
|
||
|
||
Context compilation hydrates nodes and edges from the pinned derived snapshot while retaining
|
||
profiles from the immutable descriptor. It no longer loads or parses canonical sources. The public
|
||
full `Project.load()` and deep `ProjectIndex.check()` behavior remains the recovery and equivalence
|
||
oracle.
|
||
|
||
Focused tests prove that fresh-process-style generic reads can run exact, search, filter,
|
||
backlinks, dependency, impact, context, and no-change synchronization operations while
|
||
`Project.load()` is forbidden. They also prove a final source-generation change is rejected before
|
||
return and missing/corrupt receipts fall back and repair.
|
||
|
||
The final clean 1,000-file run recorded:
|
||
|
||
| Operation | Milestone 0 median | Milestone 1 median | Milestone 1 p95 |
|
||
|---|---:|---:|---:|
|
||
| Warm no-change synchronize | 142.479 ms | 9.192 ms | 9.324 ms |
|
||
| Exact node | 286.306 ms | 17.887 ms | 18.577 ms |
|
||
| Search, limit 20 | 288.793 ms | 19.518 ms | 19.884 ms |
|
||
| Dependencies, depth 8 | 287.791 ms | 18.117 ms | 19.061 ms |
|
||
| Context, 32k, page 20 | 436.897 ms | 25.395 ms | 25.867 ms |
|
||
| Render status | 150.591 ms | 18.969 ms | 19.613 ms |
|
||
| Visualization status | — | 9.875 ms | 10.836 ms |
|
||
|
||
The recorded run came from clean commit `6253c45a5eca01efa8c73ea3dfe4d85c55878ada`.
|
||
Every measured operation passed its p95 threshold and work-counter contract.
|
||
|
||
#### Read-only audit reconciliation
|
||
|
||
The three Milestone 1 audits agreed on the main architecture:
|
||
|
||
- Keep complete loading and deep checking as independent truth oracles.
|
||
- Trust only versioned, identity-bound disposable generation receipts.
|
||
- Use one pinned read transaction and retain a final dirty check.
|
||
- Hydrate context from the current index.
|
||
- Replace full-edge traversal scans with bounded indexed frontier reads.
|
||
- Add compact success receipts before allowing large mutations to report post-write size errors.
|
||
- Replace hidden render-status rendering with a receipt comparison.
|
||
- Add algorithmic counters and parse-count gates alongside wall-clock thresholds.
|
||
|
||
One audit identified a correctness risk beyond latency: a large mutating MCP operation can commit
|
||
successfully and then be replaced by `result_too_large`. This must be fixed in Milestone 1 so
|
||
exactly-once operations never report a false failure after mutation.
|
||
|
||
#### Mutation success receipts
|
||
|
||
Proposal, preview, and canonical-application MCP mutations now declare an internal response policy.
|
||
Before runtime validation or mutation, the service proves that a minimum receipt containing the
|
||
actual input identity and fixed-length hash fields fits the configured output limit. If it cannot,
|
||
the operation returns a preflight size error with `mutation_committed = false` and does not call the
|
||
mutation.
|
||
|
||
Small results retain the existing full payload. Oversized successful results become a version-1
|
||
compact receipt that preserves exact changeset identity, hash, workflow scalars, and lifecycle
|
||
state while omitting full operations. Application receipts also preserve changed-source counts and
|
||
derived-refresh status/counts. If the compact form is still too large, the service returns the
|
||
minimum receipt proven by preflight. It never converts committed success into a post-write size
|
||
failure.
|
||
|
||
End-to-end MCP tests exercise two large hash-chained appends followed by canonical application.
|
||
Each response stays within 1,600 compact JSON characters, exposes the new exact hash, and reports
|
||
committed success. A separate 700-character preflight test proves the callback and changeset file
|
||
are never created. Changeset lifecycle receipts now obey the configured changeset byte limit on
|
||
both write and read.
|
||
|
||
#### Receipt-based render status
|
||
|
||
Successful declared renders now publish a bounded, atomic version-1 receipt below the disposable
|
||
cache. It binds project/root/adapter/source identity, the normalized view configuration, renderer
|
||
identity, template and output hashes, byte size, and safe regular-file identities. Generic renders
|
||
also publish the verified source generation used by cheap status.
|
||
|
||
Normal status compares only source-generation, descriptor/view, template-file, output-file, and
|
||
receipt identities. It does not call `Project.load()`, prepare the renderer, construct HTML, read
|
||
the full output, rebuild the index, or repair missing state. Missing and corrupt receipts are
|
||
`unverified`; source, template, or output changes are `stale`. An explicit `deep` option on the
|
||
Python, CLI, and MCP status surfaces preserves the old side-effect-free full-render equivalence
|
||
oracle.
|
||
|
||
Receipt failure after atomic output replacement is reported as degraded publication success, not a
|
||
false render failure. Canonical application converts the same condition into a degraded
|
||
derived-refresh report while retaining canonical success. Focused tests forbid source loading and
|
||
renderer preparation during warm status and cover output, template, missing-receipt, corrupt-
|
||
receipt, and post-publication receipt-failure behavior.
|
||
|
||
After race hardening, a 50-sample three-node receipt-status check measured a 3.202 ms median and
|
||
3.509 ms p95, compared with the 1.941 ms Milestone 0 three-node full-render status. The small
|
||
fixture does not show the scaling benefit; the 1,000-node Milestone 0 status baseline was
|
||
150.591 ms and will be rerun in the final Milestone 1 evidence pass.
|
||
|
||
#### Bounded indexed retrieval
|
||
|
||
Search, metadata filtering, backlinks, dependency traversal, and impact traversal now query one
|
||
extra row beyond the requested bound and report `limit` plus `truncated`. Backlinks, dependency,
|
||
and impact APIs accept the same additive `limit` option through Python, CLI, and MCP surfaces.
|
||
Omitted limits are capped by the project `max_results` policy.
|
||
|
||
Traversal no longer loads the complete edge table and repeatedly scans it. It performs
|
||
deterministically ordered frontier queries through the existing source primary key or target index.
|
||
Each request also has a deterministic edge-examination budget derived from its result limit. The
|
||
response includes `candidate_edges_consumed`, `candidate_edges_limit`, and `truncation_reason`
|
||
counters so algorithmic work can be asserted independently of machine timing. `truncated` is true
|
||
when either another unique result exists or the work budget prevents proving completeness.
|
||
|
||
The read-only query-plan audit found that source-ordered unfiltered incoming traversal required a
|
||
temporary SQLite sort with the version-2 `(target_id, relation, source_id)` index. Direct
|
||
`EXPLAIN QUERY PLAN` evidence showed `USE TEMP B-TREE FOR ORDER BY`. A measured additive
|
||
`(target_id, source_id, relation)` index removes that sort. The disposable index schema is now
|
||
version 3, so existing version-2 indexes rebuild without changing canonical source or proposals.
|
||
Frontier cursors are streamed and stop immediately on the first omitted unique result. A focused
|
||
core, CLI, MCP, Ruff, and Pyright gate passes for this work-in-progress slice.
|
||
|
||
#### Structured profiling and zero-work gates
|
||
|
||
DocForge now has an opt-in, request-local diagnostics collector backed by `ContextVar`. It emits
|
||
one bounded version-1 aggregate with a fixed operation name, outcome, total elapsed nanoseconds,
|
||
fixed stage timing keys, and fixed integer counters. It never records paths, node IDs, queries,
|
||
source text, or SQL. Disabled mode reads no clock and adds no response field, preserving the
|
||
existing CLI and MCP payloads.
|
||
|
||
The generic loader, adapter projection and extraction paths, source-generation checks, index
|
||
checks/synchronization/build/read transactions, render status/preparation/output hashing, MCP
|
||
runtime validation, and viewer-manager requests now expose direct proof counters. A warm
|
||
incremental adapter cache hit still counts the enclosing project load, so the counters cannot hide
|
||
full adapter assembly merely because extraction was reused.
|
||
|
||
MCP servers and the CLI accept the additive `--diagnostics` startup option. Diagnostics are
|
||
attached to structured successes and errors only when the complete MCP response still fits its
|
||
configured output budget; they are discarded before any primary result or compact mutation
|
||
receipt. Warm generic error decoration now reads the persisted source generation before falling
|
||
back to complete loading. Render- and visualization-status error paths explicitly disable both
|
||
recovery synchronization and complete identity loading.
|
||
|
||
Context isolation tests cover threads, concurrent async tasks, repeated stages, nested collectors,
|
||
exceptions, and disabled collection. Repository tests assert that warm success and error reads,
|
||
render status, and visualization status perform zero project loads, source parses, adapter
|
||
projection/extraction, index builds, render preparation, output construction, and output hashing.
|
||
The result JSON schema contains the same closed operation, stage, and counter sets as the
|
||
implementation.
|
||
|
||
The maintained `tools/milestone1_benchmark.py` harness adds hard counter and p95 latency gates to a
|
||
disposable generic project. The smoke target is part of `make gate`; the final 1,000-node evidence
|
||
is recorded in `benchmarks/milestone1-2026-07-29.json`. The historical Milestone 0 harness remains
|
||
behaviorally unchanged as comparison evidence; it only exposes shared fixture and measurement
|
||
helpers to the Milestone 1 harness.
|
||
|
||
#### Visualization snapshot freshness
|
||
|
||
Visualization workers now receive a version-1 snapshot specification containing the exact
|
||
validated index publication signature: device, inode, size, modification time, and change time.
|
||
Both the manager and worker reject a launch if that publication changes before startup. The
|
||
transmitted project root, root fingerprint, source identity, adapter, counts, limits, and confined
|
||
index path are strictly validated before the worker may serve source or graph data.
|
||
|
||
Worker health reports index freshness through stat-only comparison. It does not open SQLite and
|
||
does not renew the browser activity lease. The version-2 viewer-manager protocol validates the
|
||
complete worker identity and distinguishes an unreachable worker from a live stale worker. A stale
|
||
worker stays lifecycle `running` for accurate diagnosis, but the next visualize request stops it
|
||
and launches a newly validated snapshot instead of reusing it.
|
||
|
||
Client status separately compares the worker's pinned source identity with
|
||
`IncrementalStateProject.incremental_state()`. The composite snapshot is stale if either proof is
|
||
stale, current only when both proofs are current, and unknown otherwise. A stopped worker has
|
||
unknown snapshot identity. MCP preserves this state at the top-level `staleness` field and disables
|
||
recovery synchronization and full-load error decoration.
|
||
|
||
Tests cover signature mutation before worker startup, malformed identity, missing and symlinked
|
||
indexes, stat-only health, unchanged activity, current/unknown/stale source states, live stale
|
||
workers, non-reuse, and zero-load status. The Milestone 1 benchmark now measures current, stale,
|
||
not-running, and unavailable visualization status separately with the same zero-work and 50 ms p95
|
||
gates as other receipt status operations.
|
||
|
||
#### Bounded pagination and exact large-result review
|
||
|
||
The final Milestone 1 contract audit found that count limits and the global MCP output ceiling were
|
||
not sufficient. A 1,000-node context response already exceeded the normal 200,000-character tool
|
||
limit, and one allowed changeset operation can be larger than that limit. Returning
|
||
`result_too_large` kept transport bounded but stranded useful evidence.
|
||
|
||
Version-1 pagination now uses canonical, base64url cursors with a domain-separated SHA-256
|
||
corruption checksum. Cursors bind the project, adapter, canonical generation, semantic query,
|
||
collection hash, and position. They are deliberately unkeyed read tokens rather than authorization
|
||
credentials. Corrupt tokens fail as `invalid_cursor`; changed generations or collections fail as
|
||
`stale_cursor` with explicit pagination-restart remediation.
|
||
|
||
Context transport flattens the compiler's deterministic selected entries followed by all explicit
|
||
omissions, then partitions each page back into the existing arrays. Both item count and exact
|
||
compact-JSON response size constrain packing. An individually oversized entry becomes a bounded,
|
||
hash-identified omission and advances the cursor, avoiding an infinite retry while preserving the
|
||
fact that evidence was excluded.
|
||
|
||
Changeset list, inspection, validation, and diff reads preserve direct full-result defaults while
|
||
MCP uses bounded pages. Pages retain exact changeset identity and hash. Large operation pages
|
||
compact content-bearing fields into hashes and character counts. A single oversized structured
|
||
diff is serialized once as canonical ASCII JSON and returned through hash-bound chunks that
|
||
reconstruct the exact legacy `operations` and `changes` arrays. This solves transport growth
|
||
without lowering canonical changeset limits or adding cursor storage.
|
||
|
||
The benchmark now validates zero-work and operation-specific counters for every warmup and measured
|
||
sample, records bounded semantic response summaries, covers filter, backlinks, outgoing and
|
||
incoming traversal, and measures current/stale/missing/corrupt render receipts plus all
|
||
visualization lifecycle states. A maintained query-plan test prevents the incoming traversal
|
||
temporary sort from returning.
|
||
|
||
#### Milestone closeout
|
||
|
||
The complete repository gate passed with 138 tests and 77 subtests, zero Pyright diagnostics,
|
||
warning-strict execution, package builds, contract checks, and both benchmark smoke gates. The
|
||
clean 1,000-node benchmark passed all maintained thresholds and is interpreted in
|
||
`docs/MILESTONE_1_BASELINE.md`. Compatibility, measured decisions, limitations, and scope evidence
|
||
are frozen in `docs/MILESTONE_1_CLOSEOUT.md`.
|
||
|
||
Milestone 1 made no storage rewrite, self-hosting change, production integration change, tag, or
|
||
release.
|
||
|
||
### Initial design constraints
|
||
|
||
- Full rebuild remains the recovery and equivalence oracle.
|
||
- Canonical content remains authoritative.
|
||
- Existing one-method `load_projection()` adapters remain unchanged.
|
||
- No-AST adapters remain first-class.
|
||
- Indexes, source-generation receipts, and caches remain disposable.
|
||
- Cheap reads may trust only identity-bound, versioned, corruption-checked receipts.
|
||
- Any optimization must fail closed on source mutation and must preserve stale-read refusal.
|
||
|
||
### Future ideas and suggestions
|
||
|
||
These are notes, not commitments:
|
||
|
||
- A stable source-generation provider may deserve a public adapter capability only after both the
|
||
generic project and one incremental adapter prove the same boundary.
|
||
- Profiling receipts could eventually feed the human-facing project control panel, but Milestone 1
|
||
should expose structured data before adding UI.
|
||
- A durable telemetry exporter remains deliberately deferred. Request-local bounded aggregates are
|
||
enough to prove compiler work in Milestone 1 without adding persistence, cardinality, or privacy
|
||
risks.
|
||
- The stat identity is a cheap publication proof, not a cryptographic integrity scan. Full index
|
||
validation remains the launch and query oracle.
|
||
- Cursor authentication remains deliberately absent. If read cursors ever carry authority rather
|
||
than bounded positions, they will need a different versioned security contract and persisted key
|
||
lifecycle.
|
||
|
||
## Milestone 2 — active: agent retrieval and MCP experience
|
||
|
||
### Audit reconciliation
|
||
|
||
Three independent read-only audits covered effective policy and bootstrap, task-shaped retrieval
|
||
and context capsules, and generation diffs plus client configuration and doctor checks.
|
||
|
||
They agreed on these boundaries:
|
||
|
||
- Keep the project descriptor at schema version 1. Process capability and client configuration are
|
||
machine-specific bindings, not canonical project content.
|
||
- Preserve the legacy adapter-policy payload, no-AST shorthand, tool names, default tool ordering,
|
||
one-method adapters, and custom context provider.
|
||
- Add one versioned effective-policy authority and derive bootstrap, contract, instructions, and
|
||
access reporting from it.
|
||
- Add one task-context operation with a closed task-kind vocabulary and one immutable,
|
||
generation-pinned retrieval plan. Do not create a tool for every task kind.
|
||
- Produce evidence gaps only from declared plan requirements and completed bounded checks. Never
|
||
infer missing facts from arbitrary project naming.
|
||
- Record only the latest bounded generation transition as disposable evidence. Do not add a
|
||
history database.
|
||
- Preview client configuration by default. Any write must be explicit, atomic, merge-preserving,
|
||
and backed by a verified client-format driver.
|
||
- Keep doctor strictly read-only. It must not bootstrap, synchronize, build, render, start a
|
||
viewer, or rewrite client configuration.
|
||
|
||
### Versioned effective policy and session contract
|
||
|
||
The binding now composes an immutable version-1 policy containing capability mode, adapter
|
||
evolution, AST and Logic behavior, synchronization and integrity levels, render and viewer
|
||
behavior, profiling, blocked tools, prohibitions, and explicit precedence. `--no-ast` is a
|
||
restrictive override. The exact legacy `adapter_policy` response remains a projection of the new
|
||
object.
|
||
|
||
Bootstrap reuses the identity already proven by synchronization and no longer reloads the complete
|
||
project. Its additive version-1 session contract reports binding, generation, effective policy,
|
||
actual registered surfaces and mutation access, render policies, first operation, filtered
|
||
workflow, and prohibitions. Read mode does not recommend proposals. Proposal mode recommends
|
||
registration and review only with writer access. Application is recommended only when the
|
||
exact-hash applier is enabled.
|
||
|
||
Existing factory defaults and tool order remain unchanged. Explicit application mode fails closed
|
||
without an applier. Operator mode is reserved and currently adds no tools.
|
||
|
||
### Versioned task retrieval and context capsules
|
||
|
||
The first Milestone 2 retrieval slice adds one `docforge_get_task_context` read tool rather than a
|
||
family of task-specific tools. Its closed task kinds are change, implementation, failure,
|
||
ownership, test, operation, and release. One immutable `RetrievalPlanV1` derives exact or lexical
|
||
focus, bounded bidirectional graph traversal, metadata hydration, required evidence categories,
|
||
and fixed work budgets from the project descriptor and effective policy.
|
||
|
||
The public executor re-derives every submitted plan before opening SQLite. It rejects modified
|
||
steps, task identity, requirements, category order, bounds, policy identity, or hashes as
|
||
`invalid_retrieval_plan`. Traversal binds the project relation set by canonical hash and queries
|
||
the already-validated edge table by endpoint, avoiding relation-sized SQL parameter lists.
|
||
Version-1 internal ceilings are 1,000 evidence items, 100,000 candidate edges, and 10,000 task
|
||
query characters.
|
||
|
||
The executor uses one immutable SQLite read generation. It rejects missing explicit focus, blocks
|
||
unresolved or tied lexical focus, stops at deterministic evidence and candidate-edge limits, and
|
||
checks source identity again when the transaction closes. `ContextCapsuleV1` binds the generation,
|
||
policy, request, plan, evidence collection, and complete capsule with canonical hashes.
|
||
|
||
Project relation names remain authoritative. The core recognizes only a versioned alias map for
|
||
structure, implementation, dependency, execution, data, evidence, and context. Unknown allowed
|
||
relations stay visible under their raw names as `unclassified`. Required evidence diagnostics
|
||
distinguish categories the project never declared, completed bounded checks with no selected
|
||
evidence, and incomplete checks caused by a result, work, token, or response limit.
|
||
|
||
Each evidence item carries a stable content hash, confined source identity, shortest selected graph
|
||
path, every additional qualifying relationship reason observed during traversal, and explicit
|
||
limitations where the current graph cannot prove evidence type, extractor identity, relationship
|
||
source provenance, or observation time. The planner contains no Logic operation, so no-AST
|
||
bindings can use task context without weakening their existing Logic prohibition.
|
||
|
||
Path direction is relative to the preceding traversal node. Additional relationship reasons use
|
||
the evidence node as their direction subject. Candidate-edge and unclassified-relation ceilings
|
||
produce explicit omissions and bounded summaries.
|
||
|
||
MCP pagination preserves the complete plan, collection, and capsule hashes while returning bounded
|
||
pages. Its cursor additionally binds the effective policy and task request. An individually
|
||
oversized item advances once as a hash-identified omission. A later generation or policy change
|
||
fails closed as `stale_cursor`.
|
||
|
||
The legacy profile-context contract remains intact. A custom context provider does not silently
|
||
gain core task planning. Version 1 defines no custom task-planner extension, so the additive tool
|
||
returns `task_context_unavailable` without synchronization or a complete projection load.
|
||
|
||
Two independent pre-commit audits reproduced and closed plan-forgery, relation-sized SQL,
|
||
SQLite-parameter portability, ambiguous relationship-direction, missing work-limit evidence,
|
||
schema/runtime drift, incomplete page hashing, and custom-provider hidden-load defects. Regression
|
||
coverage includes 33,005 valid relation names, fixed extreme project limits, tampered plans,
|
||
evidence-relative directions, edge and unclassified limits, schema-valid pages, changed cursor
|
||
semantics, oversized evidence advancement, no-AST retrieval, and legacy complete-projection
|
||
adapters.
|
||
|
||
The complete repository gate passes with 158 tests and 101 subtests, zero Pyright diagnostics,
|
||
warning-strict execution, package builds, public-contract validation, and the maintained Milestone
|
||
0 and Milestone 1 smoke benchmarks. Gitleaks 8.30.1 reports no secret findings in the working tree.
|
||
|
||
### Latest-generation diff receipt
|
||
|
||
Three read-only audits reconciled the index publication, public transport, compatibility, and
|
||
no-AST boundaries before implementation. The selected design stores one disposable
|
||
`generation-diff.json` receipt. It does not add a history database, arbitrary generation
|
||
selectors, source text, rendered content, or Logic details.
|
||
|
||
Before a build loads current source, it accepts an existing index only when its exact main-file
|
||
inode has a matching stable whole-file attestation and no WAL, journal, or shared-memory sidecar.
|
||
It then captures that predecessor through an immutable main-file transaction. The capture validates
|
||
the SQLite application and schema IDs, project/root/adapter binding, integrity, complete node and
|
||
edge rows, Logic aggregate identity, FTS count, metadata hashes and counts, and final file
|
||
signature. It never calls normal check or synchronization and never repairs predecessor evidence.
|
||
|
||
The final source revalidation now compares exact nodes and edges in addition to source hash,
|
||
revision, and Logic. A verified predecessor that maps the same source identity to different graph
|
||
content fails before publication as `generation_collision`. This closes a pre-existing adapter
|
||
determinism gap found during the generation-diff audit.
|
||
|
||
SQLite replacement is now the explicit derived mutation commit point. Whole-file attestation,
|
||
cheap source-generation, and generation-diff receipts publish independently afterward. Any
|
||
post-commit receipt failure returns `status = ok`, `index = published`, a bounded degraded
|
||
publication record, and receipt-stage names. It never rolls back the new index or reports a false failed
|
||
mutation. Attestation hashing checks the exact index signature before, during, and immediately
|
||
before receipt publication.
|
||
|
||
Version-1 diff semantics compare every core `Node` field by stable node ID and exact edge triples.
|
||
Node renames are removal plus addition. Edge changes are removal plus addition. Exact summary
|
||
counts and a full ordered item-hash collection cover every change. Retained details are
|
||
deterministically ordered and independently capped at 1,000 items and 1 MiB with explicit item- or
|
||
byte-limit evidence. A first build or untrusted predecessor is a baseline with no fabricated
|
||
all-added result. A same-generation reindex republishes the existing meaningful transition against
|
||
the new index file identity instead of erasing it with an empty diff.
|
||
|
||
The additive public surfaces are:
|
||
|
||
- CLI `generation-diff [--limit N] [--cursor OPAQUE]`.
|
||
- MCP `docforge_get_generation_diff(limit=None, cursor=None)`.
|
||
- Telemetry operations `cli.generation-diff` and `mcp.generation_diff`.
|
||
|
||
Public reads do not open SQLite, call `project.load()`, extract an adapter projection, parse source,
|
||
check, synchronize, build, or repair. They strictly validate the bounded receipt, compare stable
|
||
receipt and index file identities, require two matching cheap source-generation checks, and report
|
||
unknown for legacy adapters without that capability. Missing, corrupt, foreign, oversized,
|
||
symlinked, stale, or concurrently changed evidence remains a read-only status outcome.
|
||
|
||
Generation-diff pagination binds the complete stored receipt hash and effective policy. That hash
|
||
already covers project, generation, graph, collection, and committed-index identity. Page size may
|
||
change. A replaced receipt returns `stale_cursor`. One top-level pagination object owns the only
|
||
cursor. The nested version-1 page uses `receipt_header.stored_receipt_hash` so it never
|
||
misrepresents the complete receipt hash as the hash of a partial header. The summary distinguishes
|
||
additional retained pages from details permanently omitted by the fixed publication limits.
|
||
|
||
Adversarial coverage now includes strict runtime/schema rejection, predecessor attestation and
|
||
generation identity, live and synthetic SQLite sidecars, cache-root symlink substitution,
|
||
source/sidecar changes during diff preparation, independent receipt failures, and degraded
|
||
post-commit identity and durability failures. Focused verification passes the direct, CLI, MCP,
|
||
schema, pagination, legacy, incremental no-AST, and zero-work suites. The complete repository gate
|
||
passes with 176 tests and 113 subtests, zero Pyright diagnostics, package builds, web checks, and
|
||
the maintained Milestone 0 and Milestone 1 smoke benchmarks. Final independent re-audit is in
|
||
progress before this slice is committed.
|
||
|
||
The final dense benchmark uses a 1,000-node transition with 1,000 changed details. Its receipt is
|
||
775,663 bytes. Direct status is 37.659 ms median and 39.108 ms p95. A maximum-size MCP request
|
||
returns 307 items in 199,754 bytes at 54.037 ms median and 56.617 ms p95. Four pages reconstruct
|
||
all 1,000 retained details in 652,798 bytes at 201.55 ms median. Peak RSS is 79,096 KiB.
|
||
Every hidden-work counter remains zero; the read performs exactly two cheap source-generation
|
||
checks.
|
||
|
||
Measurement found and removed two avoidable costs before commit. Receipt loading had repeated the
|
||
complete 1,000-item validator solely to check project identity; it now validates once and compares
|
||
the three binding fields directly. Page fitting had encoded every growing prefix; it now uses an
|
||
exact logarithmic search and retains the hash-only oversized-item omission path. The maximum page
|
||
fell from 272.06 ms p95 to 56.617 ms p95, while full traversal fell from roughly 859 ms to
|
||
203.55 ms p95. Regression tests require one receipt validation and at most 15 response encodes for
|
||
1,000 page candidates. Final independent publication, contract, and performance audits approve
|
||
the slice for commit.
|