1
0
Fork 0
Code Issues Pull requests Projects Releases 2 Packages Wiki Activity Actions Pages

Add incremental adapter compiler boundary

This commit is contained in:
Andraxion 2026-07-25 19:08:39 -04:00
parent 82b3b90521
commit 696b62f9f8
20 changed files with 1592 additions and 122 deletions

View file

@ -1,4 +1,4 @@
# DocForge 0.14 contract
# DocForge 1.1 development contract
## Authority boundary
@ -19,7 +19,8 @@ commit when Git is available; it cannot change repository state.
- Result envelope: `schemas/result.schema.json`, version 1.
- Changeset schema: `schemas/changeset.schema.json`, version 1.
- Index schema: version 1, disposable and reproducible.
- Core, CLI, and MCP server: version 1.0.0.
- Core, CLI, and MCP server: version 1.1.0.dev0 on `Dev-Rewrite`.
- Incremental extraction cache: version 1, disposable and reproducible.
Schema files describe the generic interchange contract. Runtime validation remains responsible for
path confinement, source hashing, relationship resolution, dependency cycles, project limits, stale
@ -48,6 +49,10 @@ operation names its expected base hash. A move preserves the stable node ID. A d
every incident relationship. Proposal validation and storage are atomic. Application requires the
exact final changeset hash; prose is never auto-merged.
Relationship-only additions and removals use the validated update operation without changing node
metadata or content. They remain bound to the complete changeset base hash and the anchor node's
expected content hash.
The MCP process binds to one configured writer identity at startup. The project descriptor grants
that writer explicit families and operation types. A changeset records its creator, project root
fingerprint, base revision, canonical source hash, and ordered operations. Every append requires the
@ -200,3 +205,15 @@ An explicit integration may construct the full fixed MCP surface for a configure
and one startup-bound writer. Canonical application is registered only when the integration also
supplies a startup-bound applier identity and project-owned `CanonicalApplier`. An adapter without
proposal settings or validation remains read-only.
An adapter may additionally implement the opt-in incremental contract. Its manifest inventories
stable source IDs, fingerprints, extractor versions, and source dependencies without parsing the
complete project. Each extraction owns deterministic nodes, relationships, and optional
function-scoped logic. Added, changed, deleted, and reverse-dependent sources are invalidated.
Cached and refreshed facts are always assembled into a complete projection and pass normal graph
validation before publication. The full projection loader remains the fallback and equivalence
oracle.
Logic projections are not primary graph nodes. They remain source-scoped, function-owned,
independently cached control-flow data so ordinary search, Nodes, Flow, and Web do not become
statement graphs.

View file

@ -0,0 +1,127 @@
# Incremental Adapter Indexing
DocForge Release 1 adapters return one complete immutable projection. That contract remains
supported. The `Dev-Rewrite` compiler adds an opt-in source-scoped contract that avoids reparsing
unchanged files while preserving the same validated, atomically published graph.
## Safety model
Incremental indexing is an extraction optimization. It does not weaken publication:
1. The adapter returns a cheap, deterministic `AdapterManifest`.
2. DocForge compares every source fingerprint and extractor version with the last cache generation.
3. Added, changed, deleted, and reverse-dependent sources are invalidated.
4. The adapter reparses only invalidated sources.
5. DocForge assembles cached and refreshed contributions into a complete candidate projection.
6. The complete graph passes the same validation as a full adapter projection.
7. DocForge rereads the manifest to prove sources remained stable.
8. The extraction cache and SQLite graph are published with atomic file replacement.
An interrupted extraction never replaces the last validated SQLite index. A malformed,
incompatible, or missing cache is a cache miss, not a partial graph.
## Adapter contract
An incremental loader implements all three methods:
```python
class MyAdapter:
def load_manifest(self) -> AdapterManifest: ...
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ...
def load_projection(self) -> AdapterProjection: ...
```
`load_projection()` remains the deterministic full-rebuild fallback and equivalence oracle.
Each `AdapterSource` declares:
- A stable source ID.
- A safe project-relative source path.
- A SHA-256 content fingerprint.
- An extractor version.
- Other source IDs whose changes can alter this source's extracted facts.
Each `AdapterSourceProjection` owns:
- Its primary graph nodes.
- Its primary graph relationships, including cross-source relationships owned by that source.
- Optional function-scoped logic projections.
Ownership must be deterministic. Two sources may not produce the same primary node or the same
function logic projection.
## Invalidation
DocForge invalidates a source when:
- It is new.
- Its fingerprint changed.
- Its extractor version changed.
- A declared dependency was added, changed, or deleted.
- Any source in its reverse-dependency chain was invalidated.
Deleted sources are omitted from the candidate projection. Their cached dependency declarations
remain available long enough to invalidate surviving dependents.
If an adapter cannot precisely describe the affected sources, it should declare broader
dependencies or change its adapter/extractor version. Incorrectly retaining a stale relationship
is never an acceptable optimization.
## Build reporting
`build` and `reindex` include an extraction report:
```json
{
"build": {
"mode": "incremental",
"cache_hits": 391,
"reparsed_sources": 3,
"invalidated_sources": 3,
"deleted_sources": 0,
"total_sources": 394,
"cache_hit_ids": ["..."],
"reparsed_source_ids": ["..."]
}
}
```
Adapters can call `AdapterProject.verify_incremental_equivalence()` in release and contract tests.
The check compares project identity, revision, source hash, nodes, and relationships against
`load_projection()`.
## Manual changes and relationships
Changesets remain an approval queue, not a compiler queue. Compilation never silently applies a
proposal.
After explicit application:
1. The project-owned applier updates canonical files.
2. Changed manual files receive new fingerprints.
3. Incremental extraction reparses those files and affected dependents.
4. The complete candidate graph is validated and published.
5. Declared renders are regenerated.
`docforge_propose_relationship_update` queues relationship-only additions and removals without
rewriting node content. It is still bound to the changeset's complete base source hash and the
anchor node's expected content hash.
## Lazy logic boundary
`LogicProjection` stores control flow separately from the primary architecture graph. It is owned
by one function or method node and one source extraction.
Logic nodes can represent entries, conditions, basic blocks, calls, merges, loops, returns, and
raises. Logic edges retain relation, display label, and deterministic ordinal. Adapters may leave
logic empty until they implement a language analyzer.
This boundary prevents thousands of boolean expressions and basic blocks from polluting Nodes,
Flow, Web, ordinary search, or architectural traversal. A future Logic view can request one
function-scoped projection on demand.
## Full rebuilds
Full rebuilds remain mandatory as a fallback and equivalence oracle. Change the adapter version,
extractor version, or cache schema whenever old cached facts are no longer valid. Removing the
confined extraction cache also forces a clean reparse without affecting canonical files.

View file

@ -49,6 +49,7 @@ gate.
- `docforge_propose_node_create`
- `docforge_propose_node_update`
- `docforge_propose_node_move`
- `docforge_propose_relationship_update`
- `docforge_propose_node_delete`
- `docforge_validate_changeset`
- `docforge_get_changeset_diff`
@ -58,6 +59,8 @@ Proposal tools may write only below the configured changeset or isolated preview
change canonical files or declared project output. Without `--proposal-writer`, changeset mutation
tools return `proposal_access_disabled`. Validation, diff retrieval, and preview remain available
for existing changesets. A preview accepts a declared view ID, not a renderer name or command.
The relationship-update tool queues additions and removals without rewriting node content and
rejects an empty relationship list.
## Canonical application tool

View file

@ -16,7 +16,10 @@ declared rendering, and the Nodes/Flow/Web visualization model documented below.
- Disposable SQLite indexing with lexical search, filters, backlinks, dependencies, and impact.
- Bounded context profiles for AI agents, including source paths and content hashes.
- Isolated, optimistic changesets with create, update, move, delete, validation, diffs, and previews.
- Relationship-only changeset operations that do not rewrite node content.
- Hash-bound canonical application through both CLI and an explicitly enabled MCP tool.
- Opt-in incremental adapter extraction with reverse-dependency invalidation.
- Lazy function-scoped logic projections that do not densify the primary graph.
- Declared HTML render views. Arbitrary templates, render commands, and output paths are rejected.
- A loopback-only graph browser with Nodes, semantic Flow, and convergence Web views,
relationship keys, source inspection, branch-aware node hiding, panel resizing, zooming, and
@ -463,6 +466,7 @@ Example MCP client configuration:
- `docforge_propose_node_create`
- `docforge_propose_node_update`
- `docforge_propose_node_move`
- `docforge_propose_relationship_update`
- `docforge_propose_node_delete`
- `docforge_validate_changeset`
- `docforge_get_changeset_diff`
@ -487,9 +491,34 @@ Recommended agent sequence:
7. Call `docforge_apply_changeset` with that exact hash.
8. Report changed canonical files and derived refresh results.
Use `docforge_propose_relationship_update` when the intended change is only an edge addition or
removal. It uses the same underlying validated update contract, but rejects empty relationship
lists and makes it explicit that node content will remain unchanged.
Custom adapters may expose the application tool only when they supply a project-owned
`CanonicalApplier`. Core DocForge will not guess how adapter nodes map back to canonical sources.
## Incremental adapter compilation
Release 1 complete-projection adapters remain supported. Adapters with large source trees can
implement the optional source-scoped manifest and extraction contract. DocForge then fingerprints
sources, reuses unchanged facts, reparses changed sources and their reverse dependents, validates a
complete candidate graph, and publishes the index atomically.
Build results report cache hits, reparsed sources, invalidated sources, deleted sources, and total
sources. A full projection remains the fallback and equivalence oracle.
Manual proposals remain separate from compilation. Applying an approved changeset updates
canonical sources first. Incremental compilation then notices those changed source fingerprints;
it never treats an unapplied proposal as canonical.
Function-scoped `LogicProjection` data is cached alongside its owning source but remains separate
from the primary Nodes, Flow, and Web graph. This is the storage boundary for a future boolean and
control-flow view without adding every condition and basic block to ordinary graph traversal.
See [Incremental Adapter Indexing](INCREMENTAL_INDEXING.md) for the complete contract, cache
invalidation rules, manual-application lifecycle, and lazy Logic boundary.
## Troubleshooting
### `stale_index` or `visualization_stale`