Add incremental adapter compiler boundary
This commit is contained in:
parent
82b3b90521
commit
696b62f9f8
20 changed files with 1592 additions and 122 deletions
|
|
@ -1,4 +1,4 @@
|
|||
# DocForge 0.14 contract
|
||||
# DocForge 1.1 development contract
|
||||
|
||||
## Authority boundary
|
||||
|
||||
|
|
@ -19,7 +19,8 @@ commit when Git is available; it cannot change repository state.
|
|||
- Result envelope: `schemas/result.schema.json`, version 1.
|
||||
- Changeset schema: `schemas/changeset.schema.json`, version 1.
|
||||
- Index schema: version 1, disposable and reproducible.
|
||||
- Core, CLI, and MCP server: version 1.0.0.
|
||||
- Core, CLI, and MCP server: version 1.1.0.dev0 on `Dev-Rewrite`.
|
||||
- Incremental extraction cache: version 1, disposable and reproducible.
|
||||
|
||||
Schema files describe the generic interchange contract. Runtime validation remains responsible for
|
||||
path confinement, source hashing, relationship resolution, dependency cycles, project limits, stale
|
||||
|
|
@ -48,6 +49,10 @@ operation names its expected base hash. A move preserves the stable node ID. A d
|
|||
every incident relationship. Proposal validation and storage are atomic. Application requires the
|
||||
exact final changeset hash; prose is never auto-merged.
|
||||
|
||||
Relationship-only additions and removals use the validated update operation without changing node
|
||||
metadata or content. They remain bound to the complete changeset base hash and the anchor node's
|
||||
expected content hash.
|
||||
|
||||
The MCP process binds to one configured writer identity at startup. The project descriptor grants
|
||||
that writer explicit families and operation types. A changeset records its creator, project root
|
||||
fingerprint, base revision, canonical source hash, and ordered operations. Every append requires the
|
||||
|
|
@ -200,3 +205,15 @@ An explicit integration may construct the full fixed MCP surface for a configure
|
|||
and one startup-bound writer. Canonical application is registered only when the integration also
|
||||
supplies a startup-bound applier identity and project-owned `CanonicalApplier`. An adapter without
|
||||
proposal settings or validation remains read-only.
|
||||
|
||||
An adapter may additionally implement the opt-in incremental contract. Its manifest inventories
|
||||
stable source IDs, fingerprints, extractor versions, and source dependencies without parsing the
|
||||
complete project. Each extraction owns deterministic nodes, relationships, and optional
|
||||
function-scoped logic. Added, changed, deleted, and reverse-dependent sources are invalidated.
|
||||
Cached and refreshed facts are always assembled into a complete projection and pass normal graph
|
||||
validation before publication. The full projection loader remains the fallback and equivalence
|
||||
oracle.
|
||||
|
||||
Logic projections are not primary graph nodes. They remain source-scoped, function-owned,
|
||||
independently cached control-flow data so ordinary search, Nodes, Flow, and Web do not become
|
||||
statement graphs.
|
||||
|
|
|
|||
127
docs/INCREMENTAL_INDEXING.md
Normal file
127
docs/INCREMENTAL_INDEXING.md
Normal file
|
|
@ -0,0 +1,127 @@
|
|||
# Incremental Adapter Indexing
|
||||
|
||||
DocForge Release 1 adapters return one complete immutable projection. That contract remains
|
||||
supported. The `Dev-Rewrite` compiler adds an opt-in source-scoped contract that avoids reparsing
|
||||
unchanged files while preserving the same validated, atomically published graph.
|
||||
|
||||
## Safety model
|
||||
|
||||
Incremental indexing is an extraction optimization. It does not weaken publication:
|
||||
|
||||
1. The adapter returns a cheap, deterministic `AdapterManifest`.
|
||||
2. DocForge compares every source fingerprint and extractor version with the last cache generation.
|
||||
3. Added, changed, deleted, and reverse-dependent sources are invalidated.
|
||||
4. The adapter reparses only invalidated sources.
|
||||
5. DocForge assembles cached and refreshed contributions into a complete candidate projection.
|
||||
6. The complete graph passes the same validation as a full adapter projection.
|
||||
7. DocForge rereads the manifest to prove sources remained stable.
|
||||
8. The extraction cache and SQLite graph are published with atomic file replacement.
|
||||
|
||||
An interrupted extraction never replaces the last validated SQLite index. A malformed,
|
||||
incompatible, or missing cache is a cache miss, not a partial graph.
|
||||
|
||||
## Adapter contract
|
||||
|
||||
An incremental loader implements all three methods:
|
||||
|
||||
```python
|
||||
class MyAdapter:
|
||||
def load_manifest(self) -> AdapterManifest: ...
|
||||
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ...
|
||||
def load_projection(self) -> AdapterProjection: ...
|
||||
```
|
||||
|
||||
`load_projection()` remains the deterministic full-rebuild fallback and equivalence oracle.
|
||||
|
||||
Each `AdapterSource` declares:
|
||||
|
||||
- A stable source ID.
|
||||
- A safe project-relative source path.
|
||||
- A SHA-256 content fingerprint.
|
||||
- An extractor version.
|
||||
- Other source IDs whose changes can alter this source's extracted facts.
|
||||
|
||||
Each `AdapterSourceProjection` owns:
|
||||
|
||||
- Its primary graph nodes.
|
||||
- Its primary graph relationships, including cross-source relationships owned by that source.
|
||||
- Optional function-scoped logic projections.
|
||||
|
||||
Ownership must be deterministic. Two sources may not produce the same primary node or the same
|
||||
function logic projection.
|
||||
|
||||
## Invalidation
|
||||
|
||||
DocForge invalidates a source when:
|
||||
|
||||
- It is new.
|
||||
- Its fingerprint changed.
|
||||
- Its extractor version changed.
|
||||
- A declared dependency was added, changed, or deleted.
|
||||
- Any source in its reverse-dependency chain was invalidated.
|
||||
|
||||
Deleted sources are omitted from the candidate projection. Their cached dependency declarations
|
||||
remain available long enough to invalidate surviving dependents.
|
||||
|
||||
If an adapter cannot precisely describe the affected sources, it should declare broader
|
||||
dependencies or change its adapter/extractor version. Incorrectly retaining a stale relationship
|
||||
is never an acceptable optimization.
|
||||
|
||||
## Build reporting
|
||||
|
||||
`build` and `reindex` include an extraction report:
|
||||
|
||||
```json
|
||||
{
|
||||
"build": {
|
||||
"mode": "incremental",
|
||||
"cache_hits": 391,
|
||||
"reparsed_sources": 3,
|
||||
"invalidated_sources": 3,
|
||||
"deleted_sources": 0,
|
||||
"total_sources": 394,
|
||||
"cache_hit_ids": ["..."],
|
||||
"reparsed_source_ids": ["..."]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Adapters can call `AdapterProject.verify_incremental_equivalence()` in release and contract tests.
|
||||
The check compares project identity, revision, source hash, nodes, and relationships against
|
||||
`load_projection()`.
|
||||
|
||||
## Manual changes and relationships
|
||||
|
||||
Changesets remain an approval queue, not a compiler queue. Compilation never silently applies a
|
||||
proposal.
|
||||
|
||||
After explicit application:
|
||||
|
||||
1. The project-owned applier updates canonical files.
|
||||
2. Changed manual files receive new fingerprints.
|
||||
3. Incremental extraction reparses those files and affected dependents.
|
||||
4. The complete candidate graph is validated and published.
|
||||
5. Declared renders are regenerated.
|
||||
|
||||
`docforge_propose_relationship_update` queues relationship-only additions and removals without
|
||||
rewriting node content. It is still bound to the changeset's complete base source hash and the
|
||||
anchor node's expected content hash.
|
||||
|
||||
## Lazy logic boundary
|
||||
|
||||
`LogicProjection` stores control flow separately from the primary architecture graph. It is owned
|
||||
by one function or method node and one source extraction.
|
||||
|
||||
Logic nodes can represent entries, conditions, basic blocks, calls, merges, loops, returns, and
|
||||
raises. Logic edges retain relation, display label, and deterministic ordinal. Adapters may leave
|
||||
logic empty until they implement a language analyzer.
|
||||
|
||||
This boundary prevents thousands of boolean expressions and basic blocks from polluting Nodes,
|
||||
Flow, Web, ordinary search, or architectural traversal. A future Logic view can request one
|
||||
function-scoped projection on demand.
|
||||
|
||||
## Full rebuilds
|
||||
|
||||
Full rebuilds remain mandatory as a fallback and equivalence oracle. Change the adapter version,
|
||||
extractor version, or cache schema whenever old cached facts are no longer valid. Removing the
|
||||
confined extraction cache also forces a clean reparse without affecting canonical files.
|
||||
|
|
@ -49,6 +49,7 @@ gate.
|
|||
- `docforge_propose_node_create`
|
||||
- `docforge_propose_node_update`
|
||||
- `docforge_propose_node_move`
|
||||
- `docforge_propose_relationship_update`
|
||||
- `docforge_propose_node_delete`
|
||||
- `docforge_validate_changeset`
|
||||
- `docforge_get_changeset_diff`
|
||||
|
|
@ -58,6 +59,8 @@ Proposal tools may write only below the configured changeset or isolated preview
|
|||
change canonical files or declared project output. Without `--proposal-writer`, changeset mutation
|
||||
tools return `proposal_access_disabled`. Validation, diff retrieval, and preview remain available
|
||||
for existing changesets. A preview accepts a declared view ID, not a renderer name or command.
|
||||
The relationship-update tool queues additions and removals without rewriting node content and
|
||||
rejects an empty relationship list.
|
||||
|
||||
## Canonical application tool
|
||||
|
||||
|
|
|
|||
|
|
@ -16,7 +16,10 @@ declared rendering, and the Nodes/Flow/Web visualization model documented below.
|
|||
- Disposable SQLite indexing with lexical search, filters, backlinks, dependencies, and impact.
|
||||
- Bounded context profiles for AI agents, including source paths and content hashes.
|
||||
- Isolated, optimistic changesets with create, update, move, delete, validation, diffs, and previews.
|
||||
- Relationship-only changeset operations that do not rewrite node content.
|
||||
- Hash-bound canonical application through both CLI and an explicitly enabled MCP tool.
|
||||
- Opt-in incremental adapter extraction with reverse-dependency invalidation.
|
||||
- Lazy function-scoped logic projections that do not densify the primary graph.
|
||||
- Declared HTML render views. Arbitrary templates, render commands, and output paths are rejected.
|
||||
- A loopback-only graph browser with Nodes, semantic Flow, and convergence Web views,
|
||||
relationship keys, source inspection, branch-aware node hiding, panel resizing, zooming, and
|
||||
|
|
@ -463,6 +466,7 @@ Example MCP client configuration:
|
|||
- `docforge_propose_node_create`
|
||||
- `docforge_propose_node_update`
|
||||
- `docforge_propose_node_move`
|
||||
- `docforge_propose_relationship_update`
|
||||
- `docforge_propose_node_delete`
|
||||
- `docforge_validate_changeset`
|
||||
- `docforge_get_changeset_diff`
|
||||
|
|
@ -487,9 +491,34 @@ Recommended agent sequence:
|
|||
7. Call `docforge_apply_changeset` with that exact hash.
|
||||
8. Report changed canonical files and derived refresh results.
|
||||
|
||||
Use `docforge_propose_relationship_update` when the intended change is only an edge addition or
|
||||
removal. It uses the same underlying validated update contract, but rejects empty relationship
|
||||
lists and makes it explicit that node content will remain unchanged.
|
||||
|
||||
Custom adapters may expose the application tool only when they supply a project-owned
|
||||
`CanonicalApplier`. Core DocForge will not guess how adapter nodes map back to canonical sources.
|
||||
|
||||
## Incremental adapter compilation
|
||||
|
||||
Release 1 complete-projection adapters remain supported. Adapters with large source trees can
|
||||
implement the optional source-scoped manifest and extraction contract. DocForge then fingerprints
|
||||
sources, reuses unchanged facts, reparses changed sources and their reverse dependents, validates a
|
||||
complete candidate graph, and publishes the index atomically.
|
||||
|
||||
Build results report cache hits, reparsed sources, invalidated sources, deleted sources, and total
|
||||
sources. A full projection remains the fallback and equivalence oracle.
|
||||
|
||||
Manual proposals remain separate from compilation. Applying an approved changeset updates
|
||||
canonical sources first. Incremental compilation then notices those changed source fingerprints;
|
||||
it never treats an unapplied proposal as canonical.
|
||||
|
||||
Function-scoped `LogicProjection` data is cached alongside its owning source but remains separate
|
||||
from the primary Nodes, Flow, and Web graph. This is the storage boundary for a future boolean and
|
||||
control-flow view without adding every condition and basic block to ordinary graph traversal.
|
||||
|
||||
See [Incremental Adapter Indexing](INCREMENTAL_INDEXING.md) for the complete contract, cache
|
||||
invalidation rules, manual-application lifecycle, and lazy Logic boundary.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### `stale_index` or `visualization_stale`
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue