DocForge2/docs/INCREMENTAL_INDEXING.md

Incremental Adapter Indexing

DocForge Release 1 adapters return one complete immutable projection. That contract remains supported. The incremental compiler adds an opt-in source-scoped contract that avoids reparsing unchanged files while preserving the same validated, atomically published graph.

Import these contracts from the public docforge.adapter_sdk facade. See Legacy Adapters and No-AST Policy for the preserved one-method contract and Reference Adapters for the four maintained implementations.

Release 1 compatibility

The incremental interface is additive:

• An existing adapter implementing only load_projection() continues to work unchanged.
• Existing generic projects, descriptors, canonical sources, changesets, and indexes require no migration.
• Only adapters implementing both load_manifest() and extract_source() use the incremental path.
• Incremental adapters must still implement load_projection() for clean rebuilds and equivalence testing.
• Existing adapters receive identical correctness behavior but no incremental speedup until they opt in.

Safety model

Incremental indexing is an extraction optimization. It does not weaken publication:

1. The adapter returns a cheap, deterministic AdapterManifest.
2. DocForge compares every source fingerprint and extractor version with the last cache generation.
3. Added, changed, deleted, and reverse-dependent sources are invalidated.
4. The adapter reparses only invalidated sources.
5. DocForge assembles cached and refreshed contributions into a complete candidate projection.
6. The complete graph passes the same validation as a full adapter projection.
7. DocForge rereads the manifest to prove sources remained stable.
8. The extraction cache and SQLite graph are published with atomic file replacement.

An interrupted extraction never replaces the last validated SQLite index. A malformed, incompatible, or missing cache is a cache miss, not a partial graph.

One cache generation is bounded to 10,000 source contributions and 64,000,000 encoded bytes. Aggregate graph and Logic assembly limits are described in the Language Adapter Authoring Guide.

Adapter contract

An incremental loader implements the first, second, and fourth methods. It implements load_complete_assembly() as well when it publishes Logic:

class MyAdapter:
    def load_manifest(self) -> AdapterManifest: ...
    def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ...
    def load_complete_assembly(self) -> AdapterAssembly: ...
    def load_projection(self) -> AdapterProjection: ...

load_projection() remains the deterministic full-rebuild primary-graph oracle. An incremental adapter that publishes Logic must additionally implement load_complete_assembly() as the cache-independent complete graph-plus-Logic oracle.

Each AdapterSource declares:

• A stable source ID.
• A safe project-relative source path.
• A SHA-256 content fingerprint.
• An extractor version.
• Other source IDs whose changes can alter this source's extracted facts.

Each AdapterSourceProjection owns:

• Its primary graph nodes.
• Its primary graph relationships, including cross-source relationships owned by that source.
• Optional function-scoped logic projections.

Ownership must be deterministic. Two sources may not produce the same primary node or the same function logic projection.

Some language tools emit overlapping raw evidence before ownership can be resolved. An incremental loader may additionally implement:

def assemble_projection(
    manifest: AdapterManifest,
    contributions: tuple[AdapterSourceProjection, ...],
) -> AdapterAssembly: ...

DocForge caches and invalidates the source contributions normally, then passes the complete current contribution set to this pure assembly step. The assembler must deterministically select or merge overlapping evidence and return one valid final graph and Logic set. It may not read hidden source state or create a second extraction cache. The final identity, revision, and source hash must match the manifest exactly.

Without an assembler, the stricter default remains in force: two contributions may not publish the same primary node or Logic owner.

Invalidation

DocForge invalidates a source when:

• It is new.
• Its fingerprint changed.
• Its extractor version changed.
• A declared dependency was added, changed, or deleted.
• Any source in its reverse-dependency chain was invalidated.

Deleted sources are omitted from the candidate projection. Their cached dependency declarations remain available long enough to invalidate surviving dependents.

The manifest describes the current supported filesystem snapshot. It must not retain a missing file only because Git still tracks it, and it must not require staging before a deletion disappears. Git-backed adapters must prove that staged and unstaged deletions produce the same current source set. DocForge then removes the omitted contribution and invalidates its surviving reverse dependents.

If an adapter cannot precisely describe the affected sources, it should declare broader dependencies or change its adapter/extractor version. Incorrectly retaining a stale relationship is never an acceptable optimization.

Adapter implementation lifecycle

Graph source changes are synchronizable. Changes to the code or configuration implementing the adapter are not.

AdapterProject fingerprints the inferred or explicitly declared implementation boundary when the project process starts. Every MCP operation validates that boundary before work begins. Changes to implementation paths or bytes return adapter_restart_required with bounded added, changed, and deleted path evidence. The error is intentionally not auto-repaired through docforge_sync; a fresh process must import and validate the current adapter.

Build reporting

build and reindex include an extraction report:

{
  "build": {
    "mode": "incremental",
    "cache_hits": 391,
    "reparsed_sources": 3,
    "invalidated_sources": 3,
    "deleted_sources": 0,
    "total_sources": 394,
    "cache_hit_ids": ["..."],
    "reparsed_source_ids": ["..."]
  }
}

Adapters can call AdapterProject.verify_incremental_equivalence() in release and contract tests. The check compares project identity, revision, source hash, nodes, relationships, and Logic against the independent complete assembly. It also requires the complete assembly's primary graph to match load_projection().

Manual changes and relationships

Changesets remain an approval queue, not a compiler queue. Compilation never silently applies a proposal.

After explicit application:

1. The project-owned applier updates canonical files.
2. Changed manual files receive new fingerprints.
3. Incremental extraction reparses those files and affected dependents.
4. The complete candidate graph is validated and published.
5. Declared renders are regenerated.

docforge_propose_relationship_update queues relationship-only additions and removals without rewriting node content. It is still bound to the changeset's complete base source hash and the anchor node's expected content hash.

Lazy logic boundary

LogicProjection stores control flow separately from the primary architecture graph. It is owned by one function or method node and one source extraction.

Logic nodes can represent entries, conditions, basic blocks, calls, convergence points, loops, returns, and raises. Logic edges retain relation, display label, and deterministic ordinal. Adapters may leave logic empty until they implement a language analyzer.

This boundary prevents thousands of boolean expressions and basic blocks from polluting Nodes, Flow, Web, ordinary search, or architectural traversal. The Logic tab and docforge_get_logic request one function-scoped projection on demand. The built-in analyzers cover Python, JavaScript, TypeScript, and C++. Python uses the standard-library AST. JavaScript, TypeScript, and C++ use distinct optional Tree-sitter grammars with thin language-aware control-flow profiles. Ordinary graph reads use stored projections and do not load or execute these parsers. A grammar alone supplies syntax, not control-flow meaning, so each new language still needs a profile for its branch, loop, case, exception, and termination constructs. All analyzers report possible static paths; they do not claim runtime branch outcomes.

Manifest and warm-parser scope

Parser work is language-specific and must be measured at the correct boundary:

• Python manifest construction fingerprints source and tokenizes local imports without calling ast.parse.
• JavaScript and TypeScript manifest construction lexes static relative module specifiers without invoking their distinct Tree-sitter extraction parsers. Focused tests prove this behavior on an unchanged warm build.
• The C++ reference manifest parses inventoried sources with tree-sitter-cpp to discover quoted include dependencies. A warm C++ extraction-cache hit is not a zero-parser claim.

The maintained Python benchmark instruments the unchanged warm path and requires zero ast.parse calls and zero extract_source calls. That exact zero-parser benchmark claim is Python only. JavaScript and TypeScript retain focused parser-free-manifest tests; C++ deliberately does not.

Full rebuilds

Full rebuilds remain mandatory as a fallback and equivalence oracle. Change the adapter version, extractor version, or cache schema whenever old cached facts are no longer valid. Removing the confined extraction cache also forces a clean reparse without affecting canonical files.