DocForge2/docs/INCREMENTAL_INDEXING.md

# Incremental Adapter Indexing

DocForge Release 1 adapters return one complete immutable projection. That contract remains
supported. The incremental compiler adds an opt-in source-scoped contract that avoids reparsing
unchanged files while preserving the same validated, atomically published graph.

Import these contracts from the public `docforge.adapter_sdk` facade. See
[Legacy Adapters and No-AST Policy](LEGACY_AND_NO_AST.md) for the preserved one-method contract and
[Reference Adapters](REFERENCE_ADAPTERS.md) for the four maintained implementations.

## Release 1 compatibility

The incremental interface is additive:

- An existing adapter implementing only `load_projection()` continues to work unchanged.
- Existing generic projects, descriptors, canonical sources, changesets, and indexes require no
  migration.
- Only adapters implementing both `load_manifest()` and `extract_source()` use the incremental
  path.
- Incremental adapters must still implement `load_projection()` for clean rebuilds and equivalence
  testing.
- Existing adapters receive identical correctness behavior but no incremental speedup until they
  opt in.

## Safety model

Incremental indexing is an extraction optimization. It does not weaken publication:

1. The adapter returns a cheap, deterministic `AdapterManifest`.
2. DocForge compares every source fingerprint and extractor version with the last cache generation.
3. Added, changed, deleted, and reverse-dependent sources are invalidated.
4. The adapter reparses only invalidated sources.
5. DocForge assembles cached and refreshed contributions into a complete candidate projection.
6. The complete graph passes the same validation as a full adapter projection.
7. DocForge rereads the manifest to prove sources remained stable.
8. The extraction cache and SQLite graph are published with atomic file replacement.

An interrupted extraction never replaces the last validated SQLite index. A malformed,
incompatible, or missing cache is a cache miss, not a partial graph.

One cache generation is bounded to 10,000 source contributions and 64,000,000 encoded bytes.
Aggregate graph and Logic assembly limits are described in the
[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md#current-aggregate-bounds).

## Adapter contract

An incremental loader implements the first, second, and fourth methods. It implements
`load_complete_assembly()` as well when it publishes Logic:

```python
class MyAdapter:
    def load_manifest(self) -> AdapterManifest: ...
    def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ...
    def load_complete_assembly(self) -> AdapterAssembly: ...
    def load_projection(self) -> AdapterProjection: ...
```

`load_projection()` remains the deterministic full-rebuild primary-graph oracle. An incremental
adapter that publishes Logic must additionally implement `load_complete_assembly()` as the
cache-independent complete graph-plus-Logic oracle.

Each `AdapterSource` declares:

- A stable source ID.
- A safe project-relative source path.
- A SHA-256 content fingerprint.
- An extractor version.
- Other source IDs whose changes can alter this source's extracted facts.

Each `AdapterSourceProjection` owns:

- Its primary graph nodes.
- Its primary graph relationships, including cross-source relationships owned by that source.
- Optional function-scoped logic projections.

Ownership must be deterministic. Two sources may not produce the same primary node or the same
function logic projection.

Some language tools emit overlapping raw evidence before ownership can be resolved. An incremental
loader may additionally implement:

```python
def assemble_projection(
    manifest: AdapterManifest,
    contributions: tuple[AdapterSourceProjection, ...],
) -> AdapterAssembly: ...
```

DocForge caches and invalidates the source contributions normally, then passes the complete current
contribution set to this pure assembly step. The assembler must deterministically select or merge
overlapping evidence and return one valid final graph and Logic set. It may not read hidden source
state or create a second extraction cache. The final identity, revision, and source hash must match
the manifest exactly.

Without an assembler, the stricter default remains in force: two contributions may not publish the
same primary node or Logic owner.

## Invalidation

DocForge invalidates a source when:

- It is new.
- Its fingerprint changed.
- Its extractor version changed.
- A declared dependency was added, changed, or deleted.
- Any source in its reverse-dependency chain was invalidated.

Deleted sources are omitted from the candidate projection. Their cached dependency declarations
remain available long enough to invalidate surviving dependents.

The manifest describes the current supported filesystem snapshot. It must not retain a missing file
only because Git still tracks it, and it must not require staging before a deletion disappears.
Git-backed adapters must prove that staged and unstaged deletions produce the same current source
set. DocForge then removes the omitted contribution and invalidates its surviving reverse
dependents.

If an adapter cannot precisely describe the affected sources, it should declare broader
dependencies or change its adapter/extractor version. Incorrectly retaining a stale relationship
is never an acceptable optimization.

## Adapter implementation lifecycle

Graph source changes are synchronizable. Changes to the code or configuration implementing the
adapter are not.

`AdapterProject` fingerprints the inferred or explicitly declared implementation boundary when the
project process starts. Every MCP operation validates that boundary before work begins. Changes to
implementation paths or bytes return `adapter_restart_required` with bounded added, changed, and
deleted path evidence. The error is intentionally not auto-repaired through `docforge_sync`; a
fresh process must import and validate the current adapter.

## Build reporting

`build` and `reindex` include an extraction report:

```json
{
  "build": {
    "mode": "incremental",
    "cache_hits": 391,
    "reparsed_sources": 3,
    "invalidated_sources": 3,
    "deleted_sources": 0,
    "total_sources": 394,
    "cache_hit_ids": ["..."],
    "reparsed_source_ids": ["..."]
  }
}
```

Adapters can call `AdapterProject.verify_incremental_equivalence()` in release and contract tests.
The check compares project identity, revision, source hash, nodes, relationships, and Logic against
the independent complete assembly. It also requires the complete assembly's primary graph to match
`load_projection()`.

## Manual changes and relationships

Changesets remain an approval queue, not a compiler queue. Compilation never silently applies a
proposal.

After explicit application:

1. The project-owned applier updates canonical files.
2. Changed manual files receive new fingerprints.
3. Incremental extraction reparses those files and affected dependents.
4. The complete candidate graph is validated and published.
5. Declared renders are regenerated.

`docforge_propose_relationship_update` queues relationship-only additions and removals without
rewriting node content. It is still bound to the changeset's complete base source hash and the
anchor node's expected content hash.

## Lazy logic boundary

`LogicProjection` stores control flow separately from the primary architecture graph. It is owned
by one function or method node and one source extraction.

Logic nodes can represent entries, conditions, basic blocks, calls, convergence points, loops,
returns, and raises. Logic edges retain relation, display label, and deterministic ordinal.
Adapters may leave
logic empty until they implement a language analyzer.

This boundary prevents thousands of boolean expressions and basic blocks from polluting Nodes,
Flow, Web, ordinary search, or architectural traversal. The Logic tab and `docforge_get_logic`
request one function-scoped projection on demand. The built-in analyzers cover Python,
JavaScript, TypeScript, and C++. Python uses the standard-library AST. JavaScript, TypeScript, and
C++ use distinct optional Tree-sitter grammars with thin language-aware control-flow profiles.
Ordinary graph reads use stored projections and do not load or execute these parsers. A grammar
alone supplies syntax, not control-flow meaning, so each new language still needs a profile for
its branch, loop, case, exception, and termination constructs. All analyzers report possible
static paths; they do not claim runtime branch outcomes.

## Manifest and warm-parser scope

Parser work is language-specific and must be measured at the correct boundary:

- Python manifest construction fingerprints source and tokenizes local imports without calling
  `ast.parse`.
- JavaScript and TypeScript manifest construction lexes static relative module specifiers without
  invoking their distinct Tree-sitter extraction parsers. Focused tests prove this behavior on an
  unchanged warm build.
- The C++ reference manifest parses inventoried sources with `tree-sitter-cpp` to discover quoted
  include dependencies. A warm C++ extraction-cache hit is not a zero-parser claim.

The maintained Python benchmark instruments the unchanged warm path and requires zero
`ast.parse` calls and zero `extract_source` calls. That exact zero-parser benchmark claim is Python
only. JavaScript and TypeScript retain focused parser-free-manifest tests; C++ deliberately does
not.

## Full rebuilds

Full rebuilds remain mandatory as a fallback and equivalence oracle. Change the adapter version,
extractor version, or cache schema whenever old cached facts are no longer valid. Removing the
confined extraction cache also forces a clean reparse without affecting canonical files.