Document language adapter construction
This commit is contained in:
parent
cd54cae71d
commit
7bc2ac1e3f
4 changed files with 459 additions and 0 deletions
|
|
@ -124,6 +124,9 @@ DocForge describes them as a source graph.
|
|||
invalidation, equivalence, relationship changes, and the lazy Logic boundary.
|
||||
- [Project onboarding](docs/PROJECT_ONBOARDING.md) — repository assessment, safe manual
|
||||
scaffolding, language frontends, source/manual integration, proof, and MCP activation.
|
||||
- [Language adapter authoring](docs/ADAPTER_AUTHORING_GUIDE.md) — implementation sequence,
|
||||
stable identities, overlap ownership, normalization, incremental equivalence, troubleshooting,
|
||||
and the complete adapter proof matrix.
|
||||
|
||||
## Development
|
||||
|
||||
|
|
|
|||
448
docs/ADAPTER_AUTHORING_GUIDE.md
Normal file
448
docs/ADAPTER_AUTHORING_GUIDE.md
Normal file
|
|
@ -0,0 +1,448 @@
|
|||
# Language Adapter Authoring Guide
|
||||
|
||||
This guide explains how to turn compiler, parser, build-system, or documentation evidence into a
|
||||
deterministic DocForge project graph. It covers the design work that is easy to miss when a small
|
||||
fixture is expanded into a complete repository.
|
||||
|
||||
Use this guide after the repository assessment in
|
||||
[Project Onboarding](PROJECT_ONBOARDING.md). The onboarding checklist decides whether an adapter
|
||||
is needed. This guide defines how to build and prove one.
|
||||
|
||||
## Required outcome
|
||||
|
||||
A production adapter must provide one reproducible public graph from authoritative project
|
||||
evidence. It must not guess facts from filenames, preserve unstable parser identities, publish the
|
||||
same declaration from several extraction units, or let an incremental cache become a second
|
||||
source of truth.
|
||||
|
||||
The complete and incremental paths must publish exactly the same:
|
||||
|
||||
- project identity, adapter identity, revision, and source hash;
|
||||
- nodes and stable node identities;
|
||||
- relationships and their deterministic metadata;
|
||||
- function Logic projections and owners;
|
||||
- source paths and anchors;
|
||||
- validation failures for invalid input.
|
||||
|
||||
Performance does not relax this requirement. A fast graph that sometimes retains stale or
|
||||
translation-unit-dependent facts is invalid.
|
||||
|
||||
## The four adapter layers
|
||||
|
||||
Keep these layers separate:
|
||||
|
||||
1. **Evidence discovery** finds the authoritative build and source inputs.
|
||||
2. **Source extraction** converts one extraction unit into raw, deterministic evidence.
|
||||
3. **Assembly** resolves overlap and assigns each published fact to one owner.
|
||||
4. **DocForge publication** validates, caches, indexes, queries, and visualizes the assembled graph.
|
||||
|
||||
DocForge supplies the publication contracts. A language frontend owns the first three layers
|
||||
because only the frontend understands the language's build semantics, identity rules, generated
|
||||
evidence, and source ownership.
|
||||
|
||||
The contracts are:
|
||||
|
||||
```python
|
||||
class MyAdapter:
|
||||
def load_manifest(self) -> AdapterManifest: ...
|
||||
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ...
|
||||
def assemble_projection(
|
||||
self,
|
||||
manifest: AdapterManifest,
|
||||
contributions: tuple[AdapterSourceProjection, ...],
|
||||
) -> AdapterAssembly: ...
|
||||
def load_projection(self) -> AdapterProjection: ...
|
||||
```
|
||||
|
||||
`assemble_projection()` is optional only when extraction units already have disjoint ownership.
|
||||
`load_projection()` is always required. It is the clean rebuild and equivalence oracle.
|
||||
|
||||
## Step 1: define authority before parsing
|
||||
|
||||
Write down which tool owns each fact before implementing extraction.
|
||||
|
||||
Typical authorities include:
|
||||
|
||||
| Fact | Suitable authority |
|
||||
|---|---|
|
||||
| Source inventory | Build graph, workspace manifest, or declared source roots |
|
||||
| Active flags and features | Build-system output |
|
||||
| Declaration identity | Compiler or language-service semantic identity |
|
||||
| Definition location | Compiler or parser source location |
|
||||
| Calls and inheritance | Resolved semantic evidence |
|
||||
| Include or module dependencies | Compiler or build graph |
|
||||
| Function control-flow shape | A language-aware analyzer |
|
||||
| Documentation meaning | Canonical documentation sources |
|
||||
|
||||
Do not let a syntax parser overrule a compiler on semantics. Do not infer a resolved call merely
|
||||
because names match. When the selected frontend cannot prove a fact, omit it or label it as
|
||||
explicitly derived or proposed.
|
||||
|
||||
Record:
|
||||
|
||||
- frontend and compiler versions;
|
||||
- build-system and feature configuration;
|
||||
- supported source and generated-source roots;
|
||||
- supported declaration and relationship kinds;
|
||||
- unsupported facts;
|
||||
- any normalization applied to frontend output.
|
||||
|
||||
Changing one of these rules normally requires an extractor or adapter version change.
|
||||
|
||||
## Step 2: define the extraction unit
|
||||
|
||||
An extraction unit is the smallest input that can be fingerprinted, invalidated, and re-extracted
|
||||
without hidden state.
|
||||
|
||||
Examples include:
|
||||
|
||||
- one compiler translation unit;
|
||||
- one module or crate target;
|
||||
- one Java source set or compilation unit;
|
||||
- one canonical documentation file;
|
||||
- one generated API registry snapshot.
|
||||
|
||||
Use the build system's real unit rather than an arbitrary file grouping. One source file is not
|
||||
necessarily one semantic unit when features, generated files, module resolution, or compiler flags
|
||||
change its meaning.
|
||||
|
||||
Every unit needs:
|
||||
|
||||
- a stable source ID;
|
||||
- a project-confined relative path;
|
||||
- a content or semantic fingerprint;
|
||||
- an extractor version;
|
||||
- direct dependencies whose changes can alter its evidence.
|
||||
|
||||
Do not include output paths, temporary directories, wall time, process IDs, pointer values, or
|
||||
unordered container iteration in any identity or fingerprint.
|
||||
|
||||
## Step 3: prove one bounded extraction
|
||||
|
||||
Begin with one representative unit. Include enough language behavior to expose identity and
|
||||
ownership problems:
|
||||
|
||||
- declaration and definition;
|
||||
- nested and out-of-line ownership;
|
||||
- resolved call;
|
||||
- inheritance or interface implementation;
|
||||
- an internal or private symbol;
|
||||
- one function suitable for Logic extraction;
|
||||
- one shared declaration imported by another unit.
|
||||
|
||||
The first proof should establish:
|
||||
|
||||
- repeated byte-stable extraction;
|
||||
- project-root confinement;
|
||||
- stable source and symbol IDs;
|
||||
- exact source paths and anchors;
|
||||
- relationship endpoint validity;
|
||||
- exact Logic ownership;
|
||||
- explicit rejection of unsafe or ambiguous build input.
|
||||
|
||||
Do not activate the adapter in a routine project session at this point. A correct single-unit
|
||||
projection does not prove whole-project ownership.
|
||||
|
||||
## Step 4: design stable identities
|
||||
|
||||
Stable identity is a semantic design decision, not a serialization detail.
|
||||
|
||||
Prefer, in order:
|
||||
|
||||
1. a stable compiler or language-tool symbol identity;
|
||||
2. a normalized semantic key built from qualified ownership, symbol kind, and signature;
|
||||
3. a source-qualified identity only for symbols whose language visibility is source-local.
|
||||
|
||||
Avoid:
|
||||
|
||||
- parser object addresses or transient declaration IDs;
|
||||
- traversal order;
|
||||
- result-set position;
|
||||
- source line as the entire identity;
|
||||
- a display name without namespace, owner, or signature;
|
||||
- one identity policy for both externally shared and source-local symbols.
|
||||
|
||||
Definitions and declarations of the same externally visible symbol must converge. Anonymous,
|
||||
private-to-unit, or internal-linkage symbols must remain distinct when the language makes them
|
||||
distinct.
|
||||
|
||||
Every normalization used inside an identity must be tested against more than one extraction unit.
|
||||
Some compiler fields change only after a symbol is instantiated, referenced, or fully evaluated.
|
||||
If such a field is not part of authored identity, remove or normalize it before hashing.
|
||||
|
||||
## Step 5: separate raw evidence from the public graph
|
||||
|
||||
Many semantic tools repeat the same declaration in every unit that imports a header, module,
|
||||
crate, package, or generated interface. That repetition is valid raw evidence but invalid public
|
||||
ownership.
|
||||
|
||||
Do not force raw extraction to guess the final owner before the complete dependency inventory is
|
||||
known. Instead:
|
||||
|
||||
1. extract deterministic raw contributions;
|
||||
2. inventory the complete current contribution set;
|
||||
3. assign each shared source or symbol to one deterministic owner;
|
||||
4. publish each node, relationship, and Logic record once;
|
||||
5. reject conflicting evidence instead of silently choosing incompatible values.
|
||||
|
||||
The raw contribution cache may contain overlap. The assembled DocForge graph may not.
|
||||
|
||||
### Ownership rules
|
||||
|
||||
Define ownership for every published fact. A common policy is:
|
||||
|
||||
| Fact | Recommended owner |
|
||||
|---|---|
|
||||
| Shared source declaration | Deterministically selected dependent unit or declared module owner |
|
||||
| Unit-private symbol | Its extraction unit |
|
||||
| Function Logic | The unit owning the exact function definition |
|
||||
| Containment | The owner of the contained member |
|
||||
| Call relationship | The owner of the caller |
|
||||
| Inheritance relationship | The owner of the derived type |
|
||||
| Source declaration/definition edge | The owner of the source or symbol selected by the adapter |
|
||||
|
||||
The correct policy depends on the language. What matters is that it is explicit, deterministic,
|
||||
and identical in complete and incremental assembly.
|
||||
|
||||
Relationship metadata also needs deterministic reduction. Repeated evidence locations must not
|
||||
grow with the number or order of extraction units. Select one stable evidence record, or define a
|
||||
bounded ordered representation with a documented reason.
|
||||
|
||||
## Step 6: normalize frontend-generated noise
|
||||
|
||||
Whole-project extraction exposes facts that a single fixture will not. Compilers and language
|
||||
services may emit:
|
||||
|
||||
- implicit template or generic instantiations;
|
||||
- synthesized methods and bridge functions;
|
||||
- default constructors or defaulted functions;
|
||||
- inferred exception or effect annotations;
|
||||
- generated annotation-processor output;
|
||||
- macro expansions;
|
||||
- duplicate declarations with different amounts of semantic completion;
|
||||
- declarations from libraries outside the project root.
|
||||
|
||||
For each category, choose one policy:
|
||||
|
||||
- publish as authored evidence;
|
||||
- publish as derived evidence with a stable identity;
|
||||
- attach to an authored owner without becoming a primary node;
|
||||
- omit as compiler-use noise.
|
||||
|
||||
Do not keep a field merely because the frontend emits it. If it changes based on whether another
|
||||
unit uses the symbol, it will break determinism or complete/incremental parity unless normalized.
|
||||
|
||||
Add a regression fixture for every normalization rule. The fixture should demonstrate the
|
||||
unstable input and the expected stable output.
|
||||
|
||||
## Step 7: compose the complete reference graph
|
||||
|
||||
Run the frontend over the complete supported source inventory and apply the final ownership
|
||||
partition.
|
||||
|
||||
The composition must:
|
||||
|
||||
- sort units and outputs deterministically;
|
||||
- converge declarations and definitions;
|
||||
- preserve source-local identity;
|
||||
- resolve semantic parents rather than relying only on visual parser nesting;
|
||||
- reject duplicate node identities with conflicting semantic metadata;
|
||||
- reject conflicting Logic for one owner;
|
||||
- reject duplicate source IDs;
|
||||
- reject missing relationship endpoints;
|
||||
- retain explicit status when optional Logic analysis cannot parse a function.
|
||||
|
||||
Run two independent complete extractions and compare exact serialized projections or their
|
||||
canonical fingerprints.
|
||||
|
||||
Record:
|
||||
|
||||
- extraction-unit count;
|
||||
- node, relationship, and Logic counts;
|
||||
- relationship counts by important type;
|
||||
- duration and peak memory;
|
||||
- output or index size;
|
||||
- exact fingerprint.
|
||||
|
||||
These measurements establish the reference shape. They are not performance promises across
|
||||
machines.
|
||||
|
||||
## Step 8: add dependency-aware incremental extraction
|
||||
|
||||
Use authoritative dependency evidence whenever possible:
|
||||
|
||||
- compiler dependency output;
|
||||
- module or crate graph;
|
||||
- Maven or Gradle compilation graph;
|
||||
- generated-source and annotation-processor inputs;
|
||||
- explicitly declared documentation dependencies.
|
||||
|
||||
The manifest must include every unit that can invalidate a cached contribution. A dependency-only
|
||||
unit may publish an empty contribution; it still needs a fingerprint and stable ID so reverse
|
||||
dependents are invalidated.
|
||||
|
||||
The incremental path is:
|
||||
|
||||
1. build the current manifest;
|
||||
2. compare fingerprints, extractor versions, additions, and deletions;
|
||||
3. invalidate changed units and their reverse dependents;
|
||||
4. reuse only valid raw contributions;
|
||||
5. extract invalidated units;
|
||||
6. assemble the complete current contribution set;
|
||||
7. validate the candidate graph;
|
||||
8. reread the manifest to detect concurrent source changes;
|
||||
9. atomically publish the cache and index.
|
||||
|
||||
Missing, incompatible, or corrupt cache data is a cache miss. It must never become a partial graph
|
||||
or replace the last valid index.
|
||||
|
||||
## Step 9: keep the complete path independent
|
||||
|
||||
The full rebuild must not read the incremental extraction cache. Otherwise equivalence compares
|
||||
the cache with itself and cannot detect stale or incorrectly owned facts.
|
||||
|
||||
The complete path must independently:
|
||||
|
||||
- rediscover the supported source inventory;
|
||||
- extract every semantic unit;
|
||||
- apply the same normalization rules;
|
||||
- apply the same public ownership partition;
|
||||
- generate the same project metadata and source hash;
|
||||
- publish the same nodes, relationships, and Logic.
|
||||
|
||||
Raw unpartitioned frontend output is not the equivalence oracle when the incremental assembler
|
||||
publishes a partitioned graph. Both paths must compare the same public projection shape.
|
||||
|
||||
## Step 10: integrate manual and source projections
|
||||
|
||||
Keep canonical manual and derived source projections independently rebuildable. Compose them at a
|
||||
session boundary rather than making source extraction rewrite documentation.
|
||||
|
||||
Decide:
|
||||
|
||||
- which sessions receive source nodes;
|
||||
- which documentation families remain isolated;
|
||||
- whether active context includes source nodes or manual guidance only;
|
||||
- how manual nodes link to implementation nodes;
|
||||
- what happens when required build evidence is absent;
|
||||
- whether source support is required, optional with a null fallback, or disabled.
|
||||
|
||||
The source graph has no authority to edit runtime code or canonical documentation. MCP, viewer,
|
||||
cache, and index operation remain bound to one explicit project root.
|
||||
|
||||
## Required proof matrix
|
||||
|
||||
An adapter is not complete until these cases pass:
|
||||
|
||||
| Case | Required result |
|
||||
|---|---|
|
||||
| Repeated single-unit extraction | Exact stable output |
|
||||
| Two independent complete builds | Exact public projection equality |
|
||||
| Cold incremental build | Every current unit extracted once |
|
||||
| Unchanged warm build | Zero reparses |
|
||||
| Implementation/source change | Only the unit and declared dependents reparse |
|
||||
| Shared header/module change | Every reverse dependent reparses |
|
||||
| Added source | New contribution appears without stale duplicates |
|
||||
| Renamed source | Old contribution disappears and new identity follows policy |
|
||||
| Deleted source | Owned nodes, relationships, and Logic disappear |
|
||||
| Build flags/features change | Affected units invalidate |
|
||||
| Extractor version change | Old contributions invalidate |
|
||||
| Corrupt cache | Clean recovery without partial publication |
|
||||
| Interrupted extraction | Last validated index remains active |
|
||||
| Complete versus incremental | Exact equality |
|
||||
| Cross-session isolation | Unconfigured sessions cannot see the source graph |
|
||||
| Viewer and MCP retrieval | Exact symbols, relationships, and Logic are retrievable |
|
||||
|
||||
Fixtures must include overlapping shared declarations. A single-file fixture cannot prove
|
||||
assembly.
|
||||
|
||||
## Performance review
|
||||
|
||||
Measure before and after adding incremental extraction:
|
||||
|
||||
- complete extraction wall time and peak resident memory;
|
||||
- warm manifest, assembly, validation, and index time;
|
||||
- raw cache and final index size;
|
||||
- cache-unit count and cache-hit count;
|
||||
- node, relationship, and Logic counts;
|
||||
- checked query latency.
|
||||
|
||||
If warm operation remains expensive, identify whether the cost is dependency discovery,
|
||||
fingerprinting, assembly, validation, or indexing. Do not hide staleness behind an arbitrary time
|
||||
window. An optimization must retain same-operation source-change detection or replace it with an
|
||||
equally explicit freshness contract.
|
||||
|
||||
## Troubleshooting by symptom
|
||||
|
||||
### Node counts change between identical complete builds
|
||||
|
||||
Check for unstable frontend IDs, unordered output, generated declarations, inferred type or
|
||||
exception information, and source paths containing temporary directories.
|
||||
|
||||
### The same header or module node appears in many contributions
|
||||
|
||||
Keep the overlap in raw evidence and add deterministic assembly ownership. Do not publish every
|
||||
copy and rely on index deduplication.
|
||||
|
||||
### Complete and incremental graphs differ
|
||||
|
||||
Compare the public ownership partition first. Confirm that the complete path does not compare
|
||||
unpartitioned raw output with assembled incremental output. Then compare normalization versions,
|
||||
dependency inventories, deleted units, and relationship ownership.
|
||||
|
||||
### Warm builds consume nearly complete-build memory
|
||||
|
||||
Check whether cached raw contributions are too verbose, whether the assembler retains all
|
||||
frontend AST data, and whether dependency discovery reparses semantic source. Cache only the
|
||||
bounded projection required for deterministic assembly.
|
||||
|
||||
### Relationship counts grow when another unit includes the same source
|
||||
|
||||
Define a relationship owner and deterministic evidence reduction. Do not concatenate repeated
|
||||
evidence from every unit.
|
||||
|
||||
### A function has conflicting Logic
|
||||
|
||||
Attach Logic only to the exact semantic definition owner. A syntax analyzer may find a similar
|
||||
function, but line or symbol agreement with the semantic frontend is required before publication.
|
||||
|
||||
### An unchanged query is slow
|
||||
|
||||
Measure manifest revalidation separately from SQLite lookup. Preserve freshness; optimize the
|
||||
authoritative dependency and fingerprint path rather than skipping it silently.
|
||||
|
||||
## Change and release rules
|
||||
|
||||
Change the extractor version when parsing, normalization, identity, relationship, Logic, or
|
||||
dependency behavior can alter a source contribution.
|
||||
|
||||
Change the adapter version when assembly, project metadata, session composition, or public graph
|
||||
policy changes.
|
||||
|
||||
Change the cache schema version when older serialized contributions cannot be read safely.
|
||||
|
||||
For every such change:
|
||||
|
||||
1. add a fixture for the behavior;
|
||||
2. run the complete proof matrix;
|
||||
3. prove clean recovery from the previous disposable cache;
|
||||
4. record before-and-after graph counts and fingerprints;
|
||||
5. update the adapter's operating guide and unsupported-fact list.
|
||||
|
||||
## Completion checklist
|
||||
|
||||
- [ ] Authority for every extracted fact is recorded.
|
||||
- [ ] Build evidence and source inventory are reproducible.
|
||||
- [ ] Stable identities distinguish shared and source-local symbols correctly.
|
||||
- [ ] One representative unit extracts deterministically.
|
||||
- [ ] Shared declarations and relationships have explicit public owners.
|
||||
- [ ] Frontend-generated noise has documented normalization rules.
|
||||
- [ ] Two independent complete builds match exactly.
|
||||
- [ ] Incremental extraction uses authoritative dependencies.
|
||||
- [ ] The complete oracle is independent of the cache.
|
||||
- [ ] Complete and incremental public projections match exactly.
|
||||
- [ ] Corrupt, missing, and interrupted cache cases fail safely.
|
||||
- [ ] Session composition and family isolation are proven.
|
||||
- [ ] Viewer, query, context, and Logic retrieval are proven.
|
||||
- [ ] Performance, graph shape, unsupported facts, and version rules are recorded.
|
||||
|
||||
|
|
@ -136,6 +136,11 @@ optional assembly contract. Cache the raw source contributions through DocForge,
|
|||
deterministically select or merge ownership from the complete contribution set. Do not hide a
|
||||
second extraction cache inside the project adapter.
|
||||
|
||||
Before implementing a frontend, read the
|
||||
[Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md). It defines the complete extraction,
|
||||
identity, ownership, normalization, incremental-equivalence, troubleshooting, and proof route that
|
||||
this checklist summarizes.
|
||||
|
||||
### 5. Build-system evidence
|
||||
|
||||
#### C and C++
|
||||
|
|
|
|||
|
|
@ -117,6 +117,9 @@ docforge --project-root /absolute/path/MyProject onboard \
|
|||
Scaffolding refuses to replace existing target files. It leaves source-graph status at
|
||||
`adapter_required` until a project integration implements and proves the adapter contract.
|
||||
See [Project onboarding](PROJECT_ONBOARDING.md) for the complete language-neutral checklist.
|
||||
Use the [Language Adapter Authoring Guide](ADAPTER_AUTHORING_GUIDE.md) when implementing that
|
||||
frontend. It covers stable identities, overlapping compiler evidence, deterministic ownership,
|
||||
normalization, incremental equivalence, failure recovery, and the required proof matrix.
|
||||
|
||||
### Configure a generic project
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue