xinmi/docs/native-graph-migration.md

Native Historical Evidence Graph

Acceptance database: yawan_native_acceptance_20260915, tenant 10059. Source: read-only snapshot /tmp/yawan-graph-source-20260917.sqlite. Snapshot SHA-256: 2643990033bdde26a19729f114f3274600801d703476cfc8f339f572802e324f.

Preserved Records

Migration 200 stores original table names, IDs, graph version IDs, complete JSON payloads, tenant/knowledge-base ownership, and source snapshot hash in PostgreSQL. No model extraction is needed to retain the original graph. The old SQLite and production database were not modified.

Original table	Preserved rows
knowledgegraphversion	5
knowledgeentity	56,149
knowledgerelation	7,627
knowledgegraphevidence	178,985
knowledgegraphchunkstate	107,452
knowledgegraphsourceunit	0
knowledgegraphclaim	0
knowledgeentityalias	0
knowledgegraphcommunity	0
knowledgegraphcommunityreport	0
Total	350,218

The zero-row tables were inspected and are empty in the snapshot. Import replay checks payload equality and table counts, refuses conflicting source records, and preserves existing verified mappings.

Verified Native Mapping

There are 107,397 distinct source chunks referenced by preserved evidence. Of these, 17,025 have verified native counterparts; 90,372 remain available as original evidence without a native chunk link. The verified sources back 24,077 of 178,985 evidence rows. The separate chunk-state table contains 107,452 historical state records, including sources not referenced by evidence.

Knowledge base	Evidence source chunks	Native mappings
产品信息	37	3
功法	9,801	2,073
心法	66,204	7,000
技法	17,154	4,601
销售逻辑	14,201	3,348

Exact raw SHA-256 matching produced zero matches because native ingestion changed chunk boundaries. The successful mapping requires a uniquely identified native source document in the same tenant and knowledge base, then a unique native chunk containing the entire normalized historical excerpt. Normalization is Unicode NFKC plus whitespace removal; excerpts shorter than 80 Unicode codepoints are excluded. Filename conversion from .jsonl to .md restricts candidate documents only and is never sufficient to map a chunk. Duplicate candidate documents and ambiguous chunk matches are left unmapped; the initial pass found 72 ambiguous excerpt matches.

Each mapping stores the current native chunk SHA-256. Evidence links and retrieval recheck that hash, chunk enablement, and live chunk/document/knowledge-base ownership. Modified, disabled, or deleted resources lose their usable link. Overview counts describe migration-time associations, as the UI explicitly labels them; they do not rescan and hash the full corpus on each click.

Retrieval And UI

LEGACY_GRAPH_ENABLE=true activates the historical PostgreSQL graph adapter through the existing graph retrieval interface. Neo4j may remain disabled. A configured Neo4j driver is still supported for existing graph writes, deletes, and additional retrieval. Historical records remain archived when native documents are removed; unavailable native links are not returned.

Search accepts up to eight terms of 200 codepoints, selects at most 16 seeds, expands at most 16 neighboring entities and 32 relations, then fetches up to four verified chunks per entity in a single bounded evidence query. Terms are matched against preserved canonical entity names. The original entity labels and extraction quality are preserved; this migration does not invent new labels or relationships. Legacy chunk IDs are never passed into native RAG.

The authenticated endpoint is GET /api/v1/knowledge-bases/:id/evidence-graph. It uses existing viewer, knowledge-base read, and scoped API-key guards. The knowledge-base Evidence Graph tab supports entity search, pagination, original evidence, relationships, mapping status, and native document opening independently of Wiki.

Reproduction

The scripts restrict writes to isolated acceptance databases matching yawan_native_acceptance_[0-9]+; their default is the database above. The Docker context is colima-yawan, container WeKnora-postgres.

node scripts/migrate_native_graph.mjs /tmp/yawan-graph-source-20260917.sqlite --apply
node scripts/map_native_graph_evidence.mjs --apply
node scripts/verify_native_graph.mjs
node scripts/verify_native_graph_ui.cjs

The UI verifier requires Node 20+, the bundled Playwright package and installed Chrome. It reads the private acceptance login file without printing credentials and targets API port 18080 and frontend port 15173. No credentials are stored in this document.

Validation

• Import replay passed with identical counts and all 17,025 mappings retained.
• PostgreSQL integration tests passed: missing scope, tenant isolation, document scope, unchanged native retrieval, relation-neighbor expansion, bounded results, and rolled-back modified/disabled/deleted resource fixtures.
• Largest knowledge-base overview measured 152-237 ms after removing corpus-wide per-request hashing; representative graph retrieval measured 117-238 ms. These are local acceptance observations, not a production performance guarantee.
• Graph source count query uses an index-only scan; a measured 66,204-source count took 8.6 ms. The migration includes its covering index and replay performs VACUUM ANALYZE after commit.
• Browser verification passed at 1440x1000 and 390x844: authorized 200 and unauthenticated 401, entity details, mapped and unmapped evidence, native document drawer, empty search and reset, no erroneous empty-document guide, no browser errors, and no mobile horizontal overflow.
• vue-tsc --noEmit passed. Targeted Go package compilation and chat-pipeline tests passed.

Browser artifacts are in /Volumes/Extreme SSD/yawan-storage/weknora/acceptance/: native-graph-desktop.png, native-graph-mobile.png, and native-graph-ui-verification.json. Import replay log: /tmp/native-graph-replay-final.log.

The latest repository batching and availability-filter changes require the parent-managed API rebuild. Frontend changes are served by the existing dev server. This task did not restart the acceptance API.