yawan
f7fc992f57
Anydoc / anydoc parser engine (push) Canceled after 0s
App / Format, vet, test, and build (push) Canceled after 0s
CLI / test (macos-latest / go 1.26) (push) Canceled after 0s
CLI / test (ubuntu-latest / go 1.26) (push) Canceled after 0s
CLI / test (windows-latest / go 1.26) (push) Canceled after 0s
Build and Push Docker Image / build-ui (push) Canceled after 0s
Build and Push Docker Image / build-docreader (push) Canceled after 0s
Build and Push Docker Image / build-sandbox (push) Canceled after 0s
Build and Push Docker Image / build-app (amd64, linux/amd64, ubuntu-latest) (push) Canceled after 0s
Build and Push Docker Image / build-app (arm64, linux/arm64, ubuntu-24.04-arm) (push) Canceled after 0s
DocReader / Python tests and Go client tests (push) Canceled after 0s
Build & Publish (dsh-weknora) / Typecheck, test, pack (push) Canceled after 0s
Build & Publish (dsh-weknora) / End-to-end inside dsh (pinned) (push) Canceled after 0s
Build & Publish (dsh-weknora) / End-to-end inside dsh (latest, advisory) (push) Canceled after 0s
Frontend / Test, type-check, and build (push) Canceled after 0s
Go Lint Cache / warm lint cache (.) (push) Canceled after 0s
Go Lint Cache / warm lint cache (cli) (push) Canceled after 0s
Go Lint Cache / warm lint cache (client) (push) Canceled after 0s
Build & Publish (mcp-server) / Test (Python 3.10) (push) Canceled after 0s
Build & Publish (mcp-server) / Test (Python 3.11) (push) Canceled after 0s
Build & Publish (mcp-server) / Test (Python 3.12) (push) Canceled after 0s
Build & Publish (mcp-server) / Test (Python 3.13) (push) Canceled after 0s
Build and Push Docker Image / merge (push) Canceled after 0s
Build & Publish (dsh-weknora) / Publish to npm (push) Canceled after 0s
Build & Publish (mcp-server) / Build sdist + wheel (push) Canceled after 0s
Build & Publish (mcp-server) / Publish to PyPI (push) Canceled after 0s
124 lines
5.1 KiB
Markdown
124 lines
5.1 KiB
Markdown
---
|
||
title: 集成向量数据库
|
||
tags: [集成扩展, 向量数据库, 检索引擎, 扩展]
|
||
aliases: [向量数据库, VecDB, 向量检索, RetrieveEngine]
|
||
source: 使用其他向量数据库.md
|
||
---
|
||
|
||
# 集成新的向量数据库
|
||
|
||
本文提供了向 WeKnora 项目添加新向量数据库支持的完整指南。通过实现标准化接口和遵循结构化流程，开发者可以高效地集成自定义向量数据库。
|
||
|
||
> 类似的扩展开发模式参见 [添加网络搜索引擎](../集成扩展/添加网络搜索引擎.md)
|
||
|
||
## 集成流程
|
||
|
||
### 1. 实现基础检索引擎接口
|
||
|
||
```go
|
||
type RetrieveEngine interface {
|
||
EngineType() types.RetrieverEngineType
|
||
Retrieve(ctx context.Context, params types.RetrieveParams) ([]*types.RetrieveResult, error)
|
||
Support() []types.RetrieverType
|
||
}
|
||
```
|
||
|
||
### 2. 实现存储层接口
|
||
|
||
实现 `RetrieveEngineRepository` 接口，扩展基础检索引擎能力，添加索引管理功能：
|
||
|
||
- `Save` / `BatchSave` — 保存索引
|
||
- `EstimateStorageSize` — 估算存储空间
|
||
- `DeleteByChunkIDList` / `DeleteByKnowledgeIDList` — 删除索引
|
||
- `CopyIndices` — 复制索引
|
||
|
||
### 3. 实现服务层接口
|
||
|
||
创建 `RetrieveEngineService` 实现，负责索引创建和管理的业务逻辑。
|
||
|
||
### 4. 添加环境变量配置
|
||
|
||
```
|
||
RETRIEVE_DRIVER=postgres,elasticsearch_v8,your_database
|
||
YOUR_DATABASE_ADDR=host:port
|
||
YOUR_DATABASE_USERNAME=username
|
||
YOUR_DATABASE_PASSWORD=password
|
||
```
|
||
|
||
### 5. 注册检索引擎
|
||
|
||
在 `internal/container/container.go` 的 `initRetrieveEngineRegistry` 中添加初始化与注册逻辑。
|
||
|
||
### 6. 定义检索引擎类型常量
|
||
|
||
在 `internal/types/retriever.go` 中添加新的引擎类型常量。
|
||
|
||
## 参考实现
|
||
|
||
建议参考现有实现：
|
||
|
||
- PostgreSQL: `internal/application/repository/retriever/postgres/`
|
||
- ElasticsearchV7: `internal/application/repository/retriever/elasticsearch/v7/`
|
||
- ElasticsearchV8: `internal/application/repository/retriever/elasticsearch/v8/`
|
||
- Apache Doris 4.1: `internal/application/repository/retriever/doris/`
|
||
- Tencent VectorDB: `internal/application/repository/retriever/tencentvectordb/`
|
||
|
||
## Apache Doris 4.1 集成要点
|
||
|
||
Doris 是 MPP 风格的分析型 SQL 库，其接入方式与 NoSQL 向量库有几处特殊点：
|
||
|
||
- **协议**：MySQL 协议（FE 9030）走主链路；HTTP API（FE 8030）走 Stream Load partial update。
|
||
- **表结构**：每维度一张 `<DORIS_TABLE_PREFIX>_<dim>` 表，`UNIQUE KEY(id)` + `enable_unique_key_merge_on_write=true`。
|
||
- **索引**：INVERTED 索引覆盖过滤字段、`content` 字段使用 `chinese` parser；ANN 索引使用 HNSW + cosine_distance（异步构建，建表后自动轮询 30s 等待 `FINISHED`）。
|
||
- **分数语义**：使用 `1 - cosine_distance_approximate(...)`，threshold 比较 `>=`、`ORDER BY DESC`，与 Qdrant cosine 相似度方向一致。
|
||
- **关键词检索**：基于 Doris 倒排索引的 `MATCH_ANY`，无需在 Go 端做 jieba 分词。
|
||
- **批量字段更新**：`BatchUpdateChunkEnabledStatus` 与 `BatchUpdateChunkTagID` 通过 Stream Load partial update 协议实现，支持 1MiB 自动分批。
|
||
|
||
环境变量：
|
||
|
||
```
|
||
RETRIEVE_DRIVER=doris
|
||
DORIS_ADDR=doris-fe:9030
|
||
DORIS_HTTP_PORT=8030
|
||
DORIS_DATABASE=weknora
|
||
DORIS_USERNAME=root
|
||
DORIS_PASSWORD=
|
||
DORIS_TABLE_PREFIX=weknora_embeddings
|
||
```
|
||
|
||
启动方式：`docker compose --profile doris up -d`，然后在 FE 上 `CREATE DATABASE weknora;`。
|
||
|
||
## Tencent VectorDB
|
||
|
||
WeKnora 内置 Tencent VectorDB 适配器，驱动名为 `tencent_vectordb`。该适配器支持向量检索、基于 BM25 sparse vector 的关键词检索和索引管理，可参与 WeKnora 上层混合检索。
|
||
|
||
```env
|
||
RETRIEVE_DRIVER=tencent_vectordb
|
||
TENCENT_VECTORDB_ADDR=http://your-instance.tencentvectordb.com
|
||
TENCENT_VECTORDB_USERNAME=root
|
||
TENCENT_VECTORDB_API_KEY=your_tencent_vectordb_api_key
|
||
TENCENT_VECTORDB_DATABASE=weknora
|
||
TENCENT_VECTORDB_COLLECTION=weknora_embeddings
|
||
TENCENT_VECTORDB_REPLICA_NUMBER=1
|
||
```
|
||
|
||
`TENCENT_VECTORDB_COLLECTION` 是集合名前缀。WeKnora 会按向量维度创建实际集合，例如 `weknora_embeddings_768`。
|
||
`TENCENT_VECTORDB_REPLICA_NUMBER` 是创建集合时使用的副本数，默认 `1`；单节点 QA 环境可设为 `0`，生产环境可按 Tencent VectorDB 集群规模调整。
|
||
|
||
关键词检索依赖 Tencent VectorDB sparse vector 索引。新建集合会自动创建 `sparse_vector` 索引；旧版本已创建的向量集合如果没有该索引，需要重建集合并重新导入知识库数据后才能启用关键词检索。
|
||
|
||
## 相关主题
|
||
|
||
- [添加网络搜索引擎](../集成扩展/添加网络搜索引擎.md) — 同类扩展开发模式（接口实现 + 注册 + DI）
|
||
- [知识图谱](../核心功能/知识图谱.md) — 知识图谱功能依赖 Neo4j 而非向量数据库
|
||
- [常见问题](../运维排障/常见问题.md) — Embedding 模型配置与向量维度相关
|
||
|
||
---
|
||
|
||
## 反向链接
|
||
|
||
- [Home](../Home.md) — Wiki 首页导航
|
||
- [添加网络搜索引擎](../集成扩展/添加网络搜索引擎.md) — 同类扩展开发模式
|
||
- [知识图谱](../核心功能/知识图谱.md) — 另一种知识检索方式（基于图而非向量）
|
||
- [版本路线图](../项目概述/版本路线图.md) — 路线图中的检索能力扩展方向
|