WeKnora deployment source with the reviewed wiki identity, summary and recovery fixes

docreader/Makefile

@@ -0,0 +1,23 @@
+.PHONY: proto build run docker-build docker-run clean
+
+# 生成 protobuf 代码
+proto:
+	@echo "Generating protobuf code..."
+	@sh ./scripts/generate_proto.sh
+
+# 构建 Go 客户端
+build:
+	@echo "Building Go client..."
+	@go build -o bin/client ./src/client
+
+# 运行 Python 服务
+run:
+	@echo "Running Python server..."
+	@python src/server/server.py
+
+# 清理
+clean:
+	@echo "Cleaning up..."
+	@rm -rf bin/
+	@find . -name "*.pyc" -delete
+	@find . -name "__pycache__" -delete 

docreader/README.md

@@ -0,0 +1,229 @@
+# DocReader Service
+
+DocReader 是 WeKnora 项目中负责文档解析和处理的 gRPC 服务。它支持多种文档格式的读取、OCR 识别、多模态处理等功能。
+
+## Docker Compose 环境变量配置
+
+在 `docker-compose.yml` 文件中，docreader 服务配置了以下环境变量：
+
+```yaml
+docreader:
+  image: wechatopenai/weknora-docreader:${WEKNORA_VERSION:-latest}
+  environment:
+    - MINIO_ENDPOINT=minio:9000
+    - MINIO_PUBLIC_ENDPOINT=http://localhost:${MINIO_PORT:-9000}
+    - MINERU_ENDPOINT=${MINERU_ENDPOINT:-}
+    - MAX_FILE_SIZE_MB=${MAX_FILE_SIZE_MB:-}
+```
+
+### 环境变量说明
+
+#### 1. MINIO_ENDPOINT
+
+- **说明**: MinIO 服务的内部访问地址（容器间通信）
+- **默认值**: `minio:9000`
+- **用途**: DocReader 服务使用此地址连接到 MinIO 对象存储服务，用于读取和存储文档处理过程中的文件
+- **配置示例**:
+  ```yaml
+  - MINIO_ENDPOINT=minio:9000  # Docker 网络内部地址
+  ```
+
+#### 2. MINIO_PUBLIC_ENDPOINT
+
+- **说明**: MinIO 服务的公开访问地址（外部访问）
+- **默认值**: `http://localhost:9000`
+- **用途**: 用于生成可从外部访问的文件 URL，例如在文档解析后返回图片链接时使用
+- **重要提示**: 
+  - 如果需要从其他设备或容器访问，需要将 `localhost` 替换为实际的 IP 地址
+  - 可以在 `.env` 文件中配置 `MINIO_PORT` 来自定义端口
+- **配置示例**:
+  ```bash
+  # .env 文件
+  MINIO_PORT=9000
+  ```
+  或直接在 docker-compose.yml 中修改：
+  ```yaml
+  - MINIO_PUBLIC_ENDPOINT=http://192.168.1.100:9000  # 使用实际 IP
+  ```
+
+#### 3. MINERU_ENDPOINT
+
+- **说明**: MinerU 服务的访问地址（可选）
+- **默认值**: 空（不使用 MinerU）
+- **用途**: MinerU 是一个高级文档解析服务，支持更复杂的文档结构识别和处理。配置此变量后，DocReader 可以调用 MinerU 进行文档解析
+- **配置示例**:
+  ```bash
+  # .env 文件
+  MINERU_ENDPOINT=http://mineru-service:8080
+  ```
+
+#### 4. MAX_FILE_SIZE_MB
+
+- **说明**: 允许上传的最大文件大小（单位：MB）
+- **默认值**: `50` MB
+- **用途**: 限制 gRPC 服务接收的文件大小，防止过大的文件导致服务崩溃或性能问题
+- **配置示例**:
+  ```bash
+  # .env 文件
+  MAX_FILE_SIZE_MB=100  # 允许最大 100MB 的文件
+  ```
+
+## 其他可配置的环境变量
+
+除了 docker-compose.yml 中已配置的变量外，DocReader 还支持以下环境变量（可根据需要添加）：
+
+### gRPC 配置
+
+- `DOCREADER_GRPC_MAX_WORKERS`: gRPC 服务的最大工作线程数（默认：4）
+- `DOCREADER_GRPC_PORT`: gRPC 服务监听端口（默认：50051）
+
+### 解析器资源控制
+
+- `DOCREADER_MARKITDOWN_MAX_WORKERS`: MarkItDown 解析的最大并发数（默认：1，设为 0 可关闭限流）
+- `DOCREADER_PDF_RENDER_MAX_WORKERS`: 扫描 PDF 渲染为图片的最大并发数（默认：1，设为 0 可关闭限流）
+- `DOCREADER_PDF_RENDER_DPI`: 扫描 PDF 渲染 DPI（默认：200）
+- `DOCREADER_PDF_JPEG_QUALITY`: 扫描 PDF 输出 JPEG 质量（默认：85，范围会自动限制在 1-95）
+
+### OCR / VLM
+
+DocReader 自身不再内置 OCR 与 VLM 后端。扫描 PDF 会被渲染为 JPEG 图片后交由 Go App 侧调用 OCR/VLM 服务处理，相关配置请参考主项目文档。
+
+### 存储配置
+
+DocReader 支持多种存储后端：
+
+#### MinIO/S3 存储（推荐）
+
+- `STORAGE_TYPE`: 设置为 `minio`
+- `MINIO_ACCESS_KEY_ID`: MinIO 访问密钥 ID（默认：minioadmin）
+- `MINIO_SECRET_ACCESS_KEY`: MinIO 访问密钥（默认：minioadmin）
+- `MINIO_BUCKET_NAME`: MinIO 存储桶名称（默认：WeKnora）
+- `MINIO_PATH_PREFIX`: 文件路径前缀
+- `MINIO_USE_SSL`: 是否使用 SSL（默认：false）
+
+#### 腾讯云 COS 存储
+
+- `STORAGE_TYPE`: 设置为 `cos`
+- `COS_SECRET_ID`: COS 访问密钥 ID
+- `COS_SECRET_KEY`: COS 访问密钥
+- `COS_REGION`: COS 区域
+- `COS_BUCKET_NAME`: COS 存储桶名称
+- `COS_APP_ID`: COS 应用 ID
+- `COS_PATH_PREFIX`: 文件路径前缀
+- `COS_ENABLE_OLD_DOMAIN`: 是否使用旧域名（默认：true）
+
+#### 阿里云 OSS 存储
+
+- `STORAGE_TYPE`: 设置为 `oss`
+- `OSS_ACCESS_KEY_ID`: OSS 访问密钥 ID
+- `OSS_ACCESS_KEY_SECRET`: OSS 访问密钥
+- `OSS_ENDPOINT`: OSS 端点（如 `oss-cn-hangzhou.aliyuncs.com`）
+- `OSS_BUCKET_NAME`: OSS 存储桶名称
+- `OSS_REGION`: OSS 区域（如 `cn-hangzhou`）
+- `OSS_PATH_PREFIX`: 文件路径前缀
+
+### 代理配置
+
+如果需要通过代理访问外部服务：
+
+- `EXTERNAL_HTTP_PROXY`: HTTP 代理地址
+- `EXTERNAL_HTTPS_PROXY`: HTTPS 代理地址
+
+### 图像处理配置
+
+扫描 PDF 会被渲染为 JPEG 图片后交给 Go App 侧 OCR 处理。如果在导入多个大 PDF 时出现资源占用过高，
+可以优先调低 `DOCREADER_PDF_RENDER_MAX_WORKERS` 或 `DOCREADER_MARKITDOWN_MAX_WORKERS`。
+
+## 配置示例
+
+### 基础配置（使用 MinIO）
+
+```yaml
+docreader:
+  environment:
+    - MINIO_ENDPOINT=minio:9000
+    - MINIO_PUBLIC_ENDPOINT=http://localhost:9000
+    - MAX_FILE_SIZE_MB=50
+```
+
+### 高级配置（启用 MinerU）
+
+```yaml
+docreader:
+  environment:
+    - MINIO_ENDPOINT=minio:9000
+    - MINIO_PUBLIC_ENDPOINT=http://192.168.1.100:9000
+    - MINERU_ENDPOINT=http://mineru:8080
+    - MAX_FILE_SIZE_MB=100
+```
+
+### 使用腾讯云 COS
+
+```yaml
+docreader:
+  environment:
+    - STORAGE_TYPE=cos
+    - COS_SECRET_ID=your_secret_id
+    - COS_SECRET_KEY=your_secret_key
+    - COS_REGION=ap-guangzhou
+    - COS_BUCKET_NAME=your-bucket
+    - COS_APP_ID=your_app_id
+    - MAX_FILE_SIZE_MB=50
+```
+
+### 使用阿里云 OSS
+
+```yaml
+docreader:
+  environment:
+    - STORAGE_TYPE=oss
+    - OSS_ACCESS_KEY_ID=your_access_key_id
+    - OSS_ACCESS_KEY_SECRET=your_access_key_secret
+    - OSS_ENDPOINT=oss-cn-hangzhou.aliyuncs.com
+    - OSS_BUCKET_NAME=your-bucket
+    - OSS_REGION=cn-hangzhou
+    - MAX_FILE_SIZE_MB=50
+```
+
+## 常见问题
+
+### 1. DocReader 服务无法启动？
+
+检查容器日志中是否存在依赖缺失或权限相关错误，必要时确认 `MINIO_ENDPOINT` / 存储相关环境变量是否正确配置。
+
+### 2. 图片无法显示？
+
+检查 `MINIO_PUBLIC_ENDPOINT` 配置：
+- 确保使用的是可从浏览器访问的地址
+- 如果从其他设备访问，不要使用 `localhost`，应使用实际 IP 地址
+
+### 3. 文件上传失败？
+
+检查 `MAX_FILE_SIZE_MB` 配置，确保限制足够大。同时需要确保前端和后端服务的文件大小限制保持一致。
+
+## 服务健康检查
+
+DocReader 服务配置了健康检查：
+
+```yaml
+healthcheck:
+  test: ["CMD", "grpc_health_probe", "-addr=localhost:50051"]
+  interval: 30s
+  timeout: 10s
+  retries: 3
+  start_period: 60s
+```
+
+可以通过以下命令检查服务状态：
+
+```bash
+docker ps | grep docreader
+docker logs WeKnora-docreader
+```
+
+## 更多信息
+
+- 服务端口：50051（gRPC）
+- 容器名称：WeKnora-docreader
+- 网络：WeKnora-network
+- 重启策略：unless-stopped

docreader/auth.py

@@ -0,0 +1,198 @@
+"""gRPC TLS 和认证模块
+
+环境变量配置：
+    TLS 相关：
+        GRPC_TLS_ENABLED: 是否启用 TLS（true/false），默认 false
+        GRPC_TLS_CERT: TLS 证书文件路径（GRPC_TLS_ENABLED=true 时必填）
+        GRPC_TLS_KEY: TLS 私钥文件路径（GRPC_TLS_ENABLED=true 时必填）
+        GRPC_TLS_CA: CA 证书路径
+        GRPC_MTLS_REQUIRE_CLIENT_CERT: 设为 true 时启用 mTLS，要求客户端
+            出示由 GRPC_TLS_CA 签发的证书。未设置时默认按 GRPC_TLS_CA 是否
+            存在自动判断（保留向后兼容）。
+
+    认证相关：
+        GRPC_AUTH_TOKEN: 认证 Token，如果设置则启用认证
+
+注意：当 GRPC_TLS_ENABLED=true 但任何 TLS 配置项缺失或加载失败时，
+本模块会抛出异常以触发 fail-fast，避免静默降级到明文。
+"""
+
+import hmac
+import logging
+import os
+from typing import Optional
+
+import grpc
+
+logger = logging.getLogger(__name__)
+
+
+class TLSConfigError(RuntimeError):
+    """TLS 配置错误，用于 fail-fast。"""
+
+
+def _env_bool(name: str, default: bool = False) -> bool:
+    raw = os.getenv(name)
+    if raw is None:
+        return default
+    return raw.strip().lower() in ("true", "1", "yes", "on")
+
+
+def load_tls_credentials() -> Optional[grpc.ServerCredentials]:
+    """构建 server 端 TLS 凭据。
+
+    GRPC_TLS_ENABLED=false 时返回 None；为 true 时如果配置无效会抛出
+    TLSConfigError，由调用方决定是否终止启动。
+    """
+    if not _env_bool("GRPC_TLS_ENABLED", False):
+        logger.info("TLS disabled (GRPC_TLS_ENABLED is not 'true')")
+        return None
+
+    cert_path = os.getenv("GRPC_TLS_CERT")
+    key_path = os.getenv("GRPC_TLS_KEY")
+
+    if not cert_path or not key_path:
+        raise TLSConfigError(
+            "GRPC_TLS_ENABLED=true but GRPC_TLS_CERT/GRPC_TLS_KEY not set; "
+            "refusing to start in plaintext mode"
+        )
+
+    try:
+        with open(cert_path, "rb") as f:
+            cert_chain = f.read()
+        with open(key_path, "rb") as f:
+            private_key = f.read()
+    except OSError as e:
+        raise TLSConfigError(f"failed to read TLS cert/key: {e}") from e
+
+    ca_path = os.getenv("GRPC_TLS_CA")
+    require_client_auth = _env_bool(
+        "GRPC_MTLS_REQUIRE_CLIENT_CERT",
+        default=bool(ca_path),
+    )
+
+    if require_client_auth and not ca_path:
+        raise TLSConfigError(
+            "GRPC_MTLS_REQUIRE_CLIENT_CERT=true requires GRPC_TLS_CA to be set"
+        )
+
+    if ca_path:
+        try:
+            with open(ca_path, "rb") as f:
+                ca_cert = f.read()
+        except OSError as e:
+            raise TLSConfigError(f"failed to read CA cert: {e}") from e
+        credentials = grpc.ssl_server_credentials(
+            [(private_key, cert_chain)],
+            root_certificates=ca_cert,
+            require_client_auth=require_client_auth,
+        )
+        if require_client_auth:
+            logger.info("TLS enabled with mTLS (mutual authentication)")
+        else:
+            logger.info("TLS enabled with CA configured (client auth optional)")
+    else:
+        credentials = grpc.ssl_server_credentials([(private_key, cert_chain)])
+        logger.info("TLS enabled (1-way)")
+
+    return credentials
+
+
+# gRPC 健康检查标准服务路径，需在鉴权前放行，便于 K8s/Docker 探活。
+_HEALTH_METHODS = frozenset(
+    {
+        "/grpc.health.v1.Health/Check",
+        "/grpc.health.v1.Health/Watch",
+    }
+)
+
+
+def _make_abort_handler(original: Optional[grpc.RpcMethodHandler]) -> grpc.RpcMethodHandler:
+    """为给定的原 handler 构造一个匹配 RPC kind 的鉴权失败 handler。
+
+    如果原 handler 不可用（理论上不会发生，但作为兜底），返回 unary_unary。
+    直接返回 None 而不是 set_code 也可，但显式调用 abort 能确保框架按
+    UNAUTHENTICATED 收尾，且匹配 kind 防止 grpc 触发 INTERNAL。
+    """
+    def _abort(_request, context):
+        context.abort(
+            grpc.StatusCode.UNAUTHENTICATED,
+            "Invalid or missing authentication token",
+        )
+
+    def _abort_stream(_request, context):
+        context.abort(
+            grpc.StatusCode.UNAUTHENTICATED,
+            "Invalid or missing authentication token",
+        )
+        return
+        yield  # pragma: no cover - make this a generator
+
+    if original is None or original.unary_unary is not None:
+        return grpc.unary_unary_rpc_method_handler(
+            _abort,
+            request_deserializer=getattr(original, "request_deserializer", None),
+            response_serializer=getattr(original, "response_serializer", None),
+        )
+    if original.unary_stream is not None:
+        return grpc.unary_stream_rpc_method_handler(
+            _abort_stream,
+            request_deserializer=original.request_deserializer,
+            response_serializer=original.response_serializer,
+        )
+    if original.stream_unary is not None:
+        return grpc.stream_unary_rpc_method_handler(
+            _abort,
+            request_deserializer=original.request_deserializer,
+            response_serializer=original.response_serializer,
+        )
+    return grpc.stream_stream_rpc_method_handler(
+        _abort_stream,
+        request_deserializer=original.request_deserializer,
+        response_serializer=original.response_serializer,
+    )
+
+
+class AuthInterceptor(grpc.ServerInterceptor):
+    """Token 认证拦截器
+
+    环境变量配置：
+        GRPC_AUTH_TOKEN: 认证 Token，如果设置则启用认证
+
+    客户端需要在 metadata 中传递 Token：
+        - key: "authorization"
+        - value: "Bearer <token>" 或直接 "<token>"
+    """
+
+    def __init__(self) -> None:
+        token = os.getenv("GRPC_AUTH_TOKEN") or ""
+        self.auth_token: Optional[bytes] = token.encode("utf-8") if token else None
+        if self.auth_token:
+            if len(self.auth_token) < 16:
+                logger.warning(
+                    "GRPC_AUTH_TOKEN is shorter than 16 bytes; consider a stronger token"
+                )
+            logger.info("Token authentication enabled")
+        else:
+            logger.warning("Token authentication disabled (GRPC_AUTH_TOKEN not set)")
+
+    def intercept_service(self, continuation, handler_call_details):
+        if not self.auth_token:
+            return continuation(handler_call_details)
+
+        method = handler_call_details.method
+        if method in _HEALTH_METHODS:
+            return continuation(handler_call_details)
+
+        metadata = dict(handler_call_details.invocation_metadata or [])
+        raw = metadata.get("authorization", "") or ""
+        if raw.startswith("Bearer "):
+            raw = raw[7:]
+        token_bytes = raw.encode("utf-8")
+
+        if not hmac.compare_digest(token_bytes, self.auth_token):
+            logger.warning("Authentication failed for method: %s", method)
+            original = continuation(handler_call_details)
+            return _make_abort_handler(original)
+
+        return continuation(handler_call_details)

docreader/client/auth.go

@@ -0,0 +1,134 @@
+// Package client provides a docreader gRPC client and the shared TLS / token
+// authentication helpers used by both the standalone Go SDK in this package
+// and the internal docparser wrapper. Keep all auth/TLS construction here so
+// the two call sites cannot drift on security defaults.
+package client
+
+import (
+	"context"
+	"crypto/tls"
+	"crypto/x509"
+	"fmt"
+	"os"
+
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/credentials"
+	"google.golang.org/grpc/credentials/insecure"
+)
+
+// AuthConfig holds the docreader gRPC client TLS / token configuration.
+type AuthConfig struct {
+	TLSEnabled bool
+	CertFile   string
+	KeyFile    string
+	CAFile     string
+	// ServerName overrides the SNI / certificate-host check on the client.
+	// When empty, the address passed to Dial is used by Go's TLS stack.
+	ServerName string
+
+	AuthToken string
+}
+
+// LoadAuthConfigFromEnv reads docreader gRPC auth knobs from the process
+// environment. The caller must pass the result to BuildDialOptions to apply
+// them to a gRPC connection.
+func LoadAuthConfigFromEnv() *AuthConfig {
+	return &AuthConfig{
+		TLSEnabled: os.Getenv("GRPC_TLS_ENABLED") == "true",
+		CertFile:   os.Getenv("GRPC_TLS_CERT"),
+		KeyFile:    os.Getenv("GRPC_TLS_KEY"),
+		CAFile:     os.Getenv("GRPC_TLS_CA"),
+		ServerName: os.Getenv("GRPC_TLS_SERVER_NAME"),
+		AuthToken:  os.Getenv("GRPC_AUTH_TOKEN"),
+	}
+}
+
+// BuildDialOptions returns the gRPC DialOptions that apply the configured
+// transport credentials and per-RPC token. Callers should append their own
+// per-call options (load balancer, message size, etc.).
+func (c *AuthConfig) BuildDialOptions(maxMsgSize int) ([]grpc.DialOption, error) {
+	opts := []grpc.DialOption{
+		grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy":"round_robin"}`),
+		grpc.WithDefaultCallOptions(
+			grpc.MaxCallRecvMsgSize(maxMsgSize),
+			grpc.MaxCallSendMsgSize(maxMsgSize),
+		),
+	}
+
+	if c.TLSEnabled {
+		creds, err := c.buildTLSCredentials()
+		if err != nil {
+			return nil, fmt.Errorf("failed to build TLS credentials: %w", err)
+		}
+		opts = append(opts, grpc.WithTransportCredentials(creds))
+		Logger.Printf("INFO: TLS enabled for gRPC client")
+	} else {
+		opts = append(opts, grpc.WithTransportCredentials(insecure.NewCredentials()))
+	}
+
+	if c.AuthToken != "" {
+		// Only allow per-RPC tokens to ride a secured channel. This mirrors
+		// gRPC's own oauth2 credentials behaviour and prevents the bearer
+		// token from leaking on plaintext connections.
+		opts = append(opts, grpc.WithPerRPCCredentials(&tokenAuth{
+			token:           c.AuthToken,
+			requireTLSGuard: c.TLSEnabled,
+		}))
+		Logger.Printf("INFO: Token authentication enabled for gRPC client (TLS=%v)", c.TLSEnabled)
+	}
+
+	return opts, nil
+}
+
+func (c *AuthConfig) buildTLSCredentials() (credentials.TransportCredentials, error) {
+	tlsConfig := &tls.Config{
+		MinVersion: tls.VersionTLS12,
+		ServerName: c.ServerName,
+	}
+
+	if c.CAFile != "" {
+		caCert, err := os.ReadFile(c.CAFile)
+		if err != nil {
+			return nil, fmt.Errorf("failed to read CA certificate: %w", err)
+		}
+		certPool := x509.NewCertPool()
+		if !certPool.AppendCertsFromPEM(caCert) {
+			return nil, fmt.Errorf("failed to parse CA certificate")
+		}
+		tlsConfig.RootCAs = certPool
+	}
+
+	switch {
+	case c.CertFile != "" && c.KeyFile != "":
+		cert, err := tls.LoadX509KeyPair(c.CertFile, c.KeyFile)
+		if err != nil {
+			return nil, fmt.Errorf("failed to load client certificate: %w", err)
+		}
+		tlsConfig.Certificates = []tls.Certificate{cert}
+		Logger.Printf("INFO: mTLS enabled (client certificate loaded)")
+	case c.CertFile != "" || c.KeyFile != "":
+		return nil, fmt.Errorf(
+			"GRPC_TLS_CERT and GRPC_TLS_KEY must be set together for mTLS",
+		)
+	}
+
+	return credentials.NewTLS(tlsConfig), nil
+}
+
+type tokenAuth struct {
+	token string
+	// requireTLSGuard mirrors AuthConfig.TLSEnabled; we expose it via
+	// RequireTransportSecurity so gRPC will refuse to send the bearer token
+	// over an insecure connection when the operator has enabled TLS.
+	requireTLSGuard bool
+}
+
+func (t *tokenAuth) GetRequestMetadata(ctx context.Context, uri ...string) (map[string]string, error) {
+	return map[string]string{
+		"authorization": "Bearer " + t.token,
+	}, nil
+}
+
+func (t *tokenAuth) RequireTransportSecurity() bool {
+	return t.requireTLSGuard
+}

docreader/client/client.go

@@ -0,0 +1,113 @@
+package client
+
+import (
+	"fmt"
+	"log"
+	"os"
+	"strconv"
+	"time"
+
+	"github.com/Tencent/WeKnora/docreader/proto"
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/resolver"
+)
+
+func getMaxMessageSize() int {
+	if sizeStr := os.Getenv("MAX_FILE_SIZE_MB"); sizeStr != "" {
+		if size, err := strconv.Atoi(sizeStr); err == nil && size > 0 {
+			if size > 20*1024 {
+				size = 20 * 1024
+			}
+			return size * 1024 * 1024
+		}
+	}
+	return 50 * 1024 * 1024
+}
+
+var Logger = log.New(os.Stdout, "[DocReader] ", log.LstdFlags|log.Lmicroseconds)
+
+// ImageRefInfo represents an image reference from a converted document.
+type ImageRefInfo struct {
+	Filename    string
+	OriginalRef string
+	MimeType    string
+	StorageKey  string
+}
+
+// Client represents a DocReader service client.
+type Client struct {
+	conn *grpc.ClientConn
+	proto.DocReaderClient
+	debug bool
+}
+
+func NewClient(addr string) (*Client, error) {
+	authConfig := LoadAuthConfigFromEnv()
+	return NewClientWithAuth(addr, authConfig)
+}
+
+func NewClientWithAuth(addr string, authConfig *AuthConfig) (*Client, error) {
+	Logger.Printf("INFO: Creating new DocReader client connecting to %s", addr)
+
+	maxMsgSize := getMaxMessageSize()
+
+	if authConfig == nil {
+		authConfig = &AuthConfig{}
+	}
+
+	opts, err := authConfig.BuildDialOptions(maxMsgSize)
+	if err != nil {
+		Logger.Printf("ERROR: Failed to build dial options: %v", err)
+		return nil, err
+	}
+
+	resolver.SetDefaultScheme("dns")
+
+	startTime := time.Now()
+	conn, err := grpc.Dial("dns:///"+addr, opts...)
+	if err != nil {
+		Logger.Printf("ERROR: Failed to connect to DocReader service: %v", err)
+		return nil, err
+	}
+	Logger.Printf("INFO: Successfully connected to DocReader service in %v", time.Since(startTime))
+
+	return &Client{
+		conn:            conn,
+		DocReaderClient: proto.NewDocReaderClient(conn),
+		debug:           false,
+	}, nil
+}
+
+func (c *Client) Close() error {
+	Logger.Printf("INFO: Closing DocReader client connection")
+	return c.conn.Close()
+}
+
+func (c *Client) SetDebug(debug bool) {
+	c.debug = debug
+}
+
+func (c *Client) Log(level string, format string, args ...interface{}) {
+	if level == "DEBUG" && !c.debug {
+		return
+	}
+	Logger.Printf("%s: %s", level, fmt.Sprintf(format, args...))
+}
+
+// GetImageRefsFromResponse extracts image references from a ReadResponse.
+func GetImageRefsFromResponse(resp *proto.ReadResponse) []ImageRefInfo {
+	if resp == nil || len(resp.ImageRefs) == 0 {
+		return nil
+	}
+
+	refs := make([]ImageRefInfo, 0, len(resp.ImageRefs))
+	for _, ref := range resp.ImageRefs {
+		refs = append(refs, ImageRefInfo{
+			Filename:    ref.Filename,
+			OriginalRef: ref.OriginalRef,
+			MimeType:    ref.MimeType,
+			StorageKey:  ref.StorageKey,
+		})
+	}
+	return refs
+}

docreader/client/client_test.go

@@ -0,0 +1,95 @@
+package client
+
+import (
+	"context"
+	"log"
+	"os"
+	"testing"
+	"time"
+
+	"github.com/Tencent/WeKnora/docreader/proto"
+)
+
+func init() {
+	log.SetOutput(os.Stdout)
+	log.SetFlags(log.LstdFlags | log.Lmicroseconds | log.Lshortfile)
+	log.Println("INFO: Initializing DocReader client tests")
+}
+
+func requireLiveDocReaderClient(t *testing.T) *Client {
+	t.Helper()
+
+	client, err := NewClient("localhost:50051")
+	if err != nil {
+		t.Fatalf("Failed to create client: %v", err)
+	}
+	t.Cleanup(func() { client.Close() })
+
+	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
+	defer cancel()
+	if _, err := client.ListEngines(ctx, &proto.ListEnginesRequest{}); err != nil {
+		t.Skipf("DocReader gRPC server not available at localhost:50051: %v", err)
+	}
+
+	return client
+}
+
+func TestReadURL(t *testing.T) {
+	client := requireLiveDocReaderClient(t)
+	client.SetDebug(true)
+
+	startTime := time.Now()
+	resp, err := client.Read(
+		context.Background(),
+		&proto.ReadRequest{
+			Url:   "https://example.com",
+			Title: "test",
+		},
+	)
+	log.Printf("INFO: Read(URL) completed in %v", time.Since(startTime))
+
+	if err != nil {
+		t.Fatalf("Read failed: %v", err)
+	}
+	if resp.Error != "" {
+		t.Fatalf("Read returned error: %s", resp.Error)
+	}
+	if resp.MarkdownContent == "" {
+		t.Error("Expected non-empty markdown content")
+	}
+	log.Printf("INFO: content_len=%d, images=%d", len(resp.MarkdownContent), len(resp.ImageRefs))
+}
+
+func TestReadFile(t *testing.T) {
+	client := requireLiveDocReaderClient(t)
+	client.SetDebug(true)
+
+	fileContent, err := os.ReadFile("../testdata/test.md")
+	if err != nil {
+		t.Fatalf("Failed to read test file: %v", err)
+	}
+
+	startTime := time.Now()
+	resp, err := client.Read(
+		context.Background(),
+		&proto.ReadRequest{
+			FileContent: fileContent,
+			FileName:    "test.md",
+			FileType:    "md",
+		},
+	)
+	log.Printf("INFO: Read(file) completed in %v", time.Since(startTime))
+
+	if err != nil {
+		t.Fatalf("Read failed: %v", err)
+	}
+	if resp.Error != "" {
+		t.Fatalf("Read returned error: %s", resp.Error)
+	}
+	if resp.MarkdownContent == "" {
+		t.Error("Expected non-empty markdown content")
+	}
+
+	imageRefs := GetImageRefsFromResponse(resp)
+	log.Printf("INFO: content_len=%d, images=%d", len(resp.MarkdownContent), len(imageRefs))
+}

docreader/config.py

@@ -0,0 +1,187 @@
+import logging
+import os
+from dataclasses import dataclass
+from typing import Any, Dict, Iterable, Optional, Tuple
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+def _get_first_env(keys: Iterable[str]) -> Tuple[Optional[str], Optional[str]]:
+    """Return (value, key) for the first existing env var in keys."""
+    for k in keys:
+        if k in os.environ:
+            return os.environ.get(k), k
+    return None, None
+
+
+def _get_str(keys: Iterable[str], default: str = "") -> str:
+    v, _ = _get_first_env(keys)
+    return default if v is None else str(v)
+
+
+def _get_int(keys: Iterable[str], default: int) -> int:
+    v, _ = _get_first_env(keys)
+    if v is None or str(v).strip() == "":
+        return default
+    try:
+        return int(str(v).strip())
+    except Exception:
+        return default
+
+
+def _get_bool(keys: Iterable[str], default: bool) -> bool:
+    v, _ = _get_first_env(keys)
+    if v is None or str(v).strip() == "":
+        return default
+    return str(v).strip().lower() in {"1", "true", "yes", "y", "on"}
+
+
+def _mask_secret(v: str) -> str:
+    if not v:
+        return ""
+    if len(v) <= 6:
+        return "***"
+    return f"{v[:2]}***{v[-2:]}"
+
+
+@dataclass(frozen=True)
+class DocReaderConfig:
+    # gRPC
+    grpc_max_workers: int
+    grpc_max_file_size_mb: int
+    grpc_port: int
+
+    # Parser
+    docx_max_pages: int
+    markitdown_max_workers: int
+    odl_max_workers: int
+    odl_hybrid: str
+    odl_hybrid_url: str
+    odl_hybrid_mode: str
+    odl_hybrid_fallback: bool
+    odl_markdown_with_html: bool
+    pdf_render_max_workers: int
+    pdf_render_parallelism: int
+    pdf_render_dpi: int
+    pdf_jpeg_quality: int
+    pdf_render_max_edge: int
+
+    # Proxy
+    external_http_proxy: str
+    external_https_proxy: str
+
+    # Temp image output directory (shared with Go app via volume, local mode fallback)
+    image_output_dir: str
+
+
+def load_config() -> DocReaderConfig:
+    """Load config from environment variables (lightweight version)."""
+
+    grpc_max_workers = _get_int(["DOCREADER_GRPC_MAX_WORKERS", "GRPC_MAX_WORKERS"], 4)
+    grpc_max_file_size_mb = (
+        _get_int(["DOCREADER_GRPC_MAX_FILE_SIZE_MB", "MAX_FILE_SIZE_MB"], 50)
+        * 1024
+        * 1024
+    )
+    grpc_port = _get_int(["DOCREADER_GRPC_PORT", "PORT"], 50051)
+    docx_max_pages = _get_int(["DOCREADER_DOCX_MAX_PAGES"], 0)
+    markitdown_max_workers = _get_int(["DOCREADER_MARKITDOWN_MAX_WORKERS"], 1)
+    odl_max_workers = _get_int(["DOCREADER_ODL_MAX_WORKERS"], 1)
+    odl_hybrid = _get_str(["DOCREADER_ODL_HYBRID"], "off")
+    odl_hybrid_url = _get_str(
+        ["DOCREADER_ODL_HYBRID_URL"],
+        "http://127.0.0.1:5002",
+    )
+    odl_hybrid_mode = _get_str(["DOCREADER_ODL_HYBRID_MODE"], "auto")
+    odl_hybrid_fallback = _get_bool(["DOCREADER_ODL_HYBRID_FALLBACK"], False)
+    odl_markdown_with_html = _get_bool(
+        ["DOCREADER_ODL_MARKDOWN_WITH_HTML"], False
+    )
+    pdf_render_max_workers = _get_int(["DOCREADER_PDF_RENDER_MAX_WORKERS"], 1)
+    # Intra-document render parallelism: how many worker processes render the
+    # scanned pages of a SINGLE PDF in parallel. pdfium is not thread-safe, so
+    # page rendering is fanned out across processes (each opens its own
+    # document). A large scanned PDF rendered serially can take >1h on
+    # CPU-constrained containers; this is the main lever to cut that wall time.
+    # Default scales with CPU count but is capped so we don't oversubscribe.
+    _cpu = os.cpu_count() or 1
+    pdf_render_parallelism = _get_int(
+        ["DOCREADER_PDF_RENDER_PARALLELISM"], max(1, min(4, _cpu))
+    )
+    pdf_render_dpi = _get_int(["DOCREADER_PDF_RENDER_DPI"], 200)
+    pdf_jpeg_quality = _get_int(["DOCREADER_PDF_JPEG_QUALITY"], 85)
+    # Cap the long edge (px) of rendered/extracted page images. Without this,
+    # PDFs declaring very large page boxes render to 100+ MP JPEGs that blow the
+    # gRPC message limit (and are far higher-res than OCR needs). ~2000px keeps
+    # dense CJK text legible for OCR while keeping page images well under ~1MB.
+    pdf_render_max_edge = _get_int(["DOCREADER_PDF_RENDER_MAX_EDGE"], 2000)
+
+    external_http_proxy = _get_str(
+        ["DOCREADER_EXTERNAL_HTTP_PROXY", "EXTERNAL_HTTP_PROXY"], ""
+    )
+    external_https_proxy = _get_str(
+        ["DOCREADER_EXTERNAL_HTTPS_PROXY", "EXTERNAL_HTTPS_PROXY"], ""
+    )
+
+    image_output_dir = _get_str(
+        ["DOCREADER_IMAGE_OUTPUT_DIR", "IMAGE_OUTPUT_DIR"], "/tmp/docreader"
+    )
+
+    return DocReaderConfig(
+        grpc_max_workers=grpc_max_workers,
+        grpc_max_file_size_mb=grpc_max_file_size_mb,
+        grpc_port=grpc_port,
+        docx_max_pages=docx_max_pages,
+        markitdown_max_workers=markitdown_max_workers,
+        odl_max_workers=odl_max_workers,
+        odl_hybrid=odl_hybrid,
+        odl_hybrid_url=odl_hybrid_url,
+        odl_hybrid_mode=odl_hybrid_mode,
+        odl_hybrid_fallback=odl_hybrid_fallback,
+        odl_markdown_with_html=odl_markdown_with_html,
+        pdf_render_max_workers=pdf_render_max_workers,
+        pdf_render_parallelism=pdf_render_parallelism,
+        pdf_render_dpi=pdf_render_dpi,
+        pdf_jpeg_quality=pdf_jpeg_quality,
+        pdf_render_max_edge=pdf_render_max_edge,
+        external_http_proxy=external_http_proxy,
+        external_https_proxy=external_https_proxy,
+        image_output_dir=image_output_dir,
+    )
+
+
+CONFIG = load_config()
+
+
+def dump_config(mask_secrets: bool = True) -> Dict[str, Any]:
+    cfg = CONFIG
+    d: Dict[str, Any] = {
+        "DOCREADER_GRPC_MAX_WORKERS": cfg.grpc_max_workers,
+        "DOCREADER_GRPC_MAX_FILE_SIZE_MB": cfg.grpc_max_file_size_mb,
+        "DOCREADER_GRPC_PORT": cfg.grpc_port,
+        "DOCREADER_DOCX_MAX_PAGES": cfg.docx_max_pages,
+        "DOCREADER_MARKITDOWN_MAX_WORKERS": cfg.markitdown_max_workers,
+        "DOCREADER_ODL_MAX_WORKERS": cfg.odl_max_workers,
+        "DOCREADER_ODL_HYBRID": cfg.odl_hybrid,
+        "DOCREADER_ODL_HYBRID_URL": cfg.odl_hybrid_url,
+        "DOCREADER_ODL_HYBRID_MODE": cfg.odl_hybrid_mode,
+        "DOCREADER_ODL_HYBRID_FALLBACK": cfg.odl_hybrid_fallback,
+        "DOCREADER_ODL_MARKDOWN_WITH_HTML": cfg.odl_markdown_with_html,
+        "DOCREADER_PDF_RENDER_MAX_WORKERS": cfg.pdf_render_max_workers,
+        "DOCREADER_PDF_RENDER_PARALLELISM": cfg.pdf_render_parallelism,
+        "DOCREADER_PDF_RENDER_DPI": cfg.pdf_render_dpi,
+        "DOCREADER_PDF_JPEG_QUALITY": cfg.pdf_jpeg_quality,
+        "DOCREADER_PDF_RENDER_MAX_EDGE": cfg.pdf_render_max_edge,
+        "DOCREADER_EXTERNAL_HTTP_PROXY": cfg.external_http_proxy,
+        "DOCREADER_EXTERNAL_HTTPS_PROXY": cfg.external_https_proxy,
+        "DOCREADER_IMAGE_OUTPUT_DIR": cfg.image_output_dir,
+    }
+    return d
+
+
+def print_config() -> None:
+    d = dump_config(mask_secrets=True)
+    logger.info("DocReader env/config (effective values):")
+    for k in sorted(d.keys()):
+        logger.info("%s=%s", k, d[k])

docreader/main.py

@@ -0,0 +1,338 @@
+import logging
+import os
+import re
+import sys
+import traceback
+import uuid
+from concurrent import futures
+from typing import Optional
+
+import grpc
+from grpc_health.v1 import health_pb2_grpc
+from grpc_health.v1.health import HealthServicer
+
+from docreader.auth import AuthInterceptor, TLSConfigError, load_tls_credentials
+from docreader import config
+from docreader.config import CONFIG
+from docreader.parser import Parser
+from docreader.proto import docreader_pb2_grpc
+from docreader.parser.registry import registry
+from docreader.proto.docreader_pb2 import (
+    ReadRequest,
+    ReadResponse,
+    ImageRef,
+    ReadStreamMeta,
+    ReadStreamResponse,
+    ListEnginesResponse,
+    ParserEngineInfo,
+)
+from docreader.utils.request import init_logging_request_id, request_id_context
+
+_SURROGATE_RE = re.compile(r"[\ud800-\udfff]")
+
+
+def to_valid_utf8_text(s: Optional[str]) -> str:
+    if not s:
+        return ""
+    s = _SURROGATE_RE.sub("\ufffd", s)
+    return s.encode("utf-8", errors="replace").decode("utf-8")
+
+
+for handler in logging.root.handlers[:]:
+    logging.root.removeHandler(handler)
+
+handler = logging.StreamHandler(sys.stdout)
+logging.root.addHandler(handler)
+
+_level_name = (os.environ.get("LOG_LEVEL") or "INFO").upper()
+_level = getattr(logging, _level_name, logging.INFO)
+logging.root.setLevel(_level)
+
+logger = logging.getLogger(__name__)
+logger.info("Initializing server logging, level=%s", _level_name)
+
+init_logging_request_id()
+
+
+def _resolve_images(
+    images: dict, request_id: str, storage_map: dict | None = None
+) -> tuple[str, list]:
+    """Resolve document images into inline bytes for the Go App to persist.
+
+    ``images`` is a dict of {relative_path: raw_data} where raw_data is
+    base64-encoded string or raw bytes.
+
+    The Go App is solely responsible for persisting images to the configured
+    storage backend (local/minio/cos/tos). This function only decodes images
+    and returns them as inline bytes via ImageRef.
+
+    Returns ("", list[ImageRef]).  image_dir_path is always empty.
+    """
+    import base64
+
+    if not images:
+        return "", []
+
+    mime_map = {
+        ".png": "image/png",
+        ".jpg": "image/jpeg",
+        ".jpeg": "image/jpeg",
+        ".gif": "image/gif",
+        ".webp": "image/webp",
+        ".bmp": "image/bmp",
+    }
+
+    refs = []
+    for ref_path, b64data in images.items():
+        try:
+            img_bytes = base64.b64decode(b64data)
+        except Exception:
+            img_bytes = b64data.encode("utf-8") if isinstance(b64data, str) else b64data
+
+        fname = os.path.basename(ref_path) or f"{uuid.uuid4().hex}.png"
+        ext = os.path.splitext(fname)[1].lower()
+        mime = mime_map.get(ext, "application/octet-stream")
+
+        refs.append(
+            ImageRef(
+                filename=fname,
+                original_ref=ref_path,
+                mime_type=mime,
+                image_data=img_bytes,
+            )
+        )
+
+    logger.info("Resolved %d images (mode=inline)", len(refs))
+    return "", refs
+
+
+def _mime_for_ref(ref_path: str) -> tuple[str, str]:
+    """Return (filename, mime_type) for an image reference path."""
+    mime_map = {
+        ".png": "image/png",
+        ".jpg": "image/jpeg",
+        ".jpeg": "image/jpeg",
+        ".gif": "image/gif",
+        ".webp": "image/webp",
+        ".bmp": "image/bmp",
+    }
+    fname = os.path.basename(ref_path) or f"{uuid.uuid4().hex}.png"
+    ext = os.path.splitext(fname)[1].lower()
+    return fname, mime_map.get(ext, "application/octet-stream")
+
+
+def _iter_image_refs(images: dict):
+    """Yield ImageRef one at a time, freeing each source entry as we go.
+
+    Used by the streaming RPC so we never hold every decoded image plus its
+    base64 source in memory simultaneously (the inline path's peak-memory and
+    message-size problem for large scanned PDFs).
+    """
+    import base64
+
+    for ref_path in list(images.keys()):
+        b64data = images.pop(ref_path)
+        try:
+            img_bytes = base64.b64decode(b64data)
+        except Exception:
+            img_bytes = b64data.encode("utf-8") if isinstance(b64data, str) else b64data
+        del b64data
+        fname, mime = _mime_for_ref(ref_path)
+        yield ImageRef(
+            filename=fname,
+            original_ref=ref_path,
+            mime_type=mime,
+            image_data=img_bytes,
+        )
+
+
+class DocReaderServicer(docreader_pb2_grpc.DocReaderServicer):
+    def __init__(self):
+        super().__init__()
+        self.parser = Parser()
+
+    def _parse_request(self, request: ReadRequest):
+        """Run the parser for a ReadRequest, returning (result, source_desc).
+
+        Shared by the unary Read and streaming ReadStream RPCs.
+        """
+        cfg = request.config
+        parser_engine = cfg.parser_engine if cfg else ""
+        engine_overrides = dict(cfg.parser_engine_overrides) if cfg else {}
+
+        if request.url:
+            logger.info("Read(URL): url=%s", request.url)
+            result = self.parser.parse_url(
+                request.url,
+                request.title,
+                parser_engine=parser_engine,
+                engine_overrides=engine_overrides,
+            )
+            return result, request.url
+
+        file_type = request.file_type or os.path.splitext(request.file_name)[1][1:]
+        logger.info(
+            "Read(File): file=%s, type=%s, size=%d bytes",
+            request.file_name,
+            file_type,
+            len(request.file_content),
+        )
+        result = self.parser.parse_file(
+            request.file_name,
+            file_type,
+            request.file_content,
+            parser_engine=parser_engine,
+            engine_overrides=engine_overrides,
+        )
+        return result, request.file_name
+
+    def Read(self, request: ReadRequest, context):
+        """Unified read: file mode (file_content set) or URL mode (url set)."""
+        request_id = request.request_id or str(uuid.uuid4())
+
+        with request_id_context(request_id):
+            try:
+                result, source_desc = self._parse_request(request)
+
+                if not result or not result.content:
+                    error_msg = f"Failed to parse: {source_desc}"
+                    logger.error(error_msg)
+                    return ReadResponse(error=error_msg)
+
+                _c = to_valid_utf8_text
+                image_dir, image_refs = _resolve_images(result.images, request_id)
+
+                response = ReadResponse(
+                    markdown_content=_c(result.content),
+                    image_refs=image_refs,
+                    image_dir_path=image_dir,
+                    metadata={k: _c(str(v)) for k, v in result.metadata.items()}
+                    if result.metadata
+                    else {},
+                )
+                logger.info(
+                    "Read response: content_len=%d, images=%d",
+                    len(result.content),
+                    len(image_refs),
+                )
+                return response
+
+            except Exception as e:
+                error_msg = f"Error reading document: {e}"
+                logger.error(error_msg)
+                logger.info("Traceback: %s", traceback.format_exc())
+                return ReadResponse(error=str(e))
+
+    def ReadStream(self, request: ReadRequest, context):
+        """Streaming read: yields one meta frame, then one frame per image.
+
+        Each frame is a small, independent gRPC message, so documents with many
+        page images (large scanned PDFs) are returned without hitting the unary
+        message-size cap, and neither side has to hold the whole payload at once.
+        """
+        request_id = request.request_id or str(uuid.uuid4())
+
+        with request_id_context(request_id):
+            _c = to_valid_utf8_text
+            try:
+                result, source_desc = self._parse_request(request)
+            except Exception as e:
+                logger.error("Error reading document: %s", e)
+                logger.info("Traceback: %s", traceback.format_exc())
+                yield ReadStreamResponse(meta=ReadStreamMeta(error=str(e)))
+                return
+
+            if not result or not result.content:
+                error_msg = f"Failed to parse: {source_desc}"
+                logger.error(error_msg)
+                yield ReadStreamResponse(meta=ReadStreamMeta(error=error_msg))
+                return
+
+            images = result.images or {}
+            image_count = len(images)
+            yield ReadStreamResponse(
+                meta=ReadStreamMeta(
+                    markdown_content=_c(result.content),
+                    image_dir_path="",
+                    metadata={k: _c(str(v)) for k, v in result.metadata.items()}
+                    if result.metadata
+                    else {},
+                    image_count=image_count,
+                )
+            )
+
+            sent = 0
+            for ref in _iter_image_refs(images):
+                yield ReadStreamResponse(image=ref)
+                sent += 1
+
+            logger.info(
+                "ReadStream response: content_len=%d, images=%d",
+                len(result.content),
+                sent,
+            )
+
+    def ListEngines(self, request, context):
+        overrides = dict(getattr(request, "config_overrides", None) or {})
+        engines_data = registry.list_engines(overrides=overrides or None)
+        engines = [
+            ParserEngineInfo(
+                name=e["name"],
+                description=e["description"],
+                file_types=e["file_types"],
+                available=e.get("available", True),
+                unavailable_reason=e.get("unavailable_reason", ""),
+            )
+            for e in engines_data
+        ]
+        return ListEnginesResponse(engines=engines)
+
+
+def main():
+    config.print_config()
+
+    interceptors = [AuthInterceptor()]
+
+    server = grpc.server(
+        futures.ThreadPoolExecutor(max_workers=CONFIG.grpc_max_workers),
+        options=[
+            ("grpc.max_send_message_length", CONFIG.grpc_max_file_size_mb),
+            ("grpc.max_receive_message_length", CONFIG.grpc_max_file_size_mb),
+        ],
+        interceptors=interceptors,
+    )
+
+    docreader_pb2_grpc.add_DocReaderServicer_to_server(DocReaderServicer(), server)
+
+    health_servicer = HealthServicer()
+    health_pb2_grpc.add_HealthServicer_to_server(health_servicer, server)
+
+    try:
+        tls_credentials = load_tls_credentials()
+    except TLSConfigError as e:
+        logger.error("Refusing to start: %s", e)
+        sys.exit(1)
+
+    if tls_credentials:
+        server.add_secure_port(f"[::]:{CONFIG.grpc_port}", tls_credentials)
+        logger.info("Server starting on port %d with TLS", CONFIG.grpc_port)
+    else:
+        server.add_insecure_port(f"[::]:{CONFIG.grpc_port}")
+        logger.warning(
+            "Server starting on port %d WITHOUT TLS (insecure mode)", CONFIG.grpc_port
+        )
+
+    server.start()
+
+    logger.info("Server started on port %d", CONFIG.grpc_port)
+    logger.info("Server is ready to accept connections")
+
+    try:
+        server.wait_for_termination()
+    except KeyboardInterrupt:
+        logger.info("Received termination signal, shutting down server")
+        server.stop(0)
+
+
+if __name__ == "__main__":
+    main()

docreader/models/document.py

@@ -0,0 +1,87 @@
+"""Chunk document schema."""
+
+import json
+from typing import Any, Dict, List
+
+from pydantic import BaseModel, Field
+
+
+class Chunk(BaseModel):
+    """Document Chunk including chunk content, chunk metadata."""
+
+    content: str = Field(default="", description="chunk text content")
+    seq: int = Field(default=0, description="Chunk sequence number")
+    start: int = Field(default=0, description="Chunk start position")
+    end: int = Field(description="Chunk end position")
+    images: List[Dict[str, Any]] = Field(
+        default_factory=list, description="Images in the chunk"
+    )
+
+    metadata: Dict[str, Any] = Field(
+        default_factory=dict,
+        description="metadata fields",
+    )
+
+    def to_dict(self, **kwargs: Any) -> Dict[str, Any]:
+        """Convert Chunk to dict."""
+
+        data = self.model_dump()
+        data.update(kwargs)
+        data["class_name"] = self.__class__.__name__
+        return data
+
+    def to_json(self, **kwargs: Any) -> str:
+        """Convert Chunk to json."""
+        data = self.to_dict(**kwargs)
+        return json.dumps(data)
+
+    def __hash__(self):
+        """Hash function."""
+        return hash((self.content,))
+
+    def __eq__(self, other):
+        """Equal function."""
+        return self.content == other.content
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any], **kwargs: Any):  # type: ignore
+        """Create Chunk from dict."""
+        if isinstance(kwargs, dict):
+            data.update(kwargs)
+
+        data.pop("class_name", None)
+        return cls(**data)
+
+    @classmethod
+    def from_json(cls, data_str: str, **kwargs: Any):  # type: ignore
+        """Create Chunk from json."""
+        data = json.loads(data_str)
+        return cls.from_dict(data, **kwargs)
+
+
+class Document(BaseModel):
+    """Document including document content, document metadata."""
+
+    model_config = {"arbitrary_types_allowed": True}
+
+    content: str = Field(default="", description="document text content")
+    images: Dict[str, str] = Field(
+        default_factory=dict, description="Images in the document"
+    )
+
+    chunks: List[Chunk] = Field(default_factory=list, description="document chunks")
+    metadata: Dict[str, Any] = Field(
+        default_factory=dict,
+        description="metadata fields",
+    )
+
+    def set_content(self, content: str) -> None:
+        """Set document content."""
+        self.content = content
+
+    def get_content(self) -> str:
+        """Get document content."""
+        return self.content
+
+    def is_valid(self) -> bool:
+        return self.content != ""

docreader/models/read_config.py

@@ -0,0 +1,17 @@
+from dataclasses import dataclass
+
+
+@dataclass
+class ChunkingConfig:
+    """Legacy config kept for backward compatibility.
+
+    After the lightweight refactoring, chunking is done in Go.
+    This class is only kept so existing parser constructors don't break.
+    """
+
+    chunk_size: int = 512
+    chunk_overlap: int = 50
+    separators: list[str] | None = None
+    enable_multimodal: bool = False
+    storage_config: dict[str, str] | None = None
+    vlm_config: dict[str, str] | None = None

docreader/parser/__init__.py

@@ -0,0 +1,40 @@
+"""
+Parser module for WeKnora document processing system.
+
+This module provides document parsers for various file formats including:
+- Microsoft Word documents (.doc, .docx)
+- PDF documents
+- Markdown files
+- Plain text files
+- Images with text content
+- Web pages
+
+The parsers extract content from documents and can split them into
+meaningful chunks for further processing and indexing.
+"""
+
+from .doc_parser import DocParser
+from .docx2_parser import Docx2Parser
+from .excel_parser import ExcelParser
+from .html_parser import HTMLParser
+from .image_parser import ImageParser
+from .markdown_parser import MarkdownParser
+from .parser import Parser
+from .pdf_parser import PDFParser
+from .registry import ParserEngineRegistry, registry
+from .web_parser import WebParser
+
+# Export public classes and modules
+__all__ = [
+    "Docx2Parser",
+    "DocParser",
+    "PDFParser",
+    "MarkdownParser",
+    "ImageParser",
+    "WebParser",
+    "Parser",
+    "ExcelParser",
+    "HTMLParser",
+    "ParserEngineRegistry",
+    "registry",
+]

docreader/parser/base_parser.py

@@ -0,0 +1,61 @@
+# -*- coding: utf-8 -*-
+import logging
+import os
+from abc import ABC, abstractmethod
+from typing import Optional
+
+from docreader.models.document import Document
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+class BaseParser(ABC):
+    """Base parser interface.
+
+    After the lightweight refactoring, BaseParser only extracts markdown text
+    and raw image references from documents. Chunking, image storage, OCR,
+    and VLM caption are handled by the Go App module.
+    """
+
+    def __init__(
+        self,
+        file_name: str = "",
+        file_type: Optional[str] = None,
+        **kwargs,
+    ):
+        self.file_name = file_name
+        self.file_type = file_type or os.path.splitext(file_name)[1].lstrip(".")
+
+        logger.info(
+            "Initializing parser for file=%s, type=%s",
+            file_name,
+            self.file_type,
+        )
+
+    @abstractmethod
+    def parse_into_text(self, content: bytes) -> Document:
+        """Parse document content into markdown text.
+
+        Returns:
+            Document with ``content`` (markdown string) and optional
+            ``images`` dict mapping storage-relative paths to base64 data.
+        """
+
+    def parse(self, content: bytes) -> Document:
+        """Parse document and return markdown + image references.
+
+        No chunking, no OCR, no VLM caption — those are done in Go.
+        """
+        logger.info(
+            "Parsing document with %s, bytes: %d",
+            self.__class__.__name__,
+            len(content),
+        )
+        document = self.parse_into_text(content)
+        logger.info(
+            "Extracted %d characters from %s",
+            len(document.content),
+            self.file_name,
+        )
+        return document

docreader/parser/chain_parser.py

@@ -0,0 +1,179 @@
+"""
+Chain Parser Module
+
+This module provides two chain-of-responsibility pattern implementations for document parsing:
+1. FirstParser: Tries multiple parsers sequentially until one succeeds
+2. PipelineParser: Chains parsers where each parser processes the output of the previous one
+"""
+
+import logging
+from typing import Dict, List, Tuple, Type
+
+from docreader.models.document import Document
+from docreader.parser.base_parser import BaseParser
+from docreader.utils import endecode
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+
+class FirstParser(BaseParser):
+    """
+    First-success parser that tries multiple parsers in sequence.
+
+    This parser attempts to parse content using each registered parser in order.
+    It returns the result from the first parser that successfully produces a valid document.
+    If all parsers fail, it returns an empty Document.
+
+    Usage:
+        # Create a custom FirstParser with specific parser classes
+        CustomParser = FirstParser.create(MarkdownParser, HTMLParser)
+        parser = CustomParser()
+        document = parser.parse_into_text(content_bytes)
+    """
+
+    # Tuple of parser classes to be instantiated
+    _parser_cls: Tuple[Type["BaseParser"], ...] = ()
+
+    def __init__(self, *args, **kwargs):
+        """Initialize FirstParser with configured parser classes."""
+        super().__init__(*args, **kwargs)
+
+        # Instantiate all parser classes into parser instances
+        self._parsers: List[BaseParser] = []
+        for parser_cls in self._parser_cls:
+            parser = parser_cls(*args, **kwargs)
+            self._parsers.append(parser)
+
+    def parse_into_text(self, content: bytes) -> Document:
+        """Parse content using the first parser that succeeds.
+
+        Args:
+            content: Raw bytes content to be parsed
+
+        Returns:
+            Document: Parsed document from the first successful parser,
+                     or an empty Document if all parsers fail
+        """
+        for p in self._parsers:
+            logger.info(f"FirstParser: using parser {p.__class__.__name__}")
+            try:
+                document = p.parse_into_text(content)
+            except Exception:
+                logger.exception(
+                    "FirstParser: parser %s raised exception; trying next parser",
+                    p.__class__.__name__,
+                )
+                continue
+
+            if document.is_valid():
+                logger.info(f"FirstParser: parser {p.__class__.__name__} succeeded")
+                return document
+        return Document()
+
+    @classmethod
+    def create(cls, *parser_classes: Type["BaseParser"]) -> Type["FirstParser"]:
+        """Factory method to create a FirstParser subclass with specific parsers.
+
+        Args:
+            *parser_classes: Variable number of BaseParser subclasses to try in order
+
+        Returns:
+            Type[FirstParser]: A new FirstParser subclass configured with the given parsers
+
+        Example:
+            CustomParser = FirstParser.create(MarkdownParser, HTMLParser)
+            parser = CustomParser()
+        """
+        # Generate a descriptive class name based on parser names
+        names = "_".join([p.__name__ for p in parser_classes])
+        # Dynamically create a new class with the parser configuration
+        return type(f"FirstParser_{names}", (cls,), {"_parser_cls": parser_classes})
+
+
+class PipelineParser(BaseParser):
+    """
+    Pipeline parser that chains multiple parsers sequentially.
+
+    This parser processes content through a series of parsers where each parser
+    receives the output of the previous parser as input. Images from all parsers
+    are accumulated and merged into the final document.
+
+    Usage:
+        # Create a custom PipelineParser with specific parser classes
+        CustomParser = PipelineParser.create(PreParser, MarkdownParser, PostParser)
+        parser = CustomParser()
+        document = parser.parse_into_text(content_bytes)
+    """
+
+    # Tuple of parser classes to be instantiated and chained
+    _parser_cls: Tuple[Type["BaseParser"], ...] = ()
+
+    def __init__(self, *args, **kwargs):
+        """Initialize PipelineParser with configured parser classes."""
+        super().__init__(*args, **kwargs)
+
+        # Instantiate all parser classes into parser instances
+        self._parsers: List[BaseParser] = []
+        for parser_cls in self._parser_cls:
+            parser = parser_cls(*args, **kwargs)
+            self._parsers.append(parser)
+
+    def parse_into_text(self, content: bytes) -> Document:
+        """Parse content through a pipeline of parsers.
+
+        Each parser in the pipeline processes the output of the previous parser.
+        Images from all parsers are accumulated and merged into the final document.
+
+        Args:
+            content: Raw bytes content to be parsed
+
+        Returns:
+            Document: Final document after processing through all parsers,
+                     with accumulated images from all stages
+        """
+        # Accumulate images and metadata from all parsers
+        images: Dict[str, str] = {}
+        metadata: Dict = {}
+        document = Document()
+        for p in self._parsers:
+            logger.info(f"PipelineParser: using parser {p.__class__.__name__}")
+            # Parse content with current parser
+            document = p.parse_into_text(content)
+            # Convert document content back to bytes for next parser
+            content = endecode.encode_bytes(document.content)
+            # Accumulate images and metadata from this parser
+            images.update(document.images)
+            metadata.update(document.metadata)
+        # Merge all accumulated images and metadata into final document
+        document.images.update(images)
+        document.metadata.update(metadata)
+        return document
+
+    @classmethod
+    def create(cls, *parser_classes: Type["BaseParser"]) -> Type["PipelineParser"]:
+        """Factory method to create a PipelineParser subclass with specific parsers.
+
+        Args:
+            *parser_classes: Variable number of BaseParser subclasses to chain in order
+
+        Returns:
+            Type[PipelineParser]: A new PipelineParser subclass configured with the given parsers
+
+        Example:
+            CustomParser = PipelineParser.create(PreprocessParser, MarkdownParser)
+            parser = CustomParser()
+        """
+        # Generate a descriptive class name based on parser names
+        names = "_".join([p.__name__ for p in parser_classes])
+        # Dynamically create a new class with the parser configuration
+        return type(f"PipelineParser_{names}", (cls,), {"_parser_cls": parser_classes})
+
+
+if __name__ == "__main__":
+    from docreader.parser.markdown_parser import MarkdownParser
+
+    # Example: Create and use a FirstParser with MarkdownParser
+    FpCls = FirstParser.create(MarkdownParser)
+    lparser = FpCls()
+    print(lparser.parse_into_text(b"aaa"))

docreader/parser/concurrency.py

@@ -0,0 +1,39 @@
+import logging
+import threading
+from contextlib import contextmanager
+from typing import Dict, Iterator
+
+logger = logging.getLogger(__name__)
+
+_LIMITERS: Dict[str, threading.BoundedSemaphore] = {}
+_LIMITERS_LOCK = threading.Lock()
+
+
+def _get_limiter(name: str, max_workers: int) -> threading.BoundedSemaphore:
+    with _LIMITERS_LOCK:
+        limiter = _LIMITERS.get(name)
+        if limiter is None:
+            limiter = threading.BoundedSemaphore(max_workers)
+            _LIMITERS[name] = limiter
+        return limiter
+
+
+@contextmanager
+def parser_worker_limit(name: str, max_workers: int) -> Iterator[None]:
+    """Limit concurrent access to heavy, process-wide parser operations.
+
+    Set max_workers <= 0 to disable throttling for deployments that have enough
+    CPU/GPU resources and know the parser backend is safe under concurrency.
+    """
+
+    if max_workers <= 0:
+        yield
+        return
+
+    limiter = _get_limiter(name, max_workers)
+    logger.debug("Waiting for %s parser slot (max_workers=%d)", name, max_workers)
+    limiter.acquire()
+    try:
+        yield
+    finally:
+        limiter.release()

docreader/parser/doc_parser.py

@@ -0,0 +1,362 @@
+import logging
+import os
+import subprocess
+import time
+import uuid
+from pathlib import Path
+from typing import List, Optional
+
+import textract
+
+from docreader.config import CONFIG
+from docreader.models.document import Document
+from docreader.parser.docx2_parser import Docx2Parser
+from docreader.utils.tempfile import TempDirContext, TempFileContext
+
+logger = logging.getLogger(__name__)
+
+
+class SandboxExecutor:
+    """Sandbox executor for running commands with proxy configuration"""
+
+    def __init__(self, proxy: Optional[str] = None, default_timeout: int = 60):
+        """Initialize sandbox executor with configuration
+
+        Args:
+            proxy: Proxy URL to use for network access. If None, will use WEB_PROXY environment variable
+            default_timeout: Default timeout in seconds for command execution
+        """
+        # Get proxy from parameter, environment variable, or use default blocking proxy
+        # Use 'or None' to convert empty string to None, then apply default value
+        self.proxy = proxy or CONFIG.external_https_proxy or "http://128.0.0.1:1"
+        self.default_timeout = default_timeout
+
+    def execute_in_sandbox(self, cmd: List[str]) -> tuple:
+        """Execute command in sandbox with proxy configuration
+
+        Args:
+            cmd: Command to execute
+
+        Returns:
+            Tuple of (stdout, stderr, returncode)
+        """
+        # Try different sandbox methods in order of preference
+        sandbox_methods = [
+            self._execute_with_proxy,
+        ]
+
+        for method in sandbox_methods:
+            try:
+                return method(cmd)
+            except Exception as e:
+                logger.warning(f"Sandbox method {method.__name__} failed: {e}")
+                continue
+
+        raise RuntimeError("All sandbox methods failed")
+
+    def _execute_with_proxy(self, cmd: List[str]) -> tuple:
+        """Execute command with proxy configuration
+
+        Args:
+            cmd: Command to execute
+
+        Returns:
+            Tuple of (stdout, stderr, returncode)
+        """
+        # Set up environment with proxy configuration
+        env = os.environ.copy()
+        if self.proxy:
+            env["http_proxy"] = self.proxy
+            env["https_proxy"] = self.proxy
+            env["HTTP_PROXY"] = self.proxy
+            env["HTTPS_PROXY"] = self.proxy
+
+        logger.info(f"Executing command with proxy: {' '.join(cmd)}")
+        if self.proxy:
+            logger.info(f"Using proxy: {self.proxy}")
+
+        process = subprocess.Popen(
+            cmd,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            env=env,
+        )
+
+        try:
+            stdout, stderr = process.communicate(timeout=self.default_timeout)
+            return stdout, stderr, process.returncode
+        except subprocess.TimeoutExpired:
+            process.kill()
+            raise RuntimeError(
+                f"Command execution timeout after {self.default_timeout} seconds"
+            )
+
+
+logger = logging.getLogger(__name__)
+
+
+class DocParser(Docx2Parser):
+    """DOC document parser"""
+
+    def __init__(self, *args, **kwargs):
+        """Initialize DOC parser with sandbox executor"""
+        super().__init__(*args, **kwargs)
+        self.sandbox_executor = SandboxExecutor()
+
+    def parse_into_text(self, content: bytes) -> Document:
+        logger.info(f"Parsing DOC document, content size: {len(content)} bytes")
+
+        handle_chain = [
+            # 1. Try to convert to docx format to extract images
+            self._parse_with_docx,
+            # 2. If image extraction is not needed or conversion failed,
+            # try using antiword to extract text
+            self._parse_with_antiword,
+            # 3. If antiword extraction fails, use textract
+            # NOTE: _parse_with_textract is disabled due to SSRF vulnerability
+            # self._parse_with_textract,
+        ]
+
+        # Save byte content as a temporary file
+        with TempFileContext(content, ".doc") as temp_file_path:
+            for handle in handle_chain:
+                try:
+                    document = handle(temp_file_path)
+                    if document:
+                        return document
+                except Exception as e:
+                    logger.warning(f"Failed to parse DOC with {handle.__name__} {e}")
+
+            return Document(content="")
+
+    def _parse_with_docx(self, temp_file_path: str) -> Document:
+        logger.info("Multimodal enabled, attempting to extract images from DOC")
+
+        docx_content = self._try_convert_doc_to_docx(temp_file_path)
+        if not docx_content:
+            raise RuntimeError("Failed to convert DOC to DOCX")
+
+        logger.info("Successfully converted DOC to DOCX, using DocxParser")
+        # Use existing DocxParser to parse the converted docx
+        document = super(Docx2Parser, self).parse_into_text(docx_content)
+        logger.info(f"Extracted {len(document.content)} characters using DocxParser")
+        return document
+
+    def _parse_with_antiword(self, temp_file_path: str) -> Document:
+        logger.info("Attempting to parse DOC file with antiword")
+
+        # Check if antiword is installed
+        antiword_path = self._try_find_antiword()
+        if not antiword_path:
+            raise RuntimeError("antiword not found in PATH")
+
+        # Use antiword to extract text directly in sandbox
+        cmd = [antiword_path, temp_file_path]
+        logger.info("Executing antiword in sandbox with proxy configuration")
+
+        stdout, stderr, returncode = self.sandbox_executor.execute_in_sandbox(cmd)
+
+        if returncode != 0:
+            raise RuntimeError(
+                f"antiword extraction failed: {stderr.decode('utf-8', errors='ignore')}"
+            )
+        text = stdout.decode("utf-8", errors="ignore")
+        logger.info(f"Successfully extracted {len(text)} characters using antiword")
+        return Document(content=text)
+
+    def _parse_with_textract(self, temp_file_path: str) -> Document:
+        logger.info(f"Parsing DOC file with textract: {temp_file_path}")
+        text = textract.process(temp_file_path, method="antiword").decode("utf-8")
+        logger.info(f"Successfully extracted {len(text)} bytes of DOC using textract")
+        return Document(content=str(text))
+
+    def _try_convert_doc_to_docx(self, doc_path: str) -> Optional[bytes]:
+        """Convert DOC file to DOCX format
+
+        Uses LibreOffice/OpenOffice for conversion
+
+        Args:
+            doc_path: DOC file path
+
+        Returns:
+            Byte stream of DOCX file content, or None if conversion fails
+        """
+        logger.info(f"Converting DOC to DOCX: {doc_path}")
+
+        # Check if LibreOffice or OpenOffice is installed
+        soffice_path = self._try_find_soffice()
+        if not soffice_path:
+            return None
+
+        # Execute conversion command
+        logger.info(f"Using {soffice_path} to convert DOC to DOCX")
+
+        # LibreOffice shares a single user profile by default, so concurrent
+        # `soffice` invocations contend for the same profile lock and the loser
+        # silently fails to convert. Give each attempt a dedicated profile dir
+        # and retry a few times so concurrent requests don't fall back to the
+        # lower-fidelity antiword path.
+        max_attempts = 3
+        for attempt in range(1, max_attempts + 1):
+            # Create a temporary directory to store the converted file
+            with TempDirContext() as temp_dir, TempDirContext() as profile_dir:
+                user_installation = Path(profile_dir).as_uri()
+                cmd = [
+                    soffice_path,
+                    "--headless",
+                    f"-env:UserInstallation={user_installation}",
+                    "--convert-to",
+                    "docx",
+                    "--outdir",
+                    temp_dir,
+                    doc_path,
+                ]
+                logger.info(
+                    f"Running command in sandbox (attempt {attempt}/{max_attempts}): "
+                    f"{' '.join(cmd)}"
+                )
+
+                # Execute in sandbox with proxy configuration
+                stdout, stderr, returncode = self.sandbox_executor.execute_in_sandbox(
+                    cmd
+                )
+
+                if returncode != 0:
+                    logger.warning(
+                        f"Error converting DOC to DOCX (attempt {attempt}/"
+                        f"{max_attempts}): {stderr.decode('utf-8', errors='ignore')}"
+                    )
+                    if attempt < max_attempts:
+                        time.sleep(0.5 * attempt)
+                        continue
+                    return None
+
+                # Find the converted file
+                docx_file = [
+                    file for file in os.listdir(temp_dir) if file.endswith(".docx")
+                ]
+                logger.info(
+                    f"Found {len(docx_file)} DOCX file(s) in temporary directory"
+                )
+                for file in docx_file:
+                    converted_file = os.path.join(temp_dir, file)
+                    logger.info(f"Found converted file: {converted_file}")
+
+                    # Read the converted file content
+                    with open(converted_file, "rb") as f:
+                        docx_content = f.read()
+                        logger.info(
+                            f"Successfully read DOCX file, size: {len(docx_content)}"
+                        )
+                        return docx_content
+
+                # Conversion reported success but produced no docx; retry.
+                logger.warning(
+                    f"No DOCX produced despite success (attempt {attempt}/"
+                    f"{max_attempts})"
+                )
+                if attempt < max_attempts:
+                    time.sleep(0.5 * attempt)
+        return None
+
+    def _try_find_executable_path(
+        self,
+        executable_name: str,
+        possible_path: List[str] = [],
+        environment_variable: List[str] = [],
+    ) -> Optional[str]:
+        """Find executable path
+        Args:
+            executable_name: Executable name
+            possible_path: List of possible paths
+            environment_variable: List of environment variables to check
+            Returns:
+                Executable path, or None if not found
+        """
+        # Common executable paths
+        paths: List[str] = []
+        paths.extend(possible_path)
+        paths.extend(os.environ.get(env_var, "") for env_var in environment_variable)
+        paths = list(set(paths))
+
+        # Check if path is set in environment variable
+        for path in paths:
+            if os.path.exists(path):
+                logger.info(f"Found {executable_name} at {path}")
+                return path
+
+        # Try to find in PATH
+        result = subprocess.run(
+            ["which", executable_name], capture_output=True, text=True
+        )
+        if result.returncode == 0 and result.stdout.strip():
+            path = result.stdout.strip()
+            logger.info(f"Found {executable_name} at {path}")
+            return path
+
+        logger.warning(f"Failed to find {executable_name}")
+        return None
+
+    def _try_find_soffice(self) -> Optional[str]:
+        """Find LibreOffice/OpenOffice executable path
+
+        Returns:
+            Executable path, or None if not found
+        """
+        # Common LibreOffice/OpenOffice executable paths
+        possible_paths = [
+            # Linux
+            "/usr/bin/soffice",
+            "/usr/lib/libreoffice/program/soffice",
+            "/opt/libreoffice25.2/program/soffice",
+            # macOS
+            "/Applications/LibreOffice.app/Contents/MacOS/soffice",
+            # Windows
+            "C:\\Program Files\\LibreOffice\\program\\soffice.exe",
+            "C:\\Program Files (x86)\\LibreOffice\\program\\soffice.exe",
+        ]
+        return self._try_find_executable_path(
+            executable_name="soffice",
+            possible_path=possible_paths,
+            environment_variable=["LIBREOFFICE_PATH"],
+        )
+
+    def _try_find_antiword(self) -> Optional[str]:
+        """Find antiword executable path
+
+        Returns:
+            Executable path, or None if not found
+        """
+        # Common antiword executable paths
+        possible_paths = [
+            # Linux/macOS
+            "/usr/bin/antiword",
+            "/usr/local/bin/antiword",
+            # Windows
+            "C:\\Program Files\\Antiword\\antiword.exe",
+            "C:\\Program Files (x86)\\Antiword\\antiword.exe",
+        ]
+        return self._try_find_executable_path(
+            executable_name="antiword",
+            possible_path=possible_paths,
+            environment_variable=["ANTIWORD_PATH"],
+        )
+
+
+if __name__ == "__main__":
+    logging.basicConfig(level=logging.DEBUG)
+
+    file_name = "/path/to/your/test.doc"
+    logger.info(f"Processing file: {file_name}")
+    doc_parser = DocParser(
+        file_name=file_name,
+        enable_multimodal=True,
+        chunk_size=512,
+        chunk_overlap=60,
+    )
+    with open(file_name, "rb") as f:
+        content = f.read()
+
+    document = doc_parser.parse_into_text(content)
+    logger.info(f"Processing complete, extracted text length: {len(document.content)}")
+    logger.info(f"Sample text: {document.content[:200]}...")

docreader/parser/docx2_parser.py

@@ -0,0 +1,28 @@
+import logging
+
+from docreader.parser.chain_parser import FirstParser
+from docreader.parser.docx_parser import DocxParser
+from docreader.parser.markitdown_parser import MarkitdownParser
+
+logger = logging.getLogger(__name__)
+
+
+class Docx2Parser(FirstParser):
+    _parser_cls = (MarkitdownParser, DocxParser)
+
+
+if __name__ == "__main__":
+    logging.basicConfig(level=logging.DEBUG)
+
+    your_file = "/path/to/your/file.docx"
+    parser = Docx2Parser(separators=[".", "?", "!", "。", "？", "！"])
+    with open(your_file, "rb") as f:
+        content = f.read()
+
+        document = parser.parse(content)
+        for cc in document.chunks:
+            logger.info(f"chunk: {cc}")
+
+        # document = parser.parse_into_text(content)
+        # logger.info(f"docx content: {document.content}")
+        # logger.info(f"find images {document.images.keys()}")

docreader/parser/docx_merge.py

@@ -0,0 +1,177 @@
+"""Expand vertically merged DOCX table cells before Markdown conversion."""
+
+from __future__ import annotations
+
+import copy
+import io
+import logging
+import zipfile
+
+from lxml import etree
+
+logger = logging.getLogger(__name__)
+
+_WORD_NAMESPACE = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
+_W = f"{{{_WORD_NAMESPACE}}}"
+_TC_PR = f"{_W}tcPr"
+_VMERGE = f"{_W}vMerge"
+_GRID_SPAN = f"{_W}gridSpan"
+_TR_PR = f"{_W}trPr"
+_GRID_BEFORE = f"{_W}gridBefore"
+_VAL = f"{_W}val"
+
+
+def fill_vertical_merged_cells_docx(content: bytes) -> bytes:
+    """Repeat a vertical merge's master content in every covered table row.
+
+    DOCX stores vertical merges as a ``vMerge=restart`` cell followed by empty
+    ``vMerge=continue`` cells. Mammoth preserves that as an HTML ``rowspan``,
+    but Markdown tables cannot represent row spans and the later rows lose the
+    merged value. This rewrites only Word XML parts that contain ``vMerge``;
+    all other package parts and documents without vertical merges pass through
+    unchanged.
+    """
+    source = io.BytesIO(content)
+    if not zipfile.is_zipfile(source):
+        return content
+
+    with zipfile.ZipFile(source, "r") as archive:
+        normalized_names = {
+            info.filename.replace("\\", "/") for info in archive.infolist()
+        }
+        if "word/document.xml" not in normalized_names:
+            return content
+
+        rewritten_parts: dict[str, bytes] = {}
+        filled_count = 0
+        for info in archive.infolist():
+            name = info.filename.replace("\\", "/")
+            if not name.startswith("word/") or not name.endswith(".xml"):
+                continue
+
+            data = archive.read(info.filename)
+            if b"vMerge" not in data:
+                continue
+            rewritten, part_filled_count = _fill_vertical_merges_in_xml(data)
+            if rewritten is not None:
+                rewritten_parts[info.filename] = rewritten
+                filled_count += part_filled_count
+
+        if not rewritten_parts:
+            return content
+
+        output = io.BytesIO()
+        with zipfile.ZipFile(output, "w") as rewritten_archive:
+            for info in archive.infolist():
+                data = rewritten_parts.get(info.filename)
+                if data is None:
+                    data = archive.read(info.filename)
+                rewritten_archive.writestr(info, data)
+
+    logger.info(
+        "Expanded %d vertically merged DOCX continuation cells before parse",
+        filled_count,
+    )
+    return output.getvalue()
+
+
+def _fill_vertical_merges_in_xml(data: bytes) -> tuple[bytes | None, int]:
+    parser = etree.XMLParser(resolve_entities=False, no_network=True)
+    try:
+        root = etree.fromstring(data, parser=parser)
+    except etree.XMLSyntaxError:
+        return None, 0
+
+    changed = False
+    filled_count = 0
+    # Process nested tables first so copied cell content is already normalized.
+    for table in reversed(root.xpath(".//w:tbl", namespaces={"w": _WORD_NAMESPACE})):
+        table_changed, table_filled_count = _fill_table_vertical_merges(table)
+        changed = changed or table_changed
+        filled_count += table_filled_count
+
+    if not changed:
+        return None, 0
+    return etree.tostring(root, xml_declaration=True, encoding="UTF-8"), filled_count
+
+
+def _fill_table_vertical_merges(table: etree._Element) -> tuple[bool, int]:
+    active_merges: dict[int, tuple[etree._Element, ...]] = {}
+    changed = False
+    filled_count = 0
+
+    for row in table.findall(f"{_W}tr"):
+        next_active_merges: dict[int, tuple[etree._Element, ...]] = {}
+        column = _grid_before(row)
+
+        for cell in row.findall(f"{_W}tc"):
+            cell_properties = cell.find(_TC_PR)
+            span = _grid_span(cell_properties)
+            occupied_columns = range(column, column + span)
+            vertical_merge = (
+                cell_properties.find(_VMERGE) if cell_properties is not None else None
+            )
+
+            if vertical_merge is not None:
+                merge_type = vertical_merge.get(_VAL, "continue")
+                if merge_type == "restart":
+                    template = tuple(
+                        copy.deepcopy(child) for child in cell if child.tag != _TC_PR
+                    )
+                    for occupied_column in occupied_columns:
+                        next_active_merges[occupied_column] = template
+                    cell_properties.remove(vertical_merge)
+                    changed = True
+                else:
+                    template = next(
+                        (
+                            active_merges[occupied_column]
+                            for occupied_column in occupied_columns
+                            if occupied_column in active_merges
+                        ),
+                        None,
+                    )
+                    if template is not None:
+                        for child in list(cell):
+                            if child.tag != _TC_PR:
+                                cell.remove(child)
+                        for child in template:
+                            cell.append(copy.deepcopy(child))
+                        for occupied_column in occupied_columns:
+                            next_active_merges[occupied_column] = template
+                        cell_properties.remove(vertical_merge)
+                        changed = True
+                        filled_count += 1
+
+            column += span
+
+        active_merges = next_active_merges
+
+    return changed, filled_count
+
+
+def _grid_span(cell_properties: etree._Element | None) -> int:
+    if cell_properties is None:
+        return 1
+    grid_span = cell_properties.find(_GRID_SPAN)
+    if grid_span is None:
+        return 1
+    return _positive_int(grid_span.get(_VAL), default=1)
+
+
+def _grid_before(row: etree._Element) -> int:
+    row_properties = row.find(_TR_PR)
+    if row_properties is None:
+        return 0
+    grid_before = row_properties.find(_GRID_BEFORE)
+    if grid_before is None:
+        return 0
+    return _positive_int(grid_before.get(_VAL), default=0)
+
+
+def _positive_int(value: str | None, default: int) -> int:
+    try:
+        parsed = int(value or "")
+    except ValueError:
+        return default
+    return parsed if parsed >= 0 else default

docreader/parser/epub_parser.py

@@ -0,0 +1,332 @@
+"""EPUB parser.
+
+Parses EPUB files into markdown text and optional embedded images.
+"""
+
+import base64
+import logging
+import os
+import posixpath
+import tempfile
+from urllib.parse import unquote
+import uuid
+from typing import Dict
+
+from bs4 import BeautifulSoup
+import ebooklib
+from ebooklib import epub
+
+from docreader.models.document import Document
+from docreader.parser.base_parser import BaseParser
+
+logger = logging.getLogger(__name__)
+
+
+class EPUBParser(BaseParser):
+    """Parser for EPUB e-book files."""
+
+    def __init__(self, *args, extract_images: bool = True, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.extract_images = extract_images
+
+    def parse_into_text(self, content: bytes) -> Document:
+        logger.info(
+            "Parsing EPUB file: %s, size: %d bytes", self.file_name, len(content)
+        )
+        try:
+            with tempfile.NamedTemporaryFile(
+                suffix=".epub", delete=False, mode="wb"
+            ) as epub_file:
+                epub_file.write(content)
+                epub_path = epub_file.name
+            try:
+                book = epub.read_epub(epub_path)
+                metadata = self._extract_metadata(book)
+                markdown_content, images = self._extract_content(book)
+
+                metadata["source_format"] = "epub"
+                metadata["file_size"] = len(content)
+                metadata["chapter_count"] = len(
+                    [part for part in markdown_content.split("\n## ") if part.strip()]
+                )
+                metadata["image_count"] = len(images)
+                return Document(
+                    content=markdown_content, images=images, metadata=metadata
+                )
+            finally:
+                if os.path.exists(epub_path):
+                    os.unlink(epub_path)
+        except ImportError:
+            logger.error("ebooklib not installed")
+            raise
+        except Exception as e:
+            logger.warning(
+                "ebooklib failed to parse EPUB: %s, trying ZIP fallback", str(e)
+            )
+            return self._parse_epub_fallback(content)
+
+    def _parse_epub_fallback(self, content: bytes) -> Document:
+        """Parse EPUB directly as a ZIP when ebooklib cannot read it."""
+        import re
+        import zipfile
+        from io import BytesIO
+
+        metadata = {"source_format": "epub", "file_size": len(content)}
+        images: Dict[str, str] = {}
+        image_aliases: Dict[str, str] = {}
+        markdown_parts = []
+
+        with zipfile.ZipFile(BytesIO(content), "r") as epub_zip:
+            html_files = [
+                f
+                for f in epub_zip.namelist()
+                if f.endswith((".html", ".xhtml", ".htm"))
+            ]
+
+            def chapter_num(filename: str) -> int:
+                match = re.search(r"chapter(\d+)", filename, re.IGNORECASE)
+                return int(match.group(1)) if match else 999999
+
+            html_files.sort(key=chapter_num)
+
+            if self.extract_images:
+                img_exts = (".jpg", ".jpeg", ".png", ".gif", ".webp", ".svg")
+                for img_file in epub_zip.namelist():
+                    if img_file.lower().endswith(img_exts):
+                        try:
+                            img_data = epub_zip.read(img_file)
+                            ext = os.path.splitext(img_file)[1]
+                            img_path = f"images/{uuid.uuid4().hex}{ext}"
+                            images[img_path] = base64.b64encode(img_data).decode(
+                                "utf-8"
+                            )
+                            self._add_image_aliases(image_aliases, img_file, img_path)
+                        except Exception as e:
+                            logger.warning(
+                                "Failed to extract image %s: %s", img_file, e
+                            )
+
+            for html_file in html_files:
+                try:
+                    html_content = epub_zip.read(html_file).decode(
+                        "utf-8", errors="ignore"
+                    )
+                    chapter_md = self._html_to_markdown(
+                        html_content,
+                        image_aliases=image_aliases,
+                        base_path=posixpath.dirname(html_file),
+                    )
+                    base = os.path.basename(html_file)
+                    title = base.replace(".html", "").replace(".xhtml", "")
+                    title = re.sub(
+                        r"chapter[_-]?", "Chapter ", title, flags=re.IGNORECASE
+                    )
+                    title = title.replace("_", " ").replace("-", " ").title()
+                    if chapter_md.strip():
+                        markdown_parts.append(f"## {title}\n\n{chapter_md}")
+                except Exception as e:
+                    logger.warning("Failed to process %s: %s", html_file, e)
+
+        metadata["chapter_count"] = len(markdown_parts)
+        metadata["image_count"] = len(images)
+        return Document(
+            content="\n\n".join(markdown_parts), images=images, metadata=metadata
+        )
+
+    def _extract_metadata(self, book) -> Dict[str, str]:
+        metadata: Dict[str, str] = {}
+        mapping = {
+            "title": "title",
+            "creator": "author",
+            "publisher": "publisher",
+            "language": "language",
+            "description": "description",
+            "date": "date",
+            "identifier": "isbn",
+        }
+        for dc_key, out_key in mapping.items():
+            try:
+                values = book.get_metadata("DC", dc_key)
+            except Exception:
+                values = None
+            if values:
+                if out_key == "author":
+                    metadata[out_key] = ", ".join(value[0] for value in values)
+                else:
+                    metadata[out_key] = values[0][0]
+        return metadata
+
+    def _extract_content(self, book) -> tuple[str, Dict[str, str]]:
+        markdown_parts = []
+        images: Dict[str, str] = {}
+        image_aliases: Dict[str, str] = {}
+
+        try:
+            toc = book.get_table_of_contents()
+        except Exception as e:
+            logger.debug("Failed to get TOC: %s, processing all HTML items", e)
+            toc = []
+
+        html_items = {}
+        for item in book.get_items():
+            if item.get_type() == ebooklib.ITEM_DOCUMENT:
+                html_items[item.get_name()] = item
+
+        if self.extract_images:
+            for item in book.get_items():
+                if item.get_type() == ebooklib.ITEM_IMAGE:
+                    img_data = item.get_content()
+                    ext = os.path.splitext(item.get_name())[1]
+                    img_path = f"images/{uuid.uuid4().hex}{ext}"
+                    images[img_path] = base64.b64encode(img_data).decode("utf-8")
+                    self._add_image_aliases(image_aliases, item.get_name(), img_path)
+
+        if toc:
+            for item in toc:
+                entries = item if isinstance(item, tuple) else (item,)
+                for sub in entries:
+                    if hasattr(sub, "get_name") and sub.get_name() in html_items:
+                        markdown_parts.append(
+                            self._process_chapter(
+                                html_items[sub.get_name()],
+                                toc_index=len(markdown_parts),
+                                image_aliases=image_aliases,
+                            )
+                        )
+
+        if not markdown_parts:
+            for _name, item in html_items.items():
+                markdown_parts.append(
+                    self._process_chapter(
+                        item,
+                        toc_index=len(markdown_parts),
+                        image_aliases=image_aliases,
+                    )
+                )
+
+        return "\n\n".join(part for part in markdown_parts if part.strip()), images
+
+    def _process_chapter(
+        self,
+        html_item,
+        toc_index: int = 0,
+        image_aliases: Dict[str, str] | None = None,
+    ) -> str:
+        try:
+            html_content = html_item.get_content()
+            soup = BeautifulSoup(html_content, "lxml")
+            title_tag = soup.find(["h1", "h2"])
+            if title_tag:
+                chapter_title = title_tag.get_text().strip()
+                title_tag.decompose()
+            else:
+                chapter_title = html_item.get_name().replace("/", "")
+                chapter_title = chapter_title.replace(".xhtml", "")
+                chapter_title = chapter_title.replace("-", " ").title()
+            body_html = str(soup.body) if soup.body else str(html_content)
+            chapter_md = self._html_to_markdown(
+                body_html,
+                image_aliases=image_aliases,
+                base_path=posixpath.dirname(html_item.get_name()),
+            )
+            return f"## {chapter_title}\n\n{chapter_md}"
+        except Exception as e:
+            logger.error(
+                "Failed to process chapter %s: %s", html_item.get_name(), e
+            )
+            return f"## Chapter {toc_index + 1}\n\n[Error processing chapter: {e}]"
+
+    def _html_to_markdown(
+        self,
+        html_content: str,
+        image_aliases: Dict[str, str] | None = None,
+        base_path: str = "",
+    ) -> str:
+        try:
+            from bs4 import Comment
+            from markdownify import markdownify as md
+
+            soup = BeautifulSoup(html_content, "lxml")
+            for element in soup(["script", "style"]):
+                element.decompose()
+            for comment in soup.find_all(
+                string=lambda text: isinstance(text, Comment)
+            ):
+                comment.extract()
+            self._strip_internal_links(soup)
+            if image_aliases:
+                self._rewrite_image_sources(soup, image_aliases, base_path)
+            markdown_text = md(str(soup), heading_style="ATX")
+            return "\n".join(
+                line.strip() for line in markdown_text.split("\n") if line.strip()
+            )
+        except ImportError:
+            logger.warning("markdownify not available, using HTML as-is")
+            return f"```html\n{html_content}\n```"
+        except Exception as e:
+            logger.error("HTML to Markdown conversion failed: %s", e)
+            return f"```html\n{html_content}\n```"
+
+    @staticmethod
+    def _strip_internal_links(soup: BeautifulSoup) -> None:
+        """Unwrap links that don't point to an external resource.
+
+        EPUB internal links (other chapter files, ``#fragment`` anchors, TOC
+        entries) become dead links after extraction. Keep only external links
+        and replace everything else with its text.
+        """
+        external = ("http://", "https://", "mailto:", "tel:")
+        for link in soup.find_all("a"):
+            href = (link.get("href") or "").strip().lower()
+            if not href or not href.startswith(external):
+                link.unwrap()
+
+    @staticmethod
+    def _add_image_aliases(
+        image_aliases: Dict[str, str],
+        original_path: str,
+        image_path: str,
+    ) -> None:
+        normalized = EPUBParser._normalize_epub_path(original_path)
+        aliases = {
+            original_path,
+            normalized,
+            unquote(original_path),
+            unquote(normalized),
+            posixpath.basename(normalized),
+        }
+        for alias in aliases:
+            if alias:
+                image_aliases[alias] = image_path
+
+    @staticmethod
+    def _rewrite_image_sources(
+        soup: BeautifulSoup,
+        image_aliases: Dict[str, str],
+        base_path: str = "",
+    ) -> None:
+        for img in soup.find_all("img"):
+            src = (img.get("src") or "").strip()
+            if not src:
+                continue
+            normalized_src = EPUBParser._normalize_epub_path(src)
+            candidates = [
+                src,
+                normalized_src,
+                unquote(src),
+                unquote(normalized_src),
+                posixpath.basename(normalized_src),
+            ]
+            if base_path:
+                joined = EPUBParser._normalize_epub_path(posixpath.join(base_path, src))
+                candidates.extend([joined, unquote(joined)])
+            for candidate in candidates:
+                if candidate in image_aliases:
+                    img["src"] = image_aliases[candidate]
+                    break
+
+    @staticmethod
+    def _normalize_epub_path(path: str) -> str:
+        path = unquote(path).split("#", 1)[0].split("?", 1)[0].replace("\\", "/")
+        normalized = posixpath.normpath(path)
+        return "" if normalized == "." else normalized.lstrip("/")

docreader/parser/excel_convert.py

@@ -0,0 +1,149 @@
+"""LibreOffice helpers for normalizing legacy or unusual Excel uploads."""
+
+from __future__ import annotations
+
+import logging
+import os
+import subprocess
+import tempfile
+import time
+from pathlib import Path
+from typing import Optional
+
+logger = logging.getLogger(__name__)
+
+_XLS_MAGIC = b"\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1"
+_ZIP_MAGIC = b"PK\x03\x04"
+
+
+def detect_excel_format(content: bytes) -> str | None:
+    """Return pandas/excel format id: xlsx, xls, xlsb, ods, or None."""
+    if not content:
+        return None
+
+    from pandas.io.excel._base import inspect_excel_format
+
+    ext = inspect_excel_format(content_or_path=content)
+    if ext in ("xlsx", "xls", "xlsb", "ods"):
+        return ext
+    if ext == "zip":
+        return "xlsx"
+
+    if content.startswith(_ZIP_MAGIC):
+        return "xlsx"
+    if len(content) >= len(_XLS_MAGIC) and content.startswith(_XLS_MAGIC):
+        return "xls"
+    return None
+
+
+def engine_for_format(ext: str | None) -> str:
+    if ext == "xls":
+        return "xlrd"
+    if ext in ("xlsx", "xlsb"):
+        return "openpyxl"
+    if ext == "ods":
+        return "odf"
+    return "openpyxl"
+
+
+def convert_excel_to_xlsx_bytes(content: bytes, suffix: str = ".xlsx") -> bytes | None:
+    """Convert arbitrary spreadsheet bytes to XLSX using LibreOffice, if available."""
+    soffice = find_soffice()
+    if not soffice:
+        return None
+
+    max_attempts = 3
+    for attempt in range(1, max_attempts + 1):
+        with tempfile.TemporaryDirectory() as temp_dir, tempfile.TemporaryDirectory() as profile_dir:
+            src = os.path.join(temp_dir, f"input{suffix}")
+            with open(src, "wb") as handle:
+                handle.write(content)
+
+            user_installation = Path(profile_dir).as_uri()
+            cmd = [
+                soffice,
+                "--headless",
+                f"-env:UserInstallation={user_installation}",
+                "--convert-to",
+                "xlsx",
+                "--outdir",
+                temp_dir,
+                src,
+            ]
+            try:
+                result = subprocess.run(cmd, capture_output=True, timeout=120)
+            except (OSError, subprocess.TimeoutExpired) as exc:
+                logger.warning("LibreOffice convert failed to start: %s", exc)
+                return None
+
+            if result.returncode != 0:
+                stderr = result.stderr.decode("utf-8", errors="ignore")
+                logger.warning(
+                    "LibreOffice convert failed (attempt %s/%s): %s",
+                    attempt,
+                    max_attempts,
+                    stderr,
+                )
+                if attempt < max_attempts:
+                    time.sleep(0.5 * attempt)
+                    continue
+                return None
+
+            for name in os.listdir(temp_dir):
+                if name.endswith(".xlsx"):
+                    with open(os.path.join(temp_dir, name), "rb") as handle:
+                        converted = handle.read()
+                    logger.info(
+                        "Converted spreadsheet via LibreOffice (%s -> xlsx, %d bytes)",
+                        suffix,
+                        len(converted),
+                    )
+                    return converted
+
+            if attempt < max_attempts:
+                time.sleep(0.5 * attempt)
+    return None
+
+
+def normalize_excel_bytes(content: bytes, file_type: str | None = None) -> bytes:
+    """Return bytes readable by pandas, converting via LibreOffice when needed."""
+    ext = detect_excel_format(content)
+    if ext is not None:
+        return content
+
+    suffixes = []
+    if file_type:
+        suffixes.append(f".{file_type.lstrip('.')}")
+    suffixes.extend([".xlsx", ".xls", ".et", ".csv"])
+    seen: set[str] = set()
+    for suffix in suffixes:
+        if suffix in seen:
+            continue
+        seen.add(suffix)
+        converted = convert_excel_to_xlsx_bytes(content, suffix=suffix)
+        if converted and detect_excel_format(converted) is not None:
+            return converted
+
+    raise ValueError(
+        "Unrecognized Excel file format; the file may be corrupt, encrypted, "
+        "or not a spreadsheet"
+    )
+
+
+def find_soffice() -> Optional[str]:
+    possible_paths = [
+        "/usr/bin/soffice",
+        "/usr/lib/libreoffice/program/soffice",
+        "/opt/libreoffice25.2/program/soffice",
+        "/Applications/LibreOffice.app/Contents/MacOS/soffice",
+        "C:\\Program Files\\LibreOffice\\program\\soffice.exe",
+        "C:\\Program Files (x86)\\LibreOffice\\program\\soffice.exe",
+    ]
+    for path in possible_paths:
+        if path and os.path.exists(path):
+            return path
+
+    result = subprocess.run(["which", "soffice"], capture_output=True, text=True)
+    if result.returncode == 0 and result.stdout.strip():
+        return result.stdout.strip()
+    return None

docreader/parser/excel_parser.py

@@ -0,0 +1,265 @@
+"""
+Excel Parser Module
+
+This module provides functionality to parse Excel files (.xlsx, .xls) into
+structured Document objects with text content and chunks. It supports multiple
+sheets and handles various Excel formats using pandas.
+"""
+import logging
+import re
+from io import BytesIO
+from typing import Any, List
+
+import pandas as pd
+
+from docreader.models.document import Chunk, Document
+from docreader.parser.base_parser import BaseParser
+from docreader.parser.excel_convert import (
+    convert_excel_to_xlsx_bytes,
+    detect_excel_format,
+    engine_for_format,
+    normalize_excel_bytes,
+)
+from docreader.parser.xlsx_merge import fill_merged_cells_xlsx
+from docreader.parser.xlsx_repair import repair_xlsx_bytes
+
+logger = logging.getLogger(__name__)
+
+# Pattern to detect Excel image function strings that should be excluded from
+# parsed text content.  WPS uses =DISPIMG("ID",mode) to embed images in cells;
+# when opened by other tools the formula may appear as plain text prefixed with
+# "_xlfn." or "=".  Office 365 uses =_xlfn.IMAGE(url, ...) similarly.
+# The _xlfn. prefix is optional — WPS may omit it (e.g. =DISPIMG("ID",1)).
+_IMAGE_FUNC_RE = re.compile(
+    r"^=?(_xlfn\.)?(DISPIMG|IMAGE)\(", re.IGNORECASE
+)
+
+
+def _is_image_function(value: object) -> bool:
+    """Return True if *value* looks like an embedded-image function string."""
+    if not isinstance(value, str):
+        return False
+    return _IMAGE_FUNC_RE.match(value) is not None
+
+
+class ExcelParser(BaseParser):
+    """Parser for Excel files (.xlsx, .xls).
+    
+    This parser extracts text content from Excel files by processing all sheets
+    and converting each row into a structured text format. Each row becomes a
+    separate chunk with key-value pairs.
+    
+    Features:
+        - Supports multiple sheets in a single Excel file
+        - Automatically removes completely empty rows
+        - Converts each row to "column: value" format
+        - Creates individual chunks for each row for better granularity
+        
+    Example:
+        >>> parser = ExcelParser()
+        >>> with open("data.xlsx", "rb") as f:
+        ...     content = f.read()
+        ...     document = parser.parse_into_text(content)
+        >>> print(document.content)
+        Name: John,Age: 30,City: NYC
+        Name: Jane,Age: 25,City: LA
+    """
+
+    def __init__(
+        self,
+        file_name: str = "",
+        file_type: str | None = None,
+        xlsx_first_row_as_header: Any = False,
+        **kwargs: Any,
+    ):
+        super().__init__(file_name=file_name, file_type=file_type, **kwargs)
+        self.xlsx_first_row_as_header = _parse_bool(xlsx_first_row_as_header)
+    
+    def parse_into_text(self, content: bytes) -> Document:
+        """Parse Excel file bytes into a Document object.
+        
+        Args:
+            content: Raw bytes of the Excel file
+            
+        Returns:
+            Document: Parsed document containing:
+                - content: Full text with all rows from all sheets
+                - chunks: List of Chunk objects, one per row
+                
+        Note:
+            - Empty rows (all NaN values) are automatically skipped
+            - Each row is formatted as: "col1: val1,col2: val2,..."
+            - Chunks maintain sequential ordering across all sheets
+        """
+        chunks: List[Chunk] = []
+        text: List[str] = []
+        start, end = 0, 0
+
+        excel_file = _open_excel_file(content, file_type=self.file_type)
+        
+        # Process each sheet in the Excel file
+        for excel_sheet_name in excel_file.sheet_names:
+            df = _read_sheet_dataframe(
+                excel_file,
+                excel_sheet_name,
+                xlsx_first_row_as_header=self.xlsx_first_row_as_header,
+            )
+            # Remove rows where all values are NaN (completely empty rows)
+            df.dropna(how="all", inplace=True)
+
+            # Process each row in the DataFrame
+            for _, row in df.iterrows():
+                page_content = []
+                # Build key-value pairs for non-null values
+                for k, v in row.items():
+                    if pd.notna(v) and not _is_image_function(v):
+                        page_content.append(f"{k}: {v}")
+                
+                # Skip rows with no valid content
+                if not page_content:
+                    continue
+                
+                # Format row as comma-separated key-value pairs
+                content_row = ",".join(page_content) + "\n"
+                end += len(content_row)
+                text.append(content_row)
+                
+                # Create a chunk for this row with position tracking
+                chunks.append(
+                    Chunk(content=content_row, seq=len(chunks), start=start, end=end)
+                )
+                start = end
+
+        # Combine all text and return as Document
+        return Document(content="".join(text), chunks=chunks)
+
+
+def _read_sheet_dataframe(
+    excel_file: pd.ExcelFile,
+    sheet_name: str,
+    xlsx_first_row_as_header: bool = False,
+) -> pd.DataFrame:
+    """Read a worksheet into a DataFrame with stable column labels."""
+    from openpyxl.utils import get_column_letter
+
+    # Keep row 1 as data by default for both XLSX and legacy XLS. Users can
+    # explicitly restore the historical behavior where row 1 supplies semantic
+    # labels for every row.
+    df = excel_file.parse(sheet_name=sheet_name, header=None)
+    if xlsx_first_row_as_header and len(df.index) >= 2:
+        df.columns = _stable_header_labels(df.iloc[0].tolist())
+        return df.iloc[1:].copy()
+
+    df.columns = [get_column_letter(idx + 1) for idx in range(len(df.columns))]
+    return df
+
+
+def _stable_header_labels(values: List[object]) -> List[str]:
+    """Build non-empty, unique labels from an explicitly selected header row."""
+    from openpyxl.utils import get_column_letter
+
+    labels: List[str] = []
+    counts: dict[str, int] = {}
+    for index, value in enumerate(values, start=1):
+        label = ""
+        if pd.notna(value) and not _is_image_function(value):
+            label = str(value).strip()
+        if not label:
+            label = get_column_letter(index)
+
+        count = counts.get(label, 0) + 1
+        counts[label] = count
+        labels.append(label if count == 1 else f"{label}_{count}")
+    return labels
+
+
+def _parse_bool(value: Any) -> bool:
+    if isinstance(value, bool):
+        return value
+    return str(value).strip().lower() in {"1", "true", "yes", "on"}
+
+
+def _prepare_xlsx_bytes(data: bytes) -> bytes:
+    repaired = repair_xlsx_bytes(data)
+    if repaired is not None:
+        data = repaired
+    return fill_merged_cells_xlsx(data)
+
+
+def _open_excel_file(content: bytes, file_type: str | None = None) -> pd.ExcelFile:
+    """Open an Excel workbook with explicit engine selection and fallbacks."""
+    data = content
+    converted_via_soffice = False
+
+    while True:
+        ext = detect_excel_format(data)
+        if ext is None:
+            if converted_via_soffice:
+                raise ValueError(
+                    "Excel file format cannot be determined, you must specify an engine manually."
+                )
+            try:
+                data = normalize_excel_bytes(data, file_type=file_type)
+            except ValueError as exc:
+                raise ValueError(
+                    "Excel file format cannot be determined, you must specify an engine manually."
+                ) from exc
+            converted_via_soffice = True
+            continue
+
+        if ext == "ods":
+            converted = convert_excel_to_xlsx_bytes(data, suffix=".ods")
+            if converted:
+                data = converted
+                continue
+
+        engine = engine_for_format(ext)
+        if ext == "xlsx":
+            data = _prepare_xlsx_bytes(data)
+            engine = "openpyxl"
+        try:
+            return pd.ExcelFile(BytesIO(data), engine=engine)
+        except ImportError as exc:
+            raise ValueError(
+                f"Excel engine {engine!r} is not available for .{ext} files"
+            ) from exc
+        except KeyError as exc:
+            if "sharedStrings.xml" not in str(exc) or engine != "openpyxl":
+                raise
+            repaired = repair_xlsx_bytes(data)
+            if repaired is None:
+                raise
+            logger.info("Repaired XLSX sharedStrings packaging before parse")
+            data = _prepare_xlsx_bytes(repaired)
+            continue
+        except ValueError as exc:
+            if converted_via_soffice or "cannot be determined" not in str(exc):
+                raise
+            try:
+                data = normalize_excel_bytes(content, file_type=file_type)
+            except ValueError:
+                raise
+            converted_via_soffice = True
+            continue
+
+
+if __name__ == "__main__":
+    # Example usage: Parse an Excel file and display results
+    logging.basicConfig(level=logging.DEBUG)
+
+    # Specify the path to your Excel file
+    your_file = "/path/to/your/file.xlsx"
+    parser = ExcelParser()
+    
+    # Read and parse the Excel file
+    with open(your_file, "rb") as f:
+        content = f.read()
+        document = parser.parse_into_text(content)
+        
+        # Display the full document content
+        logger.error(document.content)
+
+        # Display the first chunk as an example
+        for chunk in document.chunks:
+            logger.error(chunk.content)
+            break  # Only show the first chunk

docreader/parser/html_parser.py

@@ -0,0 +1,37 @@
+"""Static HTML file parser."""
+
+from bs4 import BeautifulSoup
+
+from docreader.models.document import Document
+from docreader.parser.base_parser import BaseParser
+from docreader.parser.chain_parser import PipelineParser
+from docreader.parser.markdown_parser import MarkdownParser
+from docreader.parser.mhtml_parser import MHTMLParser
+
+
+class HTMLToMarkdownParser(BaseParser):
+    """Convert uploaded HTML bytes to Markdown without browser or network access."""
+
+    def parse_into_text(self, content: bytes) -> Document:
+        if not content.strip():
+            return Document()
+
+        # Inspect the original bytes so BOMs and HTML charset declarations are
+        # honored before the shared Markdown conversion runs.
+        html = BeautifulSoup(content, "lxml").decode()
+        markdown = MHTMLParser(
+            file_name=self.file_name,
+            file_type=self.file_type,
+            extract_images=False,
+        ).html_to_markdown(
+            html,
+            strip_internal_links=False,
+            fallback_to_raw_html=False,
+        )
+        return Document(content=markdown or "")
+
+
+class HTMLParser(PipelineParser):
+    """Extract static HTML content and normalize the resulting Markdown."""
+
+    _parser_cls = (HTMLToMarkdownParser, MarkdownParser)

docreader/parser/image_parser.py

@@ -0,0 +1,28 @@
+import base64
+import logging
+import os
+
+from docreader.models.document import Document
+from docreader.parser.base_parser import BaseParser
+
+logger = logging.getLogger(__name__)
+
+
+class ImageParser(BaseParser):
+    """Parser for standalone image files.
+
+    Returns the image as a markdown reference with the raw image data
+    in Document.images so that the Go-side ImageResolver (or main.py's
+    _resolve_images) can handle storage upload.
+    """
+
+    def parse_into_text(self, content: bytes) -> Document:
+        logger.info("Parsing image file=%s, size=%d bytes", self.file_name, len(content))
+
+        ext = os.path.splitext(self.file_name)[1].lower() or ".png"
+        ref_path = f"images/{self.file_name}"
+
+        text = f"![{self.file_name}]({ref_path})"
+        images = {ref_path: base64.b64encode(content).decode()}
+
+        return Document(content=text, images=images)

docreader/parser/markdown_parser.py

@@ -0,0 +1,489 @@
+"""
+Markdown Parser Module
+
+This module provides comprehensive Markdown parsing functionality including:
+- Table formatting and standardization
+- Base64 image extraction and conversion
+- Image path replacement and URL generation
+- Pipeline-based parsing with multiple stages
+
+The parser uses a pipeline approach to process Markdown content through
+multiple stages: table formatting -> image processing.
+"""
+
+import base64
+import logging
+import os
+import re
+import uuid
+from typing import Dict, List, Match, Optional, Tuple
+
+from docreader.models.document import Document
+from docreader.parser.base_parser import BaseParser
+from docreader.parser.chain_parser import PipelineParser
+from docreader.utils import endecode
+
+# Get logger object
+logger = logging.getLogger(__name__)
+
+_SEPARATOR_CELL = re.compile(r"^:?-{3,}:?$")
+# Cells of a GFM alignment row as matched by the previous ``align_pattern``.
+# Looser than ``_SEPARATOR_CELL``: any non-empty run of ``:``/``-`` qualifies.
+_ALIGNMENT_CELL = re.compile(r"^[:-]+$")
+
+
+class MarkdownTableUtil:
+    """Utility class for formatting Markdown tables.
+
+    This class standardizes Markdown table formatting by:
+    - Normalizing column alignment markers (e.g., :---, :---:, ---:)
+    - Adding consistent spacing around pipes (|)
+    - Preserving indentation levels
+    - Handling both header rows and data rows
+
+    Example:
+        Input:  |姓名|年龄|城市|
+                |:---|---:|:---:|
+                |张三|25|北京|
+
+        Output: | 姓名 | 年龄 | 城市 |
+                | :--- | ---: | :---: |
+                | 张三 | 25 | 北京 |
+    """
+
+    @staticmethod
+    def _split_row_cells(row_line: str) -> List[str]:
+        """Split a markdown table row into cells, preserving empty cells."""
+        inner = row_line.strip()
+        if not inner.startswith("|"):
+            return []
+        parts = inner.split("|")
+        if parts and parts[0].strip() == "":
+            parts = parts[1:]
+        if parts and parts[-1].strip() == "":
+            parts = parts[:-1]
+        return [part.strip() for part in parts]
+
+    @staticmethod
+    def _is_table_row(line: str) -> bool:
+        stripped = line.strip()
+        return stripped.startswith("|") and "|" in stripped[1:]
+
+    @classmethod
+    def _is_separator_row(cls, line: str) -> bool:
+        cells = cls._split_row_cells(line)
+        return bool(cells) and all(_SEPARATOR_CELL.match(cell) for cell in cells)
+
+    @classmethod
+    def _is_alignment_row(cls, line: str) -> bool:
+        """Detect a GFM alignment row, matching the previous ``align_pattern``.
+
+        Deliberately looser than :meth:`_is_separator_row` (which needs three
+        or more dashes per cell) so formatting output is unchanged for rows
+        like ``| - | - |``.
+        """
+        cells = cls._split_row_cells(line)
+        return bool(cells) and all(_ALIGNMENT_CELL.match(cell) for cell in cells)
+
+    @classmethod
+    def _is_empty_row(cls, line: str) -> bool:
+        cells = cls._split_row_cells(line)
+        return bool(cells) and all(cell == "" for cell in cells)
+
+    @classmethod
+    def _separator_row_for(cls, header_line: str) -> str:
+        cells = cls._split_row_cells(header_line)
+        return "| " + " | ".join("---" for _ in cells) + " |"
+
+    @classmethod
+    def _normalize_table_block(cls, block: List[str]) -> List[str]:
+        """Fix MarkItDown-style tables: drop bogus prefix rows, ensure GFM delimiter."""
+        while block and cls._is_empty_row(block[0]):
+            block.pop(0)
+        if block and cls._is_separator_row(block[0]):
+            block.pop(0)
+        # GFM/marked need "| --- |" after the first row. Headerless Word tables
+        # only have data rows after we strip the fake empty+separator prefix.
+        if len(block) >= 2 and not cls._is_separator_row(block[1]):
+            sep = cls._separator_row_for(block[0])
+            block = [block[0], sep] + block[1:]
+        return block
+
+    def normalize_spurious_table_prefixes(self, content: str) -> str:
+        """Remove bogus empty/separator prefix rows from MarkItDown table output."""
+        lines = content.split("\n")
+        out: List[str] = []
+        i = 0
+        while i < len(lines):
+            line = lines[i]
+            if not self._is_table_row(line):
+                out.append(line)
+                i += 1
+                continue
+            block: List[str] = []
+            while i < len(lines) and self._is_table_row(lines[i]):
+                block.append(lines[i])
+                i += 1
+            out.extend(self._normalize_table_block(block))
+        return "\n".join(out)
+
+    def format_table(self, content: str) -> str:
+        """Format all Markdown tables in the content.
+
+        Line-based and regex-free: only well-formed rows (leading *and*
+        trailing ``|``, with at least one cell) are treated as tables, so a
+        malformed row can never trigger the catastrophic backtracking
+        described in https://github.com/Tencent/WeKnora/issues/2768.
+        CRLF/CR input is normalized to LF before scanning.
+
+        Args:
+            content: Raw Markdown text containing tables
+
+        Returns:
+            Formatted Markdown text with standardized table formatting
+        """
+
+        def format_alignment_row(prefix: str, columns: List[str]) -> str:
+            """Standardize an alignment row, preserving ``:`` markers."""
+            processed = []
+            for col in columns:
+                # Preserve left alignment marker (:---)
+                left_colon = ":" if col.startswith(":") else ""
+                # Preserve right alignment marker (---:)
+                right_colon = ":" if col.endswith(":") else ""
+                processed.append(left_colon + "---" + right_colon)
+            return prefix + "| " + " | ".join(processed) + " |"
+
+        def format_data_row(prefix: str, columns: List[str]) -> str:
+            """Standardize a header or data row."""
+            return prefix + "| " + " | ".join(columns) + " |"
+
+        # Unify CRLF/CR first so a trailing ``\r`` left by split("\n") cannot
+        # be mistaken for "ends with |" via str.rstrip(), and so we do not
+        # emit mixed line endings. Trailing newlines are preserved.
+        unified = content.replace("\r\n", "\n").replace("\r", "\n")
+        lines = unified.split("\n")
+        out: List[str] = []
+        for line in lines:
+            stripped = line.lstrip(" \t")
+            # A row must both begin and end with a pipe to be a table row;
+            # anything else is passed through unchanged. This also keeps
+            # unclosed rows (the old regex's exponential worst case) cheap.
+            # rstrip only spaces/tabs (old ``\|[\t ]*$``), not all whitespace.
+            if stripped.startswith("|") and line.rstrip(" \t").endswith("|"):
+                prefix = line[: len(line) - len(stripped)]
+                columns = self._split_row_cells(line)
+                # Lone "|" is not a table row: the old line_pattern required
+                # opening *and* closing pipes. Formatting it to "|  |" would
+                # let normalize_spurious_table_prefixes drop the line.
+                if not columns:
+                    out.append(line)
+                elif self._is_alignment_row(line):
+                    out.append(format_alignment_row(prefix, columns))
+                else:
+                    out.append(format_data_row(prefix, columns))
+            else:
+                out.append(line)
+        return self.normalize_spurious_table_prefixes("\n".join(out))
+
+    @staticmethod
+    def _self_test():
+        test_content = """
+# 测试表格
+普通文本---不会被匹配
+
+## 表格1（无前置空格）
+
+| 姓名   | 年龄  | 城市          |
+|      :---------- | -------: | :------      |
+| 张三 | 25 | 北京 |
+
+## 表格3（前置4个空格+首尾|）
+    |   产品   |   价格   |   库存   |
+    | :-------------: | ----------- | :-----------: |
+    | 手机 | 5999       | 100 |
+"""
+        util = MarkdownTableUtil()
+        format_content = util.format_table(test_content)
+        print(format_content)
+
+
+class MarkdownTableFormatter(BaseParser):
+    """Parser for formatting Markdown tables.
+
+    This parser standardizes the formatting of all Markdown tables in the
+    document to ensure consistent spacing and alignment markers.
+
+    Example:
+        >>> formatter = MarkdownTableFormatter()
+        >>> content = b"|Name|Age|\n|---|---|\n|John|30|"
+        >>> doc = formatter.parse_into_text(content)
+        >>> print(doc.content)
+        | Name | Age |
+        | --- | --- |
+        | John | 30 |
+    """
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        self.table_helper = MarkdownTableUtil()
+
+    def parse_into_text(self, content: bytes) -> Document:
+        """Parse and format Markdown tables.
+
+        Args:
+            content: Raw Markdown content as bytes
+
+        Returns:
+            Document with formatted table content
+        """
+        # Decode bytes to string with automatic encoding detection
+        text = endecode.decode_bytes(content)
+        # Format all tables in the content
+        text = self.table_helper.format_table(text)
+        return Document(content=text)
+
+
+class MarkdownImageUtil:
+    """Utility class for handling images in Markdown.
+
+    This class provides functionality to:
+    - Extract base64-encoded images from Markdown
+    - Extract image paths from Markdown
+    - Replace image paths with new URLs
+    - Convert base64 images to binary format
+
+    Supported formats:
+    - Base64 embedded images: ![alt](data:image/png;base64,iVBORw0...)
+    - Regular image links: ![alt](path/to/image.png)
+    """
+
+    def __init__(self):
+        # Pattern to match base64 embedded images
+        # Captures: (1) alt text, (2) image format, (3) base64 data
+        # Alt text uses .*? (non-greedy) to allow literal ] (e.g. Windows paths).
+        # MIME subtype uses [^;]+ to handle types with hyphens like x-emf.
+        self.b64_pattern = re.compile(
+            r"!\[(.*?)\]\(data:image/([^;]+);base64,([^\)]+)\)"
+        )
+        # Pattern to match regular image syntax (alt text allows ])
+        self.image_pattern = re.compile(r"!\[(.*?)\]\(([^)]+)\)")
+        # Pattern for replacing image paths
+        self.replace_pattern = re.compile(r"!\[(.*?)\]\(([^)]+)\)")
+
+    def extract_image(
+        self,
+        content: str,
+        path_prefix: Optional[str] = None,
+        replace: bool = True,
+    ) -> Tuple[str, List[str]]:
+        """Extract image paths from Markdown content.
+
+        Args:
+            content: Markdown text containing images
+            path_prefix: Optional prefix to add to image paths
+            replace: Whether to replace image syntax in content
+
+        Returns:
+            Tuple of (processed_text, list_of_image_paths)
+
+        Example:
+            >>> util = MarkdownImageUtil()
+            >>> text, images = util.extract_image("![logo](img/logo.png)")
+            >>> print(images)
+            ['img/logo.png']
+        """
+        # List to store extracted image paths
+        images: List[str] = []
+
+        def repl(match: Match[str]) -> str:
+            """Replacement function for each image match."""
+            title = match.group(1)  # Alt text
+            image_path = match.group(2)  # Image path
+
+            # Add prefix if specified
+            if path_prefix:
+                image_path = f"{path_prefix}/{image_path}"
+
+            images.append(image_path)
+
+            # Keep original if replace is False
+            if not replace:
+                return match.group(0)
+
+            # Replace image path with potentially prefixed path
+            return f"![{title}]({image_path})"
+
+        text = self.image_pattern.sub(repl, content)
+        logger.debug(f"Extracted {len(images)} images from markdown")
+        return text, images
+
+    def extract_base64(
+        self,
+        content: str,
+        path_prefix: Optional[str] = None,
+        replace: bool = True,
+    ) -> Tuple[str, Dict[str, bytes]]:
+        """Extract and decode base64 embedded images from Markdown.
+
+        This method finds all base64-encoded images in the Markdown content,
+        decodes them to binary format, generates unique filenames, and
+        optionally replaces them with file path references.
+
+        Args:
+            content: Markdown text containing base64 images
+            path_prefix: Optional directory prefix for generated paths
+            replace: Whether to replace base64 syntax with file paths
+
+        Returns:
+            Tuple of (processed_text, dict_of_path_to_bytes)
+
+        Example:
+            >>> util = MarkdownImageUtil()
+            >>> text = "![logo](data:image/png;base64,iVBORw0KGg...)"
+            >>> new_text, images = util.extract_base64(text, "images")
+            >>> print(new_text)
+            ![logo](images/uuid.png)
+            >>> print(len(images))
+            1
+        """
+        # Dictionary mapping generated file paths to binary image data
+        images: Dict[str, bytes] = {}
+
+        def repl(match: Match[str]) -> str:
+            """Replacement function for each base64 image match."""
+            title = match.group(1)  # Alt text
+            img_ext = match.group(2)  # Image format (png, jpg, etc.)
+            img_b64 = match.group(3)  # Base64 encoded data
+
+            # Decode base64 string to bytes
+            image_byte = endecode.encode_image(img_b64, errors="ignore")
+            if not image_byte:
+                logger.error(f"Failed to decode base64 image skip it: {img_b64}")
+                return title  # Return just the alt text if decode fails
+
+            # Generate unique filename with original extension
+            image_path = f"{uuid.uuid4()}.{img_ext}"
+            if path_prefix:
+                image_path = f"{path_prefix}/{image_path}"
+            images[image_path] = image_byte
+
+            # Keep original base64 if replace is False
+            if not replace:
+                return match.group(0)
+
+            # Replace base64 data with file path reference
+            return f"![{title}]({image_path})"
+
+        text = self.b64_pattern.sub(repl, content)
+        logger.debug(f"Extracted {len(images)} base64 images from markdown")
+        return text, images
+
+    def replace_path(self, content: str, images: Dict[str, str]) -> str:
+        """Replace image paths in Markdown with new URLs.
+
+        This method is typically used to replace local file paths with
+        uploaded URLs after images have been stored.
+
+        Args:
+            content: Markdown text with image references
+            images: Mapping of old paths to new URLs
+
+        Returns:
+            Markdown text with updated image URLs
+
+        Example:
+            >>> util = MarkdownImageUtil()
+            >>> content = "![logo](temp/img.png)"
+            >>> mapping = {"temp/img.png": "https://cdn.com/img.png"}
+            >>> result = util.replace_path(content, mapping)
+            >>> print(result)
+            ![logo](https://cdn.com/img.png)
+        """
+        # Track which paths were actually replaced
+        content_replace: set = set()
+
+        def repl(match: Match[str]) -> str:
+            """Replacement function for each image match."""
+            title = match.group(1)  # Alt text
+            image_path = match.group(2)  # Current image path
+
+            # Only replace if path exists in mapping
+            if image_path not in images:
+                return match.group(0)  # Keep original
+
+            content_replace.add(image_path)
+            # Get new URL from mapping
+            image_path = images[image_path]
+            return f"![{title}]({image_path})" if image_path else title
+
+        text = self.replace_pattern.sub(repl, content)
+        logger.debug(f"Replaced {len(content_replace)} images in markdown")
+        return text
+
+    @staticmethod
+    def _self_test():
+        your_content = "test![](data:image/png;base64,iVBORw0KGgoAAAA)test"
+        image_handle = MarkdownImageUtil()
+        text, images = image_handle.extract_base64(your_content)
+        print(text)
+
+        for image_url, image_byte in images.items():
+            with open(image_url, "wb") as f:
+                f.write(image_byte)
+
+
+class MarkdownImageBase64(BaseParser):
+    """Parser for extracting base64 images from Markdown.
+
+    Extracts base64-encoded images, replaces them with path references,
+    and returns the raw image data in Document.images for the Go-side
+    ImageResolver (or main.py _resolve_images) to handle storage.
+    """
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        self.image_helper = MarkdownImageUtil()
+
+    def parse_into_text(self, content: bytes) -> Document:
+        text = endecode.decode_bytes(content)
+        text, img_b64 = self.image_helper.extract_base64(text, path_prefix="images")
+
+        images: Dict[str, str] = {}
+        for ipath, raw_bytes in img_b64.items():
+            images[ipath] = base64.b64encode(raw_bytes).decode()
+
+        logger.debug("Extracted %d base64 images from markdown", len(images))
+        return Document(content=text, images=images)
+
+
+class MarkdownParser(PipelineParser):
+    """Complete Markdown parser using pipeline approach.
+
+    This parser processes Markdown content through multiple stages:
+    1. MarkdownTableFormatter: Standardizes table formatting
+    2. MarkdownImageBase64: Extracts and uploads base64 images
+
+    The pipeline ensures that content flows through each parser in sequence,
+    with each stage's output becoming the next stage's input.
+    """
+
+    _parser_cls = (MarkdownTableFormatter, MarkdownImageBase64)
+
+
+if __name__ == "__main__":
+    # Example usage and testing
+    logging.basicConfig(level=logging.DEBUG)
+
+    # Test the complete MarkdownParser pipeline
+    your_content = "test![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAMgA)test"
+    parser = MarkdownParser()
+
+    # Parse content and display results
+    document = parser.parse_into_text(your_content.encode())
+    logger.info(document.content)
+    logger.info(f"Images: {len(document.images)}, name: {document.images.keys()}")
+
+    # Run individual utility tests
+    MarkdownImageUtil._self_test()
+    MarkdownTableUtil._self_test()

docreader/parser/markitdown_parser.py

@@ -0,0 +1,83 @@
+import io
+import logging
+
+from markitdown import MarkItDown
+
+from docreader.config import CONFIG
+from docreader.models.document import Document
+from docreader.parser.base_parser import BaseParser
+from docreader.parser.chain_parser import PipelineParser
+from docreader.parser.concurrency import parser_worker_limit
+from docreader.parser.docx_merge import fill_vertical_merged_cells_docx
+from docreader.parser.markdown_parser import MarkdownParser
+from docreader.parser.ppt_convert import normalize_ppt_bytes
+from docreader.parser.pptx_media import (
+    attach_pptx_media_to_markdown,
+    markdown_needs_pptx_media_attach,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class StdMarkitdownParser(BaseParser):
+    """
+    Standard MarkItDown Parser Wrapper
+
+    This parser uses the markitdown library to convert various document formats
+    (docx, pptx, pdf, etc.) into text/markdown.
+    """
+
+    def __init__(self, *args, **kwargs):
+        # 这里的 super() 会调用 BaseParser 的初始化，确保 self.file_type 被正确赋值
+        super().__init__(*args, **kwargs)
+        self.markitdown = MarkItDown()
+
+    def parse_into_text(self, content: bytes) -> Document:
+        """
+        Parses content using MarkItDown.
+        Uses self.file_type (inherited from BaseParser) to hint the stream format.
+        """
+        ext = self.file_type
+        ft = (ext or "").lstrip(".").lower()
+        pptx_bytes: bytes | None = None
+        if ft in ("ppt", "pptx"):
+            content, ext = normalize_ppt_bytes(content, ft)
+            pptx_bytes = content
+            ft = "pptx"
+        elif ft == "docx":
+            content = fill_vertical_merged_cells_docx(content)
+        if ext and not ext.startswith("."):
+            ext = "." + ext
+
+        with parser_worker_limit("markitdown", CONFIG.markitdown_max_workers):
+            result = self._convert_markitdown(content, ext, keep_data_uris=True)
+            if result is None:
+                logger.warning(
+                    "MarkItDown failed with embedded images for %s; retrying without data URIs",
+                    ft or ext,
+                )
+                result = self._convert_markitdown(content, ext, keep_data_uris=False)
+
+        text = result.text_content
+        images: dict[str, str] = {}
+        if pptx_bytes is not None and markdown_needs_pptx_media_attach(text):
+            text, images = attach_pptx_media_to_markdown(text, pptx_bytes)
+        return Document(content=text, images=images)
+
+    def _convert_markitdown(
+        self, content: bytes, ext: str | None, *, keep_data_uris: bool
+    ):
+        try:
+            return self.markitdown.convert(
+                io.BytesIO(content),
+                file_extension=ext,
+                keep_data_uris=keep_data_uris,
+            )
+        except Exception:
+            if keep_data_uris:
+                return None
+            raise
+
+
+class MarkitdownParser(PipelineParser):
+    _parser_cls = (StdMarkitdownParser, MarkdownParser)

docreader/parser/mhtml_parser.py

@@ -0,0 +1,395 @@
+"""MHTML parser.
+
+Parses MIME HTML web archives into markdown text and optional embedded images.
+"""
+
+import base64
+import email
+import html
+import logging
+import os
+from email.header import decode_header
+from urllib.parse import unquote, urljoin, urlparse
+import uuid
+from typing import Dict
+
+from bs4 import BeautifulSoup
+
+from docreader.models.document import Document
+from docreader.parser.base_parser import BaseParser
+
+logger = logging.getLogger(__name__)
+
+_AD_DOMAINS = (
+    "googleads",
+    "doubleclick",
+    "googlesyndication",
+    "facebook.com/tr",
+    "analytics",
+    "pixel",
+)
+
+_UNKNOWN_8BIT = frozenset({"unknown-8bit", "unknown"})
+
+
+def _header_str(value) -> str:
+    """Normalize a MIME header to str, recovering 8-bit UTF-8 bytes.
+
+    ``email.message_from_bytes`` uses compat32 by default. Browser-saved
+    ``.mhtml`` files often store raw UTF-8 in headers such as Content-Location.
+    Those values come back as ``email.header.Header`` with charset
+    ``unknown-8bit``: ``.strip()`` / ``.lower()`` raise AttributeError, and
+    ``str(Header)`` replaces the bytes with U+FFFD so image aliases no longer
+    match the HTML body.
+    """
+    if value is None:
+        return ""
+    if isinstance(value, bytes):
+        return value.decode("utf-8", errors="replace")
+    if isinstance(value, str):
+        try:
+            return value.encode("ascii", "surrogateescape").decode("utf-8")
+        except UnicodeError:
+            return value
+    try:
+        chunks = decode_header(value)
+    except (TypeError, ValueError, LookupError, UnicodeError):
+        return str(value)
+
+    parts: list[str] = []
+    for chunk, charset in chunks:
+        if isinstance(chunk, bytes):
+            encoding = (charset or "utf-8").lower()
+            if encoding in _UNKNOWN_8BIT:
+                encoding = "utf-8"
+            try:
+                parts.append(chunk.decode(encoding, errors="replace"))
+            except LookupError:
+                parts.append(chunk.decode("utf-8", errors="replace"))
+        elif chunk:
+            try:
+                parts.append(
+                    chunk.encode("ascii", "surrogateescape").decode("utf-8")
+                )
+            except UnicodeError:
+                parts.append(chunk)
+    return "".join(parts)
+
+
+class MHTMLParser(BaseParser):
+    """Parser for MHTML web archives."""
+
+    def __init__(self, *args, extract_images: bool = True, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.extract_images = extract_images
+
+    def parse_into_text(self, content: bytes) -> Document:
+        logger.info(
+            "Parsing MHTML file: %s, size: %d bytes", self.file_name, len(content)
+        )
+        msg = email.message_from_bytes(content)
+
+        html_parts = []
+        images: Dict[str, str] = {}
+        image_aliases: Dict[str, str] = {}
+        metadata: Dict[str, object] = {}
+
+        for part in msg.walk():
+            content_type = part.get_content_type()
+            location = _header_str(part.get("Content-Location", ""))
+
+            if content_type == "text/html":
+                payload = part.get_payload(decode=True)
+                if not payload:
+                    continue
+                charset = part.get_content_charset() or "utf-8"
+                try:
+                    html_text = payload.decode(charset, errors="ignore")
+                except LookupError:
+                    html_text = payload.decode("utf-8", errors="ignore")
+                html_parts.append(
+                    {
+                        "content": html_text,
+                        "location": location,
+                        "size": len(html_text),
+                    }
+                )
+            elif content_type.startswith("image/") and self.extract_images:
+                image_data = part.get_payload(decode=True)
+                if image_data:
+                    image_path = self._image_path_for_part(part, content_type, images)
+                    images[image_path] = base64.b64encode(image_data).decode("utf-8")
+                    self._add_image_aliases(image_aliases, part, image_path)
+
+        main_html = self._select_main_html(html_parts)
+        if not main_html:
+            logger.warning("No HTML content found in MHTML file")
+            return Document(
+                content="", images=images, metadata={"source_format": "mhtml"}
+            )
+        html_content = main_html["content"]
+
+        try:
+            markdown_text = self.html_to_markdown(
+                html_content,
+                image_aliases=image_aliases,
+                base_location=main_html.get("location", ""),
+            )
+        except Exception as e:
+            logger.error("Failed to convert HTML to Markdown: %s", e)
+            markdown_text = f"```html\n{html_content}\n```"
+
+        metadata["source_format"] = "mhtml"
+        metadata["file_size"] = len(content)
+        metadata["image_count"] = len(images)
+        return Document(content=markdown_text, images=images, metadata=metadata)
+
+    def _select_main_html(self, html_parts) -> dict:
+        """Pick the largest non-ad HTML part as the main document body."""
+        if not html_parts:
+            return {}
+
+        def is_ad(location: str) -> bool:
+            loc = _header_str(location)
+            if not loc:
+                return False
+            loc = loc.lower()
+            return any(ad in loc for ad in _AD_DOMAINS)
+
+        non_ad = sorted(
+            (part for part in html_parts if not is_ad(part.get("location", ""))),
+            key=lambda part: part["size"],
+            reverse=True,
+        )
+        if non_ad:
+            logger.info("Selected main HTML: %d bytes", non_ad[0]["size"])
+            return non_ad[0]
+
+        largest = max(html_parts, key=lambda part: part["size"])
+        logger.warning(
+            "Only ad content found, using largest: %d bytes", largest["size"]
+        )
+        return largest
+
+    @staticmethod
+    def _add_image_aliases(
+        image_aliases: Dict[str, str], part, image_path: str
+    ) -> None:
+        """Register the refs an MHTML document may use for an image part."""
+        for raw in (
+            part.get("Content-Location", ""),
+            part.get("Content-ID", ""),
+            part.get("X-Attachment-Id", ""),
+        ):
+            raw = _header_str(raw).strip()
+            if not raw:
+                continue
+            values = {raw, html.unescape(raw), unquote(html.unescape(raw))}
+            cid = raw.strip("<>")
+            if cid:
+                values.add(f"cid:{cid}")
+                values.add(f"cid:{unquote(cid)}")
+            for value in values:
+                if value:
+                    image_aliases[value] = image_path
+
+    @staticmethod
+    def _image_extension(content_type: str) -> str:
+        return {
+            "image/png": ".png",
+            "image/jpeg": ".jpg",
+            "image/gif": ".gif",
+            "image/webp": ".webp",
+            "image/bmp": ".bmp",
+            "image/tiff": ".tiff",
+            "image/x-icon": ".ico",
+        }.get(content_type, ".png")
+
+    @classmethod
+    def _image_path_for_part(
+        cls, part, content_type: str, images: Dict[str, str]
+    ) -> str:
+        """Choose a stable image path when the MHTML part exposes a filename."""
+        ext = cls._image_extension(content_type)
+        location = _header_str(part.get("Content-Location", "")).strip()
+        filename = cls._filename_from_content_location(location)
+        if not filename:
+            return f"images/{uuid.uuid4().hex}{ext}"
+
+        stem, location_ext = os.path.splitext(filename)
+        if not location_ext:
+            filename = f"{filename}{ext}"
+        image_path = f"images/{filename}"
+        if image_path not in images:
+            return image_path
+
+        suffix = 2
+        stem, location_ext = os.path.splitext(filename)
+        while True:
+            candidate = f"images/{stem}_{suffix}{location_ext}"
+            if candidate not in images:
+                return candidate
+            suffix += 1
+
+    @staticmethod
+    def _filename_from_content_location(location: str) -> str:
+        decoded = unquote(html.unescape(_header_str(location).strip()))
+        if not decoded or decoded.lower().startswith("cid:"):
+            return ""
+        path = urlparse(decoded).path or decoded
+        filename = os.path.basename(path)
+        if not filename or filename in {".", ".."}:
+            return ""
+        if "/" in filename or "\\" in filename:
+            return ""
+        return filename
+
+    def html_to_markdown(
+        self,
+        html_content: str,
+        image_aliases: Dict[str, str] | None = None,
+        base_location: str = "",
+        *,
+        strip_internal_links: bool = True,
+        fallback_to_raw_html: bool = True,
+    ) -> str:
+        """Convert HTML to Markdown with explicit link and fallback policies."""
+
+        def raw_html_fallback() -> str:
+            if not fallback_to_raw_html:
+                return ""
+            return f"```html\n{html_content[:50000]}\n```"
+
+        try:
+            from markdownify import markdownify as md
+
+            soup = BeautifulSoup(html_content, "lxml")
+            for tag in soup(["script", "style", "noscript", "iframe"]):
+                tag.decompose()
+            if strip_internal_links:
+                self._strip_internal_links(soup)
+            if image_aliases:
+                self._rewrite_image_sources(soup, image_aliases, base_location)
+            text_fallback = soup.get_text(separator="\n", strip=True)
+            markdown_text = md(str(soup), heading_style="ATX")
+            result = self._normalize_markdown(markdown_text)
+            if not result and text_fallback:
+                logger.warning("Markdown empty, falling back to text extraction")
+                return text_fallback
+            if not result:
+                return raw_html_fallback()
+            return result
+        except ImportError:
+            logger.warning("markdownify not available, returning raw HTML")
+            return raw_html_fallback()
+        except Exception as e:
+            logger.error("HTML to Markdown conversion failed: %s", e)
+            return raw_html_fallback()
+
+    def _html_to_markdown(
+        self,
+        html_content: str,
+        image_aliases: Dict[str, str] | None = None,
+        base_location: str = "",
+    ) -> str:
+        """Backward-compatible wrapper for existing internal callers and tests."""
+        return self.html_to_markdown(html_content, image_aliases, base_location)
+
+    @staticmethod
+    def _normalize_markdown(markdown_text: str) -> str:
+        text = markdown_text.replace("\r\n", "\n").replace("\r", "\n")
+        output: list[str] = []
+        pending_blank = False
+        fence_char: str | None = None
+        fence_len = 0
+
+        for line in text.split("\n"):
+            if fence_char is not None:
+                output.append(line)
+                if MHTMLParser._is_closing_fence(line, fence_char, fence_len):
+                    fence_char = None
+                    fence_len = 0
+                continue
+
+            opening = MHTMLParser._opening_fence(line)
+            if opening is not None:
+                if pending_blank and output:
+                    output.append("")
+                pending_blank = False
+                output.append(line)
+                fence_char, fence_len = opening
+                continue
+
+            if not line.strip(" \t"):
+                pending_blank = True
+                continue
+
+            if pending_blank and output:
+                output.append("")
+            pending_blank = False
+
+            trailing_spaces = len(line) - len(line.rstrip(" "))
+            if trailing_spaces >= 2:
+                line = line.rstrip(" \t") + "  "
+            else:
+                line = line.rstrip(" \t")
+            output.append(line)
+
+        return "\n".join(output).strip("\n")
+
+    @staticmethod
+    def _opening_fence(line: str) -> tuple[str, int] | None:
+        stripped = line.lstrip(" ")
+        if len(line) - len(stripped) > 3 or not stripped:
+            return None
+        fence_char = stripped[0]
+        if fence_char not in {"`", "~"}:
+            return None
+        fence_len = len(stripped) - len(stripped.lstrip(fence_char))
+        if fence_len < 3:
+            return None
+        return fence_char, fence_len
+
+    @staticmethod
+    def _is_closing_fence(line: str, fence_char: str, fence_len: int) -> bool:
+        stripped = line.lstrip(" ")
+        if len(line) - len(stripped) > 3:
+            return False
+        closing_len = len(stripped) - len(stripped.lstrip(fence_char))
+        if closing_len < fence_len:
+            return False
+        return not stripped[closing_len:].strip(" \t")
+
+    @staticmethod
+    def _strip_internal_links(soup: BeautifulSoup) -> None:
+        """Unwrap links that don't point to an external resource."""
+        external = ("http://", "https://", "mailto:", "tel:")
+        for link in soup.find_all("a"):
+            href = (link.get("href") or "").strip().lower()
+            if not href or not href.startswith(external):
+                link.unwrap()
+
+    @staticmethod
+    def _rewrite_image_sources(
+        soup: BeautifulSoup,
+        image_aliases: Dict[str, str],
+        base_location: str = "",
+    ) -> None:
+        for img in soup.find_all("img"):
+            src = (img.get("src") or "").strip()
+            if not src:
+                continue
+            candidates = [
+                src,
+                html.unescape(src),
+                unquote(html.unescape(src)),
+            ]
+            if base_location:
+                candidates.append(urljoin(base_location, src))
+            base_name = os.path.basename(unquote(html.unescape(src)))
+            if base_name:
+                candidates.append(base_name)
+            for candidate in candidates:
+                if candidate in image_aliases:
+                    img["src"] = image_aliases[candidate]
+                    break

docreader/parser/opendataloader_parser.py

@@ -0,0 +1,378 @@
+"""PDF parser backed by OpenDataLoader PDF (Apache-2.0).
+
+Requires Java 11+ on PATH and the ``opendataloader-pdf`` Python package.
+Each ``convert()`` spawns a JVM; concurrency is limited via
+``DOCREADER_ODL_MAX_WORKERS``.
+
+Hybrid mode (``docling-fast``, etc.) needs a running
+``opendataloader-pdf-hybrid`` server — configure ``DOCREADER_ODL_HYBRID_URL``.
+"""
+
+from __future__ import annotations
+
+import base64
+import html
+import logging
+import os
+import re
+import shutil
+import tempfile
+import urllib.error
+import urllib.request
+from typing import Any, Dict, Mapping, Optional, Tuple
+
+from docreader.config import CONFIG
+from docreader.models.document import Document
+from docreader.parser.base_parser import BaseParser
+from docreader.parser.concurrency import parser_worker_limit
+from docreader.utils.ssrf import is_ssrf_safe_url
+
+logger = logging.getLogger(__name__)
+
+_MIN_CHARS_PER_PAGE = 20
+_IMAGE_SUFFIXES = (".png", ".jpg", ".jpeg", ".gif", ".webp", ".bmp")
+_MD_IMAGE_RE = re.compile(r"!\[([^\]]*)\]\(([^)]+)\)")
+_IMAGE_FILE_NUM_RE = re.compile(r"^imageFile(\d+)\.", re.I)
+
+
+class _NoRedirectHandler(urllib.request.HTTPRedirectHandler):
+    """Health probes never need redirects; refusing them closes redirect SSRF."""
+
+    def redirect_request(self, req, fp, code, msg, headers, newurl):
+        return None
+
+
+def _override_str(overrides: Optional[Mapping[str, Any]], key: str, default: str = "") -> str:
+    if overrides:
+        v = overrides.get(key)
+        if v is not None and str(v).strip() != "":
+            return str(v).strip()
+    return default
+
+
+def _override_bool(overrides: Optional[Mapping[str, Any]], key: str, default: bool) -> bool:
+    if overrides:
+        v = overrides.get(key)
+        if v is not None and str(v).strip() != "":
+            return str(v).strip().lower() in {"1", "true", "yes", "y", "on"}
+    return default
+
+
+def _java_available() -> Tuple[bool, str]:
+    if not shutil.which("java"):
+        return False, "需要 Java 11+（JRE），请安装并在 PATH 中配置 java"
+    return True, ""
+
+
+def _package_available() -> Tuple[bool, str]:
+    try:
+        import opendataloader_pdf  # noqa: F401
+    except ImportError as e:
+        return False, f"opendataloader-pdf 未安装: {e}"
+    return True, ""
+
+
+def _ping_hybrid(
+    url: str,
+    *,
+    timeout_sec: float = 5.0,
+    retries: int = 3,
+    retry_delay_sec: float = 2.0,
+) -> Tuple[bool, str]:
+    import time
+
+    base = url.rstrip("/")
+    health_url = f"{base}/health"
+    safe, reason = is_ssrf_safe_url(health_url)
+    if not safe:
+        return False, f"OpenDataLoader hybrid URL 被 SSRF 防护拦截: {reason}"
+
+    opener = urllib.request.build_opener(_NoRedirectHandler())
+    last_err = ""
+    for attempt in range(max(1, retries)):
+        try:
+            req = urllib.request.Request(health_url, method="GET")
+            with opener.open(req, timeout=timeout_sec) as resp:
+                if 200 <= resp.status < 300:
+                    return True, ""
+                last_err = f"hybrid 健康检查 HTTP {resp.status}: {health_url}"
+        except urllib.error.URLError as e:
+            last_err = f"无法连接 OpenDataLoader hybrid 服务 ({health_url}): {e}"
+        except Exception as e:
+            last_err = f"hybrid 健康检查失败: {e}"
+        if attempt + 1 < retries:
+            time.sleep(retry_delay_sec)
+    hint = (
+        "；若刚执行 make dev-start --odl-hybrid，请等待镜像构建/服务就绪"
+        "（docker logs WeKnora-odl-hybrid）"
+    )
+    return False, last_err + hint
+
+
+def opendataloader_available(
+    overrides: Optional[Mapping[str, Any]] = None,
+    quick: bool = False,
+) -> Tuple[bool, str]:
+    """Registry / ListEngines availability probe.
+
+    When ``quick`` is set the hybrid health check uses a single short-timeout
+    attempt with no retries, so it can be used for fast status listing without
+    blocking on long retry/backoff loops. Parsing keeps the patient retry
+    behavior to tolerate a hybrid service that is still starting up.
+    """
+    ok, msg = _java_available()
+    if not ok:
+        return False, msg
+    ok, msg = _package_available()
+    if not ok:
+        return False, msg
+
+    hybrid = _resolve_hybrid(overrides)
+    if hybrid and hybrid.lower() not in ("off", ""):
+        url = _resolve_hybrid_url(overrides)
+        if url:
+            if quick:
+                return _ping_hybrid(url, retries=1, timeout_sec=2.0)
+            return _ping_hybrid(url, retries=6, retry_delay_sec=5.0, timeout_sec=5.0)
+    return True, ""
+
+
+def _resolve_hybrid(overrides: Optional[Mapping[str, Any]] = None) -> str:
+    return _override_str(overrides, "odl_hybrid", CONFIG.odl_hybrid)
+
+
+def _resolve_hybrid_url(overrides: Optional[Mapping[str, Any]] = None) -> str:
+    return _override_str(overrides, "odl_hybrid_url", CONFIG.odl_hybrid_url)
+
+
+def _find_markdown_file(output_dir: str, pdf_stem: str) -> str:
+    candidates = []
+    for root, _, files in os.walk(output_dir):
+        for name in files:
+            if name.lower().endswith(".md"):
+                path = os.path.join(root, name)
+                candidates.append(path)
+    if not candidates:
+        raise FileNotFoundError(f"OpenDataLoader 未在 {output_dir} 生成 markdown 文件")
+    for path in candidates:
+        base = os.path.splitext(os.path.basename(path))[0]
+        if base == pdf_stem or base.startswith(pdf_stem):
+            return path
+    candidates.sort(key=lambda p: os.path.getmtime(p), reverse=True)
+    return candidates[0]
+
+
+def _normalize_odl_image_url(raw: str) -> str:
+    """OpenDataLoader wraps paths as ``<images/foo.png>``; storage may HTML-escape them."""
+    s = html.unescape((raw or "").strip())
+    s = s.replace("&lt;", "<").replace("&gt;", ">").replace("&amp;", "&")
+    s = s.strip().strip("<>").strip().strip('"').strip("'")
+    if s.startswith("./"):
+        s = s[2:]
+    return s.replace("\\", "/")
+
+
+def _canonical_image_ref(abs_path: str, output_dir: str) -> str:
+    """Use ``images/<file>`` keys to match OpenDataLoader markdown conventions."""
+    rel = os.path.relpath(abs_path, output_dir).replace("\\", "/")
+    name = os.path.basename(abs_path)
+    if rel.startswith("images/"):
+        return rel
+    return f"images/{name}"
+
+
+def _collect_images_under_output(output_dir: str) -> Dict[str, str]:
+    """Collect every extracted image under the convert output tree."""
+    images: Dict[str, str] = {}
+    for root, _, files in os.walk(output_dir):
+        for name in files:
+            if not name.lower().endswith(_IMAGE_SUFFIXES):
+                continue
+            abs_path = os.path.join(root, name)
+            ref = _canonical_image_ref(abs_path, output_dir)
+            if ref in images:
+                continue
+            with open(abs_path, "rb") as f:
+                images[ref] = base64.b64encode(f.read()).decode("utf-8")
+    return images
+
+
+def _register_image_alias(aliases: Dict[str, str], alias: str, canonical: str) -> None:
+    key = _normalize_odl_image_url(alias)
+    if key:
+        aliases[key] = canonical
+
+
+def _build_path_alias_map(images: Dict[str, str]) -> Dict[str, str]:
+    """Map ODL markdown spellings (angle brackets, entities, basenames) to dict keys."""
+    aliases: Dict[str, str] = {}
+    for ref in images:
+        base = os.path.basename(ref)
+        variants = [
+            ref,
+            base,
+            f"images/{base}",
+            f"<{ref}>",
+            f"<images/{base}>",
+            f"&lt;{ref}&gt;",
+            f"&lt;images/{base}&gt;",
+        ]
+        for variant in variants:
+            _register_image_alias(aliases, variant, ref)
+    return aliases
+
+
+def _resolve_image_ref(url: str, aliases: Dict[str, str]) -> Optional[str]:
+    key = _normalize_odl_image_url(url)
+    if not key or key.startswith("data:"):
+        return None
+    if key in aliases:
+        return aliases[key]
+    base = os.path.basename(key)
+    for candidate in (base, f"images/{base}"):
+        if candidate in aliases:
+            return aliases[candidate]
+    m = _IMAGE_FILE_NUM_RE.match(base)
+    if m:
+        num = int(m.group(1))
+        numbered = []
+        for ref in {aliases[k] for k in aliases}:
+            bm = _IMAGE_FILE_NUM_RE.match(os.path.basename(ref))
+            if bm:
+                numbered.append((int(bm.group(1)), ref))
+        numbered.sort(key=lambda x: x[0])
+        for n, ref in numbered:
+            if n == num:
+                return ref
+        if numbered and 1 <= num <= len(numbered):
+            return numbered[num - 1][1]
+    return None
+
+
+def _rewrite_markdown_image_refs(
+    markdown: str, images: Dict[str, str]
+) -> str:
+    if not images:
+        return markdown
+    aliases = _build_path_alias_map(images)
+
+    def repl(match: re.Match[str]) -> str:
+        alt, raw_url = match.group(1), match.group(2)
+        url = raw_url.strip().split()[0] if raw_url else ""
+        canonical = _resolve_image_ref(url, aliases)
+        if canonical is None:
+            return match.group(0)
+        return f"![{alt}]({canonical})"
+
+    return _MD_IMAGE_RE.sub(repl, markdown)
+
+
+def _run_convert(
+    pdf_path: str,
+    output_dir: str,
+    image_dir: str,
+    overrides: Optional[Mapping[str, Any]] = None,
+) -> None:
+    import opendataloader_pdf
+
+    kwargs: Dict[str, Any] = {
+        "input_path": pdf_path,
+        "output_dir": output_dir,
+        "format": "markdown",
+        "image_output": "external",
+        "image_dir": image_dir,
+        "quiet": True,
+        "markdown_with_html": _override_bool(
+            overrides, "odl_markdown_with_html", CONFIG.odl_markdown_with_html
+        ),
+    }
+    hybrid = _resolve_hybrid(overrides)
+    if hybrid and hybrid.lower() not in ("off", ""):
+        kwargs["hybrid"] = hybrid
+        hybrid_url = _resolve_hybrid_url(overrides)
+        if hybrid_url:
+            safe, reason = is_ssrf_safe_url(hybrid_url)
+            if not safe:
+                raise RuntimeError(
+                    f"OpenDataLoader hybrid URL 被 SSRF 防护拦截: {reason}"
+                )
+            kwargs["hybrid_url"] = hybrid_url
+        hybrid_mode = _override_str(overrides, "odl_hybrid_mode", CONFIG.odl_hybrid_mode)
+        if hybrid_mode:
+            kwargs["hybrid_mode"] = hybrid_mode
+        if _override_bool(overrides, "odl_hybrid_fallback", CONFIG.odl_hybrid_fallback):
+            kwargs["hybrid_fallback"] = True
+
+    opendataloader_pdf.convert(**kwargs)
+
+
+class OpenDataLoaderParser(BaseParser):
+    """Parse PDFs with OpenDataLoader (layout-aware markdown + external images)."""
+
+    def __init__(self, *args: Any, **kwargs: Any):
+        self._engine_overrides: Dict[str, Any] = {
+            k: v
+            for k, v in kwargs.items()
+            if k.startswith("odl_") or k in ("mineru_endpoint", "mineru_api_key")
+        }
+        super().__init__(*args, **kwargs)
+
+    def parse_into_text(self, content: bytes) -> Document:
+        ok, msg = opendataloader_available(self._engine_overrides)
+        if not ok:
+            raise RuntimeError(msg)
+
+        safe_name = os.path.basename(self.file_name) or "document.pdf"
+        if not safe_name.lower().endswith(".pdf"):
+            safe_name = f"{os.path.splitext(safe_name)[0] or 'document'}.pdf"
+        pdf_stem = os.path.splitext(safe_name)[0]
+
+        max_workers = CONFIG.odl_max_workers
+        with parser_worker_limit("opendataloader", max_workers):
+            with tempfile.TemporaryDirectory(prefix="weknora-odl-") as tmp_dir:
+                pdf_path = os.path.join(tmp_dir, safe_name)
+                with open(pdf_path, "wb") as f:
+                    f.write(content)
+                image_dir = os.path.join(tmp_dir, "images")
+                os.makedirs(image_dir, exist_ok=True)
+
+                _run_convert(
+                    pdf_path,
+                    tmp_dir,
+                    image_dir,
+                    overrides=self._engine_overrides,
+                )
+
+                md_path = _find_markdown_file(tmp_dir, pdf_stem)
+                with open(md_path, encoding="utf-8", errors="replace") as f:
+                    text = f.read()
+
+                images = _collect_images_under_output(tmp_dir)
+                text = _rewrite_markdown_image_refs(text, images)
+
+        if len(text.strip()) < _MIN_CHARS_PER_PAGE:
+            logger.info(
+                "OpenDataLoaderParser: %s yielded little text; "
+                "falling back to builtin scanned renderer",
+                self.file_name,
+            )
+            from docreader.parser.pdf_parser import PDFScannedParser
+
+            return PDFScannedParser(
+                file_name=self.file_name, file_type=self.file_type
+            ).parse_into_text(content)
+
+        logger.info(
+            "OpenDataLoaderParser: %s -> content_len=%d images=%d",
+            self.file_name,
+            len(text),
+            len(images),
+        )
+        return Document(
+            content=text,
+            images=images,
+            metadata={
+                "parser_engine": "opendataloader",
+                "odl_hybrid": _resolve_hybrid(self._engine_overrides) or "off",
+            },
+        )

docreader/parser/parser.py

@@ -0,0 +1,103 @@
+import logging
+from typing import Any, Optional
+
+from docreader.models.document import Document
+from docreader.parser.registry import registry
+from docreader.parser.web_parser import WebParser
+
+logger = logging.getLogger(__name__)
+
+
+# OLE Compound File magic used by legacy binary Microsoft Office files.
+# Some WPS/Word documents keep this payload while being renamed to .docx.
+_OLE_COMPOUND_FILE_MAGIC = b"\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1"
+
+
+def detect_effective_file_type(file_type: str, content: bytes) -> str:
+    """Return the parser file type after checking trustworthy file magic.
+
+    OOXML ``.docx`` files are ZIP containers, while legacy ``.doc`` files
+    use the OLE Compound File format.  Word and WPS tolerate a legacy file
+    renamed to ``.docx``, so route that well-known mismatch through the DOC
+    parser instead of feeding binary OLE data to the DOCX parser.
+    """
+    normalized = file_type.lower().lstrip(".")
+    if normalized == "docx" and content.startswith(_OLE_COMPOUND_FILE_MAGIC):
+        logger.warning(
+            "Detected legacy DOC content with a DOCX extension; using DOC parser"
+        )
+        return "doc"
+    return normalized
+
+
+class Parser:
+    """Document parser facade (lightweight version).
+
+    Converts files/URLs to markdown + image references.
+    No chunking, no storage, no OCR, no VLM.
+    """
+
+    def __init__(self):
+        self.registry = registry
+        logger.info(
+            "Parser initialized with engines: %s",
+            ", ".join(self.registry.get_engine_names()),
+        )
+
+    def parse_file(
+        self,
+        file_name: str,
+        file_type: str,
+        content: bytes,
+        parser_engine: Optional[str] = None,
+        engine_overrides: Optional[dict[str, Any]] = None,
+    ) -> Document:
+        """Parse file content to markdown."""
+        engine = parser_engine or ""
+        overrides = engine_overrides or {}
+        logger.info(
+            "Parsing file: %s, type: %s, engine: %s",
+            file_name,
+            file_type,
+            engine or "builtin",
+        )
+
+        effective_file_type = detect_effective_file_type(file_type, content)
+        cls = self.registry.get_parser_class(engine, effective_file_type)
+        logger.info(
+            "Creating %s parser instance for %s file",
+            cls.__name__,
+            effective_file_type,
+        )
+        parser = cls(
+            file_name=file_name,
+            file_type=effective_file_type,
+            **overrides,
+        )
+
+        logger.info("Starting to parse file content, size: %d bytes", len(content))
+        result = parser.parse(content)
+
+        if not result.content:
+            logger.warning("Parser returned empty content for file: %s", file_name)
+        logger.info("Parsed file %s, content length=%d", file_name, len(result.content))
+        return result
+
+    def parse_url(
+        self,
+        url: str,
+        title: str,
+        parser_engine: Optional[str] = None,
+        engine_overrides: Optional[dict[str, Any]] = None,
+    ) -> Document:
+        """Parse content from a URL to markdown."""
+        logger.info("Parsing URL: %s, title: %s", url, title)
+
+        parser = WebParser(title=title)
+        logger.info("Starting to parse URL content")
+        result = parser.parse(url.encode())
+
+        if not result.content:
+            logger.warning("Parser returned empty content for url: %s", url)
+        logger.info("Parsed url %s, content length=%d", url, len(result.content))
+        return result

docreader/parser/ppt_convert.py

@@ -0,0 +1,116 @@
+"""LibreOffice helpers for legacy binary PowerPoint (.ppt) uploads."""
+
+from __future__ import annotations
+
+import logging
+import os
+import subprocess
+import tempfile
+import time
+from pathlib import Path
+from docreader.parser.excel_convert import find_soffice
+
+logger = logging.getLogger(__name__)
+
+_OLE_MAGIC = b"\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1"
+_ZIP_MAGIC = b"PK\x03\x04"
+
+
+def is_ole_compound(content: bytes) -> bool:
+    return len(content) >= len(_OLE_MAGIC) and content.startswith(_OLE_MAGIC)
+
+
+def is_zip_openxml(content: bytes) -> bool:
+    return len(content) >= len(_ZIP_MAGIC) and content.startswith(_ZIP_MAGIC)
+
+
+def needs_ppt_to_pptx_conversion(content: bytes, file_type: str | None) -> bool:
+    """True when content is legacy .ppt (OLE), not modern .pptx (ZIP)."""
+    ext = (file_type or "").lstrip(".").lower()
+    if ext == "pptx" or is_zip_openxml(content):
+        return False
+    if ext == "ppt" or is_ole_compound(content):
+        return is_ole_compound(content) or ext == "ppt"
+    return False
+
+
+def convert_ppt_to_pptx_bytes(content: bytes, suffix: str = ".ppt") -> bytes | None:
+    """Convert legacy PowerPoint bytes to PPTX using LibreOffice, if available."""
+    soffice = find_soffice()
+    if not soffice:
+        return None
+
+    max_attempts = 3
+    for attempt in range(1, max_attempts + 1):
+        with tempfile.TemporaryDirectory() as temp_dir, tempfile.TemporaryDirectory() as profile_dir:
+            src = os.path.join(temp_dir, f"input{suffix}")
+            with open(src, "wb") as handle:
+                handle.write(content)
+
+            user_installation = Path(profile_dir).as_uri()
+            cmd = [
+                soffice,
+                "--headless",
+                f"-env:UserInstallation={user_installation}",
+                "--convert-to",
+                "pptx",
+                "--outdir",
+                temp_dir,
+                src,
+            ]
+            try:
+                result = subprocess.run(cmd, capture_output=True, timeout=120)
+            except (OSError, subprocess.TimeoutExpired) as exc:
+                logger.warning("LibreOffice PPT convert failed to start: %s", exc)
+                return None
+
+            if result.returncode != 0:
+                stderr = result.stderr.decode("utf-8", errors="ignore")
+                logger.warning(
+                    "LibreOffice PPT convert failed (attempt %s/%s): %s",
+                    attempt,
+                    max_attempts,
+                    stderr,
+                )
+                if attempt < max_attempts:
+                    time.sleep(0.5 * attempt)
+                    continue
+                return None
+
+            for name in os.listdir(temp_dir):
+                if name.endswith(".pptx"):
+                    with open(os.path.join(temp_dir, name), "rb") as handle:
+                        converted = handle.read()
+                    logger.info(
+                        "Converted presentation via LibreOffice (%s -> pptx, %d bytes)",
+                        suffix,
+                        len(converted),
+                    )
+                    return converted
+
+            if attempt < max_attempts:
+                time.sleep(0.5 * attempt)
+    return None
+
+
+def normalize_ppt_bytes(content: bytes, file_type: str | None) -> tuple[bytes, str]:
+    """Return (bytes, extension) suitable for MarkItDown (pptx when converted)."""
+    ext = (file_type or "").lstrip(".").lower()
+
+    if is_zip_openxml(content):
+        return content, ".pptx"
+
+    if not needs_ppt_to_pptx_conversion(content, ext):
+        dotted = f".{ext}" if ext else ".pptx"
+        return content, dotted
+
+    suffix = ".ppt" if ext in ("", "ppt") else f".{ext}"
+    converted = convert_ppt_to_pptx_bytes(content, suffix=suffix)
+    if converted:
+        return converted, ".pptx"
+
+    raise ValueError(
+        "Legacy PowerPoint (.ppt) is not supported by MarkItDown directly; "
+        "LibreOffice is required to convert it to .pptx. Install LibreOffice "
+        "(soffice) in the docreader environment or upload .pptx instead."
+    )

docreader/parser/pptx_media.py

@@ -0,0 +1,154 @@
+"""Extract and rasterize images embedded in PPTX (e.g. WMF) when MarkItDown cannot inline them."""
+
+from __future__ import annotations
+
+import base64
+import io
+import logging
+import os
+import re
+import subprocess
+import tempfile
+import uuid
+import zipfile
+from typing import Dict, List, Tuple
+
+logger = logging.getLogger(__name__)
+
+_MARKDOWN_IMAGE = re.compile(r"!\[([^\]]*)\]\(([^)]+)\)")
+_RASTER_EXT = {".png", ".jpg", ".jpeg", ".gif", ".webp", ".bmp"}
+_VECTOR_EXT = {".wmf", ".emf", ".svg"}
+
+
+def _find_convert() -> str | None:
+    for path in ("/usr/bin/convert", "/usr/local/bin/convert"):
+        if os.path.isfile(path):
+            return path
+    try:
+        result = subprocess.run(
+            ["which", "convert"], capture_output=True, text=True, check=False
+        )
+        if result.returncode == 0 and result.stdout.strip():
+            return result.stdout.strip()
+    except OSError:
+        pass
+    return None
+
+
+def _rasterize_with_imagemagick(data: bytes, suffix: str) -> bytes | None:
+    convert = _find_convert()
+    if not convert:
+        return None
+    with tempfile.TemporaryDirectory() as temp_dir:
+        src = os.path.join(temp_dir, f"input{suffix}")
+        dst = os.path.join(temp_dir, "output.png")
+        with open(src, "wb") as handle:
+            handle.write(data)
+        try:
+            result = subprocess.run(
+                [convert, src, dst],
+                capture_output=True,
+                timeout=60,
+            )
+        except (OSError, subprocess.TimeoutExpired) as exc:
+            logger.warning("ImageMagick convert failed: %s", exc)
+            return None
+        if result.returncode != 0 or not os.path.isfile(dst):
+            stderr = (result.stderr or b"").decode("utf-8", errors="ignore")
+            logger.warning("ImageMagick convert exit %s: %s", result.returncode, stderr)
+            return None
+        with open(dst, "rb") as handle:
+            return handle.read()
+
+
+def _rasterize_with_pillow(data: bytes) -> bytes | None:
+    try:
+        from PIL import Image
+    except ImportError:
+        return None
+    try:
+        img = Image.open(io.BytesIO(data))
+        if img.mode not in ("RGB", "L"):
+            img = img.convert("RGB")
+        out = io.BytesIO()
+        img.save(out, format="PNG")
+        return out.getvalue()
+    except Exception as exc:
+        logger.debug("Pillow could not open media bytes: %s", exc)
+        return None
+
+
+def rasterize_media_bytes(name: str, data: bytes) -> bytes | None:
+    ext = os.path.splitext(name)[1].lower()
+    if ext in _RASTER_EXT:
+        png = _rasterize_with_pillow(data)
+        if png:
+            return png
+    if ext in _VECTOR_EXT or ext in _RASTER_EXT:
+        return _rasterize_with_imagemagick(data, ext or ".bin")
+    return _rasterize_with_imagemagick(data, ext or ".bin")
+
+
+def list_pptx_media(pptx_bytes: bytes) -> List[Tuple[str, bytes]]:
+    """Return (zip path, raw bytes) for each file under ppt/media/, in archive order."""
+    items: List[Tuple[str, bytes]] = []
+    with zipfile.ZipFile(io.BytesIO(pptx_bytes)) as archive:
+        for name in archive.namelist():
+            if not name.startswith("ppt/media/"):
+                continue
+            base = os.path.basename(name)
+            if not base or base.startswith("."):
+                continue
+            items.append((name, archive.read(name)))
+    return items
+
+
+def extract_pptx_media_rasterized(pptx_bytes: bytes) -> List[bytes]:
+    """Rasterize all ppt/media assets to PNG bytes, skipping failures."""
+    rasterized: List[bytes] = []
+    for path, raw in list_pptx_media(pptx_bytes):
+        png = rasterize_media_bytes(os.path.basename(path), raw)
+        if png:
+            rasterized.append(png)
+            logger.info("Rasterized pptx media %s (%d -> %d bytes)", path, len(raw), len(png))
+        else:
+            logger.warning("Failed to rasterize pptx media %s", path)
+    return rasterized
+
+
+def _is_unresolved_image_ref(url: str) -> bool:
+    if not url or url.startswith("data:") or url.startswith("images/"):
+        return False
+    if url.startswith(("http://", "https://")):
+        return False
+    return True
+
+
+def attach_pptx_media_to_markdown(
+    markdown: str, pptx_bytes: bytes
+) -> Tuple[str, Dict[str, str]]:
+    """Replace unresolved ![](...) refs with images/ paths and inline image payloads."""
+    media = extract_pptx_media_rasterized(pptx_bytes)
+    if not media:
+        return markdown, {}
+
+    images: Dict[str, str] = {}
+    media_iter = iter(media)
+
+    def repl(match: re.Match[str]) -> str:
+        alt, url = match.group(1), match.group(2)
+        if not _is_unresolved_image_ref(url):
+            return match.group(0)
+        try:
+            png = next(media_iter)
+        except StopIteration:
+            return match.group(0)
+        ref = f"images/{uuid.uuid4()}.png"
+        images[ref] = base64.b64encode(png).decode()
+        return f"![{alt}]({ref})"
+
+    return _MARKDOWN_IMAGE.sub(repl, markdown), images
+
+
+def markdown_needs_pptx_media_attach(markdown: str) -> bool:
+    return any(_is_unresolved_image_ref(m.group(2)) for m in _MARKDOWN_IMAGE.finditer(markdown))

docreader/parser/registry.py

@@ -0,0 +1,216 @@
+import logging
+from typing import Callable, Dict, List, Optional, Tuple, Type
+
+from docreader.parser.base_parser import BaseParser
+from docreader.parser.doc_parser import DocParser
+from docreader.parser.docx2_parser import Docx2Parser
+from docreader.parser.epub_parser import EPUBParser
+from docreader.parser.excel_parser import ExcelParser
+from docreader.parser.html_parser import HTMLParser
+from docreader.parser.image_parser import ImageParser
+from docreader.parser.markdown_parser import MarkdownParser
+from docreader.parser.markitdown_parser import MarkitdownParser
+from docreader.parser.mhtml_parser import MHTMLParser
+from docreader.parser.opendataloader_parser import (
+    OpenDataLoaderParser,
+    opendataloader_available,
+)
+from docreader.parser.pdf_parser import PDFParser
+from docreader.parser.xmind_parser import XMindParser
+
+logger = logging.getLogger(__name__)
+
+BUILTIN_ENGINE = "builtin"
+
+# File types that builtin does not implement natively. When the requested
+# engine (including empty / builtin) cannot parse them, route to this engine
+# instead of raising ValueError — PPT/PPTX/CSV are markitdown-only.
+_DEFAULT_ENGINE_BY_TYPE = {
+    "ppt": "markitdown",
+    "pptx": "markitdown",
+    "csv": "markitdown",
+}
+
+
+class ParserEngineRegistry:
+    """Registry for parser engines.
+
+    Each engine maps file extensions to parser classes.
+    When a requested engine doesn't support a file type, the registry
+    tries a type-level default engine, then builtin.
+    """
+
+    def __init__(self):
+        self._engines: Dict[str, Dict[str, Type[BaseParser]]] = {}
+        self._descriptions: Dict[str, str] = {}
+        self._check_available: Dict[str, Callable[..., Tuple[bool, str]]] = {}
+        self._unavailable_hint: Dict[str, str] = {}
+
+    def register(
+        self,
+        name: str,
+        file_types: Dict[str, Type[BaseParser]],
+        description: str = "",
+        check_available: Callable[..., Tuple[bool, str]] | None = None,
+        unavailable_hint: str = "",
+    ):
+        self._engines[name] = file_types
+        self._descriptions[name] = description
+        if check_available is not None:
+            self._check_available[name] = check_available
+            self._unavailable_hint[name] = unavailable_hint
+        logger.info(
+            "Registered parser engine '%s' with file types: %s",
+            name,
+            ", ".join(file_types.keys()),
+        )
+
+    def get_parser_class(self, engine: str, file_type: str) -> Type[BaseParser]:
+        """Resolve parser class for the given engine and file type.
+
+        Fallback order when the requested engine is empty or does not
+        support the type: type-level default (e.g. ppt → markitdown),
+        then builtin. Raises an actionable error listing engines that
+        do support the type.
+        """
+        ft = file_type.lower().lstrip(".")
+
+        if engine and engine in self._engines:
+            cls = self._engines[engine].get(ft)
+            if cls:
+                logger.info("Using engine '%s' for file type '%s'", engine, ft)
+                return cls
+            logger.info(
+                "Engine '%s' does not support '%s', falling back",
+                engine,
+                ft,
+            )
+
+        default_engine = _DEFAULT_ENGINE_BY_TYPE.get(ft)
+        if default_engine:
+            cls = self._engines.get(default_engine, {}).get(ft)
+            if cls:
+                logger.info(
+                    "Using default engine '%s' for file type '%s'",
+                    default_engine,
+                    ft,
+                )
+                return cls
+
+        builtin = self._engines.get(BUILTIN_ENGINE, {})
+        cls = builtin.get(ft)
+        if cls:
+            return cls
+
+        supported = sorted(
+            name for name, parsers in self._engines.items() if ft in parsers
+        )
+        engine_label = engine or BUILTIN_ENGINE
+        if supported:
+            raise ValueError(
+                f"Unsupported file type {file_type!r} for engine {engine_label!r}. "
+                f"Configure one of: {', '.join(supported)}"
+            )
+        raise ValueError(f"Unsupported file type: {file_type}")
+
+    def list_engines(self, overrides: Optional[Dict[str, str]] = None) -> List[Dict]:
+        """Return metadata for all registered engines, including availability.
+
+        Args:
+            overrides: tenant-level config overrides (e.g. mineru_endpoint, mineru_api_key)
+                       forwarded to each engine's check_available function.
+        """
+        result = []
+        for name, parsers in self._engines.items():
+            available = True
+            unavailable_reason = ""
+            check = self._check_available.get(name)
+            if check is not None:
+                try:
+                    available, unavailable_reason = check(overrides)
+                except Exception as e:
+                    available = False
+                    unavailable_reason = str(e) or self._unavailable_hint.get(name, "")
+            if not available and not unavailable_reason:
+                unavailable_reason = self._unavailable_hint.get(name, "不可用")
+            result.append(
+                {
+                    "name": name,
+                    "description": self._descriptions.get(name, ""),
+                    "file_types": sorted(parsers.keys()),
+                    "available": available,
+                    "unavailable_reason": unavailable_reason,
+                }
+            )
+        return result
+
+    def get_engine_names(self) -> List[str]:
+        return list(self._engines.keys())
+
+
+def _build_default_registry() -> ParserEngineRegistry:
+    """Create and populate the default registry with all known engines."""
+    reg = ParserEngineRegistry()
+
+    _image_types = {
+        ext: ImageParser for ext in ("jpg", "jpeg", "png", "gif", "bmp", "tiff", "webp")
+    }
+
+    reg.register(
+        BUILTIN_ENGINE,
+        {
+            "docx": Docx2Parser,
+            "doc": DocParser,
+            "pdf": PDFParser,
+            "md": MarkdownParser,
+            "markdown": MarkdownParser,
+            "xlsx": ExcelParser,
+            "xls": ExcelParser,
+            "pptx": MarkitdownParser,
+            "ppt": MarkitdownParser,
+            "epub": EPUBParser,
+            "html": HTMLParser,
+            "htm": HTMLParser,
+            "mhtml": MHTMLParser,
+            "xmind": XMindParser,
+            **_image_types,
+        },
+        description="内置解析引擎",
+    )
+
+    reg.register(
+        "markitdown",
+        {
+            "md": MarkitdownParser,
+            "markdown": MarkitdownParser,
+            "pdf": MarkitdownParser,
+            "docx": MarkitdownParser,
+            "doc": MarkitdownParser,
+            "pptx": MarkitdownParser,
+            "ppt": MarkitdownParser,
+            "xlsx": MarkitdownParser,
+            "xls": MarkitdownParser,
+            "csv": MarkitdownParser,
+        },
+        description="MarkItDown 解析引擎（微软 MarkItDown 库）",
+    )
+
+    reg.register(
+        "opendataloader",
+        {"pdf": OpenDataLoaderParser},
+        description="OpenDataLoader PDF（版面分析，需 Java 11+）",
+        check_available=lambda overrides: opendataloader_available(
+            overrides, quick=True
+        ),
+        unavailable_hint="请安装 opendataloader-pdf 与 Java 11+",
+    )
+
+    # NOTE: Engine listing is managed by Go-side engine registry
+    # (docparser.ListAllEngines). The Python list_engines method is kept for
+    # backward compatibility with the gRPC ListEngines RPC but the Go app
+    # no longer calls it. MinerU engines are handled natively by Go.
+
+    return reg
+
+
+registry = _build_default_registry()

docreader/parser/web_parser.py

@@ -0,0 +1,353 @@
+import asyncio
+import logging
+import re
+from dataclasses import dataclass
+from typing import Optional
+
+from lxml.etree import XPath
+from playwright.async_api import Page, async_playwright
+from trafilatura import extract, utils, xpaths
+
+from docreader.config import CONFIG
+from docreader.models.document import Document
+from docreader.parser.base_parser import BaseParser
+from docreader.parser.chain_parser import PipelineParser
+from docreader.parser.markdown_parser import MarkdownParser
+from docreader.utils import endecode
+from docreader.utils.ssrf import is_ssrf_safe_url
+
+logger = logging.getLogger(__name__)
+
+
+class WebParseError(RuntimeError):
+    """Raised when a URL cannot be scraped or has no extractable content.
+
+    This must propagate as a parse failure. Returning the error string as
+    document body would index it as knowledge.
+    """
+
+
+_GOTO_TIMEOUT_MS = 30_000
+_NETWORK_IDLE_TIMEOUT_MS = 10_000
+_SPA_WAIT_TIMEOUT_MS = 15_000
+# Minimum visible characters before treating an SPA shell as "rendered".
+_SPA_MIN_TEXT_LEN = 80
+# Minimum visible characters for Playwright text fallback when trafilatura fails.
+_MIN_FALLBACK_TEXT_LEN = 50
+
+# Monkey-patch trafilatura internals to better support WeChat Official Account
+# articles, whose images live on `mmbiz.qpic.cn` without a standard file
+# extension and whose main content sits inside `#js_content` /
+# `.rich_media_content`. Trafilatura's `utils.IMAGE_EXTENSION` and
+# `xpaths.BODY_XPATH` are internal APIs, so we guard the patch and skip
+# silently if they are renamed/removed in a future release.
+try:
+    _WECHAT_IMAGE_EXTENSION = re.compile(
+        r"[^\s]+\.(avif|bmp|gif|hei[cf]|jpe?g|png|webp)(\b|$)|"  # Standard extensions
+        r"mmbiz\.qpic\.cn/[^\s]*wx_fmt=(jpeg|jpg|png|gif|webp)"  # WeChat query format
+    )
+    utils.IMAGE_EXTENSION = _WECHAT_IMAGE_EXTENSION
+
+    _WECHAT_BODY_XPATH = XPath(
+        '(.//*[@id="js_content" or contains(@class, "rich_media_content")])[1]'
+    )
+    _wechat_xpath_str = str(_WECHAT_BODY_XPATH)
+    if not any(str(x) == _wechat_xpath_str for x in xpaths.BODY_XPATH):
+        xpaths.BODY_XPATH.insert(0, _WECHAT_BODY_XPATH)
+except (AttributeError, ImportError) as e:
+    logger.warning(
+        "Failed to patch trafilatura internals for WeChat support: %s", e
+    )
+
+
+@dataclass(frozen=True)
+class _ScrapeResult:
+    html: str
+    visible_text: str
+    page_title: str
+    error: str = ""
+
+
+def extract_markdown_from_html(html: str) -> Optional[str]:
+    """Run trafilatura on HTML; return markdown or None if nothing extracted."""
+    if not html or not html.strip():
+        return None
+    md_text = extract(
+        html,
+        output_format="markdown",
+        with_metadata=True,
+        include_images=True,
+        include_tables=True,
+        include_links=True,
+    )
+    if not md_text or not md_text.strip():
+        return None
+    return md_text
+
+
+def build_visible_text_fallback(visible_text: str, page_title: str = "") -> Optional[str]:
+    """Build markdown from Playwright-visible text when trafilatura finds no article body."""
+    text = (visible_text or "").strip()
+    if len(text) < _MIN_FALLBACK_TEXT_LEN:
+        return None
+    title = (page_title or "").strip()
+    if title and not text.startswith(title):
+        return f"# {title}\n\n{text}"
+    return text
+
+
+async def wait_for_rendered_content(page: Page) -> None:
+    """Wait for SPA/JS pages beyond the initial HTML shell."""
+    try:
+        await page.wait_for_load_state("networkidle", timeout=_NETWORK_IDLE_TIMEOUT_MS)
+        logger.info("Network idle after navigation")
+    except Exception:
+        logger.info("Network idle wait timed out, continuing")
+
+    try:
+        await page.wait_for_function(
+            """(minLen) => {
+                const root = document.querySelector('#app')
+                    || document.querySelector('main')
+                    || document.body;
+                return ((root?.innerText || '').trim().length >= minLen);
+            }""",
+            arg=_SPA_MIN_TEXT_LEN,
+            timeout=_SPA_WAIT_TIMEOUT_MS,
+        )
+        logger.info("SPA/root visible text reached minimum length")
+    except Exception:
+        logger.info("SPA text wait timed out, using current DOM")
+
+
+async def read_visible_text(page: Page) -> str:
+    """Prefer #app/main innerText, then fall back to body."""
+    return await page.evaluate(
+        """() => {
+            const root = document.querySelector('#app')
+                || document.querySelector('main')
+                || document.querySelector('[role="main"]')
+                || document.body;
+            return (root?.innerText || '').trim();
+        }"""
+    )
+
+
+async def install_ssrf_route_guard(page: Page) -> None:
+    """Block navigation/subresource requests to SSRF-restricted targets (incl. redirects)."""
+
+    async def handle_route(route) -> None:
+        safe, reason = is_ssrf_safe_url(route.request.url)
+        if not safe:
+            logger.warning(
+                "SSRF guard blocked request to %s: %s", route.request.url, reason
+            )
+            await route.abort("blockedbyclient")
+            return
+        await route.continue_()
+
+    await page.route("**/*", handle_route)
+
+
+class StdWebParser(BaseParser):
+    """Standard web page parser using Playwright and Trafilatura.
+
+    This parser scrapes web pages using Playwright's WebKit browser and extracts
+    clean content using Trafilatura library. It supports proxy configuration and
+    converts HTML content to markdown format.
+    """
+
+    def __init__(self, title: str, **kwargs):
+        """Initialize the web parser.
+
+        Args:
+            title: Title of the web page to be used as file name
+            **kwargs: Additional arguments passed to BaseParser
+        """
+        self.title = title
+        # Get proxy configuration from config if available
+        self.proxy = CONFIG.external_https_proxy
+        super().__init__(file_name=title, **kwargs)
+        logger.info(f"Initialized WebParser with title: {title}")
+
+    async def scrape(self, url: str) -> _ScrapeResult:
+        """Scrape web page content using Playwright.
+
+        Args:
+            url: The URL of the web page to scrape
+
+        Returns:
+            HTML, visible text, and document title. On hard failure the
+            content fields are empty and ``error`` explains why.
+        """
+        logger.info(f"Starting web page scraping for URL: {url}")
+        safe, reason = is_ssrf_safe_url(url)
+        if not safe:
+            logger.error("URL blocked by SSRF guard before navigation: %s", reason)
+            return _ScrapeResult(
+                html="", visible_text="", page_title="",
+                error=f"URL blocked by SSRF guard: {reason}",
+            )
+        try:
+            async with async_playwright() as p:
+                kwargs = {}
+                # Configure proxy if available
+                if self.proxy:
+                    kwargs["proxy"] = {"server": self.proxy}
+                logger.info("Launching WebKit browser")
+                browser = await p.webkit.launch(**kwargs)
+                page = await browser.new_page()
+                await install_ssrf_route_guard(page)
+
+                logger.info(f"Navigating to URL: {url}")
+                try:
+                    await page.goto(
+                        url,
+                        timeout=_GOTO_TIMEOUT_MS,
+                        wait_until="domcontentloaded",
+                    )
+                    logger.info("Initial page load complete")
+                except Exception as e:
+                    logger.error(f"Error navigating to URL: {str(e)}")
+                    await browser.close()
+                    return _ScrapeResult(
+                        html="", visible_text="", page_title="",
+                        error=f"navigation failed: {e}",
+                    )
+
+                await wait_for_rendered_content(page)
+
+                page_title = await page.title()
+                visible_text = await read_visible_text(page)
+                content = await page.content()
+                logger.info(
+                    "Retrieved %d bytes HTML, %d chars visible text, title=%r",
+                    len(content),
+                    len(visible_text),
+                    page_title[:80] if page_title else "",
+                )
+
+                await browser.close()
+                logger.info("Browser closed")
+
+            logger.info("Successfully retrieved HTML content")
+            return _ScrapeResult(
+                html=content,
+                visible_text=visible_text,
+                page_title=page_title or "",
+            )
+
+        except Exception as e:
+            logger.error(f"Failed to scrape web page: {str(e)}")
+            return _ScrapeResult(
+                html="", visible_text="", page_title="",
+                error=f"scrape failed: {e}",
+            )
+
+    def parse_into_text(self, content: bytes) -> Document:
+        """Parse web page content into a Document object.
+
+        Args:
+            content: URL encoded as bytes
+
+        Returns:
+            Document object containing the parsed markdown content
+
+        Raises:
+            WebParseError: If scraping fails or the page has no extractable
+                content. Callers must surface this as a parse failure rather
+                than indexing the error string as document body.
+        """
+        url = endecode.decode_bytes(content)
+
+        logger.info(f"Scraping web page: {url}")
+        scrape_result = asyncio.run(self.scrape(url))
+        if scrape_result.error or (
+            not scrape_result.html and not scrape_result.visible_text
+        ):
+            detail = scrape_result.error or "no HTML or visible text"
+            logger.error("Failed to scrape web page %s: %s", url, detail)
+            raise WebParseError(f"Failed to scrape web page: {url} ({detail})")
+
+        md_text = extract_markdown_from_html(scrape_result.html)
+        if not md_text:
+            md_text = build_visible_text_fallback(
+                scrape_result.visible_text,
+                scrape_result.page_title,
+            )
+            if md_text:
+                logger.info(
+                    "Trafilatura empty; using Playwright visible-text fallback (%d chars)",
+                    len(md_text),
+                )
+
+        if not md_text:
+            logger.error("Failed to parse web page: %s", url)
+            raise WebParseError(f"Failed to parse web page: {url}")
+
+        metadata = {}
+        title_match = re.search(r"^title:\s*(.+)", md_text, re.MULTILINE)
+        if title_match:
+            extracted_title = title_match.group(1).strip()
+            if extracted_title:
+                metadata["title"] = extracted_title
+                logger.info(
+                    f"Extracted article title from trafilatura: {extracted_title}"
+                )
+        elif scrape_result.page_title:
+            metadata["title"] = scrape_result.page_title.strip()
+            logger.info(
+                "Using page title from Playwright: %s", metadata["title"]
+            )
+        else:
+            logger.info(
+                "No title found in trafilatura output, first 200 chars: %r",
+                md_text[:200],
+            )
+        return Document(content=md_text, metadata=metadata)
+
+
+class WebParser(PipelineParser):
+    """Web parser using pipeline pattern.
+
+    This parser chains StdWebParser (for web scraping and HTML to markdown conversion)
+    with MarkdownParser (for markdown processing). The pipeline processes content
+    sequentially through both parsers.
+    """
+
+    # Parser classes to be executed in sequence
+    _parser_cls = (StdWebParser, MarkdownParser)
+
+
+if __name__ == "__main__":
+    import sys
+
+    logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s")
+
+    url = sys.argv[1] if len(sys.argv) > 1 else "https://cloud.tencent.com/document/product/457/6759"
+    print(f"\n{'='*60}")
+    print(f"URL: {url}")
+    print(f"{'='*60}\n")
+
+    parser = WebParser(title="")
+    doc = parser.parse_into_text(url.encode())
+
+    print(f"--- metadata ---")
+    for k, v in doc.metadata.items():
+        print(f"  {k}: {v}")
+
+    print(f"\n--- images ({len(doc.images)}) ---")
+    for path in list(doc.images.keys())[:10]:
+        print(f"  {path}  ({len(doc.images[path])} chars base64)")
+
+    print(f"\n--- content ({len(doc.content)} chars) ---")
+    print(doc.content[:300000])
+    if len(doc.content) > 300000:
+        print(f"\n... (truncated, total {len(doc.content)} chars)")
+
+    print(f"\n--- chunks ({len(doc.chunks)}) ---")
+    for i, chunk in enumerate(doc.chunks[:5]):
+        print(f"  [{i}] seq={chunk.seq} range=[{chunk.start}:{chunk.end}] len={len(chunk.content)}")
+        print(f"      {chunk.content[:120]}{'...' if len(chunk.content) > 120 else ''}")
+    if len(doc.chunks) > 5:
+        print(f"  ... ({len(doc.chunks) - 5} more chunks)")

docreader/parser/xlsx_merge.py

@@ -0,0 +1,56 @@
+"""Fill merged cell values before pandas reads an XLSX workbook."""
+
+from __future__ import annotations
+
+import logging
+import zipfile
+from io import BytesIO
+
+logger = logging.getLogger(__name__)
+
+
+def fill_merged_cells_xlsx(content: bytes) -> bytes:
+    """Unmerge ranges and copy the master cell value into every covered cell.
+
+    openpyxl only stores values on the top-left cell of a merge; pandas then
+    sees NaN in the rest. Filling makes row-wise RAG chunks retain context.
+    """
+    if not zipfile.is_zipfile(BytesIO(content)):
+        return content
+
+    from openpyxl import load_workbook
+
+    wb = load_workbook(BytesIO(content), data_only=True)
+    changed = False
+    for ws in wb.worksheets:
+        if not ws.merged_cells.ranges:
+            continue
+        for merge_range in list(ws.merged_cells.ranges):
+            master_value = ws.cell(merge_range.min_row, merge_range.min_col).value
+            ws.unmerge_cells(str(merge_range))
+            for row in range(merge_range.min_row, merge_range.max_row + 1):
+                for col in range(merge_range.min_col, merge_range.max_col + 1):
+                    ws.cell(row, col).value = master_value
+            changed = True
+
+    if not changed:
+        return content
+
+    # Charts are not part of the cell text extracted by DocReader. Remove
+    # dedicated chart sheets through the public workbook API, then clear
+    # embedded worksheet charts (openpyxl has no public removal API for them)
+    # before saving this temporary workbook.
+    active_sheet = wb.active
+    for chart_sheet in list(wb.chartsheets):
+        wb.remove(chart_sheet)
+    if active_sheet in wb.worksheets:
+        wb.active = active_sheet
+    elif wb.worksheets:
+        wb.active = wb.worksheets[0]
+    for ws in wb.worksheets:
+        ws._charts.clear()
+
+    out = BytesIO()
+    wb.save(out)
+    logger.info("Filled merged cells in XLSX before parse")
+    return out.getvalue()

docreader/parser/xlsx_repair.py

@@ -0,0 +1,126 @@
+"""Repair common XLSX packaging issues before openpyxl/pandas read."""
+
+from __future__ import annotations
+
+import io
+import re
+import zipfile
+from typing import Callable, Dict, Iterable, Set
+
+SST_PART = "xl/sharedStrings.xml"
+_SST_OVERRIDE_RE = re.compile(
+    r'<Override[^>]*PartName="[^"]*sharedStrings\.xml"[^>]*/>',
+    re.IGNORECASE,
+)
+_SST_REL_RE = re.compile(
+    r'<Relationship[^>]*Type="[^"]*sharedStrings"[^>]*/>',
+    re.IGNORECASE,
+)
+
+
+def repair_xlsx_bytes(content: bytes) -> bytes | None:
+    """Return repaired XLSX bytes, or None if no repair was applied.
+
+    Handles workbooks that reference ``xl/sharedStrings.xml`` in package
+    metadata but omit the part (common with some exporters). When worksheets
+    only use inline strings, manifest references are stripped so openpyxl can
+    read the file.
+    """
+    if not zipfile.is_zipfile(io.BytesIO(content)):
+        return None
+
+    with zipfile.ZipFile(io.BytesIO(content), "r") as zin:
+        names = _normalized_names(zin.namelist())
+        sst_path = _find_shared_strings_path(names)
+        if sst_path:
+            if sst_path == SST_PART:
+                return None
+            return _rewrite_zip(
+                zin, lambda files: _rename_shared_strings_part(files, sst_path)
+            )
+        if not _package_references_shared_strings(zin, names):
+            return None
+        if _worksheets_use_shared_string_cells(zin, names):
+            return None
+        return _rewrite_zip(zin, _strip_shared_strings_manifest)
+
+
+def _normalized_names(namelist: Iterable[str]) -> Set[str]:
+    return {name.replace("\\", "/") for name in namelist}
+
+
+def _find_shared_strings_path(names: Set[str]) -> str | None:
+    for name in names:
+        if name.lower().endswith("sharedstrings.xml"):
+            return name
+    return None
+
+
+def _package_references_shared_strings(
+    zin: zipfile.ZipFile, names: Set[str]
+) -> bool:
+    content_types = "[Content_Types].xml"
+    if content_types in names:
+        ct = zin.read(content_types).decode("utf-8", errors="replace")
+        if "sharedstrings.xml" in ct.lower():
+            return True
+
+    rels_path = "xl/_rels/workbook.xml.rels"
+    if rels_path in names:
+        rels = zin.read(rels_path).decode("utf-8", errors="replace")
+        if "sharedstrings" in rels.lower():
+            return True
+    return False
+
+
+def _worksheets_use_shared_string_cells(
+    zin: zipfile.ZipFile, names: Set[str]
+) -> bool:
+    for name in names:
+        if not name.startswith("xl/worksheets/") or not name.endswith(".xml"):
+            continue
+        sheet = zin.read(name).decode("utf-8", errors="replace")
+        if re.search(r'\bt="s"', sheet):
+            return True
+    return False
+
+
+def _rename_shared_strings_part(
+    files: Dict[str, bytes], source_path: str
+) -> Dict[str, bytes]:
+    updated = dict(files)
+    updated[SST_PART] = updated.pop(source_path)
+    return updated
+
+
+def _strip_shared_strings_manifest(files: Dict[str, bytes]) -> Dict[str, bytes]:
+    updated = dict(files)
+    ct_path = "[Content_Types].xml"
+    if ct_path in updated:
+        ct = updated[ct_path].decode("utf-8")
+        ct = _SST_OVERRIDE_RE.sub("", ct)
+        updated[ct_path] = ct.encode("utf-8")
+
+    rels_path = "xl/_rels/workbook.xml.rels"
+    if rels_path in updated:
+        rels = updated[rels_path].decode("utf-8")
+        rels = _SST_REL_RE.sub("", rels)
+        updated[rels_path] = rels.encode("utf-8")
+    return updated
+
+
+def _rewrite_zip(
+    zin: zipfile.ZipFile,
+    transform: Callable[[Dict[str, bytes]], Dict[str, bytes]],
+) -> bytes:
+    files: Dict[str, bytes] = {}
+    for info in zin.infolist():
+        name = info.filename.replace("\\", "/")
+        files[name] = zin.read(info.filename)
+    files = transform(files)
+
+    out = io.BytesIO()
+    with zipfile.ZipFile(out, "w", zipfile.ZIP_DEFLATED) as zout:
+        for name, data in files.items():
+            zout.writestr(name, data)
+    return out.getvalue()

docreader/parser/xmind_parser.py

@@ -0,0 +1,246 @@
+"""Parse XMind archives into Markdown outlines."""
+
+import io
+import json
+import zipfile
+from dataclasses import dataclass, field
+from xml.etree import ElementTree
+from xml.etree.ElementTree import Element
+
+from docreader.models.document import Document
+from docreader.parser.base_parser import BaseParser
+
+
+MAX_CONTENT_BYTES = 32 * 1024 * 1024
+
+
+@dataclass
+class _Topic:
+    title: str = ""
+    note: str = ""
+    children: list["_Topic"] = field(default_factory=list)
+
+
+@dataclass
+class _Sheet:
+    title: str = ""
+    root_topic: _Topic | None = None
+
+
+def _clean_text(value: object) -> str:
+    return value.strip() if isinstance(value, str) else ""
+
+
+def _topic_from_json(value: object) -> _Topic | None:
+    if not isinstance(value, dict):
+        return None
+
+    note = ""
+    notes = value.get("notes")
+    if isinstance(notes, dict):
+        plain = notes.get("plain")
+        if isinstance(plain, dict):
+            note = _clean_text(plain.get("content"))
+
+    topics: list[_Topic] = []
+    children = value.get("children")
+    if isinstance(children, dict):
+        attached = children.get("attached")
+        if isinstance(attached, list):
+            for child in attached:
+                topic = _topic_from_json(child)
+                if topic is not None:
+                    topics.append(topic)
+
+    return _Topic(
+        title=_clean_text(value.get("title")),
+        note=note,
+        children=topics,
+    )
+
+
+def _parse_json_sheets(payload: bytes) -> list[_Sheet]:
+    values = json.loads(payload)
+    if not isinstance(values, list):
+        raise ValueError("invalid XMind content.json: expected a sheet list")
+
+    sheets: list[_Sheet] = []
+    for value in values:
+        if not isinstance(value, dict):
+            continue
+        sheets.append(
+            _Sheet(
+                title=_clean_text(value.get("title")),
+                root_topic=_topic_from_json(value.get("rootTopic")),
+            )
+        )
+    return sheets
+
+
+def _local_name(tag: str) -> str:
+    return tag.rsplit("}", 1)[-1].split(":", 1)[-1]
+
+
+def _direct_children(element: Element, name: str) -> list[Element]:
+    return [child for child in element if _local_name(child.tag) == name]
+
+
+def _first_child(element: Element, name: str) -> Element | None:
+    children = _direct_children(element, name)
+    return children[0] if children else None
+
+
+def _element_text(element: Element | None) -> str:
+    if element is None:
+        return ""
+    return "".join(element.itertext()).strip()
+
+
+def _topic_from_xml(element: Element) -> _Topic:
+    notes = _first_child(element, "notes")
+    plain = _first_child(notes, "plain") if notes is not None else None
+
+    topics: list[_Topic] = []
+    for children in _direct_children(element, "children"):
+        for topic_group in _direct_children(children, "topics"):
+            topics.extend(
+                _topic_from_xml(topic)
+                for topic in _direct_children(topic_group, "topic")
+            )
+
+    return _Topic(
+        title=_element_text(_first_child(element, "title")),
+        note=_element_text(plain),
+        children=topics,
+    )
+
+
+def _parse_xml_sheets(payload: bytes) -> list[_Sheet]:
+    try:
+        root = ElementTree.fromstring(payload)
+    except ElementTree.ParseError as exc:
+        raise ValueError("invalid XMind content.xml") from exc
+
+    sheets: list[_Sheet] = []
+    for sheet in _direct_children(root, "sheet"):
+        root_topic = _first_child(sheet, "topic")
+        sheets.append(
+            _Sheet(
+                title=_element_text(_first_child(sheet, "title")),
+                root_topic=(
+                    _topic_from_xml(root_topic) if root_topic is not None else None
+                ),
+            )
+        )
+    return sheets
+
+
+def _read_content_entry(content: bytes) -> tuple[str, bytes]:
+    try:
+        with zipfile.ZipFile(io.BytesIO(content)) as archive:
+            try:
+                info = archive.getinfo("content.json")
+                content_format = "json"
+            except KeyError:
+                try:
+                    info = archive.getinfo("content.xml")
+                except KeyError as exc:
+                    raise ValueError(
+                        "XMind archive is missing content.json or content.xml"
+                    ) from exc
+                content_format = "xml"
+
+            if info.flag_bits & 0x1:
+                raise ValueError("encrypted XMind content is not supported")
+            if info.file_size > MAX_CONTENT_BYTES:
+                raise ValueError("XMind content entry exceeds the 32 MiB limit")
+
+            with archive.open(info) as entry:
+                payload = entry.read(MAX_CONTENT_BYTES + 1)
+            if len(payload) > MAX_CONTENT_BYTES:
+                raise ValueError("XMind content entry exceeds the 32 MiB limit")
+            return content_format, payload
+    except zipfile.BadZipFile as exc:
+        raise ValueError("invalid XMind archive") from exc
+
+
+def _render_topic(topic: _Topic, depth: int) -> tuple[list[str], int, int]:
+    lines: list[str] = []
+    topic_count = 0
+    note_count = 0
+    child_depth = depth
+
+    if topic.title:
+        lines.append(f"{'  ' * depth}- {topic.title}")
+        topic_count = 1
+        child_depth += 1
+        if topic.note:
+            for note_line in topic.note.splitlines():
+                normalized = note_line.strip()
+                quote = f"> {normalized}" if normalized else ">"
+                lines.append(f"{'  ' * child_depth}{quote}")
+            note_count = 1
+
+    for child in topic.children:
+        child_lines, child_topics, child_notes = _render_topic(child, child_depth)
+        lines.extend(child_lines)
+        topic_count += child_topics
+        note_count += child_notes
+
+    return lines, topic_count, note_count
+
+
+def _render_sheets(sheets: list[_Sheet]) -> tuple[str, int, int, int]:
+    rendered_sheets: list[str] = []
+    topic_count = 0
+    note_count = 0
+
+    for index, sheet in enumerate(sheets, start=1):
+        if sheet.root_topic is None:
+            continue
+        lines, sheet_topics, sheet_notes = _render_topic(sheet.root_topic, 0)
+        if not lines:
+            continue
+        title = sheet.title or f"Sheet {index}"
+        rendered_sheets.append(f"# {title}\n\n" + "\n".join(lines))
+        topic_count += sheet_topics
+        note_count += sheet_notes
+
+    return (
+        "\n\n---\n\n".join(rendered_sheets),
+        len(rendered_sheets),
+        topic_count,
+        note_count,
+    )
+
+
+class XMindParser(BaseParser):
+    """Extract topic hierarchy and plain-text notes from XMind files."""
+
+    def parse_into_text(self, content: bytes) -> Document:
+        content_format, payload = _read_content_entry(content)
+
+        if content_format == "json":
+            try:
+                sheets = _parse_json_sheets(payload)
+            except (json.JSONDecodeError, UnicodeDecodeError) as exc:
+                raise ValueError("invalid XMind content.json") from exc
+        else:
+            sheets = _parse_xml_sheets(payload)
+
+        markdown, sheet_count, topic_count, note_count = _render_sheets(
+            sheets
+        )
+        if not markdown:
+            raise ValueError("XMind archive contains no renderable topics")
+        return Document(
+            content=markdown,
+            metadata={
+                "source_format": "xmind",
+                "xmind_content_format": content_format,
+                "file_size": len(content),
+                "sheet_count": sheet_count,
+                "topic_count": topic_count,
+                "note_count": note_count,
+            },
+        )

docreader/proto/docreader.pb.go

@@ -0,0 +1,796 @@
+// Code generated by protoc-gen-go. DO NOT EDIT.
+// versions:
+// 	protoc-gen-go v1.36.7
+// 	protoc        v5.29.3
+// source: docreader.proto
+
+package proto
+
+import (
+	protoreflect "google.golang.org/protobuf/reflect/protoreflect"
+	protoimpl "google.golang.org/protobuf/runtime/protoimpl"
+	reflect "reflect"
+	sync "sync"
+	unsafe "unsafe"
+)
+
+const (
+	// Verify that this generated code is sufficiently up-to-date.
+	_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
+	// Verify that runtime/protoimpl is sufficiently up-to-date.
+	_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
+)
+
+type ReadConfig struct {
+	state                 protoimpl.MessageState `protogen:"open.v1"`
+	ParserEngine          string                 `protobuf:"bytes,1,opt,name=parser_engine,json=parserEngine,proto3" json:"parser_engine,omitempty"`
+	ParserEngineOverrides map[string]string      `protobuf:"bytes,2,rep,name=parser_engine_overrides,json=parserEngineOverrides,proto3" json:"parser_engine_overrides,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
+	unknownFields         protoimpl.UnknownFields
+	sizeCache             protoimpl.SizeCache
+}
+
+func (x *ReadConfig) Reset() {
+	*x = ReadConfig{}
+	mi := &file_docreader_proto_msgTypes[0]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *ReadConfig) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ReadConfig) ProtoMessage() {}
+
+func (x *ReadConfig) ProtoReflect() protoreflect.Message {
+	mi := &file_docreader_proto_msgTypes[0]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use ReadConfig.ProtoReflect.Descriptor instead.
+func (*ReadConfig) Descriptor() ([]byte, []int) {
+	return file_docreader_proto_rawDescGZIP(), []int{0}
+}
+
+func (x *ReadConfig) GetParserEngine() string {
+	if x != nil {
+		return x.ParserEngine
+	}
+	return ""
+}
+
+func (x *ReadConfig) GetParserEngineOverrides() map[string]string {
+	if x != nil {
+		return x.ParserEngineOverrides
+	}
+	return nil
+}
+
+// Unified read request: set file_content for file mode, url for URL mode.
+type ReadRequest struct {
+	state         protoimpl.MessageState `protogen:"open.v1"`
+	FileContent   []byte                 `protobuf:"bytes,1,opt,name=file_content,json=fileContent,proto3" json:"file_content,omitempty"`
+	FileName      string                 `protobuf:"bytes,2,opt,name=file_name,json=fileName,proto3" json:"file_name,omitempty"`
+	FileType      string                 `protobuf:"bytes,3,opt,name=file_type,json=fileType,proto3" json:"file_type,omitempty"`
+	Url           string                 `protobuf:"bytes,4,opt,name=url,proto3" json:"url,omitempty"`
+	Title         string                 `protobuf:"bytes,5,opt,name=title,proto3" json:"title,omitempty"`
+	Config        *ReadConfig            `protobuf:"bytes,6,opt,name=config,proto3" json:"config,omitempty"`
+	RequestId     string                 `protobuf:"bytes,7,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *ReadRequest) Reset() {
+	*x = ReadRequest{}
+	mi := &file_docreader_proto_msgTypes[1]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *ReadRequest) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ReadRequest) ProtoMessage() {}
+
+func (x *ReadRequest) ProtoReflect() protoreflect.Message {
+	mi := &file_docreader_proto_msgTypes[1]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use ReadRequest.ProtoReflect.Descriptor instead.
+func (*ReadRequest) Descriptor() ([]byte, []int) {
+	return file_docreader_proto_rawDescGZIP(), []int{1}
+}
+
+func (x *ReadRequest) GetFileContent() []byte {
+	if x != nil {
+		return x.FileContent
+	}
+	return nil
+}
+
+func (x *ReadRequest) GetFileName() string {
+	if x != nil {
+		return x.FileName
+	}
+	return ""
+}
+
+func (x *ReadRequest) GetFileType() string {
+	if x != nil {
+		return x.FileType
+	}
+	return ""
+}
+
+func (x *ReadRequest) GetUrl() string {
+	if x != nil {
+		return x.Url
+	}
+	return ""
+}
+
+func (x *ReadRequest) GetTitle() string {
+	if x != nil {
+		return x.Title
+	}
+	return ""
+}
+
+func (x *ReadRequest) GetConfig() *ReadConfig {
+	if x != nil {
+		return x.Config
+	}
+	return nil
+}
+
+func (x *ReadRequest) GetRequestId() string {
+	if x != nil {
+		return x.RequestId
+	}
+	return ""
+}
+
+type ImageRef struct {
+	state         protoimpl.MessageState `protogen:"open.v1"`
+	Filename      string                 `protobuf:"bytes,1,opt,name=filename,proto3" json:"filename,omitempty"`
+	OriginalRef   string                 `protobuf:"bytes,2,opt,name=original_ref,json=originalRef,proto3" json:"original_ref,omitempty"`
+	MimeType      string                 `protobuf:"bytes,3,opt,name=mime_type,json=mimeType,proto3" json:"mime_type,omitempty"`
+	StorageKey    string                 `protobuf:"bytes,4,opt,name=storage_key,json=storageKey,proto3" json:"storage_key,omitempty"` // download URL from shared storage
+	ImageData     []byte                 `protobuf:"bytes,5,opt,name=image_data,json=imageData,proto3" json:"image_data,omitempty"`    // inline bytes fallback
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *ImageRef) Reset() {
+	*x = ImageRef{}
+	mi := &file_docreader_proto_msgTypes[2]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *ImageRef) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ImageRef) ProtoMessage() {}
+
+func (x *ImageRef) ProtoReflect() protoreflect.Message {
+	mi := &file_docreader_proto_msgTypes[2]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use ImageRef.ProtoReflect.Descriptor instead.
+func (*ImageRef) Descriptor() ([]byte, []int) {
+	return file_docreader_proto_rawDescGZIP(), []int{2}
+}
+
+func (x *ImageRef) GetFilename() string {
+	if x != nil {
+		return x.Filename
+	}
+	return ""
+}
+
+func (x *ImageRef) GetOriginalRef() string {
+	if x != nil {
+		return x.OriginalRef
+	}
+	return ""
+}
+
+func (x *ImageRef) GetMimeType() string {
+	if x != nil {
+		return x.MimeType
+	}
+	return ""
+}
+
+func (x *ImageRef) GetStorageKey() string {
+	if x != nil {
+		return x.StorageKey
+	}
+	return ""
+}
+
+func (x *ImageRef) GetImageData() []byte {
+	if x != nil {
+		return x.ImageData
+	}
+	return nil
+}
+
+type ReadResponse struct {
+	state           protoimpl.MessageState `protogen:"open.v1"`
+	MarkdownContent string                 `protobuf:"bytes,1,opt,name=markdown_content,json=markdownContent,proto3" json:"markdown_content,omitempty"`
+	ImageRefs       []*ImageRef            `protobuf:"bytes,2,rep,name=image_refs,json=imageRefs,proto3" json:"image_refs,omitempty"`
+	ImageDirPath    string                 `protobuf:"bytes,3,opt,name=image_dir_path,json=imageDirPath,proto3" json:"image_dir_path,omitempty"`
+	Metadata        map[string]string      `protobuf:"bytes,4,rep,name=metadata,proto3" json:"metadata,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
+	Error           string                 `protobuf:"bytes,5,opt,name=error,proto3" json:"error,omitempty"`
+	unknownFields   protoimpl.UnknownFields
+	sizeCache       protoimpl.SizeCache
+}
+
+func (x *ReadResponse) Reset() {
+	*x = ReadResponse{}
+	mi := &file_docreader_proto_msgTypes[3]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *ReadResponse) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ReadResponse) ProtoMessage() {}
+
+func (x *ReadResponse) ProtoReflect() protoreflect.Message {
+	mi := &file_docreader_proto_msgTypes[3]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use ReadResponse.ProtoReflect.Descriptor instead.
+func (*ReadResponse) Descriptor() ([]byte, []int) {
+	return file_docreader_proto_rawDescGZIP(), []int{3}
+}
+
+func (x *ReadResponse) GetMarkdownContent() string {
+	if x != nil {
+		return x.MarkdownContent
+	}
+	return ""
+}
+
+func (x *ReadResponse) GetImageRefs() []*ImageRef {
+	if x != nil {
+		return x.ImageRefs
+	}
+	return nil
+}
+
+func (x *ReadResponse) GetImageDirPath() string {
+	if x != nil {
+		return x.ImageDirPath
+	}
+	return ""
+}
+
+func (x *ReadResponse) GetMetadata() map[string]string {
+	if x != nil {
+		return x.Metadata
+	}
+	return nil
+}
+
+func (x *ReadResponse) GetError() string {
+	if x != nil {
+		return x.Error
+	}
+	return ""
+}
+
+// Metadata header for a streamed read. Sent exactly once, before any image.
+type ReadStreamMeta struct {
+	state           protoimpl.MessageState `protogen:"open.v1"`
+	MarkdownContent string                 `protobuf:"bytes,1,opt,name=markdown_content,json=markdownContent,proto3" json:"markdown_content,omitempty"`
+	ImageDirPath    string                 `protobuf:"bytes,2,opt,name=image_dir_path,json=imageDirPath,proto3" json:"image_dir_path,omitempty"`
+	Metadata        map[string]string      `protobuf:"bytes,3,rep,name=metadata,proto3" json:"metadata,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
+	Error           string                 `protobuf:"bytes,4,opt,name=error,proto3" json:"error,omitempty"`
+	ImageCount      uint32                 `protobuf:"varint,5,opt,name=image_count,json=imageCount,proto3" json:"image_count,omitempty"` // best-effort total image count (0 if unknown)
+	unknownFields   protoimpl.UnknownFields
+	sizeCache       protoimpl.SizeCache
+}
+
+func (x *ReadStreamMeta) Reset() {
+	*x = ReadStreamMeta{}
+	mi := &file_docreader_proto_msgTypes[4]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *ReadStreamMeta) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ReadStreamMeta) ProtoMessage() {}
+
+func (x *ReadStreamMeta) ProtoReflect() protoreflect.Message {
+	mi := &file_docreader_proto_msgTypes[4]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use ReadStreamMeta.ProtoReflect.Descriptor instead.
+func (*ReadStreamMeta) Descriptor() ([]byte, []int) {
+	return file_docreader_proto_rawDescGZIP(), []int{4}
+}
+
+func (x *ReadStreamMeta) GetMarkdownContent() string {
+	if x != nil {
+		return x.MarkdownContent
+	}
+	return ""
+}
+
+func (x *ReadStreamMeta) GetImageDirPath() string {
+	if x != nil {
+		return x.ImageDirPath
+	}
+	return ""
+}
+
+func (x *ReadStreamMeta) GetMetadata() map[string]string {
+	if x != nil {
+		return x.Metadata
+	}
+	return nil
+}
+
+func (x *ReadStreamMeta) GetError() string {
+	if x != nil {
+		return x.Error
+	}
+	return ""
+}
+
+func (x *ReadStreamMeta) GetImageCount() uint32 {
+	if x != nil {
+		return x.ImageCount
+	}
+	return 0
+}
+
+// One frame of a ReadStream. The first frame carries `meta`; every subsequent
+// frame carries a single `image`.
+type ReadStreamResponse struct {
+	state protoimpl.MessageState `protogen:"open.v1"`
+	// Types that are valid to be assigned to Payload:
+	//
+	//	*ReadStreamResponse_Meta
+	//	*ReadStreamResponse_Image
+	Payload       isReadStreamResponse_Payload `protobuf_oneof:"payload"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *ReadStreamResponse) Reset() {
+	*x = ReadStreamResponse{}
+	mi := &file_docreader_proto_msgTypes[5]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *ReadStreamResponse) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ReadStreamResponse) ProtoMessage() {}
+
+func (x *ReadStreamResponse) ProtoReflect() protoreflect.Message {
+	mi := &file_docreader_proto_msgTypes[5]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use ReadStreamResponse.ProtoReflect.Descriptor instead.
+func (*ReadStreamResponse) Descriptor() ([]byte, []int) {
+	return file_docreader_proto_rawDescGZIP(), []int{5}
+}
+
+func (x *ReadStreamResponse) GetPayload() isReadStreamResponse_Payload {
+	if x != nil {
+		return x.Payload
+	}
+	return nil
+}
+
+func (x *ReadStreamResponse) GetMeta() *ReadStreamMeta {
+	if x != nil {
+		if x, ok := x.Payload.(*ReadStreamResponse_Meta); ok {
+			return x.Meta
+		}
+	}
+	return nil
+}
+
+func (x *ReadStreamResponse) GetImage() *ImageRef {
+	if x != nil {
+		if x, ok := x.Payload.(*ReadStreamResponse_Image); ok {
+			return x.Image
+		}
+	}
+	return nil
+}
+
+type isReadStreamResponse_Payload interface {
+	isReadStreamResponse_Payload()
+}
+
+type ReadStreamResponse_Meta struct {
+	Meta *ReadStreamMeta `protobuf:"bytes,1,opt,name=meta,proto3,oneof"`
+}
+
+type ReadStreamResponse_Image struct {
+	Image *ImageRef `protobuf:"bytes,2,opt,name=image,proto3,oneof"`
+}
+
+func (*ReadStreamResponse_Meta) isReadStreamResponse_Payload() {}
+
+func (*ReadStreamResponse_Image) isReadStreamResponse_Payload() {}
+
+type ListEnginesRequest struct {
+	state           protoimpl.MessageState `protogen:"open.v1"`
+	ConfigOverrides map[string]string      `protobuf:"bytes,1,rep,name=config_overrides,json=configOverrides,proto3" json:"config_overrides,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
+	unknownFields   protoimpl.UnknownFields
+	sizeCache       protoimpl.SizeCache
+}
+
+func (x *ListEnginesRequest) Reset() {
+	*x = ListEnginesRequest{}
+	mi := &file_docreader_proto_msgTypes[6]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *ListEnginesRequest) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ListEnginesRequest) ProtoMessage() {}
+
+func (x *ListEnginesRequest) ProtoReflect() protoreflect.Message {
+	mi := &file_docreader_proto_msgTypes[6]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use ListEnginesRequest.ProtoReflect.Descriptor instead.
+func (*ListEnginesRequest) Descriptor() ([]byte, []int) {
+	return file_docreader_proto_rawDescGZIP(), []int{6}
+}
+
+func (x *ListEnginesRequest) GetConfigOverrides() map[string]string {
+	if x != nil {
+		return x.ConfigOverrides
+	}
+	return nil
+}
+
+type ParserEngineInfo struct {
+	state             protoimpl.MessageState `protogen:"open.v1"`
+	Name              string                 `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
+	Description       string                 `protobuf:"bytes,2,opt,name=description,proto3" json:"description,omitempty"`
+	FileTypes         []string               `protobuf:"bytes,3,rep,name=file_types,json=fileTypes,proto3" json:"file_types,omitempty"`
+	Available         bool                   `protobuf:"varint,4,opt,name=available,proto3" json:"available,omitempty"`
+	UnavailableReason string                 `protobuf:"bytes,5,opt,name=unavailable_reason,json=unavailableReason,proto3" json:"unavailable_reason,omitempty"`
+	unknownFields     protoimpl.UnknownFields
+	sizeCache         protoimpl.SizeCache
+}
+
+func (x *ParserEngineInfo) Reset() {
+	*x = ParserEngineInfo{}
+	mi := &file_docreader_proto_msgTypes[7]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *ParserEngineInfo) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ParserEngineInfo) ProtoMessage() {}
+
+func (x *ParserEngineInfo) ProtoReflect() protoreflect.Message {
+	mi := &file_docreader_proto_msgTypes[7]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use ParserEngineInfo.ProtoReflect.Descriptor instead.
+func (*ParserEngineInfo) Descriptor() ([]byte, []int) {
+	return file_docreader_proto_rawDescGZIP(), []int{7}
+}
+
+func (x *ParserEngineInfo) GetName() string {
+	if x != nil {
+		return x.Name
+	}
+	return ""
+}
+
+func (x *ParserEngineInfo) GetDescription() string {
+	if x != nil {
+		return x.Description
+	}
+	return ""
+}
+
+func (x *ParserEngineInfo) GetFileTypes() []string {
+	if x != nil {
+		return x.FileTypes
+	}
+	return nil
+}
+
+func (x *ParserEngineInfo) GetAvailable() bool {
+	if x != nil {
+		return x.Available
+	}
+	return false
+}
+
+func (x *ParserEngineInfo) GetUnavailableReason() string {
+	if x != nil {
+		return x.UnavailableReason
+	}
+	return ""
+}
+
+type ListEnginesResponse struct {
+	state         protoimpl.MessageState `protogen:"open.v1"`
+	Engines       []*ParserEngineInfo    `protobuf:"bytes,1,rep,name=engines,proto3" json:"engines,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *ListEnginesResponse) Reset() {
+	*x = ListEnginesResponse{}
+	mi := &file_docreader_proto_msgTypes[8]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *ListEnginesResponse) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*ListEnginesResponse) ProtoMessage() {}
+
+func (x *ListEnginesResponse) ProtoReflect() protoreflect.Message {
+	mi := &file_docreader_proto_msgTypes[8]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use ListEnginesResponse.ProtoReflect.Descriptor instead.
+func (*ListEnginesResponse) Descriptor() ([]byte, []int) {
+	return file_docreader_proto_rawDescGZIP(), []int{8}
+}
+
+func (x *ListEnginesResponse) GetEngines() []*ParserEngineInfo {
+	if x != nil {
+		return x.Engines
+	}
+	return nil
+}
+
+var File_docreader_proto protoreflect.FileDescriptor
+
+const file_docreader_proto_rawDesc = "" +
+	"\n" +
+	"\x0fdocreader.proto\x12\tdocreader\"\xeb\x01\n" +
+	"\n" +
+	"ReadConfig\x12#\n" +
+	"\rparser_engine\x18\x01 \x01(\tR\fparserEngine\x12h\n" +
+	"\x17parser_engine_overrides\x18\x02 \x03(\v20.docreader.ReadConfig.ParserEngineOverridesEntryR\x15parserEngineOverrides\x1aH\n" +
+	"\x1aParserEngineOverridesEntry\x12\x10\n" +
+	"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
+	"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01J\x04\b\x03\x10\x04\"\xe0\x01\n" +
+	"\vReadRequest\x12!\n" +
+	"\ffile_content\x18\x01 \x01(\fR\vfileContent\x12\x1b\n" +
+	"\tfile_name\x18\x02 \x01(\tR\bfileName\x12\x1b\n" +
+	"\tfile_type\x18\x03 \x01(\tR\bfileType\x12\x10\n" +
+	"\x03url\x18\x04 \x01(\tR\x03url\x12\x14\n" +
+	"\x05title\x18\x05 \x01(\tR\x05title\x12-\n" +
+	"\x06config\x18\x06 \x01(\v2\x15.docreader.ReadConfigR\x06config\x12\x1d\n" +
+	"\n" +
+	"request_id\x18\a \x01(\tR\trequestId\"\xa6\x01\n" +
+	"\bImageRef\x12\x1a\n" +
+	"\bfilename\x18\x01 \x01(\tR\bfilename\x12!\n" +
+	"\foriginal_ref\x18\x02 \x01(\tR\voriginalRef\x12\x1b\n" +
+	"\tmime_type\x18\x03 \x01(\tR\bmimeType\x12\x1f\n" +
+	"\vstorage_key\x18\x04 \x01(\tR\n" +
+	"storageKey\x12\x1d\n" +
+	"\n" +
+	"image_data\x18\x05 \x01(\fR\timageData\"\xa9\x02\n" +
+	"\fReadResponse\x12)\n" +
+	"\x10markdown_content\x18\x01 \x01(\tR\x0fmarkdownContent\x122\n" +
+	"\n" +
+	"image_refs\x18\x02 \x03(\v2\x13.docreader.ImageRefR\timageRefs\x12$\n" +
+	"\x0eimage_dir_path\x18\x03 \x01(\tR\fimageDirPath\x12A\n" +
+	"\bmetadata\x18\x04 \x03(\v2%.docreader.ReadResponse.MetadataEntryR\bmetadata\x12\x14\n" +
+	"\x05error\x18\x05 \x01(\tR\x05error\x1a;\n" +
+	"\rMetadataEntry\x12\x10\n" +
+	"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
+	"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x9a\x02\n" +
+	"\x0eReadStreamMeta\x12)\n" +
+	"\x10markdown_content\x18\x01 \x01(\tR\x0fmarkdownContent\x12$\n" +
+	"\x0eimage_dir_path\x18\x02 \x01(\tR\fimageDirPath\x12C\n" +
+	"\bmetadata\x18\x03 \x03(\v2'.docreader.ReadStreamMeta.MetadataEntryR\bmetadata\x12\x14\n" +
+	"\x05error\x18\x04 \x01(\tR\x05error\x12\x1f\n" +
+	"\vimage_count\x18\x05 \x01(\rR\n" +
+	"imageCount\x1a;\n" +
+	"\rMetadataEntry\x12\x10\n" +
+	"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
+	"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"}\n" +
+	"\x12ReadStreamResponse\x12/\n" +
+	"\x04meta\x18\x01 \x01(\v2\x19.docreader.ReadStreamMetaH\x00R\x04meta\x12+\n" +
+	"\x05image\x18\x02 \x01(\v2\x13.docreader.ImageRefH\x00R\x05imageB\t\n" +
+	"\apayload\"\xb7\x01\n" +
+	"\x12ListEnginesRequest\x12]\n" +
+	"\x10config_overrides\x18\x01 \x03(\v22.docreader.ListEnginesRequest.ConfigOverridesEntryR\x0fconfigOverrides\x1aB\n" +
+	"\x14ConfigOverridesEntry\x12\x10\n" +
+	"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
+	"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\xb4\x01\n" +
+	"\x10ParserEngineInfo\x12\x12\n" +
+	"\x04name\x18\x01 \x01(\tR\x04name\x12 \n" +
+	"\vdescription\x18\x02 \x01(\tR\vdescription\x12\x1d\n" +
+	"\n" +
+	"file_types\x18\x03 \x03(\tR\tfileTypes\x12\x1c\n" +
+	"\tavailable\x18\x04 \x01(\bR\tavailable\x12-\n" +
+	"\x12unavailable_reason\x18\x05 \x01(\tR\x11unavailableReason\"L\n" +
+	"\x13ListEnginesResponse\x125\n" +
+	"\aengines\x18\x01 \x03(\v2\x1b.docreader.ParserEngineInfoR\aengines2\xdf\x01\n" +
+	"\tDocReader\x129\n" +
+	"\x04Read\x12\x16.docreader.ReadRequest\x1a\x17.docreader.ReadResponse\"\x00\x12G\n" +
+	"\n" +
+	"ReadStream\x12\x16.docreader.ReadRequest\x1a\x1d.docreader.ReadStreamResponse\"\x000\x01\x12N\n" +
+	"\vListEngines\x12\x1d.docreader.ListEnginesRequest\x1a\x1e.docreader.ListEnginesResponse\"\x00B5Z3github.com/Tencent/WeKnora/internal/docreader/protob\x06proto3"
+
+var (
+	file_docreader_proto_rawDescOnce sync.Once
+	file_docreader_proto_rawDescData []byte
+)
+
+func file_docreader_proto_rawDescGZIP() []byte {
+	file_docreader_proto_rawDescOnce.Do(func() {
+		file_docreader_proto_rawDescData = protoimpl.X.CompressGZIP(unsafe.Slice(unsafe.StringData(file_docreader_proto_rawDesc), len(file_docreader_proto_rawDesc)))
+	})
+	return file_docreader_proto_rawDescData
+}
+
+var file_docreader_proto_msgTypes = make([]protoimpl.MessageInfo, 13)
+var file_docreader_proto_goTypes = []any{
+	(*ReadConfig)(nil),          // 0: docreader.ReadConfig
+	(*ReadRequest)(nil),         // 1: docreader.ReadRequest
+	(*ImageRef)(nil),            // 2: docreader.ImageRef
+	(*ReadResponse)(nil),        // 3: docreader.ReadResponse
+	(*ReadStreamMeta)(nil),      // 4: docreader.ReadStreamMeta
+	(*ReadStreamResponse)(nil),  // 5: docreader.ReadStreamResponse
+	(*ListEnginesRequest)(nil),  // 6: docreader.ListEnginesRequest
+	(*ParserEngineInfo)(nil),    // 7: docreader.ParserEngineInfo
+	(*ListEnginesResponse)(nil), // 8: docreader.ListEnginesResponse
+	nil,                         // 9: docreader.ReadConfig.ParserEngineOverridesEntry
+	nil,                         // 10: docreader.ReadResponse.MetadataEntry
+	nil,                         // 11: docreader.ReadStreamMeta.MetadataEntry
+	nil,                         // 12: docreader.ListEnginesRequest.ConfigOverridesEntry
+}
+var file_docreader_proto_depIdxs = []int32{
+	9,  // 0: docreader.ReadConfig.parser_engine_overrides:type_name -> docreader.ReadConfig.ParserEngineOverridesEntry
+	0,  // 1: docreader.ReadRequest.config:type_name -> docreader.ReadConfig
+	2,  // 2: docreader.ReadResponse.image_refs:type_name -> docreader.ImageRef
+	10, // 3: docreader.ReadResponse.metadata:type_name -> docreader.ReadResponse.MetadataEntry
+	11, // 4: docreader.ReadStreamMeta.metadata:type_name -> docreader.ReadStreamMeta.MetadataEntry
+	4,  // 5: docreader.ReadStreamResponse.meta:type_name -> docreader.ReadStreamMeta
+	2,  // 6: docreader.ReadStreamResponse.image:type_name -> docreader.ImageRef
+	12, // 7: docreader.ListEnginesRequest.config_overrides:type_name -> docreader.ListEnginesRequest.ConfigOverridesEntry
+	7,  // 8: docreader.ListEnginesResponse.engines:type_name -> docreader.ParserEngineInfo
+	1,  // 9: docreader.DocReader.Read:input_type -> docreader.ReadRequest
+	1,  // 10: docreader.DocReader.ReadStream:input_type -> docreader.ReadRequest
+	6,  // 11: docreader.DocReader.ListEngines:input_type -> docreader.ListEnginesRequest
+	3,  // 12: docreader.DocReader.Read:output_type -> docreader.ReadResponse
+	5,  // 13: docreader.DocReader.ReadStream:output_type -> docreader.ReadStreamResponse
+	8,  // 14: docreader.DocReader.ListEngines:output_type -> docreader.ListEnginesResponse
+	12, // [12:15] is the sub-list for method output_type
+	9,  // [9:12] is the sub-list for method input_type
+	9,  // [9:9] is the sub-list for extension type_name
+	9,  // [9:9] is the sub-list for extension extendee
+	0,  // [0:9] is the sub-list for field type_name
+}
+
+func init() { file_docreader_proto_init() }
+func file_docreader_proto_init() {
+	if File_docreader_proto != nil {
+		return
+	}
+	file_docreader_proto_msgTypes[5].OneofWrappers = []any{
+		(*ReadStreamResponse_Meta)(nil),
+		(*ReadStreamResponse_Image)(nil),
+	}
+	type x struct{}
+	out := protoimpl.TypeBuilder{
+		File: protoimpl.DescBuilder{
+			GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
+			RawDescriptor: unsafe.Slice(unsafe.StringData(file_docreader_proto_rawDesc), len(file_docreader_proto_rawDesc)),
+			NumEnums:      0,
+			NumMessages:   13,
+			NumExtensions: 0,
+			NumServices:   1,
+		},
+		GoTypes:           file_docreader_proto_goTypes,
+		DependencyIndexes: file_docreader_proto_depIdxs,
+		MessageInfos:      file_docreader_proto_msgTypes,
+	}.Build()
+	File_docreader_proto = out.File
+	file_docreader_proto_goTypes = nil
+	file_docreader_proto_depIdxs = nil
+}

docreader/proto/docreader.proto

@@ -0,0 +1,86 @@
+syntax = "proto3";
+
+package docreader;
+
+option go_package = "github.com/Tencent/WeKnora/internal/docreader/proto";
+
+service DocReader {
+  rpc Read(ReadRequest) returns (ReadResponse) {}
+  // ReadStream is the streaming counterpart of Read. It first emits one
+  // ReadStreamResponse carrying the parse metadata (markdown / metadata /
+  // error), then emits one message per image. This keeps every gRPC message
+  // small so large scanned PDFs (hundreds of page images, far exceeding the
+  // unary message-size cap) can be returned without RESOURCE_EXHAUSTED and
+  // with bounded memory on both ends.
+  rpc ReadStream(ReadRequest) returns (stream ReadStreamResponse) {}
+  rpc ListEngines(ListEnginesRequest) returns (ListEnginesResponse) {}
+}
+
+message ReadConfig {
+  string parser_engine = 1;
+  map<string, string> parser_engine_overrides = 2;
+  // image_storage removed: image persistence is now handled entirely by the Go App.
+  // Field number 3 is reserved for backward compatibility.
+  reserved 3;
+}
+
+// Unified read request: set file_content for file mode, url for URL mode.
+message ReadRequest {
+  bytes  file_content = 1;
+  string file_name = 2;
+  string file_type = 3;
+  string url = 4;
+  string title = 5;
+  ReadConfig config = 6;
+  string request_id = 7;
+}
+
+message ImageRef {
+  string filename = 1;
+  string original_ref = 2;
+  string mime_type = 3;
+  string storage_key = 4;    // download URL from shared storage
+  bytes  image_data = 5;     // inline bytes fallback
+}
+
+message ReadResponse {
+  string markdown_content = 1;
+  repeated ImageRef image_refs = 2;
+  string image_dir_path = 3;
+  map<string, string> metadata = 4;
+  string error = 5;
+}
+
+// Metadata header for a streamed read. Sent exactly once, before any image.
+message ReadStreamMeta {
+  string markdown_content = 1;
+  string image_dir_path = 2;
+  map<string, string> metadata = 3;
+  string error = 4;
+  uint32 image_count = 5; // best-effort total image count (0 if unknown)
+}
+
+// One frame of a ReadStream. The first frame carries `meta`; every subsequent
+// frame carries a single `image`.
+message ReadStreamResponse {
+  oneof payload {
+    ReadStreamMeta meta = 1;
+    ImageRef image = 2;
+  }
+}
+
+message ListEnginesRequest {
+  map<string, string> config_overrides = 1;
+}
+
+message ParserEngineInfo {
+  string name = 1;
+  string description = 2;
+  repeated string file_types = 3;
+  bool available = 4;
+  string unavailable_reason = 5;
+}
+
+message ListEnginesResponse {
+  repeated ParserEngineInfo engines = 1;
+}

docreader/proto/docreader_grpc.pb.go

@@ -0,0 +1,213 @@
+// Code generated by protoc-gen-go-grpc. DO NOT EDIT.
+// versions:
+// - protoc-gen-go-grpc v1.5.1
+// - protoc             v5.29.3
+// source: docreader.proto
+
+package proto
+
+import (
+	context "context"
+	grpc "google.golang.org/grpc"
+	codes "google.golang.org/grpc/codes"
+	status "google.golang.org/grpc/status"
+)
+
+// This is a compile-time assertion to ensure that this generated file
+// is compatible with the grpc package it is being compiled against.
+// Requires gRPC-Go v1.64.0 or later.
+const _ = grpc.SupportPackageIsVersion9
+
+const (
+	DocReader_Read_FullMethodName        = "/docreader.DocReader/Read"
+	DocReader_ReadStream_FullMethodName  = "/docreader.DocReader/ReadStream"
+	DocReader_ListEngines_FullMethodName = "/docreader.DocReader/ListEngines"
+)
+
+// DocReaderClient is the client API for DocReader service.
+//
+// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
+type DocReaderClient interface {
+	Read(ctx context.Context, in *ReadRequest, opts ...grpc.CallOption) (*ReadResponse, error)
+	// ReadStream is the streaming counterpart of Read. It first emits one
+	// ReadStreamResponse carrying the parse metadata (markdown / metadata /
+	// error), then emits one message per image. This keeps every gRPC message
+	// small so large scanned PDFs (hundreds of page images, far exceeding the
+	// unary message-size cap) can be returned without RESOURCE_EXHAUSTED and
+	// with bounded memory on both ends.
+	ReadStream(ctx context.Context, in *ReadRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[ReadStreamResponse], error)
+	ListEngines(ctx context.Context, in *ListEnginesRequest, opts ...grpc.CallOption) (*ListEnginesResponse, error)
+}
+
+type docReaderClient struct {
+	cc grpc.ClientConnInterface
+}
+
+func NewDocReaderClient(cc grpc.ClientConnInterface) DocReaderClient {
+	return &docReaderClient{cc}
+}
+
+func (c *docReaderClient) Read(ctx context.Context, in *ReadRequest, opts ...grpc.CallOption) (*ReadResponse, error) {
+	cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+	out := new(ReadResponse)
+	err := c.cc.Invoke(ctx, DocReader_Read_FullMethodName, in, out, cOpts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+func (c *docReaderClient) ReadStream(ctx context.Context, in *ReadRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[ReadStreamResponse], error) {
+	cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+	stream, err := c.cc.NewStream(ctx, &DocReader_ServiceDesc.Streams[0], DocReader_ReadStream_FullMethodName, cOpts...)
+	if err != nil {
+		return nil, err
+	}
+	x := &grpc.GenericClientStream[ReadRequest, ReadStreamResponse]{ClientStream: stream}
+	if err := x.ClientStream.SendMsg(in); err != nil {
+		return nil, err
+	}
+	if err := x.ClientStream.CloseSend(); err != nil {
+		return nil, err
+	}
+	return x, nil
+}
+
+// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
+type DocReader_ReadStreamClient = grpc.ServerStreamingClient[ReadStreamResponse]
+
+func (c *docReaderClient) ListEngines(ctx context.Context, in *ListEnginesRequest, opts ...grpc.CallOption) (*ListEnginesResponse, error) {
+	cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+	out := new(ListEnginesResponse)
+	err := c.cc.Invoke(ctx, DocReader_ListEngines_FullMethodName, in, out, cOpts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+// DocReaderServer is the server API for DocReader service.
+// All implementations must embed UnimplementedDocReaderServer
+// for forward compatibility.
+type DocReaderServer interface {
+	Read(context.Context, *ReadRequest) (*ReadResponse, error)
+	// ReadStream is the streaming counterpart of Read. It first emits one
+	// ReadStreamResponse carrying the parse metadata (markdown / metadata /
+	// error), then emits one message per image. This keeps every gRPC message
+	// small so large scanned PDFs (hundreds of page images, far exceeding the
+	// unary message-size cap) can be returned without RESOURCE_EXHAUSTED and
+	// with bounded memory on both ends.
+	ReadStream(*ReadRequest, grpc.ServerStreamingServer[ReadStreamResponse]) error
+	ListEngines(context.Context, *ListEnginesRequest) (*ListEnginesResponse, error)
+	mustEmbedUnimplementedDocReaderServer()
+}
+
+// UnimplementedDocReaderServer must be embedded to have
+// forward compatible implementations.
+//
+// NOTE: this should be embedded by value instead of pointer to avoid a nil
+// pointer dereference when methods are called.
+type UnimplementedDocReaderServer struct{}
+
+func (UnimplementedDocReaderServer) Read(context.Context, *ReadRequest) (*ReadResponse, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method Read not implemented")
+}
+func (UnimplementedDocReaderServer) ReadStream(*ReadRequest, grpc.ServerStreamingServer[ReadStreamResponse]) error {
+	return status.Errorf(codes.Unimplemented, "method ReadStream not implemented")
+}
+func (UnimplementedDocReaderServer) ListEngines(context.Context, *ListEnginesRequest) (*ListEnginesResponse, error) {
+	return nil, status.Errorf(codes.Unimplemented, "method ListEngines not implemented")
+}
+func (UnimplementedDocReaderServer) mustEmbedUnimplementedDocReaderServer() {}
+func (UnimplementedDocReaderServer) testEmbeddedByValue()                   {}
+
+// UnsafeDocReaderServer may be embedded to opt out of forward compatibility for this service.
+// Use of this interface is not recommended, as added methods to DocReaderServer will
+// result in compilation errors.
+type UnsafeDocReaderServer interface {
+	mustEmbedUnimplementedDocReaderServer()
+}
+
+func RegisterDocReaderServer(s grpc.ServiceRegistrar, srv DocReaderServer) {
+	// If the following call pancis, it indicates UnimplementedDocReaderServer was
+	// embedded by pointer and is nil.  This will cause panics if an
+	// unimplemented method is ever invoked, so we test this at initialization
+	// time to prevent it from happening at runtime later due to I/O.
+	if t, ok := srv.(interface{ testEmbeddedByValue() }); ok {
+		t.testEmbeddedByValue()
+	}
+	s.RegisterService(&DocReader_ServiceDesc, srv)
+}
+
+func _DocReader_Read_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(ReadRequest)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(DocReaderServer).Read(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: DocReader_Read_FullMethodName,
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(DocReaderServer).Read(ctx, req.(*ReadRequest))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+func _DocReader_ReadStream_Handler(srv interface{}, stream grpc.ServerStream) error {
+	m := new(ReadRequest)
+	if err := stream.RecvMsg(m); err != nil {
+		return err
+	}
+	return srv.(DocReaderServer).ReadStream(m, &grpc.GenericServerStream[ReadRequest, ReadStreamResponse]{ServerStream: stream})
+}
+
+// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
+type DocReader_ReadStreamServer = grpc.ServerStreamingServer[ReadStreamResponse]
+
+func _DocReader_ListEngines_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(ListEnginesRequest)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(DocReaderServer).ListEngines(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: DocReader_ListEngines_FullMethodName,
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(DocReaderServer).ListEngines(ctx, req.(*ListEnginesRequest))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
+// DocReader_ServiceDesc is the grpc.ServiceDesc for DocReader service.
+// It's only intended for direct use with grpc.RegisterService,
+// and not to be introspected or modified (even as a copy)
+var DocReader_ServiceDesc = grpc.ServiceDesc{
+	ServiceName: "docreader.DocReader",
+	HandlerType: (*DocReaderServer)(nil),
+	Methods: []grpc.MethodDesc{
+		{
+			MethodName: "Read",
+			Handler:    _DocReader_Read_Handler,
+		},
+		{
+			MethodName: "ListEngines",
+			Handler:    _DocReader_ListEngines_Handler,
+		},
+	},
+	Streams: []grpc.StreamDesc{
+		{
+			StreamName:    "ReadStream",
+			Handler:       _DocReader_ReadStream_Handler,
+			ServerStreams: true,
+		},
+	},
+	Metadata: "docreader.proto",
+}

docreader/proto/docreader_pb2.py

@@ -0,0 +1,71 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler.  DO NOT EDIT!
+# NO CHECKED-IN PROTOBUF GENCODE
+# source: docreader.proto
+# Protobuf Python Version: 6.31.1
+"""Generated protocol buffer code."""
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import descriptor_pool as _descriptor_pool
+from google.protobuf import runtime_version as _runtime_version
+from google.protobuf import symbol_database as _symbol_database
+from google.protobuf.internal import builder as _builder
+_runtime_version.ValidateProtobufRuntimeVersion(
+    _runtime_version.Domain.PUBLIC,
+    6,
+    31,
+    1,
+    '',
+    'docreader.proto'
+)
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+
+
+DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x0f\x64ocreader.proto\x12\tdocreader\"\xba\x01\n\nReadConfig\x12\x15\n\rparser_engine\x18\x01 \x01(\t\x12Q\n\x17parser_engine_overrides\x18\x02 \x03(\x0b\x32\x30.docreader.ReadConfig.ParserEngineOverridesEntry\x1a<\n\x1aParserEngineOverridesEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01J\x04\x08\x03\x10\x04\"\xa0\x01\n\x0bReadRequest\x12\x14\n\x0c\x66ile_content\x18\x01 \x01(\x0c\x12\x11\n\tfile_name\x18\x02 \x01(\t\x12\x11\n\tfile_type\x18\x03 \x01(\t\x12\x0b\n\x03url\x18\x04 \x01(\t\x12\r\n\x05title\x18\x05 \x01(\t\x12%\n\x06\x63onfig\x18\x06 \x01(\x0b\x32\x15.docreader.ReadConfig\x12\x12\n\nrequest_id\x18\x07 \x01(\t\"n\n\x08ImageRef\x12\x10\n\x08\x66ilename\x18\x01 \x01(\t\x12\x14\n\x0coriginal_ref\x18\x02 \x01(\t\x12\x11\n\tmime_type\x18\x03 \x01(\t\x12\x13\n\x0bstorage_key\x18\x04 \x01(\t\x12\x12\n\nimage_data\x18\x05 \x01(\x0c\"\xe2\x01\n\x0cReadResponse\x12\x18\n\x10markdown_content\x18\x01 \x01(\t\x12\'\n\nimage_refs\x18\x02 \x03(\x0b\x32\x13.docreader.ImageRef\x12\x16\n\x0eimage_dir_path\x18\x03 \x01(\t\x12\x37\n\x08metadata\x18\x04 \x03(\x0b\x32%.docreader.ReadResponse.MetadataEntry\x12\r\n\x05\x65rror\x18\x05 \x01(\t\x1a/\n\rMetadataEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"\xd2\x01\n\x0eReadStreamMeta\x12\x18\n\x10markdown_content\x18\x01 \x01(\t\x12\x16\n\x0eimage_dir_path\x18\x02 \x01(\t\x12\x39\n\x08metadata\x18\x03 \x03(\x0b\x32\'.docreader.ReadStreamMeta.MetadataEntry\x12\r\n\x05\x65rror\x18\x04 \x01(\t\x12\x13\n\x0bimage_count\x18\x05 \x01(\r\x1a/\n\rMetadataEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"p\n\x12ReadStreamResponse\x12)\n\x04meta\x18\x01 \x01(\x0b\x32\x19.docreader.ReadStreamMetaH\x00\x12$\n\x05image\x18\x02 \x01(\x0b\x32\x13.docreader.ImageRefH\x00\x42\t\n\x07payload\"\x9a\x01\n\x12ListEnginesRequest\x12L\n\x10\x63onfig_overrides\x18\x01 \x03(\x0b\x32\x32.docreader.ListEnginesRequest.ConfigOverridesEntry\x1a\x36\n\x14\x43onfigOverridesEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"x\n\x10ParserEngineInfo\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x13\n\x0b\x64\x65scription\x18\x02 \x01(\t\x12\x12\n\nfile_types\x18\x03 \x03(\t\x12\x11\n\tavailable\x18\x04 \x01(\x08\x12\x1a\n\x12unavailable_reason\x18\x05 \x01(\t\"C\n\x13ListEnginesResponse\x12,\n\x07\x65ngines\x18\x01 \x03(\x0b\x32\x1b.docreader.ParserEngineInfo2\xdf\x01\n\tDocReader\x12\x39\n\x04Read\x12\x16.docreader.ReadRequest\x1a\x17.docreader.ReadResponse\"\x00\x12G\n\nReadStream\x12\x16.docreader.ReadRequest\x1a\x1d.docreader.ReadStreamResponse\"\x00\x30\x01\x12N\n\x0bListEngines\x12\x1d.docreader.ListEnginesRequest\x1a\x1e.docreader.ListEnginesResponse\"\x00\x42\x35Z3github.com/Tencent/WeKnora/internal/docreader/protob\x06proto3')
+
+_globals = globals()
+_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
+_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'docreader_pb2', _globals)
+if not _descriptor._USE_C_DESCRIPTORS:
+  _globals['DESCRIPTOR']._loaded_options = None
+  _globals['DESCRIPTOR']._serialized_options = b'Z3github.com/Tencent/WeKnora/internal/docreader/proto'
+  _globals['_READCONFIG_PARSERENGINEOVERRIDESENTRY']._loaded_options = None
+  _globals['_READCONFIG_PARSERENGINEOVERRIDESENTRY']._serialized_options = b'8\001'
+  _globals['_READRESPONSE_METADATAENTRY']._loaded_options = None
+  _globals['_READRESPONSE_METADATAENTRY']._serialized_options = b'8\001'
+  _globals['_READSTREAMMETA_METADATAENTRY']._loaded_options = None
+  _globals['_READSTREAMMETA_METADATAENTRY']._serialized_options = b'8\001'
+  _globals['_LISTENGINESREQUEST_CONFIGOVERRIDESENTRY']._loaded_options = None
+  _globals['_LISTENGINESREQUEST_CONFIGOVERRIDESENTRY']._serialized_options = b'8\001'
+  _globals['_READCONFIG']._serialized_start=31
+  _globals['_READCONFIG']._serialized_end=217
+  _globals['_READCONFIG_PARSERENGINEOVERRIDESENTRY']._serialized_start=151
+  _globals['_READCONFIG_PARSERENGINEOVERRIDESENTRY']._serialized_end=211
+  _globals['_READREQUEST']._serialized_start=220
+  _globals['_READREQUEST']._serialized_end=380
+  _globals['_IMAGEREF']._serialized_start=382
+  _globals['_IMAGEREF']._serialized_end=492
+  _globals['_READRESPONSE']._serialized_start=495
+  _globals['_READRESPONSE']._serialized_end=721
+  _globals['_READRESPONSE_METADATAENTRY']._serialized_start=674
+  _globals['_READRESPONSE_METADATAENTRY']._serialized_end=721
+  _globals['_READSTREAMMETA']._serialized_start=724
+  _globals['_READSTREAMMETA']._serialized_end=934
+  _globals['_READSTREAMMETA_METADATAENTRY']._serialized_start=674
+  _globals['_READSTREAMMETA_METADATAENTRY']._serialized_end=721
+  _globals['_READSTREAMRESPONSE']._serialized_start=936
+  _globals['_READSTREAMRESPONSE']._serialized_end=1048
+  _globals['_LISTENGINESREQUEST']._serialized_start=1051
+  _globals['_LISTENGINESREQUEST']._serialized_end=1205
+  _globals['_LISTENGINESREQUEST_CONFIGOVERRIDESENTRY']._serialized_start=1151
+  _globals['_LISTENGINESREQUEST_CONFIGOVERRIDESENTRY']._serialized_end=1205
+  _globals['_PARSERENGINEINFO']._serialized_start=1207
+  _globals['_PARSERENGINEINFO']._serialized_end=1327
+  _globals['_LISTENGINESRESPONSE']._serialized_start=1329
+  _globals['_LISTENGINESRESPONSE']._serialized_end=1396
+  _globals['_DOCREADER']._serialized_start=1399
+  _globals['_DOCREADER']._serialized_end=1622
+# @@protoc_insertion_point(module_scope)

docreader/proto/docreader_pb2.pyi

@@ -0,0 +1,137 @@
+from google.protobuf.internal import containers as _containers
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from collections.abc import Iterable as _Iterable, Mapping as _Mapping
+from typing import ClassVar as _ClassVar, Optional as _Optional, Union as _Union
+
+DESCRIPTOR: _descriptor.FileDescriptor
+
+class ReadConfig(_message.Message):
+    __slots__ = ("parser_engine", "parser_engine_overrides")
+    class ParserEngineOverridesEntry(_message.Message):
+        __slots__ = ("key", "value")
+        KEY_FIELD_NUMBER: _ClassVar[int]
+        VALUE_FIELD_NUMBER: _ClassVar[int]
+        key: str
+        value: str
+        def __init__(self, key: _Optional[str] = ..., value: _Optional[str] = ...) -> None: ...
+    PARSER_ENGINE_FIELD_NUMBER: _ClassVar[int]
+    PARSER_ENGINE_OVERRIDES_FIELD_NUMBER: _ClassVar[int]
+    parser_engine: str
+    parser_engine_overrides: _containers.ScalarMap[str, str]
+    def __init__(self, parser_engine: _Optional[str] = ..., parser_engine_overrides: _Optional[_Mapping[str, str]] = ...) -> None: ...
+
+class ReadRequest(_message.Message):
+    __slots__ = ("file_content", "file_name", "file_type", "url", "title", "config", "request_id")
+    FILE_CONTENT_FIELD_NUMBER: _ClassVar[int]
+    FILE_NAME_FIELD_NUMBER: _ClassVar[int]
+    FILE_TYPE_FIELD_NUMBER: _ClassVar[int]
+    URL_FIELD_NUMBER: _ClassVar[int]
+    TITLE_FIELD_NUMBER: _ClassVar[int]
+    CONFIG_FIELD_NUMBER: _ClassVar[int]
+    REQUEST_ID_FIELD_NUMBER: _ClassVar[int]
+    file_content: bytes
+    file_name: str
+    file_type: str
+    url: str
+    title: str
+    config: ReadConfig
+    request_id: str
+    def __init__(self, file_content: _Optional[bytes] = ..., file_name: _Optional[str] = ..., file_type: _Optional[str] = ..., url: _Optional[str] = ..., title: _Optional[str] = ..., config: _Optional[_Union[ReadConfig, _Mapping]] = ..., request_id: _Optional[str] = ...) -> None: ...
+
+class ImageRef(_message.Message):
+    __slots__ = ("filename", "original_ref", "mime_type", "storage_key", "image_data")
+    FILENAME_FIELD_NUMBER: _ClassVar[int]
+    ORIGINAL_REF_FIELD_NUMBER: _ClassVar[int]
+    MIME_TYPE_FIELD_NUMBER: _ClassVar[int]
+    STORAGE_KEY_FIELD_NUMBER: _ClassVar[int]
+    IMAGE_DATA_FIELD_NUMBER: _ClassVar[int]
+    filename: str
+    original_ref: str
+    mime_type: str
+    storage_key: str
+    image_data: bytes
+    def __init__(self, filename: _Optional[str] = ..., original_ref: _Optional[str] = ..., mime_type: _Optional[str] = ..., storage_key: _Optional[str] = ..., image_data: _Optional[bytes] = ...) -> None: ...
+
+class ReadResponse(_message.Message):
+    __slots__ = ("markdown_content", "image_refs", "image_dir_path", "metadata", "error")
+    class MetadataEntry(_message.Message):
+        __slots__ = ("key", "value")
+        KEY_FIELD_NUMBER: _ClassVar[int]
+        VALUE_FIELD_NUMBER: _ClassVar[int]
+        key: str
+        value: str
+        def __init__(self, key: _Optional[str] = ..., value: _Optional[str] = ...) -> None: ...
+    MARKDOWN_CONTENT_FIELD_NUMBER: _ClassVar[int]
+    IMAGE_REFS_FIELD_NUMBER: _ClassVar[int]
+    IMAGE_DIR_PATH_FIELD_NUMBER: _ClassVar[int]
+    METADATA_FIELD_NUMBER: _ClassVar[int]
+    ERROR_FIELD_NUMBER: _ClassVar[int]
+    markdown_content: str
+    image_refs: _containers.RepeatedCompositeFieldContainer[ImageRef]
+    image_dir_path: str
+    metadata: _containers.ScalarMap[str, str]
+    error: str
+    def __init__(self, markdown_content: _Optional[str] = ..., image_refs: _Optional[_Iterable[_Union[ImageRef, _Mapping]]] = ..., image_dir_path: _Optional[str] = ..., metadata: _Optional[_Mapping[str, str]] = ..., error: _Optional[str] = ...) -> None: ...
+
+class ReadStreamMeta(_message.Message):
+    __slots__ = ("markdown_content", "image_dir_path", "metadata", "error", "image_count")
+    class MetadataEntry(_message.Message):
+        __slots__ = ("key", "value")
+        KEY_FIELD_NUMBER: _ClassVar[int]
+        VALUE_FIELD_NUMBER: _ClassVar[int]
+        key: str
+        value: str
+        def __init__(self, key: _Optional[str] = ..., value: _Optional[str] = ...) -> None: ...
+    MARKDOWN_CONTENT_FIELD_NUMBER: _ClassVar[int]
+    IMAGE_DIR_PATH_FIELD_NUMBER: _ClassVar[int]
+    METADATA_FIELD_NUMBER: _ClassVar[int]
+    ERROR_FIELD_NUMBER: _ClassVar[int]
+    IMAGE_COUNT_FIELD_NUMBER: _ClassVar[int]
+    markdown_content: str
+    image_dir_path: str
+    metadata: _containers.ScalarMap[str, str]
+    error: str
+    image_count: int
+    def __init__(self, markdown_content: _Optional[str] = ..., image_dir_path: _Optional[str] = ..., metadata: _Optional[_Mapping[str, str]] = ..., error: _Optional[str] = ..., image_count: _Optional[int] = ...) -> None: ...
+
+class ReadStreamResponse(_message.Message):
+    __slots__ = ("meta", "image")
+    META_FIELD_NUMBER: _ClassVar[int]
+    IMAGE_FIELD_NUMBER: _ClassVar[int]
+    meta: ReadStreamMeta
+    image: ImageRef
+    def __init__(self, meta: _Optional[_Union[ReadStreamMeta, _Mapping]] = ..., image: _Optional[_Union[ImageRef, _Mapping]] = ...) -> None: ...
+
+class ListEnginesRequest(_message.Message):
+    __slots__ = ("config_overrides",)
+    class ConfigOverridesEntry(_message.Message):
+        __slots__ = ("key", "value")
+        KEY_FIELD_NUMBER: _ClassVar[int]
+        VALUE_FIELD_NUMBER: _ClassVar[int]
+        key: str
+        value: str
+        def __init__(self, key: _Optional[str] = ..., value: _Optional[str] = ...) -> None: ...
+    CONFIG_OVERRIDES_FIELD_NUMBER: _ClassVar[int]
+    config_overrides: _containers.ScalarMap[str, str]
+    def __init__(self, config_overrides: _Optional[_Mapping[str, str]] = ...) -> None: ...
+
+class ParserEngineInfo(_message.Message):
+    __slots__ = ("name", "description", "file_types", "available", "unavailable_reason")
+    NAME_FIELD_NUMBER: _ClassVar[int]
+    DESCRIPTION_FIELD_NUMBER: _ClassVar[int]
+    FILE_TYPES_FIELD_NUMBER: _ClassVar[int]
+    AVAILABLE_FIELD_NUMBER: _ClassVar[int]
+    UNAVAILABLE_REASON_FIELD_NUMBER: _ClassVar[int]
+    name: str
+    description: str
+    file_types: _containers.RepeatedScalarFieldContainer[str]
+    available: bool
+    unavailable_reason: str
+    def __init__(self, name: _Optional[str] = ..., description: _Optional[str] = ..., file_types: _Optional[_Iterable[str]] = ..., available: bool = ..., unavailable_reason: _Optional[str] = ...) -> None: ...
+
+class ListEnginesResponse(_message.Message):
+    __slots__ = ("engines",)
+    ENGINES_FIELD_NUMBER: _ClassVar[int]
+    engines: _containers.RepeatedCompositeFieldContainer[ParserEngineInfo]
+    def __init__(self, engines: _Optional[_Iterable[_Union[ParserEngineInfo, _Mapping]]] = ...) -> None: ...

docreader/proto/docreader_pb2_grpc.py

@@ -0,0 +1,189 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+"""Client and server classes corresponding to protobuf-defined services."""
+import grpc
+import warnings
+
+from docreader.proto import docreader_pb2 as docreader__pb2
+
+GRPC_GENERATED_VERSION = '1.80.0'
+GRPC_VERSION = grpc.__version__
+_version_not_supported = False
+
+try:
+    from grpc._utilities import first_version_is_lower
+    _version_not_supported = first_version_is_lower(GRPC_VERSION, GRPC_GENERATED_VERSION)
+except ImportError:
+    _version_not_supported = True
+
+if _version_not_supported:
+    raise RuntimeError(
+        f'The grpc package installed is at version {GRPC_VERSION},'
+        + ' but the generated code in docreader_pb2_grpc.py depends on'
+        + f' grpcio>={GRPC_GENERATED_VERSION}.'
+        + f' Please upgrade your grpc module to grpcio>={GRPC_GENERATED_VERSION}'
+        + f' or downgrade your generated code using grpcio-tools<={GRPC_VERSION}.'
+    )
+
+
+class DocReaderStub(object):
+    """Missing associated documentation comment in .proto file."""
+
+    def __init__(self, channel):
+        """Constructor.
+
+        Args:
+            channel: A grpc.Channel.
+        """
+        self.Read = channel.unary_unary(
+                '/docreader.DocReader/Read',
+                request_serializer=docreader__pb2.ReadRequest.SerializeToString,
+                response_deserializer=docreader__pb2.ReadResponse.FromString,
+                _registered_method=True)
+        self.ReadStream = channel.unary_stream(
+                '/docreader.DocReader/ReadStream',
+                request_serializer=docreader__pb2.ReadRequest.SerializeToString,
+                response_deserializer=docreader__pb2.ReadStreamResponse.FromString,
+                _registered_method=True)
+        self.ListEngines = channel.unary_unary(
+                '/docreader.DocReader/ListEngines',
+                request_serializer=docreader__pb2.ListEnginesRequest.SerializeToString,
+                response_deserializer=docreader__pb2.ListEnginesResponse.FromString,
+                _registered_method=True)
+
+
+class DocReaderServicer(object):
+    """Missing associated documentation comment in .proto file."""
+
+    def Read(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+    def ReadStream(self, request, context):
+        """ReadStream is the streaming counterpart of Read. It first emits one
+        ReadStreamResponse carrying the parse metadata (markdown / metadata /
+        error), then emits one message per image. This keeps every gRPC message
+        small so large scanned PDFs (hundreds of page images, far exceeding the
+        unary message-size cap) can be returned without RESOURCE_EXHAUSTED and
+        with bounded memory on both ends.
+        """
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+    def ListEngines(self, request, context):
+        """Missing associated documentation comment in .proto file."""
+        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+        context.set_details('Method not implemented!')
+        raise NotImplementedError('Method not implemented!')
+
+
+def add_DocReaderServicer_to_server(servicer, server):
+    rpc_method_handlers = {
+            'Read': grpc.unary_unary_rpc_method_handler(
+                    servicer.Read,
+                    request_deserializer=docreader__pb2.ReadRequest.FromString,
+                    response_serializer=docreader__pb2.ReadResponse.SerializeToString,
+            ),
+            'ReadStream': grpc.unary_stream_rpc_method_handler(
+                    servicer.ReadStream,
+                    request_deserializer=docreader__pb2.ReadRequest.FromString,
+                    response_serializer=docreader__pb2.ReadStreamResponse.SerializeToString,
+            ),
+            'ListEngines': grpc.unary_unary_rpc_method_handler(
+                    servicer.ListEngines,
+                    request_deserializer=docreader__pb2.ListEnginesRequest.FromString,
+                    response_serializer=docreader__pb2.ListEnginesResponse.SerializeToString,
+            ),
+    }
+    generic_handler = grpc.method_handlers_generic_handler(
+            'docreader.DocReader', rpc_method_handlers)
+    server.add_generic_rpc_handlers((generic_handler,))
+    server.add_registered_method_handlers('docreader.DocReader', rpc_method_handlers)
+
+
+ # This class is part of an EXPERIMENTAL API.
+class DocReader(object):
+    """Missing associated documentation comment in .proto file."""
+
+    @staticmethod
+    def Read(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(
+            request,
+            target,
+            '/docreader.DocReader/Read',
+            docreader__pb2.ReadRequest.SerializeToString,
+            docreader__pb2.ReadResponse.FromString,
+            options,
+            channel_credentials,
+            insecure,
+            call_credentials,
+            compression,
+            wait_for_ready,
+            timeout,
+            metadata,
+            _registered_method=True)
+
+    @staticmethod
+    def ReadStream(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_stream(
+            request,
+            target,
+            '/docreader.DocReader/ReadStream',
+            docreader__pb2.ReadRequest.SerializeToString,
+            docreader__pb2.ReadStreamResponse.FromString,
+            options,
+            channel_credentials,
+            insecure,
+            call_credentials,
+            compression,
+            wait_for_ready,
+            timeout,
+            metadata,
+            _registered_method=True)
+
+    @staticmethod
+    def ListEngines(request,
+            target,
+            options=(),
+            channel_credentials=None,
+            call_credentials=None,
+            insecure=False,
+            compression=None,
+            wait_for_ready=None,
+            timeout=None,
+            metadata=None):
+        return grpc.experimental.unary_unary(
+            request,
+            target,
+            '/docreader.DocReader/ListEngines',
+            docreader__pb2.ListEnginesRequest.SerializeToString,
+            docreader__pb2.ListEnginesResponse.FromString,
+            options,
+            channel_credentials,
+            insecure,
+            call_credentials,
+            compression,
+            wait_for_ready,
+            timeout,
+            metadata,
+            _registered_method=True)

docreader/pyproject.toml

@@ -0,0 +1,30 @@
+[project]
+name = "docreader"
+version = "0.1.0"
+description = "Add your description here"
+readme = "README.md"
+requires-python = ">=3.10.18"
+dependencies = [
+    "beautifulsoup4>=4.14.2",
+    "ebooklib>=0.18",
+    "grpcio>=1.78.0",
+    "grpcio-health-checking>=1.78.0",
+    "grpcio-tools>=1.78.0",
+    "lxml>=6.1.0",
+    "markdownify>=0.13.1",
+    "markitdown[docx,pdf,xls,xlsx,pptx]>=0.1.3",
+    "opendataloader-pdf>=2.4.7",
+    "openpyxl>=3.1.0",
+    "pandas>=2.0.0",
+    "xlrd>=2.0.0",
+    "pillow>=12.2.0",
+    "playwright>=1.55.0",
+    "protobuf>=6.33.5",
+    "pydantic>=2.13.4",
+    "pypdf>=6.13.3",
+    "pypdfium2>=5.8.0",
+    "python-docx>=1.2.0",
+    "requests>=2.33.0",
+    "textract==1.5.0",
+    "trafilatura>=2.0.0",
+]

docreader/scripts/generate_proto.sh

@@ -0,0 +1,36 @@
+#!/bin/bash
+set -ex
+
+# 设置目录
+PROTO_DIR="docreader/proto"
+PYTHON_OUT="docreader/proto"
+GO_OUT="docreader/proto"
+
+# 生成Python代码
+python3 -m grpc_tools.protoc -I${PROTO_DIR} \
+    --python_out=${PYTHON_OUT} \
+    --pyi_out=${PYTHON_OUT} \
+    --grpc_python_out=${PYTHON_OUT} \
+    ${PROTO_DIR}/docreader.proto
+
+# 生成Go代码（仅在 protoc-gen-go 可用时执行）
+if command -v protoc-gen-go &> /dev/null; then
+    protoc -I${PROTO_DIR} --go_out=${GO_OUT} \
+        --go_opt=paths=source_relative \
+        --go-grpc_out=${GO_OUT} \
+        --go-grpc_opt=paths=source_relative \
+        ${PROTO_DIR}/docreader.proto
+else
+    echo "protoc-gen-go not found, skipping Go code generation"
+fi
+
+# 修复Python导入问题（MacOS兼容版本）
+if [ "$(uname)" == "Darwin" ]; then
+    # MacOS版本
+    sed -i '' 's/import docreader_pb2/from docreader.proto import docreader_pb2/g' ${PYTHON_OUT}/docreader_pb2_grpc.py
+else
+    # Linux版本
+    sed -i 's/import docreader_pb2/from docreader.proto import docreader_pb2/g' ${PYTHON_OUT}/docreader_pb2_grpc.py
+fi
+
+echo "Proto files generated successfully!"

docreader/scripts/parse_local.py

@@ -0,0 +1,119 @@
+"""本地解析调试脚本：直接调用 Parser 解析本地文件，不经过 gRPC 服务。
+
+用法（在仓库根目录 WeKnora 下执行）：
+    PYTHONPATH=. docreader/.venv/bin/python docreader/scripts/parse_local.py <文件路径> [--engine markitdown] [--out out.md]
+
+示例：
+    PYTHONPATH=. docreader/.venv/bin/python docreader/scripts/parse_local.py docreader/testdata/test.md
+    PYTHONPATH=. docreader/.venv/bin/python docreader/scripts/parse_local.py ~/Desktop/demo.pdf --out /tmp/demo.md
+"""
+
+import argparse
+import logging
+import os
+import sys
+import time
+
+from docreader.parser import Parser
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="解析本地文件并输出 markdown")
+    parser.add_argument("path", help="待解析的本地文件路径")
+    parser.add_argument(
+        "--engine",
+        default="",
+        help="解析引擎名称（builtin / markitdown），留空使用内置引擎",
+    )
+    parser.add_argument(
+        "--type",
+        default="",
+        help="文件类型（如 pdf/docx/md），留空则按扩展名推断",
+    )
+    parser.add_argument(
+        "--out",
+        default="",
+        help="将完整 markdown 写入该文件，并把图片导出到同目录的 images/ 下",
+    )
+    parser.add_argument(
+        "--scanned",
+        action="store_true",
+        help="跳过 markitdown 文本抽取，直接把 PDF 每页渲染成图片（扫描件用，避免 pdfminer 卡死）",
+    )
+    parser.add_argument(
+        "--log-level",
+        default="INFO",
+        help="日志级别（DEBUG/INFO/WARNING/ERROR）",
+    )
+    args = parser.parse_args()
+
+    logging.basicConfig(
+        level=getattr(logging, args.log_level.upper(), logging.INFO),
+        format="%(asctime)s %(levelname)s %(name)s: %(message)s",
+        stream=sys.stderr,
+    )
+
+    if not os.path.isfile(args.path):
+        print(f"文件不存在: {args.path}", file=sys.stderr)
+        return 1
+
+    file_name = os.path.basename(args.path)
+    file_type = args.type or os.path.splitext(file_name)[1].lstrip(".")
+    with open(args.path, "rb") as f:
+        content = f.read()
+
+    started = time.monotonic()
+    if args.scanned:
+        from docreader.parser.pdf_parser import PDFScannedParser
+
+        doc = PDFScannedParser(file_name=file_name, file_type=file_type).parse_into_text(
+            content
+        )
+    else:
+        doc = Parser().parse_file(
+            file_name=file_name,
+            file_type=file_type,
+            content=content,
+            parser_engine=args.engine or None,
+        )
+    elapsed = time.monotonic() - started
+
+    print("=" * 60, file=sys.stderr)
+    print(f"file       : {file_name}", file=sys.stderr)
+    print(f"type       : {file_type}", file=sys.stderr)
+    print(f"engine     : {args.engine or 'builtin'}", file=sys.stderr)
+    print(f"scanned    : {args.scanned}", file=sys.stderr)
+    print(f"content_len: {len(doc.content)}", file=sys.stderr)
+    print(f"images     : {len(doc.images)}", file=sys.stderr)
+    print(f"metadata   : {doc.metadata}", file=sys.stderr)
+    print(f"elapsed    : {elapsed:.2f}s", file=sys.stderr)
+    print("=" * 60, file=sys.stderr)
+
+    if args.out:
+        import base64
+
+        out_dir = os.path.dirname(os.path.abspath(args.out))
+        os.makedirs(out_dir, exist_ok=True)
+        with open(args.out, "w", encoding="utf-8") as f:
+            f.write(doc.content)
+        if doc.images:
+            img_root = os.path.join(out_dir, "images")
+            os.makedirs(img_root, exist_ok=True)
+            for ref_path, b64data in doc.images.items():
+                try:
+                    raw = base64.b64decode(b64data)
+                except Exception:
+                    raw = b64data if isinstance(b64data, bytes) else b64data.encode()
+                dest = os.path.join(out_dir, ref_path)
+                os.makedirs(os.path.dirname(dest), exist_ok=True)
+                with open(dest, "wb") as imgf:
+                    imgf.write(raw)
+        print(f"已写入: {args.out}（图片导出到 {out_dir}/images/）", file=sys.stderr)
+    else:
+        print(doc.content)
+
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

docreader/splitter/header_hook.py

@@ -0,0 +1,216 @@
+import re
+from typing import Callable, Dict, List, Match, Pattern, Union
+
+from pydantic import BaseModel, Field
+
+
+class HeaderTrackerHook(BaseModel):
+    """表头追踪Hook的配置类，支持多种场景的表头识别"""
+
+    start_pattern: Pattern[str] = Field(
+        description="表头开始匹配（正则表达式或字符串）"
+    )
+    end_pattern: Pattern[str] = Field(description="表头结束匹配（正则表达式或字符串）")
+    extract_header_fn: Callable[[Match[str]], str] = Field(
+        default=lambda m: m.group(0),
+        description="从开始匹配结果中提取表头内容的函数（默认取匹配到的整个内容）",
+    )
+    priority: int = Field(default=0, description="优先级（多个配置时，高优先级先匹配）")
+    case_sensitive: bool = Field(
+        default=True, description="是否大小写敏感（仅当传入字符串pattern时生效）"
+    )
+
+    def __init__(
+        self,
+        start_pattern: Union[str, Pattern[str]],
+        end_pattern: Union[str, Pattern[str]],
+        **kwargs,
+    ):
+        flags = 0 if kwargs.get("case_sensitive", True) else re.IGNORECASE
+        if isinstance(start_pattern, str):
+            start_pattern = re.compile(start_pattern, flags | re.DOTALL)
+        if isinstance(end_pattern, str):
+            end_pattern = re.compile(end_pattern, flags | re.DOTALL)
+        super().__init__(
+            start_pattern=start_pattern,
+            end_pattern=end_pattern,
+            **kwargs,
+        )
+
+
+# 初始化表头Hook配置（提供默认配置：支持Markdown表格、代码块）
+DEFAULT_CONFIGS = [
+    # 代码块配置（```开头，```结尾）
+    # HeaderTrackerHook(
+    #     # 代码块开始（支持语言指定）
+    #     start_pattern=r"^\s*```(\w+).*(?!```$)",
+    #     # 代码块结束
+    #     end_pattern=r"^\s*```.*$",
+    #     extract_header_fn=lambda m: f"```{m.group(1)}" if m.group(1) else "```",
+    #     priority=20,  # 代码块优先级高于表格
+    #     case_sensitive=True,
+    # ),
+    # Markdown表格配置（表头带下划线）
+    HeaderTrackerHook(
+        # 表头行 + 分隔行
+        start_pattern=r"^\s*(?:\|[^|\n]*)+[\r\n]+\s*(?:\|\s*:?-{3,}:?\s*)+\|?[\r\n]+$",
+        # 空行或非表格内容
+        end_pattern=r"^\s*$|^\s*[^|\s].*$",
+        priority=15,
+        case_sensitive=False,
+    ),
+]
+DEFAULT_CONFIGS.sort(key=lambda x: -x.priority)
+
+_TABLE_ROW_PATTERN = re.compile(r"^\s*(?:\|[^|\n]*)+\|\s*$", re.MULTILINE)
+_MARKDOWN_TABLE_PRIORITY = 15
+
+
+def _is_empty_table_header_row(header: str) -> bool:
+    """True when the column-name line is only pipes/whitespace (MarkItDown quirk)."""
+    newline = header.find("\n")
+    if newline < 0:
+        return False
+    row = header[:newline].strip()
+    return bool(row) and all(ch in "| \t" for ch in row)
+
+
+def _extract_separator_line(header: str) -> str:
+    for line in header.split("\n"):
+        if "---" in line:
+            return line + "\n"
+    return ""
+
+
+def _table_row_column_count(line: str) -> int:
+    line = line.strip()
+    if not line.startswith("|"):
+        return 0
+    parts = line.split("|")
+    if parts and parts[0].strip() == "":
+        parts = parts[1:]
+    if parts and parts[-1].strip() == "":
+        parts = parts[:-1]
+    return len(parts)
+
+
+def _first_table_row_column_count(text: str) -> int:
+    for line in text.split("\n"):
+        line = line.strip()
+        if line and _TABLE_ROW_PATTERN.match(line):
+            return _table_row_column_count(line)
+    return 0
+
+
+def _header_table_column_count(header: str) -> int:
+    for line in header.split("\n"):
+        line = line.strip()
+        if not line or "---" in line:
+            continue
+        count = _table_row_column_count(line)
+        if count > 0:
+            return count
+    return 0
+
+
+def _split_ends_with_paragraph_break(split: str) -> bool:
+    trimmed = split.rstrip(" \t\r")
+    return trimmed.endswith("\n\n") or trimmed.endswith("\r\n\r\n")
+
+
+def header_column_mismatch(headers: str, next_unit: str) -> bool:
+    header_cols = _header_table_column_count(headers)
+    row_cols = _first_table_row_column_count(next_unit)
+    return header_cols > 0 and row_cols > 0 and header_cols != row_cols
+
+
+# 定义Hook状态数据结构
+class HeaderTracker(BaseModel):
+    """表头追踪 Hook 的状态类"""
+
+    header_hook_configs: List[HeaderTrackerHook] = Field(default=DEFAULT_CONFIGS)
+    active_headers: Dict[int, str] = Field(default_factory=dict)
+    ended_headers: set[int] = Field(default_factory=set)
+    pending_extend: Dict[int, bool] = Field(default_factory=dict)
+    pending_table_break: bool = Field(default=False)
+    header_ended_this_unit: bool = Field(default=False)
+
+    def _clear_table_header(self) -> None:
+        self.ended_headers.add(_MARKDOWN_TABLE_PRIORITY)
+        self.active_headers.pop(_MARKDOWN_TABLE_PRIORITY, None)
+        self.pending_extend.pop(_MARKDOWN_TABLE_PRIORITY, None)
+
+    def update(self, split: str) -> Dict[int, str]:
+        """检测当前split中的表头开始/结束，更新Hook状态"""
+        new_headers: Dict[int, str] = {}
+        self.header_ended_this_unit = False
+
+        if self.pending_table_break:
+            self.pending_table_break = False
+            if _MARKDOWN_TABLE_PRIORITY in self.active_headers:
+                if _first_table_row_column_count(split) > 0:
+                    self._clear_table_header()
+                    self.header_ended_this_unit = True
+                else:
+                    self._clear_table_header()
+
+        # 1. 检查是否有表头结束标记
+        for config in self.header_hook_configs:
+            if config.priority in self.active_headers and config.end_pattern.search(
+                split
+            ):
+                self.ended_headers.add(config.priority)
+                del self.active_headers[config.priority]
+                self.pending_extend.pop(config.priority, None)
+
+        # 1b. \n\n 分块会吞掉表间空行：段尾 \n\n 或列数变化时结束表头追踪
+        if (
+            _MARKDOWN_TABLE_PRIORITY in self.active_headers
+            and not self.pending_extend.get(_MARKDOWN_TABLE_PRIORITY)
+        ):
+            if _split_ends_with_paragraph_break(split):
+                self.pending_table_break = True
+            else:
+                header = self.active_headers[_MARKDOWN_TABLE_PRIORITY]
+                row_cols = _first_table_row_column_count(split)
+                header_cols = _header_table_column_count(header)
+                if row_cols > 0 and header_cols > 0 and row_cols != header_cols:
+                    self._clear_table_header()
+                    self.header_ended_this_unit = True
+
+        # 2. 空表头行：用首个数据行补全列名（与 Go header_tracker 一致）
+        for priority in list(self.pending_extend.keys()):
+            if priority in self.active_headers and _TABLE_ROW_PATTERN.search(split):
+                sep = _extract_separator_line(self.active_headers[priority])
+                self.active_headers[priority] = split + sep
+            self.pending_extend.pop(priority, None)
+
+        # 3. 检查是否有新的表头开始标记（只处理未活跃且未结束的）
+        for config in self.header_hook_configs:
+            if (
+                config.priority not in self.active_headers
+                and config.priority not in self.ended_headers
+            ):
+                match = config.start_pattern.search(split)
+                if match:
+                    header = config.extract_header_fn(match)
+                    self.active_headers[config.priority] = header
+                    new_headers[config.priority] = header
+                    if _is_empty_table_header_row(header):
+                        self.pending_extend[config.priority] = True
+
+        # 4. 检查是否所有活跃表头都已结束（清空结束标记）
+        if not self.active_headers:
+            self.ended_headers.clear()
+
+        return new_headers
+
+    def get_headers(self) -> str:
+        """获取当前所有活跃表头的拼接文本（按优先级排序）"""
+        # 按优先级降序排列表头
+        sorted_headers = sorted(self.active_headers.items(), key=lambda x: -x[0])
+        return (
+            "\n".join([header for _, header in sorted_headers])
+            if sorted_headers
+            else ""
+        )

docreader/splitter/splitter.py

@@ -0,0 +1,599 @@
+"""Token splitter.
+
+This module provides text splitting functionality with support for:
+- Configurable chunk size and overlap
+- Protected regex patterns (e.g., math formulas, images, links, tables)
+- Header tracking for context preservation
+- Smart merging with overlap handling
+"""
+
+import itertools
+import logging
+import re
+from typing import Callable, Generic, List, Pattern, Tuple, TypeVar
+
+from pydantic import BaseModel, Field, PrivateAttr
+
+from docreader.splitter.header_hook import (
+    HeaderTracker,
+    header_column_mismatch,
+)
+from docreader.utils.split import split_by_char, split_by_sep
+
+# Default configuration for text chunking
+# Aligned with internal/infrastructure/chunker/splitter.go (DefaultChunkOverlap = 80,
+# DefaultChunkSize = 512). The Go splitter is now the production path; this
+# Python splitter is kept for the docreader sidecar where it's still used.
+DEFAULT_CHUNK_OVERLAP = 80  # Number of characters to overlap between chunks (~15% of chunk size)
+DEFAULT_CHUNK_SIZE = 512  # Maximum size of each chunk in characters
+
+T = TypeVar("T")
+
+logger = logging.getLogger(__name__)
+
+
+class TextSplitter(BaseModel, Generic[T]):
+    """Text splitter with support for protected patterns and header tracking.
+
+    This class splits text into chunks while:
+    - Respecting chunk size and overlap constraints
+    - Preserving protected patterns (formulas, tables, code blocks)
+    - Tracking headers for context preservation
+    - Maintaining text integrity with smart merging
+    """
+
+    chunk_size: int = Field(description="The token chunk size for each chunk.")
+    chunk_overlap: int = Field(
+        description="The token overlap of each chunk when splitting."
+    )
+    separators: List[str] = Field(
+        description="Default separators for splitting into words"
+    )
+
+    # Try to keep the matched characters as a whole.
+    # If it's too long, the content will be further segmented.
+    # 尝试将匹配的字符作为整体保留，如果太长则进一步分段
+    protected_regex: List[str] = Field(
+        description="Protected regex for splitting into words"
+    )
+    len_function: Callable[[str], int] = Field(description="The length function.")
+    # Header tracking Hook related attributes
+    # 标题跟踪钩子相关属性
+    header_hook: HeaderTracker = Field(default_factory=HeaderTracker, exclude=True)
+
+    # Compiled regex patterns for protected content
+    _protected_fns: List[Pattern] = PrivateAttr()
+    # Split functions for different separators
+    _split_fns: List[Callable] = PrivateAttr()
+
+    def __init__(
+        self,
+        chunk_size: int = DEFAULT_CHUNK_SIZE,
+        chunk_overlap: int = DEFAULT_CHUNK_OVERLAP,
+        separators: List[str] = ["\n", "。", " "],
+        protected_regex: List[str] = [
+            # math formula - LaTeX style formulas enclosed in $$
+            r"\$\$[\s\S]*?\$\$",
+            # image - Markdown image syntax ![alt](url)
+            r"!\[.*?\]\(.*?\)",
+            # link - Markdown link syntax [text](url)
+            r"\[.*?\]\(.*?\)",
+            # table header - Markdown table header with separator line
+            r"[ ]*(?:\|[^|\n]*)+\|[\r\n]+\s*(?:\|\s*:?-{3,}:?\s*)+\|[\r\n]+",
+            # table body - Markdown table rows
+            r"[ ]*(?:\|[^|\n]*)+\|[\r\n]+",
+            # code header - Code block start with language identifier
+            r"```(?:\w+)[\r\n]+[^\r\n]*",
+        ],
+        length_function: Callable[[str], int] = lambda x: len(x),
+    ):
+        """Initialize with parameters.
+
+        Args:
+            chunk_size: Maximum size of each chunk
+            chunk_overlap: Number of tokens to overlap between chunks
+            separators: List of separators to use for splitting (in priority order)
+            protected_regex: Regex patterns for content that should be kept intact
+            length_function: Function to calculate text length (default: character count)
+
+        Raises:
+            ValueError: If chunk_overlap is larger than chunk_size
+        """
+        if chunk_overlap > chunk_size:
+            raise ValueError(
+                f"Got a larger chunk overlap ({chunk_overlap}) than chunk size "
+                f"({chunk_size}), should be smaller."
+            )
+
+        super().__init__(
+            chunk_size=chunk_size,
+            chunk_overlap=chunk_overlap,
+            separators=separators,
+            protected_regex=protected_regex,
+            len_function=length_function,
+        )
+        # Compile all protected regex patterns for efficient matching
+        self._protected_fns = [re.compile(reg) for reg in protected_regex]
+        # Create split functions: one for each separator, plus character-level splitting as fallback
+        self._split_fns = [split_by_sep(sep) for sep in separators] + [split_by_char()]
+
+    def split_text(self, text: str) -> List[Tuple[int, int, str]]:
+        """Split text into chunks with overlap and protected pattern handling.
+
+        Args:
+            text: The input text to split
+
+        Returns:
+            List of tuples (start_pos, end_pos, chunk_text) representing each chunk
+        """
+        if text == "":
+            return []
+
+        # Step 1: Split text by separators recursively
+        splits = self._split(text)
+        # Step 2: Extract protected content positions
+        protect = self._split_protected(text)
+        # Step 3: Merge splits with protected content to ensure integrity
+        splits = self._join(splits, protect)
+
+        # Verify that joining all splits reconstructs the original text
+        assert "".join(splits) == text
+
+        # Step 4: Merge splits into final chunks with overlap
+        chunks = self._merge(splits)
+
+        # Step 5: Validate chunks and test restoration
+        # self._validate_chunks(chunks, text)
+
+        return chunks
+
+    def _split(self, text: str) -> List[str]:
+        """Break text into splits that are smaller than chunk size.
+
+        This method recursively splits text using separators in priority order.
+        It tries each separator until it finds one that can split the text,
+        then recursively processes any splits that are still too large.
+
+        NOTE: the splits contain the separators.
+
+        Args:
+            text: The text to split
+
+        Returns:
+            List of text splits, each smaller than chunk_size
+        """
+        # If text is already small enough, return as-is
+        if self.len_function(text) <= self.chunk_size:
+            return [text]
+
+        # Try each split function in order until one successfully splits the text
+        splits = []
+        for split_fn in self._split_fns:
+            splits = split_fn(text)
+            if len(splits) > 1:
+                break
+
+        # Process each split: keep if small enough, otherwise recursively split further
+        new_splits = []
+        for split in splits:
+            split_len = self.len_function(split)
+            if split_len <= self.chunk_size:
+                new_splits.append(split)
+            else:
+                # Recursively split oversized chunks
+                new_splits.extend(self._split(split))
+        return new_splits
+
+    def _merge(self, splits: List[str]) -> List[Tuple[int, int, str]]:
+        """Merge splits into chunks with overlap and header tracking.
+
+        The high-level idea is to keep adding splits to a chunk until we
+        exceed the chunk size, then we start a new chunk with overlap.
+
+        When we start a new chunk, we pop off the first element of the previous
+        chunk until the total length is less than the chunk size.
+
+        Headers are tracked and prepended to chunks for context preservation.
+
+        Args:
+            splits: List of text splits to merge
+
+        Returns:
+            List of tuples (start_pos, end_pos, chunk_text) representing merged chunks
+        """
+        # Final list of chunks with their positions
+        chunks: List[Tuple[int, int, str]] = []
+
+        # Current chunk being built: list of (start, end, text) tuples
+        cur_chunk: List[Tuple[int, int, str]] = []
+
+        # Track current headers and chunk length
+        cur_headers, cur_len = "", 0
+        # Track position in original text
+        cur_start, cur_end = 0, 0
+
+        for split in splits:
+            # Calculate position of current split in original text
+            cur_end = cur_start + len(split)
+            split_len = self.len_function(split)
+
+            # Warn if a single split exceeds chunk size (shouldn't happen after _split)
+            if split_len > self.chunk_size:
+                logger.error(
+                    f"Got a split of size {split_len}, ",
+                    f"larger than chunk size {self.chunk_size}.",
+                )
+
+            # Update header tracking with current split
+            self.header_hook.update(split)
+            if self.header_hook.header_ended_this_unit and len(cur_chunk) > 0:
+                chunks.append(
+                    (
+                        cur_chunk[0][0],
+                        cur_chunk[-1][1],
+                        "".join([c[2] for c in cur_chunk]),
+                    )
+                )
+                cur_chunk = []
+                cur_len = 0
+            cur_headers = self.header_hook.get_headers()
+            cur_headers_len = self.len_function(cur_headers)
+
+            # If headers are too large, skip them to avoid oversized chunks
+            if cur_headers_len > self.chunk_size:
+                logger.error(
+                    f"Got headers of size {cur_headers_len}, ",
+                    f"larger than chunk size {self.chunk_size}.",
+                )
+                cur_headers, cur_headers_len = "", 0
+
+            # Check if adding this split would exceed chunk size
+            # If so, finalize current chunk and start a new one with overlap
+            if cur_len + split_len + cur_headers_len > self.chunk_size:
+                # Finalize the previous chunk if it has content
+                if len(cur_chunk) > 0:
+                    chunks.append(
+                        (
+                            cur_chunk[0][0],  # Start position of first element
+                            cur_chunk[-1][1],  # End position of last element
+                            "".join([c[2] for c in cur_chunk]),  # Concatenated text
+                        )
+                    )
+
+                # Start a new chunk with overlap from previous chunk
+                # Keep popping off the first element of the previous chunk until:
+                #   1. the current chunk length is less than chunk overlap
+                #   2. the total length is less than chunk size
+                while cur_chunk and (
+                    cur_len > self.chunk_overlap
+                    or cur_len + split_len + cur_headers_len > self.chunk_size
+                ):
+                    # Remove the first element to reduce overlap.
+                    # If the first element is a prepended header (start==end), also remove it.
+                    first_chunk = cur_chunk.pop(0)
+                    cur_len -= self.len_function(first_chunk[2])
+
+                    # If we just popped a real content piece, there may be a header right after it
+                    # (depending on previous iterations). Pop it only if it is actually a header.
+                    if cur_chunk and first_chunk[0] == first_chunk[1]:
+                        first_chunk = cur_chunk.pop(0)
+                        cur_len -= self.len_function(first_chunk[2])
+
+                # Prepend headers to new chunk if:
+                # 1. Headers exist
+                # 2. Headers + split fit in chunk size
+                # 3. Headers are not already in the split
+                if (
+                    cur_headers
+                    and split_len + cur_headers_len < self.chunk_size
+                    and cur_headers not in split
+                    and not header_column_mismatch(cur_headers, split)
+                ):
+                    next_start = cur_chunk[0][0] if cur_chunk else cur_start
+
+                    cur_chunk.insert(0, (next_start, next_start, cur_headers))
+                    cur_len += cur_headers_len
+
+            # Add current split to the chunk
+            cur_chunk.append((cur_start, cur_end, split))
+            cur_len += split_len
+            cur_start = cur_end
+
+        # Handle the last chunk (there should always be at least one)
+        assert cur_chunk
+        chunks.append(
+            (
+                cur_chunk[0][0],
+                cur_chunk[-1][1],
+                "".join([c[2] for c in cur_chunk]),
+            )
+        )
+
+        return chunks
+
+    def _split_protected(self, text: str) -> List[Tuple[int, str]]:
+        """Extract protected content from text based on regex patterns.
+
+        Args:
+            text: The input text to scan for protected patterns
+
+        Returns:
+            List of tuples (start_position, protected_text) for each protected match
+        """
+        # Find all matches for all protected patterns
+        matches = [
+            (match.start(), match.end())
+            for pattern in self._protected_fns
+            for match in pattern.finditer(text)
+        ]
+        # Sort by start position (ascending), then by length (descending) to handle overlaps
+        matches.sort(key=lambda x: (x[0], -x[1]))
+
+        res = []
+
+        def fold(initial: int, current: Tuple[int, int]) -> int:
+            """Accumulator function to filter overlapping matches."""
+            # Only process if match starts after previous match ended
+            if current[0] >= initial:
+                # Only keep protected content if it fits within chunk size
+                if current[1] - current[0] < self.chunk_size:
+                    res.append((current[0], text[current[0] : current[1]]))
+                else:
+                    logger.warning(f"Protected text ignore: {current}")
+            # Return the end position of the furthest match so far
+            return max(initial, current[1])
+
+        # Filter overlapping matches using accumulate
+        list(itertools.accumulate(matches, fold, initial=-1))
+        return res
+
+    def _join(self, splits: List[str], protect: List[Tuple[int, str]]) -> List[str]:
+        """Merge splits with protected content to ensure protected patterns remain intact.
+
+        Merges and splits elements in splits array based on protected substrings.
+
+        The function processes the input splits to ensure all protected substrings
+        remain as single items. If a protected substring is concatenated with preceding
+        or following content in any split element, it will be separated from
+        the adjacent content. The final result maintains the original order of content
+        while enforcing the integrity of protected substrings.
+
+        Key behaviors:
+        1. Preserves the complete structure of each protected substring
+        2. Separates protected substrings from any adjacent non-protected content
+        3. Maintains the original sequence of all content
+        4. Handles cases where protected substrings are partially concatenated
+
+        Args:
+            splits: List of text splits from _split()
+            protect: List of (position, text) tuples for protected content
+
+        Returns:
+            List of text splits with protected content properly isolated
+        """
+        j = 0  # Index for protected content list
+        point, start = 0, 0  # Track current position in original text
+        res = []  # Result list of merged splits
+
+        for split in splits:
+            # Calculate end position of current split
+            end = start + len(split)
+
+            # Get the portion of split starting from current point
+            cur = split[point - start :]
+
+            # Process all protected content that overlaps with current split
+            while j < len(protect):
+                p_start, p_content = protect[j]
+                p_end = p_start + len(p_content)
+
+                # If protected content is beyond current split, move to next split
+                if end <= p_start:
+                    break
+
+                # Add content before protected section
+                if point < p_start:
+                    local_end = p_start - point
+                    res.append(cur[:local_end])
+                    cur = cur[local_end:]
+                    point = p_start
+
+                # Add the protected content as a single unit
+                res.append(p_content)
+                j += 1
+
+                # Skip content that's part of the protected section
+                if point < p_end:
+                    local_start = p_end - point
+                    cur = cur[local_start:]
+                    point = p_end
+
+                # If no more content in current split, break
+                if not cur:
+                    break
+
+            # Add any remaining content from current split
+            if cur:
+                res.append(cur)
+                point = end
+
+            # Move to next split
+            start = end
+        return res
+
+    def _validate_chunks(
+        self, chunks: List[Tuple[int, int, str]], original_text: str
+    ) -> None:
+        """Validate chunks order and test text restoration.
+
+        This method performs two validations:
+        1. Checks if chunk start positions are in ascending order
+        2. Tests if the original text can be restored from chunks
+
+        If validation fails, saves debug information to /tmp/chunk_error_<timestamp>.md
+
+        Args:
+            chunks: List of tuples (start_pos, end_pos, chunk_text) to validate
+            original_text: The original text that was split
+        """
+        import datetime
+
+        errors = []
+
+        # Validation 1: Check if start positions are in ascending order
+        for i in range(1, len(chunks)):
+            prev_start = chunks[i - 1][0]
+            curr_start = chunks[i][0]
+            if curr_start < prev_start:
+                error_msg = (
+                    f"Chunk order error: chunk[{i}] start position ({curr_start}) "
+                    f"is less than chunk[{i - 1}] start position ({prev_start})"
+                )
+                errors.append(error_msg)
+                logger.error(error_msg)
+
+        # Validation 2: Test text restoration
+        try:
+            restored_text = self.restore_text(chunks)
+            if restored_text != original_text:
+                error_msg = (
+                    f"Restoration failed: restored text differs from original. "
+                    f"Original length: {len(original_text)}, "
+                    f"Restored length: {len(restored_text)}"
+                )
+                errors.append(error_msg)
+                logger.error(error_msg)
+
+                # Find first difference position
+                min_len = min(len(original_text), len(restored_text))
+                diff_pos = -1
+                for i in range(min_len):
+                    if original_text[i] != restored_text[i]:
+                        diff_pos = i
+                        break
+
+                if diff_pos >= 0:
+                    context_start = max(0, diff_pos - 50)
+                    context_end = min(len(original_text), diff_pos + 50)
+                    errors.append(
+                        f"First difference at position {diff_pos}:\n"
+                        f"Original: {repr(original_text[context_start:context_end])}\n"
+                        f"Restored: {repr(restored_text[context_start:context_end])}"
+                    )
+                elif len(original_text) != len(restored_text):
+                    errors.append(
+                        f"Texts match up to position {min_len}, but lengths differ"
+                    )
+        except Exception as e:
+            error_msg = f"Restoration exception: {str(e)}"
+            errors.append(error_msg)
+            logger.error(error_msg)
+
+        # If there are errors, save debug information to file
+        if errors:
+            timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
+            error_file = f"/tmp/chunk_error_{timestamp}.md"
+
+            with open(error_file, "w", encoding="utf-8") as f:
+                f.write("# Chunk Validation Error Report\n\n")
+                f.write(f"Timestamp: {timestamp}\n\n")
+
+                f.write("## Errors\n\n")
+                for error in errors:
+                    f.write(f"- {error}\n\n")
+
+                f.write("\n## Original Text\n\n")
+                f.write(f"Length: {len(original_text)}\n\n")
+                f.write("```\n")
+                f.write(original_text)
+                f.write("\n```\n\n")
+
+                f.write("\n## Chunks Information\n\n")
+                f.write(f"Total chunks: {len(chunks)}\n\n")
+                for i, (start, end, chunk_text) in enumerate(chunks):
+                    f.write(f"### Chunk {i}\n\n")
+                    f.write(f"- Position: [{start}:{end}]\n")
+                    f.write(f"- Length: {len(chunk_text)}\n")
+                    f.write(f"- Content:\n\n```\n{chunk_text}\n```\n\n")
+
+                try:
+                    restored_text = self.restore_text(chunks)
+                    f.write("\n## Restored Text\n\n")
+                    f.write(f"Length: {len(restored_text)}\n\n")
+                    f.write("```\n")
+                    f.write(restored_text)
+                    f.write("\n```\n")
+                except Exception as e:
+                    f.write("\n## Restoration Failed\n\n")
+                    f.write(f"Error: {str(e)}\n")
+
+            logger.error(f"Validation errors saved to: {error_file}")
+
+    def restore_text(self, chunks: List[Tuple[int, int, str]]) -> str:
+        """Restore original text from chunks with overlap handling.
+
+        This method reconstructs the original text from chunks that may contain:
+        - Overlapping content between consecutive chunks
+        - Prepended headers that were added during merging (headers have start==end position)
+
+        The algorithm:
+        1. Sort chunks by their start position (and end position as tiebreaker)
+        2. Track the maximum end position seen so far
+        3. For each chunk, extract only the new content (after max_end_pos)
+        4. Concatenate all new content pieces
+
+        Args:
+            chunks: List of tuples (start_pos, end_pos, chunk_text) from split_text()
+
+        Returns:
+            The restored original text
+
+        Example:
+            >>> splitter = TextSplitter(chunk_size=10, chunk_overlap=3)
+            >>> chunks = splitter.split_text("Hello World!")
+            >>> restored = splitter.restore_text(chunks)
+            >>> assert restored == "Hello World!"
+        """
+        if not chunks:
+            return ""
+
+        # Sort chunks by start position, then by end position
+        sorted_chunks = sorted(chunks, key=lambda x: (x[1], x[0]))
+
+        result_parts = []
+        last_end = 0
+
+        for start_pos, end_pos, chunk_text in sorted_chunks:
+            result_parts.append(chunk_text[last_end - end_pos :])
+            last_end = end_pos
+
+        return "".join(result_parts)
+
+
+if __name__ == "__main__":
+    s = """
+    这是一些普通文本。
+
+    | 姓名 | 年龄 | 城市 |
+    |------|------|------|
+    | 张三 | 25   | 北京 |
+    | 李四 | 30   | 上海 |
+    | 王五 | 28   | 广州 |
+    | 张三 | 25   | 北京 |
+    | 李四 | 30   | 上海 |
+    | 王五 | 28   | 广州 |
+
+    这是文本结束。
+
+"""
+
+    sp = TextSplitter(
+        chunk_size=200,
+        chunk_overlap=10,
+        separators=["\n\n", "\n", "。", "？", "！", "，", "；", "："],
+    )
+    ck = sp.split_text(s)
+    for c in ck:
+        print("------", len(c))
+        print(c)
+    pass

docreader/testdata/test.html

@@ -0,0 +1,56 @@
+<!DOCTYPE html>
+<html lang="zh-CN">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>测试 HTML 文档</title>
+</head>
+<body>
+    <h1>测试 HTML 文档</h1>
+    
+    <p>这是一个测试 HTML 文档，用于测试 HTML 解析功能。</p>
+    
+    <h2>包含图片</h2>
+    <img src="https://example.com/image.jpg" alt="测试图片">
+    
+    <h2>包含链接</h2>
+    <p>这是一个<a href="https://example.com">测试链接</a>。</p>
+    
+    <h2>包含代码块</h2>
+    <pre><code>
+def hello_world():
+    print("Hello, World!")
+    </code></pre>
+    
+    <h2>包含表格</h2>
+    <table>
+        <thead>
+            <tr>
+                <th>表头1</th>
+                <th>表头2</th>
+            </tr>
+        </thead>
+        <tbody>
+            <tr>
+                <td>内容1</td>
+                <td>内容2</td>
+            </tr>
+            <tr>
+                <td>内容3</td>
+                <td>内容4</td>
+            </tr>
+        </tbody>
+    </table>
+    
+    <h2>测试分块功能</h2>
+    <p>这部分内容用于测试分块功能，确保 HTML 结构在分块时保持完整。</p>
+    <ul>
+        <li>第一块内容</li>
+        <li>第二块内容</li>
+        <li>第三块内容</li>
+    </ul>
+    
+    <h2>测试重叠功能</h2>
+    <p>这部分内容可能会在分块时与前后块重叠，以确保上下文的连续性。</p>
+</body>
+</html> 

docreader/testdata/test.md

@@ -0,0 +1,37 @@
+# 测试 Markdown 文档
+
+这是一个测试 Markdown 文档，用于测试 Markdown 解析功能。
+
+## 包含图片
+
+![测试图片](https://geektutu.com/post/quick-go-protobuf/go-protobuf.jpg)
+
+## 包含链接
+
+这是一个[测试链接](https://example.com)。
+
+## 包含代码块
+
+```python
+def hello_world():
+    print("Hello, World!")
+```
+
+## 包含表格
+
+| 表头1 | 表头2 |
+|-------|-------|
+| 内容1 | 内容2 |
+| 内容3 | 内容4 |
+
+## 测试分块功能
+
+这部分内容用于测试分块功能，确保 Markdown 结构在分块时保持完整。
+
+- 第一块内容
+- 第二块内容
+- 第三块内容
+
+## 测试重叠功能
+
+这部分内容可能会在分块时与前后块重叠，以确保上下文的连续性。 

docreader/testdata/test.txt

@@ -0,0 +1,16 @@
+这是一个测试文档
+包含多行内容
+用于测试文档解析功能
+
+这个文档包含以下内容：
+1. 基本文本内容
+2. 多行段落
+3. 列表项
+
+测试分块功能：
+- 第一块内容
+- 第二块内容
+- 第三块内容
+
+测试重叠功能：
+这部分内容可能会在分块时与前后块重叠，以确保上下文的连续性。 

docreader/testdata/test_download.txt

@@ -0,0 +1,19 @@
+这是一个测试文档
+包含多行内容
+用于测试文档解析功能
+
+这个文档包含以下内容：
+1. 基本文本内容
+2. 多行段落
+3. 列表项
+
+测试分块功能：
+- 第一块内容
+- 第二块内容
+- 第三块内容
+
+测试重叠功能：
+这部分内容可能会在分块时与前后块重叠，以确保上下文的连续性。 
+
+
+test

docreader/tests/fixtures/generate_issue_2634_fixture.py

@@ -0,0 +1,57 @@
+"""Generate the real DOCX regression fixture for GitHub issue #2634."""
+
+from pathlib import Path
+
+from docx import Document
+from docx.enum.table import WD_CELL_VERTICAL_ALIGNMENT
+
+OUTPUT = Path(__file__).with_name("issue_2634_vertical_merge.docx")
+
+
+def main() -> None:
+    document = Document()
+    document.add_heading("遗传病检测项目表", level=1)
+    document.add_paragraph(
+        "回归场景：检测方法纵向合并，但每条检测项目在转换后都必须保留该方法。"
+    )
+
+    rows = [
+        ("Q0101", "遗传性乳腺癌", "BRCA1", "15个工作日"),
+        ("Q0102", "遗传性卵巢癌", "BRCA2", "15个工作日"),
+        ("Q0103", "林奇综合征", "MLH1", "15个工作日"),
+        ("Q0104", "家族性腺瘤性息肉病", "APC", "15个工作日"),
+    ]
+    method = (
+        "检测方法：提取外周血基因组 DNA，采用高通量测序，并对候选位点进行"
+        "Sanger 测序验证；检测结果需结合临床表现综合判断。"
+    )
+
+    table = document.add_table(rows=len(rows) + 1, cols=5)
+    table.style = "Table Grid"
+    headers = ("项目编号", "检测项目", "相关基因", "周期", "检测方法")
+    for column, value in enumerate(headers):
+        table.cell(0, column).text = value
+
+    for row_index, row in enumerate(rows, start=1):
+        for column, value in enumerate(row):
+            table.cell(row_index, column).text = value
+
+    merged_method = table.cell(1, 4).merge(table.cell(len(rows), 4))
+    merged_method.text = method
+    merged_method.vertical_alignment = WD_CELL_VERTICAL_ALIGNMENT.CENTER
+
+    document.add_paragraph("非合并对照表：相邻单元格内容允许相同，但不应被当作合并。")
+    control = document.add_table(rows=2, cols=3)
+    control.style = "Table Grid"
+    for column, value in enumerate(("列A", "列B", "列C")):
+        control.cell(0, column).text = value
+    for column, value in enumerate(("相同值", "相同值", "独立值")):
+        control.cell(1, column).text = value
+
+    document.save(OUTPUT)
+    print(f"generated={OUTPUT}")
+    print(f"size={OUTPUT.stat().st_size}")
+
+
+if __name__ == "__main__":
+    main()

docreader/tests/test_config.py

@@ -0,0 +1,43 @@
+import os
+import unittest
+from unittest.mock import patch
+
+from docreader import config
+
+
+class DocReaderConfigTest(unittest.TestCase):
+    def test_parser_concurrency_defaults_are_conservative(self):
+        with patch.dict(os.environ, {}, clear=True):
+            cfg = config.load_config()
+
+        self.assertEqual(cfg.markitdown_max_workers, 1)
+        self.assertEqual(cfg.pdf_render_max_workers, 1)
+        self.assertEqual(cfg.pdf_render_dpi, 200)
+        self.assertEqual(cfg.pdf_jpeg_quality, 85)
+
+    def test_loads_parser_concurrency_env(self):
+        env = {
+            "DOCREADER_MARKITDOWN_MAX_WORKERS": "3",
+            "DOCREADER_PDF_RENDER_MAX_WORKERS": "2",
+            "DOCREADER_PDF_RENDER_DPI": "180",
+            "DOCREADER_PDF_JPEG_QUALITY": "85",
+        }
+        with patch.dict(os.environ, env):
+            cfg = config.load_config()
+
+        self.assertEqual(cfg.markitdown_max_workers, 3)
+        self.assertEqual(cfg.pdf_render_max_workers, 2)
+        self.assertEqual(cfg.pdf_render_dpi, 180)
+        self.assertEqual(cfg.pdf_jpeg_quality, 85)
+
+    def test_dump_config_includes_parser_limits(self):
+        dumped = config.dump_config()
+
+        self.assertIn("DOCREADER_MARKITDOWN_MAX_WORKERS", dumped)
+        self.assertIn("DOCREADER_PDF_RENDER_MAX_WORKERS", dumped)
+        self.assertIn("DOCREADER_PDF_RENDER_DPI", dumped)
+        self.assertIn("DOCREADER_PDF_JPEG_QUALITY", dumped)
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/tests/test_docx_merge.py

@@ -0,0 +1,187 @@
+import io
+import unittest
+import zipfile
+from pathlib import Path
+from types import SimpleNamespace
+from unittest.mock import patch
+
+from docx import Document as WordDocument
+from lxml import etree
+
+from docreader.parser.docx2_parser import Docx2Parser
+from docreader.parser.docx_merge import fill_vertical_merged_cells_docx
+from docreader.parser.markitdown_parser import StdMarkitdownParser
+
+FIXTURE = Path(__file__).with_name("fixtures") / "issue_2634_vertical_merge.docx"
+METHOD_PREFIX = "检测方法：提取外周血基因组 DNA"
+PROJECT_CODES = ("Q0101", "Q0102", "Q0103", "Q0104")
+WORD_NAMESPACE = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
+
+
+class DocxVerticalMergeFillTest(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.fixture_bytes = FIXTURE.read_bytes()
+
+    def test_fixture_is_a_real_docx_with_vertical_merge(self):
+        self.assertTrue(zipfile.is_zipfile(io.BytesIO(self.fixture_bytes)))
+        with zipfile.ZipFile(io.BytesIO(self.fixture_bytes)) as archive:
+            self.assertIsNone(archive.testzip())
+            root = etree.fromstring(archive.read("word/document.xml"))
+
+        vertical_merges = root.xpath(
+            ".//w:tbl[1]//w:vMerge", namespaces={"w": WORD_NAMESPACE}
+        )
+        merge_values = [
+            node.get(f"{{{WORD_NAMESPACE}}}val", "continue") for node in vertical_merges
+        ]
+        self.assertEqual(
+            merge_values,
+            ["restart", "continue", "continue", "continue"],
+        )
+
+    def test_fill_repeats_master_content_in_every_covered_row(self):
+        filled = fill_vertical_merged_cells_docx(self.fixture_bytes)
+        document = WordDocument(io.BytesIO(filled))
+        table = document.tables[0]
+
+        for row_index, project_code in enumerate(PROJECT_CODES, start=1):
+            self.assertEqual(table.cell(row_index, 0).text, project_code)
+            self.assertIn(METHOD_PREFIX, table.cell(row_index, 4).text)
+
+        with zipfile.ZipFile(io.BytesIO(filled)) as archive:
+            root = etree.fromstring(archive.read("word/document.xml"))
+        self.assertEqual(
+            root.xpath("count(.//w:vMerge)", namespaces={"w": WORD_NAMESPACE}),
+            0.0,
+        )
+
+    def test_non_merged_equal_cells_are_not_collapsed(self):
+        filled = fill_vertical_merged_cells_docx(self.fixture_bytes)
+        document = WordDocument(io.BytesIO(filled))
+        control_row = document.tables[1].rows[1]
+        self.assertEqual(len(control_row.cells), 3)
+        self.assertEqual(
+            [cell.text for cell in control_row.cells],
+            ["相同值", "相同值", "独立值"],
+        )
+
+    def test_non_docx_bytes_pass_through_unchanged(self):
+        content = b"not a docx package"
+        self.assertIs(fill_vertical_merged_cells_docx(content), content)
+
+    def test_docx_without_vertical_merges_passes_through_unchanged(self):
+        document = WordDocument()
+        table = document.add_table(rows=1, cols=2)
+        table.cell(0, 0).text = "left"
+        table.cell(0, 1).text = "right"
+        output = io.BytesIO()
+        document.save(output)
+        content = output.getvalue()
+
+        self.assertIs(fill_vertical_merged_cells_docx(content), content)
+
+    def test_horizontal_merge_only_passes_through_unchanged(self):
+        document = WordDocument()
+        table = document.add_table(rows=1, cols=3)
+        table.cell(0, 0).merge(table.cell(0, 1)).text = "horizontal merge"
+        table.cell(0, 2).text = "independent"
+        output = io.BytesIO()
+        document.save(output)
+        content = output.getvalue()
+
+        self.assertIs(fill_vertical_merged_cells_docx(content), content)
+
+    def test_separate_merge_groups_keep_distinct_values_and_other_columns(self):
+        document = WordDocument()
+        table = document.add_table(rows=6, cols=2)
+        table.cell(0, 0).text = "group"
+        table.cell(0, 1).text = "item"
+        for row_index in range(1, 6):
+            table.cell(row_index, 1).text = f"item-{row_index}"
+
+        table.cell(1, 0).merge(table.cell(2, 0)).text = "group-a"
+        table.cell(3, 0).text = "standalone"
+        table.cell(4, 0).merge(table.cell(5, 0)).text = "group-b"
+
+        output = io.BytesIO()
+        document.save(output)
+        filled = fill_vertical_merged_cells_docx(output.getvalue())
+        parsed_table = WordDocument(io.BytesIO(filled)).tables[0]
+
+        self.assertEqual(
+            [parsed_table.cell(row_index, 0).text for row_index in range(1, 6)],
+            ["group-a", "group-a", "standalone", "group-b", "group-b"],
+        )
+        self.assertEqual(
+            [parsed_table.cell(row_index, 1).text for row_index in range(1, 6)],
+            [f"item-{row_index}" for row_index in range(1, 6)],
+        )
+
+
+class DocxMarkitdownDispatchTest(unittest.TestCase):
+    def test_non_docx_types_do_not_run_docx_preprocessor(self):
+        for file_type in ("doc", "txt", "md"):
+            with self.subTest(file_type=file_type):
+                parser = StdMarkitdownParser(file_type=file_type)
+                result = SimpleNamespace(text_content="parsed")
+                with (
+                    patch(
+                        "docreader.parser.markitdown_parser.fill_vertical_merged_cells_docx"
+                    ) as preprocess,
+                    patch.object(parser, "_convert_markitdown", return_value=result),
+                ):
+                    document = parser.parse_into_text(b"payload")
+
+                self.assertEqual(document.content, "parsed")
+                preprocess.assert_not_called()
+
+    def test_uppercase_docx_type_runs_docx_preprocessor(self):
+        parser = StdMarkitdownParser(file_type=".DOCX")
+        result = SimpleNamespace(text_content="parsed")
+        with (
+            patch(
+                "docreader.parser.markitdown_parser.fill_vertical_merged_cells_docx",
+                return_value=b"preprocessed",
+            ) as preprocess,
+            patch.object(parser, "_convert_markitdown", return_value=result) as convert,
+        ):
+            document = parser.parse_into_text(b"payload")
+
+        self.assertEqual(document.content, "parsed")
+        preprocess.assert_called_once_with(b"payload")
+        convert.assert_called_once_with(b"preprocessed", ".DOCX", keep_data_uris=True)
+
+
+class Issue2634ParserRegressionTest(unittest.TestCase):
+    def test_markitdown_output_repeats_merged_value_for_every_data_row(self):
+        document = StdMarkitdownParser(
+            file_name=FIXTURE.name, file_type="docx"
+        ).parse_into_text(FIXTURE.read_bytes())
+
+        data_lines = {
+            project_code: next(
+                line for line in document.content.splitlines() if project_code in line
+            )
+            for project_code in PROJECT_CODES
+        }
+        for line in data_lines.values():
+            self.assertIn(METHOD_PREFIX, line)
+        self.assertEqual(document.content.count(METHOD_PREFIX), len(PROJECT_CODES))
+        self.assertIn("| 相同值 | 相同值 | 独立值 |", document.content)
+
+    def test_full_docx_entry_keeps_context_on_all_four_rows(self):
+        document = Docx2Parser(
+            file_name=FIXTURE.name, file_type="docx"
+        ).parse_into_text(FIXTURE.read_bytes())
+
+        for project_code in PROJECT_CODES:
+            data_line = next(
+                line for line in document.content.splitlines() if project_code in line
+            )
+            self.assertIn(METHOD_PREFIX, data_line)
+        self.assertEqual(document.content.count(METHOD_PREFIX), len(PROJECT_CODES))
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/tests/test_docx_tables.py

@@ -0,0 +1,213 @@
+import importlib.util
+import io
+import sys
+import types
+import unittest
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+from unittest.mock import patch
+
+from docx import Document as WordDocument
+
+
+def _load_docx_parser():
+    """Load docx_parser without triggering the heavy package __init__.
+
+    ``docreader/parser/__init__.py`` imports doc_parser -> textract, a heavy
+    dependency unrelated to DOCX parsing. Registering the packages as bare
+    namespaces lets us import only the modules docx_parser actually needs.
+    """
+    root = Path(__file__).resolve().parents[2]
+    docreader_pkg = types.ModuleType("docreader")
+    docreader_pkg.__path__ = [str(root / "docreader")]
+    sys.modules.setdefault("docreader", docreader_pkg)
+    parser_pkg = types.ModuleType("docreader.parser")
+    parser_pkg.__path__ = [str(root / "docreader" / "parser")]
+    sys.modules["docreader.parser"] = parser_pkg
+
+    spec = importlib.util.spec_from_file_location(
+        "docreader.parser.docx_parser", root / "docreader" / "parser" / "docx_parser.py"
+    )
+    module = importlib.util.module_from_spec(spec)
+    sys.modules["docreader.parser.docx_parser"] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+docx_parser = _load_docx_parser()
+DocxParser = docx_parser.DocxParser
+table_to_gfm_markdown = docx_parser.table_to_gfm_markdown
+
+
+def _parse(content):
+    """Parse a DOCX through the real Docx processor.
+
+    The production path uses a ProcessPoolExecutor backed by a multiprocessing
+    Manager. Some CI sandboxes forbid POSIX semaphores, so run the same page
+    task pool on threads and swap the manager for a plain list - parse
+    behavior is identical, only the execution backend changes.
+    """
+
+    class _FakeManager:
+        def __enter__(self):
+            self._items = []
+            return self
+
+        def __exit__(self, *exc):
+            return False
+
+        def list(self):
+            return self._items
+
+    with patch.object(docx_parser, "Manager", _FakeManager), patch.object(
+        docx_parser, "ProcessPoolExecutor", ThreadPoolExecutor
+    ):
+        return DocxParser(max_pages=100).parse_into_text(content)
+
+
+def _docx_bytes(build):
+    doc = WordDocument()
+    build(doc)
+    buf = io.BytesIO()
+    doc.save(buf)
+    return buf.getvalue()
+
+
+class DocxTableContentTest(unittest.TestCase):
+    """Regression test: DOCX tables must be kept in the parsed text.
+
+    Docx.__call__ used to return tables separately from the text lines, and
+    parse_into_text dropped them, silently losing all table content.
+    """
+
+    def _docx_with_table(self, cells):
+        def build(doc):
+            doc.add_paragraph("Introduction paragraph")
+            table = doc.add_table(rows=len(cells), cols=len(cells[0]))
+            for r, row in enumerate(cells):
+                for c, value in enumerate(row):
+                    table.cell(r, c).text = value
+
+        return _docx_bytes(build)
+
+    def test_table_content_is_kept_in_parsed_text(self):
+        content = self._docx_with_table(
+            [["City", "Population"], ["Beijing", "21.5M"]]
+        )
+        document = _parse(content)
+        self.assertIn("Introduction paragraph", document.content)
+        for cell in ("City", "Population", "Beijing", "21.5M"):
+            self.assertIn(cell, document.content)
+        self.assertIn("| City | Population |", document.content)
+        self.assertIn("| --- | --- |", document.content)
+
+    def test_table_only_document_is_not_empty(self):
+        def build(doc):
+            table = doc.add_table(rows=2, cols=1)
+            table.cell(0, 0).text = "Header"
+            table.cell(1, 0).text = "Value"
+
+        document = _parse(_docx_bytes(build))
+        self.assertIn("Header", document.content)
+        self.assertIn("Value", document.content)
+
+    def test_pipe_in_cell_does_not_break_table(self):
+        def build(doc):
+            table = doc.add_table(rows=2, cols=1)
+            table.cell(0, 0).text = "A|B"
+            table.cell(1, 0).text = "C"
+
+        document = _parse(_docx_bytes(build))
+        self.assertIn(r"A\|B", document.content)
+
+    def test_empty_adjacent_cells_keep_columns(self):
+        def build(doc):
+            table = doc.add_table(rows=2, cols=3)
+            table.cell(0, 0).text = "A"
+            table.cell(0, 1).text = "B"
+            table.cell(0, 2).text = "C"
+            table.cell(1, 0).text = "1"
+            table.cell(1, 1).text = ""
+            table.cell(1, 2).text = ""
+
+        document = _parse(_docx_bytes(build))
+        self.assertIn("| A | B | C |", document.content)
+        self.assertIn("| 1 |  |  |", document.content)
+
+    def test_equal_adjacent_cells_are_not_collapsed(self):
+        """Adjacent independent cells with the same text must stay separate.
+
+        Regression for the #2634 control row: ``相同值 | 相同值 | 独立值``.
+        """
+
+        def build(doc):
+            table = doc.add_table(rows=2, cols=3)
+            table.cell(0, 0).text = "相同值"
+            table.cell(0, 1).text = "相同值"
+            table.cell(0, 2).text = "独立值"
+            table.cell(1, 0).text = "x"
+            table.cell(1, 1).text = "y"
+            table.cell(1, 2).text = "z"
+
+        document = _parse(_docx_bytes(build))
+        self.assertIn("| 相同值 | 相同值 | 独立值 |", document.content)
+        self.assertIn("| x | y | z |", document.content)
+
+    def test_horizontal_merge_keeps_grid_width(self):
+        def build(doc):
+            table = doc.add_table(rows=2, cols=3)
+            table.cell(0, 0).merge(table.cell(0, 1)).text = "merged"
+            table.cell(0, 2).text = "right"
+            table.cell(1, 0).text = "a"
+            table.cell(1, 1).text = "b"
+            table.cell(1, 2).text = "c"
+
+        document = _parse(_docx_bytes(build))
+        self.assertIn("| merged | merged | right |", document.content)
+        self.assertIn("| a | b | c |", document.content)
+
+    def test_table_stays_between_paragraphs(self):
+        def build(doc):
+            doc.add_paragraph("before table")
+            table = doc.add_table(rows=2, cols=1)
+            table.cell(0, 0).text = "H"
+            table.cell(1, 0).text = "V"
+            doc.add_paragraph("after table")
+
+        document = _parse(_docx_bytes(build))
+        before = document.content.index("before table")
+        header = document.content.index("| H |")
+        after = document.content.index("after table")
+        self.assertLess(before, header)
+        self.assertLess(header, after)
+
+    def test_html_like_cell_text_is_literal(self):
+        def build(doc):
+            table = doc.add_table(rows=1, cols=2)
+            table.cell(0, 0).text = "a < b"
+            table.cell(0, 1).text = "</td><td>injected"
+
+        document = _parse(_docx_bytes(build))
+        self.assertIn("| a < b | </td><td>injected |", document.content)
+
+    def test_table_to_gfm_markdown_helper(self):
+        doc = WordDocument()
+        table = doc.add_table(rows=2, cols=2)
+        table.cell(0, 0).text = "A"
+        table.cell(0, 1).text = "B"
+        table.cell(1, 0).text = "C"
+        table.cell(1, 1).text = "D"
+        self.assertEqual(
+            table_to_gfm_markdown(table),
+            "| A | B |\n| --- | --- |\n| C | D |",
+        )
+
+    def test_table_to_gfm_markdown_skips_empty(self):
+        class _Empty:
+            rows = []
+
+        self.assertEqual(table_to_gfm_markdown(_Empty()), "")
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/tests/test_epub_parser.py

@@ -0,0 +1,92 @@
+import os
+import tempfile
+import unittest
+
+from ebooklib import epub
+
+from docreader.parser.epub_parser import EPUBParser
+from docreader.parser.registry import registry
+
+
+def _minimal_epub_bytes() -> bytes:
+    book = epub.EpubBook()
+    book.set_identifier("test-epub")
+    book.set_title("Tiny EPUB")
+    book.set_language("en")
+    book.add_author("WeKnora")
+
+    chapter = epub.EpubHtml(
+        title="Chapter One", file_name="text/chapter_01.xhtml", lang="en"
+    )
+    chapter.content = (
+        "<html><body><h1>Chapter One</h1>"
+        "<p>Hello EPUB world.</p>"
+        '<p><a href="chapter_02.xhtml#sec2">Chapter 2</a> '
+        '<a href="#footnote1">note</a> '
+        '<a href="https://example.com">the site</a></p>'
+        '<img alt="cover" src="../images/pic.png">'
+        "</body></html>"
+    )
+    book.add_item(chapter)
+    book.add_item(
+        epub.EpubItem(
+            uid="pic",
+            file_name="images/pic.png",
+            media_type="image/png",
+            content=b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR",
+        )
+    )
+    book.toc = (epub.Link("text/chapter_01.xhtml", "Chapter One", "chapter-one"),)
+    book.spine = ["nav", chapter]
+    book.add_item(epub.EpubNcx())
+    book.add_item(epub.EpubNav())
+
+    with tempfile.NamedTemporaryFile(suffix=".epub", delete=False) as handle:
+        path = handle.name
+    try:
+        epub.write_epub(path, book)
+        with open(path, "rb") as handle:
+            return handle.read()
+    finally:
+        if os.path.exists(path):
+            os.unlink(path)
+
+
+class EPUBParserTest(unittest.TestCase):
+    def test_parse_minimal_epub(self):
+        document = EPUBParser(
+            file_name="tiny.epub", file_type="epub"
+        ).parse_into_text(_minimal_epub_bytes())
+
+        self.assertIn("Hello EPUB world", document.content)
+        self.assertEqual(document.metadata["source_format"], "epub")
+        self.assertEqual(len(document.images), 1)
+        image_ref = next(iter(document.images))
+        self.assertTrue(image_ref.startswith("images/"))
+        self.assertIn(image_ref, document.content)
+        self.assertNotIn("../images/pic.png", document.content)
+
+    def test_internal_links_are_unwrapped_but_external_links_remain(self):
+        document = EPUBParser(
+            file_name="tiny.epub", file_type="epub"
+        ).parse_into_text(_minimal_epub_bytes())
+
+        self.assertIn("Chapter 2", document.content)
+        self.assertIn("note", document.content)
+        self.assertNotIn("chapter_02.xhtml#sec2", document.content)
+        self.assertNotIn("#footnote1", document.content)
+        self.assertIn("[the site](https://example.com)", document.content)
+
+    def test_parse_without_images(self):
+        document = EPUBParser(
+            file_name="tiny.epub", file_type="epub", extract_images=False
+        ).parse_into_text(_minimal_epub_bytes())
+
+        self.assertEqual(document.images, {})
+
+    def test_registry_resolves_epub(self):
+        self.assertIs(registry.get_parser_class("", "epub"), EPUBParser)
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/tests/test_excel_parser.py

@@ -0,0 +1,513 @@
+import io
+import os
+import shutil
+import subprocess
+import tempfile
+import unittest
+import zipfile
+from unittest.mock import patch
+
+import openpyxl
+import pandas as pd
+from openpyxl.chart import BarChart, Reference
+
+from docreader.parser.excel_convert import detect_excel_format, engine_for_format
+from docreader.parser.excel_parser import ExcelParser
+from docreader.parser.xlsx_merge import fill_merged_cells_xlsx
+from docreader.parser.xlsx_repair import repair_xlsx_bytes
+
+
+def _xlsx_with_phantom_shared_strings() -> bytes:
+    """Workbook with inline strings but a dangling sharedStrings manifest entry."""
+    wb = openpyxl.Workbook()
+    ws = wb.active
+    ws["A1"] = "hello"
+    ws["B1"] = 42
+    bio = io.BytesIO()
+    wb.save(bio)
+
+    with tempfile.TemporaryDirectory() as tmpdir:
+        with zipfile.ZipFile(io.BytesIO(bio.getvalue()), "r") as zin:
+            zin.extractall(tmpdir)
+
+        ct_path = f"{tmpdir}/[Content_Types].xml"
+        with open(ct_path, encoding="utf-8") as f:
+            ct = f.read()
+        override = (
+            '<Override PartName="/xl/sharedStrings.xml" '
+            'ContentType="application/vnd.openxmlformats-officedocument.'
+            'spreadsheetml.sharedStrings+xml"/>'
+        )
+        with open(ct_path, "w", encoding="utf-8") as f:
+            f.write(ct.replace("</Types>", override + "</Types>"))
+
+        out = io.BytesIO()
+        with zipfile.ZipFile(out, "w", zipfile.ZIP_DEFLATED) as zout:
+            for root, _, files in os.walk(tmpdir):
+                for name in files:
+                    path = os.path.join(root, name)
+                    arc = os.path.relpath(path, tmpdir)
+                    zout.write(path, arc)
+        return out.getvalue()
+
+
+class ExcelFormatDetectionTest(unittest.TestCase):
+    def test_detect_xlsx_and_engine(self):
+        wb = openpyxl.Workbook()
+        bio = io.BytesIO()
+        wb.save(bio)
+        content = bio.getvalue()
+        self.assertEqual(detect_excel_format(content), "xlsx")
+        self.assertEqual(engine_for_format("xlsx"), "openpyxl")
+
+    def test_detect_xls_magic(self):
+        content = b"\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1" + b"\x00" * 512
+        self.assertEqual(detect_excel_format(content), "xls")
+        self.assertEqual(engine_for_format("xls"), "xlrd")
+
+    def test_open_legacy_xls_bytes_with_xlsx_extension(self):
+        if not shutil.which("soffice"):
+            self.skipTest("LibreOffice not available")
+        wb = openpyxl.Workbook()
+        ws = wb.active
+        ws["A1"] = "legacy"
+        xlsx_bio = io.BytesIO()
+        wb.save(xlsx_bio)
+        with tempfile.TemporaryDirectory() as tmpdir:
+            src = os.path.join(tmpdir, "sheet.xlsx")
+            with open(src, "wb") as handle:
+                handle.write(xlsx_bio.getvalue())
+            subprocess.run(
+                [
+                    "soffice",
+                    "--headless",
+                    "--convert-to",
+                    "xls",
+                    "--outdir",
+                    tmpdir,
+                    src,
+                ],
+                check=True,
+                capture_output=True,
+            )
+            xls_path = os.path.join(tmpdir, "sheet.xls")
+            with open(xls_path, "rb") as handle:
+                xls_bytes = handle.read()
+
+        document = ExcelParser(file_name="fake.xlsx", file_type="xlsx").parse_into_text(
+            xls_bytes
+        )
+        self.assertIn("legacy", document.content)
+
+
+class XlsxRepairTest(unittest.TestCase):
+    def test_repair_removes_phantom_shared_strings_reference(self):
+        broken = _xlsx_with_phantom_shared_strings()
+        with self.assertRaises(KeyError):
+            pd.read_excel(io.BytesIO(broken))
+
+        repaired = repair_xlsx_bytes(broken)
+        self.assertIsNotNone(repaired)
+        df = pd.read_excel(io.BytesIO(repaired), header=None)
+        self.assertEqual(df.values.tolist(), [["hello", 42]])
+
+    def test_repair_skips_when_shared_string_cells_need_table(self):
+        import xlsxwriter
+
+        bio = io.BytesIO()
+        wb = xlsxwriter.Workbook(bio, {"in_memory": True})
+        ws = wb.add_worksheet()
+        ws.write(0, 0, "hello")
+        wb.close()
+
+        with tempfile.TemporaryDirectory() as tmpdir:
+            with zipfile.ZipFile(io.BytesIO(bio.getvalue()), "r") as zin:
+                zin.extractall(tmpdir)
+            os.remove(f"{tmpdir}/xl/sharedStrings.xml")
+
+            out = io.BytesIO()
+            with zipfile.ZipFile(out, "w", zipfile.ZIP_DEFLATED) as zout:
+                for root, _, files in os.walk(tmpdir):
+                    for name in files:
+                        path = os.path.join(root, name)
+                        arc = os.path.relpath(path, tmpdir)
+                        zout.write(path, arc)
+            broken = out.getvalue()
+
+        self.assertIsNone(repair_xlsx_bytes(broken))
+
+
+class XlsxMergeFillTest(unittest.TestCase):
+    @staticmethod
+    def _workbook_with_merged_cells_and_chart() -> bytes:
+        wb = openpyxl.Workbook()
+        ws = wb.active
+        ws["A1"] = "title"
+        ws.merge_cells("A1:B1")
+        ws.append(["category", "value"])
+        ws.append(["one", 1])
+        ws.append(["two", 2])
+
+        chart = BarChart()
+        chart.add_data(
+            Reference(ws, min_col=2, min_row=2, max_row=4),
+            titles_from_data=True,
+        )
+        chart.set_categories(Reference(ws, min_col=1, min_row=3, max_row=4))
+        ws.add_chart(chart, "D2")
+
+        bio = io.BytesIO()
+        wb.save(bio)
+        return bio.getvalue()
+
+    def test_fill_merged_cells_propagates_master_value(self):
+        wb = openpyxl.Workbook()
+        ws = wb.active
+        ws["A1"] = "title"
+        ws.merge_cells("A1:B1")
+        ws["A2"] = "left"
+        ws["B2"] = "right"
+        ws.merge_cells("A2:A3")
+        ws["B3"] = "only-b"
+        bio = io.BytesIO()
+        wb.save(bio)
+
+        filled = fill_merged_cells_xlsx(bio.getvalue())
+        out_wb = openpyxl.load_workbook(io.BytesIO(filled), data_only=True)
+        out_ws = out_wb.active
+        self.assertEqual(out_ws["B1"].value, "title")
+        self.assertEqual(out_ws["A3"].value, "left")
+        self.assertEqual(out_ws["B3"].value, "only-b")
+
+    def test_fill_merged_cells_does_not_serialize_charts(self):
+        content = self._workbook_with_merged_cells_and_chart()
+
+        with patch(
+            "openpyxl.chart._chart.ChartBase._write",
+            side_effect=AttributeError("'Typed' object has no attribute 'to_tree'"),
+        ):
+            filled = fill_merged_cells_xlsx(content)
+
+        out_wb = openpyxl.load_workbook(io.BytesIO(filled), data_only=True)
+        out_ws = out_wb.active
+        self.assertEqual(out_ws["B1"].value, "title")
+        self.assertEqual(out_ws["A3"].value, "one")
+        self.assertEqual(out_ws["B4"].value, 2)
+        self.assertEqual(out_ws._charts, [])
+
+    def test_excel_parser_reads_merged_workbook_with_chart(self):
+        document = ExcelParser().parse_into_text(
+            self._workbook_with_merged_cells_and_chart()
+        )
+
+        self.assertIn("A: title,B: title", document.content)
+        self.assertIn("A: one,B: 1", document.content)
+        self.assertIn("A: two,B: 2", document.content)
+
+    def test_chart_workbook_without_merges_passes_through_unchanged(self):
+        content = self._workbook_with_merged_cells_and_chart()
+        wb = openpyxl.load_workbook(io.BytesIO(content))
+        wb.active.unmerge_cells("A1:B1")
+        bio = io.BytesIO()
+        wb.save(bio)
+        content = bio.getvalue()
+
+        self.assertEqual(fill_merged_cells_xlsx(content), content)
+
+    def test_fill_does_not_serialize_dedicated_chart_sheets(self):
+        content = self._workbook_with_merged_cells_and_chart()
+        wb = openpyxl.load_workbook(io.BytesIO(content))
+        ws = wb.active
+        chart = BarChart()
+        chart.add_data(
+            Reference(ws, min_col=2, min_row=2, max_row=4),
+            titles_from_data=True,
+        )
+        chart.set_categories(Reference(ws, min_col=1, min_row=3, max_row=4))
+        chart_sheet = wb.create_chartsheet("Summary")
+        chart_sheet.add_chart(chart)
+        wb.active = chart_sheet
+        bio = io.BytesIO()
+        wb.save(bio)
+
+        with patch(
+            "openpyxl.chart._chart.ChartBase._write",
+            side_effect=AttributeError("'Typed' object has no attribute 'to_tree'"),
+        ):
+            filled = fill_merged_cells_xlsx(bio.getvalue())
+
+        out_wb = openpyxl.load_workbook(io.BytesIO(filled), data_only=True)
+        self.assertEqual(out_wb.active._charts, [])
+        self.assertEqual(out_wb.active.title, "Sheet")
+        self.assertEqual(out_wb.chartsheets, [])
+
+    def test_fill_keeps_active_worksheet_after_removing_earlier_chart_sheet(self):
+        content = self._workbook_with_merged_cells_and_chart()
+        wb = openpyxl.load_workbook(io.BytesIO(content))
+        data_sheet = wb.active
+        chart = BarChart()
+        chart.add_data(
+            Reference(data_sheet, min_col=2, min_row=2, max_row=4),
+            titles_from_data=True,
+        )
+        chart_sheet = wb.create_chartsheet("Summary", 0)
+        chart_sheet.add_chart(chart)
+        wb.active = data_sheet
+        bio = io.BytesIO()
+        wb.save(bio)
+
+        filled = fill_merged_cells_xlsx(bio.getvalue())
+
+        out_wb = openpyxl.load_workbook(io.BytesIO(filled), data_only=True)
+        self.assertEqual(out_wb.active.title, "Sheet")
+        self.assertEqual(out_wb.chartsheets, [])
+
+    def test_parse_en_mergecell_workbook(self):
+        path = os.path.join(
+            os.path.dirname(__file__),
+            "..",
+            "..",
+            "testdata",
+            "rag_test",
+            "xlsx",
+            "en_mergecell.xlsx",
+        )
+        if not os.path.isfile(path):
+            self.skipTest("en_mergecell.xlsx fixture not available")
+        with open(path, "rb") as handle:
+            document = ExcelParser().parse_into_text(handle.read())
+
+        chunks = [chunk.content.strip() for chunk in document.chunks]
+        self.assertEqual(len(chunks), 12)
+        self.assertIn("A: A1", chunks[0])
+        self.assertIn("A: A2", chunks[1])
+        self.assertIn("B: B3", chunks[2])
+        self.assertNotIn("Unnamed:", document.content)
+        self.assertIn("A: A7", chunks[6])
+        self.assertIn("A: A7", chunks[7])
+        self.assertIn("D: D10", chunks[9])
+
+
+class ExcelImageFilterTest(unittest.TestCase):
+    """Tests for filtering embedded image function strings (#1779)."""
+
+    def _xlsx_with_image_functions(self) -> bytes:
+        """Create an XLSX where image functions are stored as text values.
+
+        WPS embeds images using =DISPIMG("ID",1) which may appear as plain
+        text (not a formula) in some export scenarios.
+        """
+        wb = openpyxl.Workbook()
+        ws = wb.active
+        ws["A1"] = "Name"
+        ws["B1"] = "Photo"
+        ws["A2"] = "Alice"
+        ws["B2"] = '_xlfn.DISPIMG("ID_ABCDEF123",1)'
+        ws["A3"] = "Bob"
+        ws["B3"] = '_xlfn.DISPIMG("ID_GHIJKL456",1)'
+        ws["A4"] = "Charlie"
+        ws["B4"] = "real data"
+        bio = io.BytesIO()
+        wb.save(bio)
+        return bio.getvalue()
+
+    def test_dispimg_text_values_are_excluded(self):
+        """Image function strings stored as text must not appear in output."""
+        document = ExcelParser().parse_into_text(self._xlsx_with_image_functions())
+        self.assertNotIn("DISPIMG", document.content)
+        self.assertNotIn("_xlfn", document.content)
+        # Real data must still be present
+        self.assertIn("Alice", document.content)
+        self.assertIn("Bob", document.content)
+        self.assertIn("Charlie", document.content)
+        self.assertIn("real data", document.content)
+
+    def test_dispimg_with_equals_prefix(self):
+        """=_xlfn.DISPIMG(...) as text (not formula) should also be filtered."""
+        wb = openpyxl.Workbook()
+        ws = wb.active
+        ws["A1"] = "Name"
+        ws["B1"] = "Photo"
+        ws["A2"] = "Alice"
+        # Stored as text with = prefix (some exporters do this)
+        ws["B2"] = '=_xlfn.DISPIMG("ID_123",1)'
+        bio = io.BytesIO()
+        wb.save(bio)
+        # Note: openpyxl treats strings starting with = as formulas,
+        # so data_only=True in fill_merged_cells_xlsx will turn them to None.
+        # This test verifies the formula path also works correctly.
+        document = ExcelParser().parse_into_text(bio.getvalue())
+        self.assertNotIn("DISPIMG", document.content)
+        self.assertIn("Alice", document.content)
+
+    def test_image_function_variations(self):
+        """Various image function patterns should all be filtered."""
+        wb = openpyxl.Workbook()
+        ws = wb.active
+        ws["A1"] = '_xlfn.DISPIMG("ID_001",1)'
+        ws["A2"] = 'DISPIMG("ID_002",1)'  # no prefix at all
+        ws["A3"] = '_xlfn.IMAGE("https://example.com/img.png")'
+        ws["A4"] = 'IMAGE("https://example.com/img.png",1)'
+        ws["A5"] = "Normal text"
+        bio = io.BytesIO()
+        wb.save(bio)
+        document = ExcelParser().parse_into_text(bio.getvalue())
+        self.assertNotIn("DISPIMG", document.content)
+        self.assertNotIn("IMAGE(", document.content)
+        self.assertIn("Normal text", document.content)
+
+    def test_real_world_wps_dispimg(self):
+        """Exact pattern from issue #1779 screenshot: =DISPIMG("ID_...",1)."""
+        wb = openpyxl.Workbook()
+        ws = wb.active
+        ws["A1"] = "Product"
+        ws["B1"] = "Image"
+        ws["C1"] = "Price"
+        ws["A2"] = "Basic"
+        # This is the exact format shown in the issue screenshot
+        ws["B2"] = 'DISPIMG("ID_5A60F9ED501E48A38EBEE5D326E18235",1)'
+        ws["C2"] = "39.9"
+        ws["A3"] = "Pro"
+        ws["B3"] = 'DISPIMG("ID_AABBCCDD",1)'
+        ws["C3"] = "99.9"
+        bio = io.BytesIO()
+        wb.save(bio)
+        document = ExcelParser().parse_into_text(bio.getvalue())
+        self.assertNotIn("DISPIMG", document.content)
+        self.assertNotIn("ID_5A60F9ED", document.content)
+        self.assertIn("Basic", document.content)
+        self.assertIn("39.9", document.content)
+        self.assertIn("Pro", document.content)
+        self.assertIn("99.9", document.content)
+
+
+class ExcelParserTest(unittest.TestCase):
+    @staticmethod
+    def _workbook_bytes(rows):
+        wb = openpyxl.Workbook()
+        ws = wb.active
+        for row in rows:
+            ws.append(row)
+        bio = io.BytesIO()
+        wb.save(bio)
+        return bio.getvalue()
+
+    def test_xlsx_first_row_header_mode_repeats_column_context(self):
+        content = self._workbook_bytes(
+            [
+                ["Name", "Age", "City"],
+                ["Alice", 30, "Shenzhen"],
+                ["Bob", 28, "Shanghai"],
+            ]
+        )
+
+        document = ExcelParser(
+            file_name="people.xlsx",
+            file_type="xlsx",
+            xlsx_first_row_as_header="true",
+        ).parse_into_text(content)
+
+        chunks = [chunk.content.strip() for chunk in document.chunks]
+        self.assertEqual(
+            chunks,
+            [
+                "Name: Alice,Age: 30,City: Shenzhen",
+                "Name: Bob,Age: 28,City: Shanghai",
+            ],
+        )
+        self.assertNotIn("A: Name", document.content)
+
+    def test_xlsx_keeps_first_row_as_data_by_default(self):
+        content = self._workbook_bytes(
+            [
+                ["Name", "City"],
+                ["Alice", "Shenzhen"],
+            ]
+        )
+
+        document = ExcelParser(file_name="people.xlsx", file_type="xlsx").parse_into_text(
+            content
+        )
+
+        chunks = [chunk.content.strip() for chunk in document.chunks]
+        self.assertEqual(len(chunks), 2)
+        self.assertEqual(chunks[0], "A: Name,B: City")
+        self.assertEqual(chunks[1], "A: Alice,B: Shenzhen")
+
+    def test_single_row_xlsx_is_not_consumed_in_header_mode(self):
+        content = self._workbook_bytes([["Name", "Age", "City"]])
+
+        document = ExcelParser(
+            file_name="single.xlsx",
+            file_type="xlsx",
+            xlsx_first_row_as_header=True,
+        ).parse_into_text(content)
+
+        self.assertEqual(
+            [chunk.content.strip() for chunk in document.chunks],
+            ["A: Name,B: Age,C: City"],
+        )
+
+    def test_header_mode_generates_stable_labels_for_duplicate_and_empty_cells(self):
+        content = self._workbook_bytes(
+            [
+                ["Name", "Name", None],
+                ["Alice", "Alias", "Shenzhen"],
+            ]
+        )
+
+        document = ExcelParser(
+            file_name="people.xlsx",
+            file_type="xlsx",
+            xlsx_first_row_as_header="yes",
+        ).parse_into_text(content)
+
+        self.assertEqual(
+            [chunk.content.strip() for chunk in document.chunks],
+            ["Name: Alice,Name_2: Alias,C: Shenzhen"],
+        )
+
+    def test_xlsx_explicit_false_override_keeps_first_row_as_data(self):
+        content = self._workbook_bytes(
+            [
+                ["Name", "Age"],
+                ["Alice", 30],
+            ]
+        )
+
+        document = ExcelParser(
+            file_name="people.xlsx",
+            file_type="xlsx",
+            xlsx_first_row_as_header="false",
+        ).parse_into_text(content)
+
+        chunks = [chunk.content.strip() for chunk in document.chunks]
+        self.assertEqual(chunks[0], "A: Name,B: Age")
+        self.assertEqual(chunks[1], "A: Alice,B: 30")
+
+    def test_parse_phantom_shared_strings_workbook(self):
+        document = ExcelParser().parse_into_text(_xlsx_with_phantom_shared_strings())
+        self.assertIn("hello", document.content)
+        self.assertIn("42", document.content)
+        self.assertGreater(len(document.chunks), 0)
+
+    def test_parse_en_calcchain_shared_strings_case(self):
+        path = os.path.join(
+            os.path.dirname(__file__),
+            "..",
+            "..",
+            "testdata",
+            "rag_test",
+            "xlsx",
+            "en_calcchain.xlsx",
+        )
+        if not os.path.isfile(path):
+            self.skipTest("en_calcchain.xlsx fixture not available")
+        with open(path, "rb") as f:
+            document = ExcelParser().parse_into_text(f.read())
+        self.assertGreater(len(document.content), 0)
+        self.assertGreater(len(document.chunks), 0)
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/tests/test_html_parser.py

@@ -0,0 +1,71 @@
+import unittest
+from pathlib import Path
+
+from docreader.parser.html_parser import HTMLParser
+from docreader.parser.registry import BUILTIN_ENGINE, registry
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+
+
+class HTMLParserTest(unittest.TestCase):
+    def test_parse_static_html_fixture_into_markdown(self):
+        content = (REPO_ROOT / "docreader" / "testdata" / "test.html").read_bytes()
+
+        document = HTMLParser(file_name="test.html", file_type="html").parse(content)
+
+        self.assertIn("# 测试 HTML 文档", document.content)
+        self.assertIn("[测试链接](https://example.com)", document.content)
+        self.assertIn("| 表头1 | 表头2 |", document.content)
+        self.assertIn("内容4", document.content)
+
+    def test_parse_empty_html_returns_empty_document(self):
+        document = HTMLParser(file_name="empty.htm", file_type="htm").parse(b"")
+
+        self.assertEqual("", document.content)
+
+    def test_parse_honors_declared_windows_1252_charset(self):
+        html = (
+            '<meta charset="windows-1252">'
+            "<h1>R\u00e9sum\u00e9 \u201cquoted\u201d \u20ac</h1>"
+        ).encode("windows-1252")
+
+        document = HTMLParser(file_name="legacy.html", file_type="html").parse(html)
+
+        self.assertIn("# R\u00e9sum\u00e9 \u201cquoted\u201d \u20ac", document.content)
+
+    def test_parse_honors_utf16_bom(self):
+        html = "<html><body><h1>UTF-16 title</h1></body></html>".encode("utf-16")
+
+        document = HTMLParser(file_name="utf16.html", file_type="html").parse(html)
+
+        self.assertIn("# UTF-16 title", document.content)
+        self.assertNotIn("\x00", document.content)
+
+    def test_parse_script_only_html_returns_empty_document(self):
+        html = (
+            b"<html><body><script>alert('x')</script><style>p{}</style></body></html>"
+        )
+
+        document = HTMLParser(file_name="script.html", file_type="html").parse(html)
+
+        self.assertEqual("", document.content)
+
+    def test_parse_preserves_relative_and_fragment_links(self):
+        html = b'<a href="./guide.html">Guide</a><a href="#section">Section</a>'
+
+        document = HTMLParser(file_name="links.html", file_type="html").parse(html)
+
+        self.assertIn("[Guide](./guide.html)", document.content)
+        self.assertIn("[Section](#section)", document.content)
+
+    def test_registry_routes_html_extensions_to_html_parser(self):
+        self.assertIs(registry.get_parser_class("", "html"), HTMLParser)
+        self.assertIs(registry.get_parser_class("", "htm"), HTMLParser)
+
+        builtin_types = registry._engines[BUILTIN_ENGINE]
+        self.assertIn("html", builtin_types)
+        self.assertIn("htm", builtin_types)
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/tests/test_markdown_table_util.py

@@ -0,0 +1,109 @@
+import io
+import unittest
+from pathlib import Path
+
+from markitdown import MarkItDown
+
+from docreader.parser.markdown_parser import MarkdownTableUtil
+
+
+class TestMarkdownTableUtil(unittest.TestCase):
+    def test_preserves_empty_cells(self):
+        """Interior empty cells must not be dropped during formatting."""
+        raw = "| a |  | c |\n| --- | --- | --- |\n| 1 | 2 | 3 |"
+        formatted = MarkdownTableUtil().format_table(raw)
+        self.assertIn("| a |  | c |", formatted)
+        self.assertEqual(formatted.count("|"), raw.count("|"))
+
+    def test_format_nonempty_table(self):
+        raw = "|Name|Age|\n|---|---|\n|John|30|"
+        formatted = MarkdownTableUtil().format_table(raw)
+        self.assertIn("| Name | Age |", formatted)
+        self.assertIn("| --- | --- |", formatted)
+        self.assertIn("| John | 30 |", formatted)
+
+    def test_normalize_markitdown_en_tables(self):
+        docx = (
+            Path(__file__).resolve().parents[2]
+            / "testdata"
+            / "rag_test"
+            / "docx"
+            / "en_tables.docx"
+        )
+        if not docx.is_file():
+            docx = Path(__file__).resolve().parents[2].parent / "testdata/rag_test/docx/en_tables.docx"
+        if not docx.is_file():
+            self.skipTest("en_tables.docx fixture not available")
+        raw = MarkItDown().convert(io.BytesIO(docx.read_bytes()), file_extension=".docx").text_content
+        normalized = MarkdownTableUtil().format_table(raw)
+
+        self.assertNotIn("|  |  |  |  |", normalized)
+        self.assertIn("| Name | Game | Fame | Blame |", normalized)
+        idx_name = normalized.index("| Name | Game | Fame | Blame |")
+        idx_sep = normalized.index("| --- | --- | --- | --- |", idx_name)
+        self.assertLess(idx_name, idx_sep)
+        self.assertIn("| Lebron James | Basketball |", normalized)
+
+        # Headerless 2-row tables: delimiter inserted so GFM renderers show a table
+        self.assertIn(
+            "| Sinple | Table |\n| --- | --- |\n| Without | Header |", normalized
+        )
+        self.assertIn(
+            "| Simple  Multiparagraph | Table  Full |\n| --- | --- |\n"
+            "| Of  Paragraphs | In each  Cell. |",
+            normalized,
+        )
+
+    def test_malformed_unclosed_row_is_passthrough(self):
+        """An unclosed ``|``-prefixed row must not hang the process.
+
+        Regression for Tencent/WeKnora#2768: the old line_pattern explored an
+        exponential number of backtracking paths on such rows and stalled the
+        single-threaded docreader event loop.
+        """
+        line = "| " + " | ".join(f"col{i}" for i in range(100)) + "  unclosed"
+        formatted = MarkdownTableUtil().format_table(line)
+        self.assertEqual(formatted, line)
+
+    def test_long_valid_table_formats(self):
+        """A long, well-formed table still formats in linear time."""
+        header = "| " + " | ".join(f"c{i}" for i in range(200)) + " |"
+        sep = "| " + " | ".join("---" for _ in range(200)) + " |"
+        row = "| " + " | ".join(f"v{i}" for i in range(200)) + " |"
+        formatted = MarkdownTableUtil().format_table("\n".join([header, sep, row]))
+        self.assertIn("| c0 | c1 |", formatted)
+        self.assertIn("| c199 |", formatted)
+        self.assertIn("| --- |", formatted)
+        self.assertIn("| v0 | v1 |", formatted)
+        self.assertIn("| v199 |", formatted)
+
+    def test_lone_pipe_is_passthrough(self):
+        """A single-pipe line is not a table row and must not be deleted."""
+        self.assertEqual(MarkdownTableUtil().format_table("|"), "|")
+        self.assertEqual(MarkdownTableUtil().format_table("  |  "), "  |  ")
+
+    def test_lone_pipe_above_table_is_kept(self):
+        raw = "|\n| a | b |\n| --- | --- |\n| 1 | 2 |"
+        formatted = MarkdownTableUtil().format_table(raw)
+        self.assertEqual(formatted.split("\n")[0], "|")
+        self.assertIn("| a | b |", formatted)
+        self.assertIn("| 1 | 2 |", formatted)
+
+    def test_alignment_colons_preserved(self):
+        raw = "| a | b | c |\n| :---------- | -------: | :------: |\n| 1 | 2 | 3 |"
+        formatted = MarkdownTableUtil().format_table(raw)
+        self.assertIn("| :--- | ---: | :---: |", formatted)
+
+    def test_crlf_table_formats_without_mixed_endings(self):
+        raw = "# Title\r\n\r\n|Name|Age|\r\n|---|---|\r\n|John|30|\r\n\r\nparagraph\r\n"
+        formatted = MarkdownTableUtil().format_table(raw)
+        self.assertIn("| Name | Age |", formatted)
+        self.assertIn("| --- | --- |", formatted)
+        self.assertIn("| John | 30 |", formatted)
+        self.assertIn("paragraph", formatted)
+        self.assertNotIn("\r", formatted)
+        self.assertTrue(formatted.endswith("\n"))
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/tests/test_mhtml_parser.py

@@ -0,0 +1,323 @@
+import base64
+import email
+import json
+import unittest
+from email.header import Header
+from email.message import EmailMessage
+from pathlib import Path
+
+from docreader.parser.mhtml_parser import MHTMLParser, _header_str
+from docreader.parser.registry import registry
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+
+
+def _minimal_mhtml_bytes() -> bytes:
+    root = EmailMessage()
+    root["Subject"] = "Tiny MHTML"
+    root.make_related()
+
+    main = EmailMessage()
+    main.set_content(
+        "<html><body><h1>Main Article</h1>"
+        "<p>Hello MHTML world.</p>"
+        '<p><a href="chapter03.xhtml#sec2">Chapter 3</a> '
+        '<a href="#footnote1">note</a> '
+        '<a href="https://example.com">the site</a></p>'
+        '<img alt="tiny" src="cid:tiny-image">'
+        "<script>window.noise = true</script>"
+        "</body></html>",
+        subtype="html",
+    )
+    main["Content-Location"] = "https://example.com/article"
+    root.attach(main)
+
+    ad = EmailMessage()
+    ad.set_content(
+        "<html><body><h1>Advertisement</h1>"
+        "<p>Buy this unrelated thing.</p></body></html>",
+        subtype="html",
+    )
+    ad["Content-Location"] = "https://googleads.example/frame.html"
+    root.attach(ad)
+
+    image = EmailMessage()
+    image.set_content(
+        b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR",
+        maintype="image",
+        subtype="png",
+    )
+    image["Content-Location"] = "cid:tiny-image"
+    root.attach(image)
+
+    return root.as_bytes()
+
+
+def _mhtml_with_table_image_and_caption() -> bytes:
+    root = EmailMessage()
+    root["Subject"] = "MHTML with table image"
+    root.make_related()
+
+    main = EmailMessage()
+    main.set_content(
+        "<html><body>"
+        "<table>"
+        "<tr><th>体验方向</th><th>代表内容</th></tr>"
+        "<tr><td>赛季制建立</td><td>BP、Rank</td></tr>"
+        "</table>"
+        '<img src="cid:test-image" alt="图片" title="图片">'
+        "<p>高机动性身法与独特枪械反馈</p>"
+        "</body></html>",
+        subtype="html",
+    )
+    main["Content-Location"] = "https://example.com/article"
+    root.attach(main)
+
+    image = EmailMessage()
+    image.set_content(
+        b"GIF89a\x01\x00\x01\x00\x80\x00\x00\x00\x00\x00\xff\xff\xff,\x00\x00"
+        b"\x00\x00\x01\x00\x01\x00\x00\x02\x02D\x01\x00;",
+        maintype="image",
+        subtype="gif",
+    )
+    image["Content-ID"] = "<test-image>"
+    root.attach(image)
+
+    return root.as_bytes()
+
+
+def _mhtml_with_8bit_utf8_headers() -> bytes:
+    """Browser-style MHTML: raw UTF-8 bytes in MIME headers, not RFC 2047."""
+    html_loc = "https://example.com/测试页面.html".encode("utf-8")
+    ad_loc = "https://googleads.example/广告.html".encode("utf-8")
+    image_loc = "https://example.com/图片.png".encode("utf-8")
+    content_id = "<图片@local>".encode("utf-8")
+    attachment_id = "图片".encode("utf-8")
+    main_html = (
+        "<html><body><h1>中文主文</h1>"
+        '<img alt="绝对" src="https://example.com/图片.png">'
+        '<img alt="相对" src="图片.png">'
+        '<img alt="cid" src="cid:图片@local">'
+        "</body></html>"
+    ).encode("utf-8")
+    ad_html = "<html><body><h1>广告正文</h1></body></html>".encode("utf-8")
+    png = b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR"
+    return b"".join(
+        [
+            b"MIME-Version: 1.0\r\n",
+            b'Content-Type: multipart/related; boundary="----=_Next"\r\n',
+            b"\r\n",
+            b"------=_Next\r\n",
+            b'Content-Type: text/html; charset="utf-8"\r\n',
+            b"Content-Transfer-Encoding: 8bit\r\n",
+            b"Content-Location: ",
+            html_loc,
+            b"\r\n\r\n",
+            main_html,
+            b"\r\n",
+            b"------=_Next\r\n",
+            b'Content-Type: text/html; charset="utf-8"\r\n',
+            b"Content-Transfer-Encoding: 8bit\r\n",
+            b"Content-Location: ",
+            ad_loc,
+            b"\r\n\r\n",
+            ad_html,
+            b"\r\n",
+            b"------=_Next\r\n",
+            b"Content-Type: image/png\r\n",
+            b"Content-Transfer-Encoding: base64\r\n",
+            b"Content-Location: ",
+            image_loc,
+            b"\r\nContent-ID: ",
+            content_id,
+            b"\r\nX-Attachment-Id: ",
+            attachment_id,
+            b"\r\n\r\n",
+            base64.b64encode(png),
+            b"\r\n",
+            b"------=_Next--\r\n",
+        ]
+    )
+
+
+class MHTMLParserTest(unittest.TestCase):
+    def test_parse_selects_main_html_and_filters_noise(self):
+        document = MHTMLParser(
+            file_name="article.mhtml", file_type="mhtml"
+        ).parse_into_text(_minimal_mhtml_bytes())
+
+        self.assertIn("Main Article", document.content)
+        self.assertIn("Hello MHTML world", document.content)
+        self.assertNotIn("Advertisement", document.content)
+        self.assertNotIn("window.noise", document.content)
+        self.assertEqual(document.metadata["source_format"], "mhtml")
+
+    def test_internal_links_are_unwrapped_but_external_links_remain(self):
+        document = MHTMLParser(
+            file_name="article.mhtml", file_type="mhtml"
+        ).parse_into_text(_minimal_mhtml_bytes())
+
+        self.assertIn("Chapter 3", document.content)
+        self.assertIn("note", document.content)
+        self.assertNotIn("chapter03.xhtml#sec2", document.content)
+        self.assertNotIn("#footnote1", document.content)
+        self.assertIn("[the site](https://example.com)", document.content)
+
+    def test_image_extraction_toggle(self):
+        with_images = MHTMLParser(
+            file_name="article.mhtml", file_type="mhtml", extract_images=True
+        ).parse_into_text(_minimal_mhtml_bytes())
+        without_images = MHTMLParser(
+            file_name="article.mhtml", file_type="mhtml", extract_images=False
+        ).parse_into_text(_minimal_mhtml_bytes())
+
+        self.assertEqual(len(with_images.images), 1)
+        image_ref = next(iter(with_images.images))
+        self.assertTrue(image_ref.startswith("images/"))
+        self.assertIn(image_ref, with_images.content)
+        self.assertNotIn("cid:tiny-image", with_images.content)
+        self.assertEqual(without_images.images, {})
+
+    def test_table_image_and_caption_keep_markdown_block_boundaries(self):
+        document = MHTMLParser(
+            file_name="article.mhtml", file_type="mhtml"
+        ).parse_into_text(_mhtml_with_table_image_and_caption())
+
+        self.assertEqual(len(document.images), 1)
+        image_ref = next(iter(document.images))
+        self.assertIn(f'![图片]({image_ref} "图片")', document.content)
+        self.assertIn(
+            f"| 赛季制建立 | BP、Rank |\n\n![图片]({image_ref} \"图片\")",
+            document.content,
+        )
+        self.assertIn(
+            f'![图片]({image_ref} "图片")\n\n高机动性身法与独特枪械反馈',
+            document.content,
+        )
+
+    def test_html_to_markdown_preserves_indentation_and_code_block_blanks(self):
+        markdown = MHTMLParser(
+            file_name="article.mhtml", file_type="mhtml"
+        )._html_to_markdown(
+            "<ul><li>parent<ul><li>child</li></ul></li></ul>"
+            "<blockquote><p>quoted</p></blockquote>"
+            "<pre><code>line1\n\n  indented\n</code></pre>"
+        )
+
+        self.assertIn("* parent\n  + child", markdown)
+        self.assertIn("\n\n> quoted\n\n", markdown)
+        self.assertIn("```\nline1\n\n  indented\n```", markdown)
+
+    def test_html_to_markdown_preserves_nested_list_indentation(self):
+        markdown = MHTMLParser(
+            file_name="article.mhtml", file_type="mhtml"
+        )._html_to_markdown("<ul><li>parent<ul><li>child</li></ul></li></ul>")
+
+        self.assertIn("* parent\n  + child", markdown)
+
+    def test_html_to_markdown_preserves_blockquote_boundaries(self):
+        markdown = MHTMLParser(
+            file_name="article.mhtml", file_type="mhtml"
+        )._html_to_markdown("<p>before</p><blockquote><p>quoted</p></blockquote><p>after</p>")
+
+        self.assertIn("before\n\n> quoted\n\nafter", markdown)
+
+    def test_html_to_markdown_preserves_fenced_code_blank_lines(self):
+        markdown = MHTMLParser(
+            file_name="article.mhtml", file_type="mhtml"
+        )._html_to_markdown("<pre><code>line1\n\n\nline2\n</code></pre>")
+
+        self.assertIn("```\nline1\n\n\nline2\n```", markdown)
+
+    def test_html_to_markdown_collapses_excess_blank_lines_outside_code(self):
+        markdown = MHTMLParser._normalize_markdown("alpha\n\n  \n\t\n\nbeta")
+
+        self.assertEqual(markdown, "alpha\n\nbeta")
+        self.assertNotIn("\n\n\n", markdown)
+
+    def test_html_to_markdown_preserves_hard_break_spaces(self):
+        markdown = MHTMLParser(
+            file_name="article.mhtml", file_type="mhtml"
+        )._html_to_markdown("<p>alpha<br>beta</p>")
+
+        self.assertEqual(markdown, "alpha  \nbeta")
+
+    def test_normalize_markdown_preserves_two_space_hard_break(self):
+        markdown = MHTMLParser._normalize_markdown("alpha  \nbeta")
+
+        self.assertEqual(markdown, "alpha  \nbeta")
+
+    def test_html_to_markdown_normalizes_crlf(self):
+        markdown = MHTMLParser._normalize_markdown("alpha\r\n\r\nbeta\rgamma")
+
+        self.assertEqual(markdown, "alpha\n\nbeta\ngamma")
+
+    def test_html_to_markdown_does_not_strip_leading_indentation_at_document_start(self):
+        markdown = MHTMLParser._normalize_markdown("  indented start\n")
+
+        self.assertEqual(markdown, "  indented start")
+
+    def test_mhtml_shared_contract_fixture(self):
+        fixture = REPO_ROOT / "testdata" / "mhtml" / "titled-image.mhtml"
+        contract_path = REPO_ROOT / "testdata" / "mhtml" / "titled-image-contract.json"
+        contract = json.loads(contract_path.read_text(encoding="utf-8"))
+
+        document = MHTMLParser(
+            file_name="titled-image.mhtml", file_type="mhtml"
+        ).parse_into_text(fixture.read_bytes())
+
+        self.assertEqual(document.content, contract["markdown_content"])
+        self.assertEqual(len(document.images), 1)
+        image_contract = contract["images"][0]
+        self.assertIn(image_contract["original_ref"], document.images)
+        self.assertEqual(
+            base64.b64decode(document.images[image_contract["original_ref"]]),
+            base64.b64decode(image_contract["image_data_base64"]),
+        )
+        self.assertIn(
+            '| 赛季制建立 | BP、Rank |\n\n![图片](images/第 1 页 (测试).gif "阶段 1) 图片")',
+            document.content,
+        )
+        self.assertIn(
+            '![图片](images/第 1 页 (测试).gif "阶段 1) 图片")\n\n高机动性身法与独特枪械反馈',
+            document.content,
+        )
+        self.assertNotIn("\n\n\n", document.content.split("```", 1)[0])
+        self.assertIn("```\nline1\n\n\nline2\n```", document.content)
+        self.assertIn("* parent\n  + child", document.content)
+        self.assertIn("alpha  \nbeta", document.content)
+
+    def test_header_str_recovers_unknown_8bit_utf8(self):
+        raw = (
+            b"Content-Type: text/plain\r\n"
+            b"Content-Location: https://example.com/"
+            + "图片.png".encode("utf-8")
+            + b"\r\n\r\nbody\r\n"
+        )
+        value = email.message_from_bytes(raw).get("Content-Location")
+        self.assertIsInstance(value, Header)
+        self.assertEqual(_header_str(value), "https://example.com/图片.png")
+        self.assertEqual(_header_str(None), "")
+        self.assertEqual(_header_str("ascii-location"), "ascii-location")
+        self.assertEqual(_header_str("已是unicode"), "已是unicode")
+
+    def test_parse_8bit_utf8_headers_rewrites_images_without_crashing(self):
+        document = MHTMLParser(
+            file_name="article.mhtml", file_type="mhtml"
+        ).parse_into_text(_mhtml_with_8bit_utf8_headers())
+
+        self.assertIn("中文主文", document.content)
+        self.assertNotIn("广告正文", document.content)
+        self.assertIn("images/图片.png", document.images)
+        self.assertIn("images/图片.png", document.content)
+        self.assertEqual(document.content.count("images/图片.png"), 3)
+        self.assertNotIn("cid:图片@local", document.content)
+        self.assertNotIn("\ufffd", document.content)
+        self.assertTrue(all("\ufffd" not in key for key in document.images))
+
+    def test_registry_resolves_mhtml(self):
+        self.assertIs(registry.get_parser_class("", "mhtml"), MHTMLParser)
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/tests/test_opendataloader_parser.py

@@ -0,0 +1,133 @@
+"""Unit tests for OpenDataLoader parser helpers (no JVM required)."""
+
+import os
+import sys
+import tempfile
+import unittest
+from unittest import mock
+
+from docreader.parser.opendataloader_parser import (
+    OpenDataLoaderParser,
+    _collect_images_under_output,
+    _find_markdown_file,
+    _normalize_odl_image_url,
+    _ping_hybrid,
+    _run_convert,
+    _rewrite_markdown_image_refs,
+    opendataloader_available,
+)
+
+
+class OpenDataLoaderHelpersTest(unittest.TestCase):
+    def test_hybrid_health_probe_blocks_private_url_before_request(self):
+        with mock.patch(
+            "docreader.parser.opendataloader_parser.is_ssrf_safe_url",
+            return_value=(False, "restricted test address"),
+        ), mock.patch("urllib.request.build_opener") as build_opener:
+            ok, msg = _ping_hybrid("http://127.0.0.1:8080", retries=1)
+
+        self.assertFalse(ok)
+        self.assertIn("SSRF", msg)
+        build_opener.assert_not_called()
+
+    def test_convert_blocks_private_hybrid_url_at_final_sink(self):
+        fake_module = mock.Mock()
+        with mock.patch.dict(sys.modules, {"opendataloader_pdf": fake_module}):
+            with self.assertRaisesRegex(RuntimeError, "SSRF"):
+                _run_convert(
+                    "/tmp/input.pdf",
+                    "/tmp/output",
+                    "/tmp/output/images",
+                    overrides={
+                        "odl_hybrid": "docling-fast",
+                        "odl_hybrid_url": "http://169.254.169.254/latest/meta-data",
+                    },
+                )
+
+        fake_module.convert.assert_not_called()
+
+    def test_find_markdown_prefers_stem_match(self):
+        with tempfile.TemporaryDirectory() as d:
+            other = os.path.join(d, "other.md")
+            target = os.path.join(d, "paper.md")
+            with open(other, "w") as f:
+                f.write("x")
+            with open(target, "w") as f:
+                f.write("# Title")
+            self.assertEqual(_find_markdown_file(d, "paper"), target)
+
+    def test_collect_and_rewrite_images(self):
+        with tempfile.TemporaryDirectory() as d:
+            img_dir = os.path.join(d, "images")
+            os.makedirs(img_dir)
+            png = os.path.join(img_dir, "fig1.png")
+            with open(png, "wb") as f:
+                f.write(b"\x89PNG\r\n\x1a\n")
+            images = _collect_images_under_output(d)
+            self.assertIn("images/fig1.png", images)
+            md = "See ![fig](images/fig1.png) and ![alt](./fig1.png)."
+            out = _rewrite_markdown_image_refs(md, images)
+            self.assertIn("![fig](images/fig1.png)", out)
+            self.assertIn("![alt](images/fig1.png)", out)
+
+    def test_rewrite_odl_angle_bracket_and_entity_urls(self):
+        images = {"images/imageFile1.png": "e30="}
+        for md_in in (
+            "![image 1](<images/imageFile1.png>)",
+            "![image 1](&lt;images/imageFile1.png&gt;)",
+        ):
+            out = _rewrite_markdown_image_refs(md_in, images)
+            self.assertEqual("![image 1](images/imageFile1.png)", out)
+
+    def test_normalize_odl_image_url(self):
+        self.assertEqual(
+            _normalize_odl_image_url("&lt;images/imageFile2.png&gt;"),
+            "images/imageFile2.png",
+        )
+        self.assertEqual(
+            _normalize_odl_image_url("<images/imageFile2.png>"),
+            "images/imageFile2.png",
+        )
+
+    def test_rewrite_skips_data_uris(self):
+        md = "![x](data:image/png;base64,abc)"
+        self.assertEqual(_rewrite_markdown_image_refs(md, {"images/a.png": "e30="}), md)
+
+
+class OpenDataLoaderParserTest(unittest.TestCase):
+    @mock.patch("docreader.parser.opendataloader_parser.opendataloader_available")
+    @mock.patch("docreader.parser.opendataloader_parser._run_convert")
+    def test_parse_reads_markdown_and_images(self, mock_convert, mock_avail):
+        mock_avail.return_value = (True, "")
+
+        def fake_convert(pdf_path, output_dir, image_dir, overrides=None):
+            stem = os.path.splitext(os.path.basename(pdf_path))[0]
+            md_path = os.path.join(output_dir, f"{stem}.md")
+            with open(md_path, "w") as f:
+                f.write("# Hello\n\n![pic](images/pic.png)\n")
+            os.makedirs(image_dir, exist_ok=True)
+            with open(os.path.join(image_dir, "pic.png"), "wb") as f:
+                f.write(b"png")
+
+        mock_convert.side_effect = fake_convert
+
+        parser = OpenDataLoaderParser(file_name="doc.pdf", file_type="pdf")
+        doc = parser.parse_into_text(b"%PDF-1.4 fake")
+        self.assertIn("# Hello", doc.content)
+        self.assertIn("images/pic.png", doc.content)
+        self.assertIn("images/pic.png", doc.images)
+        self.assertEqual(doc.metadata.get("parser_engine"), "opendataloader")
+
+    @mock.patch("docreader.parser.opendataloader_parser.shutil.which", return_value=None)
+    def test_availability_requires_java(self, _which):
+        with mock.patch(
+            "docreader.parser.opendataloader_parser._package_available",
+            return_value=(True, ""),
+        ):
+            ok, msg = opendataloader_available()
+        self.assertFalse(ok)
+        self.assertIn("Java", msg)
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/tests/test_parser_concurrency.py

@@ -0,0 +1,88 @@
+import base64
+import io
+import threading
+import time
+import unittest
+import uuid
+
+from PIL import Image
+
+from docreader.parser.concurrency import parser_worker_limit
+from docreader.parser.pdf_parser import PDFScannedParser, _normalize_image_quality
+
+
+class ParserConcurrencyTest(unittest.TestCase):
+    def test_parser_worker_limit_serializes_work(self):
+        limiter_name = f"test-{uuid.uuid4()}"
+        active_workers = 0
+        max_active_workers = 0
+        state_lock = threading.Lock()
+        start = threading.Barrier(3)
+
+        def worker():
+            nonlocal active_workers, max_active_workers
+            start.wait()
+            with parser_worker_limit(limiter_name, 1):
+                with state_lock:
+                    active_workers += 1
+                    max_active_workers = max(max_active_workers, active_workers)
+                time.sleep(0.02)
+                with state_lock:
+                    active_workers -= 1
+
+        threads = [threading.Thread(target=worker) for _ in range(2)]
+        for thread in threads:
+            thread.start()
+
+        start.wait()
+        for thread in threads:
+            thread.join()
+
+        self.assertEqual(max_active_workers, 1)
+
+    def test_scanned_pdf_parser_outputs_jpeg_images(self):
+        pdf_bytes = io.BytesIO()
+        pages = [
+            Image.new("RGB", (64, 64), "white"),
+            Image.new("RGB", (64, 64), "black"),
+        ]
+        pages[0].save(
+            pdf_bytes,
+            format="PDF",
+            save_all=True,
+            append_images=pages[1:],
+        )
+
+        document = PDFScannedParser(file_name="scan.pdf").parse_into_text(
+            pdf_bytes.getvalue()
+        )
+
+        image_ref = "images/scan_page_1.jpg"
+        self.assertIn(f"![scan_page_1.jpg]({image_ref})", document.content)
+        self.assertIn(image_ref, document.images)
+        self.assertEqual(document.metadata["image_source_type"], "scanned_pdf")
+        self.assertEqual(document.metadata["page_count"], 2)
+        self.assertEqual(len(document.images), 2)
+        self.assertIn("images/scan_page_2.jpg", document.images)
+        image_bytes = base64.b64decode(document.images[image_ref])
+        self.assertTrue(image_bytes.startswith(b"\xff\xd8"))
+
+    def test_scanned_pdf_parser_logs_malformed_pdf_without_format_error(self):
+        parser = PDFScannedParser(file_name="broken.pdf")
+
+        with self.assertLogs("docreader.parser.pdf_parser", level="ERROR") as logs:
+            with self.assertRaises(Exception):
+                parser.parse_into_text(b"not a pdf")
+
+        self.assertTrue(
+            any("PDFScannedParser failed to parse PDF:" in line for line in logs.output)
+        )
+
+    def test_normalize_image_quality_bounds_jpeg_quality(self):
+        self.assertEqual(_normalize_image_quality(-1), 1)
+        self.assertEqual(_normalize_image_quality(90), 90)
+        self.assertEqual(_normalize_image_quality(120), 95)
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/tests/test_parser_routing.py

@@ -0,0 +1,105 @@
+import unittest
+
+from docreader.models.document import Document
+from docreader.parser.base_parser import BaseParser
+from docreader.parser.parser import Parser, detect_effective_file_type
+from docreader.parser.markitdown_parser import MarkitdownParser
+from docreader.parser.pdf_parser import PDFParser
+from docreader.parser.registry import BUILTIN_ENGINE, ParserEngineRegistry, registry
+from docreader.parser.xmind_parser import XMindParser
+
+
+OLE_MAGIC = b"\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1"
+
+
+class _RecordingParser(BaseParser):
+    last_file_type = ""
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        type(self).last_file_type = self.file_type
+
+    def parse_into_text(self, content: bytes) -> Document:
+        return Document(content="parsed")
+
+
+class _RecordingRegistry:
+    def __init__(self):
+        self.requested_file_type = ""
+
+    def get_parser_class(self, engine: str, file_type: str):
+        self.requested_file_type = file_type
+        return _RecordingParser
+
+
+class ParserRoutingTest(unittest.TestCase):
+    def test_builtin_registry_routes_xmind_to_xmind_parser(self):
+        parser_class = registry.get_parser_class(BUILTIN_ENGINE, "xmind")
+
+        self.assertIs(parser_class, XMindParser)
+
+    def test_legacy_doc_payload_renamed_to_docx_uses_doc_parser(self):
+        registry = _RecordingRegistry()
+        parser = Parser()
+        parser.registry = registry
+
+        result = parser.parse_file(
+            "legacy.docx",
+            "docx",
+            OLE_MAGIC + b"legacy-word-payload",
+        )
+
+        self.assertEqual("parsed", result.content)
+        self.assertEqual("doc", registry.requested_file_type)
+        self.assertEqual("doc", _RecordingParser.last_file_type)
+
+    def test_real_docx_payload_keeps_docx_parser_route(self):
+        self.assertEqual(
+            "docx",
+            detect_effective_file_type(".DOCX", b"PK\x03\x04ooxml-payload"),
+        )
+
+    def test_unrelated_ole_type_is_not_reclassified(self):
+        self.assertEqual("xls", detect_effective_file_type("xls", OLE_MAGIC))
+
+    def test_empty_engine_routes_pptx_to_markitdown(self):
+        self.assertIs(registry.get_parser_class("", "pptx"), MarkitdownParser)
+        self.assertIs(registry.get_parser_class("", "ppt"), MarkitdownParser)
+        self.assertIs(registry.get_parser_class("", ".PPTX"), MarkitdownParser)
+
+    def test_builtin_engine_still_parses_pptx(self):
+        self.assertIs(registry.get_parser_class(BUILTIN_ENGINE, "pptx"), MarkitdownParser)
+
+    def test_type_default_when_builtin_lacks_pptx(self):
+        isolated = ParserEngineRegistry()
+        isolated.register(BUILTIN_ENGINE, {"pdf": PDFParser})
+        isolated.register(
+            "markitdown",
+            {"pptx": MarkitdownParser, "ppt": MarkitdownParser, "csv": MarkitdownParser},
+        )
+
+        self.assertIs(isolated.get_parser_class("", "pptx"), MarkitdownParser)
+        self.assertIs(isolated.get_parser_class(BUILTIN_ENGINE, "pptx"), MarkitdownParser)
+        self.assertIs(isolated.get_parser_class("", "csv"), MarkitdownParser)
+        self.assertIs(isolated.get_parser_class(BUILTIN_ENGINE, "csv"), MarkitdownParser)
+        self.assertIs(isolated.get_parser_class(BUILTIN_ENGINE, "pdf"), PDFParser)
+        self.assertIs(isolated.get_parser_class("", "pdf"), PDFParser)
+
+        with self.assertRaises(ValueError) as ctx:
+            isolated.get_parser_class("", "xyz")
+        self.assertIn("Unsupported file type: xyz", str(ctx.exception))
+
+    def test_actionable_error_lists_supporting_engines(self):
+        isolated = ParserEngineRegistry()
+        isolated.register(BUILTIN_ENGINE, {"pdf": PDFParser})
+        isolated.register("opendataloader", {"xmind": XMindParser})
+
+        with self.assertRaises(ValueError) as ctx:
+            isolated.get_parser_class(BUILTIN_ENGINE, "xmind")
+        message = str(ctx.exception)
+        self.assertIn("Configure one of:", message)
+        self.assertIn("opendataloader", message)
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/tests/test_pdf_embedded_images.py

@@ -0,0 +1,258 @@
+"""Embedded-figure extraction from native PDF pages.
+
+Focus: images whose visible content lives in a /SMask (soft transparency
+mask) while the base RGB plane is black. ``get_bitmap()`` alone decodes only
+the base plane, producing an all-black JPEG; the decode path must render the
+mask in and composite over white. Reproduces the all-black figures extracted
+from a production PDF exported by a plotting tool.
+"""
+
+import base64
+import gc
+import io
+import unittest
+from unittest.mock import patch
+
+import pypdfium2 as pdfium
+import pypdfium2.raw as pdfium_raw
+from PIL import Image
+
+from docreader.parser.pdf_parser import (
+    _decode_embedded_image_pil,
+    _extract_embedded_images,
+)
+
+PAGE_W, PAGE_H = 612, 792
+IMG_W, IMG_H = 120, 90
+
+
+def _pdf_from_objects(objects: list) -> bytes:
+    out = bytearray(b"%PDF-1.7\n")
+    offsets = []
+    for i, body in enumerate(objects, start=1):
+        offsets.append(len(out))
+        out += b"%d 0 obj\n" % i + body + b"\nendobj\n"
+    xref_pos = len(out)
+    out += b"xref\n0 %d\n" % (len(objects) + 1)
+    out += b"0000000000 65535 f \n"
+    for off in offsets:
+        out += b"%010d 00000 n \n" % off
+    out += b"trailer\n<< /Size %d /Root 1 0 R >>\nstartxref\n%d\n%%%%EOF\n" % (
+        len(objects) + 1,
+        xref_pos,
+    )
+    return bytes(out)
+
+
+def _page_objects(image_obj: bytes, extra_objs: list | None = None) -> list:
+    contents = b"q 480 0 0 360 66 216 cm /Im0 Do Q\n"
+    objects = [
+        b"<< /Type /Catalog /Pages 2 0 R >>",
+        b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
+        (
+            b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 %d %d] "
+            b"/Resources << /XObject << /Im0 5 0 R >> >> /Contents 4 0 R >>"
+            % (PAGE_W, PAGE_H)
+        ),
+        b"<< /Length %d >>\nstream\n" % len(contents) + contents + b"endstream",
+        image_obj,
+    ]
+    if extra_objs:
+        objects.extend(extra_objs)
+    return objects
+
+
+def _circle_mask() -> bytes:
+    cx, cy, radius = IMG_W // 2, IMG_H // 2, 30
+    mask = bytearray(IMG_W * IMG_H)
+    for y in range(IMG_H):
+        for x in range(IMG_W):
+            if (x - cx) ** 2 + (y - cy) ** 2 <= radius * radius:
+                mask[y * IMG_W + x] = 255
+    return bytes(mask)
+
+
+def _build_smask_pdf(base_rgb=(0, 0, 0)) -> bytes:
+    """Hand-built PDF: one RGB image shaped by an SMask circle.
+
+    The SMask is opaque (255) inside a filled circle and transparent (0)
+    elsewhere. A decoder that drops the mask shows the raw base plane only.
+    """
+    r, g, b = base_rgb
+    base = bytes((r, g, b)) * (IMG_W * IMG_H)
+    mask = _circle_mask()
+    image = (
+        b"<< /Type /XObject /Subtype /Image /Width %d /Height %d "
+        b"/ColorSpace /DeviceRGB /BitsPerComponent 8 /SMask 6 0 R "
+        b"/Length %d >>\nstream\n" % (IMG_W, IMG_H, len(base))
+        + base
+        + b"\nendstream"
+    )
+    mask_obj = (
+        b"<< /Type /XObject /Subtype /Image /Width %d /Height %d "
+        b"/ColorSpace /DeviceGray /BitsPerComponent 8 /Length %d >>\nstream\n"
+        % (IMG_W, IMG_H, len(mask))
+        + mask
+        + b"\nendstream"
+    )
+    return _pdf_from_objects(_page_objects(image, [mask_obj]))
+
+
+def _build_opaque_gray_pdf(level: int = 128) -> bytes:
+    data = bytes((level,)) * (IMG_W * IMG_H)
+    image = (
+        b"<< /Type /XObject /Subtype /Image /Width %d /Height %d "
+        b"/ColorSpace /DeviceGray /BitsPerComponent 8 /Length %d >>\nstream\n"
+        % (IMG_W, IMG_H, len(data))
+        + data
+        + b"\nendstream"
+    )
+    return _pdf_from_objects(_page_objects(image))
+
+
+def _first_image_object(pdf):
+    for page in pdf:
+        for obj in page.get_objects():
+            if obj.type == pdfium_raw.FPDF_PAGEOBJ_IMAGE:
+                return obj
+    raise AssertionError("test PDF has no image object")
+
+
+def _raw_decode(obj):
+    bitmap = obj.get_bitmap()
+    try:
+        pil = bitmap.to_pil()
+        if pil.mode in ("RGB", "L"):
+            return pil.copy()
+        return pil.convert("RGB")
+    finally:
+        bitmap.close()
+
+
+class DecodeEmbeddedImageTest(unittest.TestCase):
+    def test_smask_image_composites_over_white(self):
+        pdf = pdfium.PdfDocument(_build_smask_pdf())
+        try:
+            pil = _decode_embedded_image_pil(_first_image_object(pdf))
+        finally:
+            pdf.close()
+        self.assertEqual(pil.mode, "RGB")
+        # Transparent corners become white page background, not black.
+        self.assertGreater(pil.getpixel((2, 2))[0], 200)
+        self.assertGreater(pil.getpixel((IMG_W - 3, IMG_H - 3))[0], 200)
+        # Opaque circle center keeps the base-plane colour (black).
+        self.assertLess(pil.getpixel((IMG_W // 2, IMG_H // 2))[0], 40)
+
+    def test_colored_smask_keeps_base_color(self):
+        pdf = pdfium.PdfDocument(_build_smask_pdf(base_rgb=(200, 30, 40)))
+        try:
+            pil = _decode_embedded_image_pil(_first_image_object(pdf))
+        finally:
+            pdf.close()
+        self.assertEqual(pil.mode, "RGB")
+        self.assertGreater(pil.getpixel((2, 2))[0], 200)
+        center = pil.getpixel((IMG_W // 2, IMG_H // 2))
+        self.assertGreater(center[0], 150)
+        self.assertLess(center[1], 80)
+        self.assertLess(center[2], 80)
+
+    def test_opaque_image_matches_raw_pixels(self):
+        buf = io.BytesIO()
+        Image.new("RGB", (100, 80), (10, 200, 30)).save(buf, format="PDF")
+        pdf = pdfium.PdfDocument(buf.getvalue())
+        try:
+            obj = _first_image_object(pdf)
+            raw = _raw_decode(obj)
+            pil = _decode_embedded_image_pil(obj)
+        finally:
+            pdf.close()
+        self.assertEqual(pil.size, raw.size)
+        self.assertEqual(pil.mode, raw.mode)
+        self.assertEqual(pil.tobytes(), raw.tobytes())
+
+    def test_opaque_gray_fallback_keeps_l_mode(self):
+        # render=True typically returns BGRA even for DeviceGray; the raw
+        # fallback must keep L so grayscale hashes/JPEGs stay single-channel.
+        pdf = pdfium.PdfDocument(_build_opaque_gray_pdf(128))
+        try:
+            obj = _first_image_object(pdf)
+            raw = _raw_decode(obj)
+            self.assertEqual(raw.mode, "L")
+            original = obj.get_bitmap
+
+            def _boom(render=False, **kwargs):
+                if render:
+                    raise RuntimeError("render unavailable")
+                return original(render=render, **kwargs)
+
+            with patch.object(obj, "get_bitmap", side_effect=_boom):
+                with self.assertLogs("docreader.parser.pdf_parser", level="WARNING"):
+                    pil = _decode_embedded_image_pil(obj)
+        finally:
+            pdf.close()
+        self.assertEqual(pil.mode, "L")
+        self.assertEqual(pil.getpixel((IMG_W // 2, IMG_H // 2)), 128)
+        self.assertEqual(pil.tobytes(), raw.tobytes())
+
+    def test_decoded_image_survives_bitmap_gc(self):
+        buf = io.BytesIO()
+        Image.new("RGB", (100, 80), (10, 200, 30)).save(buf, format="PDF")
+        pdf = pdfium.PdfDocument(buf.getvalue())
+        try:
+            pil = _decode_embedded_image_pil(_first_image_object(pdf))
+        finally:
+            pdf.close()
+        gc.collect()
+        self.assertEqual(pil.getpixel((50, 40)), (10, 200, 30))
+
+    def test_render_failure_falls_back_to_raw(self):
+        buf = io.BytesIO()
+        Image.new("RGB", (100, 80), (10, 200, 30)).save(buf, format="PDF")
+        pdf = pdfium.PdfDocument(buf.getvalue())
+        try:
+            obj = _first_image_object(pdf)
+            raw = _raw_decode(obj)
+            original = obj.get_bitmap
+
+            def _boom(render=False, **kwargs):
+                if render:
+                    raise RuntimeError("render unavailable")
+                return original(render=render, **kwargs)
+
+            with patch.object(obj, "get_bitmap", side_effect=_boom):
+                with self.assertLogs(
+                    "docreader.parser.pdf_parser", level="WARNING"
+                ) as logs:
+                    pil = _decode_embedded_image_pil(obj)
+        finally:
+            pdf.close()
+        self.assertTrue(any("falling back to raw decode" in m for m in logs.output))
+        self.assertEqual(pil.mode, raw.mode)
+        self.assertEqual(pil.tobytes(), raw.tobytes())
+
+
+class ExtractEmbeddedImagesTest(unittest.TestCase):
+    def test_smask_figure_is_not_all_black(self):
+        pdf = pdfium.PdfDocument(_build_smask_pdf())
+        try:
+            result = _extract_embedded_images(
+                pdf, ["text"], pdfium_raw, "smask_doc", 85
+            )
+        finally:
+            pdf.close()
+        self.assertIn(0, result)
+        ref_path, b64_jpeg, _y_top = result[0][0]
+        self.assertTrue(ref_path.startswith("images/smask_doc_p1_img1"))
+        pil = Image.open(io.BytesIO(base64.b64decode(b64_jpeg))).convert("RGB")
+        total = bright = 0
+        for x in range(0, pil.width, 4):
+            for y in range(0, pil.height, 4):
+                total += 1
+                if max(pil.getpixel((x, y))) >= 200:
+                    bright += 1
+        # Dropping the mask would make this ~0: the whole figure is black.
+        self.assertGreater(bright / total, 0.5)
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/tests/test_pdf_router.py

@@ -0,0 +1,504 @@
+import io
+import unittest
+
+from PIL import Image
+
+from docreader.parser.pdf_parser import (
+    PDFParser,
+    _classify_page,
+    _filter_reading_columns,
+    _group_lines,
+    _is_artifact_column,
+    _join_line_glyphs,
+    _merge_orphan_punctuation_lines,
+    _point_in_boxes,
+    _segments_to_markdown,
+    _select_embedded_images,
+    _should_prefer_plain,
+    _split_columns,
+    _strip_repeating_lines,
+)
+
+
+def _char(ch, x0, x1, y0, y1):
+    return {"x0": x0, "x1": x1, "y0": y0, "y1": y1, "ch": ch}
+
+
+def _line(text, h):
+    return {"text": text, "h": h}
+
+
+def _make_image_only_pdf(num_pages: int = 2) -> bytes:
+    buf = io.BytesIO()
+    pages = [Image.new("RGB", (64, 64), color) for color in ("white", "black")]
+    pages = (pages * ((num_pages // 2) + 1))[:num_pages]
+    pages[0].save(buf, format="PDF", save_all=True, append_images=pages[1:])
+    return buf.getvalue()
+
+
+class ClassifyPageTest(unittest.TestCase):
+    def test_full_page_image_is_scanned_even_with_text(self):
+        # Scanned newspaper: image covers the page, embedded OCR text exists.
+        self.assertEqual(_classify_page(2.0, 1620), "scanned")
+        # Scanned UN doc with garbled OCR text layer (ratio ~1.0).
+        self.assertEqual(_classify_page(1.0, 1500), "scanned")
+
+    def test_native_text_page_is_text(self):
+        self.assertEqual(_classify_page(0.01, 673), "text")
+        self.assertEqual(_classify_page(0.0, 1200), "text")
+
+    def test_sparse_text_with_image_is_scanned(self):
+        self.assertEqual(_classify_page(0.3, 2), "scanned")
+
+    def test_blank_page_is_text(self):
+        # No image, no text -> not rendered as an image.
+        self.assertEqual(_classify_page(0.0, 0), "text")
+
+
+class StripRepeatingLinesTest(unittest.TestCase):
+    def test_removes_repeated_header_footer(self):
+        header = "ACME CONFIDENTIAL"
+        texts = [f"{header}\nbody page {i}\npage {i} footer" for i in range(6)]
+        # Make the footer identical across pages so it is detected.
+        texts = [f"{header}\nbody page {i}\nshared footer" for i in range(6)]
+        classes = ["text"] * 6
+        cleaned = _strip_repeating_lines(texts, classes)
+        for page in cleaned:
+            self.assertNotIn(header, page)
+            self.assertNotIn("shared footer", page)
+        self.assertIn("body page 0", cleaned[0])
+
+    def test_keeps_lines_when_too_few_pages(self):
+        texts = ["HEADER\nbody"] * 2
+        classes = ["text"] * 2
+        self.assertEqual(_strip_repeating_lines(texts, classes), texts)
+
+
+class SelectEmbeddedImagesTest(unittest.TestCase):
+    def _fig(self, page, h="fig", w=200, ht=200, area=0.2):
+        return {"page": page, "width": w, "height": ht, "area_ratio": area, "hash": h}
+
+    def test_keeps_real_figure(self):
+        meta = [self._fig(0, "a")]
+        self.assertEqual(_select_embedded_images(meta, 1), [0])
+
+    def test_drops_tiny_and_small_images(self):
+        meta = [
+            self._fig(0, "tiny_area", area=0.001),  # too small a share
+            self._fig(0, "tiny_px", w=20, ht=20),  # too few pixels
+        ]
+        self.assertEqual(_select_embedded_images(meta, 1), [])
+
+    def test_drops_repeated_logo_watermark(self):
+        # Same hash on 5 of 6 text pages -> running logo/watermark.
+        meta = [self._fig(p, "logo") for p in range(5)]
+        meta.append(self._fig(5, "unique"))
+        kept = _select_embedded_images(meta, 6)
+        kept_hashes = {meta[i]["hash"] for i in kept}
+        self.assertNotIn("logo", kept_hashes)
+        self.assertIn("unique", kept_hashes)
+
+    def test_dedups_identical_image_on_same_page(self):
+        meta = [self._fig(0, "dup"), self._fig(0, "dup")]
+        self.assertEqual(len(_select_embedded_images(meta, 1)), 1)
+
+    def test_respects_max_images_cap(self):
+        meta = [self._fig(i, f"h{i}") for i in range(10)]
+        self.assertEqual(len(_select_embedded_images(meta, 10, max_images=3)), 3)
+
+
+class ReadingOrderTest(unittest.TestCase):
+    def test_single_column_stays_single(self):
+        # One column of glyphs at x~100, no full-height gutter.
+        chars = [_char("a", 100, 110, 700 - i * 12, 712 - i * 12) for i in range(5)]
+        cols = _split_columns(chars, scale=12.0, width=600.0)
+        self.assertEqual(len(cols), 1)
+
+    def test_two_columns_split_left_to_right(self):
+        # Left column x~50-150, right column x~400-500, wide empty gutter between.
+        left = [_char("L", 50, 150, 700 - i * 12, 712 - i * 12) for i in range(4)]
+        right = [_char("R", 400, 500, 700 - i * 12, 712 - i * 12) for i in range(4)]
+        cols = _split_columns(left + right, scale=12.0, width=600.0)
+        self.assertEqual(len(cols), 2)
+        # Reading order: left column before right column.
+        self.assertEqual(cols[0][0]["ch"], "L")
+        self.assertEqual(cols[1][0]["ch"], "R")
+
+    def test_group_lines_orders_by_y_then_x(self):
+        # Two visual lines; within a line glyphs given out of x-order.
+        chars = [
+            _char("B", 110, 120, 700, 712),  # adjacent to A (no word-sized gap)
+            _char("A", 100, 110, 700, 712),
+            _char("C", 100, 110, 680, 692),  # next line down
+        ]
+        lines = _group_lines(chars)
+        self.assertEqual([ln["text"] for ln in lines], ["AB", "C"])
+
+    def test_join_line_glyphs_inserts_word_spaces(self):
+        # Wide gap between "copy" and "of" mimics positioned OCR / text layers.
+        chars = [
+            _char("c", 0, 4, 0, 10),
+            _char("f", 10, 14, 0, 10),
+        ]
+        self.assertEqual(_join_line_glyphs(chars), "c f")
+
+    def test_join_line_glyphs_keeps_adjacent_letters(self):
+        chars = [_char("A", 100, 110, 700, 712), _char("B", 110, 120, 700, 712)]
+        self.assertEqual(_join_line_glyphs(chars), "AB")
+
+
+class HeadingDetectionTest(unittest.TestCase):
+    def test_promotes_large_line_to_heading(self):
+        lines = [_line("Big Title", 24.0)] + [_line(f"body {i}", 10.0) for i in range(6)]
+        md = _segments_to_markdown(lines)
+        self.assertTrue(md.startswith("# Big Title"))
+        self.assertIn("\nbody 0", md)
+
+    def test_does_not_promote_when_sizes_uniform(self):
+        lines = [_line(f"line {i}", 10.0) for i in range(6)]
+        md = _segments_to_markdown(lines)
+        self.assertNotIn("#", md)
+
+    def test_skips_sentence_like_long_lines(self):
+        # Large but ends with a period and is long -> body text, not a heading.
+        lines = [_line("x" * 90 + ".", 30.0)] + [_line("y", 10.0) for _ in range(6)]
+        md = _segments_to_markdown(lines)
+        self.assertFalse(md.startswith("#"))
+
+
+class HiddenTextFilterTest(unittest.TestCase):
+    def test_point_in_boxes(self):
+        boxes = [(0.0, 0.0, 10.0, 10.0)]
+        self.assertTrue(_point_in_boxes(5.0, 5.0, boxes))
+        self.assertFalse(_point_in_boxes(20.0, 5.0, boxes))
+
+
+class MarginColumnFilterTest(unittest.TestCase):
+    def test_drops_narrow_vertical_margin_column(self):
+        # Mimics arXiv sidebar: narrow x span, one glyph per line.
+        margin = [
+            _char(c, 20, 28, 500 - i * 14, 512 - i * 14)
+            for i, c in enumerate("0202luJ22")
+        ]
+        body = [
+            _char("L", 160, 170, 700, 712),
+            _char("a", 170, 180, 700, 712),
+            _char("n", 180, 190, 700, 712),
+        ]
+        cols = _filter_reading_columns(margin + body, scale=10.0, width=612.0)
+        self.assertEqual(len(cols), 1)
+        self.assertEqual(cols[0][0]["ch"], "L")
+
+    def test_keeps_real_two_column_layout(self):
+        left = [_char("L", 50, 150, 700 - i * 12, 712 - i * 12) for i in range(4)]
+        right = [_char("R", 400, 500, 700 - i * 12, 712 - i * 12) for i in range(4)]
+        cols = _filter_reading_columns(left + right, scale=12.0, width=600.0)
+        self.assertEqual(len(cols), 2)
+
+
+class PunctuationMergeTest(unittest.TestCase):
+    def test_merges_orphan_periods(self):
+        lines = [
+            _line("Figure 1 2", 10.0),
+            _line(". .", 10.0),
+            _line("Next", 10.0),
+        ]
+        merged = _merge_orphan_punctuation_lines(lines)
+        self.assertEqual([ln["text"] for ln in merged], ["Figure 1 2..", "Next"])
+
+
+class PdfTextSanitizeTest(unittest.TestCase):
+    def test_removes_fffe_placeholder(self):
+        from docreader.parser.pdf_parser import _postprocess_pdf_text
+
+        raw = "multi\ufffelayer and non\ufffetrivial"
+        out = _postprocess_pdf_text(raw)
+        self.assertEqual(out, "multilayer and nontrivial")
+
+    def test_strips_chart_axis_run(self):
+        from docreader.parser.pdf_parser import _postprocess_pdf_text
+
+        raw = (
+            "Deep convolutional neural networks have led to breakthroughs.\n"
+            "0 1 2 3 4 5 6 0\n"
+            "10\n"
+            "20\n"
+            "iter. (1e4)\n"
+            "training error (%)\n"
+            "56-layer\n"
+            "20-layer\n"
+            "Figure 1. Training error on CIFAR-10.\n"
+        )
+        out = _postprocess_pdf_text(raw)
+        self.assertIn("breakthroughs", out)
+        self.assertNotIn("56-layer", out)
+        self.assertIn("Figure 1.", out)
+
+    def test_strips_diagram_labels_above_caption(self):
+        from docreader.parser.pdf_parser import _postprocess_pdf_text
+
+        raw = (
+            "Paragraph before.\n"
+            "identity\n"
+            "weight layer\n"
+            "relu\n"
+            "Figure 2. Residual learning block.\n"
+            "Paragraph after.\n"
+        )
+        out = _postprocess_pdf_text(raw)
+        self.assertIn("Paragraph before.", out)
+        self.assertIn("Figure 2.", out)
+        self.assertIn("Paragraph after.", out)
+        self.assertNotIn("identity", out)
+        self.assertNotIn("weight layer", out)
+
+    def test_strips_arxiv_header_line(self):
+        from docreader.parser.pdf_parser import _postprocess_pdf_text
+
+        raw = "Body text.\n1\narXiv:1512.03385v1 [cs.CV] 10 Dec 2015\nMore body."
+        out = _postprocess_pdf_text(raw)
+        self.assertNotIn("arXiv:", out)
+        self.assertIn("Body text.", out)
+
+
+class PlainWellFormedTest(unittest.TestCase):
+    def test_academic_plain_skips_layout(self):
+        from docreader.parser.pdf_parser import _plain_is_well_formed
+
+        plain = (
+            "Recent work [DL15, MBXS17] shows progress on NLP tasks "
+            "with pre-trained models."
+        )
+        self.assertTrue(_plain_is_well_formed(plain))
+
+    def test_glued_scan_plain_needs_layout(self):
+        from docreader.parser.pdf_parser import _plain_is_well_formed
+
+        self.assertFalse(_plain_is_well_formed("Thisisadigitalcopyofabook"))
+
+
+class LayoutQualityFallbackTest(unittest.TestCase):
+    def test_prefers_plain_when_many_single_char_lines(self):
+        plain = "Language Models are Few-Shot Learners\nTom Brown"
+        layout = "0\n2\n0\n2\nl\nu\nJ\nLan ua e Models"
+        self.assertTrue(_should_prefer_plain(plain, layout))
+
+    def test_keeps_good_layout(self):
+        plain = "Hello world"
+        layout = "Hello world"
+        self.assertFalse(_should_prefer_plain(plain, layout))
+
+
+class ResNetPaperFigureTest(unittest.TestCase):
+    """Regression: ResNet PDF (arXiv:1512.03385) vector figures and captions."""
+
+    def test_resnet_figures_and_captions(self):
+        import os
+
+        from docreader.parser.pdf_parser import PDFParser
+
+        for path in (
+            os.path.join(
+                os.path.dirname(__file__),
+                "..",
+                "..",
+                "testdata",
+                "rag_test",
+                "pdf_en",
+                "resnet.pdf",
+            ),
+            "/tmp/resnet.pdf",
+        ):
+            if os.path.isfile(path):
+                break
+        else:
+            self.skipTest("resnet.pdf not available")
+
+        with open(path, "rb") as f:
+            doc = PDFParser(file_name="resnet.pdf", file_type="pdf").parse_into_text(
+                f.read()
+            )
+        self.assertGreater(doc.metadata.get("vector_figure_count", 0), 0)
+        self.assertIn("![", doc.content)
+        self.assertIn("Figure 2. Residual learning", doc.content)
+        self.assertNotIn("arXiv:", doc.content)
+        fig2 = doc.content.find("Figure 2. Residual learning")
+        before = doc.content[max(0, fig2 - 120) : fig2]
+        self.assertIn("![", before)
+        self.assertNotIn("identity", before)
+
+
+class Gpt3PaperLayoutTest(unittest.TestCase):
+    """Regression: arXiv GPT-3 paper title page must not be one-glyph-per-line."""
+
+    def test_gpt3_page0_title_and_authors(self):
+        import os
+
+        import pypdfium2 as pdfium
+        import pypdfium2.raw as pdfium_r
+
+        from docreader.parser.pdf_parser import PDFParser, _extract_layout_text
+
+        pdf_path = os.path.join(
+            os.path.dirname(__file__),
+            "..",
+            "..",
+            "testdata",
+            "rag_test",
+            "pdf_en",
+            "gpt3.pdf",
+        )
+        if not os.path.isfile(pdf_path):
+            self.skipTest("gpt3.pdf not in testdata")
+        with open(pdf_path, "rb") as f:
+            content = f.read()
+        with pdfium.PdfDocument(content) as pdf:
+            page = pdf[0]
+            try:
+                layout = _extract_layout_text(page, pdfium_r)
+            finally:
+                page.close()
+        # Margin sidebar must not appear as one-glyph-per-line prefix.
+        self.assertNotRegex(layout[:300], r"^0\n2\n0\n2")
+        self.assertIn("Few-Shot Learners", layout)
+
+        doc = PDFParser(file_name="gpt3.pdf", file_type="pdf").parse_into_text(content)
+        self.assertIn("Language Models are Few-Shot Learners", doc.content)
+        self.assertIn("Tom B. Brown", doc.content[:1200])
+        self.assertIn("[DL15, MBXS17, PNZtY18]", doc.content)
+        self.assertIn("task-specific architectures), and more recently", doc.content)
+        self.assertNotIn("k ifi hi d l", doc.content)
+
+
+class ScanEnglishDictLayoutTest(unittest.TestCase):
+    """Regression: Google Books-style PDFs lose spaces without gap inference."""
+
+    def test_scan_en_dict_page0_has_word_spaces(self):
+        import os
+
+        import pypdfium2 as pdfium
+        import pypdfium2.raw as pdfium_r
+
+        from docreader.parser.pdf_parser import _extract_layout_text
+
+        pdf_path = os.path.join(
+            os.path.dirname(__file__),
+            "..",
+            "..",
+            "testdata",
+            "rag_test",
+            "pdf_scan",
+            "scan_en_dict.pdf",
+        )
+        if not os.path.isfile(pdf_path):
+            self.skipTest("scan_en_dict.pdf not in testdata")
+        with open(pdf_path, "rb") as f:
+            pdf = pdfium.PdfDocument(f.read())
+        try:
+            text = _extract_layout_text(pdf[0], pdfium_r)
+        finally:
+            pdf.close()
+        self.assertIn("This is a digital copy of a book", text)
+        self.assertNotIn("Thisisadigitalcopyofabook", text)
+
+
+class PDFRouterIntegrationTest(unittest.TestCase):
+    def test_image_only_pdf_routes_to_scanned(self):
+        pdf_bytes = _make_image_only_pdf(2)
+        doc = PDFParser(file_name="imgonly.pdf", file_type="pdf").parse_into_text(
+            pdf_bytes
+        )
+
+        self.assertEqual(doc.metadata["image_source_type"], "scanned_pdf")
+        self.assertEqual(doc.metadata["page_count"], 2)
+        self.assertEqual(doc.metadata["scanned_page_count"], 2)
+        self.assertEqual(len(doc.images), 2)
+        self.assertIn("images/imgonly_page_1.jpg", doc.images)
+        self.assertIn("![imgonly_page_1.jpg](images/imgonly_page_1.jpg)", doc.content)
+        # JPEG magic bytes after decoding.
+        import base64
+
+        self.assertTrue(
+            base64.b64decode(doc.images["images/imgonly_page_1.jpg"]).startswith(
+                b"\xff\xd8"
+            )
+        )
+
+    def test_malformed_pdf_raises_after_fallback(self):
+        # Routing fails to open the PDF, falls back to full rendering which also
+        # fails on garbage input; the error surfaces to the caller.
+        with self.assertRaises(Exception):
+            PDFParser(file_name="broken.pdf", file_type="pdf").parse_into_text(
+                b"not a pdf"
+            )
+
+
+class ForceScannedTest(unittest.TestCase):
+    """Tests for the force-scanned PDF parsing mode."""
+
+    def test_default_text_pdf_is_not_force_scanned(self):
+        """Without any override, a text PDF should NOT use scanned mode."""
+        parser = PDFParser(file_name="normal.pdf", file_type="pdf")
+        self.assertFalse(parser._force_scanned)
+
+    def test_force_scanned_via_kwarg_true(self):
+        """Per-upload override pdf_force_scanned=true enables scanned mode."""
+        parser = PDFParser(
+            file_name="test.pdf", file_type="pdf", pdf_force_scanned="true"
+        )
+        self.assertTrue(parser._force_scanned)
+
+    def test_force_scanned_via_kwarg_false(self):
+        """Per-upload override pdf_force_scanned=false disables scanned mode."""
+        parser = PDFParser(
+            file_name="test.pdf", file_type="pdf", pdf_force_scanned="false"
+        )
+        self.assertFalse(parser._force_scanned)
+
+    def test_force_scanned_via_kwarg_overrides_env(self, monkeypatch=None):
+        """Per-upload override takes priority over global env var."""
+        import docreader.parser.pdf_parser as pdf_mod
+
+        old = pdf_mod.FORCE_SCANNED_PDF
+        try:
+            pdf_mod.FORCE_SCANNED_PDF = True
+            # Explicit false override should win over env=true
+            parser = PDFParser(
+                file_name="test.pdf", file_type="pdf", pdf_force_scanned="false"
+            )
+            self.assertFalse(parser._force_scanned)
+        finally:
+            pdf_mod.FORCE_SCANNED_PDF = old
+
+    def test_force_scanned_via_env(self):
+        """Global env variable enables force scanned when no override."""
+        import docreader.parser.pdf_parser as pdf_mod
+
+        old = pdf_mod.FORCE_SCANNED_PDF
+        try:
+            pdf_mod.FORCE_SCANNED_PDF = True
+            parser = PDFParser(file_name="test.pdf", file_type="pdf")
+            self.assertTrue(parser._force_scanned)
+        finally:
+            pdf_mod.FORCE_SCANNED_PDF = old
+
+    def test_force_scanned_output_is_scanned_pdf(self):
+        """Force-scanned mode produces scanned_pdf metadata and image refs."""
+        pdf_bytes = _make_image_only_pdf(2)
+        parser = PDFParser(
+            file_name="forced.pdf", file_type="pdf", pdf_force_scanned="true"
+        )
+        doc = parser.parse_into_text(pdf_bytes)
+        self.assertEqual(doc.metadata.get("image_source_type"), "scanned_pdf")
+        self.assertEqual(doc.metadata.get("page_count"), 2)
+        self.assertEqual(doc.metadata.get("scanned_page_count"), 2)
+        self.assertEqual(doc.metadata.get("text_page_count"), 0)
+        self.assertEqual(doc.metadata.get("embedded_image_count"), 0)
+        self.assertEqual(doc.metadata.get("vector_figure_count"), 0)
+        self.assertIn("![forced_page_1.jpg]", doc.content)
+        self.assertIn("![forced_page_2.jpg]", doc.content)
+        self.assertEqual(len(doc.images), 2)
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/tests/test_ppt_convert.py

@@ -0,0 +1,94 @@
+import shutil
+import unittest
+from pathlib import Path
+
+from docreader.parser.ppt_convert import (
+    convert_ppt_to_pptx_bytes,
+    is_ole_compound,
+    is_zip_openxml,
+    needs_ppt_to_pptx_conversion,
+    normalize_ppt_bytes,
+)
+
+TESTDATA = Path(__file__).resolve().parents[2] / "testdata" / "rag_test"
+LEGACY_PPT = TESTDATA / "ppt_old" / "en_38256.ppt"
+WMF_IMAGE_PPT = LEGACY_PPT
+IMAGE_HEAVY_PPT = TESTDATA / "ppt_old" / "en_41384.ppt"
+PPTX_SAMPLE = TESTDATA / "pptx" / "en_marker.pptx"
+
+
+class TestPptConvert(unittest.TestCase):
+    def test_legacy_ppt_magic(self):
+        if not LEGACY_PPT.is_file():
+            self.skipTest("legacy PPT fixture not available")
+        content = LEGACY_PPT.read_bytes()
+        self.assertTrue(is_ole_compound(content))
+        self.assertFalse(is_zip_openxml(content))
+        self.assertTrue(needs_ppt_to_pptx_conversion(content, "ppt"))
+
+    def test_pptx_does_not_need_conversion(self):
+        if not PPTX_SAMPLE.is_file():
+            self.skipTest("PPTX fixture not available")
+        content = PPTX_SAMPLE.read_bytes()
+        self.assertTrue(is_zip_openxml(content))
+        self.assertFalse(needs_ppt_to_pptx_conversion(content, "pptx"))
+
+    def test_normalize_pptx_passthrough(self):
+        if not PPTX_SAMPLE.is_file():
+            self.skipTest("PPTX fixture not available")
+        content = PPTX_SAMPLE.read_bytes()
+        out, ext = normalize_ppt_bytes(content, "pptx")
+        self.assertEqual(out, content)
+        self.assertEqual(ext, ".pptx")
+
+    def test_legacy_ppt_requires_soffice(self):
+        if not LEGACY_PPT.is_file():
+            self.skipTest("legacy PPT fixture not available")
+        if not shutil.which("soffice"):
+            with self.assertRaises(ValueError) as ctx:
+                normalize_ppt_bytes(LEGACY_PPT.read_bytes(), "ppt")
+            self.assertIn("LibreOffice", str(ctx.exception))
+            self.skipTest("LibreOffice not available")
+        converted = convert_ppt_to_pptx_bytes(LEGACY_PPT.read_bytes(), suffix=".ppt")
+        self.assertIsNotNone(converted)
+        self.assertTrue(is_zip_openxml(converted))
+        out, ext = normalize_ppt_bytes(LEGACY_PPT.read_bytes(), "ppt")
+        self.assertEqual(ext, ".pptx")
+        self.assertTrue(is_zip_openxml(out))
+
+    def test_wmf_legacy_ppt_extracts_rasterized_image(self):
+        if not shutil.which("soffice"):
+            self.skipTest("LibreOffice not available")
+        if not shutil.which("convert"):
+            self.skipTest("ImageMagick convert not available")
+        if not WMF_IMAGE_PPT.is_file():
+            self.skipTest("testdata missing")
+
+        from docreader.parser.markitdown_parser import MarkitdownParser
+
+        doc = MarkitdownParser(file_type="ppt").parse_into_text(
+            WMF_IMAGE_PPT.read_bytes()
+        )
+        self.assertEqual(len(doc.images), 1)
+        self.assertNotIn("bd10496_.jpg", doc.content)
+        self.assertIn("images/", doc.content)
+
+    def test_image_heavy_legacy_ppt_extracts_images(self):
+        if not shutil.which("soffice"):
+            self.skipTest("LibreOffice not available")
+        if not IMAGE_HEAVY_PPT.is_file():
+            self.skipTest("testdata missing")
+
+        from docreader.parser.markitdown_parser import MarkitdownParser
+
+        doc = MarkitdownParser(file_type="ppt").parse_into_text(
+            IMAGE_HEAVY_PPT.read_bytes()
+        )
+        self.assertGreaterEqual(len(doc.images), 2)
+        self.assertNotIn("![](.jpg)", doc.content)
+        for ref in doc.images:
+            self.assertTrue(ref.startswith("images/"))
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/tests/test_ssrf.py

@@ -0,0 +1,61 @@
+import ipaddress
+import os
+import unittest
+from unittest.mock import patch
+
+from docreader.utils.ssrf import is_ssrf_safe_url, reset_ssrf_whitelist_cache_for_test
+
+
+class TestSSRFValidation(unittest.TestCase):
+    def setUp(self) -> None:
+        self._env_patch = patch.dict(
+            os.environ,
+            {"SSRF_WHITELIST": "", "SSRF_WHITELIST_EXTRA": ""},
+            clear=False,
+        )
+        self._env_patch.start()
+        reset_ssrf_whitelist_cache_for_test()
+
+    def tearDown(self) -> None:
+        self._env_patch.stop()
+        reset_ssrf_whitelist_cache_for_test()
+
+    def test_blocks_loopback_ip(self):
+        safe, reason = is_ssrf_safe_url("http://127.0.0.1:8080/page")
+        self.assertFalse(safe)
+        self.assertTrue(reason)
+
+    def test_blocks_restricted_hostname(self):
+        safe, reason = is_ssrf_safe_url("http://host.docker.internal/secret")
+        self.assertFalse(safe)
+        self.assertIn("restricted", reason)
+
+    def test_blocks_metadata_host(self):
+        safe, reason = is_ssrf_safe_url(
+            "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
+        )
+        self.assertFalse(safe)
+        self.assertTrue(reason)
+
+    def test_blocks_invalid_port(self):
+        safe, reason = is_ssrf_safe_url("https://example.com:99999/path")
+        self.assertFalse(safe)
+        self.assertIn("invalid port", reason)
+
+    def test_blocks_ipv4_mapped_private_ipv6_resolution(self):
+        with patch(
+            "docreader.utils.ssrf._resolve_host_ips",
+            return_value=((ipaddress.ip_address("::ffff:127.0.0.1"),), None),
+        ):
+            safe, reason = is_ssrf_safe_url("https://example.invalid/path")
+        self.assertFalse(safe)
+        self.assertIn("restricted", reason)
+
+    def test_allows_public_https(self):
+        safe, reason = is_ssrf_safe_url("https://example.com/article")
+        self.assertTrue(safe, reason)
+        self.assertEqual(reason, "")
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/tests/test_web_parser.py

@@ -0,0 +1,153 @@
+import os
+import unittest
+from unittest.mock import AsyncMock, patch
+
+from docreader.parser.web_parser import (
+    StdWebParser,
+    WebParseError,
+    WebParser,
+    _ScrapeResult,
+    build_visible_text_fallback,
+    extract_markdown_from_html,
+    install_ssrf_route_guard,
+)
+from docreader.utils.ssrf import is_ssrf_safe_url, reset_ssrf_whitelist_cache_for_test
+
+
+class TestWebParserHelpers(unittest.TestCase):
+    def setUp(self) -> None:
+        self._env_patch = patch.dict(
+            os.environ,
+            {"SSRF_WHITELIST": "", "SSRF_WHITELIST_EXTRA": ""},
+            clear=False,
+        )
+        self._env_patch.start()
+        reset_ssrf_whitelist_cache_for_test()
+
+    def tearDown(self) -> None:
+        self._env_patch.stop()
+        reset_ssrf_whitelist_cache_for_test()
+
+    def test_extract_markdown_empty_html(self):
+        self.assertIsNone(extract_markdown_from_html(""))
+        self.assertIsNone(extract_markdown_from_html("   "))
+
+    def test_extract_markdown_article_html(self):
+        html = """
+        <html><head><title>Demo</title></head><body>
+        <article><h1>Hello</h1><p>World paragraph with enough text for extraction.</p></article>
+        </body></html>
+        """
+        md = extract_markdown_from_html(html)
+        self.assertIsNotNone(md)
+        self.assertIn("Hello", md)
+
+    def test_build_fallback_too_short(self):
+        self.assertIsNone(build_visible_text_fallback("short"))
+        self.assertIsNone(build_visible_text_fallback(""))
+
+    def test_build_fallback_with_title(self):
+        text = "A" * 60
+        md = build_visible_text_fallback(text, page_title="WeKnora")
+        self.assertIsNotNone(md)
+        self.assertTrue(md.startswith("# WeKnora"))
+        self.assertIn(text, md)
+
+    def test_build_fallback_without_title(self):
+        text = "B" * 60
+        md = build_visible_text_fallback(text, page_title="")
+        self.assertEqual(md, text)
+
+    def test_install_ssrf_route_guard_is_importable(self):
+        self.assertTrue(callable(install_ssrf_route_guard))
+
+    def test_redirect_target_blocked_before_navigation(self):
+        safe, reason = is_ssrf_safe_url("http://127.0.0.1:39127/audit.txt")
+        self.assertFalse(safe)
+        self.assertTrue(reason)
+
+
+class TestStdWebParserFailures(unittest.TestCase):
+    """Scrape/parse failures must raise, not become indexable document body."""
+
+    def _parser(self) -> StdWebParser:
+        return StdWebParser(title="page")
+
+    def test_empty_scrape_raises_instead_of_error_body(self):
+        empty = _ScrapeResult(
+            html="",
+            visible_text="",
+            page_title="",
+            error="navigation failed: Timeout",
+        )
+        parser = self._parser()
+        with patch.object(parser, "scrape", new=AsyncMock(return_value=empty)):
+            with self.assertRaises(WebParseError) as ctx:
+                parser.parse_into_text(b"https://example.com/blocked")
+        message = str(ctx.exception)
+        self.assertIn("https://example.com/blocked", message)
+        self.assertIn("navigation failed", message)
+        self.assertNotIn("Error parsing web page:", message)
+
+    def test_empty_scrape_without_error_field_still_raises(self):
+        empty = _ScrapeResult(html="", visible_text="", page_title="")
+        parser = self._parser()
+        with patch.object(parser, "scrape", new=AsyncMock(return_value=empty)):
+            with self.assertRaises(WebParseError) as ctx:
+                parser.parse_into_text(b"https://example.com/blank")
+        self.assertIn("no HTML or visible text", str(ctx.exception))
+
+    def test_unextractable_page_raises_instead_of_error_body(self):
+        scrape = _ScrapeResult(
+            html="<html><body><div></div></body></html>",
+            visible_text="short",
+            page_title="",
+        )
+        parser = self._parser()
+        with patch.object(parser, "scrape", new=AsyncMock(return_value=scrape)):
+            with patch(
+                "docreader.parser.web_parser.extract_markdown_from_html",
+                return_value=None,
+            ):
+                with patch(
+                    "docreader.parser.web_parser.build_visible_text_fallback",
+                    return_value=None,
+                ):
+                    with self.assertRaises(WebParseError) as ctx:
+                        parser.parse_into_text(b"https://example.com/empty")
+        self.assertIn("Failed to parse web page", str(ctx.exception))
+        self.assertIn("https://example.com/empty", str(ctx.exception))
+
+    def test_successful_scrape_returns_markdown_document(self):
+        html = """
+        <html><head><title>Demo</title></head><body>
+        <article><h1>Hello</h1><p>World paragraph with enough text for extraction.</p></article>
+        </body></html>
+        """
+        scrape = _ScrapeResult(
+            html=html,
+            visible_text="Hello World paragraph with enough text for extraction.",
+            page_title="Demo",
+        )
+        parser = self._parser()
+        with patch.object(parser, "scrape", new=AsyncMock(return_value=scrape)):
+            doc = parser.parse_into_text(b"https://example.com/ok")
+        self.assertTrue(doc.is_valid())
+        self.assertNotIn("Error parsing web page:", doc.content)
+        self.assertIn("Hello", doc.content)
+
+    def test_pipeline_web_parser_does_not_index_scrape_error(self):
+        empty = _ScrapeResult(
+            html="",
+            visible_text="",
+            page_title="",
+            error="URL blocked by SSRF guard: loopback",
+        )
+        pipeline = WebParser(title="page")
+        with patch.object(StdWebParser, "scrape", new=AsyncMock(return_value=empty)):
+            with self.assertRaises(WebParseError):
+                pipeline.parse_into_text(b"https://example.com/ssrf")
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/tests/test_xmind_parser.py

@@ -0,0 +1,286 @@
+import io
+import json
+import unittest
+import zipfile
+from unittest.mock import patch
+
+from docreader.parser.xmind_parser import XMindParser
+
+
+def _xmind_zip(entries: dict[str, bytes | str]) -> bytes:
+    buffer = io.BytesIO()
+    with zipfile.ZipFile(buffer, "w", compression=zipfile.ZIP_DEFLATED) as archive:
+        for name, value in entries.items():
+            payload = value.encode("utf-8") if isinstance(value, str) else value
+            archive.writestr(name, payload)
+    return buffer.getvalue()
+
+
+def _modern_xmind_bytes(sheets: list[dict]) -> bytes:
+    return _xmind_zip({"content.json": json.dumps(sheets, ensure_ascii=False)})
+
+
+def _classic_xmind_bytes(xml: str) -> bytes:
+    return _xmind_zip({"content.xml": xml})
+
+
+def _mark_entry_encrypted(payload: bytes) -> bytes:
+    marked = bytearray(payload)
+    local_header = marked.index(b"PK\x03\x04")
+    central_header = marked.index(b"PK\x01\x02")
+    for offset in (local_header + 6, central_header + 8):
+        flags = int.from_bytes(marked[offset : offset + 2], "little") | 0x1
+        marked[offset : offset + 2] = flags.to_bytes(2, "little")
+    return bytes(marked)
+
+
+class XMindParserModernTests(unittest.TestCase):
+    def test_finds_content_without_enumerating_unrelated_entries(self):
+        payload = _xmind_zip(
+            {
+                "attachments/preview.png": b"preview",
+                "content.json": json.dumps(
+                    [{"title": "Outline", "rootTopic": {"title": "Root"}}]
+                ),
+            }
+        )
+
+        with patch.object(
+            zipfile.ZipFile,
+            "namelist",
+            side_effect=AssertionError("archive entries must not be enumerated"),
+        ):
+            document = XMindParser().parse_into_text(payload)
+
+        self.assertEqual("# Outline\n\n- Root", document.content)
+
+    def test_parses_topic_hierarchy_and_plain_notes(self):
+        payload = _modern_xmind_bytes(
+            [
+                {
+                    "title": "Launch Plan",
+                    "rootTopic": {
+                        "title": "Release",
+                        "notes": {"plain": {"content": "Coordinate teams"}},
+                        "children": {
+                            "attached": [
+                                {
+                                    "title": "Backend",
+                                    "children": {
+                                        "attached": [{"title": "API freeze"}]
+                                    },
+                                }
+                            ]
+                        },
+                    },
+                }
+            ]
+        )
+
+        document = XMindParser(file_name="launch.xmind").parse_into_text(payload)
+
+        self.assertEqual(
+            "# Launch Plan\n\n"
+            "- Release\n"
+            "  > Coordinate teams\n"
+            "  - Backend\n"
+            "    - API freeze",
+            document.content,
+        )
+        self.assertEqual(document.metadata["source_format"], "xmind")
+        self.assertEqual(document.metadata["xmind_content_format"], "json")
+        self.assertEqual(document.metadata["sheet_count"], 1)
+        self.assertEqual(document.metadata["topic_count"], 3)
+        self.assertEqual(document.metadata["note_count"], 1)
+        self.assertEqual(document.metadata["file_size"], len(payload))
+
+    def test_renders_multiple_sheets_with_fallback_title(self):
+        payload = _modern_xmind_bytes(
+            [
+                {"title": "One", "rootTopic": {"title": "Alpha"}},
+                {"title": "  ", "rootTopic": {"title": "Beta"}},
+            ]
+        )
+
+        document = XMindParser().parse_into_text(payload)
+
+        self.assertEqual(
+            "# One\n\n- Alpha\n\n---\n\n# Sheet 2\n\n- Beta",
+            document.content,
+        )
+        self.assertEqual(document.metadata["sheet_count"], 2)
+        self.assertEqual(document.metadata["topic_count"], 2)
+
+    def test_promotes_children_of_blank_topic(self):
+        payload = _modern_xmind_bytes(
+            [
+                {
+                    "title": "Outline",
+                    "rootTopic": {
+                        "title": "  ",
+                        "children": {
+                            "attached": [
+                                {
+                                    "title": "Visible",
+                                    "children": {
+                                        "attached": [{"title": "Nested"}]
+                                    },
+                                }
+                            ]
+                        },
+                    },
+                }
+            ]
+        )
+
+        document = XMindParser().parse_into_text(payload)
+
+        self.assertEqual("# Outline\n\n- Visible\n  - Nested", document.content)
+        self.assertEqual(document.metadata["topic_count"], 2)
+
+    def test_renders_multiline_note_as_blockquotes(self):
+        payload = _modern_xmind_bytes(
+            [
+                {
+                    "title": "Notes",
+                    "rootTopic": {
+                        "title": "Root",
+                        "notes": {
+                            "plain": {"content": " First line \n\n Second line "}
+                        },
+                    },
+                }
+            ]
+        )
+
+        document = XMindParser().parse_into_text(payload)
+
+        self.assertEqual(
+            "# Notes\n\n- Root\n  > First line\n  >\n  > Second line",
+            document.content,
+        )
+
+
+class XMindParserClassicTests(unittest.TestCase):
+    def test_parses_namespaced_xml_hierarchy_and_notes(self):
+        payload = _classic_xmind_bytes(
+            """<?xml version="1.0" encoding="UTF-8"?>
+<xmap-content xmlns="urn:xmind:xmap:xmlns:content:2.0">
+  <sheet>
+    <title>Architecture</title>
+    <topic>
+      <title>Platform</title>
+      <notes><plain>Owns ingress</plain></notes>
+      <children>
+        <topics type="attached">
+          <topic><title>Gateway</title></topic>
+        </topics>
+      </children>
+    </topic>
+  </sheet>
+</xmap-content>
+"""
+        )
+
+        document = XMindParser().parse_into_text(payload)
+
+        self.assertEqual(
+            "# Architecture\n\n- Platform\n  > Owns ingress\n  - Gateway",
+            document.content,
+        )
+        self.assertEqual(document.metadata["xmind_content_format"], "xml")
+        self.assertEqual(document.metadata["sheet_count"], 1)
+        self.assertEqual(document.metadata["topic_count"], 2)
+        self.assertEqual(document.metadata["note_count"], 1)
+
+    def test_prefers_content_json_when_both_entries_exist(self):
+        xml = """<xmap-content><sheet><title>XML</title>
+<topic><title>XML topic</title></topic></sheet></xmap-content>"""
+        json_content = json.dumps(
+            [{"title": "JSON", "rootTopic": {"title": "JSON topic"}}]
+        )
+        payload = _xmind_zip(
+            {"content.xml": xml, "content.json": json_content}
+        )
+
+        document = XMindParser().parse_into_text(payload)
+
+        self.assertEqual("# JSON\n\n- JSON topic", document.content)
+        self.assertEqual(document.metadata["xmind_content_format"], "json")
+
+
+class XMindParserValidationTests(unittest.TestCase):
+    def test_rejects_invalid_zip(self):
+        with self.assertRaises(ValueError) as context:
+            XMindParser().parse_into_text(b"not a ZIP archive")
+
+        self.assertEqual(str(context.exception), "invalid XMind archive")
+
+    def test_rejects_archive_without_supported_content(self):
+        payload = _xmind_zip({"manifest.json": "{}"})
+
+        with self.assertRaises(ValueError) as context:
+            XMindParser().parse_into_text(payload)
+
+        self.assertEqual(
+            str(context.exception),
+            "XMind archive is missing content.json or content.xml",
+        )
+
+    def test_rejects_malformed_json(self):
+        payload = _xmind_zip({"content.json": "{"})
+
+        with self.assertRaises(ValueError) as context:
+            XMindParser().parse_into_text(payload)
+
+        self.assertEqual(str(context.exception), "invalid XMind content.json")
+
+    def test_rejects_malformed_xml(self):
+        payload = _classic_xmind_bytes("<xmap-content>")
+
+        with self.assertRaises(ValueError) as context:
+            XMindParser().parse_into_text(payload)
+
+        self.assertEqual(str(context.exception), "invalid XMind content.xml")
+
+    def test_rejects_archive_without_renderable_topics(self):
+        payload = _modern_xmind_bytes(
+            [{"title": "Empty", "rootTopic": {"title": " "}}]
+        )
+
+        with self.assertRaises(ValueError) as context:
+            XMindParser().parse_into_text(payload)
+
+        self.assertEqual(
+            str(context.exception),
+            "XMind archive contains no renderable topics",
+        )
+
+    def test_rejects_content_entry_over_limit(self):
+        payload = _xmind_zip({"content.json": "12345"})
+
+        with patch("docreader.parser.xmind_parser.MAX_CONTENT_BYTES", 4):
+            with self.assertRaises(ValueError) as context:
+                XMindParser().parse_into_text(payload)
+
+        self.assertEqual(
+            str(context.exception),
+            "XMind content entry exceeds the 32 MiB limit",
+        )
+
+    def test_rejects_encrypted_content_entry(self):
+        payload = _mark_entry_encrypted(
+            _xmind_zip({"content.json": "[]"})
+        )
+
+        with self.assertRaises(ValueError) as context:
+            XMindParser().parse_into_text(payload)
+
+        self.assertEqual(
+            str(context.exception),
+            "encrypted XMind content is not supported",
+        )
+
+
+if __name__ == "__main__":
+    unittest.main()

docreader/utils/__init__.py

@@ -0,0 +1,83 @@
+#
+#  Copyright 2024 The InfiniFlow Authors. All Rights Reserved.
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import os
+import re
+import logging
+
+# 配置日志
+logger = logging.getLogger(__name__)
+
+
+def singleton(cls, *args, **kw):
+    instances = {}
+
+    def _singleton():
+        key = str(cls) + str(os.getpid())
+        if key not in instances:
+            logger.info(f"Creating new singleton instance with key: {key}")
+            instances[key] = cls(*args, **kw)
+        else:
+            logger.info(f"Returning existing singleton instance with key: {key}")
+        return instances[key]
+
+    return _singleton
+
+
+def rmSpace(txt):
+    logger.info(f"Removing spaces from text of length: {len(txt)}")
+    txt = re.sub(r"([^a-z0-9.,\)>]) +([^ ])", r"\1\2", txt, flags=re.IGNORECASE)
+    return re.sub(r"([^ ]) +([^a-z0-9.,\(<])", r"\1\2", txt, flags=re.IGNORECASE)
+
+
+def findMaxDt(fnm):
+    m = "1970-01-01 00:00:00"
+    logger.info(f"Finding maximum date in file: {fnm}")
+    try:
+        with open(fnm, "r") as f:
+            while True:
+                l = f.readline()
+                if not l:
+                    break
+                l = l.strip("\n")
+                if l == "nan":
+                    continue
+                if l > m:
+                    m = l
+        logger.info(f"Maximum date found: {m}")
+    except Exception as e:
+        logger.error(f"Error reading file {fnm} for max date: {str(e)}")
+    return m
+
+
+def findMaxTm(fnm):
+    m = 0
+    logger.info(f"Finding maximum time in file: {fnm}")
+    try:
+        with open(fnm, "r") as f:
+            while True:
+                l = f.readline()
+                if not l:
+                    break
+                l = l.strip("\n")
+                if l == "nan":
+                    continue
+                if int(l) > m:
+                    m = int(l)
+        logger.info(f"Maximum time found: {m}")
+    except Exception as e:
+        logger.error(f"Error reading file {fnm} for max time: {str(e)}")
+    return m

docreader/utils/endecode.py

@@ -0,0 +1,204 @@
+"""
+Encoding and Decoding Utilities Module
+
+This module provides utilities for encoding and decoding various data types,
+with a focus on image and text data conversion:
+- Image encoding/decoding (base64)
+- Text encoding/decoding (multiple character sets)
+- Bytes conversion utilities
+"""
+
+import base64
+import binascii
+import io
+import logging
+from typing import List, Union
+
+import numpy as np
+from PIL import Image
+
+logger = logging.getLogger(__name__)
+
+
+def decode_image(image: Union[str, bytes, Image.Image, np.ndarray]) -> str:
+    """Convert image to base64 encoded string.
+
+    This function handles multiple image input formats and converts them
+    to a base64 encoded string representation, which is useful for embedding
+    images in JSON, HTML, or other text-based formats.
+
+    Args:
+        image: Image in one of the following formats:
+            - str: File path to an image file
+            - bytes: Raw image bytes data
+            - Image.Image: PIL/Pillow Image object
+            - np.ndarray: NumPy array representing image data
+
+    Returns:
+        str: Base64 encoded string representation of the image
+
+    Raises:
+        ValueError: If the image type is not supported
+
+    Example:
+        >>> # From file path
+        >>> base64_str = decode_image("/path/to/image.png")
+        >>> # From PIL Image
+        >>> from PIL import Image
+        >>> img = Image.open("photo.jpg")
+        >>> base64_str = decode_image(img)
+    """
+    if isinstance(image, str):
+        # Handle file path: read file and encode to base64
+        with open(image, "rb") as image_file:
+            return base64.b64encode(image_file.read()).decode()
+
+    elif isinstance(image, bytes):
+        # Handle raw bytes: directly encode to base64
+        return base64.b64encode(image).decode()
+
+    elif isinstance(image, Image.Image):
+        # Handle PIL Image: save to buffer then encode
+        buffer = io.BytesIO()
+        # Use original format if available, otherwise default to PNG
+        img_format = image.format if image.format else "PNG"
+        image.save(buffer, format=img_format)
+        return base64.b64encode(buffer.getvalue()).decode()
+
+    elif isinstance(image, np.ndarray):
+        # Handle numpy array: convert to PIL Image, then encode as PNG
+        pil_image = Image.fromarray(image)
+        buffer = io.BytesIO()
+        pil_image.save(buffer, format="PNG")
+        return base64.b64encode(buffer.getvalue()).decode()
+
+    raise ValueError(f"Unsupported image type: {type(image)}")
+
+
+def encode_image(image: str, errors="strict") -> bytes:
+    """Decode a base64 encoded image string back to bytes.
+
+    This function converts a base64 encoded string representation of an image
+    back into its original binary bytes format.
+
+    Args:
+        image: Base64 encoded string representation of an image
+        errors: Error handling scheme for decoding errors:
+            - 'strict' (default): Raise binascii.Error on decoding errors
+            - 'ignore': Return empty bytes on decoding errors
+            - Any other name registered with codecs.register_error
+
+    Returns:
+        bytes: Decoded image bytes, or empty bytes if errors='ignore' and decoding fails
+
+    Raises:
+        binascii.Error: If decoding fails and errors='strict'
+
+    Example:
+        >>> base64_str = "iVBORw0KGgoAAAANSUhEUgAAAAUA..."
+        >>> image_bytes = encode_image(base64_str)
+        >>> # With error handling
+        >>> image_bytes = encode_image(base64_str, errors="ignore")
+    """
+    try:
+        # Attempt to decode the base64 string to bytes
+        image_bytes = base64.b64decode(image)
+    except binascii.Error as e:
+        # Handle decoding errors based on the errors parameter
+        if errors == "ignore":
+            return b""
+        else:
+            raise e
+    return image_bytes
+
+
+def encode_bytes(content: str) -> bytes:
+    """Convert a string to bytes using UTF-8 encoding.
+
+    Args:
+        content: String to be encoded
+
+    Returns:
+        bytes: UTF-8 encoded bytes representation of the string
+
+    Example:
+        >>> text = "Hello, 世界"
+        >>> encoded = encode_bytes(text)
+        >>> type(encoded)
+        <class 'bytes'>
+    """
+    return content.encode()
+
+
+def decode_bytes(
+    content: bytes,
+    encodings: List[str] = [
+        "utf-8",
+        "gb18030",
+        "gb2312",
+        "gbk",
+        "big5",
+        "ascii",
+        "latin-1",
+    ],
+) -> str:
+    """Decode bytes to string with automatic encoding detection.
+
+    This function attempts to decode bytes using multiple encoding formats
+    in order of priority. It's particularly useful for handling text files
+    with unknown or mixed encodings, especially for Chinese text.
+
+    The function tries encodings in the provided order and returns the first
+    successful decode. If all encodings fail, it falls back to latin-1 with
+    error replacement to ensure a result is always returned.
+
+    Args:
+        content: Bytes content to be decoded
+        encodings: List of encoding formats to try, in order of priority.
+            Default includes common encodings for Chinese and Western text:
+            - utf-8: Universal encoding (tried first)
+            - gb18030, gb2312, gbk: Chinese encodings (Simplified)
+            - big5: Chinese encoding (Traditional)
+            - ascii, latin-1: Western encodings
+
+    Returns:
+        str: Decoded string content
+
+    Note:
+        - If all encodings fail, latin-1 with error='replace' is used as fallback
+        - The fallback may result in character replacement (�) for invalid bytes
+        - A warning is logged when fallback encoding is used
+
+    Example:
+        >>> # Decode with default encodings
+        >>> text = decode_bytes(b"\\xe4\\xb8\\xad\\xe6\\x96\\x87")  # UTF-8 Chinese
+        >>> print(text)
+        中文
+        >>> # Decode with custom encodings
+        >>> text = decode_bytes(content, encodings=["utf-8", "gbk"])
+    """
+    # Try decoding with each encoding format in order
+    for encoding in encodings:
+        try:
+            text = content.decode(encoding)
+            logger.debug(f"Decode content with {encoding}: {len(text)} characters")
+            return text
+        except UnicodeDecodeError:
+            # This encoding didn't work, try the next one
+            continue
+
+    # Fallback: use latin-1 with error replacement if all encodings fail
+    # latin-1 can decode any byte sequence, but may produce incorrect characters
+    text = content.decode(encoding="latin-1", errors="replace")
+    logger.warning(
+        "Unable to determine correct encoding, using latin-1 as fallback. "
+        "This may cause character issues."
+    )
+    return text
+
+
+if __name__ == "__main__":
+    # Example: Test encode_image with error handling
+    # This demonstrates decoding a base64 string with 'ignore' error mode
+    img = "test![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAMgA)test"
+    encode_image(img, errors="ignore")

docreader/utils/request.py

@@ -0,0 +1,149 @@
+import contextlib
+import logging
+import time
+import uuid
+from contextvars import ContextVar
+from logging import LogRecord
+from typing import Optional
+
+# 配置日志
+logger = logging.getLogger(__name__)
+
+# 定义上下文变量
+request_id_var = ContextVar("request_id", default=None)
+_request_start_time_ctx = ContextVar("request_start_time", default=None)
+
+
+def set_request_id(request_id: str) -> None:
+    """设置当前上下文的请求ID"""
+    request_id_var.set(request_id)
+
+
+def get_request_id() -> Optional[str]:
+    """获取当前上下文的请求ID"""
+    return request_id_var.get()
+
+
+class MillisecondFormatter(logging.Formatter):
+    """自定义日志格式化器，只显示毫秒级时间戳(3位数字)而不是微秒(6位)"""
+
+    def formatTime(self, record, datefmt=None):
+        """重写formatTime方法，将微秒格式化为毫秒"""
+        # 先获取标准的格式化时间
+        result = super().formatTime(record, datefmt)
+
+        # 如果使用了包含.%f的格式，则将微秒(6位)截断为毫秒(3位)
+        if datefmt and ".%f" in datefmt:
+            # 格式化的时间字符串应该在最后有6位微秒数
+            parts = result.split(".")
+            if len(parts) > 1 and len(parts[1]) >= 6:
+                # 只保留前3位作为毫秒
+                millis = parts[1][:3]
+                result = f"{parts[0]}.{millis}"
+
+        return result
+
+
+def init_logging_request_id():
+    """
+    Initialize logging to include request ID in log messages.
+    Add the custom filter to all existing handlers
+    """
+    logger.info("Initializing request ID logging")
+    root_logger = logging.getLogger()
+
+    # 添加自定义过滤器到所有处理器
+    for handler in root_logger.handlers:
+        # 添加请求ID过滤器
+        handler.addFilter(RequestIdFilter())
+
+        # 更新格式化器以包含请求ID，调整格式使其更紧凑整齐
+        formatter = logging.Formatter(
+            fmt="%(asctime)s.%(msecs)03d [%(request_id)s] %(levelname)-5s %(name)-20s | %(message)s",
+            datefmt="%Y-%m-%d %H:%M:%S",
+        )
+        handler.setFormatter(formatter)
+
+    logger.info(
+        f"Updated {len(root_logger.handlers)} handlers with request ID formatting"
+    )
+
+    # 如果没有处理器，添加一个标准输出处理器
+    if not root_logger.handlers:
+        handler = logging.StreamHandler()
+        formatter = logging.Formatter(
+            fmt="%(asctime)s.%(msecs)03d [%(request_id)s] %(levelname)-5s %(name)-20s | %(message)s",
+            datefmt="%Y-%m-%d %H:%M:%S",
+        )
+        handler.setFormatter(formatter)
+        handler.addFilter(RequestIdFilter())
+        root_logger.addHandler(handler)
+        logger.info("Added new StreamHandler with request ID formatting")
+
+
+class RequestIdFilter(logging.Filter):
+    """Filter that adds request ID to log messages"""
+
+    def filter(self, record: LogRecord) -> bool:
+        request_id = request_id_var.get()
+        if request_id is not None:
+            # 为日志记录添加请求ID属性，使用短格式
+            if len(request_id) > 8:
+                # 截取ID的前8个字符，确保显示整齐
+                short_id = request_id[:8]
+                if "-" in request_id:
+                    # 尝试保留格式，例如 test-req-1-XXX
+                    parts = request_id.split("-")
+                    if len(parts) >= 3:
+                        # 如果格式是 xxx-xxx-n-randompart
+                        short_id = f"{parts[0]}-{parts[1]}-{parts[2]}"
+                record.request_id = short_id
+            else:
+                record.request_id = request_id
+
+            # 添加执行时间属性
+            start_time = _request_start_time_ctx.get()
+            if start_time is not None:
+                elapsed_ms = int((time.time() - start_time) * 1000)
+                record.elapsed_ms = elapsed_ms
+                # 添加执行时间到消息中
+                if not hasattr(record, "message_with_elapsed"):
+                    record.message_with_elapsed = True
+                    record.msg = f"{record.msg} (elapsed: {elapsed_ms}ms)"
+        else:
+            # 如果没有请求ID，使用占位符
+            record.request_id = "no-req-id"
+
+        return True
+
+
+@contextlib.contextmanager
+def request_id_context(request_id: str = None):
+    """Context manager that sets a request ID for the current context
+
+    Args:
+        request_id: 要使用的请求ID，如果为None则自动生成
+
+    Example:
+        with request_id_context("req-123"):
+            # 在这个代码块中的所有日志都会包含请求ID req-123
+            logging.info("Processing request")
+    """
+    # Generate or use provided request ID
+    req_id = request_id or str(uuid.uuid4())
+
+    # Set start time and request ID
+    start_time = time.time()
+    req_token = request_id_var.set(req_id)
+    time_token = _request_start_time_ctx.set(start_time)
+
+    logger.info(f"Starting new request with ID: {req_id}")
+
+    try:
+        yield request_id_var.get()
+    finally:
+        # Log completion and reset context vars
+        elapsed_ms = int((time.time() - start_time) * 1000)
+        logger.info(f"Request {req_id} completed in {elapsed_ms}ms")
+        request_id_var.reset(req_token)
+        _request_start_time_ctx.reset(time_token)

docreader/utils/split.py

@@ -0,0 +1,80 @@
+import re
+from typing import Callable, List
+
+
+def split_text_keep_separator(text: str, separator: str) -> List[str]:
+    """Split text with separator and keep the separator at the end of each split.
+    
+    Args:
+        text: The input text to split
+        separator: The separator string to split by
+        
+    Returns:
+        List of text chunks with separator preserved at the start of each chunk (except first)
+        
+    Example:
+        >>> split_text_keep_separator("Hello\nWorld\nTest", "\n")
+        ["Hello", "\nWorld", "\nTest"]
+    """
+    # Split text by separator
+    parts = text.split(separator)
+    # Add separator back to the beginning of each part (except the first one)
+    result = [separator + s if i > 0 else s for i, s in enumerate(parts)]
+    # Filter out empty strings
+    return [s for s in result if s]
+
+
+def split_by_sep(sep: str, keep_sep: bool = True) -> Callable[[str], List[str]]:
+    """Create a function that splits text by a given separator.
+    
+    Args:
+        sep: The separator string to split by
+        keep_sep: If True, keep the separator in the result; if False, discard it
+        
+    Returns:
+        A callable function that takes text and returns a list of split strings
+    """
+    if keep_sep:
+        return lambda text: split_text_keep_separator(text, sep)
+    else:
+        return lambda text: text.split(sep)
+
+
+def split_by_char() -> Callable[[str], List[str]]:
+    """Create a function that splits text into individual characters.
+    
+    Returns:
+        A callable function that takes text and returns a list of characters
+    """
+    return lambda text: list(text)
+
+
+def split_by_regex(regex: str) -> Callable[[str], List[str]]:
+    """Create a function that splits text by a regex pattern.
+    
+    Args:
+        regex: The regular expression pattern to split by
+        
+    Returns:
+        A callable function that takes text and returns a list of split strings
+        The regex pattern is captured, so the separators are included in the result
+    """
+    # Compile regex with capturing group to keep separators in result
+    pattern = re.compile(f"({regex})")
+    # Split by pattern and filter out None/empty values
+    return lambda text: list(filter(None, pattern.split(text)))
+
+
+def match_by_regex(regex: str) -> Callable[[str], bool]:
+    """Create a function that checks if text matches a regex pattern.
+    
+    Args:
+        regex: The regular expression pattern to match against
+        
+    Returns:
+        A callable function that takes text and returns True if it matches the pattern
+    """
+    # Compile the regex pattern for efficient reuse
+    pattern = re.compile(regex)
+    # Return a function that checks if text matches the pattern from the start
+    return lambda text: bool(pattern.match(text))

docreader/utils/ssrf.py

@@ -0,0 +1,285 @@
+"""SSRF URL validation for docreader outbound HTTP requests.
+
+Mirrors the core policy in internal/utils/security.go so redirect targets
+during Playwright navigation are blocked the same way as Go-side imports.
+"""
+
+from __future__ import annotations
+
+import ipaddress
+import os
+import re
+import socket
+from functools import lru_cache
+from typing import FrozenSet, Optional, Tuple, Union
+from urllib.parse import urlparse
+
+RESTRICTED_HOSTNAMES: FrozenSet[str] = frozenset(
+    {
+        "localhost",
+        "127.0.0.1",
+        "::1",
+        "0.0.0.0",
+        "metadata.google.internal",
+        "metadata.tencentyun.com",
+        "metadata.aws.internal",
+        "host.docker.internal",
+        "gateway.docker.internal",
+        "kubernetes.docker.internal",
+        "kubernetes",
+        "kubernetes.default",
+        "kubernetes.default.svc",
+        "kubernetes.default.svc.cluster.local",
+    }
+)
+
+RESTRICTED_SUFFIXES: Tuple[str, ...] = (
+    ".local",
+    ".localhost",
+    ".internal",
+    ".corp",
+    ".lan",
+    ".home",
+    ".localdomain",
+    ".svc.cluster.local",
+    ".pod.cluster.local",
+)
+
+EXTRA_RESTRICTED_CIDRS: Tuple[Union[ipaddress.IPv4Network, ipaddress.IPv6Network], ...] = tuple(
+    ipaddress.ip_network(cidr)
+    for cidr in (
+        "100.64.0.0/10",
+        "198.18.0.0/15",
+        "198.51.100.0/24",
+        "203.0.113.0/24",
+        "192.0.0.0/24",
+        "192.0.2.0/24",
+        "0.0.0.0/8",
+        "240.0.0.0/4",
+        "255.255.255.255/32",
+        "172.17.0.0/16",
+        "172.18.0.0/16",
+        "172.19.0.0/16",
+        "172.20.0.0/16",
+    )
+)
+
+BLOCKED_PORTS: FrozenSet[str] = frozenset(
+    {
+        "22",
+        "23",
+        "25",
+        "445",
+        "3389",
+        "5432",
+        "3306",
+        "6379",
+        "27017",
+        "9200",
+        "2379",
+        "2380",
+        "8500",
+        "4001",
+    }
+)
+
+_IP_LIKE_PATTERNS = (
+    re.compile(r"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$"),
+    re.compile(r"^\d{8,10}$"),
+    re.compile(r"^0[0-7]+\."),
+    re.compile(r"(?i)^0x[0-9a-f]+\."),
+    re.compile(r"(?i)^0x[0-9a-f]{6,8}$"),
+    re.compile(r"(?i)^[0-9a-f:]+::[0-9a-f:]*$"),
+    re.compile(r"(?i)^[0-9a-f]{1,4}(:[0-9a-f]{1,4}){7}$"),
+)
+
+
+def _normalize_url(raw_url: str) -> str:
+    if "://" not in raw_url:
+        return f"https://{raw_url}"
+    return raw_url
+
+
+@lru_cache(maxsize=1)
+def _load_whitelist() -> Tuple[FrozenSet[str], Tuple[str, ...], Tuple[Union[ipaddress.IPv4Network, ipaddress.IPv6Network], ...]]:
+    entries: list[str] = []
+    for env_key in ("SSRF_WHITELIST", "SSRF_WHITELIST_EXTRA"):
+        raw = os.environ.get(env_key, "")
+        if raw.strip():
+            entries.extend(part.strip() for part in raw.split(",") if part.strip())
+
+    exact_hosts: set[str] = set()
+    suffix_hosts: list[str] = []
+    cidr_nets: list[Union[ipaddress.IPv4Network, ipaddress.IPv6Network]] = []
+    for entry in entries:
+        lowered = entry.lower()
+        if lowered.startswith("*."):
+            suffix_hosts.append(lowered[1:])
+            continue
+        if "/" in lowered:
+            try:
+                cidr_nets.append(ipaddress.ip_network(lowered, strict=False))
+            except ValueError:
+                continue
+            continue
+        exact_hosts.add(lowered)
+    return frozenset(exact_hosts), tuple(suffix_hosts), tuple(cidr_nets)
+
+
+def _is_whitelisted(hostname: str) -> bool:
+    lowered = hostname.lower()
+    exact_hosts, suffix_hosts, cidr_nets = _load_whitelist()
+    if lowered in exact_hosts:
+        return True
+    for suffix in suffix_hosts:
+        if lowered.endswith(suffix) or lowered == suffix.lstrip("."):
+            return True
+    try:
+        ip = ipaddress.ip_address(lowered)
+    except ValueError:
+        return False
+    return any(ip in net for net in cidr_nets)
+
+
+def _is_ip_like_hostname(hostname: str) -> bool:
+    return any(pattern.search(hostname) for pattern in _IP_LIKE_PATTERNS)
+
+
+def _is_restricted_ip(ip: Union[ipaddress.IPv4Address, ipaddress.IPv6Address]) -> Optional[str]:
+    if ip.is_private:
+        return "private IP address"
+    if ip.is_loopback:
+        return "loopback address"
+    if ip.is_link_local:
+        return "link-local address"
+    if ip.is_multicast:
+        return "multicast address"
+    if ip.is_unspecified:
+        return "unspecified address"
+    if isinstance(ip, ipaddress.IPv4Address):
+        for net in EXTRA_RESTRICTED_CIDRS:
+            if isinstance(net, ipaddress.IPv4Network) and ip in net:
+                return f"restricted range {net}"
+    if isinstance(ip, ipaddress.IPv6Address):
+        embedded_ipv4 = ip.ipv4_mapped
+        if embedded_ipv4 is not None:
+            reason = _is_restricted_ip(embedded_ipv4)
+            if reason:
+                return f"IPv4-mapped {reason}"
+        if ip.sixtofour is not None:
+            reason = _is_restricted_ip(ip.sixtofour)
+            if reason:
+                return f"6to4-embedded {reason}"
+        if ip.teredo is not None:
+            server_ip, client_ip = ip.teredo
+            for label, embedded_ip in (("server", server_ip), ("client", client_ip)):
+                reason = _is_restricted_ip(embedded_ip)
+                if reason:
+                    return f"Teredo {label} embeds {reason}"
+        # NAT64's well-known prefix (64:ff9b::/96, RFC 6052) and the deprecated
+        # IPv4-compatible form (::a.b.c.d) also carry a plain IPv4 address, but
+        # ipaddress exposes no accessor for either, so is_private and
+        # .sixtofour never see the payload. Without these two branches
+        # ::169.254.169.254 reads as an ordinary public IPv6 address.
+        # Mirrors internal/ipclass on the Go side.
+        packed = ip.packed
+        if packed[0:4] == b"\x00\x64\xff\x9b" and packed[4:12] == bytes(8):
+            reason = _is_restricted_ip(ipaddress.IPv4Address(packed[12:16]))
+            if reason:
+                return f"NAT64-embedded {reason}"
+        if packed[0:12] == bytes(12):
+            # :: and ::1 already returned above; ::ffff:a.b.c.d is handled by
+            # the ipv4_mapped branch.
+            reason = _is_restricted_ip(ipaddress.IPv4Address(packed[12:16]))
+            if reason:
+                return f"IPv4-compatible {reason}"
+        # Site-local (fec0::/10)
+        if (ip.packed[0] == 0xFE) and (ip.packed[1] & 0xC0) == 0xC0:
+            return "site-local IPv6 address"
+    return None
+
+
+def _resolve_host_ips(hostname: str) -> Tuple[Tuple[Union[ipaddress.IPv4Address, ipaddress.IPv6Address], ...], Optional[str]]:
+    try:
+        infos = socket.getaddrinfo(hostname, None, type=socket.SOCK_STREAM)
+    except socket.gaierror as exc:
+        return (), f"DNS resolution failed for hostname {hostname}: {exc}"
+    ips: list[Union[ipaddress.IPv4Address, ipaddress.IPv6Address]] = []
+    seen: set[str] = set()
+    for info in infos:
+        sockaddr = info[4]
+        if not sockaddr:
+            continue
+        ip_str = sockaddr[0]
+        if ip_str in seen:
+            continue
+        seen.add(ip_str)
+        try:
+            ips.append(ipaddress.ip_address(ip_str))
+        except ValueError:
+            continue
+    if not ips:
+        return (), f"DNS resolution failed for hostname {hostname}: no addresses"
+    return tuple(ips), None
+
+
+def is_ssrf_safe_url(raw_url: str) -> Tuple[bool, str]:
+    """Return (safe, reason). reason is empty when safe is True."""
+    if not raw_url or not raw_url.strip():
+        return False, "URL is empty"
+
+    normalized = _normalize_url(raw_url.strip())
+    parsed = urlparse(normalized)
+    scheme = (parsed.scheme or "").lower()
+    if scheme not in {"http", "https"}:
+        return False, f"invalid scheme: {scheme or '(none)'} (only http/https allowed)"
+
+    hostname = (parsed.hostname or "").strip()
+    if not hostname:
+        return False, "URL has no hostname"
+
+    hostname_lower = hostname.lower()
+    if _is_whitelisted(hostname_lower):
+        return True, ""
+
+    if hostname_lower in RESTRICTED_HOSTNAMES:
+        return False, f"hostname {hostname_lower} is restricted"
+
+    for suffix in RESTRICTED_SUFFIXES:
+        if hostname_lower.endswith(suffix):
+            return False, f"hostname suffix {suffix} is restricted"
+
+    try:
+        ipaddress.ip_address(hostname_lower)
+        return False, "direct IP address access is not allowed, use domain name or add to SSRF_WHITELIST"
+    except ValueError:
+        pass
+
+    if _is_ip_like_hostname(hostname_lower):
+        return False, "IP-like hostname format is not allowed"
+
+    resolved_ips, resolve_err = _resolve_host_ips(hostname_lower)
+    if resolve_err:
+        return False, resolve_err
+
+    for resolved_ip in resolved_ips:
+        reason = _is_restricted_ip(resolved_ip)
+        if reason:
+            return (
+                False,
+                f"hostname {hostname_lower} resolves to restricted IP {resolved_ip}: {reason}",
+            )
+
+    try:
+        port = parsed.port
+    except ValueError as exc:
+        return False, f"invalid port: {exc}"
+    if port is not None and str(port) in BLOCKED_PORTS:
+        return False, f"port {port} is blocked for security reasons"
+
+    return True, ""
+
+
+def reset_ssrf_whitelist_cache_for_test() -> None:
+    """Clear cached whitelist entries (for unit tests only)."""
+    _load_whitelist.cache_clear()

docreader/utils/tempfile.py

@@ -0,0 +1,77 @@
+import logging
+import os
+import tempfile
+
+logger = logging.getLogger(__name__)
+
+
+class TempFileContext:
+    def __init__(self, file_content: bytes, suffix: str):
+        """
+        Initialize the context
+        :param file_content: Byte data to write to file
+        :param suffix: File suffix
+        """
+        self.file_content = file_content
+        self.suffix = suffix
+        self.file = None
+
+    def __enter__(self):
+        """
+        Create file when entering context
+        """
+        self.temp_file = tempfile.NamedTemporaryFile(suffix=self.suffix, delete=False)
+        self.temp_file.write(self.file_content)
+        self.temp_file.flush()
+        logger.info(
+            f"Saved {self.suffix} content to temporary file: {self.temp_file.name}"
+        )
+        return self.temp_file.name
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        """
+        Delete file when exiting context
+        """
+        if self.temp_file:
+            self.temp_file.close()
+            if os.path.exists(self.temp_file.name):
+                os.remove(self.temp_file.name)
+            logger.info(f"File {self.temp_file.name} has been deleted.")
+        # Return False to propagate exception (if any exception occurred)
+        return False
+
+
+class TempDirContext:
+    def __init__(self):
+        """
+        Initialize the context
+        """
+        self.temp_dir = None
+
+    def __enter__(self):
+        """
+        Create directory when entering context
+        """
+        self.temp_dir = tempfile.TemporaryDirectory()
+        logger.info(f"Created temporary directory: {self.temp_dir.name}")
+        return self.temp_dir.name
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        """
+        Delete directory when exiting context
+        """
+        if self.temp_dir and os.path.exists(self.temp_dir.name):
+            self.temp_dir.cleanup()
+            logger.info(f"Directory {self.temp_dir.name} has been deleted.")
+        # Return False to propagate exception (if any exception occurred)
+        return False
+
+
+if __name__ == "__main__":
+    example_bytes = b"Hello, this is a test file."
+    file_name = "test_file.txt"
+
+    # Using with statement
+    with TempFileContext(example_bytes, file_name) as temp_file:
+        # File operations can be performed within the context
+        print(f"Does file {file_name} exist: {os.path.exists(file_name)}")