WeKnora deployment source with the reviewed wiki identity, summary and recovery fixes
Anydoc / anydoc parser engine (push) Canceled after 0s
App / Format, vet, test, and build (push) Canceled after 0s
CLI / test (macos-latest / go 1.26) (push) Canceled after 0s
CLI / test (ubuntu-latest / go 1.26) (push) Canceled after 0s
CLI / test (windows-latest / go 1.26) (push) Canceled after 0s
Build and Push Docker Image / build-ui (push) Canceled after 0s
Build and Push Docker Image / build-docreader (push) Canceled after 0s
Build and Push Docker Image / build-sandbox (push) Canceled after 0s
Build and Push Docker Image / build-app (amd64, linux/amd64, ubuntu-latest) (push) Canceled after 0s
Build and Push Docker Image / build-app (arm64, linux/arm64, ubuntu-24.04-arm) (push) Canceled after 0s
DocReader / Python tests and Go client tests (push) Canceled after 0s
Build & Publish (dsh-weknora) / Typecheck, test, pack (push) Canceled after 0s
Build & Publish (dsh-weknora) / End-to-end inside dsh (pinned) (push) Canceled after 0s
Build & Publish (dsh-weknora) / End-to-end inside dsh (latest, advisory) (push) Canceled after 0s
Frontend / Test, type-check, and build (push) Canceled after 0s
Go Lint Cache / warm lint cache (.) (push) Canceled after 0s
Go Lint Cache / warm lint cache (cli) (push) Canceled after 0s
Go Lint Cache / warm lint cache (client) (push) Canceled after 0s
Build & Publish (mcp-server) / Test (Python 3.10) (push) Canceled after 0s
Build & Publish (mcp-server) / Test (Python 3.11) (push) Canceled after 0s
Build & Publish (mcp-server) / Test (Python 3.12) (push) Canceled after 0s
Build & Publish (mcp-server) / Test (Python 3.13) (push) Canceled after 0s
Build and Push Docker Image / merge (push) Canceled after 0s
Build & Publish (dsh-weknora) / Publish to npm (push) Canceled after 0s
Build & Publish (mcp-server) / Build sdist + wheel (push) Canceled after 0s
Build & Publish (mcp-server) / Publish to PyPI (push) Canceled after 0s

This commit is contained in:
yawan
2026-09-22 16:19:06 +08:00
commit f7fc992f57
3929 changed files with 1006607 additions and 0 deletions
+3
View File
@@ -0,0 +1,3 @@
target/
lib/
patched-anydoc/
+1526
View File
File diff suppressed because it is too large Load Diff
+39
View File
@@ -0,0 +1,39 @@
# Go bindings for anydoc.
#
# `crate-type = ["staticlib"]` produces libanydoc_go.a, which the Go module
# links at build time via cgo. Not a cdylib: cgo wants an archive it can pull
# symbols from, not a loadable shared object.
[package]
name = "anydoc-go"
version = "0.1.9"
edition = "2024"
description = "Go bindings (C ABI) for anydoc"
license = "MIT"
repository = "https://github.com/firecrawl/anydoc"
publish = false
[lib]
crate-type = ["staticlib"]
# Standalone crate: it lives inside the WeKnora tree, not the anydoc
# workspace, so it takes the published crate rather than a path dependency.
# Pinned exactly: the C ABI mirrors this version's error and format enums.
[dependencies]
anydoc = "=0.1.9"
[build-dependencies]
cbindgen = "0.29"
# document_to_markdown is crate-private in published anydoc. The build script
# copies the pinned version above and re-exports that one function so we can
# keep the official GFM serializer after rewriting asset images to External URLs.
[patch.crates-io]
anydoc = { path = "patched-anydoc" }
[workspace]
# Mirrors the anydoc workspace's release profile: without it the archive keeps
# every debug symbol and roughly doubles in size.
[profile.release]
lto = "thin"
strip = "symbols"
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Sideguide Technologies Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+197
View File
@@ -0,0 +1,197 @@
# anydoc-go (vendored)
Go bindings for [anydoc](https://github.com/firecrawl/anydoc), the Rust library
that converts Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV, and PDF
documents to GitHub-Flavored Markdown. WeKnora links them through cgo so office
documents can be parsed inside the Go process, without the Python docreader
service.
## Why this is vendored
The bindings come from the open pull request
[firecrawl/anydoc#30](https://github.com/firecrawl/anydoc/pull/30). Until it
merges and upstream tags `go/vX.Y.Z`, there is no module to `go get`: the PR
also expects a maintainer-only workflow to commit per-platform archives that do
not exist yet. So the binding source lives here and `go.mod` points at it:
```
replace github.com/firecrawl/anydoc/go => ./third_party/anydoc-go
```
When the upstream module is published, delete this directory, drop the
`replace`, and require the published version. Nothing else in WeKnora changes:
only `internal/infrastructure/docparser/anydoc/backend_cgo.go` imports it.
## Provenance
| | |
| --- | --- |
| Upstream PR | firecrawl/anydoc#30 ("feat: add Go bindings") |
| PR head | `1a7a6c0` |
| Rebased onto | `4e3089b` (`chore: release v0.1.8`) |
| anydoc crate | `0.1.9` (from crates.io, pinned with `=`) |
| License | MIT (see LICENSE) |
The PR branched before anydoc 0.1.7, so it was rebased onto v0.1.8 before
vendoring. The rebase conflicts were all release bookkeeping — the workspace
member list, the README binding sections, and the version-agreement script,
which the wasm bindings had touched in the meantime — plus the binding's own
version, bumped from 0.1.3 to 0.1.8. Taking v0.1.8 also picks up the
`pdf-inspector` 0.1.8 bump that fixes [RUSTSEC-2026-0187](https://rustsec.org/advisories/RUSTSEC-2026-0187.html),
the `lopdf` stack overflow that aborts the process on a hostile PDF.
The pin then moved to 0.1.9, which is that same binding source against a newer
PDF stack: anydoc's `src/` is byte-identical between the two releases, so the C
ABI, the document model, and the GFM serializer this binding reaches into are
unchanged, and none of the local modifications below had to be re-applied. What
0.1.9 does change is `pdf-inspector`, from the crate published as 0.1.8 to
1.14.2 (the jump in the number is that project unifying its Rust, Python, and
Node versions, not fourteen major releases). See "Why 0.1.9 matters here" below.
## Why 0.1.9 matters here
anydoc's changelog lists 0.1.9 as a single dependency bump, which reads like
housekeeping. It is nine `pdf-inspector` fixes, and seven of them are
denial-of-service bounds on PDF parsing.
Those seven bound work that had no bound at all: Form XObject expansion per
page, CID `/W` ranges, Encoding `cidrange` and ToUnicode `bfrange` expansion,
content-stream decode before operators are allocated, the detector's `Tj`/`TJ`
operand lookback, and disjoint-rect table clustering. Every one sits on the path
`pdf_inspector::process_pdf_mem` takes — the only way anydoc converts a PDF —
and every one is driven by nothing but the bytes of the uploaded file.
This is the class of bug the `guarded()` panic catcher explicitly cannot
contain, for the same reason `RUSTSEC-2026-0187` could not be: unbounded CPU
never panics, and an allocation failure aborts. The detector lookback is the
cheapest to demonstrate — before 1.14.2, each `Tj`/`TJ` whose operand was
missing rescanned the whole content stream, so a stream of bare `] TJ` tokens
was quadratic:
| `] TJ` tokens | PDF size | 0.1.8 | 0.1.9 |
| --- | --- | --- | --- |
| 40,000 | 195 KB | 1.0 s | 2.0 ms |
| 120,000 | 586 KB | 9.7 s | 4.1 ms |
| 200,000 | 977 KB | 25.9 s | 6.5 ms |
| 400,000 | 1.9 MB | 105.6 s | 12.2 ms |
One run on a 4-core VM, same Go code either side, only the linked archive
differing. Quadrupling the input cost 16× the time before the fix and roughly
4× after. A 2 MB upload holding a core for nearly two minutes needs no
privilege and no malformed container — the file parses fine, it just takes
forever, and every concurrent upload takes a core of its own.
`TestDetectorLookbackStaysLinear` in `internal/infrastructure/docparser/anydoc`
fails if a bump reintroduces it.
Two commits improve extraction rather than bound it, and both change what gets
indexed: text inside Form XObjects now tracks the text line matrix and honors
`T*`, `TL`, `'`, `"`, `Tc`, and `Tw`, so nested form text keeps its line breaks
and spacing instead of running together, and small-caps runs merge instead of
being read as extra table columns — a spurious column reaches the model as a
misaligned markdown table.
Nothing else in the range reaches WeKnora, which is worth recording so the next
upgrade does not go looking for it. The crate published as 0.1.8 was cut at
`pdf-inspector`'s `packages-2026-08-10` tag, one release behind 1.14.1, so 0.1.9
formally spans that release too — but its two functional commits both miss this
binding. One only touches the Node and Python surfaces. The other serves
invisible (`Tr 3`) OCR text layers instead of reporting `needs_ocr`, which
sounds like it would spare WeKnora an OCR round trip, except that it landed in
`extract_text_in_regions`; anydoc uses the positioned-text path, which already
retried with invisible text included. Scanned PDFs still come back as "OCR is
required" and still fall back to the docreader.
## Local modifications
Keep this list current: it is the diff a future upgrade has to re-apply. Items
2–4 are bugs in the upstream PR and are worth sending back to it.
1. `Cargo.toml` — depends on the published `anydoc = "=0.1.9"` crate instead of
the workspace path dependency, declares its own empty `[workspace]`, and
repeats the upstream release profile (`lto`, `strip`), which it would
otherwise inherit from the anydoc workspace.
2. `anydoc.go` — every ABI call runs inside `call()`, which pins the goroutine
with `runtime.LockOSThread` for the duration. The ABI reports the error
message through a thread-local slot that a *second* call
(`anydoc_last_error`) reads, and Go may resume a goroutine on a different OS
thread once a cgo call returns: without pinning, a failed conversion reports
an empty message, or one belonging to another document parsed on the thread
it landed on. Reproduced at roughly 1 in 1500 concurrent conversions; the
regression test is `TestErrorDetailSurvivesConcurrency` in
`internal/infrastructure/docparser/anydoc`.
3. `model.go` — decoder preallocations are bounded by the bytes left in the
buffer (`capFor`), and `need` rejects a negative length. A count taken
straight from the buffer is only trustworthy while the Rust encoder and this
decoder agree; on a skew, `make([]Block, 0, n)` would exhaust memory before
the first bounds check, turning a version mismatch into a dead process.
4. `src/lib.rs` — the three conversion entry points run inside `guarded()`,
which catches a panic and reports it as a malformed document. A panic
escaping an `extern "C"` function aborts the process, and WeKnora parses
untrusted uploads in the same process that serves the API. Note the limit:
this cannot contain a stack overflow or an allocation failure, which is why
the dependency pin below matters as much as the guard.
5. Removed the upstream CLI (`cmd/anydoc`) and the binding test suite, which
reads fixtures from the anydoc repository. WeKnora's own tests live in
`internal/infrastructure/docparser/anydoc`.
6. `scripts/build-anydoc-lib.sh` copies the pinned anydoc release to
`patched-anydoc/` (gitignored) and re-exports `document_to_markdown`, which
is crate-private in the published crate. It reads the version from the
`anydoc = "=X.Y.Z"` pin above and checks it against `version.go`, so a bump
is one line and cannot half-land.
`src/asset_links.rs` then rewrites `ImageSource::Asset`
to `External("images/image-N.ext")` so the official serializer emits in-place
image links. `anydoc_to_markdown_with_asset_links` is the ABI for that path.
## Dependency pinning and audit
`Cargo.lock` is committed and `scripts/build-anydoc-lib.sh` builds with
`--locked`, so the archive is always the audited dependency tree. CI runs
`cargo audit` against it, because the crate that fails here is the one parsing
untrusted uploads inside the API process.
That matters concretely: with `lopdf` 0.41 — what the upstream PR's own
lockfile resolved to — a ~100 KB PDF holding a deeply nested catalog array
kills the process with a stack overflow (`RUSTSEC-2026-0187`), which neither
`guarded()` nor Go's `recover` can contain. anydoc 0.1.8 moved to `lopdf` 0.42
and the same input comes back as an ordinary error;
`TestDeeplyNestedPDFFailsWithoutKillingTheProcess` keeps it that way.
Moving the pin to 0.1.9 added no crate and removed none: `lopdf` stays at 0.42
and the whole tree below `pdf-inspector` is unchanged, so the lockfile diff is
three version lines and one checksum. The audit result is therefore the same
one described below.
`cargo audit` currently reports one allowed warning: `ttf-parser` 0.25.1 is
unmaintained (`RUSTSEC-2026-0192`), pulled in transitively by the PDF stack. It
is not a vulnerability and nothing here can fix it, so warnings report without
failing the job.
## Known upstream limitation
Markdown rendering drops embedded images: `ImageSource::Asset` renders as its
alt text, and the bytes are only reachable through the document model. The
renderer (`document_to_markdown`) is also crate-private. WeKnora keeps that
serializer: `scripts/build-anydoc-lib.sh` re-exports the one function, then
`anydoc_to_markdown_with_asset_links` rewrites `Asset` images to
`ImageSource::External("images/image-N.ext")` so the official GFM output
places them in reading order. PDF has no document model; scanned pages fall
back to the builtin docreader so they can be rasterized for OCR.
## Building the archive
cgo links `lib/<platform>/libanydoc_go.a`, which is a build artifact (~30 MB)
and is therefore git-ignored rather than committed:
```bash
scripts/build-anydoc-lib.sh # host platform
TARGET=aarch64-unknown-linux-musl scripts/build-anydoc-lib.sh
```
Then build WeKnora with the engine linked in:
```bash
make build-anydoc # or: go build -tags anydoc ./cmd/server
```
Builds without the `anydoc` tag need no Rust toolchain and no archive; the
engine simply reports itself as unavailable.
+195
View File
@@ -0,0 +1,195 @@
// Package anydoc converts documents (Word, PowerPoint, Excel, OpenDocument,
// RTF, EPUB, CSV, and PDF) to GitHub-Flavored Markdown, with full access to
// the parsed document model and embedded assets.
//
// This is the Go binding. It links a Rust static library through cgo; the
// module packages archives for every supported platform. See the README for
// the supported platform matrix and musl build tag.
package anydoc
/*
#cgo CFLAGS: -I${SRCDIR}/include
#cgo darwin,arm64 LDFLAGS: -L${SRCDIR}/lib/darwin_arm64 -lanydoc_go -lm -lstdc++
#cgo darwin,amd64 LDFLAGS: -L${SRCDIR}/lib/darwin_amd64 -lanydoc_go -lm -lstdc++
#cgo windows,amd64 LDFLAGS: -L${SRCDIR}/lib/windows_amd64 -lanydoc_go
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include "include/anydoc.h"
*/
import "C"
import (
"fmt"
"runtime"
"unsafe"
)
// call runs one ABI entry point and turns its status code into an error.
//
// The goroutine is pinned for the duration because the ABI reports the error
// message through a thread-local slot that a *second* call
// (`anydoc_last_error`) reads. Go is free to resume a goroutine on a different
// OS thread after a cgo call returns, and a conversion is long enough for that
// to happen: the message would then be missing, or belong to another
// conversion that ran on the thread we landed on.
func call(entry func() C.int) error {
runtime.LockOSThread()
defer runtime.UnlockOSThread()
if code := entry(); code != errOK {
return convertError(code)
}
return nil
}
// ToMarkdown converts a document file to Markdown. The format is detected
// from the file content; the extension is the fallback for signature-less
// formats (CSV) and unrecognizable containers.
func ToMarkdown(path string) (string, error) {
cpath := C.CString(path)
defer C.free(unsafe.Pointer(cpath))
var out *C.char
var outLen C.uintptr_t
if err := call(func() C.int { return C.anydoc_to_markdown(cpath, &out, &outLen) }); err != nil {
return "", err
}
defer C.anydoc_string_free(out)
markdown, err := cStringN(out, outLen)
if err != nil {
return "", err
}
return markdown, nil
}
// ToMarkdownBytes converts an in-memory document to Markdown. Pass a Format
// to select the parser, or nil to detect it from the content, which
// signature-less formats (CSV) have to name explicitly.
func ToMarkdownBytes(data []byte, format *Format) (string, error) {
if len(data) == 0 {
return "", &ConvertError{Kind: "unsupported", Detail: "empty input"}
}
tag := C.int(C.ANYDOC_FORMAT_NONE)
if format != nil {
tag = formatToTag(*format)
if tag == C.int(C.ANYDOC_FORMAT_NONE) {
return "", &ConvertError{Kind: "unknown_format", Detail: "unknown format: " + string(*format)}
}
}
var out *C.char
var outLen C.uintptr_t
if err := call(func() C.int {
return C.anydoc_to_markdown_bytes(
(*C.uint8_t)(unsafe.Pointer(&data[0])), C.uintptr_t(len(data)), tag, &out, &outLen,
)
}); err != nil {
return "", err
}
defer C.anydoc_string_free(out)
markdown, err := cStringN(out, outLen)
if err != nil {
return "", err
}
return markdown, nil
}
// ToMarkdownWithAssetLinks converts an in-memory document to Markdown with
// embedded images rewritten as `![alt](images/image-N.ext)` so they keep
// their original positions. The Markdown itself is produced by anydoc's
// official GFM serializer.
func ToMarkdownWithAssetLinks(data []byte, format *Format) (string, error) {
if len(data) == 0 {
return "", &ConvertError{Kind: "unsupported", Detail: "empty input"}
}
tag := C.int(C.ANYDOC_FORMAT_NONE)
if format != nil {
tag = formatToTag(*format)
if tag == C.int(C.ANYDOC_FORMAT_NONE) {
return "", &ConvertError{Kind: "unknown_format", Detail: "unknown format: " + string(*format)}
}
}
var out *C.char
var outLen C.uintptr_t
if err := call(func() C.int {
return C.anydoc_to_markdown_with_asset_links(
(*C.uint8_t)(unsafe.Pointer(&data[0])), C.uintptr_t(len(data)), tag, &out, &outLen,
)
}); err != nil {
return "", err
}
defer C.anydoc_string_free(out)
markdown, err := cStringN(out, outLen)
if err != nil {
return "", err
}
return markdown, nil
}
// ToDocument parses an in-memory document into the document model, which also
// carries the embedded assets. Pass a Format to select the parser, or nil to
// detect it from the content.
//
// Unsupported for PDF: PDF conversion produces Markdown directly and has no
// document-model form; use ToMarkdownBytes.
func ToDocument(data []byte, format *Format) (*Document, error) {
if len(data) == 0 {
return nil, &ConvertError{Kind: "unsupported", Detail: "empty input"}
}
tag := C.int(C.ANYDOC_FORMAT_NONE)
if format != nil {
tag = formatToTag(*format)
if tag == C.int(C.ANYDOC_FORMAT_NONE) {
return nil, &ConvertError{Kind: "unknown_format", Detail: "unknown format: " + string(*format)}
}
}
var buf *C.uint8_t
var bufLen C.uintptr_t
if err := call(func() C.int {
return C.anydoc_to_document(
(*C.uint8_t)(unsafe.Pointer(&data[0])), C.uintptr_t(len(data)), tag, &buf, &bufLen,
)
}); err != nil {
return nil, err
}
defer C.anydoc_buffer_free(buf, bufLen)
if buf == nil || bufLen == 0 {
return &Document{}, nil
}
// Copy the C buffer into a Go-owned slice before decoding, so the C
// buffer can be freed immediately rather than tying its lifetime to the
// decoded Document.
raw, err := cGoBytes(unsafe.Pointer(buf), bufLen)
if err != nil {
return nil, err
}
return decodeDocument(raw)
}
// cStringN reads a length-prefixed C buffer as a Go string. The bytes are
// copied, so the caller may free the C buffer immediately after.
func cStringN(s *C.char, n C.uintptr_t) (string, error) {
if s == nil || n == 0 {
return "", nil
}
b, err := cGoBytes(unsafe.Pointer(s), n)
if err != nil {
return "", err
}
return string(b), nil
}
func cGoBytes(ptr unsafe.Pointer, n C.uintptr_t) ([]byte, error) {
// C.GoBytes accepts a C int length even on 64-bit Go builds.
if uint64(n) > 1<<31-1 {
return nil, &ConvertError{Kind: "resource_limit", Detail: fmt.Sprintf("FFI output is too large for Go: %d bytes", n)}
}
return C.GoBytes(ptr, C.int(n)), nil
}
// cString reads a NUL-terminated C string as a Go string.
func cString(s *C.char) string {
if s == nil {
return ""
}
return C.GoString(s)
}
+38
View File
@@ -0,0 +1,38 @@
//! Regenerate `include/anydoc.h` from the crate's C ABI on every build.
//! `cbindgen` is a build-dependency, so this runs whenever `cargo build
//! -p anydoc-go` runs inside the workspace. The committed header is the
//! source of truth for the Go side; consumers link the packaged static archive
//! and never run this.
use std::path::PathBuf;
fn main() {
let manifest_dir = PathBuf::from(std::env::var("CARGO_MANIFEST_DIR").unwrap());
let include_dir = manifest_dir.join("include");
let header = include_dir.join("anydoc.h");
std::fs::create_dir_all(&include_dir).expect("create include directory");
let config = cbindgen::Config {
language: cbindgen::Language::C,
include_guard: Some("ANYDOC_H".to_string()),
autogen_warning: Some(
"/* Warning: this file is generated by `cargo build -p anydoc-go`. Do not edit by hand. */"
.to_string(),
),
..Default::default()
};
match cbindgen::Builder::new().with_crate(&manifest_dir).with_config(config).generate() {
Ok(bindings) => {
let _ = bindings.write_to_file(&header);
}
Err(err) => {
println!("cargo:warning=cbindgen failed to generate anydoc.h: {err}");
}
}
println!("cargo:rerun-if-changed=src/lib.rs");
println!("cargo:rerun-if-changed=src/model.rs");
println!("cargo:rerun-if-changed=src/asset_links.rs");
}
+95
View File
@@ -0,0 +1,95 @@
package anydoc
/*
#include <stdint.h>
#include "include/anydoc.h"
*/
import "C"
import "fmt"
// Error codes mirroring the C ABI (ERR_*). Stable across versions.
const (
errOK = C.ERR_OK
errUnsupported = C.ERR_UNSUPPORTED
errMalformed = C.ERR_MALFORMED
errEncrypted = C.ERR_ENCRYPTED
errResourceLimit = C.ERR_RESOURCE_LIMIT
errMissingPart = C.ERR_MISSING_PART
errIO = C.ERR_IO
errPDFNoModel = C.ERR_PDF_NO_MODEL
errInvalidArg = C.ERR_INVALID_ARG
errUnknownFormat = C.ERR_UNKNOWN_FORMAT
)
// ConvertError is the typed error every conversion function returns. It
// carries the same variant kind the Node and Python bindings expose
// ("unsupported", "malformed", "encrypted", ...) plus the crate's
// human-readable detail.
type ConvertError struct {
// Kind is the lowercase variant name, matching the Node and Python
// bindings: "unsupported", "malformed", "encrypted", "resource_limit",
// "missing_part", "io", "pdf_no_model". Go also reports
// "unknown_format" for an invalid explicit Format.
Kind string
// Detail is the crate's Display output for the error.
Detail string
}
func (e *ConvertError) Error() string {
if e == nil {
return "<nil>"
}
if e.Detail == "" {
return "anydoc: " + e.Kind
}
return "anydoc: " + e.Kind + ": " + e.Detail
}
// convertError builds a ConvertError from an ABI error code, pulling the
// detail message from the thread-local last-error slot.
func convertError(code C.int) error {
if code == errOK {
return nil
}
kind := errorKind(code)
detail := lastError()
return &ConvertError{Kind: kind, Detail: detail}
}
func errorKind(code C.int) string {
switch code {
case errUnsupported:
return "unsupported"
case errMalformed:
return "malformed"
case errEncrypted:
return "encrypted"
case errResourceLimit:
return "resource_limit"
case errMissingPart:
return "missing_part"
case errIO:
return "io"
case errPDFNoModel:
return "pdf_no_model"
case errInvalidArg:
return "invalid_argument"
case errUnknownFormat:
return "unknown_format"
default:
return fmt.Sprintf("unknown_error(%d)", int(code))
}
}
// lastError returns the human-readable message the last ABI call stashed on
// the current OS thread, or "" when there was none. The C side returns a
// freshly allocated string the caller must free.
func lastError() string {
ptr := C.anydoc_last_error()
if ptr == nil {
return ""
}
defer C.anydoc_string_free(ptr)
return cString(ptr)
}
+143
View File
@@ -0,0 +1,143 @@
package anydoc
/*
#include "include/anydoc.h"
*/
import "C"
import (
"strings"
"unsafe"
)
// Format names a document format. The string values are the lowercase
// extension names, matching the Node and Python bindings ("doc", "docx",
// "pdf", ...). Use the exported constants rather than raw strings where
// possible: the constants are pinned to the C ABI's tag values at build
// time.
type Format string
// Format constants. Every value is stable across versions.
const (
FormatDoc Format = "doc"
FormatDocx Format = "docx"
FormatOdt Format = "odt"
FormatPdf Format = "pdf"
FormatPpt Format = "ppt"
FormatPptx Format = "pptx"
FormatRtf Format = "rtf"
FormatEpub Format = "epub"
FormatXlsx Format = "xlsx"
FormatOds Format = "ods"
FormatOdp Format = "odp"
FormatCsv Format = "csv"
)
// formatToTag maps a Format to its C ABI tag. Returns ANYDOC_FORMAT_NONE for
// an unknown format.
func formatToTag(f Format) C.int {
switch f {
case FormatDoc:
return C.int(C.ANYDOC_FORMAT_DOC)
case FormatDocx:
return C.int(C.ANYDOC_FORMAT_DOCX)
case FormatOdt:
return C.int(C.ANYDOC_FORMAT_ODT)
case FormatPdf:
return C.int(C.ANYDOC_FORMAT_PDF)
case FormatPpt:
return C.int(C.ANYDOC_FORMAT_PPT)
case FormatPptx:
return C.int(C.ANYDOC_FORMAT_PPTX)
case FormatRtf:
return C.int(C.ANYDOC_FORMAT_RTF)
case FormatEpub:
return C.int(C.ANYDOC_FORMAT_EPUB)
case FormatXlsx:
return C.int(C.ANYDOC_FORMAT_XLSX)
case FormatOds:
return C.int(C.ANYDOC_FORMAT_ODS)
case FormatOdp:
return C.int(C.ANYDOC_FORMAT_ODP)
case FormatCsv:
return C.int(C.ANYDOC_FORMAT_CSV)
}
return C.int(C.ANYDOC_FORMAT_NONE)
}
// tagToFormat maps a C ABI tag back to a Format. ok is false for NONE or an
// unknown tag.
func tagToFormat(tag C.int) (Format, bool) {
switch tag {
case C.ANYDOC_FORMAT_DOC:
return FormatDoc, true
case C.ANYDOC_FORMAT_DOCX:
return FormatDocx, true
case C.ANYDOC_FORMAT_ODT:
return FormatOdt, true
case C.ANYDOC_FORMAT_PDF:
return FormatPdf, true
case C.ANYDOC_FORMAT_PPT:
return FormatPpt, true
case C.ANYDOC_FORMAT_PPTX:
return FormatPptx, true
case C.ANYDOC_FORMAT_RTF:
return FormatRtf, true
case C.ANYDOC_FORMAT_EPUB:
return FormatEpub, true
case C.ANYDOC_FORMAT_XLSX:
return FormatXlsx, true
case C.ANYDOC_FORMAT_ODS:
return FormatOds, true
case C.ANYDOC_FORMAT_ODP:
return FormatOdp, true
case C.ANYDOC_FORMAT_CSV:
return FormatCsv, true
}
return "", false
}
// FormatFromBytes detects the format from the content itself: the signature
// and identity each container specification designates (PDF header, RTF open
// group, OLE stream names, ZIP package mimetype/content types). Plain-text
// formats (CSV) carry no signature and return the zero Format and ok=false;
// so does anything unrecognized.
func FormatFromBytes(data []byte) (f Format, ok bool) {
if len(data) == 0 {
return "", false
}
var tag C.int
code := C.anydoc_format_from_bytes((*C.uint8_t)(unsafe.Pointer(&data[0])), C.uintptr_t(len(data)), &tag)
if code != errOK {
return "", false
}
return tagToFormat(tag)
}
// FormatFromExtension returns the format a bare extension names (with or
// without a leading dot), matched case-insensitively. ok is false for
// anything unrecognized.
func FormatFromExtension(ext string) (f Format, ok bool) {
ext = strings.TrimPrefix(ext, ".")
cext := C.CString(strings.ToLower(ext))
defer C.free(unsafe.Pointer(cext))
var tag C.int
code := C.anydoc_format_from_extension(cext, &tag)
if code != errOK {
return "", false
}
return tagToFormat(tag)
}
// FormatFromPath returns the format a path's extension names. ok is false
// when the path has no extension or names nothing recognized.
func FormatFromPath(path string) (f Format, ok bool) {
cpath := C.CString(path)
defer C.free(unsafe.Pointer(cpath))
var tag C.int
code := C.anydoc_format_from_path(cpath, &tag)
if code != errOK {
return "", false
}
return tagToFormat(tag)
}
+3
View File
@@ -0,0 +1,3 @@
module github.com/firecrawl/anydoc/go
go 1.22
+264
View File
@@ -0,0 +1,264 @@
#ifndef ANYDOC_H
#define ANYDOC_H
/* Warning: this file is generated by `cargo build -p anydoc-go`. Do not edit by hand. */
#include <stdarg.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdlib.h>
/**
* Success.
*/
#define ERR_OK 0
/**
* `ConvertError::Unsupported`.
*/
#define ERR_UNSUPPORTED 1
/**
* `ConvertError::Malformed`.
*/
#define ERR_MALFORMED 2
/**
* `ConvertError::Encrypted`.
*/
#define ERR_ENCRYPTED 3
/**
* `ConvertError::ResourceLimit`.
*/
#define ERR_RESOURCE_LIMIT 4
/**
* `ConvertError::MissingPart`.
*/
#define ERR_MISSING_PART 5
/**
* `ConvertError::Io`.
*/
#define ERR_IO 6
/**
* PDF was passed to `anydoc_to_document` (unsupported there).
*/
#define ERR_PDF_NO_MODEL 7
/**
* Null pointer or invalid argument passed to the ABI.
*/
#define ERR_INVALID_ARG 8
/**
* Unknown format name passed to the ABI.
*/
#define ERR_UNKNOWN_FORMAT 9
/**
* C-side format tag. Stable; mirrors the Node/Python lowercase string names
* via `format_name`. `ANYDOC_FORMAT_NONE` is the `Option::None` sentinel.
*/
#define ANYDOC_FORMAT_NONE -1
#define ANYDOC_FORMAT_DOC 0
#define ANYDOC_FORMAT_DOCX 1
#define ANYDOC_FORMAT_ODT 2
#define ANYDOC_FORMAT_PDF 3
#define ANYDOC_FORMAT_PPT 4
#define ANYDOC_FORMAT_PPTX 5
#define ANYDOC_FORMAT_RTF 6
#define ANYDOC_FORMAT_EPUB 7
#define ANYDOC_FORMAT_XLSX 8
#define ANYDOC_FORMAT_ODS 9
#define ANYDOC_FORMAT_ODP 10
#define ANYDOC_FORMAT_CSV 11
/**
* Tag values for `Block` variants. Stable across versions; the Go side maps
* these to the same lowercase string names the Node and Python bindings use.
*/
#define BLOCK_HEADING 0
#define BLOCK_PARAGRAPH 1
#define BLOCK_LIST 2
#define BLOCK_TABLE 3
#define BLOCK_QUOTE 4
#define BLOCK_CODE 5
#define BLOCK_RULE 6
#define INLINE_TEXT 0
#define INLINE_LINK 1
#define INLINE_IMAGE 2
#define INLINE_ANCHOR 3
#define INLINE_NOTEREF 4
#define INLINE_LINEBREAK 5
#define LINK_EXTERNAL 0
#define LINK_RELATIVE 1
#define LINK_ANCHOR 2
#define IMG_EXTERNAL 0
#define IMG_ASSET 1
#define IMG_UNAVAILABLE 2
#define MARKER_BULLET 0
#define MARKER_DECIMAL 1
#define MARKER_LOWER_ALPHA 2
#define MARKER_UPPER_ALPHA 3
#define MARKER_LOWER_ROMAN 4
#define MARKER_UPPER_ROMAN 5
#define TABLE_DATA 0
#define TABLE_LAYOUT 1
#define SLOT_ORIGIN 0
#define SLOT_COVERED 1
#define NOTE_FOOTNOTE 0
#define NOTE_ENDNOTE 1
/**
* Detect the format from the content. Writes a format tag to `*out` (one of
* `ANYDOC_FORMAT_*`), or `ANYDOC_FORMAT_NONE` when nothing matches.
*
* Returns `ERR_OK` on success (including no-match), `ERR_INVALID_ARG` on null
* pointers.
*/
int anydoc_format_from_bytes(const uint8_t *bytes, uintptr_t len, int *out);
/**
* The format a bare extension names (no leading dot). Writes a format tag to
* `*out`, or `ANYDOC_FORMAT_NONE` when unrecognized.
*/
int anydoc_format_from_extension(const char *ext, int *out);
/**
* The format a path's extension names. Writes a format tag to `*out`, or
* `ANYDOC_FORMAT_NONE` when unrecognized.
*/
int anydoc_format_from_path(const char *path, int *out);
/**
* Convert a document file to Markdown. The format is detected from the file
* content, with the extension as the fallback for signature-less formats
* (CSV).
*
* On success writes a `*mut c_char` to `*out_str` and its byte length to
* `*out_len`. The caller must free the string with `anydoc_string_free`.
*/
int anydoc_to_markdown(const char *path, char **out_str, uintptr_t *out_len);
/**
* Convert an in-memory document to Markdown. `format_tag` is one of the
* `ANYDOC_FORMAT_*` constants, or `ANYDOC_FORMAT_NONE` to detect from the
* content.
*/
int anydoc_to_markdown_bytes(const uint8_t *bytes,
uintptr_t len,
int format_tag,
char **out_str,
uintptr_t *out_len);
/**
* Convert an in-memory document to Markdown with embedded images rewritten
* as `![alt](images/image-N.ext)` so they keep their original positions.
*
* This is `to_document` + anydoc's official GFM serializer. Asset images are
* first turned into `ImageSource::External` URLs; the serializer already
* emits those as ordinary Markdown image links. PDF has no document model
* and is converted the same way as `anydoc_to_markdown_bytes`.
*/
int anydoc_to_markdown_with_asset_links(const uint8_t *bytes,
uintptr_t len,
int format_tag,
char **out_str,
uintptr_t *out_len);
/**
* Parse an in-memory document into the document model. `format_tag` is one of
* the `ANYDOC_FORMAT_*` constants, or `ANYDOC_FORMAT_NONE` to detect from the
* content.
*
* On success writes a heap-allocated byte buffer to `*out_buf` and its length
* to `*out_len`. The buffer holds the flat serialization (see `model`). The
* caller must free it with `anydoc_buffer_free`.
*
* Returns `ERR_PDF_NO_MODEL` for PDF input: PDF conversion produces Markdown
* directly and has no document-model form. Use `anydoc_to_markdown_bytes`.
*/
int anydoc_to_document(const uint8_t *bytes,
uintptr_t len,
int format_tag,
uint8_t **out_buf,
uintptr_t *out_len);
/**
* Free a string returned by `anydoc_to_markdown*` or `anydoc_last_error`.
* Null is a no-op (the empty-string sentinel).
*/
void anydoc_string_free(char *s);
/**
* Free a document buffer returned by `anydoc_to_document`. Null is a no-op.
* `len` must be the length written to `*out_len` at allocation time; it is
* validated against the header for safety.
*/
void anydoc_buffer_free(uint8_t *buf, uintptr_t len);
/**
* Return the human-readable message for the last error on the current
* thread. The returned `*mut c_char` is freshly allocated and must be freed
* with `anydoc_string_free`. Returns null when there is no last error.
*/
char *anydoc_last_error(void);
/**
* Return the lowercase name of a format tag, or null for an unknown tag.
* The returned string must be freed with `anydoc_string_free`.
*/
char *anydoc_format_name(int tag);
/**
* Parse a lowercase format name into a tag. Writes the tag to `*out`, or
* `ANYDOC_FORMAT_NONE` when the name is unknown.
*/
int anydoc_format_from_name(const char *name, int *out);
#endif /* ANYDOC_H */
+9
View File
@@ -0,0 +1,9 @@
//go:build linux && !musl
package anydoc
/*
#cgo amd64 LDFLAGS: -L${SRCDIR}/lib/linux_amd64_gnu -lanydoc_go -lm -lstdc++ -ldl -lpthread
#cgo arm64 LDFLAGS: -L${SRCDIR}/lib/linux_arm64_gnu -lanydoc_go -lm -lstdc++ -ldl -lpthread
*/
import "C"
+9
View File
@@ -0,0 +1,9 @@
//go:build linux && musl
package anydoc
/*
#cgo amd64 LDFLAGS: -L${SRCDIR}/lib/linux_amd64_musl -lanydoc_go -lm -lstdc++ -ldl -lpthread
#cgo arm64 LDFLAGS: -L${SRCDIR}/lib/linux_arm64_musl -lanydoc_go -lm -lstdc++ -ldl -lpthread
*/
import "C"
+729
View File
@@ -0,0 +1,729 @@
package anydoc
/*
#include "include/anydoc.h"
*/
import "C"
import "fmt"
// Document is a parsed document: its body, its notes, and the bytes of
// everything it embedded. A Document is self-contained — embedded assets
// carry their bytes, so it stays usable after the source archive is gone.
type Document struct {
Blocks []Block
// Notes are footnote and endnote bodies, referenced from text by an
// Inline with Kind == "note_ref".
Notes []Note
// Assets holds every embedded asset, indexed by Asset.ID as referenced
// by an Inline image source.
Assets []Asset
}
// Block is one block-level piece of a document body. The Kind field selects
// which of the optional fields is populated; the rest are nil.
type Block struct {
// "heading", "paragraph", "list", "table", "block_quote", "code_block",
// or "rule".
Kind string
Level *uint8 // heading: 1-based outline depth
Anchor *string // heading: stable anchor id when the document targets it
Content []Inline // heading, paragraph
List *List // list
Table *Table // table
Blocks []Block // block_quote
Lang *string // code_block
Text *string // code_block
}
// Inline is one span of inline content. The Kind field selects which optional
// fields are populated.
type Inline struct {
// "text", "link", "image", "anchor", "note_ref", or "line_break".
Kind string
Text *string // text
Style *Style // text
Content []Inline // link
Target *LinkTarget // link
Alt *string // image
Source *ImageSource // image
Anchor *string // anchor: the anchor id
NoteID *string // note_ref: the id of the note in Document.Notes
}
// Style is a fully resolved character style.
type Style struct {
Bold bool
Italic bool
Strike bool
Code bool
}
// LinkTarget is where a link points.
type LinkTarget struct {
// "external" (absolute URL with a scheme), "relative" (scheme-less
// relative reference, preserved as written), or "anchor" (internal
// target: a heading anchor or an "anchor" inline).
Kind string
Value string // the URL, relative reference, or anchor id
}
// ImageSource is where an image's bytes live.
type ImageSource struct {
// "external" (absolute URL), "asset" (embedded image, carried in
// Document.Assets), or "unavailable" (no usable source; only the alt
// text remains).
Kind string
URL *string // external
AssetID *uint64 // asset: index into Document.Assets
}
// List is a fully resolved list: numbering identity and marker resolution
// happen in the parser frontends.
type List struct {
// "bullet", "decimal", "lower_alpha", "upper_alpha", "lower_roman", or
// "upper_roman".
Marker string
Start uint64
Items []ListItem
}
// ListItem is one item of a List, which may hold nested blocks including
// further lists.
type ListItem struct {
Blocks []Block
Checked *bool // task-list state, when the item carries a checkbox
MarkerLabel *string // literal marker text overriding the level marker
}
// Table is a canonical table grid: every logical grid position appears
// exactly once. Content and spans live on the origin slot, and each position
// a span covers holds a CellSlot with Kind == "covered" pointing back at that
// origin.
type Table struct {
Grid [][]CellSlot
HeaderRows uint32 // number of leading rows that are header rows (0 = none)
Kind string // "data" or "layout"
}
// CellSlot is one position in a Table grid: either a cell or the shadow of
// one.
type CellSlot struct {
// "origin" (the cell itself) or "covered" (a position swallowed by a
// span, pointing back at the origin that covers it).
Kind string
Cell *Cell // origin
OriginRow *uint32 // covered: row of the covering origin
OriginCol *uint32 // covered: column of the covering origin
}
// Cell is a table cell and the extent it spans.
type Cell struct {
Blocks []Block
ColSpan uint32
RowSpan uint32
}
// Note is a footnote or endnote body, referenced from text by an Inline with
// Kind == "note_ref".
type Note struct {
ID string
Kind string // "footnote" or "endnote"
Blocks []Block
}
// Asset is an embedded binary asset (image, object payload). Bytes are always
// retained, so a document stays self-contained.
type Asset struct {
ID uint64 // index into Document.Assets, as referenced by an image source
MediaType string
OriginPart string
Data []byte
}
// ---- Decoding the flat FFI buffer -----------------------------------------
//
// The buffer is the flat, length-prefixed serialization written by
// go/src/model.rs. Every field is little-endian. A decoder tracks the read
// cursor and reports a short or implausible buffer as an error — the Rust side
// always produces a well-formed buffer, so a short read is a bug or a version
// skew, not user input, and it must not take the process down.
type decoder struct {
buf []byte
pos int
}
func decodeDocument(raw []byte) (*Document, error) {
d := &decoder{buf: raw}
doc := &Document{}
var err error
if doc.Blocks, err = d.blocks(); err != nil {
return nil, err
}
if doc.Notes, err = d.notes(); err != nil {
return nil, err
}
if doc.Assets, err = d.assets(); err != nil {
return nil, err
}
return doc, nil
}
// capFor bounds a preallocation by what is left in the buffer. Every element
// costs at least one byte to encode, so a count larger than the remaining
// bytes can only come from a corrupt buffer or a Rust/Go version skew — and
// preallocating it would exhaust memory before the first bounds check runs.
func (d *decoder) capFor(n uint32) int {
remaining := len(d.buf) - d.pos
if remaining < 0 {
return 0
}
if uint64(n) > uint64(remaining) {
return remaining
}
return int(n)
}
func (d *decoder) need(n int) error {
// A negative n means a length prefix overflowed int (32-bit builds); it is
// as unreadable as a truncated buffer.
if n < 0 || d.pos+n > len(d.buf) {
return fmt.Errorf("anydoc: truncated document buffer at offset %d (need %d bytes, have %d)", d.pos, n, len(d.buf)-d.pos)
}
return nil
}
func (d *decoder) u8() (uint8, error) {
if err := d.need(1); err != nil {
return 0, err
}
v := d.buf[d.pos]
d.pos++
return v, nil
}
func (d *decoder) bool() (bool, error) {
v, err := d.u8()
if err != nil {
return false, err
}
return v != 0, nil
}
func (d *decoder) u32() (uint32, error) {
if err := d.need(4); err != nil {
return 0, err
}
v := uint32(d.buf[d.pos]) | uint32(d.buf[d.pos+1])<<8 | uint32(d.buf[d.pos+2])<<16 | uint32(d.buf[d.pos+3])<<24
d.pos += 4
return v, nil
}
func (d *decoder) u64() (uint64, error) {
if err := d.need(8); err != nil {
return 0, err
}
v := uint64(d.buf[d.pos]) | uint64(d.buf[d.pos+1])<<8 | uint64(d.buf[d.pos+2])<<16 | uint64(d.buf[d.pos+3])<<24 |
uint64(d.buf[d.pos+4])<<32 | uint64(d.buf[d.pos+5])<<40 | uint64(d.buf[d.pos+6])<<48 | uint64(d.buf[d.pos+7])<<56
d.pos += 8
return v, nil
}
func (d *decoder) i32() (int32, error) {
u, err := d.u32()
if err != nil {
return 0, err
}
return int32(u), nil
}
// optStr reads a length-prefixed string or a -1 sentinel (None).
func (d *decoder) optStr() (*string, error) {
length, err := d.i32()
if err != nil {
return nil, err
}
if length < 0 {
return nil, nil
}
s, err := d.strN(int(length))
if err != nil {
return nil, err
}
return &s, nil
}
func (d *decoder) str() (string, error) {
length, err := d.u32()
if err != nil {
return "", err
}
return d.strN(int(length))
}
func (d *decoder) strN(n int) (string, error) {
if err := d.need(n); err != nil {
return "", err
}
s := string(d.buf[d.pos : d.pos+n])
d.pos += n
return s, nil
}
func (d *decoder) bytes() ([]byte, error) {
length, err := d.u32()
if err != nil {
return nil, err
}
if err := d.need(int(length)); err != nil {
return nil, err
}
b := make([]byte, length)
copy(b, d.buf[d.pos:d.pos+int(length)])
d.pos += int(length)
return b, nil
}
func (d *decoder) blocks() ([]Block, error) {
n, err := d.u32()
if err != nil {
return nil, err
}
out := make([]Block, 0, d.capFor(n))
for i := uint32(0); i < n; i++ {
b, err := d.block()
if err != nil {
return nil, err
}
out = append(out, b)
}
return out, nil
}
func (d *decoder) block() (Block, error) {
tag, err := d.i32()
if err != nil {
return Block{}, err
}
switch tag {
case C.BLOCK_HEADING:
level, err := d.u8()
if err != nil {
return Block{}, err
}
anchor, err := d.optStr()
if err != nil {
return Block{}, err
}
content, err := d.inlines()
if err != nil {
return Block{}, err
}
lv := level
return Block{Kind: "heading", Level: &lv, Anchor: anchor, Content: content}, nil
case C.BLOCK_PARAGRAPH:
content, err := d.inlines()
if err != nil {
return Block{}, err
}
return Block{Kind: "paragraph", Content: content}, nil
case C.BLOCK_LIST:
list, err := d.list()
if err != nil {
return Block{}, err
}
return Block{Kind: "list", List: list}, nil
case C.BLOCK_TABLE:
table, err := d.table()
if err != nil {
return Block{}, err
}
return Block{Kind: "table", Table: table}, nil
case C.BLOCK_QUOTE:
inner, err := d.blocks()
if err != nil {
return Block{}, err
}
return Block{Kind: "block_quote", Blocks: inner}, nil
case C.BLOCK_CODE:
lang, err := d.optStr()
if err != nil {
return Block{}, err
}
text, err := d.str()
if err != nil {
return Block{}, err
}
return Block{Kind: "code_block", Lang: lang, Text: &text}, nil
case C.BLOCK_RULE:
return Block{Kind: "rule"}, nil
default:
return Block{}, fmt.Errorf("anydoc: unknown block kind tag %d", tag)
}
}
func (d *decoder) inlines() ([]Inline, error) {
n, err := d.u32()
if err != nil {
return nil, err
}
out := make([]Inline, 0, d.capFor(n))
for i := uint32(0); i < n; i++ {
in, err := d.inline()
if err != nil {
return nil, err
}
out = append(out, in)
}
return out, nil
}
func (d *decoder) inline() (Inline, error) {
tag, err := d.i32()
if err != nil {
return Inline{}, err
}
switch tag {
case C.INLINE_TEXT:
text, err := d.str()
if err != nil {
return Inline{}, err
}
style, err := d.style()
if err != nil {
return Inline{}, err
}
return Inline{Kind: "text", Text: &text, Style: &style}, nil
case C.INLINE_LINK:
content, err := d.inlines()
if err != nil {
return Inline{}, err
}
target, err := d.linkTarget()
if err != nil {
return Inline{}, err
}
return Inline{Kind: "link", Content: content, Target: &target}, nil
case C.INLINE_IMAGE:
alt, err := d.str()
if err != nil {
return Inline{}, err
}
src, err := d.imageSource()
if err != nil {
return Inline{}, err
}
return Inline{Kind: "image", Alt: &alt, Source: &src}, nil
case C.INLINE_ANCHOR:
id, err := d.str()
if err != nil {
return Inline{}, err
}
return Inline{Kind: "anchor", Anchor: &id}, nil
case C.INLINE_NOTEREF:
id, err := d.str()
if err != nil {
return Inline{}, err
}
return Inline{Kind: "note_ref", NoteID: &id}, nil
case C.INLINE_LINEBREAK:
return Inline{Kind: "line_break"}, nil
default:
return Inline{}, fmt.Errorf("anydoc: unknown inline kind tag %d", tag)
}
}
func (d *decoder) style() (Style, error) {
bold, err := d.bool()
if err != nil {
return Style{}, err
}
italic, err := d.bool()
if err != nil {
return Style{}, err
}
strike, err := d.bool()
if err != nil {
return Style{}, err
}
code, err := d.bool()
if err != nil {
return Style{}, err
}
return Style{Bold: bold, Italic: italic, Strike: strike, Code: code}, nil
}
func (d *decoder) linkTarget() (LinkTarget, error) {
tag, err := d.i32()
if err != nil {
return LinkTarget{}, err
}
value, err := d.str()
if err != nil {
return LinkTarget{}, err
}
switch tag {
case C.LINK_EXTERNAL:
return LinkTarget{Kind: "external", Value: value}, nil
case C.LINK_RELATIVE:
return LinkTarget{Kind: "relative", Value: value}, nil
case C.LINK_ANCHOR:
return LinkTarget{Kind: "anchor", Value: value}, nil
default:
return LinkTarget{}, fmt.Errorf("anydoc: unknown link target tag %d", tag)
}
}
func (d *decoder) imageSource() (ImageSource, error) {
tag, err := d.i32()
if err != nil {
return ImageSource{}, err
}
switch tag {
case C.IMG_EXTERNAL:
url, err := d.str()
if err != nil {
return ImageSource{}, err
}
return ImageSource{Kind: "external", URL: &url}, nil
case C.IMG_ASSET:
id, err := d.u64()
if err != nil {
return ImageSource{}, err
}
return ImageSource{Kind: "asset", AssetID: &id}, nil
case C.IMG_UNAVAILABLE:
return ImageSource{Kind: "unavailable"}, nil
default:
return ImageSource{}, fmt.Errorf("anydoc: unknown image source tag %d", tag)
}
}
func (d *decoder) list() (*List, error) {
markerTag, err := d.i32()
if err != nil {
return nil, err
}
start, err := d.u64()
if err != nil {
return nil, err
}
n, err := d.u32()
if err != nil {
return nil, err
}
items := make([]ListItem, 0, d.capFor(n))
for i := uint32(0); i < n; i++ {
item, err := d.listItem()
if err != nil {
return nil, err
}
items = append(items, item)
}
marker, err := markerName(markerTag)
if err != nil {
return nil, err
}
return &List{Marker: marker, Start: start, Items: items}, nil
}
func (d *decoder) listItem() (ListItem, error) {
blocks, err := d.blocks()
if err != nil {
return ListItem{}, err
}
checkedTag, err := d.i32()
if err != nil {
return ListItem{}, err
}
var checked *bool
switch checkedTag {
case 0:
f := false
checked = &f
case 1:
t := true
checked = &t
case -1:
checked = nil
default:
return ListItem{}, fmt.Errorf("anydoc: unknown checked tag %d", checkedTag)
}
label, err := d.optStr()
if err != nil {
return ListItem{}, err
}
return ListItem{Blocks: blocks, Checked: checked, MarkerLabel: label}, nil
}
func (d *decoder) table() (*Table, error) {
headerRows, err := d.u32()
if err != nil {
return nil, err
}
kindTag, err := d.i32()
if err != nil {
return nil, err
}
rows, err := d.u32()
if err != nil {
return nil, err
}
grid := make([][]CellSlot, 0, d.capFor(rows))
for i := uint32(0); i < rows; i++ {
cols, err := d.u32()
if err != nil {
return nil, err
}
row := make([]CellSlot, 0, d.capFor(cols))
for j := uint32(0); j < cols; j++ {
slot, err := d.slot()
if err != nil {
return nil, err
}
row = append(row, slot)
}
grid = append(grid, row)
}
kind := "data"
if kindTag == C.TABLE_LAYOUT {
kind = "layout"
}
return &Table{Grid: grid, HeaderRows: headerRows, Kind: kind}, nil
}
func (d *decoder) slot() (CellSlot, error) {
tag, err := d.i32()
if err != nil {
return CellSlot{}, err
}
switch tag {
case C.SLOT_ORIGIN:
cell, err := d.cell()
if err != nil {
return CellSlot{}, err
}
return CellSlot{Kind: "origin", Cell: &cell}, nil
case C.SLOT_COVERED:
r, err := d.u32()
if err != nil {
return CellSlot{}, err
}
c, err := d.u32()
if err != nil {
return CellSlot{}, err
}
rr := r
cc := c
return CellSlot{Kind: "covered", OriginRow: &rr, OriginCol: &cc}, nil
default:
return CellSlot{}, fmt.Errorf("anydoc: unknown cell slot tag %d", tag)
}
}
func (d *decoder) cell() (Cell, error) {
colSpan, err := d.u32()
if err != nil {
return Cell{}, err
}
rowSpan, err := d.u32()
if err != nil {
return Cell{}, err
}
blocks, err := d.blocks()
if err != nil {
return Cell{}, err
}
return Cell{Blocks: blocks, ColSpan: colSpan, RowSpan: rowSpan}, nil
}
func (d *decoder) notes() ([]Note, error) {
n, err := d.u32()
if err != nil {
return nil, err
}
out := make([]Note, 0, d.capFor(n))
for i := uint32(0); i < n; i++ {
note, err := d.note()
if err != nil {
return nil, err
}
out = append(out, note)
}
return out, nil
}
func (d *decoder) note() (Note, error) {
id, err := d.str()
if err != nil {
return Note{}, err
}
kindTag, err := d.i32()
if err != nil {
return Note{}, err
}
blocks, err := d.blocks()
if err != nil {
return Note{}, err
}
kind := "footnote"
if kindTag == C.NOTE_ENDNOTE {
kind = "endnote"
}
return Note{ID: id, Kind: kind, Blocks: blocks}, nil
}
func (d *decoder) assets() ([]Asset, error) {
n, err := d.u32()
if err != nil {
return nil, err
}
out := make([]Asset, 0, d.capFor(n))
for i := uint32(0); i < n; i++ {
asset, err := d.asset()
if err != nil {
return nil, err
}
out = append(out, asset)
}
return out, nil
}
func (d *decoder) asset() (Asset, error) {
id, err := d.u64()
if err != nil {
return Asset{}, err
}
mediaType, err := d.str()
if err != nil {
return Asset{}, err
}
originPart, err := d.str()
if err != nil {
return Asset{}, err
}
data, err := d.bytes()
if err != nil {
return Asset{}, err
}
return Asset{ID: id, MediaType: mediaType, OriginPart: originPart, Data: data}, nil
}
func markerName(tag int32) (string, error) {
switch tag {
case C.MARKER_BULLET:
return "bullet", nil
case C.MARKER_DECIMAL:
return "decimal", nil
case C.MARKER_LOWER_ALPHA:
return "lower_alpha", nil
case C.MARKER_UPPER_ALPHA:
return "upper_alpha", nil
case C.MARKER_LOWER_ROMAN:
return "lower_roman", nil
case C.MARKER_UPPER_ROMAN:
return "upper_roman", nil
}
return "", fmt.Errorf("anydoc: unknown list marker tag %d", tag)
}
+83
View File
@@ -0,0 +1,83 @@
//! Rewrite embedded images to markdown-linkable sources so anydoc's official
//! GFM serializer will emit `![alt](images/image-N.ext)` in place.
//!
//! `document_to_markdown` drops `ImageSource::Asset` to alt text because
//! Markdown cannot carry bytes. `ImageSource::External` is already rendered
//! as a normal image link. We keep the official serializer and only change
//! how those assets are addressed.
use anydoc::model::{Block, CellSlot, Document, ImageSource, Inline};
pub fn rewrite_asset_images(document: &mut Document) {
let urls: Vec<Option<String>> = document
.assets
.iter()
.enumerate()
.map(|(i, asset)| {
if asset.bytes.is_empty() {
None
} else {
Some(format!("images/image-{}{}", i + 1, extension_for(&asset.media_type)))
}
})
.collect();
rewrite_blocks(&mut document.blocks, &urls);
for note in &mut document.notes {
rewrite_blocks(&mut note.blocks, &urls);
}
}
fn rewrite_blocks(blocks: &mut [Block], urls: &[Option<String>]) {
for block in blocks {
match block {
Block::Heading { content, .. } | Block::Paragraph(content) => {
rewrite_inlines(content, urls);
}
Block::List(list) => {
for item in &mut list.items {
rewrite_blocks(&mut item.blocks, urls);
}
}
Block::Table(table) => {
for row in &mut table.grid {
for slot in row {
if let CellSlot::Origin(cell) = slot {
rewrite_blocks(&mut cell.blocks, urls);
}
}
}
}
Block::BlockQuote(inner) => rewrite_blocks(inner, urls),
Block::CodeBlock { .. } | Block::Rule => {}
}
}
}
fn rewrite_inlines(inlines: &mut [Inline], urls: &[Option<String>]) {
for inline in inlines {
match inline {
Inline::Image { source, .. } => {
if let ImageSource::Asset(id) = source {
if let Some(Some(url)) = urls.get(id.0) {
*source = ImageSource::External(url.clone());
}
}
}
Inline::Link { content, .. } => rewrite_inlines(content, urls),
_ => {}
}
}
}
fn extension_for(media_type: &str) -> &'static str {
match media_type.to_ascii_lowercase().as_str() {
"image/jpeg" | "image/jpg" => ".jpg",
"image/png" => ".png",
"image/gif" => ".gif",
"image/webp" => ".webp",
"image/bmp" => ".bmp",
"image/tiff" => ".tiff",
"image/svg+xml" => ".svg",
_ => ".bin",
}
}
+620
View File
@@ -0,0 +1,620 @@
//! C ABI surface for the Go bindings.
//!
//! Memory ownership rule: **Rust allocates, C ABI exposes a `_free` for every
//! allocation family, Go releases handles.** There are two allocation
//! families:
//!
//! - strings: `*mut c_char` + length out-params, returned by
//! `anydoc_to_markdown*` and `anydoc_last_error`. Released via
//! `anydoc_string_free`. Length is passed separately so binary-safe.
//! - document buffer: a single `*mut u8` + length, returned by
//! `anydoc_to_document`, holding the flat serialization from [`model`].
//! Released via `anydoc_buffer_free`.
//!
//! Errors surface as stable non-zero codes (see `ERR_*`); the human-readable
//! message is available through `anydoc_last_error`, thread-local.
#![allow(clippy::missing_safety_doc)]
use std::ffi::CString;
use std::os::raw::{c_char, c_int};
use std::ptr;
use anydoc::{ConvertError, Format, document_to_markdown};
mod asset_links;
mod model;
// ---- Error codes -----------------------------------------------------------
/// Success.
pub const ERR_OK: c_int = 0;
/// `ConvertError::Unsupported`.
pub const ERR_UNSUPPORTED: c_int = 1;
/// `ConvertError::Malformed`.
pub const ERR_MALFORMED: c_int = 2;
/// `ConvertError::Encrypted`.
pub const ERR_ENCRYPTED: c_int = 3;
/// `ConvertError::ResourceLimit`.
pub const ERR_RESOURCE_LIMIT: c_int = 4;
/// `ConvertError::MissingPart`.
pub const ERR_MISSING_PART: c_int = 5;
/// `ConvertError::Io`.
pub const ERR_IO: c_int = 6;
/// PDF was passed to `anydoc_to_document` (unsupported there).
pub const ERR_PDF_NO_MODEL: c_int = 7;
/// Null pointer or invalid argument passed to the ABI.
pub const ERR_INVALID_ARG: c_int = 8;
/// Unknown format name passed to the ABI.
pub const ERR_UNKNOWN_FORMAT: c_int = 9;
thread_local! {
static LAST_ERROR: std::cell::RefCell<Option<CString>> = const { std::cell::RefCell::new(None) };
}
fn set_last_error(msg: &str) {
let cstring = CString::new(msg).unwrap_or_else(|_| CString::new("<invalid utf-8>").unwrap());
LAST_ERROR.with(|cell| *cell.borrow_mut() = Some(cstring));
}
fn error_code(err: &ConvertError) -> c_int {
match err {
ConvertError::Unsupported(_) => ERR_UNSUPPORTED,
ConvertError::Malformed { .. } => ERR_MALFORMED,
ConvertError::Encrypted => ERR_ENCRYPTED,
ConvertError::ResourceLimit { .. } => ERR_RESOURCE_LIMIT,
ConvertError::MissingPart { .. } => ERR_MISSING_PART,
ConvertError::Io(_) => ERR_IO,
_ => ERR_UNSUPPORTED,
}
}
fn fail(err: ConvertError) -> c_int {
let code = error_code(&err);
set_last_error(&err.to_string());
code
}
/// Run a conversion with panics contained. A panic that escapes an
/// `extern "C"` function aborts the process (Rust 1.81+), so one malformed
/// document would take the whole host process down instead of failing a single
/// conversion. A caught panic surfaces as `Malformed`, the same as any other
/// unusable input.
///
/// This cannot contain an abort the runtime raises on its own (stack overflow,
/// allocation failure); those still kill the process.
fn guarded<T>(
entry: &str,
parse: impl FnOnce() -> Result<T, ConvertError>,
) -> Result<T, ConvertError> {
// AssertUnwindSafe: the closure only reads the input slice and returns an
// owned result. Nothing observable is left half-written when it panics,
// because the out-params are written by the caller after this returns.
match std::panic::catch_unwind(std::panic::AssertUnwindSafe(parse)) {
Ok(result) => result,
Err(_) => Err(ConvertError::Malformed {
part: None,
detail: format!("panic while converting in {entry}"),
}),
}
}
// ---- Format enum -----------------------------------------------------------
/// C-side format tag. Stable; mirrors the Node/Python lowercase string names
/// via `format_name`. `ANYDOC_FORMAT_NONE` is the `Option::None` sentinel.
pub const ANYDOC_FORMAT_NONE: c_int = -1;
pub const ANYDOC_FORMAT_DOC: c_int = 0;
pub const ANYDOC_FORMAT_DOCX: c_int = 1;
pub const ANYDOC_FORMAT_ODT: c_int = 2;
pub const ANYDOC_FORMAT_PDF: c_int = 3;
pub const ANYDOC_FORMAT_PPT: c_int = 4;
pub const ANYDOC_FORMAT_PPTX: c_int = 5;
pub const ANYDOC_FORMAT_RTF: c_int = 6;
pub const ANYDOC_FORMAT_EPUB: c_int = 7;
pub const ANYDOC_FORMAT_XLSX: c_int = 8;
pub const ANYDOC_FORMAT_ODS: c_int = 9;
pub const ANYDOC_FORMAT_ODP: c_int = 10;
pub const ANYDOC_FORMAT_CSV: c_int = 11;
fn format_to_c(format: Format) -> c_int {
match format {
Format::Doc => ANYDOC_FORMAT_DOC,
Format::Docx => ANYDOC_FORMAT_DOCX,
Format::Odt => ANYDOC_FORMAT_ODT,
Format::Pdf => ANYDOC_FORMAT_PDF,
Format::Ppt => ANYDOC_FORMAT_PPT,
Format::Pptx => ANYDOC_FORMAT_PPTX,
Format::Rtf => ANYDOC_FORMAT_RTF,
Format::Epub => ANYDOC_FORMAT_EPUB,
Format::Excel => ANYDOC_FORMAT_XLSX,
Format::Ods => ANYDOC_FORMAT_ODS,
Format::Odp => ANYDOC_FORMAT_ODP,
Format::Csv => ANYDOC_FORMAT_CSV,
}
}
fn format_from_c(tag: c_int) -> Option<Format> {
Some(match tag {
ANYDOC_FORMAT_DOC => Format::Doc,
ANYDOC_FORMAT_DOCX => Format::Docx,
ANYDOC_FORMAT_ODT => Format::Odt,
ANYDOC_FORMAT_PDF => Format::Pdf,
ANYDOC_FORMAT_PPT => Format::Ppt,
ANYDOC_FORMAT_PPTX => Format::Pptx,
ANYDOC_FORMAT_RTF => Format::Rtf,
ANYDOC_FORMAT_EPUB => Format::Epub,
ANYDOC_FORMAT_XLSX => Format::Excel,
ANYDOC_FORMAT_ODS => Format::Ods,
ANYDOC_FORMAT_ODP => Format::Odp,
ANYDOC_FORMAT_CSV => Format::Csv,
_ => return None,
})
}
/// Format names, lowercase, matching the Node/Python convention.
const FORMAT_NAMES: [&str; 12] =
["doc", "docx", "odt", "pdf", "ppt", "pptx", "rtf", "epub", "xlsx", "ods", "odp", "csv"];
fn format_name(format: Format) -> &'static str {
match format {
Format::Doc => FORMAT_NAMES[0],
Format::Docx => FORMAT_NAMES[1],
Format::Odt => FORMAT_NAMES[2],
Format::Pdf => FORMAT_NAMES[3],
Format::Ppt => FORMAT_NAMES[4],
Format::Pptx => FORMAT_NAMES[5],
Format::Rtf => FORMAT_NAMES[6],
Format::Epub => FORMAT_NAMES[7],
Format::Excel => FORMAT_NAMES[8],
Format::Ods => FORMAT_NAMES[9],
Format::Odp => FORMAT_NAMES[10],
Format::Csv => FORMAT_NAMES[11],
}
}
fn parse_format_name(name: &str) -> Option<Format> {
FORMAT_NAMES
.iter()
.zip([
Format::Doc,
Format::Docx,
Format::Odt,
Format::Pdf,
Format::Ppt,
Format::Pptx,
Format::Rtf,
Format::Epub,
Format::Excel,
Format::Ods,
Format::Odp,
Format::Csv,
])
.find(|(n, _)| **n == name)
.map(|(_, f)| f)
}
// ---- String out-param helper ----------------------------------------------
/// Box a `String` into a heap allocation the caller owns and must free with
/// `anydoc_string_free`. Writes the byte length (excluding any NUL) to
/// `*out_len`. The buffer is NUL-terminated for safety with C string readers,
/// but binary-safe content relies on `out_len`.
///
/// Returns a dangling null pointer for empty strings so the Go side can treat
/// a null result as "empty" without needing to free it. Length is still
/// written as 0.
fn string_out(s: String, out_str: *mut *mut c_char, out_len: *mut usize) -> c_int {
if out_str.is_null() {
set_last_error("anydoc: out_str is null");
return ERR_INVALID_ARG;
}
let len = s.len();
// SAFETY: we write the length unconditionally; null is the empty sentinel.
if !out_len.is_null() {
unsafe { *out_len = len };
}
if len == 0 {
// Empty string: return null sentinel, no allocation to free.
unsafe { *out_str = ptr::null_mut() };
return ERR_OK;
}
let cstring = match CString::new(s.into_bytes()) {
Ok(cs) => cs,
Err(_) => {
// Contains an interior NUL: fall back to lossy replacement so the
// caller still gets something. Markdown shouldn't contain NULs.
set_last_error("anydoc: output contained an interior NUL byte");
return ERR_INVALID_ARG;
}
};
let raw = cstring.into_raw();
// into_raw returns a buffer of capacity len+1 (with the trailing NUL).
unsafe { *out_str = raw };
ERR_OK
}
// ---- C ABI functions -------------------------------------------------------
/// Detect the format from the content. Writes a format tag to `*out` (one of
/// `ANYDOC_FORMAT_*`), or `ANYDOC_FORMAT_NONE` when nothing matches.
///
/// Returns `ERR_OK` on success (including no-match), `ERR_INVALID_ARG` on null
/// pointers.
#[unsafe(no_mangle)]
pub unsafe extern "C" fn anydoc_format_from_bytes(
bytes: *const u8,
len: usize,
out: *mut c_int,
) -> c_int {
if bytes.is_null() || out.is_null() {
set_last_error("anydoc: null pointer passed to anydoc_format_from_bytes");
return ERR_INVALID_ARG;
}
let slice = unsafe { std::slice::from_raw_parts(bytes, len) };
let tag = Format::from_bytes(slice).map_or(ANYDOC_FORMAT_NONE, format_to_c);
unsafe { *out = tag };
ERR_OK
}
/// The format a bare extension names (no leading dot). Writes a format tag to
/// `*out`, or `ANYDOC_FORMAT_NONE` when unrecognized.
#[unsafe(no_mangle)]
pub unsafe extern "C" fn anydoc_format_from_extension(
ext: *const c_char,
out: *mut c_int,
) -> c_int {
if ext.is_null() || out.is_null() {
set_last_error("anydoc: null pointer passed to anydoc_format_from_extension");
return ERR_INVALID_ARG;
}
let ext_str = match unsafe { std::ffi::CStr::from_ptr(ext) }.to_str() {
Ok(s) => s.trim_start_matches('.'),
Err(_) => {
set_last_error("anydoc: extension is not valid UTF-8");
return ERR_INVALID_ARG;
}
};
let tag = Format::from_extension(ext_str).map_or(ANYDOC_FORMAT_NONE, format_to_c);
unsafe { *out = tag };
ERR_OK
}
/// The format a path's extension names. Writes a format tag to `*out`, or
/// `ANYDOC_FORMAT_NONE` when unrecognized.
#[unsafe(no_mangle)]
pub unsafe extern "C" fn anydoc_format_from_path(path: *const c_char, out: *mut c_int) -> c_int {
if path.is_null() || out.is_null() {
set_last_error("anydoc: null pointer passed to anydoc_format_from_path");
return ERR_INVALID_ARG;
}
let path_str = match unsafe { std::ffi::CStr::from_ptr(path) }.to_str() {
Ok(s) => s,
Err(_) => {
set_last_error("anydoc: path is not valid UTF-8");
return ERR_INVALID_ARG;
}
};
let tag =
Format::from_path(std::path::Path::new(path_str)).map_or(ANYDOC_FORMAT_NONE, format_to_c);
unsafe { *out = tag };
ERR_OK
}
/// Convert a document file to Markdown. The format is detected from the file
/// content, with the extension as the fallback for signature-less formats
/// (CSV).
///
/// On success writes a `*mut c_char` to `*out_str` and its byte length to
/// `*out_len`. The caller must free the string with `anydoc_string_free`.
#[unsafe(no_mangle)]
pub unsafe extern "C" fn anydoc_to_markdown(
path: *const c_char,
out_str: *mut *mut c_char,
out_len: *mut usize,
) -> c_int {
if path.is_null() || out_str.is_null() {
set_last_error("anydoc: null pointer passed to anydoc_to_markdown");
return ERR_INVALID_ARG;
}
let path_str = match unsafe { std::ffi::CStr::from_ptr(path) }.to_str() {
Ok(s) => s,
Err(_) => {
set_last_error("anydoc: path is not valid UTF-8");
return ERR_INVALID_ARG;
}
};
match guarded("anydoc_to_markdown", || anydoc::to_markdown(path_str)) {
Ok(md) => string_out(md, out_str, out_len),
Err(e) => fail(e),
}
}
/// Convert an in-memory document to Markdown. `format_tag` is one of the
/// `ANYDOC_FORMAT_*` constants, or `ANYDOC_FORMAT_NONE` to detect from the
/// content.
#[unsafe(no_mangle)]
pub unsafe extern "C" fn anydoc_to_markdown_bytes(
bytes: *const u8,
len: usize,
format_tag: c_int,
out_str: *mut *mut c_char,
out_len: *mut usize,
) -> c_int {
if bytes.is_null() || out_str.is_null() {
set_last_error("anydoc: null pointer passed to anydoc_to_markdown_bytes");
return ERR_INVALID_ARG;
}
let format = if format_tag == ANYDOC_FORMAT_NONE {
None
} else {
match format_from_c(format_tag) {
Some(f) => Some(f),
None => {
set_last_error("anydoc: unknown format tag");
return ERR_UNKNOWN_FORMAT;
}
}
};
let slice = unsafe { std::slice::from_raw_parts(bytes, len) };
match guarded("anydoc_to_markdown_bytes", || anydoc::to_markdown_bytes(slice, format)) {
Ok(md) => string_out(md, out_str, out_len),
Err(e) => fail(e),
}
}
/// Convert an in-memory document to Markdown with embedded images rewritten
/// as `![alt](images/image-N.ext)` so they keep their original positions.
///
/// This is `to_document` + anydoc's official GFM serializer. Asset images are
/// first turned into `ImageSource::External` URLs; the serializer already
/// emits those as ordinary Markdown image links. PDF has no document model
/// and is converted the same way as `anydoc_to_markdown_bytes`.
#[unsafe(no_mangle)]
pub unsafe extern "C" fn anydoc_to_markdown_with_asset_links(
bytes: *const u8,
len: usize,
format_tag: c_int,
out_str: *mut *mut c_char,
out_len: *mut usize,
) -> c_int {
if bytes.is_null() || out_str.is_null() {
set_last_error("anydoc: null pointer passed to anydoc_to_markdown_with_asset_links");
return ERR_INVALID_ARG;
}
let format = if format_tag == ANYDOC_FORMAT_NONE {
None
} else {
match format_from_c(format_tag) {
Some(f) => Some(f),
None => {
set_last_error("anydoc: unknown format tag");
return ERR_UNKNOWN_FORMAT;
}
}
};
let slice = unsafe { std::slice::from_raw_parts(bytes, len) };
match guarded("anydoc_to_markdown_with_asset_links", || {
markdown_with_asset_links(slice, format)
}) {
Ok(md) => string_out(md, out_str, out_len),
Err(e) => fail(e),
}
}
fn markdown_with_asset_links(
bytes: &[u8],
format: Option<Format>,
) -> Result<String, ConvertError> {
let resolved = format
.or_else(|| Format::from_bytes(bytes))
.ok_or_else(|| {
ConvertError::Unsupported(
"unrecognized file content: name the format explicitly".into(),
)
})?;
if resolved == Format::Pdf {
return anydoc::to_markdown_bytes(bytes, resolved);
}
let mut document = anydoc::to_document(bytes, resolved)?;
asset_links::rewrite_asset_images(&mut document);
Ok(document_to_markdown(&document))
}
/// Parse an in-memory document into the document model. `format_tag` is one of
/// the `ANYDOC_FORMAT_*` constants, or `ANYDOC_FORMAT_NONE` to detect from the
/// content.
///
/// On success writes a heap-allocated byte buffer to `*out_buf` and its length
/// to `*out_len`. The buffer holds the flat serialization (see `model`). The
/// caller must free it with `anydoc_buffer_free`.
///
/// Returns `ERR_PDF_NO_MODEL` for PDF input: PDF conversion produces Markdown
/// directly and has no document-model form. Use `anydoc_to_markdown_bytes`.
#[unsafe(no_mangle)]
pub unsafe extern "C" fn anydoc_to_document(
bytes: *const u8,
len: usize,
format_tag: c_int,
out_buf: *mut *mut u8,
out_len: *mut usize,
) -> c_int {
if bytes.is_null() || out_buf.is_null() {
set_last_error("anydoc: null pointer passed to anydoc_to_document");
return ERR_INVALID_ARG;
}
let format = if format_tag == ANYDOC_FORMAT_NONE {
None
} else {
match format_from_c(format_tag) {
Some(f) => Some(f),
None => {
set_last_error("anydoc: unknown format tag");
return ERR_UNKNOWN_FORMAT;
}
}
};
let slice = unsafe { std::slice::from_raw_parts(bytes, len) };
// Resolve the format up front so we can give the typed PDF error before
// parsing, matching the crate's `to_document` contract.
let resolved = match format {
Some(f) => f,
None => match Format::from_bytes(slice) {
Some(f) => f,
None => {
set_last_error(
"unsupported input: unrecognized file content: name the format explicitly",
);
return ERR_UNSUPPORTED;
}
},
};
if resolved == Format::Pdf {
set_last_error(
"unsupported input: PDF conversion produces Markdown directly and has no document-model form; use to_markdown_bytes",
);
return ERR_PDF_NO_MODEL;
}
let document = match guarded("anydoc_to_document", || anydoc::to_document(slice, resolved)) {
Ok(d) => d,
Err(e) => return fail(e),
};
let mut encoder = model::Encoder::new();
model::write_document(&mut encoder, &document);
let body = encoder.into_vec();
let len = body.len();
// Layout: [usize len][usize cap][u8 data..len]. The caller receives a
// pointer to the data area and the length; `anydoc_buffer_free` walks back
// to the header to recover (len, cap) for a sound deallocation.
let header_size = std::mem::size_of::<BufferHeader>();
let total = header_size + len;
let layout = std::alloc::Layout::from_size_align(total, std::mem::align_of::<BufferHeader>())
.expect("document buffer layout");
let base = unsafe { std::alloc::alloc(layout) as *mut BufferHeader };
if base.is_null() {
set_last_error("anydoc: out of memory");
return ERR_IO;
}
// The capacity we record is the *total* allocation size, so the free path
// can reconstruct a layout of the same size to pass back to `dealloc`.
unsafe {
std::ptr::write(base, BufferHeader { len, cap: total });
let data = (base as *mut u8).add(header_size);
std::ptr::copy_nonoverlapping(body.as_ptr(), data, len);
*out_buf = data;
if !out_len.is_null() {
*out_len = len;
}
}
// `body` drops here, freeing its own backing allocation.
ERR_OK
}
/// Header prepended to every document buffer, so `anydoc_buffer_free` can
/// soundly reclaim the allocation without any thread-local state. The data
/// the caller sees starts immediately after this header.
#[repr(C)]
struct BufferHeader {
len: usize,
/// Total allocation size (header + data), used to rebuild the Layout for
/// `dealloc`.
cap: usize,
}
/// Free a string returned by `anydoc_to_markdown*` or `anydoc_last_error`.
/// Null is a no-op (the empty-string sentinel).
#[unsafe(no_mangle)]
pub unsafe extern "C" fn anydoc_string_free(s: *mut c_char) {
if s.is_null() {
return;
}
// SAFETY: the pointer was produced by `CString::into_raw`; reclaiming it
// with `from_raw` is the matching deallocator.
unsafe {
let _ = CString::from_raw(s);
}
}
/// Free a document buffer returned by `anydoc_to_document`. Null is a no-op.
/// `len` must be the length written to `*out_len` at allocation time; it is
/// validated against the header for safety.
#[unsafe(no_mangle)]
pub unsafe extern "C" fn anydoc_buffer_free(buf: *mut u8, len: usize) {
if buf.is_null() {
return;
}
let header_size = std::mem::size_of::<BufferHeader>();
// SAFETY: the pointer was produced by `anydoc_to_document` with a
// `BufferHeader` immediately before it. Walk back to read the header and
// rebuild the Layout, then dealloc the whole block.
unsafe {
let base = (buf.sub(header_size)) as *mut BufferHeader;
let header = base.read();
if header.len != len {
// Length mismatch: caller bug. Leak rather than risk a bad free.
set_last_error("anydoc: anydoc_buffer_free length does not match allocation");
return;
}
let layout = std::alloc::Layout::from_size_align_unchecked(
header.cap,
std::mem::align_of::<BufferHeader>(),
);
std::alloc::dealloc(base as *mut u8, layout);
}
}
/// Return the human-readable message for the last error on the current
/// thread. The returned `*mut c_char` is freshly allocated and must be freed
/// with `anydoc_string_free`. Returns null when there is no last error.
#[unsafe(no_mangle)]
pub unsafe extern "C" fn anydoc_last_error() -> *mut c_char {
LAST_ERROR
.with(|cell| cell.borrow_mut().take())
.and_then(|cstring| {
// Re-allocate as a fresh owned C string the caller can free.
let bytes = cstring.into_bytes_with_nul();
CString::from_vec_with_nul(bytes).ok().map(|cs| cs.into_raw())
})
.unwrap_or(ptr::null_mut())
}
// ---- Go-side helpers exposed for parity tests -----------------------------
/// Return the lowercase name of a format tag, or null for an unknown tag.
/// The returned string must be freed with `anydoc_string_free`.
#[unsafe(no_mangle)]
pub unsafe extern "C" fn anydoc_format_name(tag: c_int) -> *mut c_char {
if tag == ANYDOC_FORMAT_NONE {
return ptr::null_mut();
}
match format_from_c(tag) {
Some(f) => {
let name = format_name(f);
match CString::new(name) {
Ok(cs) => cs.into_raw(),
Err(_) => ptr::null_mut(),
}
}
None => ptr::null_mut(),
}
}
/// Parse a lowercase format name into a tag. Writes the tag to `*out`, or
/// `ANYDOC_FORMAT_NONE` when the name is unknown.
#[unsafe(no_mangle)]
pub unsafe extern "C" fn anydoc_format_from_name(name: *const c_char, out: *mut c_int) -> c_int {
if name.is_null() || out.is_null() {
set_last_error("anydoc: null pointer passed to anydoc_format_from_name");
return ERR_INVALID_ARG;
}
let name_str = match unsafe { std::ffi::CStr::from_ptr(name) }.to_str() {
Ok(s) => s,
Err(_) => {
set_last_error("anydoc: format name is not valid UTF-8");
return ERR_INVALID_ARG;
}
};
let tag = parse_format_name(name_str).map_or(ANYDOC_FORMAT_NONE, format_to_c);
unsafe { *out = tag };
ERR_OK
}
+321
View File
@@ -0,0 +1,321 @@
//! `#[repr(C)]` DTOs and a flat, self-describing serialization of the document
//! model.
//!
//! The document tree is recursive (blocks hold blocks, inlines hold inlines,
//! tables hold cells that hold blocks), so rather than surfacing it through a
//! forest of opaque handles and one `_free` per node type, the FFI serializes
//! the whole `Document` into a single length-prefixed byte buffer. That gives
//! exactly one allocation family for the tree (`anydoc_buffer_free`) and keeps
//! the C ABI tiny. Variant tags are written inline as `c_int`-compatible
//! values, mirroring the Node/Python `kind` string convention.
//!
//! Format: see `write_document` below. Every field is length-prefixed so the
//! Go side can decode without guessing. Strings carry their byte length as a
//! u32; slices carry a u32 count followed by that many elements. Booleans are
//! a single byte (0 or 1). Integers are little-endian fixed width.
use std::os::raw::c_int;
use anydoc::model;
/// Tag values for `Block` variants. Stable across versions; the Go side maps
/// these to the same lowercase string names the Node and Python bindings use.
pub const BLOCK_HEADING: c_int = 0;
pub const BLOCK_PARAGRAPH: c_int = 1;
pub const BLOCK_LIST: c_int = 2;
pub const BLOCK_TABLE: c_int = 3;
pub const BLOCK_QUOTE: c_int = 4;
pub const BLOCK_CODE: c_int = 5;
pub const BLOCK_RULE: c_int = 6;
pub const INLINE_TEXT: c_int = 0;
pub const INLINE_LINK: c_int = 1;
pub const INLINE_IMAGE: c_int = 2;
pub const INLINE_ANCHOR: c_int = 3;
pub const INLINE_NOTEREF: c_int = 4;
pub const INLINE_LINEBREAK: c_int = 5;
pub const LINK_EXTERNAL: c_int = 0;
pub const LINK_RELATIVE: c_int = 1;
pub const LINK_ANCHOR: c_int = 2;
pub const IMG_EXTERNAL: c_int = 0;
pub const IMG_ASSET: c_int = 1;
pub const IMG_UNAVAILABLE: c_int = 2;
pub const MARKER_BULLET: c_int = 0;
pub const MARKER_DECIMAL: c_int = 1;
pub const MARKER_LOWER_ALPHA: c_int = 2;
pub const MARKER_UPPER_ALPHA: c_int = 3;
pub const MARKER_LOWER_ROMAN: c_int = 4;
pub const MARKER_UPPER_ROMAN: c_int = 5;
pub const TABLE_DATA: c_int = 0;
pub const TABLE_LAYOUT: c_int = 1;
pub const SLOT_ORIGIN: c_int = 0;
pub const SLOT_COVERED: c_int = 1;
pub const NOTE_FOOTNOTE: c_int = 0;
pub const NOTE_ENDNOTE: c_int = 1;
/// Encoder for the flat document buffer. Little-endian, length-prefixed.
pub struct Encoder {
buf: Vec<u8>,
}
impl Encoder {
pub fn new() -> Self {
Encoder { buf: Vec::with_capacity(8 * 1024) }
}
pub fn into_vec(self) -> Vec<u8> {
self.buf
}
pub fn u8(&mut self, v: u8) {
self.buf.push(v);
}
pub fn bool(&mut self, v: bool) {
self.buf.push(if v { 1 } else { 0 });
}
pub fn u32(&mut self, v: u32) {
self.buf.extend_from_slice(&v.to_le_bytes());
}
pub fn u64(&mut self, v: u64) {
self.buf.extend_from_slice(&v.to_le_bytes());
}
pub fn i32(&mut self, v: i32) {
self.buf.extend_from_slice(&v.to_le_bytes());
}
/// A length-prefixed UTF-8 string. `None` writes a -1 length sentinel.
pub fn opt_str(&mut self, v: &Option<String>) {
match v {
Some(s) => self.str(s),
None => self.i32(-1),
}
}
pub fn str(&mut self, s: &str) {
self.u32(s.len() as u32);
self.buf.extend_from_slice(s.as_bytes());
}
/// A length-prefixed byte slice.
pub fn bytes(&mut self, b: &[u8]) {
self.u32(b.len() as u32);
self.buf.extend_from_slice(b);
}
}
pub fn write_document(e: &mut Encoder, doc: &model::Document) {
write_blocks(e, &doc.blocks);
e.u32(doc.notes.len() as u32);
for note in &doc.notes {
write_note(e, note);
}
e.u32(doc.assets.len() as u32);
for asset in &doc.assets {
write_asset(e, asset);
}
}
fn write_blocks(e: &mut Encoder, blocks: &[model::Block]) {
e.u32(blocks.len() as u32);
for block in blocks {
write_block(e, block);
}
}
fn write_block(e: &mut Encoder, block: &model::Block) {
match block {
model::Block::Heading { level, anchor, content } => {
e.i32(BLOCK_HEADING);
e.u8(*level);
e.opt_str(anchor);
write_inlines(e, content);
}
model::Block::Paragraph(content) => {
e.i32(BLOCK_PARAGRAPH);
write_inlines(e, content);
}
model::Block::List(list) => {
e.i32(BLOCK_LIST);
write_list(e, list);
}
model::Block::Table(table) => {
e.i32(BLOCK_TABLE);
write_table(e, table);
}
model::Block::BlockQuote(inner) => {
e.i32(BLOCK_QUOTE);
write_blocks(e, inner);
}
model::Block::CodeBlock { lang, text } => {
e.i32(BLOCK_CODE);
e.opt_str(lang);
e.str(text);
}
model::Block::Rule => {
e.i32(BLOCK_RULE);
}
}
}
fn write_inlines(e: &mut Encoder, inlines: &[model::Inline]) {
e.u32(inlines.len() as u32);
for inline in inlines {
write_inline(e, inline);
}
}
fn write_inline(e: &mut Encoder, inline: &model::Inline) {
match inline {
model::Inline::Text { text, style } => {
e.i32(INLINE_TEXT);
e.str(text);
write_style(e, *style);
}
model::Inline::Link { content, target } => {
e.i32(INLINE_LINK);
write_inlines(e, content);
write_link_target(e, target);
}
model::Inline::Image { alt, source } => {
e.i32(INLINE_IMAGE);
e.str(alt);
write_image_source(e, source);
}
model::Inline::Anchor(id) => {
e.i32(INLINE_ANCHOR);
e.str(id);
}
model::Inline::NoteRef(id) => {
e.i32(INLINE_NOTEREF);
e.str(id);
}
model::Inline::LineBreak => {
e.i32(INLINE_LINEBREAK);
}
}
}
fn write_style(e: &mut Encoder, style: model::Style) {
e.bool(style.bold);
e.bool(style.italic);
e.bool(style.strike);
e.bool(style.code);
}
fn write_link_target(e: &mut Encoder, target: &model::LinkTarget) {
match target {
model::LinkTarget::External(v) => {
e.i32(LINK_EXTERNAL);
e.str(v);
}
model::LinkTarget::Relative(v) => {
e.i32(LINK_RELATIVE);
e.str(v);
}
model::LinkTarget::Anchor(v) => {
e.i32(LINK_ANCHOR);
e.str(v);
}
}
}
fn write_image_source(e: &mut Encoder, source: &model::ImageSource) {
match source {
model::ImageSource::External(url) => {
e.i32(IMG_EXTERNAL);
e.str(url);
}
model::ImageSource::Asset(id) => {
e.i32(IMG_ASSET);
e.u64(id.0 as u64);
}
model::ImageSource::Unavailable => {
e.i32(IMG_UNAVAILABLE);
}
}
}
fn write_list(e: &mut Encoder, list: &model::List) {
e.i32(marker_tag(list.marker));
e.u64(list.start);
e.u32(list.items.len() as u32);
for item in &list.items {
write_list_item(e, item);
}
}
fn write_list_item(e: &mut Encoder, item: &model::ListItem) {
write_blocks(e, &item.blocks);
match item.checked {
None => e.i32(-1),
Some(false) => e.i32(0),
Some(true) => e.i32(1),
}
e.opt_str(&item.marker_label);
}
fn write_table(e: &mut Encoder, table: &model::Table) {
e.u32(table.header_rows as u32);
e.i32(if table.kind == model::TableKind::Layout { TABLE_LAYOUT } else { TABLE_DATA });
e.u32(table.grid.len() as u32);
for row in &table.grid {
e.u32(row.len() as u32);
for slot in row {
write_slot(e, slot);
}
}
}
fn write_slot(e: &mut Encoder, slot: &model::CellSlot) {
match slot {
model::CellSlot::Origin(cell) => {
e.i32(SLOT_ORIGIN);
write_cell(e, cell);
}
model::CellSlot::Covered { origin_row, origin_col } => {
e.i32(SLOT_COVERED);
e.u32(*origin_row as u32);
e.u32(*origin_col as u32);
}
}
}
fn write_cell(e: &mut Encoder, cell: &model::Cell) {
e.u32(cell.col_span);
e.u32(cell.row_span);
write_blocks(e, &cell.blocks);
}
fn write_note(e: &mut Encoder, note: &model::Note) {
e.str(&note.id);
e.i32(if note.kind == model::NoteKind::Endnote { NOTE_ENDNOTE } else { NOTE_FOOTNOTE });
write_blocks(e, &note.blocks);
}
fn write_asset(e: &mut Encoder, asset: &model::Asset) {
e.u64(asset.id.0 as u64);
e.str(&asset.media_type);
e.str(&asset.origin_part);
e.bytes(&asset.bytes);
}
fn marker_tag(marker: model::MarkerKind) -> c_int {
match marker {
model::MarkerKind::Bullet => MARKER_BULLET,
model::MarkerKind::Decimal => MARKER_DECIMAL,
model::MarkerKind::LowerAlpha => MARKER_LOWER_ALPHA,
model::MarkerKind::UpperAlpha => MARKER_UPPER_ALPHA,
model::MarkerKind::LowerRoman => MARKER_LOWER_ROMAN,
model::MarkerKind::UpperRoman => MARKER_UPPER_ROMAN,
}
}
+6
View File
@@ -0,0 +1,6 @@
package anydoc
// Version is the version of the anydoc-go module, bumped in lockstep with the
// Rust crate version (go/Cargo.toml) and the GitHub release tag. The
// packaged static libraries.
const Version = "0.1.9"