Local-First Markdown Indexing: From Zero-Dependency Rust to Read-Only Security Design Choices

Introduction: A Local-First Practitioner's Pain Point

As a developer who has long practiced the local-first philosophy, I've always believed: data should be fully under your own control, tools should be simple and reliable, and they shouldn't depend on uncontrollable external services. But in daily Markdown document management, a tricky problem has always plagued me.

My documents are scattered across multiple directories—some in the workspace, some in personal note folders, and others distributed across different projects. Over time, the number of documents has surged, making search increasingly difficult. Worse still, internal links between documents frequently break, yet no tool helps me detect them in advance. I've tried various solutions: Obsidian plugins, Notion search, even a self-built indexing service based on ElasticSearch—but each has its own pain points: some are SaaS-dependent, some require maintaining additional infrastructure, and some are overly bloated in functionality.

The core of the problem is actually simple: how to build a queryable, broken-link-checkable Markdown document dataset in a purely local environment?

Thus, markdown-library was born. It is a local-first, read-only Markdown indexing service that scans .md documents in a repository, builds a searchable index, and supports broken-link checking. This article will share several key design decisions behind this project—from technology stack selection to security mechanisms. Every choice is the result of thinking through "intuitive approach → actual approach → why it's better."

Overview (TL;DR)

• Local-first read-only index: Scans multi-directory .md files, builds a queryable document dataset with broken-link checking, never modifies source files.
• Zero dependencies: Core dependencies are only axum / rusqlite / walkdir / blake3 / regex; Markdown parsing and HTML rendering are fully self-contained.
• Three-layer tags: frontmatter ∪ auto (auto_tag heuristics) ∪ manual (separate manual_tags table, survives rescan).
• Security closed loop: Backend zero-dependency regex renderer with built-in XSS hardening; /docs/{id}/file and /docs/{id}/html use blake3-signed short-lived tokens (id-bound / TTL 3600s / constant-time comparison).
• Purely local, no cloud, runs as a single binary.

Section 1: Starting Point—Why Rust and Zero Dependencies

Intuitive Approach

Initially, my intuition was to choose Python or Go to implement this tool. Both languages have mature Markdown processing ecosystems and can quickly build prototypes. After all, fast implementation and seeing results as soon as possible is most developers' first reaction.

Actual Approach

Ultimately, I chose Rust and adhered to a "zero dependency" design philosophy—introducing no third-party dependencies beyond framework-level libraries.

The project's core dependencies are only five: axum for web serving, rusqlite as data storage, walkdir for filesystem traversal, blake3 for generating document hashes, and regex for text matching. All other logic, including Markdown parsing and HTML rendering, is self-contained.

Why It's Better

This choice brings several key advantages:

Single binary deployment, no runtime dependencies. The final artifact is a statically linked Rust binary that can be copied to any Linux/macOS/Windows machine and run directly, without needing to install Python, Node.js, or any package manager. For local-first users, this means "download and use, no configuration needed."

SQLite as the sole data source. The entire indexing system uses only one SQLite database file, avoiding the complexity of multi-process writes. Through WAL (Write-Ahead Logging) mode, search and write operations can execute concurrently without blocking each other.

No Markdown parsing crate introduced. Markdown parsing and HTML rendering logic are all self-contained. While this may look like "reinventing the wheel," it actually allows us to precisely control the output HTML structure and embed security hardening logic within the same codebase.

Section 2: Lessons Learned—Where Are the Boundaries of Read-Only Design

Experience and Reflections

This design once had me "step into a pit." A user reported: "I want to modify a document's tags, why doesn't the index update the source file?" At first I thought it was a bug, but after deeper reflection, I realized this is precisely the design's highlight.

The read-only design ensures data source authenticity and traceability. If the index could modify source files, then when the index is corrupted or data is inconsistent, users would be unable to restore the original state. The read-only design ensures:

• Source files remain always available, unaffected by index operations
• The index can be rebuilt by rescanning
• Any erroneous data in the index can be cleared and regenerated

Section 3: Adjustments—Independent Layer Design of the Three-Layer Tag System

Why This Is Better

This layered design brings significant advantages:

Independence. The manual_tags table survives independently—even if the user performs a full rescan, manually added tags will not be lost. auto tags can be recomputed at any time without affecting frontmatter and manual tags.

Aggregated display. In the /api/tags endpoint, the three layers of tags are displayed in an aggregated manner, but source identifiers are preserved, allowing users to clearly see where each tag originates.

Flexibility. Users can search and filter based on any layer of tags, and the system will assign different weights based on tag source.

Auto tags are heuristically generated during scanning by auto_tag.rs. The core logic converts "directory segments" and "body structure" into tags—directory segments directly become category tags, while code languages, images, tables, TODOs, math formulas, etc. in the body generate corresponding structural tags:

pub fn auto_tags(relative_path: &str, body: &str, frontmatter_tags: &[String]) -> Vec<String> {
    let mut set: std::collections::BTreeSet<String> = std::collections::BTreeSet::new();

    // 1) Directory segments (parent directories after removing the filename) become tags
    let segs: Vec<&str> = relative_path.split('/').filter(|s| !s.is_empty()).collect();
    if segs.len() > 1 {
        for seg in &segs[..segs.len() - 1] {
            let slug = slugify(seg);
            if !slug.is_empty() {
                set.insert(slug);
            }
        }
    }
    // … (body structure tags: code / lang:* / image / link / table / todo / math)
    // 4) Remove those already in frontmatter to avoid duplicate display
    let fm: std::collections::HashSet<String> =
        frontmatter_tags.iter().map(|t| t.to_ascii_lowercase()).collect();
    let out: Vec<String> = set.into_iter().filter(|t| !fm.contains(t)).collect();
    out
}

The final tags table is the union of frontmatter ∪ auto, while manual tags are stored independently in the manual_tags table, with the three layers not polluting each other.

Section 4: Verification—XSS-Hardened Self-Renderer

Verification Method

To ensure the renderer's security, I designed a series of XSS test cases:

These tests verify that the renderer can correctly handle malicious Markdown input, and the generated HTML will not contain executable scripts. The key hardening is in md_render.rs: text is first escape_html-escaped, and link URLs are validated through sanitize_url—HTML entities are decoded first before protocol checking, otherwise encoded forms like javascript:... could bypass literal checks; unknown protocols (javascript: / data: / file:) are uniformly replaced with #, only http / https / mailto are allowed through, and relative links and anchors are preserved as-is:

fn sanitize_url(url: &str) -> String {
    let t = url.trim();
    // Decode entities first before checking protocol: otherwise encoded forms like `javascript:...` could bypass literal scheme checks.
    let lower = decode_entities(t).to_ascii_lowercase();
    if lower.starts_with("http://")
        || lower.starts_with("https://")
        || lower.starts_with("mailto:")
    {
        t.to_string()
    } else if has_scheme(&lower) {
        "#".to_string()   // Unknown protocols (javascript:/data:/file:) are uniformly replaced with #
    } else {
        t.to_string()     // Relative/anchor links preserved as-is
    }
}

Section 5: Results—Security Closed Loop of Signed Tokens

Browser "open original in new tab / preview" cannot attach custom headers to cross-origin requests, so /docs/{id}/file and /docs/{id}/html cannot use x-api-key authentication. markdown-library instead uses a query-parameter token issued by the backend: a blake3 keyed-hash with TTL 3600s and id binding. make_token key-hash- signs {id}:{exp} with the secret, and verify_token checks expiration and id while verifying the signature, using constant-time comparison to prevent timing attacks:

const TOKEN_TTL_SECS: u64 = 3600;

pub(crate) fn make_token(id: u64, secret: &[u8; 32]) -> String {
    let exp = now_secs() + TOKEN_TTL_SECS;
    let payload = format!("{id}:{exp}");
    let hash = blake3::keyed_hash(secret, payload.as_bytes());
    format!("{exp}.{}", to_hex(hash.as_bytes()))
}

pub(crate) fn verify_token(id: u64, token: &str, secret: &[u8; 32]) -> bool {
    let Some((exp_str, mac)) = token.split_once('.') else { return false; };
    let Ok(exp) = exp_str.parse::<u64>() else { return false; };
    if exp <= now_secs() { return false; }
    let payload = format!("{id}:{exp}");
    let hash = blake3::keyed_hash(secret, payload.as_bytes());
    constant_time_eq(&to_hex(hash.as_bytes()), mac)  // id-bound + TTL + constant-time comparison
}

When API_KEY is left empty, local access is allowed without authentication; once a key is set, /file and /html require this signed token, thereby protecting private documents without exposing x-api-key.

---

The implementation process of markdown-library made me deeply realize: local-first tools need not only "to work" but also "to be reliable." From zero-dependency Rust implementation, to read-only design philosophy, to the three-layer tag system and XSS-hardened renderer, every decision was a result of repeated trade-offs between "intuitive approach" and "actual approach." I hope this article can provide some reference for you in building your own local tools.

Frequently Asked Questions (FAQ)

Source Code Navigation

• src/auto_tag.rs — Heuristic auto tags (the auto layer of the three-layer tag system)
• src/md_render.rs — Zero-dependency Markdown renderer with XSS hardening (sanitize_url / escape_html)
• src/file_token.rs — blake3-signed short-lived file token
• src/store.rs — SQLite index and multi-root UNIQUE constraints
• src/api.rs — REST API (/api/docs, /docs/{id}/file, /html)
• src/main.rs — Startup and binding guards

Project Address

• GitHub: https://github.com/erishen/markdown-library

Related Projects

markdown-library is part of a local-first media-library trilogy alongside photo-library and video-library:

• photo-library — local-first photo library index: content-hash exact dedupe + perceptual-hash near-dupe + EXIF indexing
• video-library — local-first video library index: transcode + dedupe + privacy guards