Purely Local Read-Only Video Indexing: blake3 Content Deduplication and Range Preview, How I Never Modify a Single Original File

From "A Pile of Files" to "Read-Only Video Index": A Local-First Engineering Practice

Overview (TL;DR)

• Local-first read-only index: Scans multi-directory videos, builds a queryable, dedupable media dataset, never modifies original files.
• Content deduplication: blake3 precise hashing (Full) + first/last-1MB fast hashing (Fast) dual modes; (size, mtime) fingerprint incremental scanning avoids full rescans.
• Secure preview: Native playback via Range requests + signed short-lived tokens (id-bound / TTL 3600s / constant-time comparison); transcoded and thumbnail artifacts isolated in cache directories.
• Privacy guardrails: Refuses to listen on non-loopback addresses when API_KEY is unset; path masking ($HOME→~) applies only to display fields.
• Purely local, no cloud, runs as a single binary.

Starting Point: The Query Dilemma of a Video Library as "A Pile of Files"

My external hard drive is piled with thousands of video files scattered across multiple disks and directories. No capture time, no tags, no unified naming convention. To find a video that is "H.265 encoded, 1080p, between 20 and 40 minutes long, and byte-for-byte identical to another file," I could only rely on my eyes, flipping through pages in the file manager. Worse, many files were collected from various sources, and their names bear no relation to their content. Duplicate files consume several times the disk space, yet I had no tool to deduplicate by "content."

My other project, photo-library, already had a solution (EXIF + perceptual hash), but videos have no classic CV equivalent—no EXIF headers, and no mature perceptual hash library that can be directly applied. Video metadata (codec, resolution, bitrate, audio track information) can only be reliably provided by ffprobe.

This presented me with a core design tension: the index must be independent of the file tree and never modify original files, yet video metadata can only be obtained through an external probe (ffprobe). Once the index layer couples with the original files, or the scanning process touches the original files, the entire tool's positioning collapses.

So from the very beginning, I locked down three design principles: Purely local (data never leaves the machine, SQLite stores the index), Read-only (no operation touches original file content), Safety guardrails (authentication required before any network exposure). This "read-only" principle permeated every subsequent engineering decision.

The startup entry point embodies this principle. Upon startup, the service checks security configuration—when API_KEY is not set, listening on non-loopback addresses is forbidden:

let is_loopback = matches!(config.host.as_str(), "127.0.0.1" | "::1" | "localhost");
let allow_insecure = std::env::var("ALLOW_INSECURE_BIND")
    .map(|v| v == "1" || v.eq_ignore_ascii_case("true") || v.eq_ignore_ascii_case("yes"))
    .unwrap_or(false);
if !config.is_secured() && !is_loopback && !allow_insecure {
    eprintln!(
        "Refusing to start: API_KEY not set yet listening on non-loopback address `{}` (would expose the entire video index and original files). ...",
        config.host
    );
    std::process::exit(1);
}

This is just the starting point for a "read-only but sensitive" service—the index stores absolute paths, metadata, and content hashes of all videos. If accessed by unauthorized parties, it's equivalent to handing over the entire inventory of the video library.

Pitfall: The First Obstacle to the Read-Only Principle—"Filename Looks Like a Video" ≠ "Is a Video"

The first step of scanning is enumerating files. My initial idea was simple: filter by extension. But reality immediately slapped me—.ts is both an MPEG transport stream (one of my supported video formats) and the extension for TypeScript source files. A .d.ts file would be misidentified as a video by my extension filter.

This was just the surface. The deeper issue is: an extension only indicates "the filename looks like a video," not that the content is actually a video. A file ending in .mp4 might contain only a few lines of text (some people intentionally change extensions when downloading, or download tools save error responses as .mp4).

So in the file collection logic, I added a special case: any file ending in .d.ts (a TypeScript declaration file, whose trailing extension is also ts) is skipped outright to avoid being mistaken for a video; then hidden directories are skipped.

Relying solely on filenames isn't enough—the real determination must be left to ffprobe. The probe function returns a three-state result—an enum distinguishing "definitely a video," "not a video," and "probe failed but might be a video":

• Ok: ffprobe confirms a video stream exists, so it's added to the index.

This three-state design is the implementation of the "read-only probing" principle: ffprobe only reads file content, never writes back; even if a single file is corrupted, it degrades gracefully by skipping it, without crashing the entire scanning process.

Another detail: when metadata extraction fails, the container field falls back to the extension. This way, even if ffprobe fails to recognize a rare container format, the index still has a record, and users can at least find it by filename. Never abandon indexing an entire directory because of one bad file.

Adjustment: Incremental Scanning + Precise Hashing—Achieving Content Deduplication Without Rescanning the Entire Library Every Time

Once scanning could add files to the index, the next question followed: how to do content deduplication?

My initial idea was idealistic: full blake3 hashing, byte-by-byte comparison, absolutely precise. But thousands of large videos, each several GB, would take hours to read in full—this violates the intuitive need to "not rescan the entire library every time."

And if I went the fuzzy deduplication route (perceptual hashing, binary similarity), there are no reliable nearest-neighbor algorithms for videos—that path was a dead end.

So I split the decision into two parallel tracks:

Track One: Fingerprint-based incremental scanning. Use (size_bytes, mtime) as the file fingerprint. Both mtime (file modification time) and size_bytes (file size) are stored in the Video struct:

pub(crate) size_bytes: u64,
/// File modification time (Unix seconds). Used with size for incremental-scan change detection.
pub(crate) mtime: i64,
pub(crate) content_hash: String,

During scanning, fingerprints are compared first; unchanged files are not re-hashed; only files with changed fingerprints are re-read. This ensures both incremental scanning efficiency and precision.

Track Two: Two hash modes to choose from. An enum provides the trade-off between precision and speed:

    /// Streaming whole-file blake3: any bit change alters the hash; most precise dedupe, but slow for large files.
    #[default]
    Full,
    /// Hashes only the first and last 1MB plus size + mtime: extremely fast, but only catches "byte-identical or differs-only-in-middle" near-dupes.
    Fast,
}

The default is Full (whole-file blake3, where any bit change alters the hash). I store the hash in the content_hash field, and duplicate detection is simply grouping by this field—identical hashes mean identical content.

But the final deduplication decision cannot be left to the machine—hashing only identifies "groups with identical content"; which files to keep and which to delete must be decided by humans. An enum handles this:

    #[default]
    Undecided,
    /// Manually marked to keep.
    Keep,
    /// Manually marked to delete (removed from index on cleanup).
    Remove,
}

Note the comment: "人工指定删除（应用清理时从索引移除）" (Manually marked for deletion—removed from the index during application cleanup). The delete operation only affects index records; the original files on disk remain untouched, byte for byte—the "read-only" principle manifests here once again.

Validation: Range Preview + Transcoding + Safety Guardrails—The Last Mile of the Read-Only Principle

With the index built and deduplication working, the final question was: how to let users preview? Directly plugging a file path into a <video> tag? No. Browsers cannot read the server's file system, and I didn't want the frontend to have direct access to absolute paths.

Preview also operates on two levels, both strictly adhering to "read-only":

Level One: Native playback (Range requests). Browser video playback requires HTTP Range support (scrubbing the progress bar and streaming both depend on it). file_url is a backend-issued URL with a token, used directly as the <video src>. However, the <video> tag cannot attach custom headers to cross-origin requests, so x-api-key authentication cannot be used—this led to the design of signed tokens: short-lived signed tokens placed in query parameters.

pub(crate) fn make_token(id: u64, secret: &[u8; 32]) -> String {
    let exp = now_secs() + TOKEN_TTL_SECS;
    let payload = format!("{id}:{exp}");
    let hash = blake3::keyed_hash(secret, payload.as_bytes());
    format!("{exp}.{}", to_hex(hash.as_bytes()))
}

Tokens are valid for 1 hour (TOKEN_TTL_SECS = 3600), and validation uses constant-time comparison to prevent timing side channels. The token only authorizes access to the byte stream of the specified video and expires after its validity period.

Level Two: Transcoding + thumbnails. Some older codecs (like MPEG-4 Part 2 in certain .avi files) simply cannot be played by browsers. Both cache directories are products of the index layer, fully isolated from original files.

Path masking is also part of the "read-only" principle—users don't want their username mary appearing in URLs. During path serialization, the $HOME prefix is replaced with ~:

fn serialize_masked_path<S: serde::Serializer>(value: &str, serializer: S) -> Result<S::Ok, S::Error> {
    let masked = if let Ok(home) = std::env::var("HOME") {
        if !home.is_empty() && value.starts_with(&home) {
            format!("~{}", &value[home.len()..])
        } else {
            value.to_string()
        }
    } else {
        value.to_string()
    };
    serializer.serialize_str(&masked)
}

Note the critical reminder in the function comment: "仅用于纯展示字段；凡前端会回传用作查询过滤的字段不能脱敏，否则回传值无法匹配数据库里的真实路径" (Only for purely display fields; any field that the frontend sends back for query filtering cannot be masked, otherwise the returned value won't match the real path in the database). This detail shows that "masking" is not "loss"—the server internally retains the real path for file reading, only replacing it for external display.

Static file hosting (frontend build artifacts) also has read-only awareness: the static file service includes path traversal protection, where the normalized request path must still reside within the frontend directory:

let candidate = frontend_dir.join(rel);
let Ok(candidate) = candidate.canonicalize() else {
    return (StatusCode::NOT_FOUND, "not found").into_response();
};
let Ok(root) = frontend_dir.canonicalize() else {
    return (StatusCode::NOT_FOUND, "not found").into_response();
};
if !candidate.starts_with(&root) {
    return (StatusCode::FORBIDDEN, "forbidden").into_response();
}

Result: A Tool That "Cannot Touch Original Files" Is Ironically More Reliable

Looking back at the entire engineering evolution, I can summarize all the implementation points along this main thread:

Principle	Implementation Location	Specific Method
Fingerprint incremental	(size_bytes, mtime)	Unchanged files are not re-hashed
Precise deduplication	content_hash (blake3)	Full mode changes hash on any bit change
Preview isolation	Signed token + transcoding cache directory	Short-lived signed token; transcoded artifacts in cache directory
Path masking	Path serialization function	~ prefix externally, real path internally
Startup guardrail	Non-loopback check at startup	Refuses to listen on external network without API_KEY

I've been using this system myself for a few months, and the results meet expectations: adding a new external drive only requires selecting the directory in the UI, and incremental scanning runs in the background; duplicate files are immediately visible—filter by duplicate_only to see all videos with hash collisions; to find "H.265 1080p with AAC audio track," filter by codec + resolution + has_audio.

What reassures me most is precisely the "cannot touch original files" aspect. Because the index layer is completely independent, I can delete the index database and rebuild it at any time, and the original files won't change a single byte. Even if a scanning or transcoding bug occurs, the worst case is corrupting a few cache files—never polluting the original material. A tool that "cannot touch original files" is ironically more reliable precisely because it can't touch them—this sounds a bit paradoxical, but once you've used it, you'll understand that this is exactly how local video management should be.

Source Code Navigation

• Startup entry — Startup entry; safety guardrail (non-loopback listening forbidden without API_KEY)
• Configuration module — All environment variable configuration;
• Storage module — File collection, fingerprint storage
• Metadata module — ffprobe metadata extraction
• Signed token module — Short-lived signed tokens for video endpoints
• Data model module — Data model;
• API routing module — HTTP routes
• Static file module — Frontend static hosting; path traversal protection

Related Projects

video-library is part of a local-first media-library trilogy alongside markdown-library and photo-library:

• markdown-library — local-first Markdown read-only index: frontmatter/headings/links/broken-link checks, zero-dependency self-contained rendering
• photo-library — local-first photo library index: content-hash exact dedupe + perceptual-hash near-dupe + EXIF indexing