Local Photo Indexing: 4 Design Decisions That Turn a File Pile into a Queryable Dataset

Introduction

I have over 100,000 photos scattered across multiple hard drives, cloud storage, and mobile devices. They are neither categorized nor deduplicated, and I don't even know exactly how many I've taken.

I tried a few approaches:

• Cloud albums (Google Photos, iCloud): convenient, but the privacy cost is high. My photos are uploaded to someone else's servers, and the indexing logic is opaque.
• File managers: you can see the files, but you can't query them. Want to find "all the blue photos containing a cat"? No chance.
• Local album apps (Lightroom, Apple Photos): powerful, but they are "viewers", not a "data layer". Once photos are imported, control over the original files is handed to the app, and they are hard to query with scripts.

What I needed sits between these two: a local, queryable photo data layer where the index is decoupled from the files.

That is the positioning of photo-library — a local photo indexing service built on SQLite + a REST API. It doesn't display photos and doesn't manage files; it does exactly one thing: make photos queryable.

There are 4 key design decisions in this system, each solving a pain point of the traditional approaches. Let's break them down one by one.

TL;DR

• Incremental scan: a (size, mtime) fingerprint skips unchanged files; when ~5% of files change, a re-scan drops from ~8 minutes to ~15 seconds
• Similar-image clustering: aHash high-16-bit bucketing + in-bucket pairwise comparison turns full-table O(n²) into near-linear; blake3 content hashing handles exact duplicates separately
• disposition state machine: a three-state undecided / keep / remove enum persists human decisions; the disk files are never touched
• Never-touch-originals principle: index-first, writing only SQLite metadata; batch_upsert contains no file move or delete

Decision 1: Incremental Scan — Why Not a Full Re-Scan Every Time

The Naive Approach

The simplest idea: on every scan, walk all images, re-decode, recompute the blake3 hash, recompute the aHash, analyze the metadata, then write it to the database.

The Pain Point

My library has 100k+ images. Full decoding and hashing takes about 8 minutes. If I only took one new photo, I'd still wait 8 minutes to see the result. That is unacceptable as a user experience.

The Actual Approach

My solution is fingerprint skipping: load the existing (size, mtime) fingerprints from the store, compare them during the scan, and only re-decode and re-analyze files whose fingerprint changed.

Concurrent decoding is bounded per batch by scan_chunk() to avoid memory blow-ups from large images (a single decoded image can reach tens of MB):

/// Concurrent decodes per batch. Large images (a ~66MB Canon original decodes to
/// tens of MB) are the main cause of scan-time memory and CPU spikes when concurrency is high.
/// Default 4: during scan the CPU is no longer saturated (on M3 8-core ~30-50%); on slow disks ~2-3x;
/// overridable via env var (e.g. set 8 for a more aggressive scan).
fn scan_chunk() -> usize {
    std::env::var("SCAN_CONCURRENCY")
        .ok()
        .and_then(|v| v.parse().ok())
        .unwrap_or(4)
}

And the real "skip" happens in read_image_file: on a fingerprint hit it returns None directly, without even decoding.

// File unchanged: skip re-decode; this is what brings incremental scan from minutes down to seconds.
if signatures.get(&relative_path) == Some(&(size_bytes, mtime)) {
    return Ok(None);
}

The Effect

On my 100k-photo library, when only about 5% of files change, re-scan time drops from ~8 minutes to ~15 seconds. A speedup of over 30x.

Decision 2: Similar-Image Clustering — Why Not Full-Table O(n²)

The Problem

Pairwise comparison across 100,000 images is C(100000, 2) ≈ 5 billion comparisons. Even at 1 microsecond each, that's 5000 seconds — about 83 minutes. And that's just the comparison phase; add preprocessing and it's completely impractical.

Detail: Perceptual Hashing Uses aHash

Similarity detection doesn't compare pixels — it compares perceptual hashes. photo-library uses aHash (average hash): shrink the image to 8×8 grayscale, compare each pixel against the mean, and produce a 64-bit value.

/// Average hash (aHash): shrink to 8x8 grayscale, compare each pixel against the mean to produce 64 bits.
pub(crate) fn compute_phash(img: &DynamicImage) -> Option<i64> {
    let gray = img.resize_exact(8, 8, image::imageops::FilterType::Lanczos3).to_luma8();
    let pixels: Vec<u32> = gray.iter().map(|v| *v as u32).collect();
    let mean = pixels.iter().sum::<u32>() / pixels.len() as u32;
    let mut hash = 0u64;
    for (i, &p) in pixels.iter().enumerate() {
        if p >= mean {
            hash |= 1u64 << (63 - i as u64);
        }
    }
    // ...
}

Detail: Exact-Duplicate Exclusion + Bucketing

An important edge case: if two images have the same content_hash (blake3 hash), they are exact duplicates and don't need to participate in similar merging. Otherwise, re-encoded-but-identical images would be misjudged as "similar" rather than "duplicate".

Clustering itself uses bucketing to avoid a full-table comparison; only within a bucket is the pairwise Hamming distance computed:

/// Similar-image clustering: merge into a group when aHash distance <= threshold but content hashes differ
/// (exact duplicates excluded). Bucket first by the high 16 bits of the perceptual hash, then compare
/// pairwise within the bucket, avoiding a full-table O(n^2).
pub(crate) async fn similar(&self) -> Result<SimilarPage, ApiError> {

That way, exact duplicates are handled by the blake3 path and near-duplicates by the aHash path, and the two never interfere.

Decision 3: Persisting Human Decisions — Why a disposition State Machine

The Advantage

Duplicate/similar detection is only a "suggestion"; whether to actually delete is a human decision. photo-library persists the decision with a three-state enum:

pub(crate) enum Disposition {
    Undecided,
    Keep,
    Remove,
}

1. Reversible: the user can change the disposition repeatedly until satisfied
2. Transparent: the user can preview which files would be deleted before confirming
3. Non-destructive: the indexing service itself never performs the deletion; the user keeps the final say

Decision 4: Never-Touch-Originals Principle — Why Index First

The Result

This principle makes photo-library a reliable underlying service: the user can safely build any functionality on top of it (e.g. auto-backup, smart classification, photo analysis) without worrying that the indexer will accidentally modify files.

Concretely: the content hash only "reads" the original, never "writes" it — hash_file streams the file to compute the blake3 digest, never persisting or moving it:

/// blake3 content hash (streaming, to avoid reading a whole large image into memory).
fn hash_file(path: &Path) -> std::io::Result<String> {
    let mut file = fs::File::open(path)?;
    let mut hasher = blake3::Hasher::new();
    // ...
}

And batch_upsert writes only metadata into SQLite — no stage contains a file move or delete. This fully decouples the indexer from "file ownership".

Conclusion

The core goal of photo-library is not "managing photos" but "making photos queryable". That goal determines its architecture:

• Incremental scanning guarantees efficiency
• Bucketed clustering guarantees scalability
• The disposition state machine guarantees safety
• The never-touch-originals principle guarantees reliability

These 4 decisions support each other, together forming a stable, scalable, user-controlled local photo data layer.

If you also have a large photo collection to manage, or want to use photo-indexing capability in your own project, feel free to check out photo-library.

FAQ

Source Navigation

• src/store.rs — Incremental scan / fingerprint skip: scan_chunk concurrency control, read_image_file skips unchanged files by (size, mtime)
• src/store.rs — Content hash: hash_file uses blake3 streaming hash for exact dedupe
• src/perception.rs + src/store.rs — Perceptual hash / similar clustering: compute_phash (aHash) and similar high-16-bit bucket clustering
• src/model.rs — Data model / disposition state: Image, Disposition (undecided/keep/remove)
• src/api.rs + src/file_token.rs — File serving & auth: image_file original serving, file_token short-lived signed token

Project Address

• GitHub repo: https://github.com/erishen/photo-library