Multi-Source Aggregation and Local Data Sovereignty: A Self-Hosted Home Network Dashboard

The average modern home has 15-30 networked devices – phones, laptops, smart speakers, cameras, TV boxes, IoT sensors. Yet most home users have no idea which devices are actually online, which link their traffic leaves through, or what that unfamiliar IP really belongs to.

Strip a router down to its single connected-devices page and what it offers is usually just IP and MAC. Hand your data to a third-party cloud service and you pay a privacy tax. Move Home Assistant into the house and you take on a full hardware stack plus ongoing maintenance.

After walking all three paths, one conclusion stands out: the real pain point is not "there is no tool" – it is that the aggregation dimension and the privacy boundary of existing tools are misaligned. Some people use the router's built-in UI, some use cloud-based monitoring, some force Home Assistant into the picture. Each leaves a blind spot unsolved. lume-lan is built to close those three gaps at once: multi-source aggregation, a self-hosted static binary, local data sovereignty – one pipeline for one problem.

---

The Three Pain Points of Existing Options

Router Web UI: You Can See, But Only One Layer

The web management page of a home router is the entry point for nearly every household user. Its capability is fixed – list currently connected devices (IP / MAC / signal strength), occasionally show the gateway's WAN status. Flat functionality, no historical trend, no cross-device correlation.

What it gives you is a static "info card" snapshot per device, while the valuable view is a traffic trend with a time series. The router's native web UI only covers the former – showing the trend requires collecting cumulative counters and computing deltas.

Cloud Monitoring: Convenient, But the Data Leaves Home

Handing network data to a third-party cloud service is the least-effort path. Open the app, see the device list and traffic curves, query from anywhere. The cost: data leaves the local network and lands on the provider's servers. For users who care about data sovereignty, that is a risk premium they must pay.

That stance is written into the project itself: the point of HTPASSWD_FILE in .env is to keep credentials local only – treating "authentication and data isolation" as a first-class concern from the start, not a patch applied later.

Heavyweight Options Like Home Assistant: Complete Features, Non-Trivial Barrier

Home Assistant, OpenWrt plugins, Netdata and the like are genuinely comprehensive. But they need a resident host system (Raspberry Pi, NAS, or a dedicated server), a long configuration chain, and complex component dependencies. For a user who just wants to see "who is connected and where the traffic goes," the return on that investment may not pay off – unless you already have an always-on host and are willing to maintain it long term.

lume-lan positions itself in the opposite direction: it is a ~1.7 MB static binary – built on lume-core's language and runtime, layering in agent-httpd and native SQLite so a single executable serves static sites and JSON APIs as well as chat and agents.

Under two megabytes of executable, covering three data sources – router API, DNS logs, ARP snapshots – deployed as a container, no Node.js runtime, no build pipeline. That is the definition of "lightweight."

Worth distinguishing: the ~1.7 MB is the server artifact – a single C11 static binary. The full application also includes the three host-side Python collection scripts, the launchd timer, the Docker runtime, and the optional router and AdGuard Home services. The actual deployment cost depends on which dependencies the host already has (Python 3, Docker, a system arp), not on this binary's size.

---

The Different Approach of lume-lan

Once the three pain points are lined up, the architecture choices of lume-lan follow naturally.

Multi-Source Aggregation Instead of a Single Interface

A single data source is the common root of all three pain points. The router API cannot see DNS queries, DNS logs cannot see traffic trends, the ARP table cannot see device vendor info. Correlating the three yields a LAN view far richer than any single source—but their observation scopes differ (ARP sees only the neighbor table, the router API depends on the exact model, DNS covers only traffic through that resolver), so the aggregate still has blind spots. It is not a "complete" LAN map.

Three scripts, three jobs:

scripts/scan_lan.py        -> arp -a snapshot -> .data/devices.json
scripts/router_fetch.py    -> router session  -> .data/router.json
scripts/adguard_fetch.py   -> DNS queries     -> .data/dns.json

The core logic of scan_lan.py extracts the gateway IP from the system routing table, parses the current ARP table with arp -a, filters out noise – 169.254 link-local, broadcast MACs, virtual interfaces (Docker bridge, VPN tun, and so on) – then writes .data/devices.json:

def is_noise(ip: str, mac: str, iface: str = "") -> bool:
    if ip.startswith("169.254."):
        return True
    mm = mac.lower()
    if mm in ("", "ff:ff:ff:ff:ff:ff", "00:00:00:00:00:00"):
        return True
    if iface and iface.startswith("bridge"):
        return True
    ...

router_fetch.py reverse-engineers the login protocol of HiLink CAT routers – fetch device model and SN without a password (/api/system/deviceinfo), complete login through a SHA256 path, then pull HostInfo for the device list and cumulative traffic:

class CatSession:
    def __init__(self, base: str):
        self.base = base
        self.cookies = {}

    def _req(self, method: str, path: str, body: bytes = None,
             timeout: int = 15) -> tuple:
        url = self.base + path
        req = urllib.request.Request(
            url, data=body, method=method,
            headers={
                "Content-Type": "application/json; charset=utf-8",
                "_ResponseFormat": "JSON",
                "X-Requested-With": "XMLHttpRequest",
                ...
            },
        )
        ...

adguard_fetch.py reads the DNS query log from AdGuard Home (new API path /control/querylog), aggregates domain access per client IP over the last hour, and outputs a Top-10 ranking plus per-device block counts:

WIN = 3600          # "last 1h" aggregation window (seconds)
TOPD = 15           # per-client domain Top N
TOTAL_TOP = 10      # whole-household domain ranking Top N

A Self-Hosted Static Binary: No Runtime, the Container Serves but Never Builds

The division of labor is drawn very clearly:

+- Host (macOS) --------------------------------------------------+
|  launchd 60s refresh loop                                       |
|   scripts/scan_lan.py        arp -a -> .data/devices.json       |
|   scripts/router_fetch.py    router session -> .data/router.json|
|   scripts/adguard_fetch.py   DNS queries -> .data/dns.json      |
|  (arp scanning needs the host NIC - host only, not container)   |
+---------------------------+--------------------------------------+
                            | .data/ snapshots mounted into the container (ro)
+- Docker container (lume-lan, port 10091) -----------------------+
|  Lume server (lan.lume, binds 0.0.0.0:8091, docroot www)        |
|   /api/devices  /api/router  /api/dns ... (reads .data)         |
+-----------------------------------------------------------------+

The host does the data collection; the container only reads snapshots and serves HTTP APIs. The Dockerfile has no Node.js, no build step, no npm install – just one pre-compiled C11 static binary. That keeps the container-side deployment complexity very low; the full pipeline, however, still depends on the collection scripts and the launchd timer already running on the host – both are required.

Local Data Sovereignty: .data/ Fully Local, .env gitignored

Security design runs through the whole architecture. The .data/ directory holds every snapshot, .env holds the router password and AdGuard credentials, and both are in .gitignore:

.data/aliases.json      # local naming layer ({ip: name}), gitignored
.runtime/htpasswd       # Basic Auth password file, gitignored

The intent behind HTPASSWD_FILE is explicit – Basic Auth passwords live in a local file, are never mixed with the router password, and are not passed in as plaintext environment variables.

One caveat: storing .data/ locally only means the data is not sent outward by default – it does not mean the data can never leave this machine. Once you wire in an external DNS upstream, a cloud LLM, or a remote-access service, data gains an outbound flow that has to be confirmed item by item.

auth-pass:
    @mkdir -p .runtime
    @if [ -z "$(PASSWORD)" ]; then \
      PASS="$$(LC_ALL=C tr -dc 'A-Za-z0-9' </dev/urandom | head -c 20)"; \
    else PASS="$(PASSWORD)"; fi; \
    /usr/sbin/htpasswd -c -b -B -C 12 .runtime/htpasswd admin "$$PASS" && \
    chmod 600 .runtime/htpasswd

---

Engineering Trade-offs Under the Architectural Comparison

With the architecture described, the valuable part is understanding the trade-off behind each choice.

Independence of Data Sources

The three scripts have no dependency on each other. A failure in scan_lan.py does not affect router_fetch.py, and a dead adguard_fetch.py does not crash the dashboard – its own degradation logic shows this plainly:

"""AdGuard not running / unreachable: keep the old snapshot and mark reachable=false (UI shows "not connected"), never crash."""

dns_watch.py has another distinctive design: it writes already-discovered clients into .runtime/dns_watch_seen.json, so a restart does not re-print them and drown the log in noise:

SEEN = ROOT / ".runtime" / "dns_watch_seen.json"
prev = {}
if SEEN.exists():
    try:
        prev = json.loads(SEEN.read_text())
    except Exception:
        prev = {}

This is essentially a snapshot isolation mechanism – each data source has its own lifecycle and they never interfere.

Boundaries of the Traffic Delta

A device's delta_1h (and the trend curve) comes from the difference between two points of the router's cumulative counters. Two boundaries are worth stating plainly – one is handled, one is a known gap:

• Counter reset / router reboot (handled): when the difference is negative it is clamped to 0 via max(0, ...) (in apply_traffic_deltas, scripts/router_fetch.py), so no negative flow or reverse jump is ever drawn. The cost: after a reset, until the baseline point moves past the reset moment (about 10 minutes), that device's increment is temporarily under-counted as 0.
• Device identity change (not yet implemented): the traffic history is keyed by IP. Once DHCP reassigns an IP, the old and new devices under that IP have their counters mixed, producing a bias. The current version does not bind identity by MAC – this is a known boundary.

(Also, while the router login is in cooldown / stale state, the script does not append the stale cumulative value to the trend history, avoiding a fake "traffic = 0" flat line.)

The Deployment Boundary

The architecture diagram shows two layers clearly:

• Host: runs the scripts, holds the privileges, scans ARP
• Container: read-only mount of .data/, serves HTTP APIs, makes no system calls

One detail in scan_lan.py deserves attention – the virtual interface filter:

def _is_virtual_iface(name: str) -> bool:
    n = (name or "").lower()
    return n.startswith(("utun", "tun", "tap", "ppp", "ipsec", "awdl",
                         "bridge", "docker", "veth", "wg", "tailscale"))

This exists because on macOS Docker and VPN can momentarily take over the default route. Without the filter, the gateway gets misidentified as a virtual-interface address such as 74.125.250.129, and the entire scanned subnet goes wrong.

Security Status: What Is Done and What Is Still Exposed

The solution applies several layers of security, but each only addresses its own layer – it does not add up to "secure":

1. Network layer: ADGUARD_HOST=127.0.0.1 – AdGuard listens only on localhost, never exposed outward
2. Application layer: the router API is protected by a cookie session; the login state is fully isolated from ARP snapshots
3. Service layer: the Lume server's Basic Auth uses a local htpasswd file whose password is never reused from the router credential

The residual risks should be stated plainly:

• Basic Auth is not a complete security boundary. It only authenticates; it does not encrypt the channel. Over plaintext HTTP on an untrusted network, credentials can still be intercepted. Public deployment also needs HTTPS, a firewall, and access control.
• A read-only mount is not "no other container risk." Mounting .data/ as :ro lowers the risk of tampering, but it does not address service exposure, dependency vulnerabilities, or process privileges.
• Local storage is not "data never leaves" (see above) – .data/ is not sent out by default, but any external service you add requires re-assessing the data flow.

Authentication: Basic Auth + local htpasswd + router password isolation

In one line: these measures reduce risk, but writing this as "current measures and residual risks" is closer to reality than claiming the setup is simply "secure."

---

Where Each Option Actually Fits

Back to the original question: is there a lightweight, local, multi-source-aggregating option? The answer depends on which layer of need you are at.

Router web UI fits one scenario: you only want to know "who is connected." It solves a one-off question, single information dimension, zero deployment cost.

Cloud monitoring fits another: you are willing to trade privacy for convenience. Queryable from a phone app anywhere, multi-site access, no maintenance – but the data is never in your hands.

lume-lan answers the third scenario: advanced users who care about data sovereignty and need multi-dimensional monitoring. It is not the "best" option – it is the option that fits a certain set of constraints best: data stays home, deployment stays simple, information coverage exceeds any single interface.

Three Layers of Value: from artifact to language validation to future direction

Reading its value in three layers is closer to reality than treating it as "some router's home panel":

• Current delivery: home-network data aggregation and a local dashboard – ARP + router + DNS correlated across three sources, self-hosted in a container, data kept local.
• Lume's validation value: carrying a real application on a self-built .lume DSL and a lightweight server – with real OS dependencies (arp / ping / the routing table), network data collection, and security boundaries, not a language demo. That is also why it can push back on and test the runtime.
• Future direction (not yet implemented): device discovery, LAN message delivery, inter-agent collaboration – a different layer from "observing the existing network," requiring separately designed identity authentication, pairing authorization, and a message protocol. The current project does observation and aggregation only; it does not provide device-to-device communication.

---

Quick Start & Verification

You don't have to take the architecture diagram on faith. Here is the shortest path to a running dashboard, plus output you can check directly (see the repo README for more).

Start it up:

make auth-pass                 # set the dashboard Basic Auth password -> .runtime/htpasswd
make docker-build              # first time: build the image (host pre-downloads the Lume static binary into vendor/)
make docker-up                 # -> http://localhost:10091/
make scan && make router-fetch # refresh once by hand (run every 60s by launchd day to day)

Multi-source correlation: one device is stitched together by IP across all three sources (IP/MAC below are sanitized).

.data/devices.json – host ARP snapshot, answers "who is online":

{ "ip": "192.168.1.42", "mac": "aa:bb:cc:dd:ee:42", "iface": "en0", "live": true }

.data/router.json – router login state, answers "what is it and how much has it moved":

{ "ip": "192.168.1.42", "hostname": "LivingRoom-TV", "vendor": "Xiaomi",
  "online": true, "freq": "5GHz", "rssi": -58, "delta_1h": 734003200 }

.data/dns.json – AdGuard last-1h, aggregated per client IP, answers "what has it looked up":

"192.168.1.42": { "total": 214, "blocked": 12,
                  "top": [ { "domain": "example-cdn.com", "count": 41 } ] }

The front end merges all three by IP: ARP gives presence, the router adds name/vendor/traffic/signal, DNS adds domain access – only when all three point at 192.168.1.42 does the device turn from "an unfamiliar IP" into a readable card.

Verifiable degradation: stop AdGuard and adguard_fetch.py does not crash – it marks the snapshot reachable=false, adds error and stale_since, and carries the previous round's clients / top_domains over verbatim (rather than writing an empty object); the UI shows "not connected" for domain access while the other sources keep refreshing:

{ "reachable": false, "error": "URLError: ...", "stale_since": 1792331063 }

This path is exactly the "independence of data sources" described above – any one source going down affects only its own block, and the dashboard does not collapse as a whole.

---

Source Code Navigation

Source is open, default branch main: https://github.com/erishen/lume-lan

• scripts/scan_lan.py – ARP snapshot collector
• scripts/router_fetch.py – router API login and device/traffic fetch
• scripts/adguard_fetch.py – AdGuard DNS log aggregation
• scripts/dns_watch.py – live DNS client watcher
• Dockerfile – static binary container image
• Makefile – build and operations command orchestration
• .env.example – environment variable template
• .runtime/htpasswd – Basic Auth password file (gitignored)