firefly-studio Architecture Retrospective: Why a Desktop Digital Human Was Electron + C++ From Day One

The Starting Point: No Transition Phase

Many architecture-evolution stories open with "we built a crude version first to get it working, then refactored step by step." firefly-studio did not. The first line of code already landed the complete Electron three-layer structure — main process, preload script, React renderer — plus a C++ native module (native/build/Release/firefly_native.node) loaded by the main process through node-addon-api.

This was not because the author likes heavy architecture, but because a digital human that talks has to meet several practical constraints in the browser at a real cost — holding the device handle continuously, not losing audio in the background, running local-model inference — so firefly-studio paid that complexity in full on day one of initialization:

1. Device-handle ownership. A digital human must hold the microphone continuously and respond to voice at any moment. The browser's holding of audio devices is "soft" — background tabs get throttled, and permission can silently lapse after the user leaves the page, so the digital human "goes deaf." The Electron main process holds the audio device handle directly via CoreAudio; its lifecycle is bound to the app, and there is no "lost its ears, regains them on refocus" problem.
2. Low audio-processing latency. Lip movement must track the sound; a few dozen milliseconds of lag means "half a beat behind," especially noticeable on plosive sounds. The JS main thread must run both three.js rendering and audio-feature computation, and the frame rate collapses once features get complex. The C++ native module sinks RMS energy computation and VAD detection down into the main process, keeping clear of the render thread.
3. Local offline privacy. Speech recognition and synthesis run on local models (Whisper.cpp / Piper), so audio and text never leave the machine. This dictates that ASR/TTS must run on the desktop, not call some cloud API.

So looking back at the first version, it was not the least bit "crude": Electron 37 + electron-vite, React 18 + three.js + @react-three/fiber + @pixiv/three-vrm (VRM digital human), a C++17 audio engine, Whisper.cpp local ASR, and Piper local TTS — all essentially in place. The complexity did not evolve; it was locked in all at once by cost.

The entry point that loads the native module in the main process is very plain — loadNative() in src/main/index.js uses createRequire to resolve an absolute path and require that .node file, trying up to 6 candidate paths across three base directories (out/main/../../native, the app root, and cwd) each in Release and Debug; if all fail it throws and lists the paths it tried, suggesting you first run npm run native:build (which uses node-gyp to compile directly against the current Electron version and architecture, not electron-rebuild). There is no intermediate "let's try the browser first" state here.

What the First Version Actually Did

Breaking down the first version's execution path, it already did four things:

1. C++ capture and feature extraction — CoreAudio captures the microphone; the native module calls back PCM at fixed frames (default 20ms / 16kHz mono); the capture pipeline first passes through RNNoise real-time denoising (on by default, which requires upsampling the 16kHz signal to RNNoise's required 48kHz / 480 samples and downsampling it back after processing), and RMS energy and VAD are computed from the denoised audio; if denoiser init fails or processing throws, it auto-degrades and falls back to the raw audio.
2. Energy-driven lip movement — the renderer's Stage component takes audioEnergy (0~1) and drives the VRM model's mouth blendshape directly: louder sound, wider mouth.
3. VAD threshold judgment — the native module's IsSpeech uses whether the energy exceeds a threshold to decide "is someone speaking"; if so, that frame is marked as speech.
4. Local voice pipeline — at end of speech, the segment is concatenated and sent to Whisper.cpp for transcription; the text goes to the LLM; the reply is synthesized by Piper and played back through CoreAudio.

The core audio-energy computation lives in native/src/audio_engine.cc, and the real code looks like this:

double AudioEngine::ComputeRms(const int16_t* data, size_t len) {
  double sum_sq = 0.0;
  for (size_t i = 0; i < len; ++i) {
    double v = static_cast<double>(data[i]) / 32768.0;  // normalize to [-1, 1]
    sum_sq += v * v;
  }
  double rms = std::sqrt(sum_sq / static_cast<double>(len));
  // one-pole low-pass smoothing: smoothed = alpha * new + (1 - alpha) * old
  smoothed_energy_ = smoothing_alpha_ * rms + (1.0 - smoothing_alpha_) * smoothed_energy_;
  return smoothed_energy_;
}

bool AudioEngine::IsSpeech(const int16_t* data, size_t len) {
  double energy = ComputeRms(data, len);
  return energy >= speech_threshold_;  // simple threshold judgment
}

The renderer gets the result through the useMic hook (real API):

const { sampleRate = 16000, frameMs = 20, threshold = 0.02,
        silenceMs = 700, minSpeechMs = 300 } = options
window.electronAPI.mic.onFrame((frame) => {
  setVolume(frame.rms)        // 0~1 real-time volume
  setIsSpeaking(frame.isSpeech)
})

One common misreading needs clarifying: some say "the first version had no time dimension, no smoothing." In fact ComputeRms has had one-pole low-pass smoothing from the very beginning (smoothing_alpha_ = 0.2), so its output smoothed_energy_ is a curve with inertia, not frame-by-frame jitter. The smoothed_energy_ field is the earliest fossil in the code of the requirement "temporal inertia."

But the first version did have two real imperfections, and both were later validated by actual bugs:

• VAD was a simple threshold, not a state machine. IsSpeech only answers "did this frame exceed the threshold," not "where does a sentence start and end." What actually decides "speech-segment boundaries" is the state machine in the upper mic_capture layer (judging start/end via silenceMs / minSpeechMs), and this layer later genuinely produced a bug (see P0-001 below).
• Lip movement used energy only, not phonemes. Stage receives a single scalar audioEnergy — it can tell "loud / quiet" but not "ah" from "mm." This is a real, currently-existing ceiling, not something a pivot solved — the earlier "VAD + ASR semantics-driven lip movement" idea has not actually landed in this project; lip movement remains energy-driven to this day.

Real Bugs and Refactors

firefly-studio never had an architectural pivot like "from browser to Electron" or "from JS to C++" — it was always Electron + C++. What actually happened was fixing real bugs and filling real UX gaps on the already-locked skeleton. These are documented in docs/TODO.md:

• P0-001: VAD state machine end-detection incorrect. Symptom: "microphone capture works, but speech recognition never triggers." Root cause was in the end-detection logic of the mic_capture state machine — silenceMs was miscalculated, so after a sentence ended the system kept waiting and ASR never fired. The fix was to refactor the VAD state machine and correct the end judgment. This is exactly where the gap between "threshold judgment" and "state-machine judgment" truly blew up.
• P0-002: TTS audio does not play. The LLM returned text and the digital human's mouth moved, but there was no sound. Root cause: the TTS service did not start correctly, or the PCM playback chain had an error. Fixed the TTS startup logic and PCM playback.
• P1-003: Streaming TTS experience. The original flow was "LLM text shows first, then voice comes out" — a disjointed feel. Changed to synthesize the first segment first, so text and lip movement appear together, and later segments synthesize and play as they go.
• P0-004: Music ducking recovers too early. While the digital human speaks, music volume drops, but on long text the music recovered before speaking finished. Root cause: the ducking timer was out of sync with TTS playback progress; changed to dynamic ducking based on actual TTS playback progress.
• P1-005: VAD sensitivity optimization. VAD state switches were too frequent, causing inaccurate segmentation. Added "confirm speech after N consecutive speech frames / confirm end after N consecutive silence frames," and introduced an adaptive threshold (dynamically adjusting speech_threshold_ based on the ambient noise baseline).

These are more down-to-earth than "three pivots": they are not rebuilds from scratch, but repeated polishing between audio_engine's smoothed_energy_, useMic's silenceMs/minSpeechMs, and TTS playback progress. Every fix lands on a concrete coordinate of the old skeleton.

What Was Never Adopted

Some directions were rejected during the evaluation phase, and rejected very specifically — they are all recorded in the "Design Decisions and Trade-offs" section of docs/ARCHITECTURE.md:

Direction One: Cloud ASR, bypassing local Whisper.cpp inference

The first seriously compared option was "use a cloud speech-recognition API" versus "run Whisper.cpp locally." Local won for hard reasons: microphone audio never leaves the machine (privacy), works offline (no network), no API call cost, and predictable local inference latency. The cost is slightly lower accuracy than cloud in noisy environments, having to download the model yourself (base ~140MB), and using local CPU/memory. For a "local-first" digital human, privacy and offline are untouchable constraints, so cloud ASR was out immediately.

Direction Two: Pure JS / WebAudio for audio, bypassing the C++ native module

Another direction was "compute all audio features with JS + Web Audio API, no C++." It was rejected for performance and latency: once real-time audio RMS/VAD is sunk into C++, it stops crowding the React render thread; native audio frameworks like CoreAudio are only comfortable in C/C++; more importantly — C++ lets you set lldb breakpoints and single-step directly inside VSCode, an irreplaceable capability when debugging the audio pipeline. The cost is a heavier build chain (the C++ native module is compiled directly against the Electron ABI with node-gyp, not electron-rebuild) and macOS-only support for now. The low-latency requirement for audio processing outweighed build complexity, so the C++ native module stayed.

Direction Three: WebSocket instead of SSE

LLM streaming output also considered WebSocket at first. SSE won because streaming only needs "server → client" one-way push; SSE is HTTP-based, simple to implement, natively supported by browsers, and even has auto-reconnect. The cost is one-way, text-only, and some proxies buffer it. For the "typewriter-style output" scenario, one-way is enough, so WebSocket was not worth introducing.

Direction Four: Cloud TTS, bypassing Piper

TTS similarly compared cloud versus local. The reasons for choosing local Piper mirror ASR: text is not uploaded (privacy), MIT-licensed and commercially usable, works offline, and low latency on the ONNX runtime. The cost is fewer voice choices (currently mainly Chinese "Xiaoya" and English "Amy") and audio quality below some online services. Privacy and offline again outweighed "voice richness."

These four rejected directions share one trait: each is more convenient in some local dimension (no model download, no C++, no bidirectional protocol, more voices), but each hits a higher constraint — privacy/offline, low latency, simplicity. They were rejected not because they are "bad," but because "which constraint they lost to" is clear.

Sedimentation

The code firefly-studio finally delivered can only answer "where it ended up," while this section wants to answer "why it could get here." Looking back over its evolution history (which is really no evolution at all — just day-one locking + continuous polishing — one transferable principle emerges: engineering structure is not "designed" into being, but "forced" into being by cost — and some of that cost was paid in full on day one.

What can be extracted from the real architecture is "the priority of constraints"

The first version paid the complexity all at once not because the author was clairvoyant, but because it recognized several untouchable hard constraints:

1. Device-handle ownership (who truly owns the microphone / audio stream) — top priority; it determines the process model: must be desktop, must hold the CoreAudio handle in the main process.
2. Latency budget (acceptable delay from sound emission to lip movement / playback) — it determines that audio features must sink into C++, and ASR/TTS must run locally.
3. Privacy and offline (audio/text never leave the machine, works without network) — it determines Whisper.cpp / Piper rather than a cloud API.
4. Build-chain complexity and code readability (direct node-gyp compilation of the native module + macOS-only) — sacrificed the most often, but never fully ignored.

The transferable principle: acknowledge "costs have priorities" before talking architecture

The answer from firefly-studio's retrospective is plain: before you start layering, first figure out which constraints are untouchable and which are negotiable (the four priorities listed in the previous section are exactly where this judgment lands). Device handle / latency budget / privacy-offline are the untouchable hard constraints; what remains — build-chain complexity, cross-platform, voice richness — is the negotiable space.

The traces in the codebase are fossils of constraint priority: useMic's silenceMs / minSpeechMs is the fossil of the requirement "VAD state machine"; loadNative() being memoized and only truly required on the first IPC hit, throwing straight through the entire audio/ASR/mic/IPC chain on load failure, is the fossil of the constraint "device-handle ownership." (The one-pole smoothed_energy_ in audio_engine as the fossil of "temporal inertia" was already pointed out next to that code block in the earlier section.) The way you read a codebase shifts from "what does this function do" to "which constraint conflict was this function originally created to resolve."

The final point: what transfers is not Electron + C++, but cost-calibrated judgment

Treating firefly-studio as a template for "Electron + C++ for digital humans" to copy would be wrong — its tech stack was forced out by concrete constraints: the physical nature of audio devices, macOS platform limits, and a local-first positioning. Change the scenario (say a purely cloud assistant, or an offline batch tool) and this structure immediately becomes a burden.

What truly transfers is the judgment of "making trade-offs on the priority of constraints." Any system's final form is "the inevitable result of hard constraints" plus "the compromise product of soft constraints." Once you separate the two, when someone asks "why is it designed this way," your answer is not "because best practice" but "because these few constraints are untouchable and everything else can yield."

firefly-studio never became a "beautiful architecture" from day one; it became a "system calibrated by cost." The difference between the two is what this section wants to sediment.

The complete source code is on GitHub: erishen/firefly-studio.