Building a Conversational 3D Avatar with React Three Fiber — Real Architecture and Deployment Lessons

Wiring an LLM into a 3D scene so a VRM character can hear you, talk back, and even follow you with her eyes sounds cool — but in practice it throws a chain of engineering problems at you: the API key must never reach the browser, Serverless functions have a hard 10-second ceiling, R3F's <Canvas> doesn't propagate React Context, and speech recognition is unreliable in some regions…

This isn't an API walkthrough. It follows the actual decision order I used while building firefly (a React + React Three Fiber 3D avatar web app, live at firefly.erishen.cn). For each piece I state the problem first, then why it was designed that way, with real code from the version running in production — not a generic template.

1. Stack and Architecture Overview

Start with the hard requirements before picking tools:

• Web-only, no desktop runtime;
• UI deeply coupled with the 3D scene: chat bubbles, input box, language switch, and a "back home" button all float above the canvas;
• LLM streaming must flow through React's state.

Final stack:

• Frontend: React 18 + Vite + React Three Fiber + drei + @pixiv/three-vrm (VRM loading/driving)
• Backend: a lightweight Node.js proxy (zero-dependency native http/fetch) + Vercel Serverless Functions, responsible for hiding the LLM key and orchestrating tool calls; the license-protected model file is hosted on a separate private backend (see section 3), decoupled from the chat backend

A common misconception: many "R3F avatar" tutorials write the backend in Python/FastAPI. firefly's chat backend is pure Node — because Vercel's Node function signature is exactly the standard (req, res), the SSE streaming proxy can be reused directly without spinning up a separate Python service for the LLM. The license-protected model asset is instead hosted independently on a private backend (see section 3), decoupled from the chat backend.

Data flow:

Browser (R3F canvas + React UI)
   │  same-origin /api/chat only
   ▼
Node proxy (proxy.mjs / Vercel api/chat.js)
   │  ① hide real LLM key  ② intent prefetch + tool orchestration
   ▼
OpenAI-compatible model gateway (key held server-side)

2. Security Baseline: Keep the LLM Key Server-Side

The scariest failure mode for a chatbot: hard-code the API key in the frontend or bundle it, so anyone opening DevTools can steal it. firefly's rule is the browser only ever talks to the same-origin /api/chat; the real key lives only in a server-side .env (local) or platform env var (prod).

server/config.mjs centralizes config and prefers process.env (Vercel-injected), falling back to the project-root .env:

export const BASE  = val('LLM_BASE_URL') || ''
export const KEY   = val('LLM_API_KEY') || ''
export const MODEL = val('LLM_MODEL') || ''

// Deploy safety: BIND_HOST defaults to 127.0.0.1 (local only); in prod set
// ALLOWED_ORIGINS to the frontend domain (e.g. https://firefly.erishen.cn).
// Never use '*' — that lets any site abuse your key.
export const ALLOWED_ORIGINS = val('ALLOWED_ORIGINS') || ''

proxy.mjs also runs a connectivity self-check at startup and gives a friendly message if unconfigured. Key point: the key never enters the browser, the source, or git — .env is git-ignored.

3. VRM Loading and "Instant" Open

VRM is an open standard for humanoid 3D models. @pixiv/three-vrm converts a .vrm into a Three.js object. firefly's loader lives in src/components/AvatarVRM.jsx: register VRMLoaderPlugin on GLTFLoader, then in the parse callback disable three-vrm's built-in lookAt (we drive bones manually) and record the head / leftEye / rightEye standard bones.

The part that actually affects UX is load speed. The model is 14.5MB, so re-downloading every time is painful. Two optimizations:

1. Progress-aware download (readWithProgress computes percentage from Content-Length, renders a loading bar);
2. Browser Cache API byte cache — cache the parsed model bytes by URL, so a refresh/revisit hits cache and opens instantly.

const cache = await caches.open(MODEL_CACHE_NAME)
const hit = await cache.match(url)
if (hit) { onProgress(100); return hit.arrayBuffer() } // cache hit → skip download

License-protected model file: the avatar's VRM license forbids redistribution, so it can't be bundled into the frontend or published with the repo. Instead it's hosted on a separate private backend service, where all authentication and distribution logic lives server-side and no credentials ever reach the client.

One-line principle: model hosting is decoupled from the frontend; authentication stays server-side and credentials never touch the client.

4. The "Alive" Feeling: Gaze, Mouth, Blink

What makes the character feel alive is per-frame bone and expression driving in useFrame. firefly's real implementation:

Gaze follows the pointer (head / both eyes use humanoid standard bones, smooth interpolation toward pointer position, while recording the rest pose so we don't overwrite it):

useFrame((state) => {
  if (!vrm) return
  vrm.update(delta) // drive spring bones (hair/skirt physics) + expression transitions

  const head = headRef.current
  if (head) {
    const tx = state.pointer.x * 0.35       // head turns with mouse/touch
    const ty = -state.pointer.y * 0.25
    head.rotation.y += (baseHead.y + tx - head.rotation.y) * 0.1
    head.rotation.x += (baseHead.x + ty - head.rotation.x) * 0.1
  }
  // eyes do the same (smaller coefficient) → "eyes follow you"
})

Mouth: while speaking (speakingRef is true), the mouth morph cycles a/i/u/e/o to a beat — a stylized lip-flap, not audio-accurate phoneme alignment. Without phoneme annotation data this still reads clearly as "she's talking".

Blink is similar: prefer expressionManager's blink, but fall back to driving the mesh's morphTargetInfluences directly (the underlying まばたき blink morph) when a VRM0.0 export ships an empty expression set. This "dual-path probe + fallback" makes blinking/talking work across differently-sourced VRM models.

5. Voice Pipeline: Input vs Output (Usable vs Region-Limited)

firefly's voice lives in src/hooks/useVoice.js, split into two independent paths:

5.1 Speech Input (STT) — unreliable in some regions

Uses the browser-native SpeechRecognition (Chrome/Edge). Its recognition backend is Google's speech service, which often can't be reached on restricted networks, so recognition fails. The UI degrades gracefully: if the API is absent it switches to text input, and recognition failures surface as a bubble error — text input is always the primary path.

5.2 Speech Output (TTS) — works offline

This is frequently misunderstood. firefly does speak aloud, using the browser-native speechSynthesis (system voices, available offline, no regional block). Before reading it strips emoji/symbols (keeping only text), and prefers a "Chinese female" voice with a slightly raised pitch to fit the "maid Xiao Fei" persona:

const u = new SpeechSynthesisUtterance(stripSpeechSymbols(text))
u.lang = lang
u.pitch = 1.2 // slightly higher, cute/young-girl feel

So the real situation is: she can be heard (TTS works offline) but may not hear you well (STT is region-limited). I also state this honestly in the UI so users don't think the button is broken.

6. The Hard Battle: Fitting "Two LLM Rounds + Tools" into Vercel's 10 Seconds

This is the most worthwhile part of firefly, and the root cause of the original repeated 504 / "no response" errors.

The avatar needs live capabilities like weather and a product list, implemented via OpenAI-compatible function calling: the model gets tools in round 1 and decides whether to call one → the server executes it → the result is fed back → a second round (without tools) produces the final natural-language answer. The frontend doesn't change.

The problem: Vercel Hobby Serverless functions have a hard 10-second ceiling; past that the platform SIGKILLs the function, which looks like a silent no-response ("asking about weather does nothing"). And "LLM1 + weather + LLM2" easily blows past 10s.

My fix is server-side intent prefetch (single-round optimization): the moment a request arrives, regex-check whether the user is asking about weather/products. If so, fetch the data up front on the server, inject it as system context, and drop the corresponding tool from round 1, so the model answers directly — collapsing the flow from "LLM1 + weather + LLM2" to "weather + LLM1" in a single round.

// chat.mjs: weather intent pre-check + server-side prefetch
const weatherIntent = WEATHER_INTENT_RE.test(lastUserMsg.content || '')
if (weatherIntent) {
  const w = await fetchWeather(wArgs)        // fetch ahead on the server
  if (w.ok) {
    weatherContext = `\n[System weather context, answer directly, do not call the weather tool] ${w.summary}`
    round1Tools = TOOL_DEFS.filter(t => t.function.name !== 'get_weather') // drop the tool
  }
}

Three more reinforcements: the upstream LLM call uses fetchWithTimeout with a 25s circuit breaker; vercel.json sets maxDuration: 60 and deploys to the nearer hkg1 region (effective on Pro; Hobby clamps to 10s but doesn't error); in direct-connect environments an undici Agent({ keepAlive: true }) reuses TLS to cut cold-start handshake overhead.

Weather/products themselves are memory-cached (10 min by lat/lng, 24 h for geocoding) so repeated demo questions are near-instant.

7. The i18n Trap: R3F Doesn't Propagate React Context

firefly is bilingual (zh / EN: UI text + conversation language following). The natural approach is a React Context I18nProvider, but there's a trap: R3F v8's <Canvas> uses its own reconciler and does not propagate outer React Context, so components inside the canvas (e.g. the loading layer) can't see the outer t().

The fix is a module-level store + useSyncExternalStore, so both in-canvas and out-of-canvas components read language from the same external store and re-render on switch:

// src/i18n/store.js: not dependent on React Context
const listeners = new Set()
export function setLang(lang) {
  currentLang = lang
  localStorage.setItem(STORAGE_KEY, lang)
  listeners.forEach(fn => fn(currentLang)) // notify all subscribers (incl. in-canvas)
}
export function subscribe(fn) { listeners.add(fn); return () => listeners.delete(fn) }

Language switch, chat bubbles, and the 3D loading layer all follow correctly, and the preference persists to localStorage.

8. Deployment and Build

firefly deploys on Vercel (not GitHub Pages). vercel.json:

{
  "framework": "vite",
  "buildCommand": "npm run build",
  "outputDirectory": "dist",
  "regions": ["hkg1"]
}

The frontend is the static dist; the backend is api/*.js Serverless Functions — which directly import and reuse server/chat.mjs's handleChat (standard (req, res) style, matching Vercel's Node function signature, no adaptation needed for streaming SSE).

Key build decisions:

• Disable Vercel's default body parsing (api: { bodyParser: false }) and let handleChat read the raw stream itself, so SSE streaming isn't broken;
• maxDuration: 60: Hobby clamps to 10s without error; on Pro this value takes effect and greatly relieves the two-round LLM timeout;
• CORS whitelist: ALLOWED_ORIGINS must be the frontend domain, never *, or any site could abuse your LLM key;
• Model hosted separately: the license-protected VRM isn't bundled into the frontend; it's hosted on a separate private backend with server-side authentication (see section 3).

9. Summary: What This Architecture Actually Solves

Looking back across the whole chain, what firefly really solves isn't "writing less Three.js" — it's plugging the easy-to-trip hazards ahead of time:

1. Security: LLM key never leaves the server; CORS whitelist prevents key abuse, and model assets are authenticated by the private backend (see section 3);
2. Performance/availability: model byte-cache for instant open, weather/product memory cache, intent prefetch collapses "two LLM rounds" into "one" to fit the Serverless 10s ceiling;
3. Cross-renderer boundary: a module-level store bypasses R3F's no-Context-propagation trap, keeping i18n consistent inside and outside the canvas;
4. Honest degradation: when speech input is region-limited it says so and falls back to text; TTS uses the browser native layer to stay usable everywhere.

The cost is R3F's abstraction overhead (virtual DOM diff + per-frame reconcile), but for a single-avatar scene that's entirely acceptable. If you're building a similar interactive 3D + LLM project, I hope this "in the order the pitfalls actually appeared" write-up saves you a few loops.

Try it live at firefly.erishen.cn. firefly currently supports Chinese / English switching, with a one-click "Back to home" button in the top-right corner.

FAQ

Source Navigation

• src/components/Stage.jsx — VRM model loading and byte cache
• src/components/AvatarVRM.jsx — VRM parsing, gaze / mouth / blink driving
• src/hooks/useVoice.js — speech input (STT) / output (TTS) pipeline and degradation
• src/i18n/store.js — module-level language store (bypassing R3F Context)
• server/chat.mjs — /api/chat streaming proxy, tool orchestration, intent prefetch
• server/config.mjs — config parsing and timeout circuit breaker
• server/weather.mjs — server-side weather tool prefetch
• vercel.json — deployment region and maxDuration config

Full source: https://github.com/erishen/firefly

Related posts

• Spring RBAC microservice
• AI contacts assistant
• Desktop AI workbench