When AX Can’t Be Trusted: How to Keep Your AI Operating macOS

Prologue: Why a Video App

At 10pm, slumped on the couch, I said to my computer: "Open the video app, see what I've been watching, then go to the movie section, pick something rated 9+ that fits my taste, and just play it." My AI assistant agreed — and then froze.

Saying "fits my taste" sounds light, but behind it an LLM is inferring my profile and making the pick — and the "pull the few movies that best fit my taste from the catalog" layer is a textbook RAG scenario: hybrid retrieval over the taste profile and the catalog (keywords + semantics). That layer is what my other open-source project implements end to end: langchain-llm-toolkit, a LangChain + LiteLLM toolkit centered on RAG. I also wrote about its hybrid retrieval design in Hybrid Retrieval Architecture: When Keywords Meet Semantic Understanding. Feel free to read them alongside this article.

A video app is one of the harder classes of apps for a macOS desktop agent to drive: the UI is largely self-drawn, the AX tree can be nearly empty, and every button hides behind pixels. To an assistant that only knows "read the AX tree, click by node", it's a black box. Staring at the failure logs, it hit me: if my assistant can't drive a video app, it isn't an "assistant that can operate your Mac" — it's a "toy that can operate a few native apps".

So I made a video app my litmus test. Not because binge-watching matters, but because it represents the worst case: no accessibility hooks at all, just pixels. Crack it and the approach finally stops being limited to native apps whose AX trees are well-behaved; fail and a desktop AI assistant stays a demo forever. This article is about the dual-channel architecture I was forced to build to crack it — and if you want your AI to operate Mac apps whose AX can't be trusted, the thinking is the same.

Motivation

I was stuck on two things: getting an LLM to actually click and type, and the fact that macOS apps render their UIs in wildly different ways — some go through Accessibility properly, others draw everything themselves and leave the AX tree almost empty.

At first I thought getting AX working would be enough. The trio of toolClick, toolTypeText, and toolFocus made perfect semantic sense: find a node, perform an action, refresh the outline. Then reality hit: on a video app and a music app, treeOf(pid, 10) returns either empty AX nodes or nothing but meaningless containers. The LLM couldn't find anything there: findNodes came back empty and toolClick simply reported "not found, read_screen first".

So I added a second channel: synthetic HID coordinate actions. toolClickAt, toolDoubleClickAt, toolDrag, toolRightClickAt, toolScroll, toolScrollTo — these inject events straight at screen coordinates. Both channels share one SessionState: pid marks the current target app, outline holds the flattened AX tree, lastObservedAt records the timestamp of the last explicit observation, and every input tool passes through a staleObservation check first.

This design was never pre-planned architecture — it was forced out of me, step by step, by real scenarios. Five specific pitfalls, one by one below.

The Pitfalls: Five Defense Layers from a 35-Step Video-Picking Task

That "just play it" is a 35-step operation chain in practice. These five pitfalls aren't a generic architecture lecture — they're the traps I found by staring at the video app's task log, each one a real stall or misjudgment.

Pitfall 1: The Semantic Channel Fails — an Empty AX Tree, or One That Refuses

The first wall is the semantic channel: the LLM wants to operate by node, but the video app's AX tree is nearly empty and treeOf(pid, 10) yields nothing usable — that's failure mode one. Failure mode two is sneakier: Finder's sidebar is a textbook pitfall. treeOf(pid, 10) returns nodes advertising AXPress/AXOpen, and findNodes matches keywords like "Finder" and "Documents" — everything looks fine. But when performAction(pid, path, "AXPress", ...) runs, Finder returns -25205. The error code isn't documented clearly in the AX docs; in practice it turned out to be Finder rejecting certain self-built elements internally. My original toolClick just propagated the raw error, so the LLM got an opaque AXPress failed: -25205 and froze — it couldn't tell whether this was "element doesn't support click", a permissions problem, or a system bug.

So I changed toolClick's error handling:

try {
  await performAction(state.pid, target.path, target.actions[0], ...);
} catch (e) {
  return {
    result:
      `Clicking "${target.label}" (${target.actions[0]}) failed: ${e instanceof Error ? e.message : String(e)}. ` +
      `The element may have changed or doesn't actually support this action — re-read the screen to relocate, or fall back to click_at by coordinates.`,
    state,
  };
}

(Note: the original runtime messages are in Chinese; shown here in English for readability — the behavior and recovery paths are identical.)

That message went through many iterations. The first version printed only the raw error, and the LLM reading -25205 still had no idea what to do. Adding the hint "the element may have changed or doesn't support this action" gave the LLM two explicit escape routes: re-run read_screen to refresh the outline, or degrade to click_at by coordinates. This step is where the "pure AX single channel" really started moving toward dual channels — for self-drawn apps like video and music clients, the AX tree is nearly empty, findNodes finds nothing, and a coordinate channel that doesn't depend on the AX tree is the only way out.

There's a principle here worth lifting out on its own: low-level errors should not be handed to the LLM as-is — they should be translated into the next executable recovery path. The LLM doesn't need to know whether -25205 means "click not supported", "permission problem", or "system bug". It only needs to know what to do next.

Pitfall 2: Coordinate Drift — Move the Window, Kill the Coordinates

Coordinate actions brought their own new problem: the moment a window moves or the resolution changes, the LLM's guessed coordinates fly off target. In that 35-step run I maximized the window halfway through, and every old coordinate died — each step needed a fresh ocr re-locate. That was the first time I realized the coordinate channel needs a "guard". The (x, y) the LLM guesses is based on the screen resolution and window position it saw. If the user drags a window mid-task, or the resolution changes (e.g. plugging in an external display), the model's coordinates can land outside the target window — even on another app entirely. At its worst, the LLM clicked three times in a row, each click drifting 50px, and the third click hit the System Settings panel.

clampToWindow solves this. It takes the target's windowBounds and clamps the model's (x, y) into the window's actual frame:

export async function clampToWindow(
  pid: number | null,
  x: number,
  y: number,
): Promise<{ x: number; y: number; note: string }> {
  if (pid === null) return { x, y, note: "" };
  try {
    const b = await windowBounds(pid);
    if (!b) return { x, y, note: "" };
    const cx = Math.min(Math.max(x, b.x), b.x + b.w - 1);
    const cy = Math.min(Math.max(y, b.y), b.y + b.h - 1);
    if (cx !== x || cy !== y) {
      return {
        x: cx,
        y: cy,
        note: `(Coordinates (${Math.round(x)}, ${Math.round(y)}) are outside the target app window, corrected to (${Math.round(cx)}, ${Math.round(cy)}))`,
      };
    }
  } catch {
    /* can't get the window bounds, skip correction */
  }
  return { x, y, note: "" };
}

Notice the return value carries a note field — it's for the LLM to read, so it knows what happened. The first version silently changed the coordinates and moved on; the LLM would see its guessed (100, 200) become (95, 180) with no explanation, assume its reasoning was accurate, and keep guessing the same wrong coordinates next round. With note, the LLM gets feedback like "coordinates outside the window, corrected to …" and adjusts its strategy next round. The function also swallows windowBounds failures in a try/catch — right after launch an app may not have a frame yet, so it simply skips correction and stays invisible to the user.

Pitfall 3: Context Explosion — the 35-Step Task Overfeeds the History

modelToolResult in chat.ts grew directly out of this one. Originally I fed every read_screen output back to the LLM in full; the video-picking task from the prologue ballooned the message history from 2,000 characters to over 50,000 across its 35 steps (each read_screen returns roughly 1,500 characters, so 35 steps accumulate to 52,500). Token usage on free endpoints spiked and response latency went from 2 seconds to 12; the context window blew up long before the task ended.

export const MAX_TOOL_RESULT_LEN = 2000;
const TOOL_RESULT_TAIL_LEN = 300;

export function modelToolResult(result: string, limit = MAX_TOOL_RESULT_LEN): string {
  if (limit <= 0 || result.length <= limit) return result;
  const headLen = Math.max(limit - TOOL_RESULT_TAIL_LEN, 0);
  const head = result.slice(0, headLen);
  const tail = result.slice(-Math.min(TOOL_RESULT_TAIL_LEN, limit));
  const omitted = result.length - head.length - tail.length;
  return `${head}\n…(result too long, ${omitted} chars truncated; re-read for the full content)…\n${tail}`;
}

The truncation strategy was tuned through trial and error. "Keep only the first N characters" lost the tail conclusions, so the LLM often misjudged the task as complete; "random sampling" destroyed the AX tree's hierarchy and broke keyword matching. The final approach keeps head + tail + a truncation marker: the head carries the tree root and key paths, the tail usually holds conclusions like counts/summaries/status checks, and the middle shows how many characters were cut. When the LLM needs the full content, it proactively re-calls read_screen for specific nodes.

MAX_TOOL_RESULT_LEN defaults to 2000, but it isn't hardcoded — it's adjustable in Settings. AX trees vary wildly between tasks: a simple app's outline may be 300 characters, while a complex IDE's single-step outline can reach 5,000. I settled on 2000 because in practice 80% of tasks' per-step output fell in that range.

Pitfall 4: The Stale Outline — One Failed AX Call Wipes the Whole Session

Pitfall 1 is "the tree is empty"; this one is "the tree is broken". Beyond the video app, I was also re-testing the same kind of self-drawn UIs on a music app — refreshOutline was originally called unconditionally after every action, and one day on the music app, fetchTree returned stale data (the app's internal state had changed but the AX tree hadn't refreshed). I used that old outline to locate a node, findNodes matched an element that no longer existed, performAction failed, toolClick returned an error — but state.outline had already been overwritten with the invalid data. Every subsequent toolTypeText that looked up input fields from that outline failed.

It took 30 minutes of debugging to realize the outline had been wiped — fetchTree had died at some point and the whole session was trashed. The fix was a try/catch fallback in refreshOutline:

export async function refreshOutline(state: SessionState, depth = 10): Promise<SessionState> {
  if (state.pid === null) return state;
  try {
    return markObserved({ ...state, outline: await treeOf(state.pid, depth) });
  } catch {
    return state;
  }
}

On failure, keep the old state and never hand the LLM dirty data. Every action still tries refreshOutline afterwards; when it fails the old outline survives — a lesson learned from multiple crashes: a single failed AX call must never wipe the whole session.

Pitfall 5: Silent Truncation in Offline Quick Commands

The first four layers are all online scenarios. Offline quick commands are the opposite extreme: no LLM attached, just one pre-set command executed. When a message carries follow-up steps (e.g. "open Calendar, then use ocr to check today's date"), offline mode can only execute the first part — the remaining steps were originally silently dropped, and the user had no idea anything was lost. I added tailNote to tell the user exactly which steps didn't run and how to recover:

const tailNote = tail
  ? `\n\n📎 This message also contains follow-up steps ("${tail.slice(0, 40)}${tail.length > 40 ? "…" : ""}"). Offline quick commands can only execute a single command; configure an LLM and resend, and I'll run it in full.`
  : "";

One line that turns "the system didn't respond" into "here's what to do next."

These five pitfalls weren't figured out all at once — each one was fixed only after the previous approach hit a dead end. Pure AX wasn't enough, so I added a coordinate channel; coordinates drifted, so clampToWindow got its note feedback; context ballooned, so modelToolResult truncated; AX occasionally died, so refreshOutline got a try/catch; silent truncation lost context, so tailNote appeared. All five defense layers were forced into existence while cracking self-drawn UI apps like streaming clients. Every adjustment shifts the original assumption from "AX is reliably there" to "AX can fail at any moment" — and that is the core design philosophy of the dual-channel architecture: don't trust the reliability of any single channel; use redundancy and explicit degradation as the safety net.

Verification

After a full afternoon and the completed 35-step video-picking task from the prologue, I looked back at these five patches and found each one corresponded to a real crash or misjudgment. The verification methods differed — some were manual reproductions, some log statistics, some deliberately triggered edge cases — but I won't merge a patch into the mainline until it's been verified.

Coordinate drift. I wrote a simple test: the window's left edge started at x=50, I ran click_at 60 200 (10px from the left edge), then manually dragged the window 50px to the right (left edge now at x=100) and ran the same command again. Without clampToWindow, the second click landed outside the window on the desktop; with clamping, the return value went from (60, 200) to (100, 200) with a note attached:

(Coordinates (60, 200) are outside the target app window; corrected to (100, 200))

On receiving that note, the LLM re-runs read_screen to get the new window position instead of continuing to guess blindly. The test itself is simple — drag the window, repeat the command, watch where the click lands — but it's the cornerstone of the whole coordinate channel.

AX resilience. I repeatedly ran toolClick on Finder's sidebar, triggering -25205 every time. With the original error handling that returned only the raw error, the LLM stopped dead at -25205 and tried nothing further. With the new message:

Clicking "Finder" (AXPress) failed: action failed: -25205.
The element may have changed or doesn't actually support this action — re-read the screen to relocate, or fall back to click_at by coordinates.

The LLM's next behavior changed completely: it re-runs read_screen to refresh the outline, and if the element still can't be found, it degrades to click_at. This behavioral shift was measurable — I recorded 10 identical operations: before the fix 5 stalled outright; after, 8 auto-degraded to coordinate clicks.

Context explosion. With modelToolResult truncation in place, I reran the 35-step task: message history held steady under 8,000 characters and latency returned to around 3 seconds (the before-numbers are in Pitfall 3). The numbers came from sessionTranscript — logSession appends every exchange to a log file, so I just cat'd the file size. The two constants weren't picked arbitrarily either: across the 35 steps, 80% of read_screen outputs fell between 300 and 2,000 characters; only a few complex-IDE outlines exceeded 2,000.

Stale outline. This was the sneakiest pitfall. I ran a series of actions on a music app and toolTypeText suddenly started reporting "no text input area found" across the board. Investigation showed fetchTree had returned an empty array after one call, and refreshOutline had unconditionally updated state.outline, invalidating every outline-based operation that followed. After the try/catch fix, I reran the same flow: when fetchTree dies, the old outline survives and toolTypeText keeps working. This one needed deliberate reproduction — I forced fetchTree to fail under specific conditions (e.g. right after app launch before initialization completed) and watched whether the session crashed.

Blind operations. The core of staleObservation:

const STALE_OBSERVATION_MS = 60_000;

export function staleObservation(state: SessionState): string | null {
  if (state.lastObservedAt === null) {
    return "⚠️ No UI observation snapshot yet (no ocr / read_screen). Coordinate and input operations would be blind — run ocr or read_screen first.";
  }
  const age = Date.now() - state.lastObservedAt;
  if (age > STALE_OBSERVATION_MS) {
    return `⚠️ UI snapshot is ${Math.round(age / 1000)}s stale — coordinate/input blind operations may hit an outdated screen. Re-run ocr or read_screen first.`;
  }
  return null;
}

I wrote a test script: call toolClickAt without first calling ocr or read_screen — it returned the refusal message immediately. Then I set lastObservedAt 60 seconds in the past and called again — refused again. The 60-second threshold came from measurement: an average LLM operation chain completes within 30-60 seconds; beyond that window the UI state may have changed and blind clicking is too risky.

Letting an Agent Operate Isn't Letting It Do Anything

By now you've probably thought of the obvious question: an assistant that can "click anything" — is it too dangerous to hand your daily work? Safety design is the other baseline beyond the dual channel, three hard rules:

• Dangerous operations require confirmation: irreversible actions like delete, send, and log-out pause until you approve; the agent never executes them on its own.
• Password fields refuse auto-input: no password field is ever auto-filled; anything involving passwords is handed back to you.
• Sensitive directories require authorization: scanning personal directories only happens after your explicit approval.

These three aren't afterthoughts — they're constraints set at the architecture level: between "can operate" and "can do whatever it wants" sits a door that only a human can open.

Outcome

This is not a single-button demo — it's a continuous 35-step real operation chain. After the 35-step task, I finally saw this architecture actually run.

In the video app, the LLM first calls read_screen to get the AX tree (nearly empty), then goes straight to click_at for the play button; in a music app it matches the AXTextArea role via findNodes and uses toolTypeText to type into the lyrics search box; on Finder's sidebar, nodes that return -25205 make the LLM auto-degrade to coordinate clicks. Those three scenarios cover the main problems in this round of testing that I originally thought couldn't be done.

Those three scenarios are just the verification set, though — ax-agent was never positioned as "some video app's assistant". It's what the README says: a general-purpose UI observation + control base, not a shortcut tool. Beyond the 35-step smart chain, the mode I reach for daily is the lightest one, offline Chat: commands are parsed and executed locally — no LLM, no network, nothing leaves the process — so "open Notes" or "click into the file-transfer chat" is literally one sentence. Smart Mode isn't tied to any vendor either: any OpenAI-compatible API (DeepSeek / Qwen / Ollama / LM Studio…) plugs in, and there's an Inspector debug mode to walk elements one by one and drive them manually.

The whole thing finally ships as a Tauri local app under 9 MB — double-click and it's ready, source at ax-agent.

But looking back, this architecture wasn't designed in advance — it was forced into its shape by the five pitfalls above, one by one. My understanding now condenses to one sentence: when you're driving an unreliable external system, don't try to fix every fault point — design multi-channel redundancy and explicit degradation paths so that failures are perceivable, recoverable, and reportable.

Source Map

Repository: github.com/erishen/ax-agent

• src-tauri/src/ax_act.rs
• src-tauri/src/ax_core.rs
• src/chat.ts
• src/llm.ts
• src/tools/input.ts
• src/tools/shared.ts