AutoGen PSE Architecture Deep Dive: Building Reliable Multi-Agent Collaboration with the Planner/Specialist/Evaluator Pattern

Introduction: The Dilemma of Multi-Agent Collaboration

Since 2024, multi-agent frameworks represented by AutoGen, LangGraph, and CrewAI have emerged one after another. They let developers orchestrate multiple AI agents to work together, but in real-world deployment three core problems tend to surface:

1. Token explosion: In multi-round conversations, context keeps accumulating, and every retry appends new content on top of the history — a single request easily blows past tens of thousands or even hundreds of thousands of tokens, sending costs out of control.
2. Role ambiguity: The boundaries between agents blur, with planning and execution mixed together and the evaluator both judging and advising, until nobody can say where the problem actually lies.
3. No auditability: The agent's decision process is a black box — why did it PASS? Why FAIL? How many tokens were used? How much did it cost? Most frameworks leave this information missing.

These problems drove me to build autogen-pse, a Planner-Specialist-Evaluator (PSE) tri-role agent framework built on Microsoft AutoGen. This article parses its core ideas and implementation details from an architectural perspective.

Project: https://github.com/erishen/autogen-pse

1. The PSE Tri-Role Model: Why These Three Roles?

The PSE pattern draws inspiration from role division in software engineering:

• Planner → equivalent to a Tech Lead: breaks down requirements, makes plans, assigns tasks, and makes delivery decisions. Does no concrete execution.
• Specialist → equivalent to a Developer: writes the concrete deliverable. Only executes the tasks it is assigned.
• Evaluator → equivalent to a Code Reviewer: independently verifies deliverable quality and outputs a PASS/PARTIAL/FAIL verdict. Gives no advice, and does not rewrite the Specialist's code.

Role Constraints

Every role has a clear definition of what it can and cannot do:

Role	Responsibility	Constraint
Planner	Analyze requirements, break down tasks, assign execution, make delivery decisions	Does not write code, does not compute
Specialist	Execute specific tasks, write the deliverable	Only does what is assigned, reports when done
Evaluator	Independently verify the deliverable, output a verdict	Does not trust the Planner, gives no advice, only outputs PASS/PARTIAL/FAIL

This strict division brings several benefits:

• Clear responsibility: Any failure can be traced to a specific step. Was the plan wrong? A bug in execution? Or a flawed evaluation standard?
• Reduced hallucination: The Evaluator does not participate in execution; it only verifies, giving it a purer perspective that makes problems easier to spot.
• Auditability: Every Plan→Execute→Evaluate round is fully recorded.

2. Architecture Overview

                      User Entry
                 CLI / Makefile / Web
                         │
                         ▼
              ┌──────────────────────┐
              │    Orchestrator      │
              │  ┌────────────────┐  │
              │  │  Loop Engine    │  │
              │  │  step_buffer    │  │
              │  │  Trace Log      │  │
              │  │  Token Stats    │  │
              │  └────────────────┘  │
              └──────────────────────┘
                         │
                         ▼
           ┌─────────────────────────┐
           │   RoundRobinGroupChat   │
           │                         │
           │  Planner → Specialist   │
           │      → Evaluator        │
           │      → ToolAgent        │
           └─────────────────────────┘
                         │
              ┌──────────┼──────────┐
              ▼          ▼          ▼
          Planner    Specialist  Evaluator
          read_file  read_file   read_file
          bash       (read-only) bash
                                 pytest
                                 ruff
                         │
                    ToolAgent
                    write_file
                    read_file
                    bash
                    pytest
                    ruff

Why a Fourth Agent — ToolAgent?

Planner, Specialist, and Evaluator do not call tools directly. They send tool-call requests through DSML (Domain-Specific Markup Language), which the ToolAgent executes uniformly.

Planner: "I need to create a file with the following content: <tool_calls>..."
                                                            ↓
ToolAgent parses the XML, executes write_file, returns the result

This design follows the principle of least privilege:

• Planner has read_file + bash (can read files and run commands, but cannot write).
• Specialist has only read_file (can read, but cannot write or run commands).
• Evaluator has read_file + bash + pytest + ruff (can verify, but cannot write).
• ToolAgent has all tools, but only executes passively and makes no decisions.

3. Loop Control and step_buffer: The Key to Solving Token Explosion

This is the most core design of autogen-pse. Each task consists of multiple "Plan → Execute → Evaluate" loops:

Loop start
  ↓
Planner makes the plan
  ↓
Specialist executes and produces the deliverable
  ↓
Evaluator verifies the deliverable
  ↓
   ├── PASS → delivery complete, exit
   ├── PARTIAL → fix and retry (max 3 times)
   ├── FAIL → re-plan (max 2 times)
   └── BLOCKED / TIMEOUT → terminate

The step_buffer Strategy

In traditional multi-round dialogue, every retry passes the full conversation history to the agent. This causes token usage to grow linearly: round 1 = 10K tokens, round 2 = 20K, round 3 = 30K…

autogen-pse's step_buffer strategy does two things:

On PARTIAL retry: only the Evaluator's verdict summary (about 2000 tokens) is passed in as the new task prompt. The Planner only needs to know "what was wrong last time," not re-read the entire conversation history.

Original task: "Write a project progress weekly report"
                     ↓ FAIL
New task: "Last round was judged FAIL (reason: missing risk-assessment section).
          Please re-plan based on the original task."

On FAIL retry: clear the entire conversation context and regenerate the plan based on "original task + failure reason." This forces the Planner to produce a brand-new plan unpolluted by previous errors.

Original task: "Write a project progress weekly report"
                     ↓ FAIL × 2
New task: "PARTIAL reached 3 times, limit hit.
          Please assess whether any deliverable remains, or declare BLOCKED."

This strategy works remarkably well in practice. Taking a typical periodic-report task as an example, a complete PSE analysis typically spans 3 rounds of dialogue, consuming about 200K prompt tokens + 28K completion tokens, at a cost of roughly ¥0.65. Without step_buffer, the same task could easily exceed 500K tokens.

Loop Control Parameters

All parameters are configurable via environment variables:

MAX_PARTIAL_RETRIES = 3   # Max PARTIAL retries
MAX_FAIL_RETRIES = 2      # Max FAIL retries
TURNS_PER_CYCLE = 20      # Max turns per round (prevents infinite loops)

4. Tool System: Least-Privilege Design

autogen-pse's tool system reflects security best practices:

# Planner: can read and execute
create_planner(client, task):
    tools=[read_file, bash]

# Specialist: read-only
create_specialist(client, task):
    tools=[read_file]

# Evaluator: can read, execute, and test
create_evaluator(client, task):
    tools=[read_file, bash, run_pytest, run_ruff]

# ToolAgent: full permissions, but passive execution
create_tool_agent(client):
    tools=[write_file, bash, read_file, run_pytest, run_ruff]

The DSML Tool-Call Protocol

Agents do not call functions directly; instead they embed calls in text messages via XML tags:

<tool_calls>
<invoke name="bash">
<parameter name="command" string="true">python3 script.py</parameter>
</invoke>
</tool_calls>

The ToolAgent parses these tags, executes them, and returns results to the conversation. This indirection has several benefits:

1. Permission isolation: An agent can only request a tool call, not execute it.
2. Centralized auditing: All tool calls pass through the ToolAgent, making logging easy.
3. Flexible extension: To add a new tool, you only need to declare it in the ToolAgent's system prompt.

Token Tracking and Cost Accounting

The token consumption of every agent is precisely tracked:

class TokenTracker:
    def feed(self, message):
        if hasattr(message, "models_usage"):
            stats = self.report.agents[source]
            stats.prompt_tokens += usage.prompt_tokens
            stats.completion_tokens += usage.completion_tokens

Sample output report:

============================
📊 Token Consumption Report
============================
  Planner     | Rounds: 4  | Input: 48672   | Output: 2008    | Total: 50680
  Specialist  | Rounds: 4  | Input: 56152   | Output: 13472   | Total: 69624
  Evaluator   | Rounds: 4  | Input: 102396  | Output: 11942   | Total: 114338
  ToolAgent   | Rounds: 3  | Input: 3970    | Output: 263     | Total: 4233
---------------------------------------------------------------
  TOTAL       | Rounds: 15 | Input: 211190  | Output: 27685   | Total: 238875
---------------------------------------------------------------
  💰 Estimated cost: ¥0.6438
============================

5. Real-World Application: Structured Report Generation

A typical application of autogen-pse is structured report generation from multi-source data. This task adopts a two-layer architecture of "zero-LLM-cost pre-processing + LLM deep analysis":

Layer 1: Rules Engine (zero LLM cost)

make summarize

It reads structured business data (JSON/CSV) and runs several categories of rule checks:

• Data completeness: missing or anomalous fields
• Consistency check: conflicts across data sources
• Threshold breach: key metrics exceeding the warning line
• Structural anomalies: unbalanced distribution ratios

It also automatically fetches related external reference data. The whole process uses no LLM and costs nothing.

Layer 2: PSE Deep Analysis (LLM-driven)

make review

It takes the rules-engine output as input and hands it to the PSE trio for deep analysis. The Planner builds the analysis framework, the Specialist writes each section, and the Evaluator independently verifies.

If a RAG knowledge base is configured, it also retrieves relevant domain knowledge documents and injects them into the analysis context.

Technical Detail: How to Extract the Final Report from the Dialogue?

An interesting question: the PSE trio's final output is a conversation history — how do I extract a complete report from it?

The approach is not to have the agent write the report to a file (which risks race conditions), but instead, after execution, to parse the trace JSON and find the section marked ## 最终结论 (Final Conclusion) in the Specialist's last output:

for msg in reversed(trace["cycles"][-1]["messages"]):
    if msg["source"] == "Specialist" and "## 最终结论" in msg["content"]:
        report = extract_under_heading(msg["content"], "## 最终结论")

This "post-hoc extraction" is more reliable than "immediate writing," unaffected by conversation order and timing issues.

6. Web Dashboard

Beyond the CLI and Makefile, autogen-pse also ships a complete Web Dashboard:

Tech Stack

• FastAPI + Server-Sent Events: SSE lets users see the LLM output stream in real time
• Vite + React + Chart.js: frontend shows asset trend charts and execution history
• Docker multi-stage build: Node builds the frontend + Python runs the backend

Core Features

1. One-click execution: run a task with a button click and stream logs in real time
2. Asset trend chart: Chart.js plots total assets over time
3. Execution history: visualized list of the last 10 traces, color-coded by verdict
4. Trace details: click to expand each round's full conversation, filter token consumption by agent

7. The Full Experience: From Development to Deployment

Local Development

cp .env.example .env   # Configure API Key
uv sync                 # Install Python dependencies
make dev                # Start frontend Dev Server + API Server
make test               # Run 31 unit tests
make lint               # ruff code check

Docker Deployment

# Multi-stage build
FROM node:18 AS frontend      # Build React frontend
FROM python:3.12 AS backend   # Run FastAPI backend
COPY --from=frontend /app/web/dist /app/web/dist

Adding a New Task

Register a new task in three steps:

mkdir -p tasks/my-task/prompts
# 1. Write the three agents' system prompts
#    tasks/my-task/prompts/planner.md
#    tasks/my-task/prompts/specialist.md
#    tasks/my-task/prompts/evaluator.md
# 2. Write the entry script tasks/my-task/run.py
# 3. Register it in tasks/_registry.json

8. Tech Stack Overview

Layer	Technology
Agent framework	AutoGen RoundRobinGroupChat
LLM backend	DeepSeek / OpenAI-compatible
Web backend	FastAPI + SSE
Web frontend	Vite + React + Chart.js
Config management	pydantic-settings
RAG knowledge base	FAISS + Ollama Embeddings
Testing	pytest (31 cases)
Project management	uv + hatchling
Containerization	Docker multi-stage build

9. Design Principles Summary

Looking back at autogen-pse's architecture, several principles are worth borrowing:

1. Separation of Responsibilities

The Planner does not execute, the Evaluator gives no advice, the Specialist writes no tests. Each role does one thing, and only one thing. This makes the system more controllable and debuggable.

2. Least Privilege

The tools an agent can call strictly match its responsibility. In particular, only the ToolAgent can call write_file, which fundamentally prevents agents from accidentally tampering with files.

3. State Management

The step_buffer strategy solves the token-explosion problem in multi-round agent dialogue. The key insight: retrying is not the same as re-executing — on PARTIAL you keep context but compress it, on FAIL you clear it and start over.

4. Auditability

Every execution is fully recorded to a trace JSON file, including each round's verdict, token consumption, and elapsed time. Old traces are auto-cleaned after 7 days.

5. Progressive Cost Control

Use a rules engine for zero-cost pre-processing, and only call the LLM for deep analysis when necessary. This "rules first, AI second" approach is more economical and reliable than a pure-LLM solution.

10. Future Directions

autogen-pse is a multi-agent collaboration framework built around verifiability and traceability; its design is domain-agnostic. Planned improvements include:

• Parallel execution: support multiple Specialists working in parallel to improve efficiency
• Memory system: cross-session knowledge persistence, so agents remember the reasoning behind past decisions
• Rules-engine extension: generalize rule checks from a single example task into a generic assertion framework
• Richer dashboard: visualize the agents' thinking process and decision chains

Conclusion

Building a reliable multi-agent system is not about which framework you use, but about how you design role division, state management, and permission control. autogen-pse proves in practice one thing: giving each agent clear boundaries and explicit responsibilities matters far more than giving them more powerful capabilities.

If you are interested in autogen-pse, visit the GitHub project page: https://github.com/erishen/autogen-pse. Issues and PRs are welcome.

---

Originally published on erishen.cn, by Erishen Sun.

In this series

• Building a PSE Tri-Role Closed Loop with AutoGen: A Retriable, Traceable Agent Collaboration Framework
• When LLMs Start Deceiving Themselves: A Multi-Agent Framework Guarding Article Credibility with Regular Expressions
• LangGraph PSE: Explicitly Modeling PSE Tri-role Collaboration with State Machines
• Preventing Agents from Lying at the Source: Building a RAG-Grounded PSE Multi-Agent Framework with LlamaIndex Workflow