When LLMs Start Deceiving Themselves: A Multi-Agent Framework Guarding Article Credibility with Regular Expressions

🇨🇳 中文版

The Trust Crisis in Multi-Agent Systems

Have you ever encountered this situation: asking an LLM to review the output of another LLM?

Imagine you have built a three-role article generation pipeline: the Planner is responsible for structuring, the Specialist for writing content, and the Evaluator for quality assurance. Sounds perfect, right? But when you scrutinize the Evaluator's work, you discover a fundamental contradiction—the Evaluator itself is also an LLM.

It reviews articles written by another LLM, much like letting a suspect judge their own testimony. When the Specialist fabricates a non-existent class name or cites a non-existent file path /nonexistent/path.py, the Evaluator is likely to "approve" this content. After all, for an LLM, the boundary between fabrication and fact is inherently blurry.

This is the trust crisis in multi-agent collaboration: we rely on LLMs to verify the outputs of other LLMs, but LLMs themselves hallucinate.

The core value of the crewai-pse project lies not in it being "yet another CrewAI application," but in its counter-intuitive architectural design decisions—replacing LLM self-assessment with programmatic verification, avoiding CrewAI overhead with an independent repair loop, and preventing unauthorized access via sandboxing. These decisions reflect a profound consideration of reliability in multi-agent systems.

Design Decision 1: Replacing LLM Self-Assessment with Programmatic Verification

Intuitive Approach

Let the Evaluator Agent use grep to check if code references exist. After all, the Evaluator has the run_bash tool and can execute shell commands.

Actual Approach

In run.py, the author implements an independent _verify_article() function that uses regular expressions to extract code references and then verifies them using filesystem grep:

def _verify_article(article: str, source_dir: Path) -> tuple[list[str], list[str]]:
    """Programmatic verification: grep checks if code references exist. Returns (fictional list, verified list)."""
    refs = set(re.findall(r"`([A-Za-z_][\w._]*(?:/[A-Za-z_][\w._]*)*)`", article))
    for match in re.finditer(r"```(?:python)?\s*\n(.*?)```", article, re.DOTALL):
        for line in match.group(1).split("\n"):
            m = re.match(r"^\s*(?:def|class)\s+(\w+)", line)
            if m:
                refs.add(m.group(1))

This function performs two key actions:

1. Extracts all backtick-wrapped code identifiers: including file paths, function names, and class names.
2. Extracts def and class declarations from code blocks: ensuring that key identifiers within code examples are also verified.

Then, it validates each reference:

    for ref in sorted(refs):
        if len(ref) < 3 or ref.startswith("http"):
            continue
        if ref in PYTHON_KEYWORDS:
            continue
        # File path -> recursive search disk (searches .py and .md files)
        if "/" in ref or ref.endswith(".py") or ref.endswith(".md"):
            found = any(
                f.name == ref.rsplit("/", 1)[-1]
                for ext in ("*.py", "*.md")
                for f in source_dir.rglob(ext)
                if ".venv" not in str(f) and "__pycache__" not in str(f)
            )

Why It Is Better

There are several subtle design details here:

1. Deterministic Checks vs. Probabilistic Judgments

Regex matching and file searching are deterministic. If _verify_article() says a class name doesn't exist, it truly doesn't exist. This stands in sharp contrast to the LLM's "probabilistic judgment"—an LLM might "feel" a class name exists because it has seen similar naming patterns, even if it doesn't.

2. Distinguishing Between "Code References" and "Python Keywords"

    PYTHON_KEYWORDS = {
        "def", "class", "import", "from", "return", "yield", "raise",
        "try", "except", "finally", "with", "as", "if", "elif", "else",
        "for", "while", "break", "continue", "pass", "lambda", "and",
        "or", "not", "in", "is", "True", "False", "None", "self", "cls",
        # ... more keywords and built-in functions
    }

If these keywords are not excluded, words like def or class appearing in the article would be falsely reported as fictional code references.

3. Recursive Search Excluding Virtual Environments

if ".venv" not in str(f) and "__pycache__" not in str(f)

This is a very practical detail. Virtual environments and cache directories should not be included in the verification scope; otherwise, they would generate a large number of false positives.

Design Decision 2: Independent Repair Loop Bypassing CrewAI Orchestration

Intuitive Approach

When the Evaluator finds issues, the Planner replans, and the Specialist rewrites. Let CrewAI's orchestration engine handle the entire repair process.

Actual Approach

In the main loop of run.py, the author uses an independent OpenAI client to call the LLM directly for repairs, completely bypassing Crew orchestration:

    # Programmatic verification + automatic correction
    max_retries = 3
    for attempt in range(1, max_retries + 1):
        fictitious, verified = _verify_article(article, source_dir)
        print(f"\n{'='*60}")
        print(f"  Check (Attempt {attempt}) — Verified {len(verified)} items")
        if fictitious:
            # Separate code references from exaggerated terms
            code_refs = [f for f in fictitious if not f.startswith("[Exaggeration]")]
            exaggerations = [f for f in fictitious if f.startswith("[Exaggeration]")]

            print(f"  ❌ Fictional content {len(fictitious)} items: {', '.join(fictitious)}")
            if attempt < max_retries:
                print("  🔄 Auto-correcting...")
                fix_parts = []
                if code_refs:
                    fix_parts.append(f"**Fictional code references (do not exist in source code, must be deleted)**: {', '.join(code_refs)}")
                if exaggerations:
                    exagg_words = [f.split("—")[0].replace("[Exaggeration]", "").strip() for f in exaggerations]
                    fix_parts.append(f"**Prohibited exaggerated terms (must be completely removed from the article)**: {', '.join(exagg_words)}")

                fix_prompt = f"""The following article was found to have issues during verification. Please correct it.

{'\n'.join(fix_parts)}

**Rules**:
1. Fictional code references: Delete sentences or code examples containing the reference; do not creatively replace them.
2. Exaggerated terms: Delete the entire sentence containing the term; do not attempt to rewrite it.
3. Keep the rest of the article unchanged.
4. Output the corrected full article (starting from Front Matter); do not output explanations.

## Current Article
{article}"""
                try:
                    resp = fix_client.chat.completions.create(
                        model=fix_model,
                        messages=[{"role": "user", "content": fix_prompt}],
                        max_tokens=8192,
                        temperature=0.7,
                    )
                    article = resp.choices[0].message.content

Why It Is Better

1. Performance Advantage

Re-running the complete three-agent pipeline (Planner → Specialist → Evaluator) for every repair is very expensive. Directly calling the LLM API for repairs requires only one API call instead of three.

2. Precise Control

The repair prompt can directly specify "delete fictional content" rather than "creatively replace." This avoids the LLM introducing new hallucinations during the repair process.

3. Programmatic Fallback

    EXAGGERATED_TERMS = {
        # Exaggerated terms have been deleted as required
    }


def _strip_exaggerated(text: str) -> str:
    """Programmatically delete sentences containing exaggerated terms (split by period/newline)."""
    for keyword in EXAGGERATED_TERMS:
        # Clean up sentence by sentence based on sentence boundaries (Chinese period, newline, semicolon)
        lines = text.split("\n")
        cleaned = []
        for line in lines:
            if keyword in line:
                # Try to delete only the clause containing the keyword (split by Chinese punctuation)
                parts = re.split(r'([。；;])', line)
                filtered = []
                for i in range(0, len(parts) - 1, 2):
                    sentence = parts[i]
                    punct = parts[i + 1] if i + 1 < len(parts) else ""
                    if keyword not in sentence:
                        filtered.append(sentence + punct)
                # Handle the last segment (no punctuation at the end)
                if len(parts) % 2 == 1 and parts[-1]:
                    if keyword not in parts[-1]:
                        filtered.append(parts[-1])
                cleaned_line = "".join(filtered).strip()
                if cleaned_line:
                    cleaned.append(cleaned_line)
            else:
                cleaned.append(line)
        text = "\n".join(cleaned)
    return text

When the LLM repair also fails (up to 3 rounds), the programmatic fallback function _strip_exaggerated() directly deletes sentences containing exaggerated terms. This is the final line of defense, ensuring that unverified feature descriptions do not appear in the output.

Design Decision 3: Sandboxed File Access

Intuitive Approach

Give Agents unrestricted filesystem permissions so they can freely read any file.

Actual Approach

In tools.py, the read_file tool forces the path to be within PSE_ROOT:

# Project root directory, limiting file access scope
_PROJECT_ROOT = Path(os.getenv("PSE_ROOT", Path.cwd())).resolve()


@tool("read_file")
def read_file(path: str) -> str:
    """Read file content. Parameter path is the file path (limited to within the project directory)."""
    p = Path(path).resolve()
    if not str(p).startswith(str(_PROJECT_ROOT)):
        return f"[Error] Path exceeds project scope: {path}"
    if not p.exists():
        return f"[Error] File does not exist: {path}"
    if not p.is_file():
        return f"[Error] Not a file: {path}"
    return p.read_text(encoding="utf-8")

Why It Is Better

1. Security

Prevents Agents from reading sensitive files outside the project. Even if an Agent is induced to read /etc/passwd or user private keys, read_file will deny access.

2. Controllability

Flexible configuration of the root directory via the PSE_ROOT environment variable. This allows the same tool to be reused across different projects.

3. Handling Symbolic Links

_PROJECT_ROOT = Path(os.getenv("PSE_ROOT", Path.cwd())).resolve()

Using Path.resolve() handles symbolic links to prevent path escape. If _PROJECT_ROOT is a symbolic link, resolve() resolves it to the actual path, ensuring the security check is effective.

Complete Pipeline: From Source Code to Bilingual Articles

Now let's look at how the entire pipeline operates:

Source Code → Planner Outline → Specialist Draft → Verification/Repair → Final Chinese Article → English Translation

CrewAI Phase

In agents.py, three Agents are created and assembled into a Crew:

def create_crew(task: str | None = None) -> Crew:
    """Create PSE three-role Crew (Sequential process)."""
    return Crew(
        agents=[create_planner(task), create_specialist(task), create_evaluator(task)],
        process=Process.sequential,
        verbose=True,
    )

Note that Process.sequential is used here, meaning Agents execute in order: Planner plans first, then Specialist writes. The Evaluator is created and added to the Crew, but its task output is not consumed in the main flow—verification is handled by the programmatic _verify_article() function. The Evaluator is kept in the Crew to maintain architectural completeness, making it easy to enable LLM-assisted review when needed (e.g., semantic consistency checks), but the core verification logic currently relies on the more deterministic programmatic approach.

Programmatic Verification Phase

After the Specialist outputs the draft, the main loop in run.py begins verification:

    crew_output = crew.kickoff()

    # Extract Specialist's output from CrewOutput
    if crew_output.tasks_output:
        article = crew_output.tasks_output[-1].raw
    elif specialist_task.output:
        article = specialist_task.output.raw
    else:
        article = ""

Then it enters the verification loop until all fictional content is cleared or the maximum retry count is reached.

Translation Phase

Once the Chinese final version is ready, it is automatically translated into English:

    translate_prompt = (
        "Translate the following Chinese technical article to English. "
        "Keep ALL code examples, file paths, class names, and function names unchanged. "
        f"Output ONLY the translated article:\n\n{article}"
    )

The translation prompt specifically emphasizes "keeping all code examples, file paths, class names, and function names unchanged" to ensure technical accuracy is not compromised during translation.

Summary of Design Philosophy

The core viewpoint of crewai-pse is clear: the reliability of multi-agent systems does not come from having more Agents, but from smarter verification mechanisms.

This project demonstrates that when LLM outputs require factual accuracy guarantees, programmatic verification is more reliable than LLM self-assessment. Regular expressions and file searches are deterministic and do not suffer from bias due to the LLM's "confidence level."

Applicable Scenarios

Any AI application requiring a "generate + verify" closed loop can draw inspiration from these design decisions:

• Code Generation: Verify if generated code references real, existing APIs.
• Technical Documentation: Ensure examples and citations in documentation match the actual codebase.
• Research Reports: Prevent models from fabricating non-existent experimental results or references.

Insights

1. Do not over-rely on LLM self-assessment: Having an LLM check the output of another LLM is unreliable.
2. Programmatic verification is key: Replace probabilistic judgments with deterministic checks.
3. Repair mechanism independent of orchestration: Bypass complex Agent flows and use the LLM API directly for precise repairs.
4. Security sandboxes are essential: Even for "assistant"-type Agents, limit their filesystem access permissions.

crewai-pse is not intended to replace CrewAI or other multi-agent frameworks, but to show how to add a layer of reliable verification on top of these frameworks. This is a pragmatic approach: acknowledging LLM limitations and compensating for them with engineering means, rather than expecting the model itself to become "perfect."

---

Source Code Navigation

Module	File	Description
Main Pipeline	run.py	Core orchestration: CrewAI calls, programmatic verification, repair loop, translation
Verification Logic	run.py::_verify_article	Regex extraction of references + filesystem grep verification
Agent Definitions	agents.py	Creation of Planner/Specialist/Evaluator and Crew assembly
Sandbox Tools	tools.py	Implementation of sandbox path limits for read_file
Planner Prompts	planner.md	Instruction templates for five narrative styles
Specialist Prompts	specialist.md	Source code verification requirements and writing standards
Evaluator Prompts	evaluator.md	Verification steps and judgment format
Configuration Management	config.py	Environment variable loading

Quick Start

Installation

pip install crewai crewai-tools python-dotenv openai

Configuration

1. Copy .env.example to .env and fill in your OpenAI API Key.
2. Set the PSE_ROOT environment variable to point to your project root directory.
3. Add project configurations in tasks/project-articles/projects.json.

Running

cd tasks/project-articles
python run.py <project_name>

For example:

python run.py my-project

This will automatically generate a Chinese technical article for the project, along with an English translation.

This article was generated using the crewai-pse framework itself—the Planner structured it, the Specialist wrote the content, and programmatic verification ensured the accuracy of all code references and class names.