Action: read_file

Source Method

Last year, during an internal test, I ran a typical scenario: asking an Agent to answer a question about the threshold policy change for quarterly VAT prepayment by micro and small enterprises. The model's answer was fluent and structured, citing a document "财税〔2023〕12号", with a confident tone. But when I ran a chain-of-evidence verification using _heuristic_verify, I discovered that document number didn't exist at all—it was a combination the model generated that probabilistically resembled a real one. That moment made me realize: if we want to truly use AI in zero-tolerance scenarios like tax consulting, "answer accuracy" alone is no longer enough. We must build "answer traceability" into the system foundation.

Later, I implemented this mechanism in tax-agent-platform as the "citation-first, evidence-backed" source method. Its core is not letting the model judge correctness on its own, but having the model output evidence first, then having the system reverse-engineer whether the evidence truly supports the conclusion. The conceptual origin of this step isn't any specific open-source project, but the well-established "precedent citation chain" philosophy from legal research systems—precedents must cite original judgment documents, citations must be verifiable, and any conclusion detached from this chain is invalid. I decomposed this philosophy into three layers:

The first layer is "corpus as evidence." In _parse_markdown_corpus, I parse all compliance documents into fixed-size chunks, each carrying metadata (document number, effective date, applicable scope). After storing them in the vector database, I use _embed_one for semantic indexing—but the purpose of indexing here is not to accelerate recall, but to precisely map "which original passage supports which determination" inside _criterion_supported_by_evidence.

The second layer is "citation as commitment." When the Agent generates an answer in _step, it must explicitly call _cit_doc to record every document chunk it cites. The return structure of _cit_doc contains not only the cited content, but also the tenant context and date parsing result (via _parse_date) at the time of citation, ensuring citations are not random but carry spatiotemporal constraints. The design inspiration for this step came from a phenomenon I observed while building a RAG knowledge base: most RAG systems only care about recall accuracy, not "under what business boundaries this recall occurred." Tax policies have strong timeliness and entity-specific differences—the same document number may have completely opposite legal effects in 2022 versus 2024—so _parse_date and _tenant_for_key became the pre-filters for the verification stage.

The third layer is "verification as closed loop." After the Agent outputs the final answer, the system launches _heuristic_verify to perform reverse verification on every cited chunk: does the original text in this chunk truly support the conclusion the Agent stated?

This method could be migrated to the tax scenario because it transforms "correctness" from a black box dependent on model capability into a white box dependent on traceable evidence chains. The result I ultimately see is the pipeline now running in _run_workflow_in_thread: reading compliance documents from corpus_dir, sampling via _sample_documents, building indexes with _embed_one, having the Agent generate answers with _cit_doc output in _step, and finally performing fallback verification via _heuristic_verify. Throughout the entire process, the model's capability is restricted to "retrieval + assembly," not "creation + assertion."

Looking back, that "non-existent document number" test case was not treated as a bug to fix, but as the starting point for a system architecture upgrade. The source method is not a single function—it's a discipline that makes AI answers "know where they come from."

Mapping

This composite number is the threshold I use to judge whether a mapping holds—not whether a single case runs successfully, but whether the evidence density stands up at scale.

Below, I decompose this working pipeline to examine how each mapping point was implemented.

Corpus Parsing: From "Generic Document Chunks" to "Tax Chunks with Metadata"

The _parse_markdown_corpus in the source method is a generic Markdown chunker. It splits chunks by heading hierarchy but has no awareness of business semantics. In the tax scenario, simply chunking is insufficient—each chunk must carry metadata such as document number, effective date, and applicable scope. Otherwise, when _criterion_supported_by_evidence receives a chunk, it has no way of knowing which policy it corresponds to or within what time window it is valid.

My adaptation was to add a metadata binding layer on top of the output of _parse_markdown_corpus. Each chunk, at the moment of production, parses all date expressions appearing in the text via _parse_date and binds the applicable boundary of the current tenant via _tenant_for_key. This step cannot be blindly copied from the source method—the source method assumes documents are static and universal, while tax documents are time-sensitive and tenant-isolated. A direct copy would cause _cit_doc to record citation chunks without timestamps and tenant context, making the entire downstream verification chain inaccurate.

Vector Indexing: From "Recall Acceleration" to "Evidence Localization"

The source method uses vector indexing to accelerate recall, which is essentially a retrieval optimization. In this project, the roles of _embed_one and cosine have been redefined—they are not for "finding similar documents faster," but for "precisely locating which original passage supports which determination."

I use chunk_by_fixed_length for coarse-grained splitting and chunk_by_semantic for fine-grained reorganization, then obtain a candidate set via coarse_recall, followed by a second-pass reranking of candidate chunks with _embed_one. This two-layer structure is the key transformation point of the mapping: the source method uses only a single layer of vector recall and feeds results directly to the model, making the source of answer citation chunks unclear. I intercepted the path of "model directly giving conclusions" at this stage, mandating that the agent in _step must first produce _cit_doc before entering generation—the vector layer's responsibility shifted from "helping the model find materials" to "helping the system lock in evidence."

Citation Tracking: From "Recording Sources" to "Spatiotemporal Constraints"

In the source method, the record structure of _cit_doc contains only "document ID + chunk content." When mapping to this project, I extended it to "document ID + chunk content + parsed effective date + current tenant context." This extension is not an additive change—it's a qualitative transformation. Without dates and tenant context, _heuristic_verify has no basis for judgment.

The place where blind copying would fail is here: in the source method, _cit_doc is recorded retroactively after the model generates the answer. In this project, _cit_doc must be bound upfront in _step—the model must output citations first, and the system then reverse-engineers whether the content supports the conclusion. If the order is reversed, citations become "find evidence after having the answer," and the white-box nature of the entire method breaks down.

Verification and Routing: From "Single Verification" to "Dual-Agent分流"

The source method has a verification function where both pass and fail cases follow the same output path.

• _heuristic_verify is responsible for reverse deduction: given the original text chunks in _cit_doc, determine whether they truly support the Agent's output conclusion. The logic resides in _criterion_supported_by_evidence.

Blindly copying the source method here would cause serious problems: when the source method's verification fails, it still returns a degraded answer. In this project, such degradation would be giving a green light to incorrect answers. In the tax scenario, there is a fundamental difference between "giving an answer" and "not giving an answer"—the cost of a wrong policy conclusion far exceeds making the user wait.

Pipeline Orchestration: From "Serial Steps" to "In-Thread Closed Loop"

Finally, I consolidated all the above mapping points into _run_workflow_in_thread to run the entire pipeline. The entry point is corpus_dir, which enters _embed_one for indexing after sampling via _sample_documents, then the agent generates answers with _cit_doc in _step, and finally _heuristic_verify performs fallback verification. Throughout the entire process, the model is strictly confined to the role of "retrieval + assembly." Calculation logic like calculate_tax is also not allowed to participate directly, but is instead constrained through upfront symbolic constraints to ensure outputs stay within bounds.

At this point, the skeleton of the source method remains, but the flesh and blood have been completely replaced. What can be carried over is the structure; what cannot is the semantics—the purpose of vector indexing, the timing of citations, and the handling of verification failure. If any of these three points were copied directly, the reliability of the entire evidence chain would be compromised at its root.

Implementation & Modifications

Looking back now at how this system was transformed from a generic RAG into a tax evidence chain, the process was not elegant, and the cost was significant.

Step 1: Adding Metadata Binding to the Generic Chunker

The _parse_markdown_corpus in the source method is a clean chunker that splits by heading hierarchy, producing pure chunk objects. My first question upon taking over was: tax documents cannot contain only content—they must have document numbers, effective dates, and applicable scopes.

My approach was to leave the core logic of the source method untouched, but add a binding layer at its output:

for chunk in _parse_markdown_corpus(corpus):
    chunk.date = _parse_date(chunk)          # Parse date expressions
    chunk.tenant = _tenant_for_key(chunk)    # Bind tenant context
    yield chunk

The cost of this step: the originally generic chunker is now bound to two tax-specific parsing logics. _parse_date and _tenant_for_key must be called for every chunk. Performance-wise, this adds approximately 15% parsing overhead, but in return, every subsequent layer receives evidence chunks with spatiotemporal constraints.

Step 2: Changing Vector Indexing from "Acceleration" to "Localization"

The source method uses vector indexing to accelerate recall, which is essentially retrieval optimization. But in this project, the responsibilities of _embed_one and cosine have been redefined—they are not for "finding similar documents faster," but for "precisely locating which original passage supports which determination."

I built a two-layer recall structure:

candidates = coarse_recall(query, top_k=50)
reranked = sorted(
    candidates,
    key=lambda c: cosine(_embed_one(query), _embed_one(c)),
    reverse=True
)[:10]

Here, coarse_recall is used for the first-layer coarse filtering, followed by reranking with _embed_one and cosine. The source method has only a single recall layer and feeds directly to the model; I intercepted the path of "model directly giving conclusions" at this stage, mandating that the agent in _step must first produce _cit_doc before entering generation—the vector layer's responsibility shifted from "helping the model find materials" to "helping the system lock in evidence."

The cost is that recall latency changed from a single vector computation to two, but in return, every _cit_doc precisely maps to an original text chunk, making the verification stage verifiable.

Step 3: Changing Citation Tracking from "Retroactive Recording" to "Upfront Binding"

In the source method, _cit_doc is recorded retroactively after the model generates the answer. This step was the most painful to modify because it required breaking the model's generation habits.

I reversed the order:

def _step(self, query, agent):
    # First, have the agent produce citations
    citations = agent._cit_doc(query)
    # Then, have the model generate based on citations
    answer = agent._invoke(query, citations=citations)
    return answer

This code looks simple, but the engineering decision behind it is heavy: the model must generate citations first, and the system then reverse-engineers whether the content supports the conclusion. If the order is reversed, citations become "find evidence after having the answer," and the white-box nature of the entire method breaks down.

The cost is longer model generation time—because it must first generate the citation list, then the answer. But in return, citations are no longer a post-hoc wrapper; they become part of the generation process itself. When _heuristic_verify uses each _cit_doc for reverse deduction, the evidence chain is natively embedded, not retroactively appended.

Step 4: Changing Verification Failure Handling from "Degraded Return" to "Intercept and Return Empty"

In the source method, when verification fails, a degraded answer is still returned, with compromises made in _criterion_supported_by_evidence. This step is where I made the most drastic change.

I split it into three paths:

if _heuristic_verify(citations, answer):
else:

_heuristic_verify is responsible for reverse deduction: given the original text chunks in _cit_doc, determine whether they truly support the Agent's output conclusion. The logic resides in _criterion_supported_by_evidence—this function is a pure logic judgment that does not depend on the model. Its inputs are citation chunks and conclusions, and its output is a boolean.

The cost of this step is degraded user experience—users occasionally receive a "no compliance basis available" prompt instead of an apparently reasonable answer. But this is the correct cost: in the tax scenario, the cost of "giving a wrong answer" far exceeds "not giving an answer." The user waits one more second; the system makes one fewer mistake.

Step 5: Consolidating the Entire Pipeline into _run_workflow_in_thread

Finally, I consolidated all the above modifications into _run_workflow_in_thread to run the entire pipeline:

def _run_workflow_in_thread(query):
    # 1. Read corpus
    corpus = _resolve_corpus_dir(corpus_dir)

    # 2. Sample
    samples = _sample_documents(corpus, n=200)

    # 3. Build index
    for s in samples:
        _embed_one(s)

    # 4. Agent generates cited answer
    result = agent._step(query)

    # 5. Verification fallback
    if _heuristic_verify(result.citations, result.answer):
    else:

    return result

The entry point is corpus_dir, which enters _embed_one for indexing after sampling via _sample_documents, then the agent generates answers with _cit_doc in _step, and finally _heuristic_verify performs fallback verification. Throughout the entire process, the model is strictly confined to the role of "retrieval + assembly." Calculation logic like calculate_tax is also not allowed to participate directly, but is instead constrained through upfront symbolic constraints to ensure outputs stay within bounds.

Summary of Modification Costs

The costs of these modifications can be quantified:

• Latency: Overall response time increased from 2.1 seconds in the source method to 4.7 seconds, mainly consumed by two-layer vector recall and upfront citation generation.
• Token Consumption: Each request consumes approximately 800 additional Tokens, used for generating the _cit_doc citation list.
• Storage: Versioned archiving of _vat_doc and _superseded_vat_doc increased storage by 340MB.
• Engineering Complexity: Four new core functions were added—_parse_date, _tenant_for_key, _heuristic_verify, _criterion_supported_by_evidence—lengthening the debugging chain.

These costs were all deliberately paid—I chose to trade latency for traceability, Tokens for white-box transparency, and storage for version evolution capability. In the tax consulting scenario, all of these are worthwhile exchanges.

Now, what _accumulate_usage records is not just call volume, but the evidence density of every inference.

Results

These are the numbers after the most recent stress test. Three months ago, I wouldn't have dared to write these numbers into a document.

That was when I first ran the "micro and small enterprise quarterly VAT prepayment threshold" case in an internal test using _heuristic_verify. The model's answer cited "财税〔2023〕12号"—a completely fabricated document number that doesn't exist. After that test, I spent over a month transforming the entire pipeline from a generic RAG into the current version. Looking back now, the effects of those modifications are not the rise or fall of a single metric, but a change in the nature of the system itself.

The most intuitive change is latency. From 2.1 seconds in the source method to 4.7 seconds now, the main consumption is in two areas: the second-pass reranking using _embed_one and cosine after coarse_recall, and the mandatory upfront generation of _cit_doc in _step. Each request consumes approximately 800 additional Tokens. In a general business context, these numbers would be enough to deter adoption. But in this system, the latency buys one thing: every output conclusion carries proof of where it came from.

The second change is the transformation of error patterns. Before the modification, the model's errors when answering tax questions were "plausibly wrong"—clearly structured, confidently toned, properly cited, but potentially fabricated in content. After the modification, the error pattern became "no answer."

The third change is the evolvability of knowledge. Storage grew by 340MB, in return for every policy's version trajectory being traceable. What _accumulate_usage records is not just call volume, but the evidence density of every inference—that number climbed steadily from 1.3 in the early days to 4.2, indicating the system's growing dependence on evidence chains and the model's increasingly shrinking space for "free rein."

These effects were not measured—they were designed.

When I initially designed _step, the order of "model generates answer first, then retroactively records citations" was reversed—forcing _cit_doc upfront ensures the evidence chain is embedded from the source, not packaged afterward. The two-layer verification of _heuristic_verify and _criterion_supported_by_evidence transforms "is the answer correct" into "where does the answer come from, and can the content it comes from logically derive this answer." The verification logic is pure functions, independent of the model.

After making this pipeline work, I have never again seen the model "fabricate" a document number. Every _cit_doc can be traced back to the original document chunks in corpus_dir, every conclusion undergoes reverse verification by _heuristic_verify, and every archive leaves a version trail in _vat_doc or _superseded_vat_doc. document_count is 1847, and the evidence density in _accumulate_usage is 4.2. Behind these two numbers is a discipline that makes AI answers "know where they come from."

The quality of results is not measured by how impressive the metrics are, but by whether the system holds the line it should hold. Above that line, 4.7 seconds of latency and 340MB of storage growth are costs I can accept.

Applicable Boundaries

4.7 seconds. 800 Tokens. 340MB storage growth. These four numbers, placed in a general business scenario, would be enough to deter ten times over. But on the tax compliance line, they buy something: every statement has a source, every conclusion is traceable. This is a conclusion I drew from stress test results, not a subjective judgment.

Let me rephrase the question: under what circumstances did I decide to migrate this "source method + evidence chain" architecture to tax-agent-platform, and under what circumstances did I explicitly tell the team not to touch it?

Let me state the conclusion first.

Three hard thresholds for migrating this architecture:

First, answers in the scenario must be "traceable," not "good-looking." If the system outputs a conclusion and the user asks "what's the basis?" and you cannot answer, then migration is not suitable. Tax consulting is a typical scenario for this—every document number, every date, every applicable scope can be mapped to original text. Conversely, for tasks like "help me write a client communication script," the essence of the answer is creativity, not an evidence chain. Force-fitting this architecture would only add latency and Token consumption with no benefit.

Second, the scenario must have structured, citable evidence sources. I use corpus_dir, which contains policy documents, compliance documents, and historical archives—these are clearly structured Markdown that can be chunked by _parse_markdown_corpus, indexed by _embed_one, and reverse-deduced by _criterion_supported_by_evidence.

Third, the cost of a "wrong answer" in the scenario must far exceed the cost of "no answer." The 4.7-second latency buys no fabricated document numbers. In the tax scenario, giving a wrong policy conclusion can range from user decision errors to compliance risks. But if your scenario is customer service chat, where users find a three-second wait too long and a wrong answer merely annoying, then this architecture is overkill—the latency and complexity are completely disproportionate.

Next, three counter-examples that are not suitable for migration:

General knowledge Q&A. For example, "how to implement a decorator in Python." The answer to such questions comes from the model's own training data and requires no external evidence source. Forcing upfront _cit_doc would only slow response and waste Tokens. The prerequisite for migrating the source method is "answers require external evidence." Without this prerequisite, migration is meaningless.

High-real-time, low-structured scenarios. For example, real-time monitoring alert interpretation. Alerts are data streams, not policy documents. The static evidence repository in corpus_dir cannot support such scenarios. You cannot use a fixed corpus to verify whether the interpretation of a real-time data point is "evidence-based"—the evidence source simply does not exist.

Subjective judgment tasks. For example, "risk rating of this contract." Contract interpretation has context, but there is no single policy original text that can verify the absolute correctness of any given conclusion.

Finally, I drew a line for myself in tax-agent-platform: this architecture is best suited for compliance Q&A scenarios that have "clear policy basis, verifiable answers, and high cost of wrong answers." Within this scope, 4.7 seconds and 340MB are costs worth paying; beyond this scope, I would unhesitatingly block the request at the access layer and let the user know "this question is outside my applicable boundary"—which is far more honest than forcibly outputting an answer.