DEV Community: Muhammad Ahmad Nazir Ahmad

Building Reliable AI Agents: Why LLMs Can't Verify Their Own Output (And What to Do Instead)

Muhammad Ahmad Nazir Ahmad — Thu, 22 Oct 2026 06:10:18 +0000

If you are building an AI agent that connects to a database, queries a knowledge base, or cites documents, you are likely dealing with a fundamental misconception: You cannot prompt-engineer an LLM into being perfectly factual.

In a recent industry survey of 800+ decision-makers, 40% cited security and compliance as the #1 barrier to scaling agentic AI. 46% specifically worry about indirect prompt injection and tool poisoning. Most teams are operating in what the report calls "leap-of-faith mode"—trusting the LLM to faithfully report what it found in a database.

I know this trap intimately. I built a local AI agent using LangGraph, Ollama, and ChromaDB connected to a live PostgreSQL ERP database. During adversarial testing, it confidently invented two fake item IDs with fake stock levels, directly contradicting the empty database result it had just received.

This is the story of how I stopped trusting the LLM to verify its own output, and built a three-layer architecture using deterministic guardrails, exact-match verification, and scope-routing to make the agent safe enough for production.

The Architecture: Prompt Establishes Intent, Code Enforces It
My first instinct was to write a massive system prompt: "Never fabricate data," "Always ground your answer in retrieved context."

It worked 90% of the time. But 90% is a failing grade in enterprise software. I had to rip the safety out of the prompt and move it into the graph state. The architecture now looks like this:

1.The Prompt: Establishes the persona and the intent to be helpful.

2.The Tool: Executes the database query or vector search.

3.The Backstop (Deterministic Code): Reads the raw tool output from the graph state and overrides the LLM's prose if a known failure shape appears.

1.The Deterministic Backstop Node

I did not use a separate validation LLM (which just adds latency and another point of failure). I used a deterministic Python wrapper that checks the state before the final response is generated.

If the database tool returns an empty array or a hard error, the code intercepts the response and replaces the LLM's output entirely:

def apply_tool_result_guards(state, response):
    """
    Deterministic backstop: if the last DB query came back empty or errored,
    do not trust the model's prose about what it found.
    """
    last_tool_message = state.get("messages", [])[-1].content

    if _is_no_data(last_tool_message) or _is_error(last_tool_message):
        return "No matching data was found for that query. The data may not " \
               "exist in the database yet, or try different filters."

    return response

The Lesson: The LLM is simply not allowed to write the final sentence about what it found if the data came back empty.

2. The Whitelist over the Correctness Checker

When asking the agent for Islamic citations, it once cited verse 5:51 (which is about interfaith alliances) for a question about usury (riba). My original regex only caught Surah X:Y formatting, so it missed (Quran 5:51).

Instead of asking the LLM to check if the citation is correct (a probabilistic, hard problem), I built a whitelist check (a deterministic, easy problem).

_ALLOWED_CITATIONS = {"2:275", "2:282"} # Pre-verified list

def enforce_citation_guard(text):
    for match in _CITATION_PATTERN.finditer(text):
        if not any(allowed in match.group(0) for allowed in _ALLOWED_CITATIONS):
            # Block even if the citation is real, if it's not on the whitelist.
            return SAFE_FALLBACK_MESSAGE 
    return text

If the model guesses a real verse that isn't on the whitelist, it gets blocked. Boring and reliable beats smart and hallucinatory.

3. Never Let the Model Do Entity-Matching

During testing, the agent correctly identified the lowest-scoring supplier from a sorted SQL query, but attached that value to the wrong row's identity. Two suppliers happened to share a display name, and the model mixed the fields.

The fix? Don't hand the LLM raw JSON and hope it reads it correctly. Modify the tool to state the answer explicitly:

# Bad: Return raw JSON
return json.dumps(rows)

# Good: Force the tool to state the entity match explicitly
return "Row 1, id=4 (Supplier: Acme Corp) has the lowest value: $4,500. Do not combine this with fields from a different row."

Layer 2: The Abstration Problem (Why 4 Local LLMs Failed)

When scaling to a 196-question benchmark, I needed the agent to refuse to answer questions that were out of scope (e.g., asking for a specific ruling that isn't in the text, or asking about modern cryptocurrencies).

I tried to use a local 7B LLM as a relevance judge. The prompt was simple: "Given this question and these 5 retrieved texts, decide if the text answers the question. If not, abstain."

I tested command-r7b-arabic, llama3.1, llama3.2:3b, and phi3. I ran 24 draws.

The result: 0 correct abstentions on out-of-corpus questions.

When asked "What is the ruling on horse meat?" (a topic that has nearby food prohibitions in the text, but no direct ruling), the models didn't say "I don't know." They grabbed the most topically-adjacent verse and confidently presented it as the answer.

This is the capability floor. LLMs are great at "Topic Matching." They are terrible at "Relevance Judgment."

The Fix: Two-Layer Scope Routing

Instead of asking the LLM to judge relevance, I split the task:

Layer 1: LLM Intent Classifier. I asked the model to classify the user's question into a category (e.g., "Is this asking for a ruling?"). LLMs are excellent at classification.

Layer 2: Regex Backstop. A deterministic keyword regex catches rule-start phrases (like "What is the ruling on..." or "How many...") to backstop the LLM.

When the gate fires, the user gets a scope message explaining that the text doesn't provide a ruling, and offers to show adjacent verses instead. It catches 8/10 out-of-scope questions with 0/4 false positives on legitimate questions.

Layer 3: The Evaluation Metric was Lying (The Reranker Bug)

For months, my lexical search bucket was stuck at ~19% recall. I blamed the reranker. I built an evaluation harness to test pre-rerank vs. post-rerank scores.

The measurement showed the opposite: The reranker was helping the Arabic path, but destroying the Roman-Urdu path by 11.7 points.

Why? Because I was asking the reranker to score (Roman-Urdu query, Arabic verse) pairs—a pairing it was never trained on. The model was scoring in a near-uniform noise band (1e-5).

The fix was to remove the reranker from the Roman-Urdu path entirely. Roman-Urdu R@5 jumped from 7.1% to 17.5% overnight.

The Final Lesson: Checkpoint Your Eval Pipeline

During this, I ran a 100-minute evaluation that crashed in the final report-generation stage because of a two-character format string error ({"n":>4d} instead of s). Because nothing was written to disk, I lost the entire 100 minutes of compute.

Always checkpoint your eval results to disk (pickle, JSON, whatever) immediately after the compute finishes, before you generate reports.

# Always checkpoint expensive compute
with open("rerank_checkpoint.pkl", "wb") as f:
    pickle.dump(rank_dicts, f)

Conclusion: How to build a reliable agent in 2026

A System Prompt is a Leash, not a Lobotomy. It establishes intent, but it cannot enforce truth.

Deterministic Backstops are Mandatory. If a tool returns an empty array or an error, override the LLM's prose entirely.

Use a Whitelist, not a Correctness Checker. Verifying if a citation is accurate is hard. Checking if it is in a pre-approved list is easy.

Don't use LLMs for Entity-Matching. If you can do the math or the sorting in Python, do it. Hand the model the answer explicitly.

Test the Test. I once had a weak assertion that passed a hallucinated quote as green because my regex only checked for the word "Hadith" nearby. A weak test is as dangerous as no test.

Building agentic AI isn't about finding the smartest model. It's about building the most resilient architecture around an inherently probabilistic system.

If you want to read the full 7-part engineering logs detailing these bugs, you can find them on my portfolio here.

By Muhammad Ahmad | AI Agent Engineer | PMP®, RMP®

Finding the Bugs I Built an AI Agent to Prevent: Building Deterministic Guardrails with LangGraph & Python

Muhammad Ahmad Nazir Ahmad — Sat, 17 Oct 2026 18:28:21 +0000

``
A real debugging log: fabrication bugs, deployment gaps, and false test passes found and closed across 6 rounds of adversarial testing.

System prompts are a leash, not a lobotomy. A system prompt can be ignored, misread, or drift under pressure. Deterministic code cannot.

When I built Ahmad-AI — a local AI agent using LangGraph, Python, and Ollama with access to live SQL databases and Islamic knowledge bases — I established one non-negotiable design rule:

It must never state a specific citation or data claim unless that claim is independently verified. Verified, or it says nothing.

Because general-purpose LLMs fabricate citations confidently and often, "confidently wrong" is worse than "honestly uncertain." I spent weeks trying to prove the system would break its own rule anyway.

It did. Repeatedly. Here is what I learned across six rounds of adversarial testing against a live agent.

Round 1: The First Adversarial Pass

I ran a set of test questions against the live agent — empty database queries, Quran & Hadith citation requests, and ambiguous sorting queries.

Two critical bugs immediately surfaced:

The Ghost Row Bug:

When querying a table with no matching rows, the agent correctly output: "The query did not return any results." In the exact same sentence, it invented two fake item IDs with fake stock levels.

The Regex Mismatch:

When asked about usury (riba), the agent cited verse 5:51 (a real Quran verse, but entirely about interfaith alliances). My citation guard failed to fire because its regex expected Surah [name] X:Y, while the model wrote (Quran X:Y). A safety mechanism that doesn't match the shape of the output is functionally non-existent.

The Fix Philosophy: Code, Not More Instructions

My first instinct was to strengthen the system prompt. I resisted. Instead of writing tighter rules for the LLM, I built a deterministic backstop in Python that overrides the model’s prose when known failure patterns appear in state:

`python
python:
def apply_tool_result_guards(state, response):
"""
Deterministic backstop: if the last DB query came back empty or hard-errored,
override entirely with a fixed, honest message regardless of what the model wrote.
"""
if is_no_data(last_tool_message):
return "No matching data was found for that query. The data may not exist yet."

ALLOWED_CITATIONS = {"20:114"} # Pre-verified whitelist

def enforce_citation_guard(text):
for match in CITATION_PATTERN.finditer(text):
if not any(allowed in match.group(0) for allowed in ALLOWED_CITATIONS):
return SAFE_ISLAMIC_FALLBACK # Blocked even if the citation is real
return text

Building an Evaluation Harness That Doesn't Lie

Manually retyping test prompts in a terminal doesn't scale. To keep the agent honest, I split automated testing into two distinct layers:

Unit Tests (22 tests): Direct execution of Python guard functions against synthetic inputs (fast, no live LLM required).

End-to-End Evaluation Harness: Drives the live LangGraph execution against Ollama and Postgres.

Treating fuzzy model outputs as binary pass/fail either hides real regressions behind noisy false failures or downgrades hard security checks to "close enough."

Key Takeaways for AI Engineers

A whitelist beats a correctness checker: Verifying "is this citation accurate?" is hard. Checking "is this citation on the pre-approved whitelist?" is simple and reliable.

Print a Build Stamp at Startup: A version stamp printed at agent startup costs nothing and prevents testing against stale code.

Distinguish Transient from Permanent Errors: Retrying a connection drop makes sense. Retrying a bad SQL column name is just a slower way to fail.

Don't ask LLMs to do arithmetic or row matching: Stating query answers explicitly in tool outputs prevents models from misreading JSON rows.

Read the Full Engineering Log
This architecture is live, tested against a 22-unit-test suite and a 5/5 E2E hard check harness.

For the complete technical breakdown — including deployment gap fixes, bounded execution limits (MAX_TOOL_ROUNDS), and verbatim adversarial outputs:

📖 Read the full interactive log on my portfolio:

👉 Engineering Log 01: Guardrails & Hallucination Prevention

Muhammad Ahmad is an AI Agent Engineer and Digital Supply Chain Architect (PMP®, RMP®) with 15+ years of operational leadership experience.