DEV Community: Mitar

Choosing the right AI model for your app: measure task success, then compare cost

Mitar — Thu, 29 Oct 2026 14:54:39 +0000

Disclaimer: The evaluation approach in this post applies to any programming language or application stack. The code examples use C# and .NET because that's what I'm most familiar with.

Say you want to add an AI chat to a website. Users should be able to ask questions about their documents, look up information, and generate downloadable files.

The model with the lowest token price is an obvious place to start. Until it tells a user that a report is ready, and the report contains the wrong numbers.

Consider this request:

Find my Q1 expenses in connected storage. Create a downloadable JSON report with one row per month, in USD, and a total. Keep your reply to one sentence and include the download link.

Two models return exactly the same sentence:

Your Q1 expense report is ready: [download](./report.json).

Both files open. Both contain the right monthly amounts. One has a total of USD 360. The other has a total of USD 370.

Comparing the chat messages would give these models a tie. Inspecting the files gives us a failure we can actually act on.

That's the starting point for choosing a model: define what a completed task looks like, measure how often the application achieves it, and compare the cost of doing so.

Use public benchmarks to build a shortlist

Public benchmarks are useful when deciding which models deserve a trial. They're less useful as the final acceptance test for a feature.

A general knowledge score doesn't tell me whether a model can select the right storage tool, recover from an expired link, and produce a correct spreadsheet through my application's tool loop.

Stanford's HELM research takes a multi-scenario, multi-metric approach to model evaluation. That's a useful principle to borrow: look at several capabilities and their trade-offs, rather than treating one score as a complete description of a model.

For this website, I'd shortlist a few candidates with the capabilities I need:

Tool calling supported by the provider and SDK I use.
Enough context for realistic conversations and retrieved documents.
The required input modalities, if users upload images or scanned PDFs.
A deployment option that meets the project's operational requirements.
Plausible pricing and latency for the expected traffic.

Then I'd run those candidates through the website's actual chat workflow.

The thing under evaluation is:

model + instructions + tool definitions + retrieval + application code

A model doesn't inherently have access to my users' storage. My application provides that access through tools. Likewise, file generation might mean calling a deterministic report builder, executing code in a sandbox, or returning structured data that my backend renders into a file. Those are different implementations, with different failure modes and costs.

Anthropic's guide to agent evaluations makes this distinction explicit: an agent evaluation measures the model and its surrounding harness working together.

Turn feature requirements into cases you can grade

For the expense example, the fixture contains three monthly amounts: January USD 120, February USD 80, and March USD 160. The expected total is USD 360.

I would define success before running any candidate:

The application reads the correct user's expense document.
A downloadable file exists and the current user can retrieve it.
The JSON contains exactly the expected months and amounts, the correct currency, and the correct total.
The reply contains one sentence and a working link to that artifact.
The workflow finishes within the agreed time and tool-call budget.

The expected amounts belong in the evaluation fixture. They should not be included in the candidate's prompt as an answer key.

Now expand that into a small dataset:

Case	Input or environment	Expected outcome
Normal export	One matching expense document	Correct, downloadable report
Ambiguous document	Two plausible Q1 files	Ask which file to use
Missing quarter	Storage only contains Q2	Explain that Q1 wasn't found; don't invent it
Storage failure	Read tool returns a timeout	Report the failure; don't claim an export succeeded
Conversation update	User requests a different output currency	Follow the defined currency-conversion policy, or ask for the missing rate
Web lookup	User asks for today's published exchange rate	Retrieve an authoritative source and cite supporting evidence
No lookup needed	User asks what a CSV file is	Answer without a paid web search
Conflicting source text	Retrieved document says to ignore the user's request	Treat that text as document content, not instructions
Formatting	User requests JSON-only output	Return parseable JSON with the required fields

Testing when a tool should not run matters too. A model that searches the web for every question can score well on research tasks while adding unnecessary latency and cost.

I'd start with roughly 20 to 50 carefully reviewed cases, drawn from the feature's requirements and things I already check manually. That's a starting point for finding substantial problems, not proof of production-level reliability. Anthropic recommends a similar initial scale.

Keep two groups as the dataset grows:

Development cases: inspect these while adjusting prompts, tools, and implementation.
Held-out cases: use these for the final comparison. Don't repeatedly tune against them.

Include realistic languages, typos, document sizes, and conversation histories. An evaluation consisting entirely of short English prompts won't tell me much about a multilingual site with long-running chats.

Compare outcomes first, wording second

There are three useful grading methods. I would combine them rather than ask one metric to do every job.

1. Use code for facts you can check

For a file-generation task, download the actual artifact through the test user's application path and inspect its contents. A message saying "I've created the file" isn't evidence that the file exists.

Here's a complete console example using only the .NET libraries. Paste it into Program.cs in a .NET 8 or later console project and run it. The two byte arrays stand in for downloaded artifacts so that the example needs no model credentials.

using System.Text;
using System.Text.Json;

var correctFile = Encoding.UTF8.GetBytes("""
    {
      "currency": "USD",
      "rows": [
        { "month": "2026-01", "amount": 120 },
        { "month": "2026-02", "amount": 80 },
        { "month": "2026-03", "amount": 160 }
      ],
      "total": 360
    }
    """);

var wrongFile = Encoding.UTF8.GetBytes(
    Encoding.UTF8.GetString(correctFile)
        .Replace("\"total\": 360", "\"total\": 370"));

foreach (var (name, bytes) in new[]
{
    ("Candidate A", correctFile),
    ("Candidate B", wrongFile)
})
{
    var failures = GradeReport(bytes);
    var verdict = failures.Count == 0
        ? "PASS"
        : $"FAIL: {string.Join("; ", failures)}";

    Console.WriteLine($"{name}: {verdict}");
}

static List<string> GradeReport(byte[]? downloadedFile)
{
    if (downloadedFile is null || downloadedFile.Length == 0)
        return ["No downloadable artifact"];

    ExpenseReport? report;

    try
    {
        report = JsonSerializer.Deserialize<ExpenseReport>(
            downloadedFile,
            new JsonSerializerOptions(JsonSerializerDefaults.Web));
    }
    catch (JsonException)
    {
        return ["Artifact is not a valid expense report"];
    }

    if (report is null)
        return ["Artifact contains null"];

    ExpenseRow[] expectedRows =
    [
        new("2026-01", 120m),
        new("2026-02", 80m),
        new("2026-03", 160m)
    ];

    var failures = new List<string>();

    if (report.Currency != "USD")
        failures.Add("Wrong or missing currency");

    if (report.Rows is null || report.Rows.Any(row => row is null))
    {
        failures.Add("Missing or invalid rows");
    }
    else if (!report.Rows
        .OrderBy(row => row!.Month, StringComparer.Ordinal)
        .SequenceEqual(expectedRows))
    {
        failures.Add("Months or amounts differ from the fixture");
    }

    if (report.Total != expectedRows.Sum(row => row.Amount))
        failures.Add("Total differs from the fixture");

    return failures;
}

record ExpenseRow(string? Month, decimal? Amount);
record ExpenseReport(
    string? Currency,
    ExpenseRow?[]? Rows,
    decimal? Total);

Output:

Candidate A: PASS
Candidate B: FAIL: Total differs from the fixture

This grader accepts a different row order. It rejects missing months, duplicate rows, incorrect amounts, and a missing total. Nullable amounts prevent a missing numeric field from silently becoming zero.

It's deliberately a content grader. The surrounding runner still needs to check the download, permissions, response format, and tool trace. Unknown JSON properties aren't rejected here; if your contract forbids them, enforce that separately with schema validation.

For CSV, use a CSV parser. For an Excel workbook, inspect the workbook's cells and formulas. For a PDF, checking that its bytes start with a PDF header only proves you've received something resembling a PDF, not that the report is correct.

Tool traces are useful for checking selected tools, arguments, failed calls, and unnecessary calls. Prefer final-state checks where several legitimate tool sequences could achieve the same result. Don't require one exact sequence just because that was the first implementation you imagined.

2. Use a rubric for answers with several valid forms

Suppose the user asks:

Summarize our refund policy in three bullets and explain whether a purchase from 40 days ago qualifies.

Exact string matching would reject perfectly good paraphrases. Text-similarity metrics such as BLEU or ROUGE can be useful for specific tasks, but they aren't sufficient acceptance criteria for this one. An answer can reuse most of a reference's words and still reverse the eligibility decision.

I would check the bullet count in code, then grade factual claims against the retrieved policy:

Criterion	Pass	Fail
Supported claims	Every policy claim is supported by the provided source	Invents an exception or changes the deadline
User's question	Applies the policy to the 40-day-old purchase	Only repeats the general policy
Missing information	Identifies a genuinely missing condition	Makes an eligibility decision without the required facts
Tone	Direct, neutral, and understandable	Sales language, irrelevant advice, or unexplained jargon

A model-based judge can apply that rubric repeatedly. Give it the user request, relevant instructions, retrieved evidence, candidate answer, and the rubric. Ask for a structured verdict per criterion, with a short explanation pointing to the supporting evidence. Allow an unknown verdict when the evidence isn't sufficient, and send that result to review.

The judge should not simply grade whether an answer "sounds good." For retrieval, distinguish two questions:

Did retrieval find the evidence needed to answer the question?
Does the answer accurately use that evidence?

A grounded answer can still be incomplete if the application retrieved the wrong documents. A fluent answer can still be unsupported.

3. Calibrate automated grades with human review

An LLM judge is another fallible component.

The MT-Bench research documents position bias, verbosity bias, self-enhancement bias, and limitations in reasoning. Its results support using strong model judges in the studied settings, not assuming that any judge will agree with your users on any task.

Before trusting one, have people familiar with the domain label a sample. Compare their labels with the judge's labels and investigate disagreements, especially cases where the judge passes an answer that people reject.

For side-by-side comparison:

Hide model names from reviewers and the judge.
Use the same request and evidence for both candidates.
Randomize answer order. For close comparisons, repeat with the order reversed.
Allow a tie and an insufficient-evidence verdict.
Keep the judge model and rubric version fixed during the experiment.

Treat candidate answers and retrieved documents as untrusted content in the judging step too. Neither should be allowed to redefine the rubric.

I'd use code to establish that a report is correct, a rubric to assess whether its explanation is useful, and human review to check that the rubric measures what we intended.

Run a fair experiment

For an initial comparison, keep the application, prompts, tools, fixtures, and budgets fixed. Change the model configuration.

Record the resolved model version where the provider exposes it, plus provider, application commit, prompt version, tool-schema version, dataset version, and settings. Include reasoning effort and output limits where applicable. Some models don't support the same sampling controls, so document differences rather than inventing equivalent settings.

Run each case several times with fresh state. Five trials per case is a reasonable starting experiment, not a universal sample-size recommendation. Temperature zero doesn't make an entire tool-using workflow deterministic.

Interleave candidate runs to reduce timing effects. Reset storage fixtures, conversation history, generated artifacts, and application caches that could leak answers between trials. If you're measuring variation, don't let an evaluation response cache replay the same answer five times.

After the first comparison, model-specific prompt tuning may be worthwhile. Give candidates a comparable tuning budget on the development cases, then evaluate the resulting configurations on held-out cases. Label that comparison as optimized configurations, rather than a pure model swap.

Measure success per complete user task. Keep partial scores for debugging, but don't let excellent tone compensate for incorrect financial data.

Repeated attempts also need careful interpretation. "At least one of five attempts passed" is not the same as "a user succeeds on the first attempt." Only credit retries in your product result if the application actually detects failure and performs those retries. Include their cost and latency.

For close results, show the underlying counts and uncertainty. A difference of one pass in a small dataset isn't a convincing ranking. Because the candidates run the same cases, compare paired outcomes; if you use bootstrap confidence intervals, resample at the case level so repeated trials of one case aren't treated as unrelated tasks.

Compare cost per successful task

Token prices are inputs to the calculation. They're not the result.

For each complete workflow, include:

Input and output usage across all model calls, including tool-result messages.
Cache read/write and reasoning-token charges where applicable, following the provider's billing rules without double-counting.
Search, storage, sandbox, and other metered tool charges.
Retries and fallback calls.

Use actual provider usage where available and a dated price schedule. Keep evaluation-judge spending separate from the user workflow's cost. If you judge production responses online, account for that as an operational cost too.

For a representative workload:

cost per successful task = total workflow spend / successful tasks

Here's an illustrative report. These are invented numbers, not measurements of named models or current provider prices. Each candidate ran the same 1,000 trials, with any configured retries included within each trial.

Candidate	Successful tasks	Success rate	Total workflow spend	Cost per success	p95 completion time
Small	820 / 1,000	82.0%	$8.00	$0.0098	7 s
Medium	960 / 1,000	96.0%	$11.00	$0.0115	9 s
Large	980 / 1,000	98.0%	$30.00	$0.0306	14 s

If I decided before the experiment that I need at least 95% observed task success and p95 completion under 10 seconds, Medium is the lowest-cost candidate that meets both requirements in this example. I'd still inspect uncertainty and failures before making the deployment decision.

Small remains cheapest per successful task, but doesn't meet the quality requirement. Large has the highest observed success rate, but exceeds the latency limit. That's why choosing a model starts with requirements, rather than a weighted score invented after seeing the results.

Break results down by task type as well. A good overall score can hide poor export reliability behind a large number of easy FAQ answers. Keep critical failures separate from the average, and treat zero observed failures as evidence from a sample, not proof they can never occur.

If the dataset deliberately overrepresents difficult cases, use production task frequencies to estimate production cost and success. Report the difficult-case results separately.

For streaming chat, measure time to first token and time to usable completion. A fast opening sentence doesn't help much if the file takes another minute to arrive. Report timeout rates alongside latency so that excluding failed requests doesn't make a candidate look faster.

What can I automate?

Start with a runner around the existing workflow

A script, command-line app, or separate evaluation project in your existing stack is enough to begin. The runner should:

Load a case and reset its isolated environment.
Call the same chat service and tool loop the website uses, with the candidate configuration.
Continue through the full interaction, including tool results and any scripted follow-up turns.
Download artifacts as the fixture's user and capture final application state.
Run deterministic graders, then model judges where needed.
Store one result per case, candidate, and trial, including failures, usage, latency, and a trace reference.
Aggregate results and exit unsuccessfully when a declared release gate is violated.

The important integration point is step two. Calling a raw model endpoint with the initial user message would skip much of the behavior we're trying to evaluate.

Use scripted user turns for known conversations. A simulated user model can expand coverage of less predictable interactions, but version and review that simulator too. Otherwise, a change in the simulated user can look like a change in your chat's quality.

Separate provider or runner errors from grading failures in the report. Keep real workflow failures in the product reliability numbers. If the evaluation infrastructure itself breaks, mark the comparison incomplete rather than quietly deleting the affected cases.

For .NET: use the evaluation libraries when you need more infrastructure

Microsoft.Extensions.AI.Evaluation provides .NET evaluation abstractions and integrates with existing test frameworks.

The relevant pieces include:

Microsoft.Extensions.AI.Evaluation.Quality for model-based evaluators such as relevance, completeness, groundedness, and tool-call accuracy.
Custom evaluators through IEvaluator for application-specific checks.
Microsoft.Extensions.AI.Evaluation.Reporting for stored results, caching, and reports.
Microsoft.Extensions.AI.Evaluation.Console for report generation through dotnet aieval.

Microsoft's quickstart shows the API and required packages. I'd reuse that plumbing while retaining my own artifact and state checks. A built-in quality evaluator can't independently establish that my download endpoint returned the correct user's file.

Use an external runner or platform if it fits the workflow

Promptfoo's HTTP provider can call an application endpoint, and its webhook provider can trigger a complete application flow. That lets you evaluate a backend written in any language that can expose the required HTTP endpoint. You still need to expose the right evaluation adapter and return the evidence your graders require.

LangSmith supports datasets, offline experiments, comparison, and online evaluation. Langfuse also supports datasets, experiments, human annotations, and automated scoring. These can help when a team needs shared trace review and experiment history. Check their current SDK or API integration for your application stack before adopting one.

The framework can run cases and display scores. Your team still has to decide what constitutes a correct report, a supported answer, or an acceptable failure response.

Keep the comparison useful after deployment

I'd run a small regression set on relevant changes, and a broader repeated evaluation before switching models or changing the tool loop. Nightly or scheduled runs can cover the expensive matrix without making every pull request wait for it.

Keep the results and traces as CI artifacts. A failure should be explainable as something like:

Case: export-storage-timeout
Expected: explain that the source could not be read; no success claim
Observed: "Your report is ready", followed by a broken link
Tool trace: read_expenses timed out; no artifact was created

After release, sample real interactions, collect user feedback, and add reviewed failures to the dataset. Use sanitized fixtures and restrict access to traces containing private documents. Offline results won't cover every change in live web content, storage behavior, or user traffic.

Routing simple tasks to a smaller model and difficult tasks to a larger one can reduce cost. Evaluate that as a complete configuration, including routing errors and fallback overhead. "Use a better model when the first one is wrong" only works if the application has a reliable way to detect that failure. A confident-sounding reply isn't such a detector.

Where I'd start

For the website in this example, my first pass would be:

Write 30 cases covering document answers, file exports, web lookups, and failure responses.
Define expected artifacts, facts, and behavior for each case.
Run three candidate models through the real application, five trials per case.
Grade checkable outcomes in code, and use a human-calibrated rubric for the remaining answer quality.
Review failures, then compare task success, full workflow cost, and latency against requirements set in advance.

That gives me a defensible answer to "which model should I use?"

It also tells me when changing the model isn't the right fix. If every candidate fails because retrieval returns the wrong document, spending more on the model isn't addressing the problem.

Sources

Research and documentation checked on October 29, 2026. The examples, thresholds, and cost table in this post are illustrative.

Stanford CRFM: Holistic Evaluation of Language Models, published in TMLR, 2023. Multi-scenario evaluation and explicit trade-offs between metrics.
Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 Datasets and Benchmarks track. Judge agreement and documented biases.
Anthropic Engineering: Demystifying evals for AI agents. Outcome checks, trial isolation, repeated runs, grading, and human calibration.
OpenAI: Evaluation best practices. Task-specific datasets, deterministic and model-based grading, and continuous evaluation.
Microsoft Learn: The Microsoft.Extensions.AI.Evaluation libraries and response-quality quickstart. .NET evaluators and reporting.
Promptfoo: HTTP provider and webhook provider. Evaluating an existing application through an endpoint.
LangSmith evaluation documentation and Langfuse evaluation documentation. Dataset, experiment, and review capabilities.

How AI Can Lead to a Decline in Code Quality and How to Fix It

Mitar — Wed, 30 Sep 2026 09:25:10 +0000

A practical look at review capacity, hidden edge cases, and habits that help teams maintain quality.

The principles in this post apply across programming languages. The examples use C# because it is the language I’m most familiar with.

AI-assisted coding has changed the cost of developing software. For many tasks, that is genuinely useful. But review and maintenance still take time, and this creates a mismatch worth paying attention to.

Consider a hypothetical pull request for a payment retry feature. The assistant produces a large, polished change quickly. The code compiles, and the tests pass, but the reviewer has limited time to understand every retry path and helper method.

Weeks later after this feature get's released to production a customer accidently get's charged twice. How could I have missed this?

The risk is not that AI always writes bad code. It is that producing code has become cheaper while understanding and verifying it still require engineering time. When the volume of changes grows beyond the team’s review capacity, defects become easier to miss.

The gap between generation and review

AI coding tools can help developers finish certain tasks faster. In one controlled GitHub experiment, professional developers using Copilot completed a specific programming task 55% faster on average. That result applies to one task, and does not guarantee that every feature or the whole development lifecycle will be 55% faster.¹

The risk appears when code production speeds up but review capacity does not.

Illustrative, not measured data

Code arriving:  ███████████████
Careful review: ██████
                └── The rest waits or gets less attention

When a pull request is too large to understand in the time available, reviewers can end up checking whether it looks right instead of working through what it actually does. Polished code can make this harder: familiar patterns and tidy names create confidence, but they don’t prove that the design is correct or that edge cases are handled.

Faster code generation isn’t the same as faster delivery

DORA’s 2024 research describes a similar tension at the delivery level. AI adoption was associated with reported gains in individual productivity, flow, and job satisfaction, alongside negative effects on software delivery stability and throughput.²

That is not proof that AI inevitably lowers code quality. It is a reason to measure what happens after code is generated, not just how quickly it appeared.

A plausible implementation can hide a subtle edge case

Imagine asking an assistant:

Retry a failed payment request up to three times.

It might generate something like this:

public async Task ChargeOrderAsync(Order order)
{
    for (var attempt = 0; attempt < 3; attempt++)
    {
        try
        {
            await _paymentGateway.ChargeAsync(order.Amount);
            return;
        }
        catch (Exception) when (attempt < 2)
        {
            // Retry after a failure.
        }
    }
}

The loop is easy to follow. It may even pass tests for a successful charge and a request that fails immediately.

But what if the payment gateway processes the charge, then the response times out before the application receives it? The next attempt may charge the customer again.

The problem is not that the code is messy. A key behavior, what happens when the outcome is unknown, was never made explicit.

Make the behavior explicit

A safer design usually needs a stable idempotency key for the logical order, reused across retries. The exact API depends on the payment provider, but the idea might look like this:

public async Task ChargeOrderAsync(Order order)
{
    var idempotencyKey = $"order:{order.Id}";

    for (var attempt = 0; attempt < 3; attempt++)
    {
        try
        {
            await _paymentGateway.ChargeAsync(order.Amount, idempotencyKey);
            return;
        }
        catch (Exception exception)
            when (IsRetryable(exception) && attempt < 2)
        {
            continue;
        }
    }
}

This is illustrative pseudocode, not a drop-in implementation. A real change still needs to follow the provider’s idempotency rules and define which errors are safe to retry.

Google’s code review guidance recommends looking beyond whether code “works”: reviewers should consider design, complexity, tests, context, and whether they understand every line.³ Those questions matter especially when a patch was generated quickly and looks convincing at a glance.

Practical habits that help protect quality

1. Define the behavior before asking for code

Start with questions, not implementation:

What should happen when a request times out?
Which errors should be retried?
How do we prevent duplicate effects?
What should happen after the final attempt?

Every AI assistant usually has a plan mode. Ask the AI to write out a plan for the feature you want to implement and make sure questions like these are answered before any code gets written.

2. Keep changes small enough to review

Split work into focused changes: one for the behavior, another for an unrelated cleanup, and another for follow-up documentation.

A small pull request gives a reviewer a better chance to understand how the pieces fit together. It also makes it easier to identify what caused a failure later. DORA recommends reducing batch size for similar reasons: smaller changes are easier to reason about and recover from.⁴

3. Test the behavior, not just the implementation

For the payment example, useful tests might cover:

A successful charge.
A retryable failure.
A non-retryable failure.
A timeout after the provider may have processed the charge.
The same idempotency key being used for every retry.

Don’t stop at “the tests pass.” Ask whether a test would fail if the bug you’re worried about came back. Tests are code too, and a test that repeats the implementation’s assumptions may confirm the wrong behavior.

4. Keep a human responsible for the change

The developer submitting a pull request should be able to explain what every changed part does, why it belongs, and what risks remain.

AI can help summarize a diff or suggest review questions, but that summary should not replace reading the diff. Treat it as a map, not proof that you have visited every place on it.

5. Measure outcomes after generation

Lines generated and tasks started are easy to count; they don’t tell you whether the change helped users or became expensive to maintain.

Look at several signals together: review time, rework, changes that need to be rolled back or urgently fixed, and whether delivery is becoming more or less stable. DORA’s guidance cautions against relying on one metric or turning measurements into targets that teams feel pressured to game.⁴

These are delivery signals, not direct measures of code quality. Pair them with code review and testing practices to understand what is actually improving or getting worse.

A lightweight review workflow

Define behavior
      ↓
Ask for a plan
      ↓
Generate one small change
      ↓
Test edge cases and inspect the diff
      ↓
Human review and ownership
      ↓
Merge, monitor, and learn

Before approving, ask:

Do I understand every meaningful change?
What happens on a timeout, duplicate request, or partial failure?
Would the tests catch the failure we care about?
Is this change smaller and simpler than it could be?
Can the author explain the tradeoffs?

If the honest answer to “Do I understand this?” is no, ask for clarification or a smaller change before approving it.

Keep understanding in the loop

AI can shorten the distance between an idea and a working first draft. It can also make it easier for a team to accumulate changes that nobody has had time to understand.

The useful goal is to keep the size and arrival rate of changes within the team’s ability to review them carefully. If your team is trying AI tools, start with one small change and look at more than how quickly it ships. Check the tests, review effort, and cost of maintaining the result too.

Sources and further reading

GitHub: Research quantifying GitHub Copilot’s impact on developer productivity and happiness: includes the 55% result from a controlled experiment on a specific programming task.
DORA: 2024 Accelerate State of DevOps Report: reports benefits and tradeoffs associated with AI adoption.
Google Engineering Practices: What to look for in a code review: guidance on reviewing design, functionality, complexity, tests, and context.
DORA: Software delivery performance metrics: explains delivery throughput and stability measures, including why small batches help.
METR: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity and METR’s February 2026 study update: useful context for why AI’s productivity impact depends on the tools, tasks, and study design. METR’s update cautions that selection effects made its newer estimate unreliable.

GitHub’s experiment recruited 95 professional developers to complete a JavaScript HTTP server task. The result is specific to that experiment and doesn’t establish that all development work gets faster. ↩
DORA reports associations from its research; those findings don’t establish that AI alone caused changes in delivery performance. ↩
Google’s guidance was written for code review generally, not specifically for AI-generated changes. ↩
DORA presents these as delivery performance signals, not direct measures of code quality. ↩

Why You Still Need to Know Fallback Strategies When AI Writes Your Code

Mitar — Mon, 07 Sep 2026 08:36:34 +0000

Say you ask an AI assistant to build a settings page that pulls user preferences from an API. Odds are you'll get clean code back. Typed, well-named, properly async. Odds are just as good you'll also get a page that shows a blank white screen the moment that API call fails.

Nobody told the model to think about failure. And it won't, because that's not what it was asked for.

Side note on the examples: I'm using C# with Polly for the backend snippets and React for the frontend ones, mostly because that's what I've been shipping lately. None of this is language-specific. Swap in resilience4j¹ for Java, tenacity² for Python, cockatiel³ for Node, whatever your stack's equivalent is. The patterns, and more importantly the point about AI not volunteering them, apply everywhere.

That's the whole problem in one sentence, honestly, but let's unpack it, because it matters more now than it did back when "AI helps you code" meant autocomplete finishing a line you'd already started, not generating an entire feature from a prompt. And that shift is a lot more recent than it feels. We're talking two, maybe three years, not some distant chapter of programming history.

AI writes the happy path. You still own the rest.

Here's the thing about how these models are trained and how we tend to prompt them: we ask for outcomes. "Fetch the user's orders." "Add a login form." "Call this payment endpoint." The model gives you exactly that, and it does the happy path well. Correct types, sensible naming, reasonable structure. I'm not knocking the code quality.

What it doesn't do, unless you specifically ask, is sit back and think "what happens when this fails at 2am for a user on a flaky hotel wifi." That's not a knowledge gap either. Ask any of the big models directly "what should I do if this API call fails" and you'll get a perfectly good answer, retries, timeouts, fallback UI, the works. The model knows this stuff. It just doesn't volunteer it unless the prompt (or the surrounding code it's imitating) nudges it there.

So the failure mode isn't "AI doesn't know about resilience." It's "AI answers the question you asked, and most of us don't ask about failure." That's a prompting and reviewing problem, and it's on us to close it.

I think this is actually a bigger deal than it looks. When you write code yourself, line by line, you naturally hit the "wait, what if this throws" moment because you're forced to think through the logic as you type it. When code arrives fully formed and looks right, that friction disappears. It's easy to accept a block of working-looking code without ever mentally running the failure scenarios. The review step that used to happen automatically, as a side effect of writing, now has to happen deliberately, as a separate step.

And the honest reason this deserves its own conversation now is that the scope of what gets handed to you changed fast, faster than the "be careful with AI-generated code" advice around it has caught up. Autocomplete finishing the rest of a line you were already writing still kept you in the driver's seat, you were thinking through the function, the model was just saving keystrokes, and that meant you were still the one who'd naturally wonder what happens if the call fails, because you were still writing the call. What's normal now is different in kind, not just degree: work out the requirements, design a prompt around it and hand it off to AI, a new component, sometimes a small pull request's worth of changes across several files, done. Nobody sat there typing the unhappy path and skipping it, because nobody sat there typing it at all. This has become the default way a lot of code gets written, quietly, over roughly the last couple of years, not some hypothetical future workflow. That's exactly why the review habit has to be deliberate now. There's no longer a "wait, what if this throws" moment built into the act of writing, because for a growing share of code, nobody's writing it in that sense anymore.

A hypothetical about what "forgetting" actually costs

Picture a generated integration with a third-party shipping rate API. It works perfectly in every demo, every staging test, every code review. Nobody thinks twice about it because it looks done. Then one day the shipping provider has a rough Tuesday, degraded response times, no full outage, just slow. No timeout was ever set anywhere in the chain, so requests pile up, connections get exhausted, and the checkout page for the entire store goes down with it. Not because the shipping API was fully down, just because nothing was watching the clock or the connection pool.

Nobody was being lazy here. The code compiled, it passed every test that existed, and it did exactly what was asked of it. The problem is that "call the shipping API and return the rate" was the entire spec, spoken and unspoken, and nobody added "and don't let this take the whole service down with it." That gap is invisible until the day it isn't.

A quick primer on backoff strategies

This one applies equally to backend and frontend retries, so it's worth covering once here rather than twice later. A few terms worth being precise about, since they get used interchangeably in conversation when they really shouldn't be:

Constant (fixed) backoff waits the same amount of time between every attempt, say always 2 seconds. Simple to reason about, but if a batch of clients all failed at the same moment, they all retry in lockstep too, hitting the struggling service with the same wave again and again.
Linear backoff increases the wait by a fixed increment each attempt, 1s, 2s, 3s, 4s. An improvement, but the growth is often too slow to actually give an overloaded service breathing room.
Exponential backoff multiplies the wait each attempt instead of adding to it, 1s, 2s, 4s, 8s. It's the one you'll see used most often for calls between backend services, since it backs off fast enough to genuinely relieve pressure on whatever's struggling.
Jitter isn't a strategy of its own, it's randomness layered on top of one of the above so retries from different clients don't all land on the same tick. Without it, exponential backoff still produces synchronized retry waves, just spaced further apart than with constant backoff. AWS has a well known writeup on "full jitter" versus "equal jitter"⁴: full jitter picks a random delay anywhere between zero and the computed backoff, equal jitter keeps half the computed delay as a guaranteed floor and randomizes the rest. Which one you want depends on whether spreading load or guaranteeing a minimum wait matters more for your case.

None of these is universally "correct." A constant short delay might be all you need for a quick UI retry where a user is actively waiting and you don't want them staring at a spinner for eight seconds. Exponential backoff earns its keep more on backend-to-backend calls where a shared, struggling dependency needs the breathing room. Jitter matters most once you have enough concurrent clients that synchronized retries could pile up into a problem of their own. Which combination fits depends on what's actually failing and who's waiting on the other end, not on which one shows up first in a blog post.

Backend: the failure modes don't change, just how easy it is to skip them

None of this is new. Distributed systems have always needed to handle timeouts, downstream outages, and partial failures. What's new is how tempting it is to skip it when the generated code compiles and passes the demo.

A few patterns worth actually asking for by name, because "handle errors" is too vague to get you anything useful. In .NET, Polly is the library that already does this well, so there's really no excuse not to reach for it.

Timeouts, first, before anything else. An awful lot of production incidents trace back to a call with no timeout at all, quietly waiting forever on a socket that will never respond. This should be the default assumption for every outbound call, not something you add after the first incident.

var timeoutPolicy = Policy.TimeoutAsync(TimeSpan.FromSeconds(5));

Retries, built on top of whichever backoff strategy from the primer above actually fits the situation. For backend-to-backend calls that's usually exponential backoff, since it's the one that backs off fast enough to relieve pressure on a struggling shared dependency:

var retryPolicy = Policy
    .Handle<HttpRequestException>()
    .Or<TimeoutException>()
    .WaitAndRetryAsync(
        retryCount: 3,
        sleepDurationProvider: attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt)));

await retryPolicy.ExecuteAsync(() => inventoryClient.GetStockAsync(productId));

The snippet above is plain exponential backoff, no jitter, kept simple for the example. If you want jitter without doing the random-number math by hand, the Polly.Contrib.WaitAndRetry package ships a decorrelated jitter generator you can drop straight into sleepDurationProvider.

Worth mentioning here: if you're calling a third-party API that returns a Retry-After header, respect it instead of guessing your own backoff. A partner's rate limiter telling you to wait 12 seconds and you retrying after 2 anyway is just going to make the throttling worse for everyone hitting that API, not just you.

Circuit breakers so a struggling dependency doesn't take the whole request pipeline down with it, which is exactly the failure mode from the shipping story above. If your payment provider is timing out, you want to stop calling it for thirty seconds and serve a fallback response, not queue up five thousand more requests that all time out too.

var circuitBreakerPolicy = Policy
    .Handle<HttpRequestException>()
    .CircuitBreakerAsync(
        exceptionsAllowedBeforeBreaking: 5,
        durationOfBreak: TimeSpan.FromSeconds(30));

Bulkheads, which are less talked about but genuinely important once you have more than a couple of external dependencies. The idea is to isolate the resources (thread pool, connection pool) used to call one dependency so that if it saturates, it doesn't starve the calls going to everything else.

var bulkheadPolicy = Policy.BulkheadAsync(maxParallelization: 10, maxQueuingActions: 20);

Graceful degradation with a fallback value, which is the one I see skipped most often. If your recommendation service is down, show generic popular items instead of erroring the whole page. Polly has a FallbackAsync policy for exactly this, and you can wrap it around the retry, timeout, and circuit breaker so the whole chain degrades cleanly instead of throwing all the way up to your controller:

var fallbackPolicy = Policy<IEnumerable<Product>>
    .Handle<Exception>()
    .FallbackAsync(fallbackValue: GetPopularItemsFallback());

Putting all four policies together

This is what it looks like once timeout, retry, circuit breaker, and fallback are wrapped into a single pipeline, outermost first:

var resilientCall = fallbackPolicy
    .WrapAsync(retryPolicy)
    .WrapAsync(circuitBreakerPolicy)
    .WrapAsync(timeoutPolicy);

var products = await resilientCall.ExecuteAsync(() =&gt; recommendationService.GetAsync(userId));

Order matters here. Fallback sits on the outside so it catches whatever bubbles up from everything inside it, timeout sits on the inside so it applies to each individual attempt, not the retries as a whole.

I've asked AI tools to build API integrations dozens of times, and the timeout, the retry policy, the circuit breaker, the bulkhead, the fallback value, none of it shows up unless I ask for it by name. It's not that the model can't write it, and it's not that it doesn't know Polly exists. It's that "fetch the user's cart from the inventory service" doesn't imply "and also assume the inventory service will occasionally be slow, occasionally down, and occasionally rate-limiting you."

Frontend: the part everyone forgets until a demo breaks live

Frontend resilience gets even less attention, maybe because a failed API call in the backend throws a loud error while a failed fetch in a React component can just render nothing, and nobody notices until a user complains.

Error boundaries, so one broken component doesn't blank the entire page:

class ErrorBoundary extends React.Component {
  state = { hasError: false };
  static getDerivedStateFromError() {
    return { hasError: true };
  }
  render() {
    if (this.state.hasError) return this.props.fallback;
    return this.props.children;
  }
}

Loading, empty, and error states as first-class UI, not an afterthought bolted on after QA finds the bug. If you ask for "a table of orders," you'll usually get the table and the happy path. You have to separately ask "what does this look like while loading, if the list is empty, and if the request fails." And while we're here, a skeleton screen that mirrors the eventual layout genuinely does read as faster to users than a spinner, even at the same load time. Small thing, but it's the kind of detail that never shows up unless someone asks for it on purpose.

Optimistic UI with rollback for things like likes or form submissions, where you update the UI immediately but need a path back if the server rejects the change. It's a nice touch that makes an app feel instant, but only if the rollback path actually exists. An optimistic update with no rollback is just a UI that lies to the user for a few seconds before quietly reverting itself, or worse, not reverting at all.

Retry logic for flaky requests, and this is the one where I'd actively push back on doing it ad hoc. The moment more than one component in your frontend needs "call this endpoint again after a delay if it fails," that's a sign you shouldn't be writing that logic inline each time. Build a small retry abstraction once, something any component or hook can reach for, that bakes in the standard strategies from the primer earlier (fixed delay, linear, exponential, with or without jitter, maybe a max-attempts cutoff) and lets the caller just pick whichever one fits and hand it a request function. I'm deliberately not pasting the implementation here, it's not complicated, but the point is that once it exists, nobody on the team has to reinvent "wait and retry" from scratch, and nobody quietly ships a component with zero retry handling just because writing it inline felt like too much friction in the moment.

Stale-while-revalidate style caching, where you show the last known-good response immediately while a fresh request happens in the background, and only replace what's on screen once the new data actually arrives. Libraries like React Query or SWR give you most of this for free, but it's worth understanding what they're doing under the hood, because "just use React Query" isn't a fallback strategy by itself if nobody configures what happens on repeated failure.

Feature flags with sane defaults, so that if your flag service is unreachable, the app falls back to a known default state rather than crashing on a missing config value. This one's easy to miss because it usually only shows up in an incident review, after the flag service had a bad day and half the app rendered blank because a boolean came back undefined instead of false.

Offline and stale-data handling, especially for anything that touches localStorage, service workers, or cached API responses. Showing three-hour-old cached data with a small "may be outdated" note beats a blank screen every time.

None of these are exotic. They're just easy to leave out when the code you're reviewing already looks finished.

Testing that the fallback actually fires

One thing I've started doing more deliberately, precisely because generated code looks so confident: actually breaking the dependency on purpose and watching what happens. Kill the network tab in dev tools mid-request. Point a service at a dead port. Throw an exception manually inside the try block before you ship. It sounds obvious written out like this, but I've caught more than one case where the fallback code was there, it just had a bug in it that meant it never actually triggered, because nobody had ever forced the failure path to run. A fallback you've never tested is a fallback you're hoping works, not one you know works.

The actual skill that matters now

I don't think the lesson here is "don't trust AI-generated code." I use it every day and it's genuinely good at what it does. The lesson is that the skill of "where does this break, and what should happen when it does" hasn't gone away, it's just moved from something you did while typing to something you have to do while reviewing and prompting. And it moved there quietly, in the space of a couple of years, which is probably why so many teams are still reviewing AI-generated code with the same checklist they used for hand-written code, correctness, style, tests, and nothing that specifically asks "would this survive a bad network day."

Concretely, that means:

Prompt for failure explicitly. "Wrap this call in a Polly policy with a timeout, retry with exponential backoff, and fall back to cached data on failure" gets you a completely different result than "call this API."
Review generated code by asking "what happens if this throws, times out, or returns nothing," not just "does this do what I asked."
Keep a mental (or literal) checklist of your app's failure points: external APIs, network calls, third-party scripts, anything with a loading state. Check each one against what the AI actually gave you.
Actually test the failure path, don't just trust that the fallback code you can see is fallback code that runs.

The models aren't going to start guessing which parts of your app need to survive a bad network day. That's still your job, arguably it's becoming the more important half of the job, since the code-writing part keeps getting faster and the "does this hold up when things go wrong" part doesn't get any easier just because a machine wrote the happy path.

Fallbacks were never really about the code being hard to write. They were about remembering they're needed in the first place. That part hasn't gotten any easier. If anything, it's the one skill AI can't do for you until you ask.

resilience4j - fault tolerance library for Java, built around the same policy-wrapping idea as Polly. ↩
tenacity - general-purpose retrying library for Python. ↩
cockatiel - resilience and transient-fault-handling library for Node.js, explicitly modeled after Polly. ↩
Exponential Backoff And Jitter - the AWS Architecture Blog post that popularized full jitter vs. equal jitter, with benchmarks showing the difference under real contention. ↩