DEV Community: Ben Johnson

What Should You Look for When Interviewing a Fractional CTO Candidate?

Ben Johnson — Sun, 01 Nov 2026 14:06:00 +0000

When evaluating a fractional CTO candidate, look for someone who operates as an accountable engineering executive rather than a detached advisor. Prioritize candidates with demonstrated pattern recognition at your exact operating stage, discipline in capital allocation, and the technical humility to simplify architectures rather than overengineer them. The strongest leaders arrive with clear frameworks to assess engineering capacity, cut unnecessary platform costs, and align technical milestones with commercial revenue targets.

Hiring well means testing whether the candidate can drop into an active team dynamic, establish immediate operational cadence, and make unvarnished build versus buy calls. Avoid candidates who treat fractional engagements as passive consulting gigs or resume maintenance. The right partner sets measurable milestones, transfers strategic capability to your permanent team, and builds toward operational independence from day one.

The Practical Guide

Defining the Technical Mandate and Engagement Scope

Before scheduling an initial interview, define the precise operational problem you expect a fractional executive to resolve. These engagements fail most often when leadership hires for generic technical oversight rather than a concrete business outcome. Determine whether your organization needs an architecture audit, preparation for investor due diligence, team restructuring, or faster product delivery.

A vague mandate invites scope creep and burns capital quickly. Without defined expectations, the fractional leader will drift into low impact tasks or get caught between executive requests and frontline firefighting. Set a written thirty day, sixty day, and ninety day milestone roadmap before the search begins, and hold the incoming leader accountable for concrete deliverables such as architecture diagrams, onboarding documentation, automated deployment configurations, or hiring plans for full time staff.

Interrogating Architectural Judgment and Commercial Literacy

Probe how candidates balance technological ambition with business constraints. Ask them to describe an architectural decision they regret and how they resolved the fallout. Experienced operators speak candidly about technical debt, failed vendor evaluations, and the pain of maintaining overly complex systems. Candidates who present only flawless case studies are either withholding operational reality or lack experience under real pressure.

This matters because overengineering can quickly exhaust an early stage or mid market balance sheet. Leaders who chase trends often introduce unnecessary microservices, niche databases, or unproven frameworks that the permanent team cannot maintain once the executive leaves. Review how the candidate evaluates your existing technical debt and look for a plan that favors proven tools and isolates risk without demanding a full rewrite.

A qualified fractional CTO also connects engineering decisions to unit margins and cash runway. Ask how they allocate engineering time between revenue generating features, maintenance, and technical debt, and how they approach cloud hosting expenses and vendor consolidation. A serious failure mode is treating the engineering department as a research lab rather than a commercial engine: when code delivery decouples from revenue, the business risks running out of capital while building infrastructure for scale it has not yet reached. Track whether the candidate can calculate hosting cost per active user, forecast hiring costs accurately, and defend technical choices based on gross margin impact.

Assessing Operational Cadence and Fractional Availability

Scrutinize how the candidate structures their hours across concurrent obligations. Most fractional executives manage more than one client portfolio at a time, and the strongest performers treat each engagement as hands on building work rather than passive advising. Research from REF Global describes effective fractional operators as embedding directly into executive teams with real decision authority rather than advising from the sidelines.

Ask how many hours per week they will allocate and what their escalation protocol looks like. Executives who overcommit across too many clients introduce costly context switching during outages or sprint planning. Measure success through predictable participation in core ceremonies, clear written updates, and emergency communication within four hours during critical incidents.

Spotting Critical Red Flags in Candidate Responses

Watch for candidates who focus only on advisory strategy and avoid operational detail. Analysis from KORE1 describes conventional consultants as delivering presentations and leaving, while an authentic fractional CTO sits in meetings, makes hard calls, and takes direct accountability. If an applicant cannot discuss pull request hygiene, continuous integration pipelines, or team morale, they will struggle to earn the respect of your developers.

Other red flags include rigid stack dogmatism, blaming former colleagues for past failures, and avoiding junior engineering staff. A toxic hire can demoralize your team and trigger unforced resignations. Gather internal feedback after panel interviews and confirm the candidate communicates with humility and clarity across all levels.

Running a Paid Practical Diagnostic Before Long Term Commitment

Replace speculative technical quizzes with a paid, two week diagnostic before finalizing a retainer. Give the finalist sanitized architectural documentation, an overview of active bottlenecks, and repository history, then ask for an executive risk assessment and a prioritized thirty day action plan.

In one illustrative example, two finalists for a scaling fintech platform were asked to diagnose payment ledger delays. One proposed replacing the core database with a distributed graph database, a nine month migration. The other audited query plans, found connection pool exhaustion, and fixed it with targeted indexing in two weeks without disrupting revenue. The second candidate showed the commercial and architectural judgment the role requires.

This diagnostic stage is also where human review earns its keep. The risk of a bad hire compounds quickly once a fractional executive holds production access and makes binding calls, so verify their findings and remediation plan against your own engineers before signing a longer engagement.

At Particle41, we embed senior engineering teams who direct AI agents to deliver real outcomes. You keep full ownership of the IP, there are no long term contracts, and you work directly with people who have built and scaled companies themselves. The result is faster progress with clear accountability.

FAQ

What is a fractional CTO?

A fractional CTO is a senior technology executive who works with a company part time rather than full time, typically a set number of hours per week, while holding the same operational authority as an in house CTO. They embed into the leadership team, make binding architectural decisions, manage engineering cadence, and answer to the chief executive and board for delivery velocity and system uptime, rather than offering outside advice from a distance.

What distinguishes a fractional CTO from an engineering advisor?

An advisor offers episodic feedback and reviews high level ideas without operational responsibility. A fractional CTO embeds directly into your leadership team, makes binding architectural decisions, manages engineering cadence, and answers to the chief executive and board for delivery velocity and system uptime.

How many hours per week should a fractional CTO commit?

As a general rule of thumb, most engagements run ten to twenty hours per week, depending on operational maturity and the complexity of the challenge. Early stage startups preparing for a raise or launch often need the full twenty hours, while stable companies maintaining an established platform may need only ten. The schedule should align with business rhythms, covering sprint planning, executive syncs, and architecture reviews while leaving room for deep work and production escalations.

What interview questions reveal if a candidate is too theoretical?

Ask about the last time they personally diagnosed a production outage or reviewed a pull request, or a time they were forced to ship a compromised solution under deadline pressure and how they managed the resulting debt. Candidates who dwell on abstract methodology will stumble when pressed for specifics, while a grounded practitioner readily describes the trade offs and tooling choices involved.

How can a nontechnical founder evaluate a fractional CTO candidate?

Focus on whether the candidate can translate engineering trade offs into clear commercial consequences. Present a real dilemma, such as latency causing customer churn, and see whether they explain the solution in terms of revenue, retention, and capital rather than jargon. Bring in a trusted senior engineer to audit the candidate's past work, and reserve your own judgment for transparency, communication cadence, and commercial sense.

When should a growing business replace a fractional CTO with a full time hire?

As a rough guideline, consider transitioning to a full time CTO once engineering headcount passes fifteen to twenty engineers, or when product complexity demands forty or more hours of continuous executive presence each week. A good fractional CTO will help identify this point themselves, documenting the roadmap, stabilizing delivery, and helping the chief executive write the job description and interview their permanent successor.

Originally published at particle41.com

How Do You Run a Sprint When Half the Work Is Done by AI Agents?

Ben Johnson — Sat, 31 Oct 2026 14:05:59 +0000

Your team adopted AI coding agents, and the early results looked incredible. Features that took days now take hours. Then the second sprint hit, and something felt off. Velocity charts went haywire. Reviews piled up. Standups got weird, "I'm done, I'm just waiting on review" from three people at once. Your estimation stopped meaning anything. The work was getting faster, but the process was groaning.

Here's what's happening. Scrum, story points, your whole agile apparatus, was designed around one quiet assumption: writing code is the slow, expensive, bottleneck part of building software. Nearly every ceremony you run optimizes for that. When agents write half the code, that assumption is no longer true, and the bottleneck moves from writing to verifying. Your process is still standing guard at the old chokepoint while the traffic jam forms somewhere else.

You don't need to abandon agile. You need to move it to where the work actually is now. Here's what changes, concretely.

Estimation: You're Now Sizing Verification, Not Implementation

Story points were always a proxy for implementation effort. How hard is this to build? That's the question every estimate answered, even when teams pretended it was about "complexity."

When an agent does the building, that question gets a lot less interesting. A feature an agent can scaffold in fifteen minutes might still take a senior engineer a full day to review, reason about, test against edge cases, and trust enough to ship. The effort didn't vanish. It relocated, from the keyboard to the judgment.

So estimate the part that's now expensive. When you size a story, ask: how hard is this to verify? A task touching authentication, money, or data integrity carries enormous verification cost no matter how fast an agent generates it. A cosmetic change is cheap to build and cheap to check. Those two are no longer the same size just because they're similar amounts of code.

A practical move many teams make: stop pointing implementation and start pointing review and verification risk. Some teams add a separate, explicit "verification effort" signal to each story so it's visible in planning. However you do it, the principle holds, the expensive thing is now trust, so estimate trust.

Review Becomes the Job, Not a Chore at the End

This is the biggest cultural shift, and the one teams underestimate most.

In the old world, your engineers spent most of their day writing code they understood deeply because they were authoring it. Review was a smaller slice, a colleague glancing over a few hundred lines of a teammate's work at the end.

Now your team spends much of its day reading and judging code it didn't write, generated by an agent, often more of it than a human would have produced by hand. That's a fundamentally different activity, and a more tiring one. Reading unfamiliar code is exhausting precisely because you have to reverse-engineer someone else's thought process, and it gets harder when AI wrote it: CodeRabbit's analysis found readability issues are roughly 3x more common in AI-generated pull requests than in human-written ones. Spotting the subtle bug, checking the edge cases the agent confidently ignored, is harder than writing, not easier. And it's now the main event.

Plan for it. Treat review as primary work with real capacity allocated, not as overhead squeezed into the cracks. If your sprint plan assumes engineers spend their time building, you've mis-budgeted, because they're spending it verifying. The teams that struggle with agents aren't the ones whose agents write bad code. They're the ones who never reallocated time from writing to reviewing, so review silently becomes the bottleneck nobody planned for.

Smaller Batches, Because Big Agent Output Is Unreviewable

Agents can generate a lot of code fast, and the instinct is to let them, one prompt, one giant pull request, one big leap forward. Resist it.

A 2,000-line agent-generated PR is effectively unreviewable. Your engineer either rubber-stamps it (and you've shipped code nobody understood) or spends two days untangling it (and you've lost the speed you adopted agents for). Both outcomes are bad.

The fix is the same discipline agile always preached, just newly urgent: small, verifiable increments. This isn't new wisdom — DORA's research identifies "working in small batches" as a core capability, and DORA now calls the discipline "especially important when using AI coding assistants," warning that "while AI excels at generating large blocks of code quickly, large changes are difficult to review, test, and integrate safely." Break work so each agent-produced change is small enough that a human can actually read it, reason about it, and trust it in a sitting. Smaller batches mean faster, more reliable review, less risk per merge, and problems caught early instead of buried in a wall of generated code. The economics flipped: when writing was expensive, big batches amortized the cost. When verifying is expensive, small batches are how you keep verification tractable.

Design the Human-in-the-Loop Deliberately

In an agent-augmented sprint, the most important planning question for each piece of work is no longer "who builds this?" It's "where does a human need to be in the loop?"

For some tasks, the answer is "lightly", an agent builds, a human reviews the diff, done. For others, especially anything touching security, compliance, money, data migrations, or core architecture, you want a human deciding the approach before the agent builds, and scrutinizing carefully after. And for a few high-stakes changes, you may want a human writing the critical path themselves and letting the agent handle the surrounding scaffolding.

Make these checkpoints explicit in how you plan and define "done." A task isn't "agent writes it." A task is "agent writes it, human verifies these specific things, human approves this specific gate." Designing where the human judgment lives, intentionally, per task, is now a core part of sprint planning, not an afterthought. Get it right and you get speed and safety. Leave it implicit and you'll oscillate between rubber-stamping and bottlenecking.

Redefine Velocity Around Verified Outcomes

Velocity has always been a slippery metric, but agents break it outright. If you measure throughput by code volume, an agent can make any sprint look heroic while shipping a pile of unreviewed, possibly broken work. Lines of code was always a bad measure; now it's an actively dangerous one.

Move your definition of progress to the only thing that still means something: verified, deployed, working outcomes. Not code generated. Code reviewed, tested, merged, and running in production doing what it was supposed to do. Count features that are genuinely done, trusted, shipped, delivering value, and ignore the raw output that an agent can inflate at will.

This also keeps incentives honest. If your team is measured on verified outcomes, nobody is rewarded for letting an agent flood the repo with code no human stands behind. The number that matters is the number that survives review and reaches users.

Putting It Together in the Sprint

None of this requires a new framework. It requires pointing your existing one at the new bottleneck.

In planning, estimate verification effort, not implementation effort, and explicitly budget capacity for review as primary work.
In refinement, decide the human-in-the-loop checkpoints per story and bake them into the definition of done.
In execution, keep agent output in small, reviewable batches, no unreviewable mega-PRs.
In standup, treat "waiting on review" as the real bottleneck signal it now is, and swarm it, rather than letting verified-but-unreviewed work pile up invisibly.
In the retro and metrics, track verified, deployed outcomes, and ignore raw code volume entirely.

The pattern underneath all of it: every ceremony you run was tuned for a world where writing was the constraint. Re-tune each one for a world where verification is. That's the whole adjustment.

At Particle41, our delivery model pairs senior engineers with AI agents on every engagement, so we've had to redesign our own sprint mechanics around exactly this shift, moving the weight of the process from authorship to verification, from big batches to small ones, from output to outcomes. The teams that get the most out of agents aren't the ones with the cleverest prompts. They're the ones who noticed the bottleneck moved, and moved their process with it.

Sources

DORA, "Working in small batches" (capability) — small batches as a core software-delivery capability, and DORA's specific guidance that this discipline is "especially important when using AI coding assistants" because large AI-generated changes are hard to review, test, and integrate safely.
Aleks Volochnev, "It's harder to review code than to write it — especially with AI code," CodeRabbit (2025) — reading unfamiliar code requires reverse-engineering another's reasoning, and readability issues are roughly 3x more common in AI-generated pull requests than human-written ones.

Originally published at particle41.com

How Do You Write a Product Requirements Doc an AI Agent Can Actually Build From?

Ben Johnson — Fri, 30 Oct 2026 14:08:52 +0000

You've written product requirements docs for years. "Users should be able to reset their password." You hand it to an engineer, they nod, they ask a couple of clarifying questions over the next week, and three weeks later you have a working password reset flow that does roughly what you meant.

Now you hand that same line to an AI coding agent. It builds you a password reset flow in an hour. It's just not the one you wanted. The token never expires. There's no rate limit, so the endpoint is a spam cannon. The email says "Click here" with no branding. And the agent never asked, because it doesn't ask.

This is the shift nobody warned you about. Your PRD was never really a specification. It was a conversation starter, a rough sketch that a human engineer completed with judgment, experience, and a hundred small questions. AI agents don't bring judgment to the gaps. They bring guesses, delivered confidently and at speed. As GitHub's engineering team puts it, we should treat coding agents "more like literal-minded pair programmers" that "excel at pattern recognition but still need unambiguous instructions". The product person who thrives in 2026 is the one who learns to write a spec precise enough that a brilliant, fast, completely literal-minded builder couldn't get it wrong.

Why Your Old PRD Quietly Worked

It's worth understanding why vague PRDs survived for so long, because it explains exactly what breaks.

A human engineer reading "users should be able to reset their password" automatically supplies the missing 80%. They know password reset tokens should expire. They know to rate-limit the endpoint. They know to check whether the email exists without leaking that information to an attacker. They've built this before. And critically, when something is genuinely ambiguous, they stop and ask you. That back-and-forth was the real spec; the document was just where it started.

An AI agent removes both halves of that safety net. It doesn't have years of "we always do it this way" baked in for your specific product, and it doesn't pause to ask. It interprets your words literally and fills every gap with whatever its training suggests is plausible. Where the document was silent, you don't get a question. You get a decision you didn't make.

So the work doesn't disappear. It moves. The thinking that used to happen in conversation now has to happen in the spec, before the agent starts.

The Five Things an Agent Needs That a Human Inferred

A PRD an agent can build from has five properties. Miss them and you're not writing requirements, you're writing wishes.

1. Precision over prose. "Fast login" means nothing to an agent. "The login response must return in under 300ms at the 95th percentile" means something it can build and verify against. Every requirement that hides a number, a threshold, or a defined behavior should expose it. If a sentence could be satisfied two different ways, it will be, and not the way you wanted.

2. Acceptance criteria you could hand to a tester. This is the single highest-leverage thing you can add. For each feature, write the conditions that make it done, concrete enough that you could check each one yes or no. "Given a valid reset token, when the user submits a new password meeting complexity rules, then the password updates and all existing sessions are invalidated." That's not just a spec; it's a test the agent can write its own code against.

3. Examples, not just rules. Agents reason exceptionally well from concrete examples — they are, as GitHub notes, "exceptional at pattern completion, but not at mind reading," so a vague prompt forces the model to guess at unstated requirements. Don't only describe the transformation, show it. "Input: a phone number like (555) 123-4567 or 555.123.4567. Expected output: +15551234567." One worked example of a tricky case removes more ambiguity than three paragraphs of description. Include the edge cases on purpose: the empty input, the malformed one, the one at the boundary.

4. Explicit constraints and non-goals. Tell the agent the playing field. What stack and framework. What existing patterns and components to reuse rather than reinvent. What libraries are approved and which are forbidden. And just as importantly, what not to do, "do not add new dependencies," "do not modify the auth module," "this is read-only, no writes." Agents will cheerfully introduce a new charting library or refactor your data layer if you didn't say don't. The non-goals section prevents more rework than almost anything else.

5. Context and intent. Explain why, not only what. An agent that understands you're building this for a compliance-sensitive healthcare workflow will make systematically better choices in the gaps than one that just sees a CRUD form. Give it the surrounding picture: who the user is, what problem this solves, what the system around it looks like. Intent is what lets a good builder make good judgment calls, and it's true for AI builders too.

What This Looks Like in Practice

Take that password reset line and rewrite it the way an agent can actually build it.

Instead of "users should be able to reset their password," you write a small spec: the user requests a reset by email; the system always responds identically whether or not the email exists, to avoid account enumeration; a single-use token is generated with a short expiry; the reset link is rate-limited per email; the new password must meet the existing complexity policy in the auth module (reuse it, do not reimplement it); on success, all active sessions for that user are invalidated; the confirmation email uses the existing transactional template. (These aren't arbitrary — each is a documented recommendation in OWASP's Forgot Password Cheat Sheet, which calls for consistent responses to prevent enumeration, rate limiting against automated submissions, tokens that are single-use and expire, and giving the user the option to invalidate existing sessions.) You add an acceptance-criteria list, a worked example of the reset email, and a non-goal: do not change the existing login flow.

That's maybe four times longer than the original sentence. It also produces, on the first try, the thing you actually meant, instead of an hour of fast work you have to throw away. This is the trade spec-driven development asks you to make: more thinking up front, far less correction after. It's the same shift GitHub describes as moving "from 'code is the source of truth' to 'intent is the source of truth.'"

The Skill Product People Are Now Hired For

Notice what this rewards. It's not writing volume, an agent doesn't need a forty-page document. It's rigor. The product people who excel at building from agents are the ones who can think clearly about edge cases, who naturally ask "what happens when the input is empty, or malformed, or arrives twice?", who can pin down a behavior precisely instead of gesturing at it.

That's a real and somewhat rare discipline, and it's becoming one of the most valuable things a product person can offer. The agent will handle the typing. You handle the thinking that determines whether the typing is worth anything.

A practical habit: before you finish a spec, read it as if you were the most literal, least charitable engineer alive, someone who will do exactly what's written and nothing more, and gleefully exploit every ambiguity. Where could they technically satisfy your words while violating your intent? Every gap you find that way is a gap the agent would have found for you, the expensive way.

How to Start

You don't need a new template tomorrow. You need to upgrade the parts of your PRD that agents actually consume.

Add an acceptance-criteria block to every feature. If you do one thing, do this. It forces precision and gives the agent something to verify against.
Replace your vaguest sentence with a worked example. Find the requirement most open to interpretation and show concrete input and expected output instead of describing it.
Write a non-goals section. List what the agent should not touch, not build, and not add. It's the cheapest insurance against expensive surprises.
State the constraints once, at the top. Stack, patterns to reuse, libraries allowed and forbidden. Say it before the agent has a chance to guess.
Read it like an adversary. Hunt your own ambiguities before the agent does.

At Particle41, this is a muscle we build with the product teams we work alongside, because our own delivery pairs senior engineers with AI agents, and we've learned firsthand that the spec is now the highest-leverage artifact in the whole process. Get it precise and the agent is astonishingly productive. Leave it vague and you're just generating fast work you'll have to redo. The PRD didn't get less important when agents arrived. It got more.

Sources

Den Delimarsky, "Spec-driven development with AI: Get started with a new open source toolkit," The GitHub Blog (2025) — coding agents as "literal-minded pair programmers" that are "exceptional at pattern completion, but not at mind reading," and the shift to "intent is the source of truth."
OWASP Forgot Password Cheat Sheet, OWASP Cheat Sheet Series — consistent responses to prevent account enumeration, rate limiting against automated submissions, single-use expiring tokens, and session invalidation on reset.

Originally published at particle41.com

What Is a Feature Store, and Does Your Team Actually Need One?

Ben Johnson — Thu, 29 Oct 2026 14:08:57 +0000

Someone on your team read a post from a company running hundreds of models and now wants to stand up a feature store. The pattern traces back to Uber's Michelangelo platform, whose shared "Feature Store" held roughly 10,000 features used to accelerate machine learning projects across the company. The pitch sounds compelling: a central place for all your machine learning features, reusable across teams, consistent between training and production. Who wouldn't want that?

Here's the question worth asking before you commit: do you actually have the problems a feature store solves, or are you about to buy enterprise-grade infrastructure to serve three features to one model?

Feature stores solve two specific, genuinely painful problems. If you have those problems, a feature store is one of the best investments you can make in your ML stack. If you don't have them yet, it's operational overhead pretending to be progress. Let's get precise about what it does, when it's worth it, and what to do before you reach for it.

First, What a Feature Store Actually Is

A feature is an input to a machine learning model, a computed signal about something. "Number of logins in the last 30 days." "Average order value." "Days since last support ticket." Models are fed features, not raw data.

A feature store is a system that does three jobs:

Computes and stores features in one central place, so a feature is defined once and lives in one location.
Serves features for training (the historical values, often large batch reads from an offline store).
Serves features for inference (the current value, fast low-latency reads from an online store, when a live model needs it in milliseconds). On Uber's Michelangelo, for instance, online models reading features from the Cassandra feature store typically saw P95 latency of under 10 milliseconds.

The crucial trick is that the same feature definition powers both the offline training path and the online serving path. That single property is the whole point, and it's what solves the first big problem.

Problem One: Training/Serving Skew

This is the failure mode that makes feature stores worth their weight.

You train a model on historical data. To do it, an engineer writes a query that computes "average order value over the last 90 days." The model trains, validates beautifully, everyone's happy. Then you deploy it. In production, a different engineer, or the same one months later, writes a different piece of code to compute that same feature in real time. Maybe it handles nulls differently. Maybe it uses a slightly different time window. Maybe it rounds.

Now the model is being fed production data that's subtly different from what it trained on. Accuracy quietly degrades. And this is brutal to debug, because both code paths look correct in isolation. The model just performs worse in the wild than it did in testing, and nobody can say why.

This is training/serving skew — which Google's Rules of Machine Learning defines as "a difference between performance during training and performance during serving," often caused by a discrepancy in how data is handled between the two pipelines — and it's one of the most common and most insidious problems in production ML. A feature store eliminates it by construction: there's one definition of the feature, and both training and serving read from it. No drift, because there's nothing to drift between. This is exactly the approach Uber took on Michelangelo, where the same feature-transformation DSL expressions are "applied at training time and at prediction time" so the identical features reach the model in both cases.

If you're already feeling this pain, models that test well and underperform in production with no obvious cause, a feature store is pointing right at your problem.

Problem Two: Feature Reuse and Consistency

The second problem shows up as you scale teams and models.

Your fraud team builds a "customer lifetime value" feature. Your marketing team builds one too. So does your churn team. Three definitions, three slightly different sets of assumptions, three places to maintain, and inevitably three different answers to "what's this customer worth?" Multiply across dozens of features and several teams and you get duplicated work, wasted compute, and a quiet erosion of trust, because no two models agree on the underlying numbers.

A feature store gives you a shared, documented catalog. Define "customer lifetime value" once, correctly, and every team consumes the same version. New models get built faster because half the features they need already exist. Governance improves because there's one definition to audit.

This is a real benefit, but notice it's fundamentally a scale benefit. With one or two teams and a handful of models, you don't have a reuse problem. You have a documentation problem, which is much cheaper to fix.

When a Feature Store Is Overkill

Here's the part the vendor decks skip. A feature store is infrastructure. It has to be deployed, monitored, kept available, paid for, and learned by your team. That cost is justified only when the problems above are biting.

You almost certainly don't need a feature store yet if:

You have one or two models in production. Skew is manageable when there's one of everything to coordinate.
Your features are mostly batch, recomputed nightly, with no hard low-latency serving requirement. Half the value of a feature store is the online serving path; if you don't need it, you're paying for capability you won't use.
You have a single ML team. The reuse problem barely exists when everyone already talks to each other.
Your features are simple and stable. If a feature is just a column in a table that already updates correctly, wrapping it in a platform adds ceremony, not safety.

In these cases, adopting a feature store is how you end up maintaining a platform that serves three features to one model, while the actual work, building things that move the business, waits.

What to Do First: The Simpler Stack

Before you buy or build a feature store, get the cheaper version of its discipline in place. It solves most of the same problems at a fraction of the operational cost.

Define features once in your transformation layer. A shared dbt model (or equivalent) that computes "customer lifetime value" in exactly one place, that every model reads from, gives you most of the consistency benefit with zero new infrastructure.
Write a feature catalog. A documented table, owner, definition, refresh cadence, gets you reuse and governance without a platform. Boring, effective.
Make training and serving read the same source. If both your training pipeline and your live model query the same materialized feature table, you've eliminated skew without a feature store. The skew problem is really a "two sources of truth" problem; collapse it to one.
Standardize your serving path. If you do need low-latency reads, a well-designed cache or key-value store in front of your feature tables may be all the "online store" you need at your scale.

This stack handles a real ML practice running a handful of models. It's the right answer for far more teams than the feature store evangelists admit.

The Honest Decision Framework

So, do you need one? Walk these questions in order:

Are models that test well underperforming in production for no clear reason? If yes, you likely have skew, and a feature store directly addresses it.
Do you have low-latency serving needs that your current setup can't meet? If yes, the online-store half of a feature store earns its keep.
Are multiple teams rebuilding the same features and disagreeing on the answers? If yes, you have a reuse problem at a scale where a catalog alone may no longer cut it.
Do you have the operational maturity to run another piece of always-on infrastructure? If no, fix that before adding it.

If you answered yes to two or more of the first three and yes to the fourth, a feature store is probably the right next investment, look hard at managed options like Tecton, or open-source Feast, before building your own. If you didn't, the simpler stack above will serve you better and longer than you'd guess.

Buy the Aspirin When You Have the Headache

The pattern we see repeatedly: teams adopt heavyweight ML infrastructure based on where they hope to be, then spend their scarce engineering capacity maintaining a platform their actual workload doesn't justify. A feature store is a fine answer to a problem you have. It's an expensive answer to a problem you might have someday.

At Particle41, we help teams make exactly this call, right-sizing the data and ML platform to the problems actually in front of them, then scaling it deliberately as those problems grow real. Sometimes that means standing up a feature store. More often, at least at first, it means a clean transformation layer, a shared catalog, and one source of truth, the simpler stack that quietly solves the same problems without the operational tax. The goal isn't the most impressive architecture. It's the one your team will still be glad they have in a year.

Sources

Mike Del Balso and Jeremy Hermann, "Meet Michelangelo: Uber's Machine Learning Platform," Uber Engineering (2017) — the shared Feature Store (~10,000 features), the feature-transformation DSL applied identically at training and prediction time, and sub-10ms P95 online serving latency.
Martin Zinkevich, "Rules of Machine Learning: Best Practices for ML Engineering," Google for Developers — defines training-serving skew and why it should be explicitly monitored.

Originally published at particle41.com

How Do You Prepare Unstructured Data for AI Without a Data Science Team?

Ben Johnson — Wed, 28 Oct 2026 14:08:47 +0000

You've got a decade of institutional knowledge sitting in PDFs, support emails, contracts, and Word documents. You're not unusual: a Box-sponsored IDC white paper estimates that 90% of data is unstructured — the documents, images, and audio that can't be loaded neatly into rows and columns. You know there's an AI use case in there somewhere, a search assistant, a knowledge bot, a way to finally make that archive useful. And then you stop, because you've convinced yourself this requires a data science team you don't have and can't afford to hire.

Here's the uncomfortable truth: it mostly doesn't.

Preparing unstructured data for AI is not a research problem. It's a plumbing problem. The hard, novel, PhD-level work, training foundation models, was already done by someone else. What's left for you is a pipeline: take messy documents in one end, get clean, searchable, AI-ready data out the other. That's engineering. And if your team can build an API and integrate a third-party service, they can build this.

Let's walk the actual pipeline, step by step, so you can see exactly what your existing engineers would be doing.

Step 1: Extraction — Getting the Text Out

Everything starts with pulling raw text out of your source files. This sounds trivial and mostly is, until you hit the exceptions.

A native PDF (one generated from a document) gives up its text easily. Libraries like pdfplumber, PyMuPDF, or unstructured will extract it cleanly. A .docx, an .html email, a .txt file, all straightforward.

The trap is scanned PDFs. A meaningful chunk of any real document archive, especially anything older or anything that's been printed and re-scanned, is actually images of text, not text. Run a normal extractor on those and you get nothing. The fix is OCR: a service like AWS Textract, Google Document AI, or an open-source engine like Tesseract — the free, Apache-licensed OCR engine originally built at HP, open-sourced in 2005, and long maintained by Google — reads the image and returns text. Build a simple check early in your pipeline: if a page extracts fewer than, say, 20 characters, route it to OCR. That one branch handles 90% of the surprises.

Tables and multi-column layouts are the other gotcha. A two-column newsletter or a financial statement will extract as a jumble if you're not careful. Tools like unstructured are layout-aware and worth the dependency for document-heavy use cases.

This step is pure engineering. There is no math.

Step 2: Cleaning — Stripping the Junk

Raw extracted text is dirty. Page headers and footers repeated on every page. "Page 4 of 12." Navigation menus from HTML. Email signatures, legal disclaimers, and "this message was scanned for viruses" boilerplate. Encoding artifacts where a smart-quote became three garbage characters.

This junk matters more than people expect. When you later search this data, boilerplate that appears in every document creates noise that pollutes results, the AI keeps "finding" the disclaimer instead of the answer.

Cleaning is mostly unglamorous, deterministic work:

Remove repeated headers and footers (detect lines that recur across many pages).
Strip HTML tags and normalize whitespace.
Drop standard signatures and disclaimers with pattern matching.
Fix common encoding issues (normalize to UTF-8, repair mojibake).

You don't need machine learning for any of this. You need a regular engineer who's willing to look at the actual output, notice the patterns, and write rules. The quality of your final AI system is hugely dependent on this step, and it's entirely within reach of a normal backend developer who cares about edge cases.

Step 3: Chunking — Breaking Documents into Usable Pieces

This is the step people underestimate most, and it's where a little thought pays off enormously.

AI systems don't retrieve whole documents. They retrieve pieces. So you split each cleaned document into chunks. Get this wrong and everything downstream suffers, even with perfect data.

Chunk too large and each piece covers multiple topics, so retrieval becomes imprecise and you blow past the context the model can use. Chunk too small and you shred the meaning, a sentence pulled out of its paragraph often answers nothing.

A sensible starting point for most text is chunks of roughly 500-1,000 tokens (think a few paragraphs) with a 10-15% overlap between adjacent chunks so you don't sever ideas at the boundary. But the real win is respecting structure. Don't blindly cut every 800 tokens. Split on natural boundaries: sections, headings, paragraphs. A chunk that aligns with a logical section retrieves far better than one that starts mid-sentence.

There's no PhD here either. There's judgment, a few parameters to tune, and a willingness to eyeball the output and adjust. Treat chunking as something you iterate on, not something you set once.

Step 4: Metadata Enrichment — Making Chunks Findable

Every chunk should carry metadata, and this is the cheapest, highest-leverage step in the whole pipeline.

Attach to each chunk: the source document, its date, author or department, document type, and where in the document the chunk came from. If you can infer a few tags or a category, attach those too.

Why does this matter so much? Because metadata is what lets you filter before you search. "Find me information about refund policy, but only from documents dated after January 2026, and only from the policy team." Without metadata, your AI searches everything and surfaces the outdated 2023 policy with equal confidence. With it, retrieval gets dramatically more precise and your answers stop being subtly wrong.

Most of this metadata is sitting right there in your source systems, file paths, document properties, email headers. Extracting it is straightforward engineering. Skipping it is one of the most common reasons AI pilots underwhelm.

Step 5: Embedding — Turning Text into Vectors

Now the part that sounds like data science but is actually one API call.

To make text searchable by meaning rather than exact keywords, you convert each chunk into an embedding, a vector of numbers that captures its semantic content. You do not train this. You call a hosted embedding model, OpenAI, Cohere, Voyage, or an open-source model like a sentence-transformer, and it hands back the vector.

You store those vectors in a vector database (Pinecone, Weaviate, pgvector if you're already on Postgres, or many others). At query time, you embed the user's question the same way and the database returns the chunks whose vectors are closest. That's semantic search, and it's the engine behind most "chat with your documents" systems.

The entire embedding step is: loop over chunks, call an API, store the result with its metadata. Any engineer who has integrated a payment provider has done strictly harder work.

What This Actually Looks Like as a Project

Put the steps together and you have a clear, buildable pipeline, no research team required:

Weeks 1-2: Stand up extraction (including the OCR branch) and validate it against a representative sample of your real documents.
Weeks 3-4: Build cleaning rules by inspecting actual output, then implement structure-aware chunking and tune the parameters.
Weeks 5-6: Add metadata enrichment, wire up an embedding model, and load a vector store.
Week 7+: Test retrieval quality end to end, fix the chunks and filters that retrieve poorly, and iterate.

A capable backend team can have a working version of this in a handful of sprints. The work that actually determines quality, cleaning and chunking, is ordinary engineering done with care, not exotic modeling.

Start Now, Not After You Hire

The teams stuck "waiting for data science" are usually waiting for a permission they already have. The barrier to making your unstructured data AI-ready isn't talent you lack. It's recognizing that the job in front of you is a pipeline, and you already employ people who build pipelines.

Pick one well-bounded corpus, your support knowledge base, one product's documentation, a contract archive, and run it through these five steps. You'll learn far more from one real pipeline than from six months of planning to hire.

At Particle41, this is exactly the kind of work our senior engineers, augmented by AI agents, stand up for teams that don't have a dedicated ML group. We build the extraction, cleaning, chunking, and retrieval layer so your data is genuinely ready for AI, then hand you a pipeline your own team can run. The data science team you think you need is, more often than not, an engineering team you already have.

Sources

Box Blog (citing the Box-sponsored IDC white paper Untapped Value: What Every Executive Needs to Know About Unstructured Data, Doc. US51128223, August 2023) — the estimate that 90% of data is unstructured.
Tesseract (software), Wikipedia — Tesseract as a free, Apache-licensed open-source OCR engine, originally developed at HP and released as open source in 2005.

Originally published at particle41.com

What Is Data Lineage, and Why Does It Matter for AI Trust?

Ben Johnson — Tue, 27 Oct 2026 14:13:58 +0000

Your data quality dashboard is green. Every metric is in range. Accuracy, completeness, freshness, all nodding along. And yet your AI agent just made a decision that nobody on your team can explain, and the customer on the other end is not happy about it.

You start digging. Where did this number come from? Which table? Which join? Which transformation introduced the error? Three hours later you're reading a dbt model that someone wrote in 2024 and left the company in 2025, and you still don't have an answer.

This is the gap data quality scoring can't fill. Quality tells you a value looks wrong. It cannot tell you why it's wrong, or where it went wrong. For that, you need data lineage. And in an era where AI agents make autonomous decisions on your data, lineage stops being a nice-to-have and becomes the foundation of whether anyone can trust the system at all.

Lineage and Quality Are Not the Same Thing

People conflate these constantly, so let's be precise.

Data quality is a measurement of state. Is this email valid? Is this field populated? Is this record fresh? It's a snapshot judgment about whether data meets a standard right now.

Data lineage is a map of movement. It traces every piece of data from its origin, through every system, transformation, join, and aggregation, all the way to where it's consumed by a dashboard, a report, or an AI model. It answers where did this come from and what happened to it along the way.

Quality is the verdict. Lineage is the trail of evidence. You can have perfect quality scores and still have no idea how your data got to its current state, which means when something breaks, you're guessing.

Here's the practical distinction: a quality check might tell you that 8% of your customer revenue records are suddenly null. Useful. But lineage tells you that those nulls appeared after a specific staging model started doing a left join against a table that lost rows in an upstream API change on a specific date. One tells you that there's a problem. The other tells you where to go fix it.

Why AI Raises the Stakes

For a decade, lineage was something most teams could get away with ignoring. Analysts are skeptical by nature. They sanity-check their own numbers, exclude weird outliers, and ask questions before they act. The human in the loop absorbed a lot of lineage debt.

AI agents don't do that. An agent fed a bad input doesn't pause to wonder whether the number looks plausible. It acts. It adjusts a price, approves a claim, routes a ticket, or generates a recommendation, and it does so with confidence and at scale. The skepticism that used to catch lineage problems is gone.

That changes three things.

First, the blast radius is bigger. A bad input no longer corrupts one analyst's spreadsheet. It propagates through every downstream decision the agent makes until someone notices.

Second, the debugging window shrinks. When an agent is making thousands of decisions an hour, you can't afford a three-day investigation to find the source of a bad input. You need to trace it in minutes.

Third, the trust question becomes formal. When a human made a judgment call, "I used my experience" was an acceptable explanation. When an AI makes the call, regulators, auditors, and customers increasingly want to know exactly what data drove it. Without lineage, you literally cannot answer.

The Four Jobs Lineage Does for AI

Lineage isn't an abstract governance virtue. It does concrete work that directly determines whether your AI is trustworthy.

Debugging and root-cause analysis. When a model drifts or an agent makes a strange call, lineage lets you walk backward from the output to the exact input that caused it. Without it, you're grep-ing through pipelines. With it, you click a node and see the upstream transformation that broke. This alone can cut incident resolution from days to under an hour.

Impact analysis before you change anything. Lineage runs forward too. Before you alter a source schema, deprecate a column, or refactor a pipeline, lineage shows you every downstream model, feature, and agent that depends on it. You learn that "harmless" column rename will silently break the fraud model before you ship it, not after.

Compliance and audit. Regulations like the EU AI Act and sector rules in finance and healthcare increasingly demand that automated decisions be explainable and traceable. The Act's Article 12 requires high-risk AI systems to keep automatic logs that ensure "a level of traceability of the functioning" of the system over its lifetime, and Article 86 gives affected people a right to an explanation of individual automated decisions. "We can show every input and transformation behind this decision" is a compliance posture. "We think it came from the CRM" is a liability.

Trust and adoption. Your own stakeholders won't rely on an AI system they can't interrogate. When a skeptical exec asks "where did this recommendation come from?", being able to show the full path turns a blocker into a believer.

What Lineage Actually Looks Like in Practice

Lineage exists at different levels of granularity, and you don't need all of them everywhere.

Table-level lineage tells you that the customer_ltv table is built from the orders, accounts, and subscriptions tables. It's coarse but cheap, and it's enough to answer most "what feeds this?" questions.

Column-level lineage goes deeper: it tells you that the predicted_churn_risk feature is derived specifically from the last_login_at and support_ticket_count columns, three transformations back. This is what you need when debugging a specific bad value or doing precise impact analysis.

Transformation-level lineage captures the actual logic, the joins, filters, and aggregations applied at each hop. This is the gold standard for AI, because the error you're chasing is often in the logic, not the source.

You build this in layers. Modern tooling captures a surprising amount automatically. dbt generates lineage from your model DAG for free. Orchestrators like Airflow and Dagster expose task dependencies. Open standards like OpenLineage — an LF AI & Data Foundation graduate project with built-in integrations for Airflow, Spark, Flink, and dbt — let you emit lineage events from across a heterogeneous stack into one graph. The hard part is rarely the first 70%. It's the long tail: the hand-rolled Python script, the spreadsheet someone uploads monthly, the third-party API with no metadata. Those gaps are exactly where AI trust quietly breaks, so they're worth the manual instrumentation.

A Pragmatic Way to Start

You don't boil the ocean. You start with the data that feeds the decisions that matter most.

Pick your highest-stakes AI use case. The model or agent whose mistakes cost the most money or trust. That's where lineage pays off first.
Trace its inputs backward, end to end. Document every source, hop, and transformation feeding it. Do it by hand if you have to. You will find things that surprise you.
Adopt automated capture where it's free. Turn on dbt docs, wire up OpenLineage, connect your orchestrator's metadata. Get the easy 70% flowing into one place.
Instrument the gaps deliberately. The manual scripts and external feeds that automation misses are where your worst incidents will originate. Add lineage there on purpose.
Make it queryable, not just documented. Lineage buried in a static diagram is dead. Lineage you can query during an incident, ideally from the same place your alerts fire, is alive.

Expect this to take a focused team a few weeks per critical pipeline to get right, not a single afternoon. But the first time an agent does something inexplicable and you trace it to root cause in ten minutes instead of three days, the investment has already paid for itself.

The Bottom Line

Data quality answers "is this value good?" Data lineage answers "how did this value come to be, and what depends on it?" Both matter, but only one of them lets you debug, audit, and ultimately trust an AI system making autonomous decisions.

As AI moves from suggesting to deciding, the organizations that win won't be the ones with the cleanest data. They'll be the ones who can explain their data, instantly, all the way back to the source. That's not a dashboard. That's lineage.

At Particle41, we help teams build the data foundations that make AI trustworthy, including the lineage layer that turns "the model did something weird" into a traceable, fixable, auditable answer. If you're putting AI on top of data you can't fully trace, that's the conversation worth having first.

Sources

EU AI Act, Article 12: Record-Keeping — requires high-risk AI systems to keep automatic logs ensuring "a level of traceability of the functioning" of the system over its lifetime.
EU AI Act, Article 86: Right to Explanation of Individual Decision-Making — gives affected persons the right to "clear and meaningful explanations of the role of the AI system in the decision-making procedure."
LF AI & Data Foundation: Graduation of the OpenLineage Project — OpenLineage as an open lineage-metadata standard with built-in integrations for Airflow, Spark, Flink, and dbt.

Originally published at particle41.com

Should You Use One Big AI Model or Several Specialized Ones?

Ben Johnson — Mon, 26 Oct 2026 14:59:30 +0000

You're designing an AI system and you hit the fork in the road. Option one: pick the best frontier model and send everything to it. Simple, one integration, one bill, one thing to reason about. Option two: route different requests to different models, a small fast one here, a fine-tuned specialist there, the big one only when you need it.

The first option is seductive because it's simple, and simple is usually the right default in engineering. But with AI models, "use the biggest one for everything" is often the expensive, slow, and surprisingly less accurate choice. It's flying a 747 to pick up groceries. It gets there. It's also absurd overkill for the trip.

The honest answer isn't "monolith good" or "specialists good." It's a tradeoff between four things, cost, latency, accuracy, and maintenance, and which way it tips depends entirely on your workload. Let's make the tradeoff concrete so you can actually decide.

The Case for One Big Model

There's a real argument for the monolith, and it's mostly about simplicity, which is worth more than engineers like to admit.

One integration, one mental model. A single model means a single API, a single set of prompts, a single thing to monitor and debug. Your whole team understands the system. There's no router to maintain, no routing logic that can misroute, no zoo of models to keep track of.

Broad capability out of the box. Frontier models are genuinely general. They handle reasoning, generation, extraction, code, and conversation without task-specific work. For a brand-new product where you don't yet know your traffic patterns, that flexibility is exactly what you want. You can build fast and figure out the distribution later.

Lower fixed engineering cost. No classifier to train, no specialists to fine-tune and evaluate separately, no orchestration layer. For a small team or an early product, that saved effort is the whole point.

If your volume is modest, your tasks are varied, or you're still in discovery, one good model is the correct call. Don't build a routing system to save money you aren't yet spending.

The Case for Several Specialized Models

Now the other side, which gets stronger fast as you scale.

Cost. This is the headline. A frontier model can cost one to two orders of magnitude more per token than a small, capable one — published 2026 Claude pricing, for instance, spans $0.25 per million input tokens on the fastest model up to $15 on the most capable. If most of your traffic is routine, classification, extraction, short answers, simple summarization, you're paying frontier prices for work a cheap model does just as well. Route that majority to the small model and reserve the expensive one for the genuinely hard requests, and your total bill can drop substantially with no real quality loss.

Latency. Bigger models are generally slower. A request that didn't need deep reasoning still waits behind the full model's response time. A smaller specialist returns faster, which matters enormously for anything interactive. Routing easy requests to fast models tightens your p95 latency where users actually feel it.

Accuracy, counterintuitively. A small model fine-tuned on one narrow task frequently beats a giant general model at that task. OpenAI reported that a fine-tuned version of GPT-3.5 Turbo "can match, or even outperform, base GPT-4-level capabilities on certain narrow tasks". The specialist has seen thousands of your exact examples; the generalist is reasoning from first principles every time. For ticket classification, structured extraction, or domain-specific formatting, a tuned small model can be both cheaper and more accurate than the frontier option.

The pattern that captures most of this value is a router: a lightweight classifier looks at each request, decides its type and difficulty, and dispatches it to the best-fit model, escalating to the big one only when needed.

The Maintenance Tax Nobody Mentions

Specialization isn't free, and the cost shows up in places that don't appear in the per-token comparison.

More models, more evaluation. Every model in your system needs its own eval set, its own quality monitoring, its own regression testing. Five specialists is five evaluation pipelines, not one. When a provider ships a new version, you have five upgrade decisions instead of one.

The router is a new failure mode. Now you can be wrong in a new way: routing a hard request to a model too weak to handle it. The router itself needs evaluation, monitoring, and tuning, and a misrouting bug can degrade quality in ways that are hard to trace because the model wasn't wrong, the routing was.

More operational surface. More integrations, more fallback logic, more places for latency to hide, more for on-call to understand. The cognitive load on the team is real and it's ongoing.

This is the crux of the decision. Specialization buys you cost, latency, and accuracy, and it bills you in complexity and maintenance. The question is whether your volume and stakes make that trade worth it.

How to Actually Decide

Run your workload through these questions and the answer usually falls out.

What's your volume? At low volume, the cost savings from routing are small in absolute dollars and won't justify the engineering. At high volume, a 50% reduction in per-request cost is a large number that easily pays for a router and then some. Volume is the biggest single factor.

How repeatable are your tasks? If your traffic is a few well-defined, high-frequency task types, specialists fit beautifully, you can fine-tune for each. If every request is unique and unpredictable, a generalist's flexibility wins and routing buys you little.

How latency-sensitive is the experience? Interactive, user-facing features benefit a lot from routing fast paths to fast models. Background and batch work cares less, so the latency argument for specialization weakens.

What's your team's capacity to maintain complexity? Be honest. A small team already stretched thin may get more value from one model they fully understand than from a routing system they maintain poorly. Complexity you can't operate well is worse than simplicity.

What are the stakes per request? High-stakes outputs may justify always using your strongest model regardless of cost, or may justify a specialist fine-tuned precisely for that critical task. Low-stakes, high-volume work is where cheap models shine.

The Pragmatic Path

You don't have to pick a side on day one, and you shouldn't.

Start with one good model. Ship the product, learn your real traffic distribution, and avoid premature optimization. You can't route intelligently until you know what your requests actually look like.
Instrument cost and latency per task type. Let the data tell you where the money and the slowness concentrate. It's almost always a specific high-volume task type.
Specialize where the data points. When one task type dominates your cost or latency, that's your first candidate to split off, route to a cheaper model or fine-tune a specialist for it. Add specialists one at a time, justified by measured savings, not by architectural ambition.
Keep the frontier model as the fallback. Route confidently handled requests to specialists; let anything uncertain or out-of-distribution escalate to the big general model. You get the savings without sacrificing the safety net.

This earns complexity only where it pays, which is the right way to spend it.

At Particle41, we treat the one-versus-many question as an evolving architecture decision, not a one-time bet. We start teams on a simple, single-model footing, instrument it so the cost and latency hotspots are visible, then introduce routing and specialized models exactly where the numbers justify the added complexity. Pairing senior engineers with AI-augmented delivery, we build the router, the eval coverage, and the monitoring that make a multi-model system trustworthy instead of fragile.

The wrong question is "which is better, one model or several." The right question is "is the complexity of specialization worth it for my volume, my tasks, and my team." Answer that honestly, start simple, and let your own traffic tell you when it's time to split.

Sources

GPT-3.5 Turbo fine-tuning and API updates — OpenAI — OpenAI's statement that a fine-tuned GPT-3.5 Turbo can match or outperform base GPT-4-level capabilities on certain narrow tasks.
Prompt caching with Claude — Anthropic — per-model input/output token pricing showing the cost spread between fast and frontier models.

Originally published at particle41.com

What Does Modern Software Look Like for Travel and Hospitality Platforms?

Ben Johnson — Sun, 25 Oct 2026 14:20:13 +0000

You run a travel or hospitality platform, and on paper it works. Customers can search, book, and pay. But the support tickets tell a different story: inventory that shows available and then isn't, a site that crawls every holiday weekend, an app that's clearly an afterthought to the desktop site, and a "concierge" experience that's really just a contact form.

The uncomfortable truth is that traveler expectations moved and a lot of travel software didn't. The platforms built for a world of desktop search and nightly inventory syncs are now competing against companies that rearchitected for real-time, mobile-first, AI-assisted travel, and travelers can feel the difference even when they can't name it.

So what does modern actually look like here? It's not a redesign. It's a set of architecture decisions, and most of the legacy pain traces back to skipping them.

Real-Time Inventory Is the Whole Ballgame

Nothing erodes trust in a travel platform faster than selling something that isn't available. A room shown as open that gets declined at checkout, a flight price that vanishes the moment you click, a tour with "2 spots left" that's actually sold out, each of these is a customer you may never get back.

The root cause is almost always inventory that updates in batch instead of in real time. Legacy platforms sync availability on a schedule, hourly, or worse, overnight, which means the inventory a traveler sees is a snapshot of the past, not the truth right now. During high-demand periods, that gap between displayed and actual availability becomes a steady stream of failed bookings and frustrated customers.

Modern platforms treat inventory as a real-time, event-driven problem. When availability changes anywhere in the system, that change propagates immediately to everything that displays it. This requires real architecture, event streaming, appropriate caching with fast invalidation, and a data model built for high-frequency updates, but it's the foundation of a platform travelers can trust. If you fix one thing, fix this.

Taming Legacy GDS and Supplier Integrations

Travel runs on a web of integrations: Global Distribution Systems, channel managers, property management systems, airline and hotel APIs, payment providers. Many of these are old, slow, occasionally flaky, and not going away. The mistake legacy platforms make is wiring directly and synchronously into all of them, so that one slow GDS response or one partner outage drags down the entire experience.

Modern architecture decouples and insulates these integrations:

An integration layer that normalizes the different formats and protocols of GDS, suppliers, and channel managers behind a clean internal interface, so the rest of your system doesn't have to know which messy upstream it's talking to.
Resilience patterns, timeouts, circuit breakers, graceful degradation, so a single slow or failing partner degrades one capability instead of crashing your whole search.
Caching and async processing where appropriate, so you're not making a traveler wait on a sluggish third party for every interaction.

The goal is to keep the necessary legacy connections without letting their age and unreliability define your customer experience. The integrations are a constraint to engineer around, not an excuse for a slow platform.

Surviving Peak Season Without Overpaying All Year

Travel demand is spiky in the extreme. Holiday booking windows, flash sales, a destination going viral, traffic can jump several-fold in hours and then recede. Two classic failures bracket this: the platform that falls over under load and loses bookings at the exact moment they're most valuable, and the platform that provisions for peak year-round and bleeds money on idle capacity for the other eleven months.

Modern cloud architecture solves both with elastic, auto-scaling infrastructure. The system scales up automatically to absorb a holiday spike and scales back down when demand normalizes, so you pay for capacity roughly in line with what you actually use. Practically, that means designing for horizontal scalability, using managed and serverless services where the scaling burden shifts to the provider, and load-testing against realistic peak scenarios before peak season arrives rather than discovering your ceiling live. The objective is simple: the platform should be invisible during the biggest traffic day of the year, not the story.

Personalization That Actually Helps

Travelers increasingly expect a platform to know them. Not in a creepy way, in a useful one: remembering preferences, surfacing relevant options, not making them re-enter the same information every trip. The expectation is now mainstream — Salesforce's research found the share of customers who feel brands treat them as unique individuals jumped to 73% in 2024, up from 39% the year before. Most legacy platforms treat every visit as if it's the first, which is both a worse experience and a missed revenue opportunity.

Modern personalization uses the data you already have, booking history, stated preferences, behavior, to make the next trip easier. ML models can recommend destinations and upgrades that fit the traveler, predict what they're likely to want, and tailor the experience without making them dig for it. Done with respect for privacy and consent, this raises conversion and loyalty simultaneously, because a platform that remembers you is one you come back to. The engineering requirement underneath is, predictably, good data infrastructure: unified customer profiles and the pipelines to act on them in real time.

Mobile-First, Not Mobile-Also

Travel is increasingly managed end to end from a phone, booked, changed, and navigated in the moment, often while standing in an airport or a hotel lobby. Booking Holdings, the largest online travel company, told investors it has seen "a significant shift of business... to mobile platforms" with "an increasing percentage of our room nights... booked on our mobile apps". Yet plenty of travel software still treats mobile as a shrunken version of the desktop site rather than the primary surface.

Modern travel platforms are mobile-first by default. That means designing the core flows for the phone first, leaning into mobile-native capabilities, location, notifications, mobile wallets, offline access to a saved itinerary when connectivity drops, and building so the mobile experience is fast and complete rather than a compromised port. The traveler standing at a delayed gate trying to rebook is your most important user, and the experience has to work for them on a small screen with bad signal. Get that right and you've designed for everyone.

The AI Concierge That Knows the Booking

The most visible recent shift is the AI concierge, and it's where expectations are climbing fastest. Travelers increasingly want to ask, in plain language, "move my reservation to next weekend" or "what's the closest restaurant still open" or "my flight got canceled, what are my options," and get a real answer immediately, at any hour.

The difference between a gimmick and genuine value is integration. A chatbot bolted on the surface that can't see the booking is worse than nothing. A concierge wired into your real systems, that knows the customer's actual reservation, can check live inventory, and can take action, rebook, upgrade, answer the specific question, is a meaningful upgrade to the experience and a serious reduction in support load. Trust still has to be engineered in, though: Salesforce found 72% of customers think it's important to know when they're communicating with an AI agent, and 71% believe a human should validate AI output — which is why the guardrails around money-touching actions matter as much as the conversational layer. The hard part isn't the conversational layer; it's connecting AI to live inventory, booking systems, and customer data so it operates on truth, with the right guardrails for actions that touch money or commitments. The 2 a.m. cancellation, handled well by an AI that actually knows the booking, is exactly the moment that earns lifetime loyalty.

Modernizing Without Stopping the Business

The catch is that you can't take a live travel platform offline to rebuild it. Bookings don't pause. The right approach is incremental: stand up a modern architecture alongside the legacy system, move real-time inventory and the integration layer first because they unblock everything else, then layer in personalization, mobile-first experiences, and the AI concierge, validating each piece in production before moving on. You ship improvements continuously instead of betting on one giant relaunch.

This is the kind of build we do at Particle41 for travel and hospitality platforms: real-time inventory architectures, resilient GDS and supplier integration layers, elastic infrastructure that survives peak season, and AI concierge experiences wired into live booking systems, delivered incrementally so the business keeps running while the platform gets modern. The constraints in travel are real, old integrations, spiky demand, zero tolerance for downtime, but none of them is a reason to keep losing customers to a platform that feels like 2015.

Travelers don't grade your architecture. They just quietly book with whoever makes the trip easier. Modern travel software is how you make sure that's you.

Sources

Originally published at particle41.com

How Do You Keep AI Costs Predictable as Usage Scales?

Ben Johnson — Sun, 25 Oct 2026 14:09:50 +0000

You set an AI budget for the year. Reasonable number, board approved, everyone moved on. Then a feature got popular, usage climbed, and the bill didn't climb with it, it leapt. Now you're explaining a variance nobody modeled, and the honest answer is that you never actually had a budget. You had a guess attached to a meter you weren't watching.

This is the trap of AI cost. It doesn't behave like a software license, a fixed number you negotiate once and forget. It behaves like a utility bill: a usage-based meter that scales with every request, every token, every retry. Treating it like a line item instead of a meter is exactly how the surprise invoice happens.

Keeping AI costs predictable isn't a finance exercise you do at planning time. It's an engineering discipline you build into the system. The unit of work is the token, and predictability starts there, not in a spreadsheet.

Know Your Token Unit Economics

You cannot control what you cannot measure, and the thing to measure is cost per request, not cost per month. The monthly number is an outcome. The per-request number is the lever.

Start with the arithmetic. LLM pricing is per token, split between input and output, and output typically costs several times more than input. As a reference point, published 2026 pricing for Claude models runs from $0.25 per million input tokens on the fastest model to $15 per million on the most capable, with output priced several times higher, so a small fast model can be one to two orders of magnitude cheaper than a frontier one. Your cost per request is:

(input tokens × input price) + (output tokens × output price)

Run that on your actual traffic. A request with 8,000 tokens of context and a 1,000-token answer on a frontier model is a very different number than the same request on a small model, and the difference is the entire game. Then multiply by volume. If a feature does 100,000 requests a day at $0.05 each, that's $5,000 a day, $150,000 a month. Triple the traffic and you triple the bill, automatically, whether or not anyone planned for it.

The single most important number in your AI program is cost per request, and most teams can't state theirs. Compute it per feature, watch how it moves, and you've turned an unpredictable bill into a model you can forecast.

Cache the Work You're Repeating

A huge fraction of AI spend is paying, over and over, to process the exact same tokens. Your system prompt, your tool definitions, your few-shot examples, the retrieved context that barely changes between similar questions, all of it gets re-sent and re-billed on every call.

Caching attacks this directly:

Prompt caching lets the provider store and reuse the unchanging prefix of your prompts. For a long, stable system prompt sent on every request, this can cut the input cost of that portion by up to about 90% (and latency by up to 85%). If 80% of your input tokens are a fixed preamble, the savings are enormous and the change is small.
Semantic response caching returns a stored answer when a new request is semantically identical or near-identical to one you've already answered. For FAQ-style or high-repetition workloads, this can deflect a meaningful share of requests away from the model entirely, dropping their cost to roughly zero.

Caching is usually the highest-return cost lever because it requires no quality tradeoff. You're not making the product worse; you're refusing to pay twice for the same computation.

Route Requests to the Right Model

Sending every request to your most capable, most expensive model is the most common cost mistake, and it's pure waste. Most requests don't need a frontier model. They need a model that's good enough, and "good enough" is often 20-50x cheaper.

Model routing means classifying each request by difficulty and sending it to the cheapest model that can handle it:

Simple, structured, or high-volume tasks, classification, extraction, short answers, routine summarization, go to a small fast model.
Genuinely hard tasks, complex reasoning, nuanced generation, multi-step problems, go to the frontier model.
Borderline cases can start cheap and escalate only when the small model signals low confidence or fails a validation check.

If 70% of your traffic is routine and you move it from a frontier model to one that costs a twentieth as much, you've cut a large chunk of total spend with negligible quality impact, because those requests never needed the expensive model in the first place. The work is in building the classifier and the eval coverage to prove the cheap model holds up, but the payoff scales with every request forever.

Batch the Work That Can Wait

Not every request needs an answer in 200 milliseconds. Overnight report generation, bulk document processing, embedding a backlog, offline enrichment, these don't need real-time serving, and real-time serving is what you're paying a premium for.

Most providers offer a batch tier that trades latency (results within hours instead of seconds) for roughly 50% off. OpenAI's Batch API, for example, offers a 50% cost discount versus the synchronous API with a 24-hour turnaround. Sort your workload into latency tiers, send everything that can tolerate delay through the batch path, and you halve the cost of that entire category for free. The only discipline required is being honest about which workloads are actually urgent. Usually fewer than you'd assume.

Monitor Cost Per Request Like You Monitor Latency

You wouldn't run a production service without latency and error dashboards. AI cost deserves the same treatment, in real time, with alerts, because the failure mode is financial and it moves fast.

A working AI FinOps setup tracks:

Cost per request, broken down by feature, model, and customer. Aggregate spend hides the problem; the breakdown shows you which feature's unit economics are underwater.
Token consumption trends, so you see usage climbing before the invoice does.
Cost anomaly alerts that fire when cost per request or total spend jumps, ideally tied to a budget threshold that can throttle or page someone.
Retry and failure costs, because failed requests still burn tokens. A feature with a high retry rate is quietly paying multiple times for one successful answer.

Tag every request with its feature and model so you can attribute cost precisely. When a number spikes, you want to know exactly which feature did it within minutes, not reconstruct it from a monthly bill three weeks later.

This Is FinOps, Not Roadmap Budgeting

It's worth being explicit about the distinction, because the two get conflated and they demand different work.

Roadmap budgeting asks "how much should we invest in AI this quarter?" It's a planning conversation about where to place bets, and it happens a few times a year in a room with leadership.

What we're describing here is FinOps, the ongoing operational discipline of managing unit economics so that spend scales sanely with usage. It lives in the architecture and the dashboards, not the planning deck. You can have a perfectly reasonable roadmap budget and still get destroyed by an invoice, because the budget set the ceiling and the unit economics decided whether you'd blow through it. Predictability comes from the FinOps layer: caching, routing, batching, and monitoring, baked into how the system runs every day.

Where to Start This Quarter

You don't need all of this at once. Sequence it by return:

Instrument cost per request first. Tag requests by feature and model and get the number on a dashboard. You can't manage what you can't see, and this alone often surfaces the biggest culprit.
Turn on prompt caching for your stable prompt prefixes. Highest return, lowest effort, no quality tradeoff.
Add model routing for your highest-volume routine traffic, backed by evals proving the cheaper model holds quality.
Move non-urgent workloads to batch for the roughly 50% discount.
Wire up anomaly alerts tied to budget thresholds, so you hear about a cost spike from your monitoring, never from your invoice.

At Particle41, we build AI systems with cost as a first-class architectural constraint, not a postmortem. Caching, routing, batching, and per-request monitoring go in alongside the feature, so the bill scales predictably as usage grows. Pairing senior engineers with AI-augmented delivery, we get teams to a place where they can state their cost per request, forecast it under load, and stop being surprised by the invoice.

The teams that keep AI costs predictable aren't the ones with the biggest budgets. They're the ones who treat the token as the unit of economics and engineer accordingly. Predictability isn't something you negotiate once a year. It's something you build.

Sources

Prompt caching with Claude — Anthropic — confirms prompt caching reduces costs by up to 90% and latency by up to 85% for long prompts, and lists per-model input/output token pricing.
Batch API — OpenAI — confirms the Batch API offers a 50% cost discount versus the synchronous API with a 24-hour turnaround.
OpenAI API Pricing — OpenAI — current per-token pricing across model tiers.

Originally published at particle41.com

What Is the Model Context Protocol, and Why Are CTOs Paying Attention?

Ben Johnson — Sat, 24 Oct 2026 14:09:04 +0000

You're standing up AI across your organization. The assistant needs to read your CRM, query your database, pull from your document store, and create tickets in your project tracker. So your team writes an integration for each one, wired into whichever AI platform you happened to start with.

Now picture the bill when you want to switch model providers, or run a second one alongside the first. Every one of those integrations was built against a specific vendor's way of doing things. You don't get to bring them along. You rebuild from scratch.

This is the quiet tax on AI adoption, and it's the problem the Model Context Protocol exists to kill. If you're a CTO, MCP isn't a niche technical detail. It's a decision about who controls the layer between your AI and your data, and that decision compounds for years.

What MCP Actually Is

The Model Context Protocol is an open standard, introduced by Anthropic in November 2024 and adopted broadly across the industry since, that defines a common way for AI models to connect to external tools and data sources.

The analogy that's stuck is USB-C, and it's the one the protocol's own documentation uses: "Just as USB-C provides a standardized way to connect electronic devices, MCP provides a standardized way to connect AI applications to external systems." Before USB-C, every device had its own connector and its own cable, and you needed an adapter for everything. USB-C standardized the interface, so any compliant device talks to any compliant port. MCP does the same for AI integrations: one protocol that any model can speak and any tool can expose.

Concretely, MCP defines two roles:

MCP servers wrap a data source or tool (your database, your CRM, a file system, an internal API) and expose its capabilities through the standard protocol.
MCP clients live inside the AI application and connect to those servers, so the model can discover and call the available tools.

Write an MCP server for your internal knowledge base once, and any MCP-compatible AI application can use it. The integration is no longer welded to a single vendor's platform. It's a reusable, portable component.

The Problem It Solves: The N times M Integration Mess

Before a standard exists, integration cost grows multiplicatively. As Anthropic put it when launching MCP, "every new data source requires its own custom implementation, making truly connected systems difficult to scale." If you have M AI applications and N data sources, the naive world makes you build and maintain up to N times M custom connectors. Five AI tools and ten data sources is potentially fifty bespoke integrations, each with its own auth, its own quirks, its own maintenance burden.

A standard collapses that to N plus M. Build one MCP server per data source, and every MCP-compatible AI application can use all of them. Ten data sources plus five tools is fifteen components instead of fifty, and the new sixth AI tool costs zero new integrations because it speaks the same protocol.

This is the whole reason CTOs are paying attention. It's not that MCP is technically elegant. It's that integration sprawl is one of the largest hidden costs in enterprise AI, and a standard is the only thing that bends the curve.

Why It Matters for Vendor Lock-In

Here's the strategic core. Without a standard, your AI integrations are an asset you can't take with you. Every connector you build deepens your dependence on the platform you built it against. The switching cost climbs with every integration, until changing providers becomes effectively impossible, and your vendor knows it.

MCP changes the ownership of that asset. Because the protocol is open and vendor-neutral, the integrations you build are portable. Your MCP server for your customer database works whether the AI on the other end is from one provider this year and another next year. You can:

Run multiple models behind the same set of integrations and route work to whichever performs best.
Swap providers without rebuilding your connection layer, turning a year-long migration into a configuration change.
Negotiate from strength, because your vendor knows you can leave.

You're no longer building on rented land. The most expensive, slowest-to-build part of your AI stack, the integrations, becomes yours.

Why It Matters for Security

A pile of bespoke integrations is also a security nightmare. Each one handles authentication its own way, logs differently or not at all, and grants access through its own ad hoc rules. Auditing that mess is nearly impossible, and every custom connector is a custom attack surface.

A standard interface gives you a standard place to enforce policy. With MCP, you can centralize the questions that matter:

Authentication and authorization: a consistent way to control which models can reach which tools, and what they're allowed to do once connected.
Scoping and least privilege: expose only the specific capabilities a model needs, rather than handing it broad credentials to an entire system.
Audit logging: a uniform record of what the AI accessed and what actions it took, which is exactly what your compliance and security teams will demand.

The standard doesn't make security automatic, you still have to configure it correctly, but it gives you one well-defined boundary to harden instead of a dozen inconsistent ones. That's a meaningful reduction in attack surface and a real improvement in auditability.

What This Doesn't Solve

MCP is plumbing, and plumbing has limits. Be clear-eyed about what it isn't.

It doesn't make your data AI-ready. If your knowledge base is a mess of stale, contradictory documents, MCP gives the model a clean pipe to a dirty source. Garbage in, garbage out, just over a standard protocol.

It doesn't handle governance for you. The protocol provides the interface to enforce access control and auditing; you still have to decide the policies and implement them. A standard connector with no auth configured is still wide open.

And it doesn't eliminate engineering. Writing a robust MCP server for a complex internal system, with proper error handling, rate limiting, and security, is real work. MCP standardizes the shape of that work; it doesn't make it free. The win is that you do it once and reuse it, not that you skip it.

What to Do Now

You don't need to rip anything out. You need to make MCP a deliberate part of your architecture posture before integration debt piles up.

Set MCP as the default for new AI integrations. Any new connection between a model and a data source should go through an MCP server unless there's a specific reason not to. Stop minting vendor-locked connectors.
Inventory your existing integrations and rank them by how locked-in and how strategic they are. The ones that are both deeply coupled to a single vendor and central to your operations are your highest-priority candidates to re-platform onto MCP.
Stand up MCP servers for your highest-value data sources first, the systems your AI will touch most: your primary database, your document store, your CRM. Get the reuse where it pays off fastest.
Centralize security at the MCP boundary. Define your auth, scoping, and audit standards once, and apply them consistently across every server.
Build the muscle now, while your AI footprint is small and the cost of getting it right is low. Retrofitting a standard onto fifty existing integrations is far more painful than adopting it from the start.

At Particle41, we treat the integration layer as an architecture decision, not an afterthought. We build MCP servers for client systems with security and governance designed in from the first line, so the connections your AI depends on are portable, auditable, and yours. Pairing senior engineers with AI-augmented delivery, we get that integration layer standing in weeks instead of quarters, and we make sure it isn't quietly handing your leverage to a single vendor.

The CTOs paying attention to MCP aren't excited about a protocol. They've recognized that whoever controls the integration layer controls the AI strategy, and they intend for that to be themselves. The right time to make that call is before you've built the integrations you can't take with you.

Sources

Introducing the Model Context Protocol (Nov 25, 2024) — Anthropic — the official announcement of MCP as an open standard, the server/client architecture, and the problem that "every new data source requires its own custom implementation."
What is the Model Context Protocol (MCP)? — Official MCP documentation — source of the "USB-C port for AI applications" analogy and the description of MCP as an open standard supported across a broad ecosystem of clients and servers.

Originally published at particle41.com

How Do You Evaluate Whether an AI Feature Is Good Enough to Ship?

Ben Johnson — Fri, 23 Oct 2026 14:11:46 +0000

You built an AI feature. It demos beautifully. The stakeholders are impressed, the question becomes "when can we ship," and everyone is feeling good. That feeling is the most dangerous moment in the entire project.

A demo is a curated highlight reel. You picked the inputs, you saw the good outputs, and the room saw three examples that worked. Real users will throw thousands of inputs at it, including the weird, the adversarial, and the boringly off-distribution. The gap between "looked great in the demo" and "works for actual users" is where AI features go to die in public.

Shipping an AI feature isn't a yes/no judgment call. It's a measurement. Here's how to replace the gut-feel decision with one you can defend with numbers.

Start With an Eval Set, Not a Demo

The single highest-leverage thing you can build is an evaluation set: a collection of representative inputs paired with known-good expected outputs. This is your AI feature's test suite, and without it you are flying blind.

A real eval set has a few properties:

Size: at least 100 examples to start, scaling to 500-1,000+ for anything customer-facing. Too few and your numbers are noise.
Coverage: pull from real or realistic usage. Include the easy happy path, but deliberately load it with edge cases, ambiguous inputs, adversarial prompts, and the categories where you suspect the model is weak.
Ground truth: each example needs an expected answer or a clear rubric for what "good" means. This is the expensive part, and it's the part you cannot skip.

Build the eval set before you tune the prompt, not after. If you build it after, you'll unconsciously construct cases your current system already passes, and your 95% score will be a lie. The eval set is the bar; you don't get to move it to wherever your feature already cleared.

Offline Evals: Catch Regressions Before Users Do

Offline evaluation means running your full eval set against the feature in a controlled environment, every time you change something. New prompt, new model version, new retrieval logic, new anything: rerun the evals and compare scores.

This matters because AI systems are maddeningly non-local. You tweak a prompt to fix one category of failure and silently break three others. Without an offline eval running on every change, you discover that breakage from an angry customer instead of a failing test.

Treat offline evals like a CI gate. A change that drops your accuracy below threshold doesn't merge, the same way a change that breaks a unit test doesn't merge. The score becomes a hard signal in your pipeline, not a thing someone remembers to check sometimes.

The metrics depend on the task. Classification gives you precision, recall, and F1. Extraction gives you field-level accuracy. Open-ended generation needs rubric-based scoring, which is where LLM-as-judge comes in.

LLM-as-Judge: Scoring at Scale

You can't have a human read 1,000 generated outputs after every prompt tweak. So you use a strong model to grade the outputs of your feature against a rubric. This is LLM-as-judge, and it's how teams evaluate open-ended generation at scale. The approach has research backing: in the benchmark study that popularized it, strong judge models like GPT-4 matched human preferences with over 80% agreement, the same level of agreement humans reach with each other.

It works, but only if you respect its limits:

Calibrate it against humans first. Have humans score a sample of 50-100 outputs, then check that the judge model agrees with them. If the judge and your humans disagree more than 10-15% of the time, your rubric is too vague or the judge is the wrong tool for this task.
Give it a concrete rubric, not "is this good?" Score specific dimensions: factual accuracy, completeness, tone, format compliance. Force a structured verdict.
Watch for known biases. Judge models favor longer answers, get swayed by answer position, and prefer outputs in their own style — documented as verbosity, position, and self-enhancement biases. Account for it.

Used carefully, LLM-as-judge lets you score thousands of outputs for a few dollars and turn subjective quality into a trackable number.

Human Review: Still Irreplaceable

Automated evals tell you what's happening at scale. Humans tell you what the numbers miss. You need both, and you need to be deliberate about where humans spend their attention.

Use human review to:

Calibrate your automated judges, as above.
Audit the failure cases your evals flag, to understand whether a failure is catastrophic or merely cosmetic.
Catch the unknown unknowns, the failure modes you didn't think to put in the eval set. Real human eyes on real outputs surface categories no rubric anticipated.
Spot-check production traffic continuously after launch, not just before it.

The smart move is to focus human attention on the low-confidence and high-stakes slices, not on random sampling of everything. Let automation clear the obvious passes and fails, and put your expensive human judgment where the decisions are hard.

Set the Acceptance Threshold Before You Measure

Here is the discipline almost everyone skips. Decide what "good enough" means as a number, in advance, and tie it to the cost of being wrong.

"92% accuracy on the eval set with zero catastrophic failures" is an acceptance threshold. "It feels solid" is not. The threshold depends entirely on the stakes:

A marketing copy suggester where a human always reviews output before use might ship at 80%, because the human is the safety net.
A customer-facing support agent answering policy questions might need 95%+ with strict guardrails, because a wrong answer reaches a customer directly.
A medical or financial decision aid might need 99%+ on safety-critical categories, with hard rules backstopping the model.

The key is separating quality dimensions from safety dimensions. You might tolerate 85% on "answer is maximally helpful" while demanding 99.9% on "answer never gives dangerous advice." Set a threshold for each, and define which failures are blockers versus which are acceptable.

If you don't write the threshold down first, you will rationalize whatever number you got into "good enough," because the pressure to ship is enormous. Pre-committing to the bar is what protects you from yourself.

Guardrails: Plan for the Failures You Can't Eliminate

No AI feature hits 100%. So the question isn't only "how good is the model," it's "what happens when the model is wrong." Guardrails are the layer that contains failures the model will inevitably produce.

Effective guardrails include:

Input validation: reject or reroute prompt injection, out-of-scope requests, and malformed inputs before they reach the model.
Output validation: check generated output against rules. Schema validation for structured output, profanity and PII filters, fact-checking against a source of truth, refusal detection.
Confidence-based routing: when the model signals low confidence, escalate to a human or a fallback path instead of guessing.
Rate and scope limits: contain the blast radius if something goes wrong, so a single bad pattern can't cascade.

Guardrails let you ship a feature that's 92% accurate by ensuring the 8% fails safely instead of loudly.

Your Pre-Launch Checklist

Before any AI feature ships, you should be able to answer yes to all of these:

Do we have an eval set of 100+ representative cases with ground truth, built before tuning?
Do offline evals run automatically on every change, gating regressions like a test suite?
Is our LLM-as-judge calibrated against human scores to within 10-15%?
Have humans audited the failure cases and confirmed none are catastrophic?
Did we set acceptance thresholds in advance, separating quality from safety, and are we meeting them?
Are guardrails in place so the inevitable failures fail safely?
Do we have online evals and monitoring ready to catch the drift production will bring?

If you can't answer yes to all seven, you're not evaluating whether to ship. You're hoping.

At Particle41, eval-first development is how we build AI features that survive contact with real users. We stand up the eval harness and guardrails alongside the feature itself, so the shipping decision is a number on a dashboard rather than a feeling in a conference room. Pairing senior engineers with AI-augmented delivery, we get teams from "the demo looked great" to "we know exactly how often this works, and what happens when it doesn't."

A demo proves your AI feature can work. An eval set proves how often it does. Only one of those is safe to ship on.

Sources

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., NeurIPS 2023) — arXiv — establishes that strong LLM judges reach over 80% agreement with human preferences (the same level humans reach with each other) and documents position, verbosity, and self-enhancement biases.

Originally published at particle41.com

What Is RAG, and When Do You Actually Need It Instead of Fine-Tuning?

Ben Johnson — Thu, 22 Oct 2026 14:30:40 +0000

You've decided to put a large language model to work on your own data. Your first instinct is probably "let's fine-tune it on our documents." Hold on. That instinct is wrong most of the time, and acting on it will cost you six weeks and a five-figure training bill before you discover it didn't solve your problem.

The confusion is understandable. Fine-tuning and retrieval-augmented generation (RAG) both "make the model know your stuff." But they work in completely different ways, fail in completely different ways, and are good at completely different things.

Here's the one sentence that should guide every decision: fine-tuning changes how a model behaves; RAG changes what a model knows. Get that distinction wrong and you'll burn budget teaching a model facts it forgets the moment your data updates.

What RAG Actually Does

RAG is mechanically simple. When a user asks a question, you don't just send the question to the model. First you search a store of your own content for the most relevant chunks, then you paste those chunks into the prompt alongside the question, and you tell the model to answer using that context.

The model itself never changes. It stays frozen. All you're doing is feeding it better context at the moment of the question. The plumbing looks like this:

Ingest: split your documents into chunks of roughly 200-500 tokens.
Embed: convert each chunk into a vector using an embedding model (roughly $0.02 per million tokens for a small embedding model as of 2026, so embedding a few hundred thousand documents costs tens of dollars, not thousands).
Store: put those vectors in a vector database like Postgres with pgvector, Pinecone, Weaviate, or Qdrant.
Retrieve: at query time, embed the question, find the closest chunks by similarity, and inject the top 3-10 into the prompt.

That's it. The model reasons over fresh context every time. Update a document, re-embed that one chunk, and the next answer reflects the change instantly. No retraining, no GPUs, no waiting.

What Fine-Tuning Actually Does

Fine-tuning takes a pre-trained model and continues training it on your examples, nudging its weights so it behaves differently. You're not handing it facts to recite; you're reshaping its defaults.

Fine-tuning shines when you need the model to:

Adopt a consistent voice or format every time, without you spelling it out in the prompt.
Perform a narrow, repetitive task like classifying support tickets into one of 40 categories, or extracting structured fields from messy text.
Follow domain conventions that are hard to express in instructions, like a specific legal citation style or a clinical note structure.
Reduce per-call cost by baking behavior in, so you can use a smaller, cheaper model with shorter prompts.

The catch is the cost and the cadence. A real fine-tuning effort means curating hundreds to thousands of clean examples, running the training job (often thousands to tens of thousands of dollars depending on model size and data volume), and evaluating the result. And the moment your underlying knowledge changes, the fine-tuned facts are stale. You'd have to do it all again.

That's the trap. Fine-tuning a model to memorize facts is like baking a cake to store a phone number. It works, technically, and it's the wrong container.

The Third Option Everyone Forgets: Long Context

Before you commit to either, consider doing nothing fancy at all. Modern models accept 200K tokens of context, and some go to a million or more. If your entire knowledge base fits in the context window, you can just paste it in.

This is the right call when your corpus is small and stable, say a single 60-page policy handbook. No vector database, no pipeline, no training. Just stuff the document into the prompt.

The limits show up fast, though:

Cost scales with every token, every call. At roughly $3 per million input tokens for a mid-tier model (Claude Sonnet pricing as of 2026), sending 150K tokens of context on every request is about $0.45 per query before the model says a word. Run that 50,000 times a month and you're at $22,500 in input cost alone.
Recall degrades in the middle. Models reliably use information at the start and end of a long context but get lossy in the middle, the well-documented "lost in the middle" effect.
It doesn't scale past a few documents. A thousand-document knowledge base will never fit, and you'd be paying to re-read all of it on every single question.

Long context is a great prototype and a poor production strategy once your data grows past a handful of documents or your query volume gets real.

The Decision Criteria That Actually Matter

Forget the hype and run your problem through five questions.

Does your data change often? If knowledge updates weekly, daily, or in real time, RAG wins by default. Fine-tuning bakes a snapshot; RAG reads live. Pricing, inventory, policy, and ticket history all change too fast to retrain around.

Do you need citations? RAG knows exactly which chunks it pulled, so you can show the source behind every answer. This is non-negotiable in legal, healthcare, finance, and anywhere a wrong answer has consequences. A fine-tuned model gives you fluent confidence with no traceable source, which is worse than useless when accuracy is auditable.

Is the problem knowledge or behavior? "The model doesn't know our return policy" is a knowledge problem, so RAG. "The model writes in the wrong tone and ignores our format" is a behavior problem, so fine-tuning. Write down which one you actually have before you build anything.

How much data, and how clean? RAG handles millions of documents and tolerates messy, unstructured content. Fine-tuning needs curated, labeled, high-quality examples, and quality matters more than quantity. A few hundred excellent examples beat ten thousand noisy ones.

What's your latency and cost budget? RAG adds a retrieval step (typically 50-200ms) and some input tokens per call. Fine-tuning front-loads the cost into training and can lower per-call cost afterward. For most teams, RAG's marginal cost per query is the safer bet.

The Mistakes We See Most

Fine-tuning to inject facts. This is the big one. Teams spend weeks fine-tuning a model on their documentation, then watch it confidently hallucinate details and go stale within a month. Facts belong in retrieval, not in weights.

Treating RAG as set-and-forget. RAG quality lives or dies on retrieval quality. If the right chunk never gets retrieved, the model can't use it, and your "AI is hallucinating" problem is actually a search problem. Chunking strategy, embedding model choice, and reranking matter more than the LLM itself.

Ignoring chunk boundaries. Splitting a document mid-sentence or mid-table destroys meaning. Chunk on semantic boundaries, keep some overlap, and preserve structure like headers and tables.

Skipping the reranker. Pure vector similarity returns "topically near" chunks, not always "actually relevant" ones. A lightweight reranking step that re-scores the top candidates often lifts answer quality meaningfully for a few extra milliseconds.

Assuming it's either/or. The strongest production systems use both. Fine-tune for behavior and format, then layer RAG on top for fresh knowledge. They're not competitors; they solve different halves of the problem.

How to Decide This Week

Start with the cheapest thing that could work and earn your way to complexity:

Prototype with long context. Paste your documents into the prompt and see if the answers are good enough. If your corpus is small and stable, you might be done.
Graduate to RAG when data grows or changes. Build a basic retrieval pipeline, measure retrieval quality first (are the right chunks coming back?), then measure answer quality. Most "we need AI on our data" projects end here.
Reach for fine-tuning only for behavior. If, after RAG is working, the model still won't adopt your tone, format, or a narrow task reliably, then fine-tune for that specific gap. Layer it on top of RAG, not instead of it.

At Particle41, this is the first conversation we have when a team says they want a model trained on their data. More often than not, we steer them away from a costly fine-tuning run toward a retrieval system that ships in a fraction of the time and stays current automatically. Pairing senior engineers with AI-augmented delivery, we get a working RAG pipeline in front of users in weeks, then tune from real usage instead of guesses.

The takeaway is blunt: if you're about to fine-tune a model so it "knows your data," stop and ask whether you have a knowledge problem or a behavior problem. Knowledge is retrieval. Behavior is fine-tuning. Most of the time, you want retrieval, and you want it now.

Sources

Originally published at particle41.com