How to Test AI That Never Gives the Same Answer Twice

Deshan Ranchhod — Wed, 28 Oct 2026 14:25:20 +0000

Most software tests work on a simple idea. You give the system an input, and you check that you get the right output. If the answer matches, the test passes. If it doesn't, something broke.

AI doesn't play by that rule. Ask the same question twice and you may get two different answers. Both might be fine. One might be wrong in a way that's hard to spot. This is one of the first things that catches teams out when they move an AI feature from a demo into daily use.

Here's how I think about testing AI features, in plain terms.

1. Test for "good enough", not "exactly right"

You can't check for one fixed answer. You need to check for qualities instead. Does the answer use the right facts? Does it stay on topic? Does it follow the format the user needs? Does it avoid things it should never say?

Write these down as a short list of checks for each feature. This list becomes your test, not a single expected reply.

2. Build a sample set from real work

Collect a set of real questions or tasks your users actually bring. Include the easy ones, the tricky ones and the odd ones. Keep the set small enough that a person can review the results, then grow it over time.

Every time the AI gets something wrong in real use, add that case to the set. This way your tests get better as the product does.

3. Run each test more than once

Because answers change from run to run, one pass tells you very little. Run the same case several times and look at the spread. If it gets it right every time, great. If it gets it right most of the time, you need to decide if that's good enough for this task. For a draft email, it might be. For a change going into a live SAP system, it isn't.

4. Match the checks to the risk

Not every AI feature needs the same level of care. A feature that suggests a summary can have lighter checks. A feature that takes an action, or helps someone approve one, needs much tighter checks and a person in the loop.

I find it useful to sort features by what happens if the AI is wrong. The worse the outcome, the stricter the test and the more human review you keep.

5. Use people and tools together

Automated checks are fast and catch a lot. They can flag missing fields, wrong formats or banned words. But some things only a person can judge, like whether an answer is actually helpful.

Set up a simple review habit. Have someone who knows the work look at a sample of results on a regular basis. Keep the review light so it actually happens.

6. Keep a record

Log what the AI was asked, what it said and what happened next. When something goes wrong, this is how you find out why. In large companies, this record also matters for audit. People need to see what the AI did, just as they would for a person.

7. Re-test when anything changes

A new model version, a change to your prompts or new data can all shift how the AI behaves. Treat each of these like a code change. Run your sample set again before it goes live.

The bottom line

Testing AI is less about proving it's perfect and more about knowing how it behaves. Clear checks, real examples, repeat runs and human review will get you most of the way. That's what lets people trust an AI feature enough to use it every day.

I'm Deshan Ranchhod, an AI Product Manager in Melbourne. You can find more of my work at deshanranchhod.com, on LinkedIn, on Medium and on my blog.

From SAP Consulting to AI Product Management: What AI Agents Need to Get Right in SAP Change

Deshan Ranchhod — Mon, 26 Oct 2026 01:43:14 +0000

For years my job was to help SAP changes reach production safely. I was a senior SAP technical consultant across APAC. In September 2026 I moved into a new role as an AI Product Manager at Rev-Trac, which makes SAP change and release software.

The move has made me look at old problems with fresh eyes. These are my own views, not a product plan. But I think they hold for anyone building AI tools for SAP teams.

Where I'm coming from

SAP change work is careful work. A small change in one system can break a process somewhere else. So teams review transports, test, check what else a change might touch, and write release notes. Then someone signs off.

A lot of that work is slow and repetitive. It is also high stakes. You can't skip steps just because you are busy. That mix is exactly where AI agents look useful. It is also where they can do the most harm if we get them wrong.

Where AI agents could help

Here are three places I think agents could take real weight off SAP teams.

Reviewing transports. An agent could read what is in a transport and flag things a person should look at. Missing objects. Odd sequencing. Changes that clash with other work in flight. It would not approve anything. It would point a human at the right spot faster.

Impact analysis. When someone asks "what will this change touch?", the answer often lives in many places. An agent could pull that together into a short list of likely risks. The person reviewing it still decides what matters.

Release notes. Writing clear notes from technical changes is tedious. An agent could draft them in plain words. A person then checks and fixes the draft before it goes out.

In each case the agent does the legwork. A person keeps the judgement.

What an AI PM has to get right

Moving into product, I have learnt that the clever part is not the hard part. The hard part is making the tool safe and useful every day. Four things stand out to me.

1. Evaluation

You can't improve what you don't test. For an AI feature, that means building a set of real examples with known good answers. Then you run the agent against them every time something changes. A new model, a new prompt or a new data source can all shift the results.

I like to start small. Pick one task, such as drafting release notes. Collect real cases. Agree with users on what "good" looks like. Then measure against that, not against a demo that worked once.

2. Guardrails

An agent should only be able to do what it needs to do. If its job is to read and suggest, it should not be able to move a transport. Limits like this should be built in, not left to good behaviour.

Guardrails also cover what the agent says. It should show where its answer came from. It should say when it is not sure. A confident wrong answer is worse than "I don't know".

3. Human sign-off

In SAP change, sign-off is not red tape. It is how an organisation stays in control and passes audits. AI should make sign-off easier, not remove it.

So I think the agent should prepare the work and the person should approve it. The approval should be clear and on the record. If something goes wrong, you need to know who decided what, and why.

4. Trust

Trust is earned slowly and lost fast. SAP teams have seen plenty of tools promise a lot. They will judge an AI feature on its worst day, not its best.

That means being honest about limits. Start with low-risk tasks. Let users see the agent's reasoning and correct it. Listen to their feedback and act on it. Trust grows when people feel the tool works for them, not the other way round.

What I'd tell other SAP people

If you work in SAP and wonder where AI fits, your knowledge matters more than ever. Someone has to know what a safe change looks like. Someone has to spot when the agent is wrong. That someone is you.

You don't need to become a data scientist. Learn how these tools fail. Learn how to test them. Ask hard questions of anyone who says their tool "just works".

For me, the shift from consulting to product has been less about new tech and more about new questions. Not "can we build this?" but "should we, and how will people know they can rely on it?" That is the work I want to do well.

Deshan Ranchhod is an AI Product Manager in Melbourne, Australia. Connect with him on LinkedIn or read more on his blog. Back at uni he built a kebab shop finder app, and SBS Food wrote about Deshan Ranchhod's Get Me HSP app.

DEV Community: Deshan Ranchhod