Why “almost right” is more expensive in production than obviously wrong

A red build doesn’t get shipped. An answer that sounds convincing but is factually wrong does. That difference captures the fundamental distinction between traditional software and AI systems. And it is why quality assurance is moving from the final step to the first.

AI writes code. AI summarizes tickets. AI suggests test cases. What it cannot reliably determine is whether a system has truly been understood, whether a result can be trusted, or whether a workflow that shines in a demo will fall apart tomorrow in real-world use. This is exactly where a new career opportunity is emerging for Quality Engineers.

“Almost right” is worse than obviously wrong

In many teams, the conversation currently revolves around productivity: implementing faster, writing less boilerplate, delivering more features per sprint. The tools are already here. According to the Stack Overflow Developer Survey 2025, 84 percent of respondents use or plan to use AI tools in their development process. The second figure from the same survey is even more interesting: more developers distrust the accuracy of AI output than trust it. 46 percent actively distrust it, while only 33 percent express trust. And 66 percent cite the same frustration: AI solutions that are almost right, but not quite.

That “almost” is the point.

A red build stops itself. A plausibly worded false statement reaches the customer. An agent that performs brilliantly in a demo but exposes confidential information in the third edge case becomes a liability. A workflow that succeeds 95 percent of the time may still cause costs, legal exposure or reputational damage in the wrong business scenario. Obviously wrong gets noticed. Almost right slips through.

That changes the guiding question of software development. It is no longer: Who can generate things fastest? It becomes: Who can demonstrate that what has been generated is actually correct?

Job postings for AI Engineers already describe Quality Engineering – they just don’t call it that

At first glance, current AI Engineer job postings read like traditional engineering roles: APIs, data pipelines, cloud, MLOps, RAG, agents, LLMs, observability. But between the lines, something else becomes clear. Companies are looking for people who can make quality manageable.

On StepStone, AI roles are appearing in which model training, evaluation, production-ready services, result validation, AI governance, data protection and understanding of the EU AI Act are listed in the same job description. One current AI Security role explicitly mentions guardrails, input and output validation, prompt injection protection, logging, monitoring, anomaly detection, penetration testing and AI red teaming. And explicit QA roles for LLM and agentic AI systems already exist, requiring outputs to be validated for correctness and consistency.

Internationally, the pattern is the same. An AI Engineer role at Onit lists “Evaluation, Quality & Iteration” as a core responsibility: evaluation frameworks, golden datasets, adversarial cases, regression tests, rubric-based scoring, release gates, online metrics, safety flags, LLM-as-Judge, observability, documented failure modes and fallback paths.

That is not just development. It is Quality Engineering under a new name.

The edge case matters more than the demo

Real-world cases already show how quickly AI systems can behave differently in everyday use than they do in concept documents.

Air Canada was held responsible in 2024 for misleading information provided by its chatbot (Moffatt v. Air Canada, 2024 BCCRT 149). A customer had relied on incorrect information about retroactively applying for a bereavement fare. The tribunal made it clear that Air Canada was responsible for the accuracy of the information provided by its chatbot. The answer was not absurd. It was fluent, helpful and wrong: exactly the kind of error that does not turn a build red.

DPD had to disable parts of its AI chatbot after a user persuaded the bot to criticize its own company, swear and describe itself as useless. The company said a system update had triggered the unusual behavior. An update. The kind of change that, in traditional software, would have triggered a regression test suite.

The third case became an internet classic: a Chevrolet dealership made headlines after its chatbot, given the right prompts, appeared to agree to sell a 2024 Chevy Tahoe for one dollar. Absurd, yes – but the lesson is serious. A system that communicates with customers needs boundaries, a clear understanding of its role and validated outputs.

Before I tested software, I worked in a kitchen. My head chef at the time had a saying that I still quote in every seminar: “You can’t think stupid enough to predict how people will behave.” What he meant was that you cannot predict what people will do with your product. Since I started testing AI systems, I have added a follow-up question: How stupid can things get when AI starts helping them?

These cases are not just footnotes in AI folklore. They reveal the same underlying problem: AI systems do not fail because of syntax errors. They fail because of context, intent, permissions, hallucinations, prompt injection, unclear responsibilities and missing validation. Quality assurance was invented for exactly these kinds of failures.

In the AI era, the order changes: test first, then build

Software development was long organized around building. Gather requirements, write code, add tests afterwards. Quality mattered, but it came late – usually once the product was already largely defined.

With traditional applications, that was manageable because many errors are deterministic: input A produces output B. With LLMs and agents, that is no longer the case. In its own Evaluation Best Practices, OpenAI points out that generative AI is variable and that traditional testing methods alone are not sufficient for AI architectures. Evals are described as structured tests for measuring accuracy, performance and reliability despite nondeterminism. Recommendations include eval-driven development, task-specific evals, logging, automation and calibrating automated evaluations using human feedback.

This is a cultural shift, and you can hear it in the sentences teams use when talking about their products.

Before After
“We build first and test later.” “We first define how we will know whether the system is behaving correctly.”
“The demo was convincing.” “Which counterexamples did we test?”
“The prompt works for me.” “What regressions occur when the model, prompt, context, data source or tool permissions change?”

The right-hand column is the language of quality assurance. It now belongs at the beginning of the process, not at the end.

This also aligns with findings from Google’s DORA research, which describes AI in software development as an amplifier: it magnifies the strengths of high-performing organizations, but also the dysfunctions of weak ones. In other words, AI makes good engineering practices more valuable, not less.

Both sides need to move – and QA’s head start will not last forever

For developers, the message is uncomfortable: anyone who defines their value primarily through implementation speed is coming under pressure. AI reduces the cost of producing things while simultaneously increasing the demands around evaluation, architecture, security and traceability.

That does not make the profession less technical. Quite the opposite. The questions are becoming deeper:

  • How do I build evaluation suites for RAG systems?
  • How do I detect hallucinations that sound plausible?
  • How do I test an agent that is allowed to call tools?
  • How do I prevent a model from using more permissions than the user should have?
  • How do I build regression gates for prompts, models and data sources?
  • How do I make failures visible before customers find them?

Anyone serious about AI Engineering needs QA thinking as a core competency. Not as an add-on, not as a “nice to have”, but as a survival skill.

Conversely, this is a major opportunity for Quality Engineers. But that opportunity is not guaranteed. QA brings something many teams urgently need right now: professional skepticism. Not cynicism, not obstruction. The ability to deliberately use a system differently from how it was intended. To question a technical statement even when it is phrased fluently. To make failures reproducible, document risks and turn individual incidents into robust testing strategies. That has always been valuable. Now it is becoming central.

But that head start will only last if QA develops its technical capabilities. Anyone who wants to become an AI Quality Engineer tomorrow needs to understand today how LLM applications are built: APIs, JSON schemas, function calling, RAG, embeddings, vector databases, prompt versioning, CI/CD, observability, tracing, data protection, security, model changes, cost metrics and fallback designs.

PyRIT shows how far the discipline has already evolved: Microsoft provides a framework for automated and human-led AI red teaming that supports multi-stage attack strategies, standardized evaluation scenarios, data leakage testing, different target systems and flexible scoring methods. The new QA does not work with Excel spreadsheets and a few manual exploratory testing sessions. It works with frameworks, pipelines, telemetry and attack simulations.

The Quality Engineer of tomorrow is not the person who takes one last look at the product before release. They design the safety nets that make it possible for AI to be used in production at all.

What this means in practice

For developers: Don’t just build features – build demonstrable correctness. Every AI feature needs an answer to the question: “How do we know it works under real-world conditions?” That answer consists of golden datasets, counterexamples, prompt regressions, model comparisons, logging, metrics, fallbacks, human-in-the-loop mechanisms and clear boundaries for tool usage.

For quality assurance: Move beyond pure test execution and into AI Quality Architecture. Learn how an agent makes decisions. Learn how RAG retrieves the wrong sources. Learn why prompt injection is not simply bad input, but an attack. Learn to automate evals and critically calibrate LLM-as-Judge systems. And learn to discuss APIs, data flows, permissions and observability with developers on equal technical footing.

For companies: Stop treating QA as the final station before release. Bring Quality Engineers in at the beginning: into architecture, product definition, risk analysis, model and tool selection, and the question of which decisions may be automated – and which must not.

The standards make clear that this is about architecture, not cosmetics. The OWASP Top 10 for LLM and GenAI Applications include Prompt Injection, Sensitive Information Disclosure, Improper Output Handling, Excessive Agency and System Prompt Leakage among the key risk categories. And the NIST AI Risk Management Framework describes trustworthy AI as valid and reliable, safe, resilient, transparent, explainable, privacy-enhanced and fair. These properties do not emerge through hope. They emerge through systematic evaluation.

The uncomfortable truth

The traditional developer does not lose their profession because AI generates code. They lose value if they fail to learn how to critically evaluate AI-generated results.

The traditional tester does not lose their profession because AI generates test cases. They lose value if they fail to learn how to technically understand, instrument and harden modern AI systems.

The future does not belong to those who can give AI tasks the fastest. It belongs to those who notice when an answer sounds dangerously convincing.

That is why the most important developer of the next few years may not be the person writing the most code today. It may be the person who asks the one question in the sprint review that nobody wants to hear:

“How do we know this is still correct when the user doesn’t do what we expect?”

For a long time, that question was the hallmark of quality assurance. Now it is becoming a core competency of software development.

My head chef would have put it more simply. Then again, he didn’t have AI to help him. QA never stood only for Quality Assurance. QA always also meant: Question Asker.

Both paths can be learned. At oose eG in Hamburg, relevant courses include the ISTQB® Certified Tester AI Testing (CT-AI), Testing with Generative AI (CT-GenAI) and Fundamentals of AI Security.

If you are just getting started: ISTQB® Certified Tester Foundation Level (CTFL). If you want to evolve your role within the team: The Quality Evangelist and RiskStorming.