AI in software testing has long been more than just a bit of ChatGPT for formulating test cases. We use it to generate test cases, analyze test results, and classify defects. Sometimes it works well; other times, not so much. This is nothing new for testers. Verifying work results is one of the cornerstones of our work, and of course we do the same for AI-generated content. The question is simply how much we actually verify, how much we trust the generated content, and how much capacity we actually have to take a closer look at the sheer volume of rapidly generated content.

This is exactly where the three concepts—Human-in-the-Loop, Human-on-the-Loop, and Human-out-of-the-Loop—come into play. They describe how much responsibility remains with humans—and how much we leave to AI.

Human-in-the-Loop: Humans Make the Decision

In the “Human-in-the-Loop” approach, the AI provides suggestions. A human reviews them and then makes the decision. While this may not sound very spectacular at first, it’s exactly the right approach for critical testing tasks.

An example
An AI derives several test cases from a requirement to reset a password:

  • valid email address,
  • invalid confirmation code,
  • expired code,
  • too many reset attempts.

That’s a good start. But does the list also cover security aspects? What happens if a user account is deactivated? Is there a rate-limit check? Could error messages potentially be misused to inflate user counts? A tester reviews the suggestions, adds any missing scenarios, and decides which test cases will actually be included. Core competencies such as critical thinking are more essential here than ever. So the AI helps with the preparation. Technical responsibility remains with humans. Incidentally, this applies to all three approaches. A human is responsible for the results.

Typical areas of application:

  • final release approvals,
  • security-critical tests,
  • evaluation of critical defects,
  • acceptance of AI-generated test cases.

Human-on-the-Loop: Human Oversight

In a human-on-the-loop setup, the AI operates largely autonomously. We no longer check every single action, but instead monitor the process and aggregated results. We intervene when something goes awry. This is particularly useful when there are large volumes of recurring test tasks. No one wants to manually verify every single automated regression test. We haven’t wanted to do that in test automation so far, and we certainly don’t want to start doing it now. However, no one should simply look the other way if the results suddenly change significantly—that’s something we’ve already been doing with test suites.

An example
After each build, an AI automatically prioritizes and runs the most important regression tests. During execution, it detects that the error rate has risen significantly and stops the pipeline.

This is where the tester comes in. They examine the affected tests and determine whether the issue is:

  • a genuine product defect,
  • an infrastructure problem,
  • outdated test data,
  • or a faulty test.

So the AI handles monitoring and responds to anomalies. A human doesn’t have to be constantly present, but should be able to intervene at any time—at least during defined working hours, provided I’m not on call. This requires clear rules: At what thresholds are alerts triggered? When is a pipeline paused? Who is notified? And how can we determine why the AI reacted in the first place?

Typical applications:

  • Regression testing,
  • Performance testing,
  • Visual comparisons,
  • Monitoring of test pipelines,
  • Anomaly detection.

Human-out-of-the-Loop: Humans Perform Audits

In a human-out-of-the-loop scenario, the process runs completely autonomously. Humans do not intervene during execution but review the results later—for example, through audits, spot checks, or, in the worst case, through error tickets from the production environment. This offers the greatest degree of freedom for AI tasks and is therefore not suitable for every testing task. However, it can be quite useful for clearly defined, low-risk tests.

An example
After each deployment, an AI automatically checks whether:

  • the application is accessible,
  • the login works,
  • central APIs respond,
  • the database connection is established.

The results are documented automatically. The team regularly conducts random spot checks to verify that these tests are still functioning correctly. In this case, therefore, humans do not decide on every single result. Rather, they ensure that the automated process remains reliable overall. However, this requires a number of conditions: The task must be clearly defined, the consequences of an error must remain manageable, and all decisions must be traceable later on. In addition, an emergency mechanism is needed to quickly halt the AI processes. After all, who wants Skynet?

Typical applications:

  • simple smoke tests,
  • technical availability checks,
  • standardized format checks,
  • low-risk test data cleansing.

TL;DR:

  • Human-in-the-Loop: The AI makes suggestions; the human makes the decision.
  • Human-on-the-Loop: The AI performs the task; the human monitors it.
  • Human-out-of-the-Loop: The AI operates autonomously; the human performs post-processing checks.

The crucial question is not: “How much can AI handle?” nor “How many testers can AI replace?” but rather:

“Which decisions is AI actually allowed to make on its own in this specific testing process?”

The greater the risk of making a wrong decision, the more closely a person should be involved in the decision-making process. In testing, AI does not simply replace humans. It shifts their role: away from manual execution, toward decision-making, monitoring, and auditing automated and AI-supported testing processes.