AI in software testing means using machine learning and language models to plan, write, run and repair tests, and also to check products that contain AI themselves. The first use is already available in mainstream tools: Playwright ships three test agents, mabl sells agents that create, run, analyze and maintain tests, and Applitools applies visual AI to rendered screens.
None of this removes the tester. The work shifts from typing scripts toward reviewing what an agent proposes, and from fixing broken locators toward asking why a test failed. For a wider view, read this take on AI transforming software testing and QA, then use the sections below as a practical checklist.
What does AI in software testing cover?
Two different things share the name. The first is AI as a helper: agents and models that assist with planning, writing, running and maintaining tests. The second is testing AI systems, where the product itself is a model or an assistant.
The ISTQB describes the second area in its Certified Tester AI Testing syllabus, which covers machine learning and generative AI systems, including large language models, and their probabilistic, non-deterministic behavior. Its separate CT-GenAI certification focuses on using generative AI inside the testing process.
Keep the two apart when you plan a budget, because the tools, skills and risks are different.
Which testing tasks can AI agents take over?
Playwright’s documentation lists three agents that can work alone or in sequence:
- The planner explores the app and writes a Markdown test plan.
- The generator turns that plan into Playwright test files and verifies selectors and assertions live.
- The healer runs the suite and repairs failing tests.
The planner needs a clear request, such as a plan for guest checkout, and a seed test that prepares the environment. A product requirements document is optional context.
Setup is light. Teams add the agent definitions to a project with the init-agents command and regenerate them whenever Playwright updates, and the docs list VS Code, Claude Code, Codex and OpenCode as supported environments.
mabl describes a similar division of labor, with agents that create, run, analyze and maintain tests across web, mobile and APIs. It frames the work in four stages: build, validate, investigate and release. In the investigate stage, agents identify what changed and what needs attention so developers and coding agents can fix problems and revalidate faster.
How does AI reduce test maintenance?
Broken tests are the usual tax on automation. When a test fails, Playwright’s healer replays the failing steps and inspects the current interface for equivalent elements. It then suggests a patch, such as a locator update, a wait adjustment or a data fix, and reruns the test until it passes or until guardrails stop the loop.
mabl lists AI auto-healing in its platform and says its agents keep tests current as software changes. QAlified’s guide to codeless automation makes the same point from the other side: tests built without code still need maintenance, and self-healing is the capability to look for.
Where does visual AI help?
A functional assertion can confirm that a button exists without saying whether anyone can read it or reach it. Applitools Eyes compares rendered screens with baselines and offers four match levels: Dynamic, Strict, Layout and Exact.
It also checks color contrast ratios, font sizes and layout overlap against WCAG standards on every visual run, and it connects to Playwright, Cypress, Selenium, WebdriverIO and Appium. Applitools says it integrates natively into more than 50 test frameworks and that visual checks can be added to an existing suite with one or two lines of code, which makes it easier to bolt onto current tests than to replace them.
How do you test the AI itself?
A model that answers differently each time cannot be checked with a single expected value. QAlified’s LLM testing guide proposes a layered plan. Unit tests feed the model one input and check, for example, that a summary names the right company and adds no unsupported claims. Regression tests compare new results with a fixed baseline, because fine-tuning, retraining or a prompt change can alter quality without any code change.
Further layers cover harmful behavior such as toxicity and bias, performance under load and security tests for prompt injection and leaks. For open-ended answers, a second model can act as judge on criteria like clarity, helpfulness and tone.
What still needs people?
Generated work needs review. Playwright’s own documentation says generated tests may include initial errors, and its healer marks a test as skipped when it believes the functionality itself is broken. A person has to decide whether the fault sits in the test or in the product.
Adoption also has a learning curve. QAlified’s consulting service for AI in testing starts by assessing the team’s current state, then identifies automation opportunities and trains people on specialized tools. Treat every agent output like a pull request: read it before it merges.