Test Automation

Test Automation Maintenance: How to Keep Automated Tests Reliable at Scale

JIN

Nov 17, 2026

Table of contents

Table of contents

    Most automated test suites do not fail on the day they are written. They fail slowly, usually in the second or third year, when a few hundred tests have grown into several thousand, and the team spends more time keeping the suite green than learning anything from it.

    The data backs this up. In mabl’s 2026 State of Quality Engineering Report, a survey of 996 software professionals in the US and UK found that 41% named test maintenance as their top testing pain point. It was the leading answer for the second year in a row, up six points from 2024. The report adds a point every team adopting AI should note: as new tests get faster and cheaper to generate, the maintenance load grows right along with them.

    Most advice online treats maintenance in one of two ways. Either it is the unavoidable tax you pay for automation, or it is a problem the next self-healing tool will make disappear. Neither holds up well in practice. Our argument is simpler: most of a test’s maintenance cost is decided when the test is designed and when the suite’s architecture is chosen. After that, a team is either doing cheap routine care or expensive rescue work.

    Why Automation Maintenance Becomes Difficult

    An automated test is a frozen description of how the product behaved on the day someone wrote it. The product keeps moving. Every release changes some screens, some APIs, and some data rules, and each change gives a test a chance to drift away from reality. That part is obvious. What is less obvious is why two teams facing the same release can end up with very different repair bills.

    The real driver is coupling, not change

    The usual explanation is “the application changes too often.” Change is the trigger, but coupling sets the price. A test that checks “a signed-in customer can place an order” survives a button restyle. A test that clicks the third div inside a checkout panel with a specific CSS class does not. Spread that difference across 3,000 tests and the same design refresh produces either a handful of fixes or a lost sprint.

    A practical way to think about it: maintenance load is roughly the number of tests that touch a changed area, multiplied by how tightly each one is tied to implementation details. Teams cannot control the first number much, because products change when the business needs them to. They have a lot of control over the second.

    Test code rarely gets treated as real code

    Production code gets code review, refactoring, style rules, and clear ownership. Test code often gets none of these. Helpers get copied instead of shared, waits get added to “make it pass,” and nobody touches a test until it breaks. Software Engineering at Google, written by Google engineers about their own practice, spends a full chapter on this. Its authors describe brittle tests, the ones that break after harmless changes that introduce no real bug, as a drag that grows as the codebase grows.

    Nobody owns the suite

    In many organizations, a separate automation team writes the tests while product teams change the code. When a test breaks, the people who changed the behavior are not the people who get the alert. Failures sit in a queue, then get rerun, then get skipped. Ownership gaps never show up on a dashboard, but they explain many suites that went from trusted to ignored within a year.

    AI-generated tests add volume faster than teams add care

    Writing tests used to be the slow part. With AI assistants, it no longer is. The same mabl report shows where the bottleneck moved: 33% of respondents say manual review of AI-generated tests is already their top barrier to scaling. When asked what would hold them back if AI grew their suites tenfold, 21% pointed to manually fixing tests that break. More tests written faster means more tests to keep alive. Volume without design discipline only pushes the cost into next year.

    Common Maintenance Problems

    Five problems account for most of the repair work in enterprise suites. They overlap, but each has its own symptoms and its own first fix.

    UI Changes

    UI tests break most often because they describe the page as a developer built it, not as a user sees it. Selectors tied to CSS classes, deep XPath chains, or element positions fail whenever a design system is updated, a component library is swapped, or a layout is rearranged. Martin Fowler made this point years ago in his note on the Page Object pattern: tests that rely on HTML elements become brittle with any UI change. The fix starts with locators based on roles, labels, and visible text, plus dedicated test IDs where those are not enough, all kept in one place so a single UI change means a single edit.

    Test Data

    Shared test accounts, hard-coded order numbers, and expiring data (promo codes, trial periods, dates that pass) cause a large share of failures that look random. One test changes a user’s settings, and a different test that relies on the old settings fails an hour later. Environments also drift: someone refreshes a staging database, and half the suite loses its data. The lasting fix is to let each test create the data it needs through APIs or factories, use unique values per run, and clean up afterward. Masked or synthetic data also keeps the suite clear of privacy rules that apply to production records.

    Dependencies

    Every test depends on more than the application. It depends on browsers and drivers, framework versions, third-party APIs, payment sandboxes, identity providers, and the CI runners themselves. A browser auto-update or a sandbox outage can turn a healthy suite red overnight with no product change at all. Good teams pin versions and upgrade on purpose with tools like Renovate or Dependabot, replace unreliable external services with stubs or virtual services in most runs, and use contract testing to confirm the real integration still matches what the stubs assume.

    Flaky Tests

    A flaky test passes and fails on the same code. From a maintenance point of view, the damage is less about any single failure and more about the triage time each one burns and the trust it erodes. Once engineers learn that red often means nothing, they start rerunning instead of reading. We cover causes, detection, and fixes in depth in our guide to flaky tests and how to fix them. For maintenance planning, the key point is to treat flakiness as tracked debt: every flaky test gets an owner, a ticket, and a deadline, not an extra retry.

    Technical Debt

    Test suites collect the same debt as production code. Duplicated login helpers in twelve files, fixed sleep() calls, tests with fifteen assertions that check five unrelated things, commented-out blocks, and “temporarily” skipped tests that have been skipped for two years. None of these fail loudly. They make every future change slower and every failure harder to read, which is why debt tends to show up as rising repair time rather than as a single broken build.

    Looking at this table as five separate problems is a little misleading. In most suites we audit, they share one cause: tests know too much about how the system is built and too little about what it is supposed to do. A test that depends on layout also tends to depend on shared data and fixed waits, because it was written by recording clicks rather than by stating intent. Fixing the design removes several symptoms at once, which is why the next sections spend more time on design than on repair.

    Test Automation Maintenance Process

    Good design reduces maintenance. It does not remove it. Teams that keep large suites reliable run maintenance as a routine with clear rules, not as a cleanup project when things get bad.

    1. Triage every failure within a fixed window

    Classify each red result on the main branch within a set time, often one working day. Five buckets cover nearly everything: product defect, test defect, environment issue, data issue, or flaky. Classification matters more than speed alone because it tells you where the next fix should go. A suite where most failures land in “environment” needs infrastructure work, not more test fixes.

    2. Give every test an owner

    Ownership works best when it follows the feature, not the tool. The team that owns the checkout service owns the checkout tests, including the end-to-end ones. A central automation group can still own the framework, shared libraries, and pipeline. This split puts the alert in front of the people who changed the behavior, which shortens repair time more than any tool can.

    3. Quarantine with an expiry date

    Quarantine, moving a known-unstable test out of the blocking gate while it keeps running and reporting, is useful. Quarantine without a deadline is just a polite way to delete coverage. Set a limit, for example, ten working days, after which the test must be fixed, rewritten, or formally retired with the coverage gap noted.

    4. Fix causes, not symptoms

    The fastest fixes are often the most harmful: a longer timeout, an added sleep(), a retry wrapped around the whole test. Each one turns a visible problem into a hidden one. A proper fix names what the test was waiting for or depending on and makes that explicit, such as waiting for a network response or creating the data the test assumed would exist.

    5. Update tests in the same pull request as the change

    If a feature change breaks a test, the test update belongs in the same pull request. Adding this to the team’s definition of done prevents the slow pile-up where the product and the suite drift apart for weeks before anyone notices.

    6. Schedule small, regular hygiene

    Many guides recommend a dedicated “test cleanup sprint.” In practice, cleanup sprints tend to fail for a simple reason: the habits that created the debt are still in place, so it returns within a quarter. A small standing budget works better. Many teams reserve a fixed slice of automation capacity each sprint for dependency upgrades, pruning skipped tests, and refactoring the worst offenders on the health report.

    The thread through all six steps is speed of feedback to the right person. Most maintenance cost is not the fix itself. It is the time a broken test spends unowned, the reruns it triggers, and the trust it drains while it waits.

    Designing Maintainable Automation

    If the process controls how fast problems get fixed, design controls how many problems appear in the first place. These principles apply across Selenium, Playwright, Cypress, Appium, and API frameworks alike.

    Test behavior, not structure

    A test should describe what the system does for a user or a caller, not how it does it internally. Software Engineering at Google puts this as testing through public APIs and aiming for tests that do not need to change unless the requirements change. For UI tests, the “public API” is what the user sees and does. For service tests, it is the published contract. Anything below that line is an implementation detail the test should not depend on.

    Choose locators that match how users see the page

    Modern frameworks make this easier than it used to be. Playwright’s best practices guide recommends user-facing attributes such as roles and accessible names over CSS or XPath selectors, and its locators wait automatically until elements are ready.

    When no user-facing attribute is reliable, a dedicated data-testid acts as an explicit contract between the product and the suite. A side benefit is that role-based locators fail when accessibility labels disappear, which is often a real defect worth catching.

    Give every test its own data

    Tests that create their own data can run in any order and in parallel, and they do not break when someone else’s test changes a shared record. Use API calls or data factories in setup, generate unique values per run, and clean up afterward. This makes tests slightly longer to write and much cheaper to keep.

    Wait for conditions, never for time

    A fixed wait is a guess about how long something will take. It is either too short on a slow day or wasted time on a fast one. Wait for the specific thing that signals readiness: an element state, a network response, a message on a queue. Most modern tools support this directly, so a fixed wait in a code review should be treated as a defect.

    Keep each test readable on its own

    When a test fails at 2 a.m., the person reading it should understand its purpose without opening five helper files. Google’s engineers summarize this as “DAMP, not DRY”: a little repetition inside test bodies is fine if it makes each test clear, while shared setup and mechanics belong in helpers. One clear purpose per test, a descriptive name, and a failure message that says what was expected all cut triage time.

    Build testability into the product

    Some of the best maintenance work happens in the product code. Stable test IDs, API endpoints for seeding data in test environments, feature flags that can be set per test, and clear loading states all make tests simpler. This requires developers and QA engineers to plan together during design, which is where a shift-left approach pays off in maintenance, not just earlier bug detection.

    Automation Architecture

    Design principles help each test. Architecture decides how the whole suite responds to change. A good architecture means one product change leads to one edit in one place.

    Separate what you test from how you reach it

    The ISTQB Test Automation Engineering syllabus describes a generic test automation architecture (gTAA) with distinct layers for generating, defining, executing, and adapting tests. You do not need to follow it to the letter, but the core idea holds for any stack: keep test intent separate from the technical details of talking to the system.

    The rule that makes this work: each layer only talks to the one below it. If a scenario file contains a CSS selector or a raw HTTP call, the layers have leaked and a UI change will spread into dozens of files.

    Make page and component objects the only home for UI details

    The Page Object pattern is still the most practical way to contain UI changes. For modern component-based front ends, modeling reusable components (a date picker, a data table, a modal) works better than one large object per page. Keep assertions in the tests rather than inside the objects, so each object stays a simple, reusable map of the interface.

    Put each check at the lowest layer that can catch the bug

    This is the biggest architectural lever, and the one most often skipped. Larger tests are slower, more expensive to run, and far more likely to be unstable. Google’s own data makes the gap clear: over one week, 0.5% of its small tests were flaky, compared with 1.6% of medium tests and 14% of large tests. The same book recommends a rough mix of about 80% unit tests and 20% broader tests as a starting point.

    The practical rule is to ask, for every new end-to-end test, whether an API or integration test could catch the same defect. Business rules, validation logic, and calculations usually can. Save the UI layer for journeys where the UI itself is the risk.

    Control environments and dependencies as code

    Environments that are created by hand drift over time. Tools like Testcontainers let tests start real databases, queues, and services in disposable containers, so each run begins from a known state. Combined with pinned dependency versions and stubbed third-party services, this removes most of the “red build with no code change” failures that consume triage time.

    Design the pipeline for fast, targeted feedback

    As suites grow, running everything on every commit stops being practical. Tag tests by risk and purpose (smoke, regression, nightly), split long suites across parallel runners, and run only the tests affected by a change where your tooling supports it. A suite that gives an answer in ten minutes gets attention. One that takes three hours gets ignored, and ignored failures are where maintenance debt grows fastest.

    Architecture is where the “maintenance is decided early” argument is most visible. A team that puts business rules in API tests, UI details in component objects, and environments in code will still have maintenance work. It will just be small, local, and predictable.

    Measuring Automation Health

    Most teams report pass rate and test count. Both can look excellent while the suite is failing at its actual job. A 98% pass rate means little if failures are rerun until they pass, and a high test count says nothing about whether those tests catch real defects. The metrics below give a more honest picture.

    There is no reliable industry benchmark for most of these, and any article that quotes one with confidence deserves some doubt. Suites differ too much in size, layer mix, and product type. The useful approach is to track your own trend over time. A maintenance ratio that climbs quarter after quarter, or a quarantine list that only grows, is a stronger warning than any single number compared with another company.

    Two of these metrics deserve special attention. Test churn per change is the best early signal of design problems, because it shows coupling directly: if a small UI change touches forty test files, the architecture is leaking. Escaped defects are the metric that keeps everyone honest, because it measures whether the maintenance effort is protecting anything. A clean, stable suite that misses the bugs customers find is well maintained in the wrong direction.

    When to Rewrite vs Maintain a Test

    Online discussions usually frame this as a two-way choice. In practice, there is a third option that is often the right one: retire the test. Every automated test is a small liability that costs time on every run and every change, so it should earn its place.

    A simple test for retirement: ask the owner to describe, in one sentence, the defect this test would catch that no other test catches. If no one can, the test is costing time without adding protection. Retire it and record the decision, so the coverage gap is visible rather than accidental.

    Before rewriting, write down what the test protects

    Many rewrites quietly lose coverage because the original test’s intent was never written down. Before touching the code, record the business rule or user outcome the test protects. Then rebuild it at the lowest layer that can check that outcome. Often a fragile end-to-end test becomes a short API test plus a much simpler UI check.

    Avoid rewriting the whole suite at once

    Framework migrations, such as moving from Selenium to Playwright, tempt teams into a full rewrite. Big-bang rewrites usually take longer than planned and leave a gap where neither suite is trusted. A safer pattern is to run old and new tests side by side, migrate the highest-value and most unstable areas first, and retire each old test only after its replacement has proven stable over a set number of runs. Work out the cost against the expected savings before committing; our guide on calculating test automation ROI covers how to frame that case.

    AI-Assisted Automation Maintenance

    AI is changing maintenance work, but less dramatically than vendor marketing suggests. The honest picture is that AI is good at the tedious parts of maintenance and still needs people for the parts that require judgment.

    Where AI helps today

    • Failure triage: Grouping hundreds of failures by likely cause, separating environment noise from probable product defects, and summarizing logs and screenshots so a person reviews five clusters instead of two hundred red tests.
    • Locator repair: Detecting that an element moved or was renamed and proposing an updated locator, which handles a large share of routine UI breakage.
    • Change-aware updates: Reading a code diff and suggesting which tests need updating, or drafting the updates for review in the same pull request.
    • Suite analysis: Finding duplicate tests, overlapping coverage, and tests that have not caught anything in a long time, which feeds directly into retire decisions.

    What the data says about autonomy

    The mabl survey gives a useful reality check. Only 11% of teams can fix broken tests without human involvement. The rest split between AI suggestions that need human approval (35%), semi-autonomous healing that needs a manual restart (29%), and people rewriting tests by hand (25%). On the wider adoption side, Capgemini, Sogeti, and OpenText’s World Quality Report 2025-26 found that only 15% of organizations had scaled generative AI in quality engineering across the enterprise. For most teams, AI is an assistant in the maintenance loop, not a replacement.

    The risk of a test that heals itself into uselessness

    Self-healing has a failure mode that rarely appears in product demos. When a tool “fixes” a test by finding a different element or relaxing a check, the test can keep passing while it no longer verifies what it was written to verify. The mabl report raises this directly, warning that healed tests can pass while losing fidelity to their original purpose. Picture a “Place order” button that moved behind a menu after a redesign. A healed locator finds it, the test passes, and nobody notices that real users can no longer find the button. The test is green, and the defect is real.

    Guardrails that keep AI repairs honest

    • Heal locators, never assertions: Let AI update how a test finds an element, but treat any change to what the test checks as a human decision.
    • Log and review every heal: Each automatic repair should appear in a reviewable change, not happen silently at runtime.
    • Track heal rate as a health metric: A rising heal rate is a signal, either of heavy UI churn or of weak locators that need fixing at the source.
    • Keep intent next to the test: A one-line description of what each test protects gives both people and AI tools a reference point for judging whether a repair keeps the test meaningful.
    • Review AI-generated tests like production code: Generated tests add to the maintenance load from the day they merge, so they should meet the same design rules as hand-written ones.

    Used this way, AI lowers the cost of routine maintenance without hiding real problems. For a broader view of how teams move from scripted automation toward AI-supervised quality, see our AI quality engineering maturity model and our earlier piece on self-healing test automation.

    Know What Your Test Suite Is Really Costing You

    SHIFT ASIA’s automation engineers in Vietnam review your suite against the measures in this article: flake rate, repair time, test churn, quarantine age, and AI healing logs. You receive a ranked list of tests to keep, repair, rewrite, or retire, along with a maintenance process your own team can run. Our work follows the quality methodology of our parent company, SHIFT Inc. in Japan, and combines Playwright, Selenium, and API automation experience with AI-assisted triage and repair, always keeping a human in the review loop. You get the depth of a Japanese QA partner at the cost of an offshore team.

    Share this article

    ContactContact

    Stay in touch with Us

    What our Clients are saying

    • We asked Shift Asia for a skillful Ruby resource to work with our team in a big and long-term project in Fintech. And we're happy with provided resource on technical skill, performance, communication, and attitude. Beside that, the customer service is also a good point that should be mentioned.

      FPT Software

    • Quick turnaround, SHIFT ASIA supplied us with the resources and solutions needed to develop a feature for a file management functionality. Also, great partnership as they accommodated our requirements on the testing as well to make sure we have zero defect before launching it.

      Jienie Lab ASIA

    • Their comprehensive test cases and efficient system updates impressed us the most. Security concerns were solved, system update and quality assurance service improved the platform and its performance.

      XENON HOLDINGS