Software Testing Strategy: How to Decide What to Test When AI Writes the Code

Most software testing strategies fail quietly. The document is written at kickoff, approved, and filed away. Six months later, the team runs a regression suite that nobody chose intentionally, and the strategy still describes a product that has changed three times since then.

That gap is expensive. CISQ estimated the cost of poor software quality in the US at no less than $2.41 trillion in 2022, and one of the report’s sponsors noted that locating and fixing bugs is the highest single cost in the software development lifecycle. Most of that cost doesn’t come from teams skipping tests. It comes from spending testing effort where the product can’t actually fail.

AI-assisted development makes this harder. Code now arrives faster than people can read it, so skilled human attention is the scarcest resource on a delivery team. A software testing strategy is where you decide how to spend it. This article treats the strategy as what it really is: a short record of decisions about scope, risk, and evidence that you review every release. If you want the broader case for why quality engineering matters more once AI enters the pipeline, we covered that in Quality Engineering in the AI Era.

What Is a Software Testing Strategy?

A software testing strategy is a documented set of decisions about how to test a product to meet its quality goals. It defines objectives, scope, testing levels and types, environments, test data, automation, roles, defect handling, metrics, and release criteria. In AI-assisted projects, it should also specify how AI-generated code is reviewed, who signs off on AI-written tests, and how limited human review time is allocated across risks.

The ISTQB glossary describes a test strategy as a description of how testing will be carried out to meet test objectives in a particular context. We find it more useful to think of it as a list of answered questions. What are we protecting? What evidence do we need before release? Where will we deliberately test less? A strategy that can’t answer that last question isn’t a strategy yet.

Test strategy vs testing process

The testing process consists of activities such as planning, analysis, design, implementation, execution, and completion, as detailed in ISO/IEC/IEEE 29119-2. The strategy determines how to prioritize these activities for a specific product. Even with identical processes, different products require distinct strategies because their failure modes differ.

Test strategy vs test plan

A practical guideline: if a statement remains valid next quarter, include it in the strategy. If it references a date or individual, place it in the plan.

Organization-level vs project-level strategy

ISO/IEC/IEEE 29119-3 separates the organizational test strategy from the project test strategy. The organizational level sets rules every team follows: standard tools, defect-severity definitions, and minimum evidence for regulated systems. The project level adapts those rules to one product’s risks.

AI adds new work to the organizational level. Rules about AI usage, such as which code AI may generate, what review it needs, and whether AI-generated tests count toward release evidence, should be set once for the whole company. Otherwise, every project team invents its own, and your quality bar ends up depending on which team built the feature.

What Should a Software Testing Strategy Include?

Each component of a strategy records a decision. The third column shows the question AI-assisted delivery adds.

Reviewing the middle column reveals that most elements concern ownership, evidence, and boundaries, rather than test execution. These areas are most affected by AI. While tools can generate test cases efficiently, they cannot determine which evidence is valid or assign accountability. These decisions must be defined in the strategy.

How to Build a Software Testing Strategy

1. Understand Business and Technical Risks

Start with a risk workshop that includes product owners, architects, developers, support leads, and QA. Ask what would hurt the business most if it broke, and what is most likely to break. Our risk-based testing guide walks through the full scoring method so that we won’t repeat it here.

For AI-assisted projects, add two inputs most risk registers miss: code provenance and change volume. A module with mostly recent AI-generated changes and only light review has a higher chance of failure than its complexity alone would suggest. The same goes for a module that has suddenly started receiving three times as many changes per sprint. Neither makes the module bad. Both make it less understood, and that’s a risk.

2. Define Quality Objectives

Translate top risks into measurable objectives. For example, “The app must be secure” is not testable, but “No high-severity findings against OWASP ASVS Level 2 on payment and authentication services” is. ISO/IEC 25010 provides a useful framework by categorizing quality into characteristics such as functional suitability, reliability, security, and maintainability. Select the most relevant characteristics for your product and assign specific targets.

If the product includes AI features, their objectives look different. They usually involve accuracy thresholds, consistency across repeated runs, and limits on harmful output. Our guide to testing generative AI applications covers how to set those.

3. Map Risks to Testing Activities

Each high-priority risk should map to at least one specific testing activity, and every major testing activity should correspond to a risk. This approach identifies both coverage gaps and unnecessary testing efforts.

Introduce an independence rule for the AI era. If an AI assistant writes both the code and its tests, undetected requirements may pass all tests. For high-risk areas, require that tests originate from a different source than the code, such as a human tester, a separate tool, or tests derived from requirements before coding.

4. Select Testing Levels

The standard advice is to push checks down to the cheapest level that can catch them.

AI changes this balance. Generated unit tests often confirm only what the code does, not what it should do. A high number of unit tests in AI-generated modules can create a false sense of security. For these modules, prioritize integration and contract tests, which validate behavior at system boundaries where isolated assumptions may fail.

5. Decide What to Automate

The fundamental principles remain: automate stable, frequently run checks with clear expected results, and reserve exploratory testing for new or changing features. However, advances such as self-healing locators and agent-driven execution have reduced the cost of maintaining UI automation. As a result, some workflows that were previously not cost-effective to automate are now viable.

The strategy should continue to assign a human owner for test assertions. Automated tests that self-repair may inadvertently remove critical checks.

6. Define Quality Gates

Gates transform the strategy from guidance to enforcement. Typical gates include commit, merge, and release gates, all of which are automated in the CI/CD pipeline. The strategy specifies the required rigor for each risk tier. Critical workflows may require contract tests, security scans, and a second reviewer, while low-risk areas may only need a smoke test.

DORA‘s 2024 research is useful here. It found AI adoption linked to lower delivery stability and concluded that basics like small batch sizes and solid testing still matter. A practical gate that follows from this is a size limit on AI-generated changes. Above a set number of changed lines, a pull request is split or gets a second human reviewer before it can merge.

7. Establish Metrics

Select metrics that indicate risk reduction, such as defect escape rate, change failure rate, time to restore service, and the proportion of critical and high-risk areas with current, passing tests. Additionally, track review load, defined as the number of changes awaiting skilled review and their wait times. An increasing review backlog suggests code is being released without adequate review, regardless of test results.

These seven steps result in a concise document, intentionally limited to a few pages. A brief strategy is more likely to be reviewed and used by the team before each release.

Risk-Based Testing Strategy Example

Example application

Consider a B2B expense management platform where employees photograph receipts, an AI model extracts amounts, managers approve claims, and payments are processed via a banking API. The team releases every two weeks, with approximately half of recent backend changes generated by AI coding assistants. Two senior QA engineers have about six days per release for review and exploratory testing. This available review time, rather than the number of test cases, is the primary constraint.

Identify critical workflows

The team identifies seven key workflows: SSO login, receipt capture and AI extraction, expense submission, approval routing, reimbursement payout, admin role permissions, and reporting with CSV export.

Assign risk

The team scores likelihood and impact from 1 to 5. The one change here is that the likelihood increases by 1 point when the most recent changes to a workflow were AI-generated.

Admin permissions often surprise teams, as they are rarely tested thoroughly and can change without notice. A single defect can expose all employee financial data. The AI provenance adjustment elevated this workflow to the highest risk tier.

Map risk to test coverage

Rather than using coverage percentages, the strategy specifies the requirements for each risk tier.

The receipt extraction feature has a dedicated objective: achieving minimum field-level accuracy, measured on a fixed set of labeled receipts each release. Accuracy that appears sufficient in a demonstration may decline as the model or prompts change.

Prioritize release testing

With six days of senior QA time, the team allocates approximately three days to the two critical workflows, two days to high-risk areas, and one day to all other tasks. If development is delayed, the plan specifies that medium-tier exploratory testing is reduced first, followed by high-tier exploratory time. Critical-tier review is always maintained. This pre-agreed approach eliminates last-minute negotiations.

Unlike a traditional risk matrix, this approach adds an extra column for AI involvement. Tracking where AI contributed changes the risk ranking and directs human review time to areas with the least team understanding.

Common Software Testing Strategy Mistakes

Testing everything equally

A strategy that does not prioritize risks allocates equal effort to low- and high-impact areas, such as settings pages and payment flows. This approach may seem thorough but ensures that the most critical areas receive insufficient attention.

Automation without strategy

Teams frequently automate the easiest tasks, resulting in lengthy, slow test suites that overlook critical workflows. Automation should be guided by risk ranking, not convenience.

Measuring test cases instead of risk

Test counts and pass rates reflect activity, not risk coverage. Reporting that “4,200 tests passed” does not inform leadership about the protection of critical workflows. Instead, report risk coverage and escaped defects, reserving test counts for internal team tracking.

Leaving testing until the end

This usually starts with the strategy itself. It gets written after development is underway, so testers never see requirements while they can still be changed. Bring QA into requirement reviews and ask for testable acceptance criteria before anyone writes code, whether that’s a person or an AI assistant.

No production feedback loop

A strategy that ignores production data relies on assumptions. Incidents, support tickets, and monitoring data reveal actual user issues. Review this information each release and adjust risk scores accordingly. A strategy that remains unchanged for a year is likely not in use.

How AI Changes Software Testing Strategy

This section doesn’t catalog the defects AI tends to introduce. We covered those in Testing AI-Generated Code: The QA Engineer’s New Blind Spot. This section focuses on what to add to the strategy document.

AI-generated code

Include code provenance as a risk factor. Perfect tracking is unnecessary; a pull request label or commit tag indicating significant AI involvement is sufficient to adjust likelihood scores and require stricter review where needed. Without this, the strategy cannot distinguish between code written by an experienced engineer and code assembled from AI suggestions.

Increased code volume

Increased code volume per sprint requires more verification without additional reviewers. GitClear‘s 2025 analysis of 211 million changed lines found that code blocks duplicating nearby code became eight times more frequent, and copy-pasted lines surpassed moved lines, indicating reduced refactoring. Duplicated logic allows defects to persist in multiple locations. Similarly, Google’s 2025 DORA research linked each 25% increase in AI adoption to a 7.2% decline in delivery stability. Therefore, the strategy should explicitly plan verification capacity, specifying available skilled review hours per release and their allocation.

AI-assisted test generation

Allow AI to draft tests, test data, and edge cases to save time. However, establish an independence rule for high-risk areas, as outlined in step 3, and determine whether AI-generated tests qualify as evidence for release. We recommend counting them only after a human has approved the assertions.

Human validation

Document which decisions require human judgment, regardless of tool capabilities. At a minimum, this includes approving assertions for critical workflows, accepting known defects at release, and signing off on releases. Including a concise sign-off matrix in the strategy helps ensure that human oversight is maintained.

AI-specific quality risks

If the product includes AI features, the strategy should incorporate testing types and objectives not found in traditional plans. At the strategy level, specify the frameworks you are aligning with, such as the NIST AI Risk Management Framework, ISO/IEC 42001, or the EU AI Act for high-risk systems, to ensure clarity for audits and future teams.

All five changes highlight the same issue: AI has reduced the cost of generating test cases while limiting the role of human judgment. Strategies designed for previous conditions now misallocate testing effort.

Find Out Where Your Verification Time Is Really Going

A test strategy review built on Japan-standard QA discipline, run by engineers who work with AI-assisted delivery every day.

Most teams we talk to don’t need more test cases. They need a clear view of which risks their current effort covers, which it misses, and how much AI-generated code is shipping with less review than anyone realized. SHIFT ASIA‘s test strategy review starts with the same risk workshop described above, adds code provenance and review capacity to the picture, and ends with a strategy your team can reread every release. It draws on SHIFT Group’s quality methodology from Japan, aligned with ISTQB, and it’s delivered by our engineering teams in Vietnam, who also build the gates, contract tests, and AI output checks the strategy calls for.

Talk to SHIFT ASIA about building a software testing strategy that fits how your code is actually written today.