As a system’s development progresses, it becomes more complex, making it harder to identify the scope of impact due to the addition or modification of functions. When the program is changed, this results in an increased risk of causing unintended effects on the system, known as regression.

Regression testing, a type of software testing, is essential to mitigate these risks. This type of testing is designed to detect regressions, ensuring that updates do not adversely affect the system’s existing functionality. 

In this article, we will introduce regression testing from the overview to its purpose, importance, and points to note when implementing it while incorporating the syllabus of the ISTQB (International Software Testing Qualifications Board)and real-world examples to provide a comprehensive overview.

What is regression testing?

Regression testing is a crucial aspect of software testing that verifies that changes to the program, such as adding or modifying functions, fixing bugs, etc., have not caused unintended impacts (regressions) on other programs. Meaning that if a change has been made, regression testing helps identify whether it has negatively impacted existing features.

The ISTQB Certified Tester Foundation Level Syllabus v4.0 describes regression testing as follows:

“Regression testing confirms that no adverse consequences have been caused by a change, including a fix that has already been confirmed tested. These adverse consequences could affect the same component where the change was made, other components in the same system, or even other connected systems. Regression testing may not be restricted to the test object itself but can also be related to the environment. It is advisable first to perform an impact analysis to optimize the extent of the regression testing. Impact analysis shows which parts of the “software could be affected.” 

In other words, regression testing detects unintended side effects caused by program changes.

Therefore, regression testing can encompass any test, either functional, non-functional, GUI, API, unit, integration, or system testing. Its importance is amplified in agile development environments where frequent changes are part of the development process.

The importance of regression testing increases as the system grows and its size and complexity increase.

Regression Testing vs. Retesting

In software testing, regression testing and retesting are two terms that are often misunderstood or used interchangeably. While both involve verifying the software after modifications, they serve distinct purposes and are used in different contexts.

Retesting is a type of testing conducted to verify whether a specific defect or issue has been resolved after it has been fixed. It is a more targeted approach, focusing only on the areas where the defect was initially detected. Retesting happens all the time at any level. We retest when a bug fix has been released or after performance tuning in a specific module.

When to perform Regression Testing

Regression testing is required in the following cases:

• When changes are made to an existing program due to the addition of new functions or changes to functions
• When a bug is found in the system, and modifications are made to the code to fix it
• When changes are made to the code due to refactoring, etc., to improve the program
• When the software in the system is upgraded
• When integrating existing systems with external systems

If code is added or changed due to the above cases, regression testing is necessary as it may unexpectedly impact the system.

Why we need Regression Testing

Regression testing is essential to detect unintended side effects and regression caused by program changes. While regression testing adds another step to the development process, neglecting it can lead to significant problems down the line. Here’s a closer look at the potential risks:

1. Improve system reliability

In today’s world, where systems tend to grow in size and complexity, and the scope of the impact of program changes is becoming increasingly difficult to see, the risk of regressions is always there. 

Imagine a seemingly minor code change causing a previously functional feature to break. Without regression testing, this issue might go unnoticed until much later, impacting the system’s overall reliability. Thus, it may have a ripple effect not only on those involved in the project inside and outside the company but also on users, leading to problems with the company’s credibility.

2. Reduce time & cost 

The earlier a defect is detected, the easier and cheaper it is to fix. Skipping regression testing allows regressions to linger, potentially requiring extensive debugging and rework later in the development cycle. This can significantly inflate development costs compared to catching regressions early.

An example: Imagine an eCommerce system where a seemingly minor code change disrupts the payment processing functionality. If not caught through regression testing, this regression could lead to lost sales and customer dissatisfaction. Fixing it later in the development cycle would be more expensive than detecting it early.

As an actual example, during an integration testing phase on a project, SHIFT ASIA identified numerous defects that originated from inadequate unit testing in the previous stage. These issues necessitated significant rework at the unit level, further introducing inconsistencies at the integration level. This highlights the domino effect of neglecting regression testing.

In the worst-case scenario, regressions might not be discovered until delivery or post-release operation. This can incur not only correction costs but also potential compensation for disruptions caused by the bugs.

As software progresses through the SDLC, the expense of resolving an issue increases at an exponential rate. For example, fixing a problem during the design phase may cost 1 unit. However, the same issue detected in the testing phase may cost 4 times more; in the worst-case scenario, it can cost up to 40 times more by delivery time.

Regression Testing process

1. Changes occur in the source code

If the source code is changed due to the addition of functions, corrections/changes, bug fixes, etc., it is necessary to consider implementing regression testing.

2. Identification of where changes have been made and the extent to which they are affected

Identify where source code changes have been made and the scope affected by those changes. These are related to the extent to which regression testing is required.

3. Select test cases that cover the scope of impact

It is not practical to test all areas affected by a change due to time, budget, human resources, etc., so you need to prioritize.

Select test cases to be used in regression testing by prioritizing them based on functional/non-functional importance to the system, business importance, urgency, frequency of use, high degree of dependence, etc.

4. Plan regression testing

Once the scope and test cases that require testing are clarified, the time and resources required for test preparation and execution will become clear, allowing you to plan the tests.

The plan also includes smoke tests to be conducted before regression tests and preparation of the test environment and test data.
It is also important to define the termination conditions for the test.

5. Prepare the test environment

Please note that if there is a large difference between the test environment and the latest production environment, the test will be meaningless. It is also important that the test environment is stable and has sufficient performance for testing.

If the test environment is unstable and performs poorly, you risk making tests take longer or giving you incorrect test results.

6. Run the test

Execute tests in a test environment based on the plan. It is better to execute test cases that are rarely executed manually, but you should consider automation for tests that are executed many times and whose test cases do not easily change.

Automating test cases that are repeated over and over again can provide significant benefits such as reduced cost, time, and increased accuracy.

7. Measure results

By measuring and analyzing regression test results, you can detect defects caused by regressions and obtain insights into quality. Test results are summarized in a report and shared with development and testing teams and stakeholders. The information will be utilized for future software development and testing.

If the testing reveals significant defects, consider additional testing.

8. Fix bugs

Fix defects detected by regression testing. After fixing a defect, always confirm that the defect has been fixed.

Efficiency regression testing approaches

While regression testing offers significant benefits, many projects struggle with resource constraints. Here are five key strategies to optimize your regression testing approach:

1. Prioritize according to importance

Not all functionalities carry the same weight. Focus your efforts on the most critical features and user flows. Regarding regression testing, the broader the scope, the more test cases, effort, and cost involved. Therefore, ensuring that important functions and flows work as specified and performing regression testing that targets the related scope is best.

While a “full regression testing” method performs a complete set of regression tests without limiting the scope, it should only be used in special cases, such as when a mission-critical system has many serious regressions. Prioritizing according to importance is effective for efficiency and cost-effectiveness.

For example, regression testing would focus on the most important parts of the system or business for an eCommerce system, such as member registration, login, product selection, and payment.

To perform regression testing efficiently, it is important to be selective and not include less essential parts in the test scope.

2. Incremental Execution, Continuous Improvement

ISTQB recommends that regression test suites be run repeatedly and expanded gradually. Start with a core set of regression tests and gradually expand them as new features are added. Since regression tests are repetitive, automation offers a significant advantage. You can build confidence in the system’s stability by repeatedly running regression tests and expanding the test cases alongside new features.

The “insecticide paradox,” one of the seven principles of software testing, states that repeating the same test many times is meaningless because its effectiveness gradually decreases. However, automated regression testing is beneficial because it reduces regressions.

Due to the nature of regression, even if something works fine at one point, it may not work properly at some point later. Therefore, repeating the test many times while expanding is believed to increase reliability.

3. Proper maintenance is key

Proper maintenance of regression testing is essential. As mentioned above, regression testing is an iterative process. Regression test suites are living documents that must be updated regularly to reflect changes in the program and ensure continued effectiveness.  

For example, let’s assume that an eCommerce system’s member registration function has been updated, and the information required for member registration has increased. In that case, the regression tests related to the member registration function must also be updated.

4. Perform at all test levels

ISTQB recommends performing regression testing at all test levels, including unit testing, integration testing, and system testing when updating the program or fixing defects.

Performing regression testing at each test level is ideal because even if a regression occurs, it is detected and fixed early. At the same time, the scope of impact is as limited as possible.

5. Start automating regression testing as early as possible

ISTQB recommends starting regression testing automation early in the project. Many regression tests require repeated execution, and continuing to execute manual tests forever is a heavy burden in terms of both cost and time. Therefore, it is recommended to automate regression testing, especially for regression tests that are repeated many times.

If regression testing is automated, the cost and time required for repeated test execution can be significantly reduced, making it easier to absorb the implementation and operation costs involved in automation. Running automated tests each time a build is performed can also help detect regressions early.

Automation is also effective for integration tests involving multiple functions and system tests that check a series of movements. 

However, manual testing performed by humans may still be more effective for tests that require ad hoc judgments that are difficult to convert into test cases or that are not repeated many times.

Designing your system with automation in mind can significantly streamline the automation process later.  This will lead to greater efficiency in testing and early detection of regressions, which brings great benefits.

For example, in a project where SHIFT ASIA was in charge of regression testing, tests were first designed and executed manually at each test level. After detecting and correcting defects, part of the test was redesigned for regression testing. Finally, it was automated and executed repeatedly with each build. As a result, no severe defects occurred after the product was released.

Summary

Regression is an ever-present threat in software development, potentially causing major issues. Regression testing is crucial for its detection. However, efficient execution is the key. To avoid unnecessary test scope expansion and excessive costs, prioritize ruthlessly and automate whenever possible.

You can effectively mitigate risks by performing regression testing at all levels and enabling early regression detection. The unpredictable nature of regressions underscores the importance of planning for regression testing and automation as early as possible during project conception. This proactive approach leads to better project management, systems stability, and improved software reliability and quality.

At SHIFT ASIA, we have extensive experience supporting clients across all aspects of regression testing – planning, design, execution, and automation. If you have any concerns or questions about regression testing, such as the appropriate way to perform it, allocating enough time for testing, or improving the efficiency of costs and labor, please contact us.