Beyond the Bug: A Software Testing Insights Report

As we embark on a new year, we often find ourselves welcoming back our customers after a short period of time, rambling on and on regarding the aftermath they have to endure once they have to part ways with us. It sounds somewhat like a compliment while fitting well into a complaint simultaneously, doesn’t it? Per Zendesk’s report in 2022, 60% of customers would stop doing business with a company after experiencing a bad incident. It’s not that the projects did not end well; it’s due to the fact that they had to be hands-on without anyone monitoring or supervising. The mutual feeling of insecurity is understated.

For records and future reference, we asked our customers to complete a survey as soon as we handed over our contract to evaluate whether our testing strategies, techniques, and planning were a good fit for their business. The findings were pretty interesting and are exactly the subject of today’s blog post. Without further ado, let’s get started!

The Project Industries Background

Forty nine clients agreed to participate in this survey and shared their project data with us. Our client projects span various sectors, including Healthcare, eCommerce, Finance, ERP, food and beverage, business tools, Entertainment, and more.

It might not seem obvious to others with no technical background to find testing an important indicator for their prosperity in the industry. Even industries not traditionally associated with technology, such as Education, increasingly rely on software solutions. Consider school administration, where automated attendance systems, parental communication platforms, and report card management software are now essential. Defects in these systems can lead to significant disruptions and inaccuracies, highlighting the universal need for robust software testing.

The “Most defective” industries

Our survey revealed some surprising insights into defect prevalence across industries. The Travel industry emerged as the most defect-prone, with a staggering 108.8% defect rate in a single project. We would not believe the statistics until we saw them in black and white. We ran over 3,490 test cases to yield 358 defects, mainly discovered by integration testing techniques. Poor responsiveness and inconsistent UI/UX elements created a disjointed customer experience.

The problem lies within the software’s predefined requirements. The final products did not meet expectations, especially when cross-device display was not considered. The layout was not responsive, and UI/UX elements were scattered and incohesive. These obstacles formed unhinged obstacles that made following the customer journey uncomfortable.

Following the Travel industry, the Construction sector ranked second with a 24.9% defect rate. Contrary to popular belief, this traditionally offline industry is increasingly reliant on digital platforms for lead generation. That was proven by the rise of GenZ and millennials, who took over the old population to become the main workforce. They eventually grow out of the traditions and hop on web surfing, Reddit, local constructions, and community groups on their social media platform. Website contact forms, emails, and DMs bring in the main sales leads.

Keeping the traditions of having the website for the sake of having one puts the construction industry far behind. Our customers contacted us to eliminate defects on their websites, apps, and systems, ensuring they capture any leads that might slip away without going unnoticed. Other industries with notable defect rates include Education (7.6%), Entertainment (5.8%), Websites (5.5%), Finance (4.4%), Business Tools (3.1%), Food & Beverage (1.7%), eCommerce (1.9%), and Retail (1.1%).

Agile vs. Waterfall Methodologies

The debate between Agile and Waterfall methodologies continues. We have long discussed our favorable methodology, the Agile approach, several times in past blogs, which you can read more about here in our dedicated columns since we are a tech-oriented business.

Our data, collected from over 104 projects using Waterfall (71.94% of clients) and 39 projects using Agile (28.06% of clients), reveal interesting trends. While the overall defect rates were similar (3.1% for Waterfall and 3.7% for Agile), a deeper analysis suggests that the choice of methodology should be tailored to the industry’s specific needs.

The statistics also offer a twist on which methodology would be beneficial for which industry, depending on its specific needs and development pace.

Agile featured industries

The agile model works for fast, fast-paced, customer orientation, and frequent updates industries, such as:

eCommerce and Retail industry:

The Agile model would allow the industry to adjust to customer preferences quickly,  seasonal sales events with promotional programs, integrating new innovations to improve customer experiences such as voice search features, AI chatbot assistants, unique customer shopping experiences based on the browsing habits analyzed by AI tech, etc. Easily implement A/B testing UI/UX elements in real-time to accommodate user feedback market trends, stay on trends, and not have underperforming features.

Finance and Banking industry (Fintech and Digital Banking):

Fintech industries have rigorously competed and gained decent recognition in the last five years, with new finance products entering the market that offer both convenience and security, such as eWallets, mobile banking apps, stock trading apps, etc. Due to Finance’s sensitive nature, software products must rapidly comply with the latest regulatory changes and third-party integrations without corrupting the existing platforms and applications. This might cause interruptions to the core services and deliver undesired experiences to the end customers.

Entertainment and Media

That would be Netflix, YouTube, Spotify, and all the other streaming platforms you can name. These giants often have to adjust their recommendation algorithms and User Interfaces to increase user engagement. The streaming industry environment contains endless possibilities of security risks and cyber-attacks.

We all know how many times they are attacked each year, as the event makes the headlines every single time. That doesn’t cover the “unofficial” downtime during which they went under the radar. Agile models certainly help these companies roll out content updates and feature enhancements frequently enough to retain their customers’ interest.

AI, Machine Learning, and Saas

The industry names themselves have well-spoken reasons why Agile is the best development methodology. AI-driven applications need constant training and iteration while the SaaS platform rolls out incremental and minor enhancements, updates, and features. The development team in these industries has to be cross-functional to achieve optimized collaborations and stay competitive.

Waterfall featured industries

From our survey statistics, it seems that industries that would benefit from the Waterfall model are those that need stability, follow strict regulatory compliance, and have complex, goal-oriented systems with high-security protocols that require planning.

Healthcare and Medical software industry

Healthcare applications involving electronic health records, online medical imaging systems, and patient management systems require an absolute confidential protocol to comply with HIPAA, GDPR, and the FDA. They must also be validated upon deployment. The waterfall model allows these industries to have comprehensive testing, a well-documented, sequential development process, regulatory compliance approvals, and long-term stability to meet legal and security requirements.

ERP and Business Tools industry

ERP system, or Enterprise Resource Planning System, integrates numerous business functions including but not limited to Human Resources, Finance, Logistics, etc. Any defect might affect and disrupt the organization’s operations and escalate quickly to a loss in revenue almost immediately. ERP systems are often tailored to individual businesses, meaning they deal with large-scale data migrations, user training, and heavy customization. The waterfall model approach enables a step-by-step process for each and every phase executed thoroughly to avoid minimal interruptions during the process.

The traditional Banking and Finance industry

The old banking and finance industry relies heavily on extremely highly secure software on a large scale that must comply with strict standards such as SOX, Basel III, PCI DSS, etc. Plus, the banking software cannot afford the frequent update deployments without extensive validation beforehand, which eventually adds up the expenses and excessive workload without bringing any significant financial trade-offs.

Government and Defense system

Including military logistics, national security systems, legal databases, and the top high-priority system that would not bear any chance of risks or threats. These applications, systems, or projects often have long development cycles, undergoing years of planning, developing, and deploying. The Waterfall linear model is the most aligned with this structure review and approval process, making it the best fit for the industry.

Construction and extensive infrastructure software

Speaking of massive-scale infrastructure software in general, such as smart city management platforms, transportation monitoring systems, construction project estimation, and management tools, these software have to be pre-planned strategically and await approvals. Both construction and infrastructure are operating on fixed schedules and budgets. Waterfall approaches can allow these industries to follow a structured timeline and have their software fully designed, then tested, and integrated without any interruptions or changes applied once every phase passes its approval stage.

The ultimate choice of either the Waterfall or Agile methodology might stress your business out. Don’t be hard on yourself about choosing either. You can use a hybrid model in which Agile and Waterfall can be deployed at different phases of your development cycle.

Defect Prevention: A Proactive Approach

Our data highlights the importance of early defect detection. Integration and system testing revealed the highest defect rates, demonstrating the value of our involvement in the testing process. That said, if we had not participated in the project testing process, these defects might not have been detected at this stage but might have somehow made their way to a much later stage.

A study from the IBM System Science Institute showed that fixing a defect during a testing phase would cost 15 times more than fixing it at the design stage and 100 times if it is found during a maintenance phase. That’s the price tag for only one defect. Imagining the 358 defects we found in the travel industry from one single client made their way to the end, how much would that cost? Here is the general table of how much it might cost if a defect is found at a certain stage:

The cost multiplies as it approaches the end of the SDLC. Therefore, the key finding from these statistics is that businesses should focus on defect prevention rather than detection. But how?

• Shift Left Testing: one of this year’s latest software testing trends yet. The approach lets developers and testers discuss the potential defects before the coding process begins. Automated unit testing and continuous integration (CI/CD) are recommended to be implemented to help identify issues early in development.
• TDD, or Test-Driven Development, was adopted mostly by the Agile methodology. In this approach, developers write the test before coding and verify its functionality from the beginning.
• Automated regression testing plays a dominant role in maintaining software integrity throughout its life cycle, whether a new feature is introduced, code is changed, or an update is released.
• Code reviews and peer testing: Regular peer code reviews can unveil up to 60% defect discovery rate.

Key findings in summary

Our survey findings offer valuable insights into the current state of software testing. Key takeaways include:

• Be clear and upfront about what your industry is heading for, either regulatory compliance, new inventions updates regularly, or security-oriented customer information to have testing strategies that work for you.
• Choosing either Agile or Waterfall or both as long as it boosts efficiency and productivity. Nothing else matters.
• Preventing and early detection always save money, reduce the time spent fixing problems, and deliver higher software performance.

If you are looking for a companion in software testing, SHIFT ASIA is here to help. Shift Asia is committed to helping businesses navigate the complexities of software testing. We provide end-to-end testing services tailored to your specific industry and development goals. We not only build high-quality testing solutions but also build strong customer relationships. Contact our team today to book your private consultant session.