The statement "testing shows the presence of defects, not their absence" expresses an important aspect of software testing. In essence, testing can reveal bugs, issues, faults, and inconsistencies in software, but it cannot prove that a product is defect-free.

The goal for management is not to demonstrate that a product is defect-free, but to ensure that it meets user needs and business requirements. Management considers those standards and user expectations when determining whether or not the product functions as planned.

It is the QA team's responsibility to regularly evaluate the product against these two requirements.

While that process is continuous in any project, everyone on the team, from management to developers to the QA team, should know that it is simply not possible to test every possible case or configuration. Some elements of the software will always be untested, and it is probable that flaws will exist in these untested regions.

Another way to look at it is that testing can only show that the software works as expected for the test cases that have been executed. As a result, it is critical to build thorough test cases and rigorously execute them, but it also accepts that there may still be faults that have not been detected.

It urges QA teams to approach testing with a continuous improvement mindset, continuously striving to develop new and better ways to test software and uncover potential faults within the team's and project's business limitations.

QA teams should assess and enhance the testing process on a regular basis. They should inform stakeholders about the testing process's limits. The fundamental argument is that, while testing is an important stage in the software development process, it cannot ensure that all errors are discovered.

Here are five tangible real-world examples of this idea in action:

1. When testing a login feature for an e-commerce platform, the QA team can test for valid and incorrect credentials, but not every possible combination of characters and symbols in a username or password input field.

2. The same e-commerce platform interfaces with numerous third-party services such as payment processors, shipping providers, and inventory management systems. However, the QA team has limited access and control to properly assess the integrity of these external systems within the e-commerce platform itself.

3. When testing the e-commerce software on Chrome, Firefox, and Safari, the QA team tests the major browser versions, operating systems, and hardware configurations but is unable to test every single specification each user has.

4. In the latest software release, the QA team overlooked a flaw in the 'Forgot Password' feature because the test case was mistakenly tagged as 'Passed' rather than 'Failed' in the test management tool due to human error.

5. Due to a tight release schedule, the QA team had to prioritize testing critical features such as 'Checkout' and 'Payment' and could not undertake extensive testing on less vital features such as 'Wishlist' or 'Product Reviews'. This is due to limited time and resources.

These examples demonstrate that it is very hard to test every potential combination of inputs, configurations, user actions, and environments to show that a software application is defect-free.

Incomplete testing will always be a reality for the QA team.

Another critical aspect to remember is that testing is performed on a continually changing codebase, particularly in Agile and DevOps environments where codebases change on a daily or weekly basis. This makes it difficult to assert that the software is defect-free at any moment. In this case, QA teams cannot be certain that the required testing methodologies, such as functional testing, integration testing, performance testing, security testing, UAT testing, and regression testing, are providing 100% defect-free test coverage.

Because this is the reality of the QA testing environment, the meaning and true result of this concept is to detect faults as early as possible so that they may be addressed before they affect end-users. Along with this, the goal must be that testing informs management, developers, product teams, business teams, and other stakeholders about the quality of the software, as well as the level of risk involved with releasing it on time.

To continuously enhance the testing flow, the QA team can investigate:

1. Feedback loops

2. Maintaining metrics and KPI’s

3. Focusing on fulfilling user acceptance criteria

In the context of QA and software development, a feedback loop is a continual cycle of collecting feedback from many sources—such as developers, management, and end users—to enhance testing processes and product quality. By creating strong feedback loops, the QA team can obtain fast and actionable feedback from these groups. This aids in the identification of gaps, the improvement of test cases, and the overall testing approach.

For example, following each release, the QA team meets with developers and product managers to discuss what went well and what didn't. During one such discussion, it was revealed that a serious flaw had slipped through the cracks due to a last-minute code change. As a result, the QA team chooses to introduce a "code freeze" period before releases to provide for appropriate testing time. Usually this would be done during a scrum retrospective meeting.

Metrics and Key Performance Indicators (KPIs) quantify the testing process empirically. They aid in the long-term tracking of efficiency, effectiveness, and overall quality. This data-driven approach helps to make educated judgments and justify additional resources or strategy modifications.

QA leaders and scrum masters, for example, can monitor "Defect Detection Percentage" (DDP), which gauges the success of the testing process in finding issues before release. They detect a deteriorating trend in DDP over a few months, triggering an internal audit. The audit finds that the team has been focused too much on automated testing while ignoring exploratory testing, resulting in a strategy shift by management.

User Acceptance Criteria (UAC) are the final evaluation of a feature's quality in the eyes of the user. Ensuring that testing is by UAC ensures that the program fits user expectations as well as corporate requirements, lowering the chance of customer unhappiness.

As another example, the User Acceptance Criteria for a new "One-Click Checkout" feature specifies that the process should not take more than 3 seconds. This criterion is included in the QA team's test cases. They discover during testing that the functionality takes 5 seconds to complete the checkout, classifying it as a critical concern. This results in performance optimization prior to the introduction of the feature.

There are mitigation strategies QA teams can enforce to ensure that a minimal amount of defects are going into production. For example, QA teams and management can implement:

1. Risk-based testing

2. Automation testing

3. Continuous testing

Risk-based testing involves prioritizing the features and components of a software application based on the risk of them failing. This approach ensures that the most critical parts of the application are thoroughly tested, thereby reducing the likelihood of severe defects slipping into production.

Automation testing involves using software tools to execute predefined test cases and compare the actual outcomes with expected results. Automated tests are particularly useful for repetitive tasks and regression testing, allowing the QA team to focus on more complex test scenarios. This increases the overall efficiency and coverage of the testing process.

Continuous testing is the practice of conducting automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate. This approach allows for quicker identification and resolution of defects, making it easier to maintain a high level of software quality throughout the development lifecycle.

As mentioned previously, the Agile methodology is inherently one where the pace of development is fast and the codebase is continuously evolving. So businesses often employ a range of strategies to manage testing the implications of this testing principle.

Here are some real-world examples of how companies use Agile systems:

1. TDD (Test-Driven Development)

2. Automated Regression Testing

3. Exploratory Testing

4. Continuous Monitoring and Logging

5. User Acceptance Testing (UAT) and Beta Testing

6. Chaos Engineering

Test-Driven Development (TDD) requires developers to write tests before producing actual code, guaranteeing that each small piece of functionality is tested as soon as possible. While this method can detect faults early on, it does not claim to render the code defect-free; rather, it reinforces the goal of defect identification. While developers are primarily responsible for writing unit tests in TDD, QA engineers can contribute in various ways to enhance the quality and coverage of unit testing. For example, they can collaboratively plan with requirement analysis and test strategy, or participate in code review, facilitating test data, or analyze test coverage reports. An example of this is a financial tech firm like Stripe using TDD while developing a new transaction engine. This allows them to detect logical problems in transaction fee calculation early in the development cycle.

Automated Regression Testing automates regression tests to guarantee that new code changes do not break current functionality. Because not all test scenarios have been automated, automation helps to execute several test scenarios relatively rapidly but does not guarantee that the product is 100% bug-free. For example, an e-commerce company automates regression tests to run after every nightly build, ensuring that new features haven't broken the existing checkout process.

Exploratory Testing involves knowledgeable testers actively exploring the application in order to uncover faults that automated and scripted tests may have missed. The emphasis is on finding new problems, recognizing that testing cannot cover all possible circumstances. For example, a streaming service employs exploratory testing to uncover user experience issues that automated tests didn't catch, such as how a video’s streaming quality changes when switching between Wi-Fi and mobile data.

When it comes to Continuous Monitoring and Logging, once the software is in production, continuous monitoring tools track system behaviour. Any discrepancies can be deemed potential flaws, demonstrating that pre-release testing did not guarantee a defect-free product. For example, a travel booking website uses real-time monitoring to flag unexpected behaviours in its search and booking engine. This allows them to detect and explore any flaws rapidly.

Chaos Engineering teams can identify how a system operates under chaotic settings by purposely injecting errors into it. The process uncovers defects but also makes it clear that the absence of known defects is not the same as proof of stability or reliability. For example, a cloud storage provider uses chaos engineering to simulate server failures, network latency, and other disruptive events to ensure their service can recover gracefully and maintain data integrity.

For User Acceptance Testing (UAT) and Beta Testing, the QA team and product team test the software in a production-like environment, providing insights into how the software performs under real-world conditions. Defects discovered here demonstrate that earlier testing rounds cannot cover all conceivable use-case scenarios. For example, a food delivery app rolls out a new feature to a subset of its user base, allowing them to provide real-world feedback. Issues discovered at this stage are resolved prior to a full-scale roll-out.

Example Scenario

Let's make a scenario to better comprehend this principle. Assume an e-commerce business is working on a new checkout feature that offers a variety of payment choices, such as credit cards, digital wallets, and a "buy now, pay later" service. The development team operates in an Agile environment, with updates released every two weeks. As soon as the first bits of functional code are ready, the testing team is incorporated into this Agile workflow and begins testing the new checkout feature.

The Agile team goes through the following testing stages:

1. Unit Testing: Unit tests are written by developers to ensure that individual components function as expected.

2. Integration Testing: The QA team determines if the new checkout functionalities are compatible with the existing shopping cart and user authentication systems.

3. Functional Testing: Testers confirm that the checkout process matches the given requirements both manually and automatically.

4. User Acceptance Testing: Selected end-users (usually the product team and management) engage in ensuring that the new checkout procedure is simple and effective.

The following events occur:

1. Presence of Defects: During the testing phases, the team discovers a number of problems.

   • A bug that prevents certain items from being purchased with the "buy now, pay later" option.

   • When using a digital wallet, the checkout page takes too long to load.

   • One of the new payment APIs has a security flaw.

2. Actions Taken: The developers resolve the identified issues, which are then re-tested and confirmed by the testing team.

   • Performance and security tests are ramped up to mitigate the risks further.

3. Release and Monitoring: - With all known critical and high-priority defects fixed, the team feels confident about releasing the feature to a subset of users.

   • Real-time monitoring tools are set up after the release to detect any irregularities in the checkout process.

Now here is why testing shows the presence of defects and not their absence.

Despite extensive testing and subsequent fixes, the customer support team begins receiving complaints about a very specific issue a week after the official release. When a consumer adds a combination of specified sale goods to their shopping cart and then proceeds to checkout, a discount coupon does not work as it should. Because of its intricacy and low possibility of occurrence, this is an edge situation that was not covered during the initial testing.

Analysis:

1. Complexity: The sheer amount of item, discount, and payment option permutations makes it nearly difficult to test every possible situation.

2. Constraints: Due to the two-week sprint cycles and resource constraints, the team prioritized the most common and high-risk regions.

3. Changing Codebase: Because the Agile environment encourages rapid changes, there is always the risk of introducing new issues.

4. Incomplete Understanding: No matter how well you believe you understand user behaviour, there may be circumstances that you overlooked.

During the development phase of this real-world scenario, the testing team effectively detected many problems. They couldn't, however, establish the "absence" of faults, as proved by an overlooked edge case that occurred after the release. This incident illustrates the idea that testing is about detecting problems and managing risk, not demonstrating that software is completely defect-free.

Finally, the principle "Testing shows the presence of defects, not their absence" serves as a reminder that testing is a continual process that aids in the identification and correction of defects, but it cannot ensure that all defects have been discovered. It emphasizes the significance of continual testing process improvement and the requirement for QA teams to approach testing with a realistic grasp of its limitations.

If you liked this post, check out my blog for more on Test Engineering, QA Automation, Software Development, and DevOps.

👉 Let's Connect: If you're interested in my work, you can also follow me on LinkedIn, or GitHub. I'm always open to collab and would love to hear from you.