1. Fundamentals of Testing
Table of Contents

1.1 Why is Testing Necessary?

  • Human -> Error (mistake) -> Defect (fault, bug) which when executed may cause -> Failure
  • Measures the quality of the software
  • Gives confidence in the quality
  • Reduces the overall level of risk
  • How much testing? Depends on technical/business risk (impact and probability), safety & project constraints (time and budget)

1.1.1 Describe, with examples, the way in which a defect in software can cause harm to a person, to the environment or to a company

  1. A defect in software can cause harm to person, environment or company
  2. A defect can cause loss of money, time or business
  3. Testing improves software quality
  4. Testing reduces the risk

Money, reputation, time, loss of life can be impacted.

Why we make mistakes:

  • Lack of experience
  • Don't have right information
  • Misunderstanding
  • Careless attitude
  • Tiredness
  • Under time pressure
  • Dealing with complexity

1.1.2 Distinguish between the root cause of a defect and its effects

Mistake => Defect => failure

Not all defects cause failure. Failures can be caused defects, environment conditions, or malicious damage.

Causes:

  • Faulty requirements definition
  • Time pressure
  • Complex code
  • Many system interactions
  • Coding errors
  • Complexity of infastructure
  • Changing technologies
  • Non-compliance with standards

Root cause analysis:

  1. Brainstorming all the possible causes
  2. Grouping them into categories

1.1.3 Give reasons why testing is necessary by giving examples

  • Because software is likely to have faults
  • To learn about the reliability of the software
  • Because failures can be very expensive
  • To avoid being sued by customers

1.1.4 Describe why testing is part of quality assurance and give examples of how testing contributes to higher quality

1.1.5 Explain and compare the terms error, defect, fault, failure, and the corresponding terms mistake and bug, using examples

According to ISTQB glossay:

  • bug: same as defect
  • defect: A flaw in a component or system that can cause the component or system to fail to perform its required function, e.g. an incorrect statement or data definition. A defect, if encountered during execution, may cause a failure of the component or system.
  • error: A human action that produces an incorrect result.
  • failure: Deviation of the component or system from its expected delivery, service or result.
  • fault: same as defect
  • mistake: same as error




1.2 What is Testing?

  • Finding Defects
  • Providing information for decision-making
  • Preventing defects
  • Gaining confidence about the level of quality

1.2.1 Recall the common objectives of testing

  1. Determine that software meets end user requirements (validation)
  2. Demonstrate that software is fit for use (verification)
  3. Find defects/bugs in the software
  • Finding Defects
  • Providing information for decision-making
  • Preventing defects
  • Gaining confidence about the level of quality

1.2.2 Provide examples for the objectives of testing in different phases of the software life cycle

V-model.jpg
from http://istqbexamcertification.com/wp-content/uploads/2012/01/V-model.jpg

1.2.3 Differentiate testing from debugging

Testers test, developers fix bugs.




1.3 Seven Testing Principles

  • Testing shows presence of defects
  • Exhaustive testing is impossible
  • Early testing
  • Defect clustering
  • Pesticide paradox
  • Testing is context dependent
  • Absence-of-error fallacy

1.3.1 Explain the seven principles in testing

Principle 1 – Testing shows presence of defects

Testing can show that defects are present, but cannot prove that there are no defects. Testing reduces the probability of undiscovered defects remaining in the software but, even if no defects are found, it is not a proof of correctness.

Principle 2 – Exhaustive testing is impossible

Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases. Instead of exhaustive testing, risk analysis and priorities should be used to focus testing efforts.

Principle 3 – Early testing

To find defects early, testing activities shall be started as early as possible in the software or system development life cycle, and shall be focused on defined objectives.

Start-Software-Testing-On-Early-Stages4.-How-Does-This-Affect-Cost.jpg
from http://qatestlab.com/assets/Start-Software-Testing-On-Early-Stages4.-How-Does-This-Affect-Cost.jpg

Principle 4 – Defect clustering

Testing effort shall be focused proportionally to the expected and later observed defect density of modules. A small number of modules usually contains most of the defects discovered during pre-release testing, or is responsible for most of the operational failures.

AKA 80/20 rule

Principle 5 – Pesticide paradox

If the same tests are repeated over and over again, eventually the same set of test cases will no longer find any new defects. To overcome this “pesticide paradox”, test cases need to be regularly reviewed and revised, and new and different tests need to be written to exercise different parts of the software or system to find potentially more defects.

Principle 6 – Testing is context dependent

Testing is done differently in different contexts. For example, safety-critical software is tested differently from an e-commerce site.

Principle 7 – Absence-of-errors fallacy

Finding and fixing defects does not help if the system built is unusable and does not fulfill the users’ needs and expectations.




1.4 Fundamental Test Process

  • Planning & Control
  • Analysis & Design
  • Implementation & Execution
  • Evaluating Exit Criteria & Reporting
  • Test Closure

1.4.1 Recall the five fundamental test activities and respective tasks from planning to closure

1. Test planning and control

Planning:

  1. Identify the objectives of testing based on the scope and risks of the project
  2. Determine the test approach according to test strategy
  3. Determine the required resources
  4. Schedule test analysis and design tasks, implementation, execution and evaluation
  5. Determine the exit criteria

Control:
Test control is the ongoing activity of comparing actual progress against the plan. It includes:

  1. Measure and analyse results of reviews and testing
  2. Monitor and document test progress, coverage and exit criteria
  3. Reporting of test progress
  4. Initiate corrective actions. e.g. more testing resources
  5. Make release decisions

Test analysis and design

  1. Reviewing the test basis
  2. Identify test conditions
  3. Design the tests
  4. Evaluate testability of requirements and system
  5. Design the test environment

Test implementation and execution

Implementation:

  1. Take test conditions and make them into test cases
  2. Build test environment
  3. Create test suites
  4. Prioritise tests based on risk assessment

Execution:

  1. execute test cases and record outcomes, including relevant details like environment, software version
  2. Report incidents, which may become defects if confirmed
  3. Re-test defects
  4. Change and regression testing

Evaluating exit criteria and reporting

  • Done for each test level
  • Exit criteria is based on risk assessment
  • Assess if more tests are needed or if the exit criteria should be changed
  • Test Summary report for stakeholders

Test closure activities

  1. Collect data from completed test activities to consolidate experience
  2. Incident reports resolved (fixed/deferred)
  3. Finalise and archive testware (e.g. for regression testing)
  4. If appropriate, handover testware to maintenance organisation (not appropriate for many organisations)
  5. Evaluate testing and lessons learned




1.5 The Psychology of Testing

  • Mindset of Developer & Tester
  • Communication in a constructive manner
  • Test Independence

1.5.1 Recall the psychological factors that influence the success of testing

Independence: Developer => Tester on Development team => Independent Permanent Test Team => Outsourced Test Team

Best practices while reporting defects:

  • Communicate findings in a neutral, fact focused way. Don't criticize.
  • Be pessimistic and start with collaborations rather than battles

And:

  • Keep the focus on delivering a quality product
  • Objective not subjective
  • Attempt to understand how others feel
  • Confirm that you have both understood and been understood

Testers should have:

  • Curiosity
  • Profesisonal pessimism
  • Critical eye
  • Attention to detail
  • Good communication with developers
  • Error quessing

As a test professional, have the desire to "find problems". Be curious, have a critical eye and attention to detail

1.5.2 Contrast the mindset of a tester and of a developer

  • Independent tester questions assumptions and is less biased
  • Easier to find defects with fresh set of eyes

Developer:

  • Show that the system does what it should, and doesn't do what it shouldn't
  • Success: system works
  • Goal: show working
  • Test cases: easy

Tester:

  • Show that the system does what it shouldn't, and doesn't do what it should
  • Success: system works
  • Goal: find defects
  • Test cases: difficult




1.6 Code of ethics

  • Code is necessary, among other reasons, to ensure information accessed by testers are not put to inappropriate use.
  1. PUBLIC - Certified software testers shall act consistently with the public interest
  2. CLIENT AND EMPLOYER - Certified software testers shall act in a manner that is in the best interests of their client and employer, consistent with the public interest
  3. PRODUCT - Certified software testers shall ensure that the deliverables they provide (on the products and systems they test) meet the highest professional standards possible
  4. JUDGMENT- Certified software testers shall maintain integrity and independence in their professional judgment
  5. MANAGEMENT - Certified software test managers and leaders shall subscribe to and promote an ethical approach to the management of software testing
  6. PROFESSION - Certified software testers shall advance the integrity and reputation of the profession consistent with the public interest
  7. COLLEAGUES - Certified software testers shall be fair to and supportive of their colleagues, and promote cooperation with software developers
  8. SELF - Certified software testers shall participate in lifelong learning regarding the practice of their profession and shall promote an ethical approach to the practice of the profession
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License