What’s A Bug?
Simplistic answer:
- A “bug” is a software defect = incorrect software
- A software defect is an instance in which the software violates the specification
More realistic answer - a “bug” can be one or more of the following:
- Failure to provide the required behavior
- Providing incorrect behavior
- Providing an undocumented behavior or behavior that is not required
- Failure to conform to a design constraint (e.g., timing, safety invariant)
- Omission or defect in requirements/specification
- An instance in which software performs as designed, but it’s the “wrong” outcome
- Any “reasonable” complaint from a customer
- … other variations on this theme …
The goal of most testing is to attempt to find bugs in this expanded sense
Smoke Testing
A quick test to see if the software is operational
- The idea comes from the hardware realm - turn the power on and see if smoke pours out
- Generally simple and easy to administer
- Makes no attempt or claim of completeness
- Smoke test for car: turn on ignition and check:
- Engine idles without stalling
- Can put into forwarding gear and move 5 feet, then brake to a stop
- Wheels turn left and right while stopped
Good for catching catastrophic errors
- Especially after a new build or major change
- Exercises any built-in internal diagnosis mechanisms
But, not usually a thorough test
- More a check that many software components are “alive”
Exploratory Testing
A person exercises the system, looking for unexpected results
- Might nor might not be using documented system behavior as a guide
- Is especially looking for “strange” behaviors that are not specifically required nor prohibited by the requirements
Advantages
- An experienced, thoughtful tester can find many defects this way
- Often, the defects found are ones that would have been missed by more rigid testing methods
Disadvantages
- Usually no documented measurement of coverage
- Can leave big holes in coverage due to tester bias/blind spots
- An inexperienced, non-thoughtful tester probably won’t find the important bugs
Tests designed with knowledge of behavior
- But without knowledge of implementation
- Often called “functional” testing
Idea is to test what software does, but not how the function is implemented
- Example: cruise control black box test
- Test operation at various speeds
- Test operation at various underspeed/overspeed amounts
- BUT, no knowledge of whether the lookup table or control equation is used
Advantages:
- Tests the final behavior of the software
- Can be written independent of software design
- Less likely to overlook the same problems as design
- Can be used to test different implementations with minimal changes
Disadvantages:
- Doesn’t necessarily know the boundary cases
- For example, won’t know to exercise every lookup table entry
- Can be difficult to cover all portions of software implementation
White Box Testing
Tests designed with knowledge of software design
- Often called “structural” testing
Idea is to exercise software, knowing how it is designed
- Example: cruise control white box test
- Test operation at every point in control loop lookup table
- Tests that exercise both paths of every conditional branch statement
Advantages:
- Usually helps to get good coverage (tests are specifically designed for coverage)\
- Good for ensuring boundary cases and special cases get tested
Disadvantages:
- 100% coverage tests might not be good at assessing functionality for “surprise” behaviors and other testing goals
- Tests based on design might miss bigger picture system problems
- Tests need to be changed if implementation/algorithm changes
Testing Coverage
“Coverage” is a notion of how completely testing has been done
- Usually a percentage (e.g., “97% branch coverage”)
White box testing coverage
(this is the usual use of the word “coverage”):
- Percent of conditional branches where both sides of the branch have been tested
- Percent of lookup table entries used in computations
Black box testing coverage:
- Percent of requirements tested
- Percent of documented exceptions exercised by tests
- But, must relate to externally visible behavior or environment, not code structure
Important note: 100% coverage is not “100% tested”
- Each coverage aspect is narrow; good coverage is necessary, but not sufficient to achieve good testing
Unit Test
A “unit” is a few lines of code to a small program
- Usually created by a single developer
- Usually tested by the programmer who created it
Purpose of the unit test:
- Try to find all the “obvious” defects
- Can be done before and/or after code review
Approaches (mostly exploratory & white box)
- Exploratory testing makes a lot of sense
- Helps programmer build intuition and understand code
- White box testing to ensure all portions of code exercised
- Often useful to ensure 100% arc and state coverage for statecharts
- Some black box testing as a “sanity check”
Subsystem Test
A “subsystem” is a relatively complete software component (e.g., engine controller software)
- Usually created by a team of developers
- Usually tested by a combination of programmers and independent testers
Purpose of subsystem test:
- Try to find all the “obvious” defects
- Can be done before and/or after code review
Approaches (mostly white box; some black box)
- White box testing is key to ensuring good coverage
- Black box testing should at a minimum check interface behaviors against specification to avoid system integration surprises
- Exploratory testing can be helpful, but shouldn’t find a lot of problems
- A smoke test is helpful in making sure a change in one part of the subsystem doesn’t completely break the entire subsystem
System Integration Test
A “system” is a complete multi-component system (e.g., a car)
- Often created by multiple teams organized in different groups
- Usually tested by independent test organizations
Purpose of system integration test:
- Assume that components are mostly correct; ensure system behaves correctly
- Find problems/holes/gaps in interface specifications that cause system problems
- Find unexpected behavior in the system
Approaches (mostly black box)
- Tends to be mostly black box testing to ensure the system meets requirements
- Want white box techniques to ensure all aspects of interfaces are tested
- Exploratory testing can help look for strange component interactions
- Smoke tests are often used to find version mismatches and other problems in independent components
Acceptance Test
Acceptance tests ensure the system provides all advertised functions
- Testing performed by a customer or surrogate
- Might also involve a certification authority or independent test observer
Purpose of acceptance test:
- Does the system meet all requirements to be used by a customer?
- Usually the last checkpoint before shipping a system
- Might be performed on all systems to check for hardware defects/manufacturing defects, not just software design problems
Approaches
- Usually black box testing of system vs. 100% of requirements
Beta Test
A “beta version” is a complete software that is “close” to done
- Theoretically, all defects have been corrected or are at least known to developers
- Idea is to give it to sample users and see if a huge problem emerges
Purpose of beta test:
- See if the software is good enough for a friendly, small user community (limit risk)
- Find out if “representative” users are likely to stimulate failure modes missed by testing
Approaches
- This is almost all exploratory testing
- The assumption is that different users have different usage profiles
- Hope is that if the small user community doesn’t find problems, there won’t be many important problems that slip through into full production
Why Do We Test - Revisited
1 Testing to find bugs
2 Testing to demonstrate the correctness
Implications of testing to find bugs in practice
- This is really the “fly-fix-fly” philosophy used historically in aviation
- Fly around until a plane crashes
- Fixes whatever caused a crash so the next plane won’t break that way
- Fly around some more
- Eventually improves quality, but never achieves perfection
- We hope that only a small percentage of bugs slip past each stage
- Important to based tests on “things that matter” to find “bugs that matter”
- Finding & fixing “all” bugs in practice never ends
- Eventually must decide to ship with some known & unknown defects
- “Software reliability” theory comes into play (“how much do we test” slide later)
- The important approach is to ensure all specified functions work
- Doesn’t prove they work in all cases, but it at least gives some confidence in the result
3 Testing to demonstrate software quality
Hypothesis - “there are no defects in this software”
- Test approach: attempt to find defects
- Can disprove the hypothesis by finding defects
- Can never prove a hypothesis, because complete testing is impossible
This is more of a manufacturing QA philosophy
- Detect errors in the “manufacturing” process (creating software)
- Detection was done through defects in created items
- BUT, the issue isn’t the item being out of specification; it is that the process is broken
- Fix process errors
Author: Philip Koopman