Embedded Software Testing

Embedded Software Testing

What’s A Bug?

Simplistic answer:

  • A “bug” is a software defect = incorrect software
  • A software defect is an instance in which the software violates the specification

More realistic answer - a “bug” can be one or more of the following:

  • Failure to provide required behavior
  • Providing an incorrect behavior
  • Providing an undocumented behavior or behavior that is not required
  • Failure to conform to a design constraint (e.g., timing, safety invariant)
  • Omission or defect in requirements/specification
  • Instance in which software performs as designed, but it’s the “wrong” outcome
  • Any “reasonable” complaint from a customer
  • … other variations on this theme …

The goal of most testing is to attempt to find bugs in this expanded sense

Smoke Testing

Quick test to see if software is operational

  • Idea comes from hardware realm - turn power on and see if smoke pours out
  • Generally simple and easy to administer
  • Makes no attempt or claim of completeness
  • Smoke test for car: turn on ignition and check:
    • Engine idles without stalling
    • Can put into forward gear and move 5 feet, then brake to a stop
    • Wheels turn left and right while stopped

 

Good for catching catastrophic errors

  • Especially after a new build or major change
  • Exercises any built-in internal diagnosis mechanisms

But, not usually a thorough test

  • More a check that many software components are “alive”

Exploratory Testing

A person exercises the system, looking for unexpected results

  • Might nor might not be using documented system behavior as a guide
  • Is especially looking for “strange” behaviors that are not specifically required nor prohibited by the requirements

Advantages

  • An experienced, thoughtful tester can find many defects this way
  •  Often, the defects found are ones that would have been missed by more rigid testing methods

Disadvantages

  • Usually no documented measurement of coverage
  • Can leave big holes in coverage due to tester bias/blind spots
  • An inexperienced, non-thoughtful tester probably won’t find the important bugs

Black Box Testing

Tests designed with knowledge of behavior

  • But without knowledge of implementation
  • Often called “functional” testing

Idea is to test what software does, but not how function is implemented

  • Example: cruise control black box test
  • Test operation at various speeds
  • Test operation at various underspeed/overspeed amounts
  • BUT, no knowledge of whether lookup table or control equation is used

Advantages:

  • Tests the final behavior of the software
  • Can be written independent of software design
    • Less likely to overlook same problems as design
  • Can be used to test different implementations with minimal changes

Disadvantages:

  • Doesn’t necessarily know the boundary cases
    • For example, won’t know to exercise every lookup table entry
  • Can be difficult to cover all portions of software implementation

White Box Testing

Tests designed with knowledge of software design

  • Often called “structural” testing

Idea is to exercise software, knowing how it is designed

  • Example: cruise control white box test
  • Test operation at every point in control loop lookup table
  • Tests that exercise both paths of every conditional branch statement

Advantages:

  • Usually helps getting good coverage (tests are specifically designed for coverage)\
  • Good for ensuring boundary cases and special cases get tested

Disadvantages:

  • 100% coverage tests might not be good at assessing functionality for “surprise” behaviors and other testing goals
  • Tests based on design might miss bigger picture system problems
  • Tests need to be changed if implementation/algorithm changes

Testing Coverage

“Coverage” is a notion how completely testing has been done

  • Usually a percentage (e.g., “97% branch coverage”)

White box testing coverage

(this is the usual use of the word “coverage”):

  • Percent of conditional branches where both sides of branch have been tested
  • Percent of lookup table entries used in computations

Black box testing coverage:

  • Percent of requirements tested
  • Percent of documented exceptions exercised by tests
  • But, must relate to externally visible behavior or environment, not code structure

Important note: 100% coverage is not “100% tested”

  • Each coverage aspect is narrow; good coverage is necessary, but not sufficient to achieve good testing

Unit Test

A “unit” is a few lines of code to a small program

  • Usually created by a single developer
  • Usually tested by the programmer who created it

Purpose of unit test:

  • Try to find all the “obvious” defects
  • Can be done before and/or after code review

Approaches (mostly exploratory & white box)

  • Exploratory testing makes a lot of sense
    • Helps programmer build intuition and understand code
  • White box testing to ensure all portions of code exercised
    • Often useful to ensure 100% arc and state coverage for statecharts
  • Some black box testing as a “sanity check”

Subsystem Test

A “subsystem” is a relatively complete software component (e.g., engine controller software)

  • Usually created by a team of developers
  • Usually tested by a combination of programmers and independent testers

Purpose of subsystem test:

  • Try to find all the “obvious” defects
  • Can be done before and/or after code review

Approaches (mostly white box; some black box)

  • White box testing is key to ensuring good coverage
  • Black box testing should at a minimum check interface behaviors against specification to avoid system integration surprises
  • Exploratory testing can be helpful, but shouldn’t find a lot of problems
  • Smoke test is helpful in making sure a change in one part of subsystem doesn’t completely break the entire subsystem

System Integration Test

A “system” is a complete multi-component system (e.g., a car)

  • Often created by multiple teams organized in different groups
  • Usually tested by independent test organizations

Purpose of system integration test:

  • Assume that components are mostly correct; ensure system behaves correctly
  • Find problems/holes/gaps in interface specifications that cause system problems
  • Find unexpected behavior in system

Approaches (mostly black box)

  • Tends to be mostly black box testing to ensure system meets requirements
  • Want white box techniques to ensure all aspects of interfaces are tested
  • Exploratory testing can help look for strange component interactions
  • Smoke tests are often used to find version mismatches and other problems in independent components

Acceptance Test

Acceptance tests ensure system provides all advertised functions

  • Testing performed by a customer or surrogate
  • Might also involve a certification authority or independent test observer

Purpose of acceptance test:

  • Does the system meet all requirements to be used by a customer?
  • Usually the last checkpoint before shipping a system
  • Might be performed on all systems to check for hardware defects/manufacturing defects, not just software design problems

Approaches

  • Usually black box testing of system vs. 100% of requirements

Beta Test

A “beta version” is complete software that is “close” to done

  • Theoretically all defects have been corrected or are at least known to developers
  • Idea is to give it to sample users and see if a huge problem emerges

Purpose of beta test:

  • See if software is good enough for a friendly, small user community (limit risk)
  • Find out if “representative” users are likely to stimulate failure modes missed by testing

Approaches

  • This is almost all exploratory testing
    • Assumption is that different users have different usage profiles
    • Hope is that if small user community doesn’t find problems, there won’t be many important problems that slip through into full production

Why Do We Test - Revisited

1 Testing to find bugs

2 Testing to demonstrate correctness

Implications of testing to find bugs in practice

  • This is really the “fly-fix-fly” philosophy used historically in aviation
    • Fly around until a plane crashes
    • Fixes whatever caused a crash so the next plane won’t break that way
    • Fly around some more
    • Eventually improves quality, but never achieves perfection
  • We hope that only a small percentage of bugs slip past each stage
  • Important to base tests on “things that matter” to find “bugs that matter”
  • Finding & fixing “all” bugs in practice never ends
    • Eventually must decide to ship with some known & unknown defects
    • “Software reliability” theory comes into play (“how much do we test” slide later)
  • Important approach is to ensure all specified functions work
    • Doesn’t prove they work in all cases, but at least gives some confidence in result

3 Testing to demonstrate software quality

Hypothesis - “there are no defects in this software”

  • Test approach: attempt to find defects
  • Can disprove hypothesis by finding defects
  • Can never prove hypothesis, because complete testing is impossible

This is more of a manufacturing QA philosophy

  • Detect errors in the “manufacturing” process (creating software)
    • Detection done through defects in created items
    • BUT, the issue isn’t the item being out of specification; it is that the process is broken
  • Fix process errors

Author: Philip Koopman

Knowledge Center
Knowledge Center
Subscribe

*- required fields