The term "integrity
testing," as it is used here is a euphemism for stress testing. It is
used in preference to the term "stress testing" because it seems to better
convey the reason for using this type of testing.
If you are trying
to field a system with a client-server based architecture, and the server
just doesn't want to behave, you can benefit from integrity testing.
It is a well known,
but frequently under-appreciated fact that computers and the software
that runs on them are essentially non-deterministic systems. Inputs from
keyboards, system devices and external peripherals typically arrive at
random intervals (your software could scan your keyboard periodically,
but it has no control over when a user will press a key). Some of these
systems respond to asynchronous interrupts, which may be stacked many
layers deep. With modern operating systems, such as Unix, Solaris, Linux,
Windows/95/98/NT/2000/XP, VMS, etc., the programming environment not only
supports multiple processes running simultaneously, but multiple threads
within processes. This is done in an attempt to use CPU cycles in a more
efficient manner in order to provide ever-increasing levels of performance.
This leads to significant problems. As parallelism is increased, complexity
skyrockets, and crashes occur as a result of unanticipated conditions
occurring at random. Exacerbating this situation are the likelihood of
hidden flaws in the operating system and the programming libraries provided
by third parties. When one of these flaws pops up, it is difficult to
diagnose and even more difficult to correct. It isn't acceptable to throw
up one's hands at this point and say, "Oh well, it just doesn't work."
This is where integrity testing comes in. This is an approach to creating
software reliability in complex systems which Mr. Barchi pioneered at
Lockheed Martin Corporation. It involves the following procedures:
- The software to
be evaluated is examined to determine if it is a good candidate for
integrity testing (many simpler applications aren't). The software architecture
is examined for potential weaknesses and a test approach is designed
to exploit these in an effort to drive out hidden problems. Typically,
the types of problems that the integrity test focuses on are:
- Boundary limit
- Race conditions
- Memory leaks
limit problems occur when system resources are temporarily
exhausted due to overloading: memory queues fill up, system heap
is used up, communications traffic uses 100% of available bandwidth,
Race conditions occur when two or more processes compete
for the same resource, set a flag, etc., and implicit assumptions
that have been made about the possible order of events aren't
Memory leaks occur when software doesn't properly free
up dynamically allocated memory once it is done with it. There
are several excellent tools available, such as Rational's "Purify"
which are designed to detect memory leaks. However, they aren't
always practical to use due to the amount of overhead they impose.
Deadlocks occur when two or more processes need something
from each other to continue processing but none of them can proceed.
- AJB works with
the software developers to create an integrity test application which
focuses on key functionality of the product and stresses it to the point
where it fails (Mr. Barchi has been doing this for years and has never
encountered a complex software product that he couldn't break)
- AJB works with
the software developers to analyze the problems which are uncovered
and recommend solutions where appropriate (not all problems need to
- As the behavior
of the product when it is under stress becomes better understood, AJB
works with the software developers to expand the test to explore newly
identified potential weaknesses.
- As the product
reaches the desired level of stability, the client may then get further
value from the integrity test application by using it for performance
testing and regression testing.
The experience gained
from integrity testing helps software developers to better understand
the characteristics of complex systems and to architect more reliable
systems in the future.