Regression testing is any type of software testing which seeks to uncover regression bugs. Regression bugs occur whenever software functionality that previously worked as desired, stops working or no longer works in the same way that was previously planned. Typically regression bugs occur as an unintended consequence of program changes.
Common methods of regression testing include re-running previously run tests and checking whether previously fixed faults have re-emerged.Experience has shown that as software is developed, this kind of reemergence of faults is quite common. Sometimes it occurs because a fix gets lost through poor revision control practices (or simple human error in revision control), but often a fix for a problem will be "fragile" in that it fixes the problem in the narrow case
where it was first observed but not in more general cases which may arise over the lifetime of the software. Finally, it has often been the case that when some feature is redesigned, the same mistakes will be made in the redesign that were made in the original implementation of the feature.
Types of regression
· Local - changes introduce new bugs.
· Unmasked - changes unmask previously existing bugs.
· Remote - Changing one part breaks another part of the program. For example, Module A writes to a database. Module B reads from the database. If changes to what Module A writes to the database break Module B, it is remote regression.
There's another way to classify regression.
· New feature regression - changes to code that is new to release 1.1 break other code that is new to release 1.1.
· Existing feature regression - changes to code that is new to release 1.1 break code that existed in release 1.0.
Mitigating regression risk
· Complete test suite repetition
· Regression test automation (GUI, API, CLI)
· Partial test repetition based on traceability and analysis of technical and business risks
· Customer or user testing
oBeta - early release to both potential and current customers
oPilot - deploy to a subset of users
oParallel - users use both old and new systems simultaneously
· Use larger releases. Testing new functions often covers existing functions. The more new features in a release, the more "accidental" regression testing.
· Emergency patches - these patches are released immediately, and will be included in future maintenance releases.
Uses
Regression testing can be used not only for testing the correctness of a program, but it is also often used to track the quality of its output. For instance in the design
of a compiler, regression testing should track the code size, simulation time and compilation time of the test suite cases.
Showing posts with label Test Techniques Types Levels. Show all posts
Showing posts with label Test Techniques Types Levels. Show all posts
Wednesday, January 21, 2009
Purpose Of Integration Testing
The purpose of integration testing is to verify functional, performance and reliability requirements placed on major design items. These "design items", i.e. assemblages (or groups of units), are exercised through their interfaces using black box testing, success and error cases being simulated via appropriate parameter and data inputs. Simulated usage of shared data areas and inter-process communication is tested and individual subsystems are exercised through their input
interface. Test cases are constructed to test that all components within assemblages interact correctly,
for example across procedure calls or process activations, and
this is done after testing individual modules, i.e. unit testing.
interface. Test cases are constructed to test that all components within assemblages interact correctly,
for example across procedure calls or process activations, and
this is done after testing individual modules, i.e. unit testing.
Types Of Testing
What kinds of testing should be considered?
* Black box testing - not based on any knowledge of internal design or code. Tests are based on requirements and functionality.
* White box testing - based on knowledge of the internal logic of an application's code. Tests are based on coverage of code statements, branches, paths, conditions.
* unit testing - the most 'micro' scale of testing; to test particular functions or code modules. Typically done by the programmer and not by testers, as it requires detailed knowledge of the internal program design and code. Not always easily done unless the application has a well-designed architecture with tight code; may require developing test driver modules or test harnesses.
* incremental integration testing - continuous testing of an application as new functionality is added; requires that various aspects of an application's functionality be independent enough to work separately before all parts of the program are completed, or that test drivers be developed as needed; done by programmers or by testers.
* integration testing - testing of combined parts of an application to determine if they function together correctly. The 'parts' can be code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to client/server and distributed systems.
* functional testing - black-box type testing geared to functional requirements of an application; this type of testing should be done by testers. This doesn't mean that the programmers shouldn't check that their code works before releasing it (which of course applies to any stage of testing.)
* system testing - black-box type testing that is based on overall requirements specifications; covers all combined parts of a system.
* end-to-end testing - similar to system testing; the 'macro' end of the test scale; involves testing of a complete application environment in a situation that mimics real-world use, such as interacting with a database, using network communications, or interacting with other hardware, applications, or systems if appropriate.
* sanity testing or smoke testing - typically an initial testing effort to determine if a new software version is performing well enough to accept it for a major testing effort. For example, if the new software is crashing systems every 5 minutes, bogging down systems to a crawl, or corrupting databases, the software may not be in a 'sane' enough condition to warrant further testing in its current state.
* regression testing - re-testing after fixes or modifications of the software or its environment. It can be difficult to determine how much re-testing is needed, especially near the end of the development cycle. Automated testing tools can be especially useful for this type of testing.
* acceptance testing - final testing based on specifications of the end-user or customer, or based on use by end-users/customers over some limited period of time.
* load testing - testing an application under heavy loads, such as testing of a web site under a range of loads to determine at what point the system's response time degrades or fails.
* stress testing - term often used interchangeably with 'load' and 'performance' testing. Also used to describe such tests as system functional testing while under unusually heavy loads, heavy repetition of certain actions or inputs, input of large numerical values, large complex queries to a database system, etc.
* performance testing - term often used interchangeably with 'stress' and 'load' testing. Ideally 'performance' testing (and any other 'type' of testing) is defined in requirements documentation or QA or Test Plans.
* usability testing - testing for 'user-friendliness'. Clearly this is subjective, and will depend on the targeted end-user or customer. User interviews, surveys, video recording of user sessions, and other techniques can be used. Programmers and testers are usually not appropriate as usability testers.
* install/uninstall testing - testing of full, partial, or upgrade install/uninstall processes.
* recovery testing - testing how well a system recovers from crashes, hardware failures, or other catastrophic problems.
* failover testing - typically used interchangeably with 'recovery testing'
* security testing - testing how well the system protects against unauthorized internal or external access, willful damage, etc; may require sophisticated testing techniques.
* compatability testing - testing how well software performs in a particular hardware/software/operating system/network/etc. environment.
* exploratory testing - often taken to mean a creative, informal software test that is not based on formal test plans or test cases; testers may be learning the software as they test it.
* ad-hoc testing - similar to exploratory testing, but often taken to mean that the testers have significant understanding of the software before testing it.
* context-driven testing - testing driven by an understanding of the environment, culture, and intended use of software. For example, the testing approach for life-critical medical
equipment software would be completely different than that for a low-cost computer game.
* user acceptance testing - determining if software is satisfactory to an end-user or customer.
* comparison testing - comparing software weaknesses and strengths to competing products.
* alpha testing - testing of an application when development is nearing completion; minor design changes may still be made as a result of such testing. Typically done by end-users or others, not by programmers or testers.
* beta testing - testing when development and testing are essentially completed and final bugs and problems need to be found before final release. Typically done by end-users or others, not by programmers or testers.
* mutation testing - a method for determining if a set of test data or test cases is useful, by deliberately introducing various code changes ('bugs') and retesting with the original test data/cases to determine if the 'bugs' are detected. Proper implementation requires large computational resources.
* Black box testing - not based on any knowledge of internal design or code. Tests are based on requirements and functionality.
* White box testing - based on knowledge of the internal logic of an application's code. Tests are based on coverage of code statements, branches, paths, conditions.
* unit testing - the most 'micro' scale of testing; to test particular functions or code modules. Typically done by the programmer and not by testers, as it requires detailed knowledge of the internal program design and code. Not always easily done unless the application has a well-designed architecture with tight code; may require developing test driver modules or test harnesses.
* incremental integration testing - continuous testing of an application as new functionality is added; requires that various aspects of an application's functionality be independent enough to work separately before all parts of the program are completed, or that test drivers be developed as needed; done by programmers or by testers.
* integration testing - testing of combined parts of an application to determine if they function together correctly. The 'parts' can be code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to client/server and distributed systems.
* functional testing - black-box type testing geared to functional requirements of an application; this type of testing should be done by testers. This doesn't mean that the programmers shouldn't check that their code works before releasing it (which of course applies to any stage of testing.)
* system testing - black-box type testing that is based on overall requirements specifications; covers all combined parts of a system.
* end-to-end testing - similar to system testing; the 'macro' end of the test scale; involves testing of a complete application environment in a situation that mimics real-world use, such as interacting with a database, using network communications, or interacting with other hardware, applications, or systems if appropriate.
* sanity testing or smoke testing - typically an initial testing effort to determine if a new software version is performing well enough to accept it for a major testing effort. For example, if the new software is crashing systems every 5 minutes, bogging down systems to a crawl, or corrupting databases, the software may not be in a 'sane' enough condition to warrant further testing in its current state.
* regression testing - re-testing after fixes or modifications of the software or its environment. It can be difficult to determine how much re-testing is needed, especially near the end of the development cycle. Automated testing tools can be especially useful for this type of testing.
* acceptance testing - final testing based on specifications of the end-user or customer, or based on use by end-users/customers over some limited period of time.
* load testing - testing an application under heavy loads, such as testing of a web site under a range of loads to determine at what point the system's response time degrades or fails.
* stress testing - term often used interchangeably with 'load' and 'performance' testing. Also used to describe such tests as system functional testing while under unusually heavy loads, heavy repetition of certain actions or inputs, input of large numerical values, large complex queries to a database system, etc.
* performance testing - term often used interchangeably with 'stress' and 'load' testing. Ideally 'performance' testing (and any other 'type' of testing) is defined in requirements documentation or QA or Test Plans.
* usability testing - testing for 'user-friendliness'. Clearly this is subjective, and will depend on the targeted end-user or customer. User interviews, surveys, video recording of user sessions, and other techniques can be used. Programmers and testers are usually not appropriate as usability testers.
* install/uninstall testing - testing of full, partial, or upgrade install/uninstall processes.
* recovery testing - testing how well a system recovers from crashes, hardware failures, or other catastrophic problems.
* failover testing - typically used interchangeably with 'recovery testing'
* security testing - testing how well the system protects against unauthorized internal or external access, willful damage, etc; may require sophisticated testing techniques.
* compatability testing - testing how well software performs in a particular hardware/software/operating system/network/etc. environment.
* exploratory testing - often taken to mean a creative, informal software test that is not based on formal test plans or test cases; testers may be learning the software as they test it.
* ad-hoc testing - similar to exploratory testing, but often taken to mean that the testers have significant understanding of the software before testing it.
* context-driven testing - testing driven by an understanding of the environment, culture, and intended use of software. For example, the testing approach for life-critical medical
equipment software would be completely different than that for a low-cost computer game.
* user acceptance testing - determining if software is satisfactory to an end-user or customer.
* comparison testing - comparing software weaknesses and strengths to competing products.
* alpha testing - testing of an application when development is nearing completion; minor design changes may still be made as a result of such testing. Typically done by end-users or others, not by programmers or testers.
* beta testing - testing when development and testing are essentially completed and final bugs and problems need to be found before final release. Typically done by end-users or others, not by programmers or testers.
* mutation testing - a method for determining if a set of test data or test cases is useful, by deliberately introducing various code changes ('bugs') and retesting with the original test data/cases to determine if the 'bugs' are detected. Proper implementation requires large computational resources.
Design predicates And Types of Calls
Design predicates
Design predicates are a method, invented by Thomas McCabe, to quantify the complexity of the integration of two units of software. Each of the four types of design predicates have an associated integration complexity rating. For pieces of code that apply more than one design predicate, integration complexity ratings can be combined.
The sum of the integration complexity for a unit of code, plus one, is the maximum number of test cases necessary to exercise the integration fully. Though a test engineer can typically reduce this by covering as many previously uncovered design predicates as possible with each new test. Also, some combinations of design predicates might be logically impossible.
Types of Calls
Unconditional Call
Unit A always calls unit B. This has an integration complexity of 0. For example:
unitA::functionA() {
unitB->functionB();
}
Conditional Call
Unit A may or may not call unit B. This integration has a complexity of 1, and needs two tests: one that calls B, and one that doesn't.
unitA::functionA() {
if (condition)
unitB->functionB();
}
Mutually Exclusive Conditional Call
This is like a programming language's switch statement. Unit A calls exactly one of several possible units. Integration complexity is n - 1, where n is the number of possible units to call.
unitA::functionA() {
switch (condition) {
case 1:
unitB->functionB();
break;
case 2:
unitC->functionC();
break;
...
default:
unitN->functionN();
break;
}
}
Iterative Call
In an iterative call, unit A calls unit B at least once, but maybe more. This integration has a complexity of 1. It also requires two tests: one that calls unit B once, and one test that calls it more than once.
unitA::functionA() {
do {
unitB->functionB();
} while (condition);
}
Combining Calls
Any particular integration can combine several types of calls. For example, unit A may or may not call unit B; and if it does, it can call it one or more times. This integration combines a conditional call, with its integration complexity of 1, and an iterative call, with its integration complexity of 1. The combined integration complexity totals 2.
unitA::functionA() {
if (someNumber > 0) {
for ( i = 0 ; i < someNumber ; i++ ) {
unitB->functionB();
}
}
}
Since the number of necessary tests is the total integration complexity plus one, this integration would require 3 tests. In one, where someNumber isn't greater than 0, unit B isn't called. In another, where someNumber is 1, unit B is called once. And in the final, someNumber is greater than 1, unit B is called more than once.
Design predicates are a method, invented by Thomas McCabe, to quantify the complexity of the integration of two units of software. Each of the four types of design predicates have an associated integration complexity rating. For pieces of code that apply more than one design predicate, integration complexity ratings can be combined.
The sum of the integration complexity for a unit of code, plus one, is the maximum number of test cases necessary to exercise the integration fully. Though a test engineer can typically reduce this by covering as many previously uncovered design predicates as possible with each new test. Also, some combinations of design predicates might be logically impossible.
Types of Calls
Unconditional Call
Unit A always calls unit B. This has an integration complexity of 0. For example:
unitA::functionA() {
unitB->functionB();
}
Conditional Call
Unit A may or may not call unit B. This integration has a complexity of 1, and needs two tests: one that calls B, and one that doesn't.
unitA::functionA() {
if (condition)
unitB->functionB();
}
Mutually Exclusive Conditional Call
This is like a programming language's switch statement. Unit A calls exactly one of several possible units. Integration complexity is n - 1, where n is the number of possible units to call.
unitA::functionA() {
switch (condition) {
case 1:
unitB->functionB();
break;
case 2:
unitC->functionC();
break;
...
default:
unitN->functionN();
break;
}
}
Iterative Call
In an iterative call, unit A calls unit B at least once, but maybe more. This integration has a complexity of 1. It also requires two tests: one that calls unit B once, and one test that calls it more than once.
unitA::functionA() {
do {
unitB->functionB();
} while (condition);
}
Combining Calls
Any particular integration can combine several types of calls. For example, unit A may or may not call unit B; and if it does, it can call it one or more times. This integration combines a conditional call, with its integration complexity of 1, and an iterative call, with its integration complexity of 1. The combined integration complexity totals 2.
unitA::functionA() {
if (someNumber > 0) {
for ( i = 0 ; i < someNumber ; i++ ) {
unitB->functionB();
}
}
}
Since the number of necessary tests is the total integration complexity plus one, this integration would require 3 tests. In one, where someNumber isn't greater than 0, unit B isn't called. In another, where someNumber is 1, unit B is called once. And in the final, someNumber is greater than 1, unit B is called more than once.
Cyclomatic complexity
Cyclomatic complexity is a software metric (measurement). It was developed by Thomas McCabe and is used to measure the complexity of a program. It directly measures the number of linearly independent paths through a program's source code.
The concept, although not the method, is somewhat similar to that of general text complexity measured by the Flesch-Kincaid Readability Test.
Cyclomatic complexity is computed using a graph that describes the control flow of the program. The nodes of the graph correspond to the commands of a program. A directed edge connects two nodes if the second command might be executed immediately after the first command.
Definition
M = E − N + 2P
where
M = cyclomatic complexity
E = the number of edges of the graph
N = the number of nodes of the graph
P = the number of connected components.
"M" is alternatively defined to be one larger than the number of decision points (if/case-statements, while-statements, etc) in a module (function, procedure, chart
node, etc.), or more generally a system.
Separate subroutines are treated as being independent, disconnected components of the program's control flow graph.
Alternative definition
v(G) = e − n + p
G is a program's flowgraph
e is the number of edges (arcs) in the flowgraph
n is the number of nodes in the flowgraph
p is the number of connected components
Alternative way
There is another simple way to determine the cyclomatic number. This is done by counting the number of closed loops in the flow graph, and incrementing that number by one.
i.e.
M = Number of closed loops + 1
where
M = Cyclomatic number.
Implications for Software Testing
·M is a lower bound for the number of possible paths through the control flow graph.
·M is an upper bound for the number of test cases that are necessary to achieve a complete branch coverage.
For example, consider a program that consists of two sequential if-then-else statements.
if (c1) {
f1();
} else {
f2();
}
if (c2) {
f3();
} else {
f4();
}
· To achieve a complete branch coverage, two test cases are sufficient here.
· For a complete path coverage, four test cases are necessary.
· The cyclomatic number M is three, falling in the range between these two values, as it does for any program.
The concept, although not the method, is somewhat similar to that of general text complexity measured by the Flesch-Kincaid Readability Test.
Cyclomatic complexity is computed using a graph that describes the control flow of the program. The nodes of the graph correspond to the commands of a program. A directed edge connects two nodes if the second command might be executed immediately after the first command.
Definition
M = E − N + 2P
where
M = cyclomatic complexity
E = the number of edges of the graph
N = the number of nodes of the graph
P = the number of connected components.
"M" is alternatively defined to be one larger than the number of decision points (if/case-statements, while-statements, etc) in a module (function, procedure, chart
node, etc.), or more generally a system.
Separate subroutines are treated as being independent, disconnected components of the program's control flow graph.
Alternative definition
v(G) = e − n + p
G is a program's flowgraph
e is the number of edges (arcs) in the flowgraph
n is the number of nodes in the flowgraph
p is the number of connected components
Alternative way
There is another simple way to determine the cyclomatic number. This is done by counting the number of closed loops in the flow graph, and incrementing that number by one.
i.e.
M = Number of closed loops + 1
where
M = Cyclomatic number.
Implications for Software Testing
·M is a lower bound for the number of possible paths through the control flow graph.
·M is an upper bound for the number of test cases that are necessary to achieve a complete branch coverage.
For example, consider a program that consists of two sequential if-then-else statements.
if (c1) {
f1();
} else {
f2();
}
if (c2) {
f3();
} else {
f4();
}
· To achieve a complete branch coverage, two test cases are sufficient here.
· For a complete path coverage, four test cases are necessary.
· The cyclomatic number M is three, falling in the range between these two values, as it does for any program.
Tuesday, January 20, 2009
Code coverage - white Box Technique
Code coverage is inherently a white box testing activity. The target software is built with special options or libraries and/or run under a special environment such that every function that is exercised (executed) in the program(s) are mapped back to the function points in the source code. This process allows developers and quality assurance personnel to look for parts of a system that are rarely or never accessed under normal conditions (error handling and the like) and helps reassure test engineers that the most important conditions (function points) have been tested.
Test engineers can look at code coverage test results to help them devise test cases and input or configuration sets that will increase the code coverage over vital functions. Two common forms of code coverage used by testers are statement (or line) coverage, and path (or edge) coverage. Line coverage reports on the execution footprint of testing in terms of which lines of code were executed to complete the test. Edge coverage reports which branches, or code decision points
were executed to complete the test. They both report a coverage metric, measured as a percentage.
Generally code coverage tools and libraries exact a performance and/or memory or other resource cost which is unacceptable to normal operations of the software. Thus they are only used in the lab. As one might expect there are classes of software that cannot be feasibly subjected to these coverage tests, though a degree of coverage mapping can be approximated through analysis rather than direct testing.
There are also some sorts of defects which are affected by such tools. In particular some race conditions or similar real time sensitive operations can be masked when run under code coverage environments; and conversely some of these defects may become easier to find as a result of the additional overhead of the testing code.
Code coverage may be regarded as a more up-to-date incarnation of debugging in that the automated tools used to achieve statement and path coverage are often referred to as “debugging utilities”. These tools allow the program code under test to be observed on screen whilst the program is executing, and commands and keyboard function keys are available to allow the code to be “stepped” through literally line by line. Alternatively it is possible to define pinpointed lines of code as “breakpoints” which will allow a large section of the code to be executed, then stopping at that point and displaying that part of the program on screen. Judging
where to put breakpoints is based on a reasonable understanding of the program indicating that a particular defect is thought to exist around that point. The data values held in program variables can also be examined and in some instances (with care) altered to try out “what if” scenarios. Clearly use of a debugging tool is more the domain of the software engineer at a unit test level, and it is more likely that the software tester will ask the software engineer to perform this. However, it is
useful for the tester to understand the concept of a debugging tool.
Test engineers can look at code coverage test results to help them devise test cases and input or configuration sets that will increase the code coverage over vital functions. Two common forms of code coverage used by testers are statement (or line) coverage, and path (or edge) coverage. Line coverage reports on the execution footprint of testing in terms of which lines of code were executed to complete the test. Edge coverage reports which branches, or code decision points
were executed to complete the test. They both report a coverage metric, measured as a percentage.
Generally code coverage tools and libraries exact a performance and/or memory or other resource cost which is unacceptable to normal operations of the software. Thus they are only used in the lab. As one might expect there are classes of software that cannot be feasibly subjected to these coverage tests, though a degree of coverage mapping can be approximated through analysis rather than direct testing.
There are also some sorts of defects which are affected by such tools. In particular some race conditions or similar real time sensitive operations can be masked when run under code coverage environments; and conversely some of these defects may become easier to find as a result of the additional overhead of the testing code.
Code coverage may be regarded as a more up-to-date incarnation of debugging in that the automated tools used to achieve statement and path coverage are often referred to as “debugging utilities”. These tools allow the program code under test to be observed on screen whilst the program is executing, and commands and keyboard function keys are available to allow the code to be “stepped” through literally line by line. Alternatively it is possible to define pinpointed lines of code as “breakpoints” which will allow a large section of the code to be executed, then stopping at that point and displaying that part of the program on screen. Judging
where to put breakpoints is based on a reasonable understanding of the program indicating that a particular defect is thought to exist around that point. The data values held in program variables can also be examined and in some instances (with care) altered to try out “what if” scenarios. Clearly use of a debugging tool is more the domain of the software engineer at a unit test level, and it is more likely that the software tester will ask the software engineer to perform this. However, it is
useful for the tester to understand the concept of a debugging tool.
System testing And Types
System testing (Testing the whole system) is actually done to the entire system against the Functional Requirement Specification(s) (FRS) and/or the System Requirement Specification (SRS). Moreover, the system testing is an investigatory testing phase, where the focus is to have almost a destructive attitude and test not only the design, but also the behaviour and even the believed expectations of the customer. It is also intended to test up to and beyond the bounds defined in the software/hardware requirements specification(s).
One could view System testing as the final destructive testing phase before user acceptance testing.
The following examples are different types of testing that should be considered during System testing:
· User interface testing
· Usability testing
· Performance testing
· Compatibility testing
· Error handling testing
· Load testing
· Volume testing
· Stress testing
· User help testing
· Security testing
· Scalability testing
· Capacity testing
· Sanity testing
· Smoke testing
· Exploratory testing
· Adhoc testing
· Regression testing
· Reliability testing
· Recovery testing
· Installation testing
· Idempotency testing
· Maintenance testing
· Accessibility testing, including compliance with:
o Americans with Disabilities Act of 1990
o Section 508 Amendment to the Rehabilitation Act of 1973
o Web Accessibility Initiative (WAI) of the World Wide Web Consortium (W3C)
Although different testing organizations may prescribe different tests as part of System testing, this list serves as a general framework or foundation to begin with
Testing Level - User acceptance testing(UAT)
4) User acceptance testing (UAT)
The same system testing done in the presents of the user is known as user acceptance testing. It s a black box testing usually done by the Test engineers.
User Acceptance testing occurs just before the software is released to the customer. The end-users along with the developers perform the User Acceptance Testing with a certain set of test cases and typical scenarios.
"A testing conducted to enable a user or customer to determine whether to accept a software project."
"Testing the system with the intent of confirming readiness of the product and customer acceptance"
"User acceptance testing is determining if software is satisfactory to an end-user or customer."
The same system testing done in the presents of the user is known as user acceptance testing. It s a black box testing usually done by the Test engineers.
User Acceptance testing occurs just before the software is released to the customer. The end-users along with the developers perform the User Acceptance Testing with a certain set of test cases and typical scenarios.
"A testing conducted to enable a user or customer to determine whether to accept a software project."
"Testing the system with the intent of confirming readiness of the product and customer acceptance"
"User acceptance testing is determining if software is satisfactory to an end-user or customer."
Testing Level - System Testing
3) System Testing
System Testing is third level of Testing In this level we check Functionality of application
"System testing concentrates on testing the complete system with a variety of techniques and methods. System Testing comes into picture after the Unit and Integration Tests."
"To verify and validate behaviors of the entire system against the original system objectives "
System level testing :
Once the application is deployed into the environment then if one performs testing on the system it is known as system level testing it is a black box testing and usually done by the test engineers.At this level of testing so many types of testing are done.
Some of those are
* Integration Testing
* Load Testing
* Performance Testing
* Stress Testing etc….
During system testing it costs even more time and may delay delivery. Finally, during operations it may cause anything from a nuisance to a system failure, possibly with catastrophic as an aircraft or an emergency service.
System Testing is third level of Testing In this level we check Functionality of application
"System testing concentrates on testing the complete system with a variety of techniques and methods. System Testing comes into picture after the Unit and Integration Tests."
"To verify and validate behaviors of the entire system against the original system objectives "
System level testing :
Once the application is deployed into the environment then if one performs testing on the system it is known as system level testing it is a black box testing and usually done by the test engineers.At this level of testing so many types of testing are done.
Some of those are
* Integration Testing
* Load Testing
* Performance Testing
* Stress Testing etc….
During system testing it costs even more time and may delay delivery. Finally, during operations it may cause anything from a nuisance to a system failure, possibly with catastrophic as an aircraft or an emergency service.
Big Bang Integration Approach
Once all the modules are ready at a time integrating them finally is known as big bang approach.
In this approach, all or most of the developed modules are coupled together to form a complete software system or major part of the system and then used for integration testing. The Big Bang method is very effective for saving time in the integration testing process. However, if the test cases and their results are not recorded properly, the entire integration process will be more complicated and may prevent the testing team from achieving the goal of integration testing.
In this approach, all or most of the developed modules are coupled together to form a complete software system or major part of the system and then used for integration testing. The Big Bang method is very effective for saving time in the integration testing process. However, if the test cases and their results are not recorded properly, the entire integration process will be more complicated and may prevent the testing team from achieving the goal of integration testing.
Hybrid Integration Approach
Hybrid approach is a mixed approach of both Top down and Bottom up approaches.
Bottom Up Integration
Bottom-up integration testing begins construction and testing with atomic modules (i.e. components at the lowest levels in the program structure). Because components are integrated from the bottom up, processing required for components subordinate to a given level is always available and the need for stubs is eliminated.
1. A Bottom-up integration strategy may be implemented with the following steps:
2. Low level components are combined into clusters that perform a specific software sub function.
3. A driver is written to coordinate test case input and output.
4. The cluster is tested.
Drivers are removed and clusters are combined moving upward in the program structure.
1. A Bottom-up integration strategy may be implemented with the following steps:
2. Low level components are combined into clusters that perform a specific software sub function.
3. A driver is written to coordinate test case input and output.
4. The cluster is tested.
Drivers are removed and clusters are combined moving upward in the program structure.
Top Down Integration
Top-down integration testing is an incremental approach to construction of program structure. Modules are integrated by moving downward through the control hierarchy, beginning with the main control module. Modules subordinate to the main control module are incorporated into the structure in either a depth-first or breadth-first manner.
1. The Integration process is performed in a series of five steps:
2. The main control module is used as a test driver and stubs are substituted for all components directly subordinate to the main control module.
3. Depending on the integration approach selected subordinate stubs are replaced one at a time with actual components.
4. Tests are conducted as each component is integrated.
5. On completion of each set of tests, stub is replaced with the real component.
6. Regression testing may be conducted to ensure that new errors have not been introduced.
1. The Integration process is performed in a series of five steps:
2. The main control module is used as a test driver and stubs are substituted for all components directly subordinate to the main control module.
3. Depending on the integration approach selected subordinate stubs are replaced one at a time with actual components.
4. Tests are conducted as each component is integrated.
5. On completion of each set of tests, stub is replaced with the real component.
6. Regression testing may be conducted to ensure that new errors have not been introduced.
Testing Level - Integration Testing
2) Integration Testing
"Testing two or more modules or functions together with the intent of finding interface defects between the modules/functions."
"Integration testing is a systematic technique for constructing the program structure while at the same time conducting tests to uncover errors associated with interfacing. The objective is to take unit tested components and build a program structure that has been dictated by design."
Once the modules are developing the developers will develop some interfaces and integrate the module with the help of those interfaces while integration they will check whether the interfaces are working fine or not. It is a white box testing and usually developers or white box testers perform it.
The developers will be integrating the modules in any one of the following approaches:
i) Top Down Approach (TDA)
In this approach the parent modules are developed first and then integrated with child modules.
ii) Bottom Up Approach (BUA)
In this approach the child modules are developed first and the integrated that to the corresponding parent modules.
iii) Hybrid Approach
This approach is a mixed approach of both Top down and Bottom up approaches.
iv) Big Bang Approach
Once all the modules are ready at a time integrating them finally is known as big bang approach.
By making unit testing for each module as explained above the process of integrated testing as a whole becomes simpler. This is because by correcting mistakes or bugs in each module the integration of all units as a system and testing process becomes easier. So one might think why the integration is testing needed. The answer is simple. It is needed because unit testing as explained test and assures correctness of only each module. But it does not cover the aspects of how the system would behave or what error would be reported when modules are integrated. This is done in the level of integration testing.
STUB
While integrating the modules in top down approach if at all any mandatory module is missing then that module is replace with a temporary program known as "STUB."
DRIVER
While integrating the modules in bottom up approach if at all any mandatory module is missing then that module is replace with a temporary program known as "DRIVER."
"Testing two or more modules or functions together with the intent of finding interface defects between the modules/functions."
"Integration testing is a systematic technique for constructing the program structure while at the same time conducting tests to uncover errors associated with interfacing. The objective is to take unit tested components and build a program structure that has been dictated by design."
Once the modules are developing the developers will develop some interfaces and integrate the module with the help of those interfaces while integration they will check whether the interfaces are working fine or not. It is a white box testing and usually developers or white box testers perform it.
The developers will be integrating the modules in any one of the following approaches:
i) Top Down Approach (TDA)
In this approach the parent modules are developed first and then integrated with child modules.
ii) Bottom Up Approach (BUA)
In this approach the child modules are developed first and the integrated that to the corresponding parent modules.
iii) Hybrid Approach
This approach is a mixed approach of both Top down and Bottom up approaches.
iv) Big Bang Approach
Once all the modules are ready at a time integrating them finally is known as big bang approach.
By making unit testing for each module as explained above the process of integrated testing as a whole becomes simpler. This is because by correcting mistakes or bugs in each module the integration of all units as a system and testing process becomes easier. So one might think why the integration is testing needed. The answer is simple. It is needed because unit testing as explained test and assures correctness of only each module. But it does not cover the aspects of how the system would behave or what error would be reported when modules are integrated. This is done in the level of integration testing.
STUB
While integrating the modules in top down approach if at all any mandatory module is missing then that module is replace with a temporary program known as "STUB."
DRIVER
While integrating the modules in bottom up approach if at all any mandatory module is missing then that module is replace with a temporary program known as "DRIVER."
Testing Level - Unit Testing
1) Unit Testing
"Unit testing is done for each module of the program to ensure the validity of each module. This type of testing is done usually by developers by writing test cases for each scenarios of the module and writing the results occurring in each step for each module. "
"If one performs testing on a unit then that level of testing is known as unit level testing. It is white box testing usually developers perform it.
Unit: -It is defined as a smallest part of an application."
"Unit testing is the process of testing a particular complied program, i.e., a window, a report, an interface, etc. independently as a stand-alone component/program. The types and degrees of unit tests can vary among modified and newly created programs. Unit testing is mostly performed by the programmers who are also responsible for the creation of the necessary unit test data. "
"Unit testing is done for each module of the program to ensure the validity of each module. This type of testing is done usually by developers by writing test cases for each scenarios of the module and writing the results occurring in each step for each module. "
"If one performs testing on a unit then that level of testing is known as unit level testing. It is white box testing usually developers perform it.
Unit: -It is defined as a smallest part of an application."
"Unit testing is the process of testing a particular complied program, i.e., a window, a report, an interface, etc. independently as a stand-alone component/program. The types and degrees of unit tests can vary among modified and newly created programs. Unit testing is mostly performed by the programmers who are also responsible for the creation of the necessary unit test data. "
This is a typical scenario of Manual Unit Testing activity-
A Unit is allocated to a Programmer for programming. Programmer has to use ‘Functional Specifications’ document as input for his work.
Programmer prepares ‘Program Specifications’ for his Unit from the Functional Specifications. Program Specifications describe the programming approach, coding tips for the Unit’s coding.
Using these ‘Program specifications’ as input, Programmer prepares ‘Unit Test Cases’ document for that particular Unit. A ‘Unit Test Cases Checklist’ may be used to check the completeness of Unit Test Cases document.
‘Program Specifications’ and ‘Unit Test Cases’ are reviewed and approved by Quality Assurance Analyst or by peer programmer.
The programmer implements some functionality for the system to be developed. The same is tested by referring the unit test cases. While testing that functionality if any defects have been found, they are recorded using the defect logging tool whichever is applicable. The programmer fixes the bugs found and tests the same for any errors.
Stubs and Drivers
A software application is made up of a number of ‘Units’, where output of one ‘Unit’ goes as an ‘Input’ of another Unit. e.g. A ‘Sales Order Printing’ program takes a ‘Sales Order’ as an input, which is actually an output of ‘Sales Order Creation’ program.
Due to such interfaces, independent testing of a Unit becomes impossible. But that is what we want to do; we want to test a Unit in isolation! So here we use ‘Stub’ and ‘Driver.
A ‘Driver’ is a piece of software that drives (invokes) the Unit being tested. A driver creates necessary ‘Inputs’ required for the Unit and then invokes the Unit.
A Unit may reference another Unit in its logic. A ‘Stub’ takes place of such subordinate unit during the Unit Testing. A ‘Stub’ is a piece of software that works similar to a unit which is referenced by the Unit being tested, but it is much simpler that the actual unit. A Stub works as a ‘Stand-in’ for the subordinate unit and provides the minimum required behavior for that unit.
Programmer needs to create such ‘Drivers’ and ‘Stubs’ for carrying out Unit Testing.
Both the Driver and the Stub are kept at a minimum level of complexity, so that they do not induce any errors while testing the Unit in question.
Example - For Unit Testing of ‘Sales Order Printing’ program, a ‘Driver’ program will have the code which will create Sales Order records using hard coded data and then call ‘Sales Order Printing’ program. Suppose this printing program uses another unit which calculates Sales discounts by some complex calculations. Then call to this unit will be replaced by a ‘Stub’, which will simply return fix discount data.
Unit Test Cases
It must be clear by now that preparing Unit Test Cases document (referred to as UTC hereafter) is an important task in Unit Testing activity. Having an UTC, which is complete with every possible test case, leads to complete Unit Testing and thus gives an assurance of defect-free Unit at the end of Unit Testing stage. So let us discuss about how to prepare a UTC.
Think of following aspects while preparing Unit Test Cases –
v Expected Functionality: Write test cases against each functionality that is expected to be provided from the Unit being developed.
e.g. If an SQL script contains commands for creating one table and altering another table then test cases should be written for testing creation of one table and alteration of another.
It is important that User Requirements should be traceable to Functional Specifications, Functional Specifications be traceable to Program Specifications and Program Specifications be traceable to Unit Test Cases. Maintaining such traceability ensures that the application fulfills User Requirements.
v Input values:
o Every input value: Write test cases for each of the inputs accepted by the Unit.
E.g. If a Data Entry Form has 10 fields on it, write test cases for all 10 fields.
o Validation of input: Every input has certain validation rule associated with it. Write test cases to validate this rule. Also, there can be cross-field validations in which one field is enabled depending upon input of another field. Test cases for these should not be missed.
E.g. A combo box or list box has a valid set of values associated with it.
A numeric field may accept only positive values.
An email address field must have ampersand (@) and period (.) in it.
A ‘Sales tax code’ entered by user must belong to the ‘State’ specified by the user.
o Boundary conditions: Inputs often have minimum and maximum possible values. Do not forget to write test cases for them.
e.g. A field that accepts ‘percentage’ on a Data Entry Form should be able to accept inputs only from 1 to 100.
o Limitations of data types: Variables that hold the data have their value limits depending upon their data types. In case of computed fields, it is very important to write cases to arrive at an upper limit value of the variables.
o Computations: If any calculations are involved in the processing, write test cases to check the arithmetic expressions with all possible combinations of values.
v Output values: Write test cases to generate scenarios, which will produce all types of output values that are expected from the Unit.
e.g. A Report can display one set of data if user chooses a particular option and another set of data if user chooses a different option. Write test cases to check each of these outputs. When the output is a result of some calculations being performed or some formulae being used, then approximations play a major role and must be checked.
v Screen / Report Layout: Screen Layout or web page layout and Report layout must be tested against the requirements. It should not happen that the screen or the report looks beautiful and perfect, but user wanted something entirely different! It should ensure that pages and screens are consistent.
v Path coverage: A Unit may have conditional processing which results in various paths the control can traverse through. Test case must be written for each of these paths.
v Assumptions: A Unit may assume certain things for it to function. For example, a Unit may need a database to be open. Then test case must be written to check that the Unit reports error if such assumptions are not met.
v Transactions: In case of database applications, it is important to make sure that transactions are properly designed and no way inconsistent data gets saved in the database.
v Abnormal terminations: Behavior of the Unit in case of abnormal termination should be tested.
v Error messages: Error messages should be short, precise and self-explanatory. They should be properly phrased and should be free of grammatical mistakes.
UTC Document
Given below is a simple format for UTC document.
| Test Case No. | Test Case purpose | Procedure | Expected Result | Actual result |
| ID which can be referred to in other documents like ‘Traceability Matrix’, Root Cause Analysis of Defects etc. | What to test | How to test | What should happen | What actually happened? This column can be omitted when Defect Recording Tool is used. |
Note that as this is a sample, we have not provided columns for Pass/Fail and Remarks.
Example:
Let us say we want to write UTC for a Data Entry Form below:
ITEM MASTER FORM
Item Number : ------------------------------
Item Name: ------------------------------
Item Price : --------------------------------
Given below are some of the Unit Test Cases for the above Form:
| Test Case purpose | Procedure | Expected Result | Actual result | |
| 1 | Item no. to start by ‘A’ or ‘B’. | 1.Create a new record. 2.Type Item no. starting with ‘A’. 3.Type item no. starting with ‘B’. 4.Type item no. starting with any character other than ‘A’ and ‘B’. | 2,3. Should get accepted and control should move to next field. 4. Should not get accepted. An error message should be displayed and control should remain in Item no. field. | |
| 2. | Item Price to be between 1000 to 2000 if Item no. starts with ‘A’. | 1.Create a new record with Item no. starting with ‘A’. 2.Specify price <> 3.Specify price >2000. 4.Specify price = 1000. 5.Specify price = 2000. 6.Specify price between 1000 and 2000. | 2,3.Error should get displayed and control should remain in Price field. 4,5,6.Should get accepted and control should move to next field. | |
| | | | | |
UTC Checklist
UTC checklist may be used while reviewing the UTC prepared by the programmer. As any other checklist, it contains a list of questions, which can be answered as either a ‘Yes’ or a ‘No’. The ‘Aspects’ list given in Section 4.3 above can be referred to while preparing UTC checklist.
e.g. Given below are some of the checkpoints in UTC checklist –
- Are test cases present for all form field validations?
- Are boundary conditions considered?
- Are Error messages properly phrased?
Defect Recording
Defect Recording can be done on the same document of UTC, in the column of ‘Expected Results’. This column can be duplicated for the next iterations of Unit Testing.
Defect Recording can also be done using some tools like Bugzilla, in which defects are stored in the database.
Defect Recording needs to be done with care. It should be able to indicate the problem in clear, unambiguous manner, and reproducing of the defects should be easily possible from the defect information.
Conclusion
Subscribe to:
Posts (Atom)
RSS Feed (xml)
