Different kind of tests

27 10 2008

Below is a list of types of tests that can be implemented by a team of recipe. I will discuss this article through the famous « white and black boxes ».

1.1-white box
Test type « white box » or test structural visibility of the component under test, test games are produced by analyzing the source code. Tests such as « white boxes » are called so because one must read and enter the code to design and run the tests (or otherwise designated as « put your hands in the dirty »).
Some examples of tests « white box » all roads have been travelled at least once, the true / false have been validated, the loops are implemented correctly. But the limits are set correctly; the internal data structures are valid and if there is no memory leak, etc.
1.1.1-Code review
The code review is a systematic review of the source code of computer software, the goal is to find anomalies to correct errors in design to improve the quality and safety software.
It is therefore to read the code, which involves reviewing thousands of lines in Java or C for example. To do this there are many tools to do almost automatically.
Thus, it inserts into the tool coding standards and management. Then, the tool parse (trades language describing the review of the code, for example), applications to display a report highlighting the deficiencies.

The review code becomes increasingly an integral part in any development process and software including agile methods known as Extreme programming.

Examples of such activities during a code review may be checking rules programming or verification of compliance of the Code in relation to document detailed engineering design.

1.1.2-Unitarian Tests
They independently test the various elements that make up the system or a portion of a program.
They are used to demonstrate that each module (class, loop, etc.). Performs any service provided and only this function. Can be distinguished in these tests of logic tests (search for error, checking the correct sequence of branches covered). As the test calculation (verification of the results of calculations, performance, accuracy algorithms).

The most widely used to perform unit testing is JUnit (Java context).

1.1.3-integration tests – with « vision developer »

They are testing the various elements that make up the system and their interactions between them.
They are used to demonstrate that each module (class, loop, etc.). Performas any service provided and only this function.

Figure: Integration Tests
1.1.4-test charges

To begin you must know that there are still several meanings in « testing charges for customers or consultants. According to the definition change. They are listed in order to provide clarification.
During performance tests (response time, duration of treatment) must verify the performance of the software meet the requirements in different conditions (normal or gradient):
Example: if it is anticipated that a website is accessed simultaneously by 500 users, the pages are displayed in less than 2 seconds.

During load a test is to measure the behaviour of the system and to deal with an operating environment limits by simulating several simultaneous sessions and / or volumetric data to monitor the maintenance performance.

The surcharge on tests intended to understand the functioning and system failures when overloaded (beyond the limits of charge).

Tests with a degraded mode verify the stability of the reaction outside the normal cases of use (duplicates, empty, empty selection, the wrong format…).

My experience returns show that 100% of the load tests are treated as performance benchmarks. These are described below.

As a general test performance and expenses to meet a number of questions like « What are the risks of performance problems in production? «  » Is what applications and infrastructure support the expected number of users? « How to measure the load / quality of service rendered in connection with the means and goals? « , Or » the response time of the application for No users are conforming to specifications? « .
Tests charges can be several times the life cycle of projects such as:
at the beginning of the project to analyze the behavior of a solution: software to integrate, prototype, to identify the limits of the solution, the potential charge and the hardware scale depending on the charge assessed in time (capacity planning), ongoing project to test the dynamic behavior of the solution in production on the problems of inter-locking, consumption of resources and memory management, end-of-project to validate the final solution to verify its performance characteristics and identify the real capacity and attitudes to climb up and peak use. These tests also maximize the final architecture by adjusting the parameters systems, finally during maintenance to ensure non-regression performance characteristics of the solution and identify its capacity and in attitudes about changing mounted charge and use pics.

A typical process of load testing involves four to five phases with pre-requisite to verify.
The pre-requisite to check about the campaign load test, which can only be achieved on the basis of an application stable and validated during a phase of functional and technical preliminary.
Prior to conducting performance tests of the application, the following must be carried out and validated: the machine should be available with provision of access (user / password), generic or not, on the application environment and software (NT, Oracle, permits networks, etc.). the application installed and available (technically, operationally and in terms of integration and recipe) with provision of access (user / password) and data set. But we must also have an application in running and stable, in a configuration iso-production of preference and / or close to reality and links with other applications in running and available.

The nature and volume of data base must be identical and / or close to production. Moreover, the application must be accessible from the platform load testing.
The four phases of tests are:

1 – Pre-study and launch:

The first step before any use of a tool (HP Mercury LoadRunner for example) is to achieve a phase of preparation. This phase is divided into several stages which are a meeting of initialization / Shift and delivery of the specification test of the campaign.

2 – Preparation of testing and development scenarios:

This step addresses the items to install in order to establish the test environment.
The development phase the construction of test scripts and scenarios specified in the previous phase. This phase is broken down as follows:

  • registration scripts,
  • setting scripts,
  • validation scripts by the MOE / MOA (sometimes)
  • running scripts,
  • calibration scripts.

3 – Campaign tests: Implementation – Analysis Report – Synthesis, Correction, Test confirmation:

This phase is an iterative process that allows you to run scenarios and analyze the results. It breaks down as follows: Creation of the test, and Shooting metric collection and analysis of results.

4 – Review:

Finally, the results describe the testing, analysis, improvements in the implementation and status of goals. This report is generated after the execution of all tests.
The test cycle may be iterative several times to take into account changes in the system.

1.1.5-Safety Tests

With the arrival of Web-type applications were developed safeties testing. The aim is to check security systems: data encryption, authentication tests, empowerment, data integrity, accessibility, access competitors…


Usability testing and verification at the Good Operability (Vabe) are tests that monitor the technical architecture requirements. Shown is the illustration in the world of French telecom:
It includes in the diagram below that test usability begin mobile phone from a client to finish at the entrance of the SI provider. Through all layers networks, supervision, etc.
There are also some performance tests that are executed.

1 – Types of test
Figure: Scope of usability tests in a French telecom operator
Figure (figure to come): Activities of type test usability in a French telecom operator.
Some other examples of test part of those usability:

  • analysis of holding file,
  • establishment of the configuration platform, configuration scheduling JCL,
  • monitoring of operating
  • the establishment of Job scheduling and data flow
  • Oversight (Tivoli Unix, MVS AutoSys)
  • scenario chronology
  • set up batch case alert
  • simulation of operating records, alert, log,
  • audit logs,

• tests backup, restore, hot, cold, incremental, weekly, etc..
This type of test addresses many peculiarities and will be the subject of a forthcoming article.

1.2-black box

Tests such as « black box » or functional test does not require visibility of the component. Entries and results are expressed in terms of behavior of the software component of an external point of view, regardless of the internal structure of the component.

The black box tests are related to functional specifications. Regardless of the internal structure of the application (unlike the white box testing), they validate the functionality of the application. As the internal workings of the application is not known, the designer of sets of test should define the expected behavior (Expected Result – RA) and check if the application is consistent with this behavior (outcome – RO).
Tests type black box functional tests are implemented in phase recipe, they are designed to search for or validate compliance with the requirements, functions incorrect or missing. But inconsistencies in the interface, error data access and the initial conditions or final of a function.

1.2.1-Functional Tests

The functional (simply called recipe) is to receive a software and to verify the proper functioning before being put into service as well as its compliance with what was requested in the specifications.
The functional tests verify that the software works properly in the context of business-users during the functional, it does not programming software, but his behaviour and his functional Ergonomic (MMI and navigation) and or that the specifications (and any other media such as user manuals, data modeling…).
The tests lead to simulate the operation of the software through use case. It is necessary to have concrete scope and data to simulate a real context of use (data sets, test environment).
The tests verify aspects of the business (business function) with features non-conforming or missing the operation and management rules.
The functional tests also take into account ergonomics. These behaviours users (limit values, values unauthorized …), the preparation and presentation screens or navigation rules, etc.

1.2.2 Tests of non-regression

After a correction or a change that should be re-tested to ensure that the change has not deteriorated (done backwards) running the software and caused abnormalities on other modules not tested?
Any re-testing would be too costly and not test that changes to the component involves many risks.
We must therefore find a compromise and develop a strategy for non-regression (or TNR). These tests should not be provided regression after any modification of software to ensure that existing functionality was not affected by the changes. It is essential that the existing system continues to operate with the same quality level as the previous version.
Practice test for non regression means that the rest of the application has not lost functionality or reliability due to the addition of new functions or modification of existing functions. They are to determine whether defects were introduced into the application in the implementation of the new feature. They can also be used when changes in the environment of the application.
The non-regression tests are applicable to all phases of testing. Indeed, the prediction errors to determine the scope and extent of tests and monitor the system status changed.
If the non-regression tests with the different stages V, they are particularly suited to the latter stages. Thus, they are usually carried out under testing system recipe or integration.

Figure: Illustration of what the non-regression tests
Any information system is in perpetual evolution. But if each new change is validated it is not necessarily throughout the application in which it operates. This leads to anomalies, sometimes blocking (or butterfly effect). Tests of non-regression (often equipped) can increase the coverage tests; the tests will produce, reduce expenses and tested to optimize the execution time.

Example – ARIANE 501

On 4 June 1996 launch of Ariane 501 and a few seconds after the launch of self destruction is pronounced. Late ARIANE 501 (about € 500 million).

After analysis it is that bad choices were made when defining the strategy of non-regression tests. Indeed, components of Ariane 4 were tested but not used because « They worked perfectly before! ».
(Source IEEE Computer 01/97)
Example – EOM E_F draft xxx

1.2.3-case studies to illustrate:

Strategic Issues: Market opening in 2007 – to the growing competence of the team – define the methodology and tool dedicated to recipe.
Director: Audit activities recipe EOM (fundamental project, process and means of revenue on VABF)> Recommendations> action plan> Training> Accompaniments project manager and analyst.
Gain score: Training and support its teams – professional activities of revenue – increased control of quality – steering dedicated to the recipe – respect for commitments integrators
Technologies: Portal – Software L_r – Batch – Flow – Business Intelligence (BO)
Exposure of the case: Closer to home, in 2007, during a mission in E_F advice I have implemented the strategy of non-regression tests next.

Figure: lie in the various stages of testing
During the audit phase here is what I advocated a strategy in two phases located at the design stage of testing. It was to train and raise the competence in team activities revenue of the EOM on non regression testing. To do this he had to be put in place a strategy in 2 times (ownership gradually facility): a non-regression manual and automated non-regression.
No manual regression:
It took gather plans tests of other players as the group (inclusive type TMA) and the MOA. Then write those of the EOM collecting information:

  • Tests of non-regression of the grouping,
  • Tests of non-regression of the MOA.

It took write the summary and analysis of information on the concepts of non-regression tests EOM via:

  • testing features and flows paramount,
  • tests of the MMI L_R (menu …)

• testing load curves, charts, reports.
No automated regression:

Figure: Test Automation of non-regression – « Royal Way »
Once the non-regression testing an advanced manual, you can automate these tests. Through a 5-day study on the feasibility of such automation.
Then (after selection of robots and validation of client), said a pilot using both tests not written regressions upstream can be defined and played.
Then, after reading and analysis of test reports this phase of automation has been validated.

  • cell MOE: no recipe book for the NRR – TNR played « on the fly »
  • MOA cell: a recipe booklet with 3 scenarios but very incomplete,

• crew Métiers: none of the existing – TNR played « on the fly. »
After the audit strategy Tests No Regression (TNR) was used to establish a workshop. Then, following the workshop with pilots and cell E_F EOM the solution was to use the recipe books of the MOA, to enrich and organize workshops with the crew Métiers to finalize specifications Recipe of TNR.
In collaboration with the MOE cell we developed a table listing the benefits to implement this strategy TNR. Benefits for 3 teams. This list has also been used to communicate / sell « the TNR to various actors.

1.2.4-testing integration with logic « recetteurs »

In the event of changes to at least two applications, it is appropriate not only to test each of them, but also to verify that communications (upstream / downstream) provided still work (or, d A new relationship, that it is operational).
From the perspective of the method, achieving a recipe calls for integration of information on each application must be amended prior to be functional: one speaks fluently of functional « unit » and basic concepts outlined in the guide are applicable to this case.
Then he must clarify the relationship between applications, carried out a second time, introduces a level of complexity that requires a strategy and specific (coordination, monitoring where appropriate, suitable platform…) and even setting up a dedicated tool for framing, the ‘strategy of integration « . This allows sharing between different actors in the recipe key elements of the project constraints (planning, resources, goals), description of the platform, timing of work, etc..
Finally, the issue of integration strategy must contain scenarios and test files « shared » needed to verify the integration and the precise timetable for their passage. Games common test should be made to verify the chain.
They are to assemble several units and testing errors interactions between these units (and possibly to detect errors not detected burn unit level).
Special 1.2.5-test software
The test software such as ERP (or PGI) are special because part of testing the most complex to implement. For the following reasons as the need for identification requirements, business objects and rules. As the impact from one version to another, the recurring tests, the need for non-regression testing or interfacing with the SI system.

But also complex because there are many specific development, migration / recovery data SI history to the package, volumetric issues, platforms owner more or less open.
The complexity is also due to the richness functional thrust (predefined process trades and adapted to the job), a specific setting or developments complementary to each company, a process of development outside standard.

Sometimes, complexity from differentiations to be made between standard and specific. This level of development and testing. Indeed, developments standards are often part of a package already defined with the customer, while developments and specific tests are subject to contract amendments.
Finally, the complexity of software testing is problematic due to flow management inter-application to problems of configuration management and the difficulties of controlling the data.
Apart from the above-mentioned problems there are also those related to the design of scenarios and completeness. Below is an example scenario following the logic of an « end user ».

Figure: Example of a test of a module R / 3
The software tests are complex and require a first team experience on one of the modules of the ERP target.
Take an example to illustrate from a mission from an insurer on a SAP Finance module. The process established to identify and prepare the tests was: it was necessary to identify the main processes (and business applications) and test scenarios associated, and then identify the management rules, study the messages impacting and identify SAP transactions (or transport order) by flows and processes (the study of processes and prototypes developed during the design of the application).
Then it took specify the test scenarios and test sheets, bringing SAP scenarios provided in standard methodology ASAP (light testing methodology own SAP), define the scenarios used and specify the test case.
Finally, it took analyze coverage tests via the matrix of test cases and analyze and specify the data needed.
I will elaborate in a forthcoming article the strategy for carrying out tests on ERP.


…Quels sont ces fameux critères d’arrêts des tests ?

20 10 2008

C’est une question souvent abordée lors des formations aux tests, parfois posée par les clients et jamais abordée par les équipes de testeurs lors des projets ! Étrange n’est-ce pas ?

Les critères d’arrêt des tests ne sont surtout pas à confondre avec la question « A-t-on assez testé ? ».  Car ceci est un autre sujet concernant la couverture des tests et leurs exhaustivité (entre autre).

Définition des critères d’arrêt des tests pour ISTQB :
« Critère de sortie : l’ensemble des conditions génériques et spécifiques, convenues avec les responsables, pour permettre de terminer officiellement un processus. L’objectif d’un critère de sortie est d’éviter qu’une tâche ne soit considérée comme achevée alors qu’il y a encore des parties de cette tâche qui n’ont pas été terminées. Les critères de sortie sont utilisés dans le test pour faire des comptes rendus et pour planifier l’arrêt du test. [D’après Gilb et Graham] ».

Certes cette définition est peu claire, et c’est pourquoi je tente de lister les critères les plus fréquents :

Critères d’arrêt commun à tous les types de tests :

  • Tous les scénarios (SCN) ont été exécutés (performance, fonctionnels, etc.)
  • Tous les dossiers, fiches et cas de test ont été joués avec succès
  • Toutes les anomalies critiques ont été résolues et les Tests de Non-Régressions (TNR) déroulés
  • Votre client à signer le PV de recette, le bilan présenté et il vous reste à ranger vos affaires
  • Après 252 livraisons l’intégrateur est incapable de corriger et le client décide d’arrêter le projet
  • Les plateformes, environnements, réseaux, etc. ne sont plus disponible
  • Atteinte du taux d’anomalie maximal découverte pendant les campagnes (définition et atteinte d’un seuil préalablement défini). Ce type de critère est souvent mis en œuvre chez les clients ayant un haut niveau de maturité des activités de test ou imposant des contraintes contractuelles très forte. Cela peut être également réparti par domaine fonctionnels. Exemple :
  • Arrêt des tests si détection de plus de 50 anomalies majeures
  • Épuisement des ressources des tests :
  • Fin du budget
  • Départ des ressources humaines
  • Fin de la durée / charge prévue
  • Véritable épuisement physique des ressources
  • Les SCN de recevabilité / testabilité / Pré-requis sont bloqués ou en NOK

Critères spécifique à certains types de test :
Techniques :

  • Tous les connecteurs et « jobs » (cela désigne : les jobs, jobstreams et les batchs) planifiés ont été lancés et tournent

Lecteurs, si tu en connais d’autre je te propose de nous en faire part afin d’en faire dons à la communauté des testeurs et teuses. Merci.