The TelQA Testing Series

This article is part of a series of documents which cover the basics of performance testing. See www.telqa.com for other articles in this series and additional downloadable testing resources.

 

Overview

In the previous article we discussed performance testing aims. Now we will consider what types of performance test can be used to achieve these aims.

 

Typically, performance testing is split into two main categories:

•  Load testing – applying a realistic load to the application while monitoring a range of performance criteria
• Stress testing – applying an abnormally high load to the application in order to test the application to its breaking point

Within these subcategories there are a number of different forms of performance tests which are determined by the test duration, the load application strategy and how the load is removed.

 

 

Load Testing

Load testing involves applying a realistic load to a web application such that the behaviour of the application can be monitored under both normal and expected peak load conditions. In other words, load testing is designed to simulate the anticipated production load. The duration for which the load is applied must also be representative of the expected load.

 

Where loads are applied over a prolonged period of time, the load test is often additionally classified as an endurance test. Endurance testing is typically used to test for accumulative effects which cause performance degradation. This degradation could result in reduced throughput levels or increased response times. Typically the test would involve the monitoring of a range of performance parameters which would be used to determine the cause of any performance related problems.

 

 

Stress Testing

Stress testing involves applying an abnormally high load to a web application. This type of testing can be used to determine the behaviour of the application during extreme load situations as the application is usually loaded to breaking point. These tests are often designed to determine the conditions required for an application to fail, the nature of the failure, and any early warning signs that can be monitored to warn of an impending failure.

 

 

Types of Load Application

So far in this article we have considered the two types of performance tests commonly carried out during both development phase testing and system phase testing. In each case we have talked about load application, but we have not yet covered how the load is applied. Since the objective of the test is to predict the behaviour of the application during production, loads must be applied in a way that is representative of the anticipated production load. This means that large test loads cannot simply be instantaneously added and removed, as this is not reflective of a real-life scenario. In this section of the article we will talk about some of the different ways load can be applied to an application.

 

Ramp Up/Ramp Down

Ramping up a performance test load (virtual users – VUs) typically involves gradually increasing the number of VUs to the point at which the application is running at a specified capacity. In practice, the virtual users are added in steps over a period of time. Ramping down virtual users means gradually removing the load on the application in steps over time.

 

The step size and the time between steps should be determined by the anticipated production load. The following charts show a number of different approaches which may be used to simulate real-life production loads.

 

Spike Testing

Spike testing involves subjecting the application under test to load volumes that repeatedly increase beyond anticipated production loads for short periods of time i.e. spiking virtual user activity suddenly.

 

 

The TelQA Testing Series

 

Part 1: The Performance Testing Process

Part 2: Performance Testing Aims

Part 3: Types of Performance Testing

Part 4: Performance Test Environments

Part 5: Performance Test Inputs and Outputs

Part 6: Capturing User Interactions

Part 7: Modelling Scripts

Part 8: Selecting Performance Monitors

Part 9: Performance Test Creation

Part 10: Configuring Performance Test Environments

Part 11: Verifying Performance Test Environments

Part 12: Running a Performance Test

Part 13: Analysing Performance Data

Part 14: Using Performance Data for Tuning and Bottleneck Removal

 

A downloadable pdf of this article is available here.

 

The TelQA Testing Series

This article is part of a series of documents which cover the basics of performance testing. See www.telqa.com for other articles in this series and additional downloadable testing resources.

 

Overview

This article provides you with an introduction to determining the aims and objectives of your performance tests.

 

During the course of this article we will look at the performance testing aims for:

 

• System testing – performance testing for a complete system/application
• Development phase testing – performance tests that support application development

System Testing

To begin with, let’s first consider performance tests which target a complete system. We can split the performance testing aims into four broad categories:

 

• Quantifying the overall performance level of a system
• Comparing the performance of a system against known performance thresholds (e.g. Service Level Agreements)
• Assisting with performance tuning or the removal of performance bottlenecks
• Assessing system scalability

Whilst it is possible to achieve all four of these aims with a single set of tests which quantify the full system performance, in practice the time and resources required to design and implement a full system performance analysis are unlikely to be justified when only specific performance related data types are required.

 

The key point to note is that the aim of the test should be used to determine the type of applied performance test, i.e. the test development process should focus on matching the type of test to the required aims and not try to provide generic performance tests which attempt to quantify the entire system with a limited number of tests.

 

Where the requirements involve a comparison with known performance thresholds, the scope of the test can be restricted and hence the test development and deployment time can be significantly reduced. However, the results of these types of tests typically provide no indication of the overall system performance.

 

Detection of performance bottlenecks may involve very specific testing using system loads which are unrepresentative of the expected production environment. Again, this type of test will provide little or no indication of the overall system performance.

 

System scalability tests are very similar to tests which are designed to quantify the overall system performance, but may use different loads which are not representative of the current production loads but are intended to be representative of future load types.

 

Development Phase Testing

The aims of a performance testing during the application development phase can be broadly categorised as:

 

• Feasibility testing for in-house or third-party hardware and software
• Unit testing
• Integration testing

Performance testing carried out as part of a feasibility test will typically use specific loads. For example, when selecting an appropriate database prior to system development, a performance test would be carried out to determine the ability of the database to provide the required level of performance. This type of test would typically utilise database specific communications which, whilst being representative of the intended use, would not necessarily be the communications used in the finished system.

 

Unit testing may utilise performance tests to determine whether individual units (or parts) of the application perform to a specified level. Alternatively a performance test may be intended to determine the overall performance of the unit prior to integration testing.

 

Performance tests are also commonly used as part of integration testing. The aim of this testing is to determine whether the performance of the individual units are affected when one or more units are integrated.

 

The TelQA Testing Series

 

Part 1: The Performance Testing Process

Part 2: Performance Testing Aims

Part 3: Types of Performance Testing

Part 4: Performance Test Environments

Part 5: Performance Test Inputs and Outputs

Part 6: Capturing User Interactions

Part 7: Modelling Scripts

Part 8: Selecting Performance Monitors

Part 9: Performance Test Creation

Part 10: Configuring Performance Test Environments

Part 11: Verifying Performance Test Environments

Part 12: Running a Performance Test

Part 13: Analysing Performance Data

Part 14: Using Performance Data for Tuning and Bottleneck Removal

 

A downloadable pdf of this article is available here.

 

The TelQA Testing Series

This article is part of a series which cover the basics of performance testing. See www.telqa.com for other articles in this series and additional downloadable testing resources.

 

Overview

This article presents a general testing strategy that will help you create and deploy successful performance tests. Although testing methods must always be flexible enough to adapt to individual projects, having a general structure can be very helpful for both experienced testers and those just getting started.

 

The Performance Testing Process

A typical performance test consists of four main stages:

 

1. Test Planning

• Determine what you want to achieve from the test
• Define the type of test(s) you are going to perform
• Select a suitable test environment
• Specify inputs (user types and number of users) and expected outputs (performance data collection, page timers, transaction timers etc.)

2. Test Development

• Capture realistic user interaction for each user type and save as editable scripts.
• Model captured scripts to create reusable virtual users (VUs)
• Select appropriate performance monitors for target application
• Create test using the defined proportion of VU types. The test should also include realistic load application, delays etc.

3. Test Deployment

• Configure test environment
• Verify correct operation of test environment
• Run test
• Clean up the test environment

4. Results Analysis

• Analyse captured performance data to determine system performance level
• Use captured performance data to assist in performance tuning or removing performance bottlenecks.

Let’s have a look at these stages in a bit more detail.

 

1. Test Planning

 

1.1 What do you want to achieve from the test?

Determining the purpose of the test is a crucial part of the overall test planning, development and deployment process. Tests which are intended to demonstrate a level of acceptable performance may differ significantly from tests which are to be used as part of scalability planning.

 

1.2 What type of tests do you want to perform?

The type of test will be determined by what you want to achieve. Broadly speaking, there are two main types of performance test, load tests and stress tests. Within these areas there is a large range of more focused tests which target specific aspects of application performance.

 

1.3 Selecting a suitable test environment

The test environment should be representative of the production environment where possible. Any performance constraints caused by the test environment must be taken into the account during the results analysis stage.

 

1.4 Inputs versus outputs

Prior to developing the test it is important to define the user types and the proportion of each user type which constitute the anticipated production load. Likewise, the expected outputs must also be defined. These may include parameters which are visible to the user such as page download times and transaction times or other less obvious parameters such memory usage and database access times.

 

2. Test Development

 

2.1 Capture user interactions

The interactions of an individual user type are captured using a combination of a standard browser and test specific software. These interactions are then stored within an editable script.

 

2.2 Model the script to create a virtual user

A virtual user (VU) is created by modifying (or modelling) the script captured in the first stage of test development. These VUs are used to load the web application during the test run.

 

2.3 Selection of performance monitors

The performance of the application under test will be determined from the values collected by the selected performance monitors. These monitors may range from standard performance counters through to bespoke remote monitoring agents.

 

2.4 Creation of the test

A performance test involves the application of a load consisting of different combinations and volumes of VUs. These VUs must be applied in a realistic manner with appropriate delays during VU application and interaction.

 

3. Test Deployment

 

3.1 Configure the test environment

Where appropriate, the test environment will need to be configured prior to the performance test. This configuration may include the creation of users within databases, passwords etc.

 

3.2 Verify correct operation of the test environment

Before running the performance test, it is advisable to verify the correct operation of the test environment, e.g. user authorisation, passwords, drive-mapping etc.

 

3.3 Running the test

This is where the application under test is loaded with the realistic VU-based load. Wherever possible, the performance test should be run multiple times to obtain more accurate results and to determine whether there were any anomalous test results.

 

3.4 Clean up the test environment

After the test has run, the test environment must be restored to its state prior to the application of load. This may involve the removal of users from databases and any other data created during the test.

 

4. Results Analysis

 

4.1 Analyse captured performance data

The data collected by the performance counters can be compared against the performance acceptance criteria to determine whether the outcome of the test is successful.

 

4.2 Performance tuning and removing bottlenecks

Although performance testing is often used to determine whether the application has met a particular performance level, it is also commonly used to assist in performance tuning as well as the identification and removal of performance bottlenecks.

 

The areas covered in this part of the TelQA Testing Series are explored in more detail in later parts of the series:

 

The TelQA Testing Series

 

Part 1: The Performance Testing Process

Part 2: Performance Testing Aims

Part 3: Types of Performance Testing

Part 4: Performance Test Environments

Part 5: Performance Test Inputs and Outputs

Part 6: Capturing User Interactions

Part 7: Modelling Scripts

Part 8: Selecting Performance Monitors

Part 9: Performance Test Creation

Part 10: Configuring Performance Test Environments

Part 11: Verifying Performance Test Environments

Part 12: Running a Performance Test

Part 13: Analysing Performance Data

Part 14: Using Performance Data for Tuning and Bottleneck Removal

 

A downloadable pdf of this article is available here.