Sun, 02/06/2011 - 11:07
Defining web performance testing sounds easy, but if you ask 100 web developers, you will probably get 100 different answers. The term is used in a variety of ways to cover what should probably be thought of as many types of testing; therefore, it is a broad name encompassing several definitions.
This is our best answer as it relates to web performance testing:
the deliberate action taken against a web system and the measurement of it's ability to function under specific circumstances. The situations presented to a web system can be normal or abnormal (e.g. heavy load, low bandwidth). The measurements can focus on front end (e.g. rendering time, response time) or on the back end (e.g. query time, throughput).
Examples from Tool Measurements
One way to explain a difficult concept is to show examples, so let's look at several websites that provide free performance monitoring that test for a wide array of measurements.
For example Yottaa.com focuses on page load times, complexity, and global reachability. They have beautiful charts and graphs to show a website owner how their site compares to all the other sites in the world. Yottaa's perspective of performance has nothing to do with stress testing because it is only analyzing one page at a time. It measures what one end-user will experience, and it checks from Washington DC, London, Singapore, Dublin, and San Francisco.
Load testing is sometimes used synonymously with performance testing, however there can be significant differences. It is probably safe to view load testing as a subset of performance testing. Load test tools such as LoadStorm.com can provide a service that focuses more on performance of a web application under heavy load conditions. Unlike Yottaa or WebPageTest, LoadStorm conducts testing against many pages and many virtual users at the same time. The goal is to simulate what real users would be doing and to see how fast the web system can operate while complying with user actions.
In this type of performance testing, the measurements are related to average response times across many types of requests or user behaviors. For example, logging into a blog site, posting a blog, and logging off would be one type of user behavior. Test plans are much more complex than with a website performance tool such as Yottaa because instead of analyzing one page in great detail, the tester must define test scripts to simulate the various types of user behavior. LoadStorm doesn't measure initial connection times or DNS lookups, but it provides more useful metrics pertaining to load performance (e.g. Throughput, Requests per Second, Error Rate).
Other Forms of Web Performance Testing
Some tools are designed to simulate virtual users at a normal load and for long duration. For example, some companies test their web applications with performance tools over a 72 hour period with a steady amount of traffic. This is referred to endurance testing or soak testing and is a subset of performance testing because it is measuring server-side data (e.g. memory consumption) to find problem such as memory leaks and heap overloads.
Other forms of performance testing include spike testing, configuration testing, and stress testing. According to Wikipedia, spike testing is "done by spiking the number of users and understanding the behavior of the application; whether performance will suffer, the application will fail, or it will be able to handle dramatic changes in load."
Configuration testing involves measuring the effects of configuration changes in the target web system. Examples include adjusting cache settings, increasing threadpool counts, and trying different load balancing setups.
Stress testing pushes a target web system to its limits. There are many ways to do this, but the most common is by applying extraordinary volumes of users - essentially load testing with traffic the site cannot handle. By creating more page hits than the application can withstand, a web developer can ascertain how many concurrent users can be supported and compare that to the goal set by the CIO or other executives.
Another form of stress testing is to intentionally disable a part of the infrastructure or application to determine the performance in a bad situation. For example, if a database server has a RAID array with 20 drives, hot swapping 5 of them while running a load test will show if the drive rebuilding process has a significant effect on response times, errors, or throughput.
Performance Testing Definitions is a page on this site that contains a dozen good definitions from various software quality assurance sources such as SearchSoftwareQuality.com and testingReflections.com. Other sources like Answers.com and Wikipedia are also quoted.