What is it?
Simply put, applications must be tested to find out if they deliver fast enough to make the users happy. The tests should include measuring aspects of one page (e.g. load speed), as well as measuring speed of server response under a heavy load of simultaneous users.
This website is designed to provide helpful information and resources to web developers, designers, testers, project managers, and product managers. In the context of web applications and websites, performance testing has the objective of applying a specific user load to calculate the responsiveness of the system.
Performance testing is commonly used synonymously with load testing, although many testing professionals would argue there are subtle differences. We will include both types of testing in the resources available here. Both forms involve measuring throughput of a web application based on requests per second, concurrent users, or bytes of data transferred.
Performance testing is very important because there is a direct correlation between fast and stable web applications and the revenue generated from them. Shoppers, researchers, and just about any type of user today will not tolerate errors. Nor do people have patience with a page that takes more than 5 seconds to load. Performance = profit.
Scalability of the system is also a primary consideration because future performance is very important for websites given the nature of web linking and the Slashdot Effect that can generate an enormous volume of users in a short period of time.
performance testing definitions
Why do it?
We do it to accomplish the following:
- Improve user experience on sites and web apps
- Increase revenue generated from websites
- Gather metrics useful for tuning the system
- Identify bottlenecks such as database configuration
- Determine if a new release is ready for production
- Provide reporting to business stakeholders regarding performance against expectations
- Evaluate if the site can handle an anticipated marketing campaign
Tools to help you
LoadRunner - The 800 pound gorilla of performance testing tools. Sold now by HP, LoadRunner can test almost any type of application or database. It is commonly used by larger companies with a big IT budget. There seems to be two major camps around this product: 1) You are a professional performance engineer and love all the functionality, or 2) You are frustrated by the complexity and don't have time for months of learning curve.
Visual Studio Team System - Microsoft has a suite of testing tools for web applications as a part of Visual Studio. Typical MS software: if you are committed to the MS stack, then it can be a good option. If you have to pay extra for it because you company didn't buy the Test Edition (or whatever licensing mechanism is in play), then you might want to look at a cheaper alternative.
Silk - Enterprise load testing tool from Borland. Generally accepted as the next-best option for big companies that don't want or can't afford LoadRunner. Lots of features and capability. Still a budget challenge for most web developers, but a great fit for many testing professionals working for or with the Fortune 1,000.
IBM Rational Performance Tester - Performance testing tool in the enterprise class and price range. Extra value from extensions to Seibel and SAP Solutions. Pricey, but a good fit for large organizations that normally buy from IBM.
LoadStorm - A cloud-based (Amazon EC2) load testing tool for web applications. They target web developers on a tight budget with a simple tool that's easy to afford. It doesn't have all the features of enterprise tools, but it does have what most developers need to simulate http traffic for form submission and following links. Nice real-time graphs show metrics like response times, errors, throughput, requests per second, and concurrent users. Can test up to 50,000 concurrent users. Free account with 25 concurrent users never expires (handy!).
Xceptance LoadTest - Load testing tool for creating and running regression and load tests, in particular for web applications. XLT combines the automation of regression tests with the execution of load tests, as the test cases already created for the automated regression test can subsequently be applied as load tests. In short: Every regression test can also be a load test. Free for up to five virtual users.
Web Page Tester - Analyzes the performance of a single page in an IE browser. You provide the URL of a page, and it is requested from one of five locations around the world. Summary and detail reports are generated that show the times for overall page loading, DNS lookups, rendering, and downloading for each resource such as HTML, CSS, images, and script files. An optimization checklist is presented to make suggestions such as compression or caching for improving the performance.
SiteBlaster - SiteBlaster is a web site load and stress testing tool built to load and stress test USGovXML. USGovXML is an index to publicly available web services and XML data sources provided by the US Government. When testing is complete, a report is available that can be viewed or printed. SiteBlaster simulates Internet Explorer web browsing functionality and is best used to test those sites that use URL query strings to pass data to its web pages.
Gomez - Gomez provides an on-demand platform that you use to optimize the performance, availability and quality of your Web and mobile applications. It identifies business-impacting issues by testing and measuring Web applications from the “outside-in” — across your users, browsers, mobile devices and geographies — using a global network of 100,000+ locations.
BrowserMob - Hosted Selenium on-demand load testing tool to simulate Firefox browsers. Captures screenshots of website load issues to help you debug. It uses Amazon EC2 cloud to run many instances of Selenium. More accurately mimics the real user experience by supporting AJAX and Flash processing. Free trial available for 25 real browser user load tests or 100 simulated virtual users.
LoadImpact - Online load testing service from Gatorhole for stress testing websites. Although not on a specific cloud infrastructure, it is a hosted solution with no download or installation, and provides user traffic against your website through their network of load generator server clusters with very fast connections to enable simulation of tens of thousands of users accessing your website concurrently. Free low level load tests for 1-50 simulated users; higher levels have monthly fees.
OpenSTA - 'Open System Testing Architecture' is a free, open source web load/stress testing application, licensed under the Gnu GPL. Utilizes a distributed software architecture based on CORBA. OpenSTA binaries available for Windows.
JMeter - Desktop open source performance testing tool written in Java by the Apache Software Foundation. It was originally designed for testing Web Applications but has since expanded to other test functions. It may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types.
Curl-Loader - (also known as "omes-nik" and "davilka") is an open-source tool written in C-language, simulating application load and application behavior of thousands and tens of thousand HTTP/HTTPS and FTP/FTPS clients, each with its own source IP-address. In contrast to other tools curl-loader is using real C-written client protocol stacks, namely, HTTP and FTP stacks of libcurl and TLS/SSL of openssl, and simulates user behavior with support for login and authentication flavors. The goal of the project is to deliver a powerful and flexible open-source testing solution as a real alternative to Spirent Avalanche and IXIA IxLoad.
The Grinder - a Java load testing framework that makes it easy to run a distributed test using many load injector machines. It is freely available under a BSD-style open-source license. It has a generic approach to load testing anything that has a Java API. This includes common cases such as HTTP web servers, SOAP and REST web services, and application servers (CORBA, RMI, JMS, EJBs), as well as custom protocols. Test scripts are written in Jython. A graphical console allows multiple load injectors to be monitored and controlled, and provides centralised script editing and distribution.
ApacheBench - (sometimes simply called "ab") a Perl API for Apache benchmarking and regression testing. Intended as the foundation for a complete benchmarking and regression testing suite for transaction-based mod_perl sites. For stress-testing server while verifying correct HTTP responses. Based on the Apache 1.3.12 ab code. Available via CPAN as .tar.gz file.
http-load - Unix load test application to generate web server loads from ACME Software. Handles HTTP and HTTPS. http_load runs multiple http fetches in parallel, to test the throughput of a web server. However unlike most such test clients, it runs in a single process, so it doesn't bog down the client machine. It can be configured to do https fetches as well. You give it a file containing a list of URLs that may be fetched, a flag specifying how to start connections (either by rate or by number of simulated users), and a flag specifying when to quit (either after a given number of fetches or a given elapsed time). There are also optional flags for checksums, throttling, random jitter, and progress reports.
Pylot - Corey Goldberg's open source load testing tool for Python enthusiasts. Uses XML to define test cases.
Loadea - Win XP based load testing tool. Control and capture modules use C# for building test scenario scripts, schedule stress test execution, and XML for test data. The analysis module provides reporting capabilities. Free evaluation version for two virtual users.
CloudIntelligence - a software development company specializing in Cloud-Computing and Open-Source technologies. By leveraging best-of-breed Open-Source technologies, Cloud-Computing resources and together with our own proprietary development, we are able to present the best and affordable Cloud-Testing solution.
StressTester - enterprise class performance / web load testing tool used by organizations of all sizes to ensure that their web, internet, extranet and web service applications can meet their users’ needs in terms of performance, scalability and stability. Free trial download available.
AutoPilot M6 Suite - Enterprise performance management syste for business transactions and complex composite applications. AutoPilot M6 gives you a 360° view of your entire IT environment to assure the successful and timely completion of your business critical transactions and peak performance the applications that run your business.
LoadManager - Focused on performance criteria for banks, telecom giants and industrialists. Runs on all platforms supported by Eclipse and Java. Consists of one controller and several agent modules that can be spread over several machines.
NeoLoad - Load testing tool that runs on Windows Vista, XP, Win 2000, Win Server 2003/2008, Linux (RedHat, Mandriva), and Solaris 10. It has two main components: the Controller and the Load Generator. The controller has a load generator pre-installed. This means only one machine is needed to carry out medium-volume tests. Additional load generators (free of charge) may be deployed on other machines to create much higher loads. Free 30-day evaluation download.
TestComplete - An enterprise automated test manager with project level support for a full range of internal and UI testing. Supports testing of Windows applications created in Visual C++, Visual Basic, Delphi, C++Builder, PowerBuilder, Visual FoxPro, .NET, WPF, Java and JavaFX applications, web applications and web services, Flash, Flex and Silverlight applications. Also tests applications running on portable devices such as PDAs, Pocket PCs and smartphones.
QTest - An enterprise performance & load testing tool specializing in complex Web applications like Siebel, SAP, and Epiphany. It offers high capacity for load simulation. It supports all Web, Web Service, J2EE, .Net, ASP, AJAX, CGI, and Mainframe Portal environments and integrate with their APM solution. With the Winload module, it can test Windows, client/server and ERP applications, in particular SAP Fat Clients, PeopleSoft, Oracle, and Citrix.
httperf - Open source web server performance/benchmarking tool from HP Research Labs. It provides a flexible facility for generating various HTTP workloads and for measuring server performance. The focus of httperf is not on implementing one particular benchmark but on providing a robust, high-performance tool that facilitates the construction of both micro- and macro-level benchmarks. The three distinguishing characteristics of httperf are its robustness, which includes the ability to generate and sustain server overload, support for the HTTP/1.1 and SSL protocols, and its extensibility to new workload generators and performance measurements.
TestMaker - A single platform for Functional Testing, Regression, Load and Performance Testing, and Business Service Monitoring, all from the same single test script. Pass smoke tests, surface performance bottlenecks, and enforce Service Level Agreements (SLAs) in one product.
- Unix-based http load tester and benchmarking utility. It was designed to let web developers measure the performance of their code under duress, to see how it will stand up to load on the internet. It lets the user hit a web server with a configurable number of concurrent simulated users. Those users place the server "under siege." Written on GNU/Linux and has been successfully ported to AIX, BSD, HP-UX and Solaris. It should compile on most System V UNIX variants and on most newer BSD systems. Because Siege relies on POSIX.1b features not supported by Microsoft, it will not run on Windows.
SiteTester1 - A load-testing utility designed to test web servers and web applications by simulating virtual users following predefined procedures for HTTP1.0/1.1 compatible requests, POST/GET methods, cookies, running in multi-threaded or single-threaded mode. Generates various reports in HTML format, keeps and reads XML formatted files for test definitions and test logs. Requires JDK1.2 or higher.
eValid LoadTest - A testing tool suite built into an IE browser for 100% client side quality checking, dynamic testing, content validation, page performance tuning, and webserver loading and capacity analysis. It performs functions needed for detailed static and dynamic website testing, regression testing, QA/Validation, page timing and tuning, transaction monitoring, and realistic & scalable server loading.
WebPerformance Load Tester - Load test tool for generating and analyzing automated load tests on a server. Permanent fixed-seat, permanent floating, and temporary fixed-seat licenses available. Automatically detects and configures the test cases for many situations. Supports all browsers and web servers. Modem simulation allows each virtual user to be bandwidth limited. For Windows and many UNIX variants.
CapCal - testing tool that works with Agile frameworks and tools to leverage those same functional tests as performance and scalability tests. No more coding and maintaining redundant functional and performance test scripts. You can track CPU speed, bandwidth, and I/O rates. Measures the entire stack from the web server to the operating system to the network. CapCal’s Server Performance provides the first ever standard performance ratings facility to quickly and easily compare server and application performance in physical, virtual and cloud configurations.
On February 8, 2011, the W3C Launched a new Web Performance Interest Group. The mission of Web Performance Interest Group, part of the Rich Web Client Activity, is dedicated to creating a faster user experience on the Web. The Interest Group will produce use cases and requirements for future deliverables of the Web Performance Working Group. The Co-Chairs of the Interest Group are Steve Souders and Jason Weber. The W3C Team Contact for the Web Performance Interest Group is Philippe Le Hégaret. The Interest Group will produce use cases and requirements for future deliverables of the Web Performance Working Group. The Interest Group will provide a forum to discuss Web performance initiatives across web publishers, vendors, developers, and users with the goal of identifying areas for standardization.