Performance Testing
What is it?
This website is designed to provide helpful information and resources to web developers, designers, testers, project managers, and product managers. In the context of web applications and websites performance testing has the objective of applying a specific user load to calculate the responsiveness of the system. The term is commonly used synonymously with load testing, although many testing professionals would argue there are subtle differences. We will include both types of testing in the resources available here. Both forms involve measuring throughput of the website based on requests per second, concurrent users, or bytes of data transferred. Scalability of the system is a primary consideration because future performance is very important for websites given the nature of web linking and the Slashdot Effect that can generate an enormous volume of users in a short period of time.
performance testing definitions
Why do it?
We do it to accomplish the following:
- Gather metrics useful for tuning the system
- Identify bottlenecks such as database configuration
- Determine if a new release is ready for production
- Provide reporting to business stakeholders regarding performance against expectations
- Evaluate if the site can handle an anticipated marketing campaign
Tools to help you
LoadStorm - A cloud-based load testing tool for websites and web applications. They target web developers on a tight budget with a simple tool that's easy to afford. Provided as a subscription service with a free account that generates up to 50 concurrent users of load - $300 for 5k vusers, wow that's cheap. That's probably why they released in April 2009 and already have over 3,000 customers. No download or install necessary. It doesn't have all the features of enterprise tools, but it does have what most developers need to simulate http traffic for form submission and following links. Nice real-time graphs show metrics like response times, errors, throughput, requests per second, and concurrent users. LoadStorm uses Amazon Web Services (the cloud) to dynamically allocate load generation servers on demand to give you as many servers as you need for huge load.
LoadRunner - The 800 pound gorilla of performance testing tools. Sold now by HP, LoadRunner can test almost any type of application or database. It is commonly used by larger companies with a big IT budget. There seems to be two major camps around this product: 1) You are a professional performance engineer and love all the functionality, or 2) You are frustrated by the complexity and don't have time for months of learning curve.
Visual Studio Team System - Microsoft has a suite of testing tools for web applications as a part of Visual Studio. Typical MS software: if you are committed to the MS stack, then it can be a good option. If you have to pay extra for it because you company didn't buy the Test Edition (or whatever licensing mechanism is in play), then you might want to look at a cheaper alternative.
OpenSTA - 'Open System Testing Architecture' is a free, open source web load/stress testing application, licensed under the Gnu GPL. Utilizes a distributed software architecture based on CORBA. OpenSTA binaries available for Windows.
Silk - Enterprise load testing tool from Borland. Generally accepted as the next-best option for big companies that don't want or can't afford LoadRunner. Lots of features and capability. Still a budget challenge for most web developers, but a great fit for many testing professionals working for or with the Fortune 1,000.
IBM Rational Performance Tester - Performance testing tool in the enterprise class and price range. Extra value from extensions to Seibel and SAP Solutions. Pricey, but a good fit for large organizations that normally buy from IBM.
JMeter - Desktop open source performance testing tool written in Java by the Apache Software Foundation. It was originally designed for testing Web Applications but has since expanded to other test functions. It may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types.
Curl-Loader - (also known as "omes-nik" and "davilka") is an open-source tool written in C-language, simulating application load and application behavior of thousands and tens of thousand HTTP/HTTPS and FTP/FTPS clients, each with its own source IP-address. In contrast to other tools curl-loader is using real C-written client protocol stacks, namely, HTTP and FTP stacks of libcurl and TLS/SSL of openssl, and simulates user behavior with support for login and authentication flavors. The goal of the project is to deliver a powerful and flexible open-source testing solution as a real alternative to Spirent Avalanche and IXIA IxLoad.
The Grinder - a Java load testing framework that makes it easy to run a distributed test using many load injector machines. It is freely available under a BSD-style open-source license. It has a generic approach to load testing anything that has a Java API. This includes common cases such as HTTP web servers, SOAP and REST web services, and application servers (CORBA, RMI, JMS, EJBs), as well as custom protocols. Test scripts are written in Jython. A graphical console allows multiple load injectors to be monitored and controlled, and provides centralised script editing and distribution.
ApacheBench - (sometimes simply called "ab") a Perl API for Apache benchmarking and regression testing. Intended as the foundation for a complete benchmarking and regression testing suite for transaction-based mod_perl sites. For stress-testing server while verifying correct HTTP responses. Based on the Apache 1.3.12 ab code. Available via CPAN as .tar.gz file.
http-load - Unix load test application to generate web server loads from ACME Software. Handles HTTP and HTTPS. http_load runs multiple http fetches in parallel, to test the throughput of a web server. However unlike most such test clients, it runs in a single process, so it doesn't bog down the client machine. It can be configured to do https fetches as well. You give it a file containing a list of URLs that may be fetched, a flag specifying how to start connections (either by rate or by number of simulated users), and a flag specifying when to quit (either after a given number of fetches or a given elapsed time). There are also optional flags for checksums, throttling, random jitter, and progress reports.
Pylot - Corey Goldberg's open source load testing tool for Python enthusiasts. Uses XML to define test cases.
