Load & Perf Testing

One of the first things we performance engineers do with a new server application is to conduct a quick throughput experiment. The goal is to find the maximum throughput that the server can deliver. In many cases, it is important that the server be capable of delivering this throughput with a certain response time bound. Thus, we always qualify the throughput with an average and 90th percentile response time (i.e. we want 90% of the requests to execute within the stated time). Any decent workload should therefore measure both the throughput and response time.

Let us assume we have such a workload. How best to estimate the maximum throughput within the required response time bounds ? The easiest way to conduct such an experiment is to run a bunch of clients (emulated users, virtual users or vusers) to drive load against the target server without any think time. Here is how the flow from a vuser will look like :

Create Request ==> Send Request ==> Receive Response ==> Log statistics

This sequence of operations is executed repeatedly (without any pauses in between i.e. no think times) for a sufficient length of time to get statistically valid results. So, to find the maximum throughput, run a series of tests, each time increasing the number of clients. Simple, isn’t it ?

A little while ago, I realized that if one doesn’t have the proper training, this isn’t that simple. I came across such an experiment with the following results :

VUsers Throughput

Requests/sec 5000 88318 10000 88407 20000 88309 25000 88429 30000 88392 35000 88440

What is wrong with these results ?
Firstly, the throughput is roughly the same at all loads. This probably means that the system saturated even below the base load level of 5,000 Vusers. Recall, that the workload does not have a think time. When you have this many users repeatedly submitting requests, the server is certain to be overloaded. I must mention that the server in this case is a single system with 12 cores having hyper-threading enabled. A multi-threaded server application typically will use one or more threads to receive requests from the network, then hand the request to a worker thread for processing. Considering the context-switching, waiting, locking etc. one can assume that at most one can run 4x the number of cores or in this case about 96 server threads. Since each Vuser submits a request and waits for a response, it probably requires 2-2.5x the number of Vusers as the number of server threads to saturate a system. Using this rule of thumb, one would need to run a maximum of 200-250 Vusers.

Throughput with Correct Vusers

After I explained the above, the tests were re-run with the following results:

VUsers Throughput 1 1833 10 18226 50 74684 100 86069 200 88455 300 88375

Notice that the maximum throughput is still nearly the same as from the previous set, but it has been achieved with a much lower number of Vusers (aka clients). So does it really matter ? Doesn’t it look better to say that the server could handle 35000 connections rather than 300 ? No, it doesn’t. The reason becomes obvious if we take a look at the response times.

The Impact of Response Times

The graph below shows how the 90% Response Time varied for both sets of experiments :

The response times for the first experiment with very large number of Vusers ranges in the hundreds of millisecs. When the number of Vusers was pared down to just reach saturation, the server responded hundred times faster ! Intuitively too, this makes sense. If the server is inundated with requests, they are just going to queue up. The longer a request waits for processing, the larger is it’s response time.

Summary

When doing throughput performance experiments, it is important to take into consideration the type of server application, the hardware characteristics etc. and run appropriate load levels. Otherwise, although you may be able to find out what the maximum throughput is, you will have no idea what the response time is.

Today we received an excellent question from a customer of ours:

We were wondering if you all have any information that says “X Unique visitors per day translates into Y simultaneous users at any given time.”

Essentially, we’re looking for a way to determine how many simultaneous users we should load test with if we know the sites normal daily traffic.

While every site is different, we recommend following this line of reasoning to help you find the answer. Suppose your site gets 100K unique visitors per day, with peak traffic in the mornings and afternoons. Assume that 40% of the traffic comes between 7AM and 11AM, 40% at 4PM to 9PM, and 20% at other times. This means during your peak hours (7AM to 11AM and 4PM to 9PM) you’ll get ~10% of your unique visitors per hour, or 10K uniques in our example.

Now that you know how many unique visitors you’ll get in an hour, you can start turning that in to concurrent users. To do that, it’s important to understand that a unique visitor is roughly equivalent of a transaction. So really you want to figure out how many users you need to reach 10K transactions in an hour.

Suppose your script (or scripts) take an average of 2 minutes to complete. That means a single user will execute 30 transactions in an hour. So to reach 10K transactions, you’d need 334 users (10K divided by 30). If you decide you want to create realistic scripts that include human think time, then the scripts will take that much longer and you’ll need that many more concurrent users. For example, if the script gets 5X think time added and now takes 10 minutes to run, then you’ll need 1,667 users (10K divided by 6).

Of course, this calculation will only get you the load on a typical day (assuming a single hour sees 10% of traffic). Your traffic patterns may vary, or you may want to prepare for a larger surge. For example, if you want to test what happens when 60% of the daily traffic visits in an hour, then you’d need 2,000 users (60K divided by 30).

Tweet This Post 

Today we received an excellent question from a customer of ours: We were wondering if you all have any information that says “X Unique visitors per day translates into Y simultaneous users at any given time.” Essentially, we’re looking for a way to determine how many simultaneous users we should load test with if we know the [...]

We recently had a customer from a large clothing retailer ask us if there was any way to ensure that data, such as a username/password combination, could be restricted such that it was “checked out” and available only for a specific concurrent user. This is very common with logins, where systems often will prevent concurrent [...]

We recently had a customer from a large clothing retailer ask us if there was any way to ensure that data, such as a username/password combination, could be restricted such that it was “checked out” and available only for a specific concurrent user. This is very common with logins, where systems often will prevent concurrent logins from multiple IP addresses.

While BrowserMob does not have a concept in which data rows can be “checked out”, some simple scripting can achieve the same results. The key is in creative use of the browserMob.getUserNum() and browserMob.getTxCount() APIs. You can learn more about them by reading up on the BrowserMob APIs.

The getUserNum function returns 0, 1, 2, etc based on the concurrent user in your load test. So if you have a 100 user test, getUserNum will return between 0 and 99. It’s important to understand that it will return the same value for the same user throughout the test.

The getTxCount function returns 1, 2, 3, etc based on the number of cycles for that specific user. This number will effectively be a counter of the unique number of transactions that that particular user has executed. So user 1 and user 100 will both have a getTxCount of 1 returned, but by the time user 100 sees it, user 1 might already be on transaction 50.

Now suppose you want to run a 1000 user test in which you never concurrently log in with the same user. All you need to do is pre-create 1000 user accounts and then write your script like so:

var userId = browserMob.getUserNum(); var username = "test-" + userId; var password = "password"; selenium.type("username", username); selenium.type("password", password);

This works great, but what if you want to use more than 1000 logins? Suppose you want to use up to 10,000 logins among the 1000 user test? This is where the getTxCount function comes in to:

var loginsPerUser = 10; var userNum = browserMob.getUserNum(); var txCount = browserMob.getTxCount(); var userId = userNum * loginsPerUser + txCount % loginsPerUser; var username = "test-" + userId; var password = "password"; selenium.type("username", username); selenium.type("password", password);

What this does is allocate 10 logins per concurrent user. So user 1 will get usernames test-0, test-1, …, test-9 while user 8 will get usernames test-80, test-81, …, test-89, etc. Because of the mod call (%) the ten usernames will simply wrap around once they’ve been used.

Tweet This Post 

Because both our load testing and website monitoring services are based on Selenium, we have a unique ability to measure the performance of things like page load times, AJAX timings, and other in-browser interactions.

Selenium has both a setTimeout command and a waitForPageToLoad command. Both can be given a timeout value, which will control how long Selenium waits for a given page to load or element to appear. When it comes to using our services, most people stick with the default time of 30 seconds. If the timeout is reached, an error is thrown, the script aborts, and the transaction is recorded.

However, sometimes people want to know when pages take more than X seconds to load, but don’t want to necessarily interrupt the flow of the script. In fact, just last week we got this request from a customer:

Ability to trigger an alert based on a set threshold (by the user) – not using the timeout. This basically came from the performance issues we are experiencing. Here’s the scenario:

• We have a specific page that takes 2+ minutes to load.
• The page load timeout in the Selenium script was set to 60s.
• BMob properly reported the “timeout” error but when this error happens, BMob quits the script.
• This is not ideal for me since I want to still be able to see how long the page takes to load.
• Increasing the page load timeout for the page in question works, but now I don’t have a way (that I know of) to still trigger an alert after 60s.

I should be able to set a threshold for any page I choose that would then send a notification alert.

In other words, this customer wanted for a way to still report a transaction as a failure and receive an alert, but also still allow the script to continue. Fortunately, our support for JavaScript as a scripting language provides the answer:

var timeout = 90000; ... // start of script ... var start = new Date().getTime(); selenium.waitForPageToLoad(timeout); var end = new Date().getTime(); ... // rest of script ... if ((end - start) > 45000) { throw "An important page took longer than 45 seconds to load"; }

What this script does is sets the timeout to a very long amount (1.5 minutes) but will still report an error if a specific page takes longer than 45 seconds. This allows the remainder of the script to execute even when the page takes more than a specified 45-second threshold.

The only problem with this script is that if the page takes longer than 90 seconds, then the rest of the script will still not run because waitForPageToLoad will throw an exception. You can solve that too with a little code:

var start = new Date().getTime(); try { selenium.waitForPageToLoad(timeout); } catch (e) { // this will happen after 90 seconds // todo: recover and send the browser to the the next URL } var end = new Date().getTime();

The only thing that is important to remember with this use of try/catch is that you’ll need to properly recover from the error. Simply catching the error and trying to continue may not work. For example, if the next Selenium command requires clicking on a button that should have loaded from the last page, there may be no way to recover. However, if the next step is simply visiting a new URL, you could possibly get away with a simple open() command in the catch block.

Tweet This Post 

Because both our load testing and website monitoring services are based on Selenium, we have a unique ability to measure the performance of things like page load times, AJAX timings, and other in-browser interactions. Selenium has both a setTimeout command and a waitForPageToLoad command. Both can be given a timeout value, which will control how long [...]

Last week I got the following two questions from one of our clients “We use ASP.NET PostBacks but can’t find the PurePath for the request triggering the PostBack handler – any hints?” “We see many ThreadAbortExceptions in our ASP.NET Application and we are not sure why they happen – are they expected?” Time for a little blog that [...]

Whether you call it Football, Futbol, Fussball, Futebol, Calcio or Soccer – if you are a fan of this game I am sure you are looking forward to the upcoming FIFA World Cup in South Africa. The tournaments web site is http://www.fifa.com/worldcup and allows the fans to follow their teams and get the latest updates [...]

When something goes wrong who’s to blame? In this post I take a closer look at who is responsible when applications have performance problems. Interestingly very often the question as to who is responsible is at least as important as how to solve the problem. In order to not have to search for the responsible [...]

We have had some great Webinars with our customers in the past year – such as Zappos, SmithMicro, Peopleclick or Monster. Next week I am co-hosting a Webinar with Eric Thomas, Director of Advanced Engineering at Thomson Reuters. Eric will talk about their online services that millions of people rely on every day and how [...]

We all know that slow performance – and service disruption even more – affects our business services and eventually our revenue. At the same time we say that major parts of companies are not willing to invest in performance. In this post I will discuss why we find ourselves in this paradox and how to [...]

With more than 3 million downloads, Selenium is the world’s most popular functional testing framework. It allows web developers and QA professionals to automatically test how an application functions on multiple browsers, such as Firefox, Internet Explorer and Chrome and across operating systems, such as different versions of Windows and Mac [...]

In his latest blog post, Steve Souders writes about the impact of performance on business success. I am not going to duplicate his content here as you can just go ahead and read his blog. I just want to highlight some very interesting points and give you some links to follow up resources. Performance is now [...]

2 weeks ago I co-hosted a Webinar with one of our users – Bill Mar, Director of Engineering Services from SmithMicro Software. SmithMicro provides the backbone of our digital life by connecting different digital devices together. In his role, Bill works in the Wireless Business unit working on Voice-related services, e.g.: VoiceSMS or Visual Voicemail [...]

Our dynaTrace Community Portal is our gateway to our users. Especially with the rapidly-growing number of world-wide users of our FREE dynaTrace AJAX Edition, it is necessary to keep track on how well our pages perform from around the globe to satisfy our “performance hungry” community There are two important questions we want to [...]

I am happy to announce that the FREE dynaTrace AJAX Edition was picked to be demoed at the upcoming Velocity 2010 Conference in Santa Clara, CA. I also just got word from the organizers of Velocity that you can save 20% when you register with the promotion code vel10fsp. If you make it to Velocity – [...]

MicroFocus and dynaTrace recently announced “SilkPerformer Assurance” and with that announcement comes a tighter integration between MicroFocus’s Load Testing Tool SilkPerformer and dynaTrace’s Application Performance Management Solution for Load Testing. Enough has been said about the partnership in the actual press release. Now it is time to look behind the marketing curtain and explore the details [...]

Factors which needs to considered while defining performance strategy in Cloud br / br / strongElasticity/strong: br / br / Elasticity is the ability to scale up your system resources when needed, and scale down when not required. Here are two important measures of cloud elasticity: spin-up elasticity and spin-down elasticity. br / br / Load testing always consider peak load scenarios while measuring the system performance where as in cloud we may also have to consider Peak, moderate and low level usage of transactions. Base lining the System performance under these various loads will help the administrator to take a decision while scaling up or scaling down the resources based on load.br / br / strongScalability/strong br / br / Scalability is the ability of an application to handle increased processing demand with increase of load.br / br / Testing the scalability of an application is very important; if the application is not scalable then the elasticity of the cloud can actually cost thousands of dollars. When moving to the cloud, it becomes even important to test the applications and to tune them properly so they are optimized when it comes to hardware consumptionbr / br / strongBandwidth/strongbr / br / Usually Cloud Providers are distributed across different geographical locations. Bandwidth testing using tools like Shunra Cloud will help to ensure bandwidth requirements between the endusers and the cloud provider br / br / Some Cloud Platform providers provide the following usage information of your application this can also be used as a basis while defining performance strategybr / br / • CPUbr / br / • Memorybr / br / • # of DML statementsbr / br / • # webservice callsbr / br / • # Calculationsdiv class="blogger-post-footer"img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/3081868611305126410-6275041804756787301?l=performancetestinginfo.blogspot.com' alt='' //div pa href="http://feedads.g.doubleclick.net/~a/RIMRLPpmRyr0UDTGC40_a4fToiY/0/da"img src="http://feedads.g.doubleclick.net/~a/RIMRLPpmRyr0UDTGC40_a4fToiY/0/di" border="0" ismap="true"/img/abr/ a href="http://feedads.g.doubleclick.net/~a/RIMRLPpmRyr0UDTGC40_a4fToiY/1/da"img src="http://feedads.g.doubleclick.net/~a/RIMRLPpmRyr0UDTGC40_a4fToiY/1/di" border="0" ismap="true"/img/a/pimg src="http://feeds.feedburner.com/~r/InfoOnPerformanceTesting/~4/4aQ7YFyZ2jY" height="1" width="1"/

pstrongLoad Impact has been chosen as one of Europe's 100 hottest companies/strong/ppimg style="float: left; border: 0;" src="../../images/blog/Europe_Winners_Logo.jpg" alt="" width="138" height="137" /strongThis/strong week saw the a href="http://redherring.com" target="_blank"Red Herring/a Top 100 Europe awards take place in Paris. Load Impact was one of the 200 finalists chosen to present and on thursday, we were declared to be one of the 100 winners of the prestigious award./ppstrongThe/strong winners were the companies judged to be the most innovative and to be positioned to grow at an explosive rate. Companies were evaluated on both quantitative and qualitative criteria, such as financial performance, technology innovation, quality of management, execution of strategy and industry integration./ppstrongWe/strong are of course extremely happy to get even further recognition that we are on the right track. Still, the single biggest positive signal of all is of course, and has always been, our astounding usage statistics. Over 114,000 load tests executed, at the time of writing, is pretty telling. Our users are giving us all the confirmation we could ever ask for, and for that we are ever grateful! We will do our best to live up to all the hype and provide you with the world's best load testing service both now and in the future!/ppnbsp;/p