Skip navigation.

Load & Perf Testing

BrowserMob Joins Neustar Webmetrics Family of Services

BrowserMob - Thu, 07/08/2010 - 16:20
I’ve got great news to share with you – BrowserMob has been acquired and is joining the Neustar family of services, which includes UltraDNS and Webmetrics! I started BrowserMob in 2008 with the belief that a combination of cloud computing and real browsers wrapped up in a self-service model could dramatically change how people used load [...]
Categories: Load & Perf Testing

6 Command Line Tools for Linux Performance Monitoring

Corey Goldberg - Thu, 07/01/2010 - 09:22
p So you need to monitor a Linux system for performance metrics... CPU, Memory, Network, Disk, etc. /p p Here are 6 of my favorite command line tools for monitoring a Linux system from the command line. /p hr / stronghtop/strongbr / a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_k7-jvtv2cLo/TCy9tDdKYrI/AAAAAAAAArQ/ZYNCUDUwb8A/s1600/screenshot_htop.png"img style="cursor:pointer; cursor:hand;width: 400px; height: 266px;" src="http://1.bp.blogspot.com/_k7-jvtv2cLo/TCy9tDdKYrI/AAAAAAAAArQ/ZYNCUDUwb8A/s400/screenshot_htop.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5488970627577176754" //a br / a href="http://htop.sourceforge.net/"http://htop.sourceforge.net//a hr / strongdstat/strongbr / a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_k7-jvtv2cLo/TCy9sgkjlGI/AAAAAAAAArI/iUDVIxjowz8/s1600/screenshot_dstat.png"img style="cursor:pointer; cursor:hand;width: 400px; height: 273px;" src="http://4.bp.blogspot.com/_k7-jvtv2cLo/TCy9sgkjlGI/AAAAAAAAArI/iUDVIxjowz8/s400/screenshot_dstat.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5488970618212947042" //a br / a href="http://dag.wieers.com/home-made/dstat/"http://dag.wieers.com/home-made/dstat//a hr / strongbmon/strongbr / a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_k7-jvtv2cLo/TCy9sPm4kHI/AAAAAAAAArA/YKmhbVGV1BU/s1600/screenshot_bmon.png"img style="cursor:pointer; cursor:hand;width: 400px; height: 359px;" src="http://3.bp.blogspot.com/_k7-jvtv2cLo/TCy9sPm4kHI/AAAAAAAAArA/YKmhbVGV1BU/s400/screenshot_bmon.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5488970613659308146" //a br / a href="http://freshmeat.net/projects/bmon/"http://freshmeat.net/projects/bmon//a hr / strongiftop/strongbr / a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_k7-jvtv2cLo/TCy9t47CtqI/AAAAAAAAArg/AL8khQaDGok/s1600/screenshot_iftop.png"img style="cursor:pointer; cursor:hand;width: 400px; height: 266px;" src="http://4.bp.blogspot.com/_k7-jvtv2cLo/TCy9t47CtqI/AAAAAAAAArg/AL8khQaDGok/s400/screenshot_iftop.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5488970641929582242" //a br / a href="http://www.ex-parrot.com/pdw/iftop/"http://www.ex-parrot.com/pdw/iftop//a hr / strongifstat/strongbr / a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_k7-jvtv2cLo/TCy9tpw1eBI/AAAAAAAAArY/0JWtGr6_E2E/s1600/screenshot_ifstat.png"img style="cursor:pointer; cursor:hand;width: 400px; height: 266px;" src="http://2.bp.blogspot.com/_k7-jvtv2cLo/TCy9tpw1eBI/AAAAAAAAArY/0JWtGr6_E2E/s400/screenshot_ifstat.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5488970637860239378" //a br / a href="http://gael.roualland.free.fr/ifstat/"http://gael.roualland.free.fr/ifstat//a hr / strongsysstat/strongbr / this is a package of utilites including strongiostat/strong, strongmpstat/strong, strongsar/strong, and others.br / a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_k7-jvtv2cLo/TCy95xUUBgI/AAAAAAAAAro/ZHzzRKH5dys/s1600/screenshot_iostat.png"img style="cursor:pointer; cursor:hand;width: 400px; height: 266px;" src="http://2.bp.blogspot.com/_k7-jvtv2cLo/TCy95xUUBgI/AAAAAAAAAro/ZHzzRKH5dys/s400/screenshot_iostat.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5488970846046520834" //a br / a href="http://pagesperso-orange.fr/sebastien.godard/"http://pagesperso-orange.fr/sebastien.godard//a hr / p These tools are all available from package managers (apt-get, yum, etc) on most Linux systems. They are also available on most other *nix platforms. /pdiv class="blogger-post-footer"img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/5236867476487043111-1588815434091547815?l=coreygoldberg.blogspot.com' alt='' //div
Categories: Load & Perf Testing

Gary Busey's Tuesday Load Testing Notes

Performance & Load Testing - Tue, 06/29/2010 - 04:31
pimg src=http://loadstorm.com/sites/loadstorm.com/files/220px-Gary_Busey_2007.jpg class=picture alt=Gary Busey is a stress test height=200 /I've have always liked a href=http://en.wikipedia.org/wiki/Gary_Busey target=_blankGary Busey/a in movies and on TV. He has starred in everything from Kung Fu to The Buddy Holly Story to the Simpsons to Gunsmoke to Saturday Night Live to Lethal Weapon. His recent appearance on Celebrity Rehab with Dr. Drew showed just how much brain damage he suffered in that motorcycle crash. In some ways, Gary's life is similar to a good stress test: Ramped up the volume until the system could not respond appropriately to requests made of it./p pOn to the good stuff about system performance testing./p h2Performance Metrics Tied to Business Metrics/h2 pMike Kelly at SoftwareSearchQuality.com has an article about developing a working understanding of the business drivers for application performance. He postulates that a skilled performance tester can build better tests through understanding performance goals better through knowing what the business truly needs. He also believes this understanding makes test results more actionable. The key question to ask is, What do the project stakeholders care about? a href=http://searchsoftwarequality.techtarget.com/tip/0,289483,sid92_gci1508137_mem1,00.html?track=sy280 target=_blankRead more aboutbr / application lifecycle performance testing and monitoring strategies./a/p h2Agile Performance Testing/h2 pMaybe the best point in Mike's post is:br / The best time to be thinking about possible performance issues is when you're actively designing and writing code. You're still close to the code. You know what it does. You know how it works. For each day away from the code, the fuzzier it gets in your head. The more developers committing to the codebase, the less you really know about the code you're interacting with. /p pSound like Agile? Yep, test early and test often. I've asked questions at conferences of Agile gurus about performance testing early in the development cycle. All of them acknowledge that frequent perf tests should be run from the beginning of coding. Every build should automatically run a load test. However when you press them to see how often their projects actually follow this model, they eventually admit that the practice doesn't meet the theory. Rarely, if ever, do teams create perf scripts and execute them until the last 20% of of the project. It's easy to stand up behind a podium and talk to a PowerPoint slide about what you SHOULD do, but it is not reality that teams put high priority on performance until all the pieces of code are pulled together./p h2Unit Versus Comprehensive System Tests/h2 pA common misconception is that there is one form of performance testing - hit the whole system with volume and see how it responds. That's one type, but in many cases it is much more effective to test a component on its own. For example, let's say you are coding a module that inserts data into the database. You should examine the efficiency of the way you coded the process before you test the performance of all the moving pieces that call your module. It will be harder to detect the root cause of a bad SQL statement later when it is hidden in the big pile of other database hits./p pMike makes an argument about using tools that will let you test the speed of units of code. Again, sounds good. I support that idea and recommend it to every developer. You may be able to find some expensive queries or poorly written loop structures. That will be a tremendous help when you are testing the speed and responsiveness of the system to a user where the UI layer is communicating with the application layer which is communicating with the database layer. Identifying resource hogs and slow functions is possible early, and it appears obvious that fixing those in unit testing would be best. Tune the individual parts before assembling them. It has a much higher probability of producing a well performing system./p h2Realistic Test Scenarios for User Types/h2 pMike talks about performance profiles, and I tend to think about the different ways people will use the system. In a typical web app, you might have 70% of your traffic from anonymous lookers viewing your product catalog or reading articles. Maybe 15% of your traffic is going through the shopping experience and buying something (if you are lucky!). Perhaps you have 10% of your traffic downloading white papers or filling out web forms to register for your newsletters. You probably have a few of your internal employees signed in to modify content or update the product catalog. All of those are hypothetical of course, but they represent realistic user patterns. Your test plan should reflect these. /p pIt is also true that during Christmas the percentage of credit card processing could go up by 100% over normal. Your application may have spikes in certain kinds of traffic at end of year. I spoke with PerfDan at the STP conference last year over lunch. He said the TurboTax Online app could see a million users per day during the second week of April. Hmmm, predictable. My suspicion is that most of them are in a panic too!/p pOne of our clients recently released a new book. Realizing that they needed to load test for the anticipated volume, they started building tests with LoadStorm. What they didn't consider was that most of the traffic would NOT come through their home page to make the purchase. They were sending a large email blast with a link to a specific landing page, and their test wasn't accounting for that. Thus, their original scenario was essentially useless for predicting site performance. Their home page was horribly slow with hundreds of images and dozens of unnecessary Javascript external libraries. The system stressed out at low volume because the test virtual users bottlenecked at the home page. In reality, they were fine because the landing page was simple and loaded quickly without a bunch of crap. We were happy to coach them to improve the home page, but the real point is that traffic patterns are situational, seasonal, and specific to user types./p pMike makes a good suggestion about finding potential performance problems before they bite you in production. He recommends usage modeling, and I agree. More importantly, you should create multiple test plans to reflect each of your usage models. Test with different scenarios together to reflect what could happen under certain circumstances that are likely to happen - even if your actual server logs don't show that happening right now.br / Usage modeling isn't always just about trying to test different worse-case scenarios either. It's also possibly looking at different ways to utilize excess capacity in the off season or testing to see if dynamic provisioning will support certain growth scenarios. You might also try different usage models just so you can build a taxonomy of different load profiles so later on, when you're monitoring production, you can go back and correlate what you're seeing in production to different load patterns during your testing phase. Sometimes that can provide insight into potential problems. 'Oh yea, I remember seeing that before. Last time we saw that we were….' /p pimg src=http://loadstorm.com/sites/loadstorm.com/files/puzzle-pieces-together.jpg class=picture-left alt=pieces fit together /br / h2Load Testing - Putting the Pieces Together/h2 /ppOne of my favorite hobbies is to trace a web transaction all the way through a system. It isn't always easy. Sometimes I get stuck and need to bring in smarter developers than me - I'm the Omega Geek. It's a challenge that I recommend for you. WARNING: it is tough to do. Worth it? Yes. Easy? No./p pIt takes a lot of time and energy investment to walk through the depths of your application code, database queries, stored procedures, UI code, web services, etc. But I tell you that if you ever do this, you will become so enlightened about how all your system components are interacting. You will know what piece of code is calling the next, which methods are heavily used, which queries are unique to a particular button click, and which third party library is referenced yet unused. /p pIt will help you be the best load tester on your team because you will have the best insight into where the bottlenecks can form. You will have a perception into the inner workings that is an outstanding jump start when you begin trying to tune the system performance./p pIf you work with your whole team (developers, architect, product manager, operations, QA), then everyone learns faster. For example, we have an architect that understands the implications of tweaking settings in a message queuing system that is squarely in my area of ignorance. He has shown me ways to employ the messaging technology to get 1000% throughput increase. I wouldn't have figured that out so quickly (or never) if I had not involved him./p pI like Mike's comment about the team concept of breaking down the system into one transaction flow:br / It's almost like practicing a fire drill for a production issue. Teams that can do this analysis well can respond to production issues well. It's almost like a capstone project – tying together all your models, metrics, and logging into one neat package./p pHere are some good monitoring ideas he presents too:br / Look at the profile of the servers while they are processing that transaction. Understand what the impact might be to the end user. Try to become the world's foremost expert on what's going on in the system, for that thin sliver of time, for that small transaction./p h2Some Things Haven't Changed/h2 pSure, web applications are much more complex in 2010 than they were in 1998 when my company started building them. But in some ways, this whole performance testing issue is just like Dr. Hood taught us back in college. He made us follow a single processing thread through the VMS operating system to understand the impact on the VAX and on our application. In many ways it is easier today because the tools for monitoring and testing are better. In other ways it is more challenging because there are so many moving parts to explore./p pThe bottom line in my not-so-humble opinion is that the best performance testing involves a lot of elbow grease. You must dig in and dig in deeply. It is not a trivial exercise, and you will hit roadblocks along the way that seem insurmountable. However, as long as you set your goal properly to fully understand the inner workings of your application, and if you have the tenacity to push through the complexity of interactions of components, then you will become a world-class performance engineer./p pMy last bit of advice: Don't wait to start. Get going on a deep dive now without reading, researching, and studying all the documentation. You can find the answers once you are clear on the questions posed by your system. Dig in and see what wall you hit first. Then start asking people for help. Your colleagues, product vendors, and online forums are great places to help you over that first challenge and on to the next one. Soon you will be amazed at how thoroughly you understand the performance aspects of your application and its environment./p
Categories: Load & Perf Testing

HP Software Universe 2010 (recap)

Performance Testing with LoadRunner Focus - Mon, 06/28/2010 - 12:52
I just got back from the US after presenting at HP Software Universe 2010. I will save the content of my presentation for another time, so this is a quick overview of what I saw at the Conference and some of the things I learned from the people who attended. Conference Overview HP really know [...]
Categories: Load & Perf Testing

Performance Testing FIFA Cloud Scalability

Performance & Load Testing - Sun, 06/27/2010 - 12:53
pimg src=http://loadstorm.com/sites/loadstorm.com/files/the-links.jpg alt=performance testing links class=picture-left /As we begin another week of summer, here are a few helpful links for articles concerning issues of interest to us load and performance testers. We at LoadStorm have been busy the past month with many new customers asking us for help. /p pWe like people, and we like web performance testing. And we really like people that like web performance testing! Our focus is on providing the best value load testing tool. That said, we often get requests from clients that want us to assist them with understanding the best way to go about it. Some want us to coach them as they figure out what to do about poorly performing web applications. In any case, we certainly enjoy working with customers to make their sites run better./p pWhile we know that we aren't a fit for everyone, it has become clear over the past 18 months that we are helping a bunch of people. Thank you to each client that has sent us a testimonial or even referred us to your geek friends. We sincerely appreciate the word of mouth support we have been getting. /p pWhen we say things like, Please let us know if we can help in any way, we mean it. Seriously. So don't hestitate to tell us if you have questions or need some coaching around performance testing. /p pIn the meantime, we offer these links to other writers and vendors that may be useful to you as you tune amp; test your web applications./p pspan /br / span /br / span /br / img src=http://loadstorm.com/sites/loadstorm.com/files/USA.jpg class=picture alt=performance World Cup //span/span/span/p h2Performance of the World Cup Site/h2 pa href=http://blog.dynatrace.com/2010/06/04/hands-on-guide-verifying-fifa-world-cup-web-site-against-performance-best-practices/ target=_blankVerifying FIFA World Cup Web Site against Performance Best Practices/a/p pAndreas Grabner predicts that the World Cup site will fail. The site has poor performance to start with, and Andreas applies some engineering best practices from which all of us can learn. /p blockquotepMy analysis of the FIFA site shows that – once the World Cup starts next week and the site gets really hit by millions of users around the globe – there is a big chance the site will run into performance and scalability issues due to several best practices that my analysis shows the site does not follow. This failure causes load times of the initial page take more than 8 seconds and requires downloads of more than 200 elements. These problems can easily be fixed by following the recommendations I highlight in this blog./p/blockquote pI especially appreciate how Andreas breaks down the Key Performance Indicators (KPI) into groups, then he shows where each metric is a problem for the FIFA site. He makes great suggestions for where to look for improvement./p pHe gives the World Cup site a failing grade for overall performance. In looking at the 4 categories he uses for measurement, none got a good grade./p ul liBrowser Caching: font color=redF/font – 175 images have a short expires header, 4 have a header in the past /liliNetwork: font color=redF/font – 201 Requests in total, 1 Redirect, 1 HTTP 400, duplicated image requests on different domains /liliServer-Side: font color=redC/font - 10 App-Server Requests with a total of 3.6s - analyze server-side processing /liliJavaScript: font color=redD/font - Use CSS Lookups by ID instead of Class Name pspan /br / span /br / span /br / img src=http://loadstorm.com/sites/loadstorm.com/files/up-arrow.jpg class=picture alt=performance moving up //span/span/span/p h2Cloud Does NOT Mean Infinite Scalability/h2 pa href=http://itexpertvoice.com/ad/scale-and-scalability-rethinking-the-most-overused-it-system-selling-point-for-the-cloud-era/ target=_blankScale and Scalability: Rethinking the Most Overused IT System Selling Point for the Cloud Era/abr / Scott Fulton does a nice job dispelling some myths about cloud computing. Application performance is vital to companies today because of the dollars tied to websites, back office systems, and many forms of business automation. New start-ups utilizing cloud infrastructures are quickly coming out of the gate with disruptive technologies that process huge volumes of data for an ever-increasing user base./p pWe geeks have been saying for years, you can't just throw hardware at the problem. Scott says:/p blockquotepIf you believe that a scalable architecture for an information system, by definition, gives you more output in proportion to the resources you throw at it, then you may be thinking a cloud-based deployment could give your existing system “infinite scalability.” Companies that are trying out that theory for the first time are discovering not just that the theory is flawed, but that their systems are flawed… and now they’re calling out for help./p/blockquote pBusiness people are waking up to the fact that scalability is as important to their success as capitalism and democracy. They usually don't know how important scalability is until their system fail. After customers and revenue are lost, then they look into the underlying causes and find their systems don't really have the performance capability they thought. So in typical manager decision making, they use money to solve the problem by re-deploying on a cloud infrastructure to get much more horsepower. Unfortunately, it is NOT that simple. Cloud platforms are not magic. There are thousands of variables in the scalability equation, and the cloud with massive virtualization is only one factor to improve performance./p pSometimes the solution is to re-architect the system. In fact, Twitter has re-architected their system 4 times already. An academic may tell us to plan for growth, think through all the possibilities and produce the most scalable system from the beginning. Yeah, good idea. But it just isn't that easy because there are too many moving parts. Many of the moving parts are on moving platforms that are getting better, faster, cheaper by the month. Scott tells us to do the best we can with what we have now, and then always be looking ahead for ways to improve performance - including throwing away what you have to build a more scalable application architecture./p pBradford Stephens calls today a transitional period where no one truly has scalability figured out. There is no single right answer for evaluating a scalable platform. But he suggests that virtualization and the cloud make it feasible/affordable to rescale by powers of ten rather than multiples of two. That's exciting to me./p pScott says this:/p blockquotepThis is uncharted territory. What we do know is that businesses can no longer afford to develop solutions around the edges of the core problem. They can’t just point to the symbols for their systems’ various ills (latency, load imbalance) and invest in the symbols that represent their solutions (scalability, the cloud, CMS platforms, social network platforms) as tools for postponing the inevitable rethinking of their business models. Throwing your business model at the cloud doesn’t make the symptoms go away; indeed, it magnifies them. Symbols aren’t solutions./p/blockquote pA quote by Sean Leach in this article matches the philosophy of our CTO, Roger Campbell. Namely, get the app into the customers hands quickly and if you need to fix a scale issue, then that's a good problem to worry about when it arrives. Over 90% of web apps never get much traffic anyway, so why waste time until you know it will be an issue? Wasting months upfront analyzing and planning the best scalable architecture could very well translate to you launching nothing...ever./p pa href= target=_blankx/a/p blockquote/blockquote pa href= target=_blankx/a/p blockquote/blockquote pa href= target=_blankx/a/p blockquote/blockquote pa href= target=_blankx/a/p blockquote/blockquote pa href= target=_blankx/a/p blockquote/blockquote pa href= target=_blankx/a/p /li/ul
Categories: Load & Perf Testing

Velocity 2010 Impressions

Performance & Open Source - Thu, 06/24/2010 - 22:14

Velocity 2010 came to an end today. I attended all 3 days – it was a great conference. I did not attend last year, but the crowds this year must have been at least 3 times that of 2008, when I first presented at Velocity. Here are some of my thoughts on the conference.

Performance

Being a performance person, I am naturally biased towards performance topics. So, I’ll cover this first. All of the performance sessions at the conference can be summed up thus :

The biggest performance issue is the time it takes for the browser to load a web page (aka page load times). Here is technique x and trick y and hack z to help you fix this problem.

I learned a lot about how to optimize css, javascript, http headers etc. But I was still disappointed that there was hardly a whisper about how to optimize the server side. The claim is that of the total response time, the server takes tens or at most 100′s of milliseconds where as the client takes several seconds.  So where do you want to focus your energy on ? I can accept that. But that seems to pre-suppose that all web applications have a scalable architecture and have solved their server-side performance and scalability issues. I find that a little hard to believe.

Operations

As expected, the details of how Facebook, Yahoo and twitter run their operations was of great interest to the audience. With so much media now being served, I was surprised to see only one session on optimizing Video serving and even that was not well attended. There was hardly any talk about optimizing infrastructure. I can’t help wondering why web operations wouldn’t be interested in optimizing their infrastructure costs. After all, we’ve been hearing a lot lately about the cost of power, how data centers are going green, more efficient etc. Aren’t these things applicable to the web world as well (not just enterprise IT) ? Even more surprising, a very small portion of the audience said they were deployed on the cloud.

Neil Gunther and I presented a session  on Hidden Scalability Gotchas in Memcached and Friends.

We had a great audience with some folks squatting on the floor in the front and a standing-room only audience in the back. There was tremendous interest in applying the USL Model to accurate data to quantify scalability. If anyone has additional feedback or comments, I would love to hear them.

Tools

I was blown away by the plethora of tools, a good many of which I had never heard of. Firebug with various add-ons (YSlow, PageSpeed) set the trend on browser-side monitoring and now even commercial vendors have versions of their product (available for free !) to do the same. This is great news for developers. If you haven’t heard of HttpWatch, showslow.com, webpagetest, check them out.DynaTrace announced a free end user response time monitoring tool as well.

Products

One real cool product I came across was Strangeloop – this is an appliance that sits in front of your web server and optimizes the response page. It’s amazing that it can do so much javascript optimization resulting in dramatic reduction in latency. I can’t help wondering why browsers don’t do this ? Surely, Mozilla and Google have enough smart engineers to come up with a an optimized javascript interpreter. It will be interesting to watch.

The usual monitoring vendors were all there – Keynote, Gomez (now part of Compuware), Webmetrics, AppDynamics etc.

Misc.

Tuesday was billed as “Workshop” day. However, there really weren’t any workshops – they were all just regular sessions just longer. I guess it’s hard to do workshops with several hundred people in the room. If Velocity really wants to do workshops, they need to have at least a dozen of them scheduled and they need to be longer.

On the whole, the conference was a great success, with sold out crowds, well attended and delivered sessions and lots of new products. Hope I can make it to Velocity 2011.


Categories: Load & Perf Testing

Quick Throughput Experiment

Performance & Open Source - Mon, 06/21/2010 - 22:05

One of the first things we performance engineers do with a new server application is to conduct a quick throughput experiment. The goal is to find the maximum throughput that the server can deliver. In many cases, it is important that the server be capable of delivering this throughput with a certain response time bound. Thus, we always qualify the throughput with an average and 90th percentile response time (i.e. we want 90% of the requests to execute within the stated time). Any decent workload should therefore measure both the throughput and response time.

Let us assume we have such a workload. How best to estimate the maximum throughput within the required response time bounds ? The easiest way to conduct such an experiment is to run a bunch of clients (emulated users, virtual users or vusers) to drive load against the target server without any think time. Here is how the flow from a vuser will look like :

Create Request ==> Send Request ==> Receive Response ==> Log statistics

This sequence of operations is executed repeatedly (without any pauses in between i.e. no think times) for a sufficient length of time to get statistically valid results. So, to find the maximum throughput, run a series of tests, each time increasing the number of clients. Simple, isn’t it ?

A little while ago, I realized that if one doesn’t have the proper training, this isn’t that simple. I came across such an experiment with the following results :

VUsers Throughput

Requests/sec 5000 88318 10000 88407 20000 88309 25000 88429 30000 88392 35000 88440

What is wrong with these results ?
Firstly, the throughput is roughly the same at all loads. This probably means that the system saturated even below the base load level of 5,000 Vusers. Recall, that the workload does not have a think time. When you have this many users repeatedly submitting requests, the server is certain to be overloaded. I must mention that the server in this case is a single system with 12 cores having hyper-threading enabled. A multi-threaded server application typically will use one or more threads to receive requests from the network, then hand the request to a worker thread for processing. Considering the context-switching, waiting, locking etc. one can assume that at most one can run 4x the number of cores or in this case about 96 server threads. Since each Vuser submits a request and waits for a response, it probably requires 2-2.5x the number of Vusers as the number of server threads to saturate a system. Using this rule of thumb, one would need to run a maximum of 200-250 Vusers.

Throughput with Correct Vusers

After I explained the above, the tests were re-run with the following results:

VUsers Throughput 1 1833 10 18226 50 74684 100 86069 200 88455 300 88375

Notice that the maximum throughput is still nearly the same as from the previous set, but it has been achieved with a much lower number of Vusers (aka clients). So does it really matter ? Doesn’t it look better to say that the server could handle 35000 connections rather than 300 ? No, it doesn’t. The reason becomes obvious if we take a look at the response times.

The Impact of Response Times

The graph below shows how the 90% Response Time varied for both sets of experiments :

The response times for the first experiment with very large number of Vusers ranges in the hundreds of millisecs. When the number of Vusers was pared down to just reach saturation, the server responded hundred times faster ! Intuitively too, this makes sense. If the server is inundated with requests, they are just going to queue up. The longer a request waits for processing, the larger is it’s response time.

Summary

When doing throughput performance experiments, it is important to take into consideration the type of server application, the hardware characteristics etc. and run appropriate load levels. Otherwise, although you may be able to find out what the maximum throughput is, you will have no idea what the response time is.


Categories: Load & Perf Testing

Estimating concurrent users based on past traffic

BrowserMob - Tue, 06/15/2010 - 22:10

Today we received an excellent question from a customer of ours:

We were wondering if you all have any information that says “X Unique visitors per day translates into Y simultaneous users at any given time.”

Essentially, we’re looking for a way to determine how many simultaneous users we should load test with if we know the sites normal daily traffic.

While every site is different, we recommend following this line of reasoning to help you find the answer. Suppose your site gets 100K unique visitors per day, with peak traffic in the mornings and afternoons. Assume that 40% of the traffic comes between 7AM and 11AM, 40% at 4PM to 9PM, and 20% at other times. This means during your peak hours (7AM to 11AM and 4PM to 9PM) you’ll get ~10% of your unique visitors per hour, or 10K uniques in our example.

Now that you know how many unique visitors you’ll get in an hour, you can start turning that in to concurrent users. To do that, it’s important to understand that a unique visitor is roughly equivalent of a transaction. So really you want to figure out how many users you need to reach 10K transactions in an hour.

Suppose your script (or scripts) take an average of 2 minutes to complete. That means a single user will execute 30 transactions in an hour. So to reach 10K transactions, you’d need 334 users (10K divided by 30). If you decide you want to create realistic scripts that include human think time, then the scripts will take that much longer and you’ll need that many more concurrent users. For example, if the script gets 5X think time added and now takes 10 minutes to run, then you’ll need 1,667 users (10K divided by 6).

Of course, this calculation will only get you the load on a typical day (assuming a single hour sees 10% of traffic). Your traffic patterns may vary, or you may want to prepare for a larger surge. For example, if you want to test what happens when 60% of the daily traffic visits in an hour, then you’d need 2,000 users (60K divided by 30).

Tweet This Post 

Categories: Load & Perf Testing

Estimating concurrent users based on past traffic

BrowserMob - Tue, 06/15/2010 - 10:08
Today we received an excellent question from a customer of ours: We were wondering if you all have any information that says “X Unique visitors per day translates into Y simultaneous users at any given time.” Essentially, we’re looking for a way to determine how many simultaneous users we should load test with if we know the [...]
Categories: Load & Perf Testing

Eliminating concurrent access to sensitive data

BrowserMob - Mon, 06/14/2010 - 11:36
We recently had a customer from a large clothing retailer ask us if there was any way to ensure that data, such as a username/password combination, could be restricted such that it was “checked out” and available only for a specific concurrent user. This is very common with logins, where systems often will prevent concurrent [...]
Categories: Load & Perf Testing

Eliminating concurrent access to sensitive data

BrowserMob - Mon, 06/14/2010 - 11:36

We recently had a customer from a large clothing retailer ask us if there was any way to ensure that data, such as a username/password combination, could be restricted such that it was “checked out” and available only for a specific concurrent user. This is very common with logins, where systems often will prevent concurrent logins from multiple IP addresses.

While BrowserMob does not have a concept in which data rows can be “checked out”, some simple scripting can achieve the same results. The key is in creative use of the browserMob.getUserNum() and browserMob.getTxCount() APIs. You can learn more about them by reading up on the BrowserMob APIs.

The getUserNum function returns 0, 1, 2, etc based on the concurrent user in your load test. So if you have a 100 user test, getUserNum will return between 0 and 99. It’s important to understand that it will return the same value for the same user throughout the test.

The getTxCount function returns 1, 2, 3, etc based on the number of cycles for that specific user. This number will effectively be a counter of the unique number of transactions that that particular user has executed. So user 1 and user 100 will both have a getTxCount of 1 returned, but by the time user 100 sees it, user 1 might already be on transaction 50.

Now suppose you want to run a 1000 user test in which you never concurrently log in with the same user. All you need to do is pre-create 1000 user accounts and then write your script like so:

var userId = browserMob.getUserNum(); var username = "test-" + userId; var password = "password"; selenium.type("username", username); selenium.type("password", password);

This works great, but what if you want to use more than 1000 logins? Suppose you want to use up to 10,000 logins among the 1000 user test? This is where the getTxCount function comes in to:

var loginsPerUser = 10; var userNum = browserMob.getUserNum(); var txCount = browserMob.getTxCount(); var userId = userNum * loginsPerUser + txCount % loginsPerUser; var username = "test-" + userId; var password = "password"; selenium.type("username", username); selenium.type("password", password);

What this does is allocate 10 logins per concurrent user. So user 1 will get usernames test-0, test-1, …, test-9 while user 8 will get usernames test-80, test-81, …, test-89, etc. Because of the mod call (%) the ten usernames will simply wrap around once they’ve been used.

Tweet This Post 

Categories: Load & Perf Testing

Advanced handling of page timeouts in Selenium

BrowserMob - Sun, 06/13/2010 - 14:45

Because both our load testing and website monitoring services are based on Selenium, we have a unique ability to measure the performance of things like page load times, AJAX timings, and other in-browser interactions.

Selenium has both a setTimeout command and a waitForPageToLoad command. Both can be given a timeout value, which will control how long Selenium waits for a given page to load or element to appear. When it comes to using our services, most people stick with the default time of 30 seconds. If the timeout is reached, an error is thrown, the script aborts, and the transaction is recorded.

However, sometimes people want to know when pages take more than X seconds to load, but don’t want to necessarily interrupt the flow of the script. In fact, just last week we got this request from a customer:

Ability to trigger an alert based on a set threshold (by the user) – not using the timeout. This basically came from the performance issues we are experiencing. Here’s the scenario:

  • We have a specific page that takes 2+ minutes to load.
  • The page load timeout in the Selenium script was set to 60s.
  • BMob properly reported the “timeout” error but when this error happens, BMob quits the script.
  • This is not ideal for me since I want to still be able to see how long the page takes to load.
  • Increasing the page load timeout for the page in question works, but now I don’t have a way (that I know of) to still trigger an alert after 60s.

I should be able to set a threshold for any page I choose that would then send a notification alert.

In other words, this customer wanted for a way to still report a transaction as a failure and receive an alert, but also still allow the script to continue. Fortunately, our support for JavaScript as a scripting language provides the answer:

var timeout = 90000; ... // start of script ... var start = new Date().getTime(); selenium.waitForPageToLoad(timeout); var end = new Date().getTime(); ... // rest of script ... if ((end - start) > 45000) { throw "An important page took longer than 45 seconds to load"; }

What this script does is sets the timeout to a very long amount (1.5 minutes) but will still report an error if a specific page takes longer than 45 seconds. This allows the remainder of the script to execute even when the page takes more than a specified 45-second threshold.

The only problem with this script is that if the page takes longer than 90 seconds, then the rest of the script will still not run because waitForPageToLoad will throw an exception. You can solve that too with a little code:

var start = new Date().getTime(); try { selenium.waitForPageToLoad(timeout); } catch (e) { // this will happen after 90 seconds // todo: recover and send the browser to the the next URL } var end = new Date().getTime();

The only thing that is important to remember with this use of try/catch is that you’ll need to properly recover from the error. Simply catching the error and trying to continue may not work. For example, if the next Selenium command requires clicking on a button that should have loaded from the last page, there may be no way to recover. However, if the next step is simply visiting a new URL, you could possibly get away with a simple open() command in the catch block.

Tweet This Post 

Categories: Load & Perf Testing

Advanced handling of page timeouts in Selenium

BrowserMob - Sun, 06/13/2010 - 14:44
Because both our load testing and website monitoring services are based on Selenium, we have a unique ability to measure the performance of things like page load times, AJAX timings, and other in-browser interactions. Selenium has both a setTimeout command and a waitForPageToLoad command. Both can be given a timeout value, which will control how long [...]
Categories: Load & Perf Testing

How ASP.NET PostBacks and Redirects Work

Last week I got the following two questions from one of our clients “We use ASP.NET PostBacks but can’t find the PurePath for the request triggering the PostBack handler – any hints?” “We see many ThreadAbortExceptions in our ASP.NET Application and we are not sure why they happen – are they expected?” Time for a little blog that [...]
Categories: Load & Perf Testing

Hands-On Guide: Verifying FIFA World Cup Web Site against Performance Best Practices

Whether you call it Football, Futbol, Fussball, Futebol, Calcio or Soccer – if you are a fan of this game I am sure you are looking forward to the upcoming FIFA World Cup in South Africa. The tournaments web site is http://www.fifa.com/worldcup and allows the fans to follow their teams and get the latest updates [...]
Categories: Load & Perf Testing

Week 16 – Who is to blame for bad application performance?

When something goes wrong who’s to blame? In this post I take a closer look at who is responsible when applications have performance problems. Interestingly very often the question as to who is responsible is at least as important as how to solve the problem. In order to not have to search for the responsible [...]
Categories: Load & Perf Testing

Thomson Reuters on How they deliver High Performance Online Services

We have had some great Webinars with our customers in the past year – such as Zappos, SmithMicro, Peopleclick or Monster. Next week I am co-hosting a Webinar with Eric Thomas, Director of Advanced Engineering at Thomson Reuters. Eric will talk about their online services that millions of people rely on every day and how [...]
Categories: Load & Perf Testing

Week 22 – Is There a Business Case for Application Performance?

We all know that slow performance – and service disruption even more – affects our business services and eventually our revenue. At the same time we say that major parts of companies are not willing to invest in performance. In this post I will discuss why we find ourselves in this paradox and how to [...]
Categories: Load & Perf Testing

How to use your Selenium Tests for automated JavaScript/AJAX Performance Analysis

With more than 3 million downloads, Selenium is the world’s most popular functional testing framework. It allows web developers and QA professionals to automatically test how an application functions on multiple browsers, such as Firefox, Internet Explorer and Chrome and across operating systems, such as different versions of Windows and Mac [...]
Categories: Load & Perf Testing

Web Performance Optimization (WPO) – As Business Critical as SEO

In his latest blog post, Steve Souders writes about the impact of performance on business success. I am not going to duplicate his content here as you can just go ahead and read his blog. I just want to highlight some very interesting points and give you some links to follow up resources. Performance is now [...]
Categories: Load & Perf Testing
Syndicate content