Load & Perf Testing

dynaTrace Forums have been launched to give the community a new platform to interact. dynaTrace AJAX community members will get access to the dynaTrace AJAX Forums – commercial dynaTrace community members will additionally get access to the dynaTrace APM Forums. Whether you have feedback on dynaTrace, feedback or ideas for new dynaTrace Community Plugins [...]

pI have customers frequently ask me why LoadStorm has a minimum pause time. They want the load test to click from step to step in less than a second. My reply is that it is unrealistic to have actions of users like that. /p pimg src=http://loadstorm.com/sites/loadstorm.com/files/load-testing-think-time.jpeg alt=load testing think time class=picture /People have to think about what they see, read it, make a decision, then click. Each person will respond to the delivered page in a slightly different time frame. Think time is variable because real users have a large deviation in their ability to process information. Some are visual learners, some are analytical, some are contemplative, some are driving personalities, /p pSo when you are load testing, it will be beneficial to have a randomized pause between steps in your scenario. This will realistically represent the length of time from when a person receives a response (page) from your server to the time that person requests a new page./p pI used to believe that interpolation for performance metrics was acceptable. However, now I have found that I was wrong. Especially when it comes to think time./p pPlease don't fall into the trap of thinking that a web application will respond linearly with additional load regardless of the think time. For example, just because your app has a two second average response time for 1,000 concurrent users with a 15 second think time does NOT mean that you will get the same response for 3,000 users with a 45 second think time. The algebra may suggest the equation is valid, but it rarely holds up in real testing results. There are simply too many variables in the performance algorithm to make the jump based on pauses between steps./p pSpecifically, it is common for the virtual users in your load test that have longer think times to consume more resources than expected. The way you have coded your application will greatly affect the resources, and thus requests per page, attributable to each virtual user. Your pages may keep requesting data from the server while the user is simply reading. Not all requests require a user action. Also, if the user is aggressively clicking as soon as they spot an interesting link on the display, chances are that several resources won't have even been requested from the server by that time – resulting in less requests per page for aggressive clickers./p pI recommend letting the tool generate a random think time for your scenarios. Most of the books I've read and performance engineers I've talked to will generally put a range on the pauses between 10 to 120 seconds. My suggestion is to review some of your web server logs to get a rough idea of how much time people take on each of your important pages, then put about 20-30 seconds on each side of that. For instance, if I find that people spend an average of 40 seconds on my site pages, I will set a random think time between 20 seconds and 60 seconds./p pNot all people are alike, and not all web applications are alike. Please take these suggestions around think time and apply them to your situation. If you have any additional thoughts or recommendations about pausing between steps in a test scenario, please post them here in the comments. We always welcome your contribution./p

pa href=http://www.marketingcharts.com/uncategorized/gartner-worldwide-mobile-device-sales-grew-138-in-q2-13890/?utm_campaign=newsletteramp;utm_source=mcamp;utm_medium=textlinkGartner reports 13.8% increase in mobile device sales/a in the second quarter of 2010 over 2009 Q2./p pLoad testers should take note that global access to web sites is becoming more common from smartphones. In fact, smartphone sales are now up 50.5% from the same period in 2009./p pAccess the web from a mobile device gets cheaper and easier all the time. As web developers, we should all be enabling our apps and sites for mobile access. /p pI was surprised to see that Apple only has a 2.7% share of the overall mobile device market./p pAndroid also overtook Apple’s iPhone OS to become the third-most-popular OS in the world./p pAny thoughts on how load testing should account for all the mobile device access?/p

File Transfer Errors are very common during initialization of load test with LoadRunner. It can be because of following reasons.br / br / br / 1. There may be situations where some of the runtime files of the script are not included by the controller while transferring files to different load generators (may be a bug in controller). It may not transfer all the files required for the script to run successfully. Under those situations if the scripts give compilation errors in different load generators other than local host then manually add the missing files using Add button shown in the snapshot below.br / br / div class="separator" style="clear: both; text-align: center;"a href="http://4.bp.blogspot.com/_QGJjHMRUTpY/TGkn2ZtzqhI/AAAAAAAAApw/XZP4TYT_Xok/s1600/filetransfer.bmp" imageanchor="1" style="clear: left; cssfloat: left; float: left; margin-bottom: 1em; margin-right: 1em;"img border="0" ox="true" src="http://4.bp.blogspot.com/_QGJjHMRUTpY/TGkn2ZtzqhI/AAAAAAAAApw/XZP4TYT_Xok/s320/filetransfer.bmp" //a/divbr / nbsp; br / nbsp; br / nbsp; br / nbsp; br / nbsp; br / nbsp; br / nbsp; br / nbsp; br / nbsp; br / nbsp; br / nbsp; br / 2. If the Length of the script name is too large then LoadRunner will throw error messages like below br / br / br / span style="color: red;"Error: Mercury File Transfer error: Cannot create file C:\Documents and Settings\autodesk-admin\Local Settings\Temp\brr_mYv.195\netdir\C\Performance_Test\QAAutomation\Performance\LR\Stage\700ExodusSFDC\CallCenter\CC_SFDC_CreateGSSRequest_fromContactScreen_Attachment\CC_SFDC_CreateGSSRequest_fromContactScreen_Attachment.prm. Reason : Check the file path for illegal characters or length./spanbr / br / To double check if the length is the issue, go the LoadGenerator and try to add the file in the specified path. You may get the error as belowbr / br / div class="separator" style="clear: both; text-align: center;"a href="http://3.bp.blogspot.com/_QGJjHMRUTpY/TGkoKnmlFuI/AAAAAAAAAp4/YK7pbp4Fvrg/s1600/error.bmp" imageanchor="1" style="clear: left; cssfloat: left; float: left; margin-bottom: 1em; margin-right: 1em;"img border="0" height="80" ox="true" src="http://3.bp.blogspot.com/_QGJjHMRUTpY/TGkoKnmlFuI/AAAAAAAAAp4/YK7pbp4Fvrg/s400/error.bmp" width="400" //a/divdiv style="border-bottom: medium none; border-left: medium none; border-right: medium none; border-top: medium none;"br / /divdiv class="blogger-post-footer"img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/3081868611305126410-6861034374838639026?l=performancetestinginfo.blogspot.com' alt='' //div pa href="http://feedads.g.doubleclick.net/~a/ADoFXNqZZ-Nkrjxf78OAhXJoQVY/0/da"img src="http://feedads.g.doubleclick.net/~a/ADoFXNqZZ-Nkrjxf78OAhXJoQVY/0/di" border="0" ismap="true"/img/abr/ a href="http://feedads.g.doubleclick.net/~a/ADoFXNqZZ-Nkrjxf78OAhXJoQVY/1/da"img src="http://feedads.g.doubleclick.net/~a/ADoFXNqZZ-Nkrjxf78OAhXJoQVY/1/di" border="0" ismap="true"/img/a/pimg src="http://feeds.feedburner.com/~r/InfoOnPerformanceTesting/~4/fNH5LJz2Vwk" height="1" width="1"/

I had always wanted to work at a company that had a library of intereresting/educational books I could borrow; so when I started at JDS Australia in late 2005, I brought in a few of my technical books in case anyone else wanted to borrow them. Gradually other people brought in books too, until there [...]

I am hosting a Webinar with IntraLinks this Wednesday. The following overview of the webinar is taken from the guest blog I wrote for the IntraLinks blog as posted on July 13th 2010: —- The Cloud has become the hot topic in the past year, with many major technology leaders (Amazon, Google, Microsoft, SalesForce.com, etc.) [...]

h3How Do We Know If We Are Successful?/h3 pOne of the over-arching objectives of load testing is to measure system capacity. When you are load testing, what are the goals? Generally, everyone wants to know how much load (user traffic) the system can handle without significant performance degradation. What is significant degradation? /p pBefore you begin load testing, I urge you determine what you need from the testing. Is there a service level agreement that your company must fulfill? Are you trying to make a decision about releasing a new web application into production? Does the CTO just want to make the CEO feel good that we've done a good job with a new version? Are you creating a performance benchmark? When will we know we are finished with load testing?/p pThink about the goal of the load test. What are the success criteria and how will we know when the objective is met?/p pThe best answer for all IT questions applies here...it depends. Ok, that's sort of a cop-out. Another good answer that is used by most IT consultants can be used for load testing success criteria...Whatever will make the customer happy. /p pYes, we need to keep customer satisfaction in mind. Yes, we need to understand who our stakeholders are. Yes, we need to figure out what our boss calls success. Yes, we need to talk to the alpha geek in the basement - he knows everything./p pRegardless of how you get there, it is imperative that your team decides upfront what constitutes acceptable performance for your web application. /p pThat said, you should define “good performance” in terms of something measurable. Specifically, we recommend picking metrics that are meaningful to your stakeholders. For instance, 5,000 concurrent users with sub-second average response time and less than 0.3% error rate. /p h3It Must Be Measurable/h3 pSo the load goals should be clearly quantified, such as 25,000 concurrent users or 400 requests per second. Some people like to measure the amount of bytes flowing in and out of the network - throughput or bandwidth. Everyone wants to understand how the user perceives the system, so a metric like Response Time is paramount. Measuring the errors returned from the server is another key performance indicator. If average response time is 300 ms but you are seeing 500 Internal Server errors, that's a very bad thing because that's what the users will see. A quick error is still an error. /p pKnow the targets for load. Know the targets for measuring performance. The objectives must be expressed as something measurable and indisputable./p h3The Specification/h3 pWhere do you get the load goals? Often, there are specifications that define the number of users that are expected in a particular period of time, and it is assumed that everyone on the product team is concerned with meeting the expectations in the specification because stakeholders typically view the spec as a contract. /p h3Find the Peak/h3 pWhen there is no spec, developers normally have a plan for the number of users they want the app to support without slowing down or generating errors. I recommend looking at the existing server logs to find the peak times of traffic. If you find that 8:00 a.m. EST is 10x the normal traffic, start there. Let's say that your web application has 1,000 users login between 7:55 and 8:15 a.m., and they hit an average of 5 pages. I would set my load testing goal to be somewhere around 2,500 because I want to know how the system handles 2.5x the regular peak traffic. It may not get there, but as the load tester, I want to know where my app is stress to failure. /p h3Current Traffic Plus Some/h3 pIf you have no existing server logs because you are developing a brand new application, then talk to the marketing people. I guarantee there is a VP of Marketing (or similar title) in your company that has a number. That person has been assigned an objective for driving traffic to the site through various marketing activities such as advertising campaigns. in fact, many times that site traffic objective will be one of the measurements used during that marketing person's job evaluation. /p h3Make a Friend in Marketing/h3 pCan you think of a better way to endear yourself to that VP than to sit in her/his office and strategize beforehand about handling that volume? They are marketing and probably just assume the site will take the traffic with no problem. They probably won't sit around and ponder your load testing metrics, but they will definitely have an idea of what they expect from the marketing perspective. Also, they understanding measurements. Their metrics may be about conversion rates, dollar to lead ratios, and views per click. Make no mistake...they understand measuring activity and results./p pWhatever you do - find out who is writing the check. Who is the financial sponsor? Where is the money coming from that is paying for the testing? Identify that person and other aspects of the project start to come into focus (surprise). I suggest you interview that person...maybe more than once. Figure out what they need from the site. What do they need from the application? How does it impact their life? If it fails completely, where do they hurt the most? Clearly and precisely understand the pain related to this site's performance./p pIt wouldn't hurt to check with the legal department too. /p pSo by reaching out to them proactively, they will view you as a leader and an ally in the geek department. That is a good thing. While you are there, ask them what other expectations they have. Write it all down. You should try to use any of that information in the development of your load testing requirements./p h3Resource Utilization/h3 pHere is where we geeks get excited. Server side metrics seem to be more scientific, and thus, more interesting to programmers and engineers. /p pIs the database server's memory more than 80% consumed when the system is supporting 12,000 kilobytes per second of throughput? How much of the web servers' CPUs are being used when we are handling 350 requests per second? Those are cool questions for us nerds. More importantly, those are metrics you can use in your success criteria. I encourage your team to brainstorm about each major component of your system - with the output of such an exercise to be goals for measurements such as percentages of utilization for memory, CPU, and network bandwidth./p pSome people include stress testing when they discuss load testing, and stress testing is about breaking the system. In light of discussing resource utilization, you should consider how any component of the application's architecture affects the overall performance. What happens if a disk drive dies during heavy web traffic? How do your bottlenecks change when one of the load balancing machines must be rebooted? /p pWhen defining success criteria for your load tests be sure to identify constraints in your system, and set some targets for extreme conditions. /p h3Conclusion/h3 pInvest the time early to clearly understand what is going to make or break your load testing efforts. Get others involved. Define the measurements for input/load and for output/performance. You need a stake in the ground, even if it is your best guess. /p pI strongly recommend that you understand what metrics are going to make your boss happy. Don't overlook the other important stakeholders that reside in other departments like marketing or sales. Having your ducks in a row will always be a benefit to you personally as others assess your success. /p pI have heard it said that response time is a user concern, resource utilization is a system concern, and throughput is a business concern. That sounds clever, but my opinion is that it's a bit too trite to be useful. However, the point can be applied generally if it makes you realize how important it is to get different perspectives. You as a system architect, programmer, tester, or performance engineer will not have the same point of view for load testing as the Program Manager./p pAs for the load testing success, the good news is that it is objective. Measurable is unambiguous. Set the goals upfront, and your job is going to be much easier./p

I occasionally get invited to do JavaScript/AJAX Performance Workshops. Last week I spent two days with a group of dynaTrace AJAX Users that work in a performance task force group within their R&D Organization. I asked them about the reasons for this Client-Side Performance Initiative. They told me the following story: One of our sales managers [...]

Last week’s Walkthrough Webinar on the new dynaTrace AJAX Edition 2.0 Beta 1 is now available for public access: View Webinar on ajax.dynatrace.com If you have any feedback on either the new dynaTrace AJAX Edition or the Best Practice Documents then please use our Community Forum an let us know about it. //

p a href="http://code.google.com/p/corey-projects/source/browse/trunk/python2/memcached/memcached_stats_rrd.py"memcached_stats_rrd.py/a is a script for monitoring and graphing stats from Memcached or Membase. /p p Memcached is a high-performance, distributed memory object caching system, and Membase is the related key-value database management system. Both are open source, with packaged/commercial versions distributed by a href="http://www.northscale.com/"NorthScale/a. They both use the memcached protocol for communication, so this script will work against a vanilla memcached installation, or against a membase server/cluster. /p p For more info: /p ul liMemcached: a href="http://memcached.org"http://memcached.org/a/li liMembase: a href="http://membase.org"http://membase.org/a/li /ul hr / p script source code: a href="http://code.google.com/p/corey-projects/source/browse/trunk/python2/memcached/memcached_stats_rrd.py"memcached_stats_rrd.py/a /p p This script is useful for ad-hoc monitoring or longer term trend/capacity analysis. It collects data, stores it in RRD databases, and outputs graphs and stats in the form of PNG images. You can monitor any stats that memcached/membase publishes and graph them over specified time spans. It will generate an image file for each stat, for each time span selected. /p pThis mini monitoring system is built with:/p ul lia href="http://pypi.python.org/pypi/python-memcached/"python-memcached/a/li lia href="http://oss.oetiker.ch/rrdtool/"RRDTool/a/li lia href="https://help.ubuntu.com/community/CronHowto"cron/a (or other task/job scheduler)/li /ul pSample Graph/Image Output:/p a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_k7-jvtv2cLo/TER96OYSzxI/AAAAAAAAAr4/aPRpI4t4pIs/s1600/memcached_rrd_bytes_read_180.png"img style="cursor:pointer; cursor:hand;width: 400px; height: 182px;" src="http://1.bp.blogspot.com/_k7-jvtv2cLo/TER96OYSzxI/AAAAAAAAAr4/aPRpI4t4pIs/s400/memcached_rrd_bytes_read_180.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5495655884544724754" //a br / (bytes_read, 3 hour timespan, 60 sec collection interval)br / a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_k7-jvtv2cLo/TER96V9-zkI/AAAAAAAAAsA/5SBkZx7_hw4/s1600/memcached_rrd_mem_used_60.png"img style="cursor:pointer; cursor:hand;width: 400px; height: 182px;" src="http://2.bp.blogspot.com/_k7-jvtv2cLo/TER96V9-zkI/AAAAAAAAAsA/5SBkZx7_hw4/s400/memcached_rrd_mem_used_60.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5495655886581845570" //a br / (mem_used, 1 hour timespan, 60 sec collection interval)br / p It will generate images in the directory you specify. I have an Apache web server installed serving content from the output directory for easy web viewing. You can wrap the images in some HTML and create a little dashboard to watch your entire memcached/membase cluster, like this: /p a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_k7-jvtv2cLo/TER95p0g-PI/AAAAAAAAArw/SHFlsaDV2jM/s1600/memcached_rrd_2node_dashboard_240.png"img style="cursor:pointer; cursor:hand;width: 400px; height: 338px;" src="http://1.bp.blogspot.com/_k7-jvtv2cLo/TER95p0g-PI/AAAAAAAAArw/SHFlsaDV2jM/s400/memcached_rrd_2node_dashboard_240.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5495655874730981618" //a br / (4 hour timespan, 60 sec collection interval, 2 nodes, 3 stats each)br / hr / p Instructions... /p p strong1)/strong Install prerequisites /p p You will need the following software installed: /p ul liPython 2.x/li lipython-memcached (memcached client for Python)/li liRRDTool (round-robin database, logging/graphing backend)/li /ul p on Debian/Ubuntu: /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;" $ sudo apt-get install -y python-memcache $ sudo apt-get install -y rrdtool /pre p strong2)/strong Configure the script /p p Near the top of the script, there are a few configuration settings: /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;" # Config Settings NODES = ('192.168.1.3:11211', '192.168.1.4:11211') INTERVAL = 60 STATS = (('curr_items', 'GAUGE'), ('bytes_written', 'COUNTER')) GRAPH_MINS = (60, 180) GRAPH_DIR = '/var/www/' /pre pConfig Setting Definitions:/p ul liNODES: list of memcached/membase nodes to monitor./li liINTERVAL: collection interval in seconds. This should be the same value as you schedule the script to run./li liSTATS: list of tuples containing (stat_name, datasource_type), where "stat_name" is a memcached/membase stat, and "datasource_type" is an RRDTool data source type (DST). The most useful data source types are GAUGE and COUNTER. GAUGE is used to report a current value, and COUNTER is used for continuous incrementing counters. (see: a href="http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html"http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html/a for more info on RRD data sources)/li liGRAPH_MINS: list of minutes, corresponding to time spans displayed in the output graphs. An image for each stat is generated for each value here./li liGRAPH_DIR: directory to generate out images in./li /ul p After you have the script configured, make the script executable: /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;" $ chmod +x memcached_stats_rrd.py /pre p strong3)/strong Schedule the script /p p You can add an entry to your crontab (crontab -e) so cron will run it regularly. The example here uses a 60 sec (1 min) interval: /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;" */1 * * * * /home/perfserver/memcached_stats_rrd.py /pre p em[These instructions are for Linux/Unix, but you can configure a similar system on Windows using Task Scheduler instead of cron. The code in memcached_stats_rrd.py works cross-platform.]/em /pdiv class="blogger-post-footer"img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/5236867476487043111-7035108896037480077?l=coreygoldberg.blogspot.com' alt='' //div

pem(originally posted at a href="http://coreygoldberg.blogspot.com/"coreygoldberg.blogspot.com/a)/em/p p I had to test and benchmark a a href="http://en.wikipedia.org/wiki/SOAP"SOAP/a web service recently, and figured I'd write up some instructions for load testing with Multi-Mechanize. /p p Multi-Mechanize is a performance and load testing framework that enables you to write plugins (in Python) to create virtual user scripts that run in parallel against your service. /p ul liProject Home: a href="http://multimechanize.com"multimechanize.com/a/li liVirtual User Script Development Guide: a href="http://code.google.com/p/multi-mechanize/wiki/DevelopingScripts"DevelopingScripts/a/li /ul p Since Multi-Mechanize scripts are pure Python, you can use any Python module inside them. In the case of a web service that uses SOAP, you have a few options. You can use any 3rd party module, so something like a href="http://pypi.python.org/pypi/suds"Suds/a (a lightweight SOAP client lib) would be a reasonable choice. However, I decided to stick with the Python standard library and build my scripts with a href="http://docs.python.org/library/urllib2.html"urllib2/a. Afterall, SOAP is just slinging a bunch of XML back-and-forth over HTTP. /p p So before I get started with Multi-Mechanize and performance/load testing... /p p Here is a Python (2.x) script using urllib2 to interact with a SOAP web service over HTTP: /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;" import urllib2 with open('soap.xml') as f: soap_body = f.read() req = urllib2.Request(url='http://www.foo.com/service', data=soap_body) req.add_header('Content-Type', 'text/xml') resp = urllib2.urlopen(req) content = resp.read() /pre p em(The script above assumes you have a file named 'soap.xml' in the local directory that contains your payload (SOAP XML message). It will send an HTTP POST request containing your payload in the body.)/em /p p After the initial script is created in Python, the next step is to convert it into a Multi-Mechanize script. To do this, you create a Transaction class with a run() method and add your code there. /p p As a Multi-Mechanize script, the same thing would be done like this: /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;" import urllib2 class Transaction(object): def __init__(self): with open('soap.xml') as f: self.soap_body = f.read() def run(self): req = urllib2.Request(url='http://www.foo.com/service', data=self.soap_body) req.add_header('Content-Type', 'text/xml') resp = urllib2.urlopen(req) content = resp.read() assert ('Example SOAP Response' in content), 'Failed Content Verification' /pre p em(Notice I also added an assert statement to verify the content returned from the service.)/em /p p Now that you have a script created, it's time to configure Multi-Mechanize and run some tests. /p p You can download Multi-Mechanize from: a href="http://code.google.com/p/multi-mechanize/downloads/list"code.google.com/abr / It requires Python 2.x (2.6 or 2.7), and if you want it to generate graphs, you must also install Matplotlib and its dependencies. See the a href="http://code.google.com/p/multi-mechanize/wiki/FAQ"FAQ/a for help. /p p Once you have Multi-Mechanize downloaded, unzip it and go to the "/projects" directory. You can create a new project directory here. You can call it "soap_project". Inside this directory, you will need 2 things: a config file ("config.cfg"), and a "test_scripts" directory (containing the script you previously created, which you can call "soap_client.py". Since the script is looking for a data file named "soap.xml", make sure you have one created in the main multi-mechanize directory. /p p The directory and file layout will look like this: /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;" /multi-mechanize /lib /projects /soap_project /test_scripts soap_client.py config.cfg multi-mechanize.py soap.xml /pre p em(you can see the "default_project" for an example of how it should be setup.)/em /p p To begin with, you can use a simple config.cfg file like this: /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;" [global] run_time: 30 rampup: 0 console_logging: on results_ts_interval: 10 [user_group-1] threads: 1 script: soap_client.py /pre p This will just a run a single thread of your virtual user script for 30 seconds; good enough for testing and getting things going. /p p To run a test, go to the topmost multi-mechanize directory and run: /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;" $ python multi-mechanize.py soap_project /pre p em(You should see timer output in the console.)/em /p p Once you have things running well, you can turn off "console_logging" and increase the workload. It will take some adjustment and plenty of trials to get the load dialed in correctly for your service. You can also get a lot more sophisticated with your workload model and create multiple virtual user scripts doing different transactions. In this case, I'll keep it simple and just test one web service request type in isolation. /p p Once I ran plenty of iterations and got a feel for how the service responded and what its limits were, I settled with a config file like this: /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;" [global] run_time: 900 rampup: 900 console_logging: off results_ts_interval: 90 [user_group-1] threads: 30 script: soap_client.py /pre p em(30 threads, increasing load over 15 mins)/em /p p After each test run you do, a new results directory is created containing your test results. Look for "results.html" and view it in your browser to see the output report. /p hr / br / p Have questions about a href="http://multimechanize.com"Multi-Mechanize/a?br / Post to the discussion group: a href="http://groups.google.com/group/multi-mechanize"groups.google.com/group/multi-mechanize/a /pdiv class="blogger-post-footer"img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/5236867476487043111-4760682998727562735?l=coreygoldberg.blogspot.com' alt='' //div

pimg src=http://loadstorm.com/sites/loadstorm.com/files/endurance-testing.jpg alt=endurance testing duration of load testing class=picture width=200 /On the Software QA and Testing Resource Center site, there is a post about a href=http://sqa.fyicenter.com/FAQ/Stress-Testing-Web-Sites/Estimating_Test_Duration.html target=_blankEstimating Test Duration/a. Here is a quote for your educational benefit:/p blockquotepThe duration of the peak is also very important-a Web site that may deal very well with a peak level for five or ten minutes may crumble if that same load level is sustained longer than that. You should use the length of the average user session as a base for determining the load test duration. /p/blockquote pFor some of my thoughts on endurance testing (or soak testing), see the post on a href=http://loadstorm.com/2010/long-duration-load-testsLong Duration Load Tests/a./p

JavaScript can save your day or it can cause you nightmares. JavaScript and XHR (XmlHttpRequest) enable what the industry considers to be Web 2.0 – meaning highly interactive web sites where some application logic is pushed down to the client into the browsers JavaScript engine. As with any application code – [...]

On increasingly rare occasions, you might be required to load test an application that has a Java-based client. While it’s nice for users to have an application that has a rich GUI, it will probably lead to a world of pain for you as a performance tester. Don’t immediately reach for LoadRunner’s Java Record-Replay virtual [...]

Over the last couple of years performance of web applications became more important to businesses as search engines (such as Google) factor in performance into their ranking. This ultimately leads to Performance == Better Visibility == More Users == More Revenue. Read more on Why Web Performance Optimization [...]

p Have you seen the network/bandwidth caps that a href="http://www.rackspacecloud.com"Rackspace/a places on their cloud servers? /p p From the a href="http://cloudservers.rackspacecloud.com/index.php/Frequently_Asked_Questions"Rackspace Cloud Server FAQ/a: /p blockquote p em "Your Cloud Server has a bandwidth cap that is applied in relation to the size of the server. Please note that the bandwidth caps are placed on outbound bandwidth only. Inbound bandwidth is not capped." /em /p table trtdServer Size/tdtdPublic Limit/tdtdServiceNet Limit/td/tr trtd256MB/tdtd10 Mbps/tdtd20 Mbps/td/tr trtd512MB/tdtd20 Mbps/tdtd40 Mbps/td/tr trtd1024MB/tdtd30 Mbps/tdtd60 Mbps/td/tr trtd2048MB/tdtd40 Mbps/tdtd80 Mbps/td/tr trtd4096MB/tdtd50 Mbps/tdtd100 Mbps/td/tr trtd8192MB/tdtd60 Mbps/tdtd120 Mbps/td/tr trtd15872MB/tdtd70 Mbps/tdtd140 Mbps/td/tr /table /blockquote p I was going to use the Rackspace Cloud to do some performance testing of a new server. I wanted to run 2 servers in a clustered mode, replicating data to each other. My first concern was the speed of the internal network interconnects between cloud nodes. /p p After doing some research, I realized the bandwidth caps make it a non-starter. 20-140 Mbps private interconnects?? That's not enough. I would saturate the network almost immediately, even on their largest server class. Sorry Rackspace. /p p In comparison, a href="http://aws.amazon.com/ec2/"Amazon's EC2 Cloud/a offers High Performance Computing nodes with 10 Gigabit interconnects: a href="http://aws.amazon.com/hpc-applications/"Amazon HPC/a /pdiv class="blogger-post-footer"img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/5236867476487043111-3217723913347784700?l=coreygoldberg.blogspot.com' alt='' //div

pI've known some web Product Managers that require at least one 72 hour load test on a monthly basis. In fact, a couple are customers of LoadStorm. /p pThe reasoning behind conducting such long tests is that memory leaks can be very difficult to find in a 1 hour test and should be detectable after 72 hours. Comparisons of memory consumption in the first hour to the 72nd hour under the same load may expose any defects in the code that leave memory wasted. /p pI call this endurance testing. Many developers refer to it as soak testing or reliability testing. The goals for success of your load testing would be better to include at least one long duration test because it checks the reliability in a way that cannot be tested in any other way. /p pWikipedia defines soak testing as:/p blockquotepSoak testing involves testing a system with a significant load extended over a significant period of time, to discover how the system behaves under sustained use./p/blockquote pMike Kelly at SearchSoftwareQuality believes endurance testing and soak testing are slightly different:/p blockquotepIt's similar to endurance testing and burn-in testing, and is often used interchangeably. For me, the key distinction to soak testing is that load remains high throughout the run, while with an endurance test my load may vary based on the usage model I'm executing./p/blockquote pFor my purpose, I'll use these terms interchangeably. The only difference seems to the volume that Mike uses. I view the duration as the significant factor that makes a soak test./p pIt's common to overlook endurance testing simply out of ignorance of its importance. Even worse, programmers may think too highly of their skills to consider memory leaks a possiblity. What is probably the most significant reason for skipping soak testing? Budget. Running a load test for a day or two consumes resources; thus, it's viewed as expensive. To justify it, calculate the cost of shutting down the system to recover from a failure. It might be easy to get your stakeholders to agree to include endurance testing in your success criteria when they factor in the downtime. /p pHow long can your web application process heavy load without errors or failures? Best to find out now rather than later./p

p I just started a new discussion group for WebInject:br / br / a href="http://groups.google.com/group/webinject"http://groups.google.com/group/webinject/a /p p The old forums (a href="http://www.webinject.org/cgi-bin/forums/YaBB.cgi"http://www.webinject.org/cgi-bin/forums/YaBB.cgi/a) have been broken for several years. I will leave them up in read-only mode, but some threads are broken and unviewable. /p p Feel free to post a message to this new discussion group.br / questions, bugs, patches, collaboration, comments, welcome... /p p Updates and new development coming soon... /p p br / ema href="http://www.webinject.org"WebInject/a is a free tool for automated testing and monitoring of web applications or services. It can be used standalone, or as a plugin for external monitoring systems./em /pdiv class="blogger-post-footer"img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/5236867476487043111-3914653040547693196?l=coreygoldberg.blogspot.com' alt='' //div

I like posting LoadRunner code snippets on My Load Test and, judging by the emails and comments that I get, a lot of people find them really useful. To make my code more readable, I have added syntax highlighting that uses the same colours you see in VuGen. If you would like to download a [...]

p em[a href="http://membase.org/"Membase/a is a high-performance, distributed key-value database (NoSQL)]/em /p p To check operational stats of your Membase server/cluster, you can use the telnet interface (port 11211) and issue the stats command to any individual node: /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;" $ telnet 192.168.12.11 11211 stats /pre p The output is unsorted and a little difficult to read at quick glance. /p hr / p I use the following Python script for getting a quick snapshot of stats from Membase in a more readable format. Since Membase speaks the a href="http://memcached.org/"Memcached/a protocol, I use the a href="http://pypi.python.org/pypi/python-memcached"python-memcached/a module. List all nodes you want stats from. /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;" #!/usr/bin/env python # Corey Goldberg - 2010 (goldb.org) # print a stats report from membase key-value database (membase.org) # python 2.x # requires python-memcached import memcache NODES = ('192.168.12.11:11211',) mc = memcache.Client(NODES) for node_stats in mc.get_stats(): server, stats = node_stats print '-----------------------------------------' print server print '-----------------------------------------' for stat_name, value in sorted(stats.iteritems()): if not stat_name.startswith('ep'): if stat_name not in ('libevent', 'version'): print stat_name.ljust(25), value.rjust(15) print '-----------------------------------------' for stat_name, value in sorted(stats.iteritems()): if stat_name.startswith('ep'): if stat_name not in ('ep_dbname', 'ep_version'): print stat_name.ljust(25), value.rjust(15) /pre p sample output (1 node): /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;" $ python membase_stats_report.py ----------------------------------------- 127.0.0.1:11211 (1) ----------------------------------------- auth_cmds 0 auth_errors 0 bytes_read 81754885 bytes_written 77239947 cas_badval 0 cas_hits 0 cas_misses 0 cmd_flush 1 cmd_get 370229 cmd_set 380230 conn_yields 0 connection_structures 16 curr_connections 16 curr_items 178679 daemon_connections 10 decr_hits 0 decr_misses 0 delete_hits 0 delete_misses 0 get_hits 370228 get_misses 1 incr_hits 0 incr_misses 0 limit_maxbytes 67108864 mem_used 40909042 pid 2009 pointer_size 64 rejected_conns 0 rusage_system 65.660000 rusage_user 113.320000 threads 4 time 1278257466 total_connections 16 uptime 592 ----------------------------------------- ep_commit_time 1 ep_data_age 286 ep_data_age_highwat 286 ep_dbinit 0 ep_flush_duration 1 ep_flush_duration_highwat 2 ep_flusher_state running ep_flusher_todo 483 ep_item_commit_failed 0 ep_item_flush_failed 0 ep_max_txn_size 50000 ep_min_data_age 1 ep_queue_age_cap 5 ep_queue_size 176786 ep_storage_age 286 ep_storage_age_highwat 286 ep_tap_keepalive 0 ep_tap_total_fetched 0 ep_tap_total_queue 0 ep_too_old 1410 ep_too_young 56372 ep_total_enqueued 380487 ep_total_persisted 198703 ep_warmed_up 479990 ep_warmup true ep_warmup_thread complete ep_warmup_time 4 /pre p script source code: a href="http://code.google.com/p/corey-projects/source/browse/trunk/python2/memcached/membase_stats_report.py"membase_stats_report.py/a /p p For more advaned stats and graphing from Memcached and Membase, see:br / a href="http://coreygoldberg.blogspot.com/2010/07/monitoring-stats-from-memcached-or.html"http://coreygoldberg.blogspot.com/2010/07/monitoring-stats-from-memcached-or.html/a /pdiv class="blogger-post-footer"img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/5236867476487043111-2493719226427323488?l=coreygoldberg.blogspot.com' alt='' //div