Load & Perf Testing

pThere is a difference between performance testing of a traditional client server application versus a web application./p pWeb performance testing involves primarily HTTP traffic. There are latencies involved with Internet technologies that you shouldn't encounter on an internal software implementation./p pWeb performance testing can be the measuring and analysis of a single web page. Taking apart the download times of each individual resource such as an image, an HTML document, or a stylesheet. Web performance testing should also involve measuring the DNS lookup times and how long it takes to get a connection to the server. /p pRendering is a key element of understanding the performance of a web page. Specifically, how long after the user makes a request does the browser start to paint something. /p pIt is also critical to measure the actual server processing time for a request. The browser side perspective of the end user is important, and that is where large images or too many files can cause poor user experience. Additionally, if the database is a bottleneck that results in 5 seconds to put together a dynamic page, that is a performance problem./p

pimg src=http://loadstorm.com/sites/loadstorm.com/files/load-testing-ignored.jpg class=picture alt=load testing disconnect /br / h2CTOs Don't Get It/h2 /ppOne of the most disruptive events at organizations is the disconnect between decisions made by managers and the views of those decisions by employees. Does this situation sound familiar? /p pYour CTO or CIO makes a decision to forego load testing until after the launch of your web application. Your team responds by talking about how the higher ups don't get it, they don't understand the issues with system performance, they are making shortsighted decisions, and they are ignoring the product development team. You have seen this before - management is saying postpone and they mean cancel./p pBefore CTOs were managers, they were individual employees. Most became managers because of their skills as individuals in technical problem solving and decision making. So why the disconnect between your team and the CTO?/p pLet's break it down so we as load testers can affect change. /p pspan /br / span /br / span //span/span/span/p h2Money, Money, Money/h2 pMost companies are always budgeting and adjusting budgets. The senior executives are paid the big bucks to make sure the investors are happy with results, and the most important result is the bottom line profitability. Profitability is driven by income and expenses, so every dollar saved on the expense side is a win for the executives (and investors). /p pLoad testing is mistakenly viewed as non-critical. It is an easy target for CTOs in budget cutting. Bugs are visible and immediately get the attention of users, which quickly reaches the CEO and other top managers. It rolls downhill to the CTO very quickly. Bugs are bad. Bugs are universally understood./p pimg src=http://loadstorm.com/sites/loadstorm.com/files/MONEY-in-trash.jpg class=picture-left alt=load testing money lost /However, if the site has 5 second response time instead of a 2 second response time, rarely does the CEO get involved. It isn't as noticeable. It isn't mission critical. Thus, it isn't nearly as high a priority to improve performance as it is to fix bugs. The conclusion is obvious and logical to put more resources on the functional testing than on load testing. In fact, CTOs cut the load testing out of the budget submitted by the development team without having to explain that decision to anyone. It's easy for them. Who is going to raise a stink about it? The testers? The QA Manager? The Product Manager? The programmers? You? Probably not. /p pIt is too hard to defend! It's tough to make an argument that gets attention. You put yourself out on a limb if you stir up trouble about wasting money on performance testing. You don't want to get called into the CTO's office to explain the importance of load testing and system performance under heavy traffic. /p pSo what I've seen is a dynamic where the testers and/or developers take an attitude of, They will find out when the system crashes, and I'll just sit here and smile. Most of us know that we are right about the importance of performance, so if the managers are out of touch and don't listen to us, then they deserve what they get./p pOf course, that isn't good for our company. Translate that to, Ignoring load testing is bad for my job security. Now it clearly becomes something that can cost our company credibility, money, and ultimately jobs. Hmmm....that isn't worth being smug about. We need to do something about it./p pspan /br / span /br / span //span/span/span/p h2Simple Psychology of I Versus We - Disconnect/h2 pWhen your CTO was a programmer, the questions often asked are What should I do?, How can I solve this problem?. However as a Manager, this evolves to What should WE do, How do WE solve this problem. The consequences that affect the WE are much more complicated than those that affect the I. Moreover, not everyone who is a part of that WE will be equally affected; therefore, there will be different levels of benefactors based on the decisions that include WE, and these benefactors have opinions. /p pThe more senior a manager is, the more distant they may become from the data and information that is needed to validate an assumption. The WE produces many views that can distort the validation. Without a good process to validate assumptions, a senior manager risks moving forward on an invalid assumption. The CTO needs you! You need to step up and contribute to the conversation about load testing./p pIt's no secret that poor communication is probably the most significant factor contributing to a disconnect about the importance of system performance. If the assumptions of the decision maker are different than those affected by the decisions, then the conclusions will likely be different. If there is no communication about what those assumptions are, it is almost certain that a disconnect will occur. /p pLet's consider an example. The CEO has made growth commitments to the investors. In a big off-site executive planning weekend, the CEO beats up the head of sales and marketing to get projections of higher revenue. Marketing budgets are increased to get more leads and closed deals. Innovative and expensive advertising campaigns will drive lots of traffic to your website where cool interactivity will create better qualified prospects. The content management system underpinning all of your web marketing, CRM, and prospect database becomes critical to the CEO's success. However, the execs on this don't realize the CMS has never been load tested or tuned for high performance. The CTO isn't concerned because he spent a bunch of money on the CMS and has lots of good hardware in the data center to run it on. In the CTO's thinking is the fact that the CMS vendor provided cool performance charts in their sales brochures showing very high volumes of traffic supported. strongDisconnect!!/strong/p pspan /br / span /br / span //span/span/span/p h2Get Your CTO and Managers on Board/h2 pCommunication between your team and the senior managers should include the assumptions being made and how they were validated. Show them how a href=http://loadstorm.com/2009/web-performance-tuningweb performance tuning affects profit/a. Get the VP of Marketing involved. Explain how slower response times has been proven to lower revenue. Teach them the importance of load testing because it turns into money. /p pimg src=http://loadstorm.com/sites/loadstorm.com/files/money-falling.gif class=picture alt=load testing helps profit /Educate them. Gather up your a href=http://performance-testing.org/performance-testing-statisticsperformance testing statistics/a and present them to the CTO. Remove the disconnect./p pFor example, explain to the CTO that just because the performance was good in the vendor's environment doesn't mean the system will respond well in your environment. The hardware is different, the web server is configured differently, the database is not tuned, etc. We MUST load test the CMS ourselves. It's a safe bet that we won't get half the load that the vendor claims. Tuning and testing will take several cycles for us to make our system perform well. If the CTO has real experience with software and operational systems, this should make a lot of sense to her or him./p pGive them articles about a href=http://loadstorm.com/category/tags/failuresperformance failures/a on websites. Let them see the downside of ignoring load testing. They should understand from examples and studies that there is a clear correlation between performance tuning and company profits. The marketing folks will be motivated by campaign results so show them how traffic spikes can bring down a site, which loses customers forever. I bet they quickly see in their mind how the dollars are walking out the proverbial door./p pspan /br / span /br / span //span/span/span/p h2Load Testing and Profit/h2 pWhen executives see the connection between profit and load testing, they will start listening to you. Now THAT'S job security!/p pspan /br / span /br / span //span/span/span/p

p This post is about Multi-Mechanize, the web performance and load testing framework.br visit the project website: a href="http://multimechanize.com"multimechanize.com/a /p p Here are some instructions for getting started on rackspace cloud. These are also general instructions for anyone using a debian/ubuntu system. /p p There are _lots_ of options for running multi-mechanize in the cloud. You have choices between several cloud vendors and hosting/vps providers. Then you have a choice of operating system to deploy onto. I've found the combination of rackpsace cloud servers and ubuntu to be a really good choice. I see it as a great platform to run cloud-based load tests from (though hopefully they will offer other geographic regions at some point). It has been a breeze to work with multi-mechanize on this infrastructure. /p p So far i've been very impressed with rackspace. It lacks some of the functionality and management feature that EC2 gives you, but it is much easier to use. you pay by the hour + bandwidth used (starting at 1.5cents/hour). It's dead simple to deploy servers, and they have lots of linux operating systems to choose from. You can login to your account and provision the latest Ubuntu Server (9.10) in literally seconds. Need multiple servers? no prob. need _lots_ of servers? no prob, use the API's they provide. very slick stuff. /p p So... to get multi-mechanize working on rackspace's service... /p p create a rackspace account:br a href="http://www.rackspacecloud.com/cloud_hosting_products/servers"http://www.rackspacecloud.com/cloud_hosting_products/servers /a /p p login to your account:br a href="https://manage.rackspacecloud.com"https://manage.rackspacecloud.com/a /p p Deploy a new server from your account (choose the latest Ubuntu image). You can start with the minimum RAM allocation, and later resize your server to run large tests. Once you create your server, Rackspace will email you the server's ip address and root password. /p p Open a terminal window and ssh into your server using the info they supplied: /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;"gt;ssh lt;ip addressgt; -l root /pre p (enter password when prompted) /p p First, setup the dependencies for multi-mechanize.br run the following commands: /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;"gt;apt-get install -y python-mechanize gt;apt-get install -y python-matplotlib gt;apt-get install -y subversion /pre p Now you can grab the latest multi-mechanize trunk from subversion: /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;"gt;cd /opt gt;svn checkout http://multi-mechanize.googlecode.com/svn/trunk/ multi-mechanize /pre p Now go to your multi-mechanize directory and run a test: /p pre style="font-size:11px;border:1px #999999 dashed;font-family:monospace;color:#990000;background-color:#EEEEEE;padding-left:10px;"gt;cd /opt/multi-mechanize gt;python multi-mechanize.py default_project /pre p thats it... /pdiv class="blogger-post-footer"img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/5236867476487043111-5983545489832353473?l=coreygoldberg.blogspot.com' alt='' //div

pI found this interesting because it shows how even large companies running on their own safe data centers can experience massive performance failure. SaaS and cloud providers may get media attention for outages, but isn't it somewhat hypocritical of Fortune 5000 CTOs to claim that their internal systems are safe? Come on, let's be real. Systems have been experiencing poor performance for 60+ years, and hardware will continue to fail, and software will always have bugs, and architects will overlook weaknesses, and CEOs will annually cut budgets for performance engineering./p h2Twitter/h2 pIf you've used Twitter much in the past year, then you aren't surprised the web application has been failing. The Fail Whale is famous. Make that infamous. Twitter has suffered more actual downtime than usual through much of June. Some reports say that we should expect more outages through July./p pThe Fail Whale officially spoke to the media and blamed the downtime on the World Cup. Yeah...right. It's been apparent for a long time to most of us that Twitter is a victim of its own success. Too many users, too many tweets, too rapid growth. /p pOutages began on June 11 and continued with poor site performance for about 5 days. They are working on improvements, but Twitter has publicly admitted to internal network deficiencies. I hope they get their system tuned soon. It would be nice if they could stay ahead of the growth curve too, but I live in the real world and expect to see the Fail Whale periodically forever. Or at least until they get a real revenue model./p h2Intuit/h2 pIntuit is well-known to say the least. Their TurboTax Online runs hundreds of thousands of concurrent users around tax time. Did you know that they are still using the original C++ application with each user's own web-enabled copy instantiated on the server? Yep, I know. Crazy. /p pI guess I'm not shocked that Intuit had an overnight outage that prevented SMB customers from processing QuickBooks Online, Quicken, TurboTax Online, and QuickBase through the night of June 15. No credit card payments, no taxes calculated, no books reconciled for an estimated 300,000 small businesses. It was reportedly caused by a power failure that occurred during routine maintenance./p h2NetSuite/h2 pPopular and aggressive SaaS ERP provider NetSuite went down on April 26 for several hours. Its apps were inaccessible to customers worldwide. According to NetSuite, the cloud applications were down for under an hour, but reports say customers had performance problems for at least 6 hours. NetSuite blamed a network issue for the outage. They publicly stated that they had no idea how many customers were affected. Duh. This is a NYSE company. My guess is that they just didn't want to put the large number in print...no number does less credibility damage than hanging the truth out there. /p h2Microsoft Live/h2 pOk, I'm a skeptic about any Microsoft software reliability. Since 1983 I've been fighting the blue screen of death. There are MS haters, but I'm not really in that camp. I have been a MS Certified Professional and my company has been a MS Certified Solution Provider. That said, I love my Mac./p pSo when on January 28 Microsoft Online Services had poor performance and outage for such offerings as Microsoft Business Productivity Online Standard Suite (BPOS), I wasn't shocked. A MS post stated it was a network infrastructure problem, but it didn't say how long the poor performance lasted or how many customers were affected. You don't think that Windows was involved do you? Me neither./p pA couple of weeks later MS's Windows Live services were reported as down on February 16. Not shocked this time either. Keyword: Windows. What was also not surprising is that they blamed hardware. However, doesn't it seem a bit trite to have the world's largest software company describe the cause of global online services outage as due to a server failure? /p pAccording to Microsoft, there was an issue with the Windows Live ID service and log-ins failed for some customers, which increased the load on remaining servers. Things were back to normal after about an hour, Microsoft said./p pSlightly understated don't you think? I'm sure it had nothing to do with the instability of Windows./p h2The Planet/h2 pThe Planet is one of the world’s largest Web hosting companies and server hosting providers. It serves more than 20,000 businesses and more than 15.7 million Websites and has more than 48,500 servers under its management./p p“On May 2 at approximately 11:45 p.m. Central time, we experienced a network issue that affected connectivity within The Planet’s core network in Houston that may have prevented your servers from communicating to the Internet,” Jason Mathews, overnight technical support supervisor for The Planet wrote in one of the community forums. “Network connectivity service was fully restored in less than 85 minutes; however, your servers may have been impacted by this incident.”/p p“In our ongoing analysis of the occurrence, we determined that one of four border routers in Houston failed to properly maintain standard routing protocols,” Mathews wrote. Yep, you are down. Poor performance. Dang hardware./p pA second outage occurred on Monday morning. “We believe the network issues this morning are unrelated to connectivity problems customers in [Houston] may have experienced around 12 a.m. CDT. Around 9:30 a.m. CDT, The Planet noted that services had been fully restored and analysis found that a circuit between Dallas and Houston caused the Monday morning disruption. Yep, hardware again. /p h2Sage/h2 pPerformance problems hit Sage North America in June, which included a 22-hour outage on June 1-2 of its order entry-system, online CRM, and internal systems such as email. out of commission, according to a report on TheProgressiveAccountant.com. Sage reported the cause was in its storage area network. Not sure if that means hardware./p h2EMC/h2 pEMC launched Atmos Online in May 2009 as a scalable online storage infrastructure built on its Atmos technology. On Feb. 17, it went down. /p pUsers reported messages such as, EMC Atmos Online Temporarily Down For Maintenance and No suitable nodes are available to serve your request./p pEMC stated that the outage was caused by maintenance issues, but did not elaborate. The day before, EMC announced a bunch of cool updates. Guess there was a breakdown in the implementation of those cool updates. Call it maintenance if you want to, but I suspect the budget for testing was cut on this project. I'll even go so far as to say they probably didn't performance test at all./p

Sometimes problems in the Network can impact load test results. It is important to monitor the rate at which data is received from the server in a Network.br / br / br / LoadRunner can help you in determining the bandwidth used duringnbsp;a load test. Web Page Diagnostics Graphs in LoadRunner provides drill down analysis across different layers.br / br / Network bandwidth by a Component = Component Size/ Component Receive Timebr / br / Component Receive Time is the Time to transfer between the first byte to the last byte arrives from the server and Component Size is the size of component downloaded from the server.br / br / In the below Example Network speed during the test is strong263.249/1.132 = 232 KB/sec/strong. This speed may slight vary from component to component and also based on Load Generators used for the testbr / br / div class="separator" style="clear: both; text-align: center;"a href="http://4.bp.blogspot.com/_QGJjHMRUTpY/TD2mHDSEYFI/AAAAAAAAAjE/pvX0iz9gsAE/s1600/WebpageDrilldown.bmp" imageanchor="1" style="clear: left; cssfloat: left; float: left; margin-bottom: 1em; margin-right: 1em;"img border="0" rw="true" src="http://4.bp.blogspot.com/_QGJjHMRUTpY/TD2mHDSEYFI/AAAAAAAAAjE/pvX0iz9gsAE/s320/WebpageDrilldown.bmp" //a/divdiv class="blogger-post-footer"img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/3081868611305126410-575815807836239068?l=performancetestinginfo.blogspot.com' alt='' //div pa href="http://feedads.g.doubleclick.net/~a/gWWQIGdKImggZ9GmafyKnSDxA18/0/da"img src="http://feedads.g.doubleclick.net/~a/gWWQIGdKImggZ9GmafyKnSDxA18/0/di" border="0" ismap="true"/img/abr/ a href="http://feedads.g.doubleclick.net/~a/gWWQIGdKImggZ9GmafyKnSDxA18/1/da"img src="http://feedads.g.doubleclick.net/~a/gWWQIGdKImggZ9GmafyKnSDxA18/1/di" border="0" ismap="true"/img/a/pimg src="http://feeds.feedburner.com/~r/InfoOnPerformanceTesting/~4/P1tOXFjxn-E" height="1" width="1"/

As announced last week in our Get involved in the dynaTrace AJAX Edition 2.0 Beta Program blog we are going to do a Walkthrough Webinar on Wednesday, June 30th. The webinar will be recorded and made available on http://ajax.dynatrace.com. If you join live it gives you the chance to address questions that you have in [...]

Velocity was probably the best stage for this announcement and it was great that we got a slot on the Lightening Demo’s to show what’s new from dynaTrace. The dynaTrace Labs team has worked hard on the next version of the dynaTrace AJAX Edition. Since its inception, dynaTrace AJAX Edition got a lot of great and [...]

Velocity 2010, Web Performance & Optimization Conference takes place in Santa Clara, CA next week from Tuesday, 22nd till Thursday 24th of June. We are proud that we are not only going to be there with a booth but that both my colleague Alois and I are hosting the following sessions: Tuesday, 7:30pm: Best Practices with [...]

In this post I discuss the seven most important steps to improve your application performance practices. These simple-to-follow practices will help you to improve the way you deal with application performance. Besides eventually improving the performance of your applications it will help you to avoid playing the classical blame game which normally happens when something [...]

For a recent edition of the Swiss Computerworld Magazine we listed our Top 10 Performance Problems as we have seen them over the years when working with our clients. I hope this list is enlightening – and I’ve included follow-up links to the blogs to help better understand how to solve these problems: #1: Too Many [...]

pimg src=http://loadstorm.com/sites/loadstorm.com/files/duct-tape-baby.jpg classs=picture alt=load testing duct tape //p pI've seen too many people ignore load testing until it was too late. Some don't test at all. Some treat it with a duct tape approach: Test a little here, test a little there, no planning, no consideration for what is a real-world traffic scenario, and sloppily thrown together at the last minute before going live in production./p pThat's why I get so many calls from web developers in a panic. They probably have a CTO or EVP of Marketing looking over their shoulder the night before a big site launch. Too bad they didn't give enough priority for testing the performance of the system. /p pThe duct tape approach to load testing will make you look as irresponsible as the parents of this little girl. Plan for web performance testing and tuning early in your project. It's worth it because a href=http://loadstorm.com/2009/web-performance-tuningweb performance tuning/a correlates directly to revenue./p

pstrongA while ago/strong, we published a blog entry here about a href="../blog/browser-behaviour-and-performance" target="_blank"browser behaviour and performance/anbsp;where we told the story of how most browsers seem to transmit AJAX POST requests (i.e. HTTP POST requests originated from Javascript code) as two separate TCP packets, while a small number of browsers - most notably recent versions of Firefox - transmit the same request using only a single TCP packet./ppstrongThis behaviour/strong was discovered by a Yahoo developer originally, and later on investigated and confirmed by a href="http://josephscott.org/archives/2009/08/xmlhttprequest-xhr-uses-multiple-packets-for-http-post/" target="_blank"Joseph Scott/anbsp;on his blog. We went further and mapped a wide range of browsers on several different operating systems, to find out in what circumstances the request was sent as two packets. The results can be read in a a href="../../info/Analysis_of_browser_specific_characteristics.pdf" target="_blank"PDF report/a we created, if you're interested. (If you are not, then you should probably stop reading now, because for non-geeks, this post is only going to get worse from here)/ppstrongBecause/strongnbsp;we are such mega nerds when it comes to performance stuff, we decided to dig deeper into this, and find out just how big a performance impact the behaviour had. I.e. we wanted to see how much slower a request would be, on average, when the browser transmitted two instead of one TCP packet. Logically, more packets mean more overhead and a bigger risk of one packet getting lost and having to be resent, causing extra delay. Yahoo actually advice people against using POST in Ajax requests because of this behaviour - seenbsp;a href="http://developer.yahoo.com/performance/rules.html#ajax_get" target="_blank"Yahoo's ySlow performance rule #15 - Use GET for Ajax requests/anbsp;- but they don't tell you what it means for performance if you do/do not use POST. To us, just like to Yahoo apparently, it seemed rather obvious that sending two packets was worse than sending one packet, but maybe it shouldn't be a general performance recommendation unless it has a emsignificant/emnbsp;impact (if not, then other recommendations would perhaps have been more important to get across to ySlow users than this #15)./ppstrongSo/strong, we wanted to see how big the impact actually was.nbsp;By sending a large number of AJAX requests using both the GET and POST methods, we reasoned that we should be able to see some statistical difference between the response times of the two. Especially if we also simulated some network delay and packet loss on the link, the difference had to be quite noticeable. Our goal was to be able to say e.g. that emusing GET instead of POST in an Ajax request can reduce average transaction time by up to X% or more, depending on network conditions/em./ppstrongWe/strong set up the test, and spent several days running hundreds of thousands of Ajax GET and POST requests using different browsers, and emulating different network characteristics (i.e. network delay and packet loss). In the end we found something we weren't prepared for - the tests showed that emsending two packets seemed to slightly outperform sending one packet/em. (Again, read thenbsp;a href="../../info/Analysis_of_browser_specific_characteristics.pdf" target="_blank"PDF report/anbsp;if you want more details)nbsp;/ppstrongAfter/strong scratching our heads, asking around, double-checking the data, reading up on TCP and its most common implementations, then scratching our heads some more, we still couldn't understand how this came to be. There might have been problems with our setup somehow, but it seems far-fetched that any problems with our setup would give a 2-packet transaction an advantage over a 1-packet transaction. We decided to publish the results and see if anyone could explain what we were seeing, or tell us what we were doing wrong. We have since had a couple of people offer things such as em"two packets allow faster resend if the first packet is lost"/emnbsp;or "emsending more packets might help TCP compute the RTT better"/em. But none of the explanations seem to suggest anything that could outweigh the penalty you get from the fact that the 2-packet transaction suffers twice as much from packet loss as the 1-packet transaction. If it does - maybe that's the assumption we make that is wrong./ppstrongAt/strong the Velocity 2010 conference, I ran into a well known web performance guru and while talking to him a guy from the Yahoo ySlow team showed up, so I mentioned our findings to them both. The guru said he didn't believe me, and tried to change the subject. The Yahoo guy didn't say anything but walked off as soon as he got the chance. The obvious conclusion from this encounter is that I have to work on my people skills. Or maybe they were just stressed, or suffering from caffeine deficiency (it was a coffee break I think) - who knows? Maybe strongI/strong was suffering from caffeine deficiency. I hope I don't come off as one ofnbsp;those annoying people who can't stop trying to find faults in stuff other people do, in an attempt to feel superior. Because I am not - I just can't help trying to improve things everywhere, all the time. It's like a drug to me. I have to figure out how things work, and then try to improve them. Have to. I can probably be a pain about it - when people do something really well and expect praise I will usually tell them "Nice! The only thing you might want to change for next time is [whatever]..." and many take that as a sign that I don't like what they've done. For me, constructive criticism is the finest form of praise!/ppstrongAnyway/strong, enough of the boring introspection and back to the important stuff - investigating the problem! I decided that if people don't believe me, and they don't have time to set up a test bed and run the tests for themselves, the obvious solution is that I set up a test bed they can run tests on remotely. If I can get the critics to observe the strange behaviour first-hand, I might force them to acknowledge that it does happen, if nothing else! Then more people will be likely to try and come up with an explanation. I would be happy to learn that there is some problem with our setup, for instance, because then I would at least strongknow/strong. Not knowing is frustrating. How can I work to improve something unless I understand it first?/ppstrongSo/strong, I have now a test system up and running where anyone can try executing 50 Javascript GET, then 50 Javascript POST operations and see which are faster. I also set up the test on both Apache 2 and Lighttpd, in case this problem is webserver-dependent somehow. I thought, why not name it "Scott's problem", after the Joseph Scott who first bothered to describe it in detail. So the address to the test pages are:/pullia href="http://scotts-problem.loadimpact.com"http://scotts-problem.loadimpact.com/a (Apache 2)/lilia href="http://scotts-problem.loadimpact.com:81"http://scotts-problem.loadimpact.com:81/a (Lighttpd)/li/uldivstrongOn/strong the page you can see the tests other people have executed using different browsers, and their results. For each unique user agent you will get a summary with average transaction times. Feel free to run multiple tests if you want, you will probably not skew the data as the system only uses the average transaction time seen from a certain IP and a certain browser, when calculating average transaction times for that particular browser.nbsp;The machine is an Ubuntu server, in case anyone cares./divdivbr //divdivstrongAnd/strong yes, I know my web hacker skills are pretty awful. Don't look too closely at the javascript, or the HTML, please. You just might go blind./div

pimg src=http://i289.photobucket.com/albums/ll228/nursesnaturally/question-mark-thumb676874.jpg alt=load testing questions class=picture-left /In many situations, the correct answer is to ask questions. Software developers, testers, and project managers have a tendency to tell rather than ask because they have so much knowledge and expertise about the subject domain. However, before you begin load testing, you need to know as much as possible to help you create better scenarios and scripts./p pFor instance, you should understand how the system will be used. Ask plenty of questions of project stakeholders and write down the answers./p pspan /br / span //span/span/p ul liWhat response time would the CEO consider acceptable to her/him? /liliHas our development team built this app from scratch? /liliAre there any previous performance reports or datapoints such as benchmarks? /liliHas anyone set goals for performance? /liliWhat measurements do stakeholders want to see? /liliHow does the project manager define success for this site's performance? /liliCan I get a copy of any system documentation? /liliWill the site be mostly for anonymous browsing? /liliWhat potential is there for spikes in traffic? /liliHow often will the content change? /liliWill content such as blog posts be candidates for the Slashdot Effect? /liliIs it designed for internal company research? /liliDoes it primarily facilitate collaboration between trading partners? /liliAre there planned marketing campaigns to promote the site? /liliB2B or B2C? /liliDo you expect the company to promote the site through social media? /liliWill employees all sign-on at the same time each day? /liliAre there regular times of higher usage such as end of month? /liliCan irregular events such as webinars or advertisements drive waves of traffic? /liliHow many people does the marketing department anticipate coming to the site monthly? Daily? /liliDoes the user interface serve as a front-end to non-integrated back-end system such as SAP? /liliIs single sign-on or other identity access management system involved? /liliHow fast should a page load? /liliDoes the VP of Marketing have any idea of the correlation between site performance and revenue? /liliHow does the executive team define success for the average response times of this application? /liliAre any open source modules being used? /liliWhat is the primary technology stack? LAMP, MS, other? /liliHow many coding languages are we using? Which ones? /liliWhich database engine is deployed? Oracle, SQL Server, mySQL, other? /liliAre we hosting this in our own center? Co-located? Dedicated hosted servers? Cloud? /liliWhat are the logical layers in our architecture? /liliWhat are the physical layers in our architecture? /liliHow, if at all, are we configuring load balancing? /liliIs caching turned on in the application? Web server? Database or messaging? /liliDoes the web server use a compression technology? Is it turned on? /liliWill we be testing against the production environment? /liliIf we have a staging environment for testing, what is different from production? /liliHow often do we anticipate releasing new versions into production? /liliDo we want to measure performance of each release to identify any degradation? /liliWho should be on the email list to receive load test reports? /liliWho is John Galt? /li/ul pThat's a decent list to start. Please submit your own via the comments. /p pThe idea is to get a broad context of which the system will operate. Ask! Listen! Take notes!/p pTry it, I promise it will help you create better pland and scenarios for your load tests. Your stakeholders will be happier as well./p

I’ve got great news to share with you – BrowserMob has been acquired and is joining the Neustar family of services, which includes UltraDNS and Webmetrics!

I started BrowserMob in 2008 with the belief that a combination of cloud computing and real browsers wrapped up in a self-service model could dramatically change how people used load testing. Neustar shares that same philosophy and has built a world-class portfolio of cloud-based performance services that currently has over 3,000 customers. I’m most excited by this partnership because it means that we can now offer our unique services to an even greater audience.

A few months ago Neustar approached us and it was immediately obvious that our visions for website testing, development, and operations were very similar. For example, just like BrowserMob, the team at Webmetrics utilizes real browsers and supports open source and cloud computing initiatives. Even better, upon closer look I found that Webmetrics Load Testing and Monitoring solutions actually complement our own services much more than they compete. Joining forces made perfect sense!

While in the short term nothing will be changing, the long term benefits of this partnership will be an even better experience for all our users. For example, Webmetrics Load Testing provides high quality, hands-on professional services that BrowserMob customers will be able to utilize if they need extra assistance. And for monitoring, BrowserMob customers will be able to benefit from the massive infrastructure and capabilities that Webmetrics already has in place including a global infrastructure with agents in over 100 major cities. Similarly, we expect to bring to the table many innovations of our own over the coming months and years.

Of course our highest priority is you – our customer. I want to assure you that the quality of service you’ve been receiving from us will not change, and that 100% of our team will remain intact and dedicated to making you successful. In fact, effective today both Ian White and I are officially Neustar employees and we are committed to enhancing our solutions and your satisfaction!

Thank you for showing your continued support to BrowserMob. I’m excited to continue working with you and updating you on our plans for a fully integrated BrowserMob + Neustar offering. If you have any questions, you’re of course always welcome to contact me directly.

Patrick
+1 (503) 828-9003 x 101
patrick.lightbody@neustar.biz

Tweet This Post 

I’ve got great news to share with you – BrowserMob has been acquired and is joining the Neustar family of services, which includes UltraDNS and Webmetrics! I started BrowserMob in 2008 with the belief that a combination of cloud computing and real browsers wrapped up in a self-service model could dramatically change how people used load [...]

p So you need to monitor a Linux system for performance metrics... CPU, Memory, Network, Disk, etc. /p p Here are 6 of my favorite command line tools for monitoring a Linux system from the command line. /p hr / stronghtop/strongbr / a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_k7-jvtv2cLo/TCy9tDdKYrI/AAAAAAAAArQ/ZYNCUDUwb8A/s1600/screenshot_htop.png"img style="cursor:pointer; cursor:hand;width: 400px; height: 266px;" src="http://1.bp.blogspot.com/_k7-jvtv2cLo/TCy9tDdKYrI/AAAAAAAAArQ/ZYNCUDUwb8A/s400/screenshot_htop.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5488970627577176754" //a br / a href="http://htop.sourceforge.net/"http://htop.sourceforge.net//a hr / strongdstat/strongbr / a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_k7-jvtv2cLo/TCy9sgkjlGI/AAAAAAAAArI/iUDVIxjowz8/s1600/screenshot_dstat.png"img style="cursor:pointer; cursor:hand;width: 400px; height: 273px;" src="http://4.bp.blogspot.com/_k7-jvtv2cLo/TCy9sgkjlGI/AAAAAAAAArI/iUDVIxjowz8/s400/screenshot_dstat.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5488970618212947042" //a br / a href="http://dag.wieers.com/home-made/dstat/"http://dag.wieers.com/home-made/dstat//a hr / strongbmon/strongbr / a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_k7-jvtv2cLo/TCy9sPm4kHI/AAAAAAAAArA/YKmhbVGV1BU/s1600/screenshot_bmon.png"img style="cursor:pointer; cursor:hand;width: 400px; height: 359px;" src="http://3.bp.blogspot.com/_k7-jvtv2cLo/TCy9sPm4kHI/AAAAAAAAArA/YKmhbVGV1BU/s400/screenshot_bmon.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5488970613659308146" //a br / a href="http://freshmeat.net/projects/bmon/"http://freshmeat.net/projects/bmon//a hr / strongiftop/strongbr / a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_k7-jvtv2cLo/TCy9t47CtqI/AAAAAAAAArg/AL8khQaDGok/s1600/screenshot_iftop.png"img style="cursor:pointer; cursor:hand;width: 400px; height: 266px;" src="http://4.bp.blogspot.com/_k7-jvtv2cLo/TCy9t47CtqI/AAAAAAAAArg/AL8khQaDGok/s400/screenshot_iftop.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5488970641929582242" //a br / a href="http://www.ex-parrot.com/pdw/iftop/"http://www.ex-parrot.com/pdw/iftop//a hr / strongifstat/strongbr / a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_k7-jvtv2cLo/TCy9tpw1eBI/AAAAAAAAArY/0JWtGr6_E2E/s1600/screenshot_ifstat.png"img style="cursor:pointer; cursor:hand;width: 400px; height: 266px;" src="http://2.bp.blogspot.com/_k7-jvtv2cLo/TCy9tpw1eBI/AAAAAAAAArY/0JWtGr6_E2E/s400/screenshot_ifstat.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5488970637860239378" //a br / a href="http://gael.roualland.free.fr/ifstat/"http://gael.roualland.free.fr/ifstat//a hr / strongsysstat/strongbr / this is a package of utilites including strongiostat/strong, strongmpstat/strong, strongsar/strong, and others.br / a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_k7-jvtv2cLo/TCy95xUUBgI/AAAAAAAAAro/ZHzzRKH5dys/s1600/screenshot_iostat.png"img style="cursor:pointer; cursor:hand;width: 400px; height: 266px;" src="http://2.bp.blogspot.com/_k7-jvtv2cLo/TCy95xUUBgI/AAAAAAAAAro/ZHzzRKH5dys/s400/screenshot_iostat.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5488970846046520834" //a br / a href="http://pagesperso-orange.fr/sebastien.godard/"http://pagesperso-orange.fr/sebastien.godard//a hr / p These tools are all available from package managers (apt-get, yum, etc) on most Linux systems. They are also available on most other *nix platforms. /pdiv class="blogger-post-footer"img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/5236867476487043111-1588815434091547815?l=coreygoldberg.blogspot.com' alt='' //div

pimg src=http://loadstorm.com/sites/loadstorm.com/files/220px-Gary_Busey_2007.jpg class=picture alt=Gary Busey is a stress test height=200 /I've have always liked a href=http://en.wikipedia.org/wiki/Gary_Busey target=_blankGary Busey/a in movies and on TV. He has starred in everything from Kung Fu to The Buddy Holly Story to the Simpsons to Gunsmoke to Saturday Night Live to Lethal Weapon. His recent appearance on Celebrity Rehab with Dr. Drew showed just how much brain damage he suffered in that motorcycle crash. In some ways, Gary's life is similar to a good stress test: Ramped up the volume until the system could not respond appropriately to requests made of it./p pOn to the good stuff about system performance testing./p h2Performance Metrics Tied to Business Metrics/h2 pMike Kelly at SoftwareSearchQuality.com has an article about developing a working understanding of the business drivers for application performance. He postulates that a skilled performance tester can build better tests through understanding performance goals better through knowing what the business truly needs. He also believes this understanding makes test results more actionable. The key question to ask is, What do the project stakeholders care about? a href=http://searchsoftwarequality.techtarget.com/tip/0,289483,sid92_gci1508137_mem1,00.html?track=sy280 target=_blankRead more aboutbr / application lifecycle performance testing and monitoring strategies./a/p h2Agile Performance Testing/h2 pMaybe the best point in Mike's post is:br / The best time to be thinking about possible performance issues is when you're actively designing and writing code. You're still close to the code. You know what it does. You know how it works. For each day away from the code, the fuzzier it gets in your head. The more developers committing to the codebase, the less you really know about the code you're interacting with. /p pSound like Agile? Yep, test early and test often. I've asked questions at conferences of Agile gurus about performance testing early in the development cycle. All of them acknowledge that frequent perf tests should be run from the beginning of coding. Every build should automatically run a load test. However when you press them to see how often their projects actually follow this model, they eventually admit that the practice doesn't meet the theory. Rarely, if ever, do teams create perf scripts and execute them until the last 20% of of the project. It's easy to stand up behind a podium and talk to a PowerPoint slide about what you SHOULD do, but it is not reality that teams put high priority on performance until all the pieces of code are pulled together./p h2Unit Versus Comprehensive System Tests/h2 pA common misconception is that there is one form of performance testing - hit the whole system with volume and see how it responds. That's one type, but in many cases it is much more effective to test a component on its own. For example, let's say you are coding a module that inserts data into the database. You should examine the efficiency of the way you coded the process before you test the performance of all the moving pieces that call your module. It will be harder to detect the root cause of a bad SQL statement later when it is hidden in the big pile of other database hits./p pMike makes an argument about using tools that will let you test the speed of units of code. Again, sounds good. I support that idea and recommend it to every developer. You may be able to find some expensive queries or poorly written loop structures. That will be a tremendous help when you are testing the speed and responsiveness of the system to a user where the UI layer is communicating with the application layer which is communicating with the database layer. Identifying resource hogs and slow functions is possible early, and it appears obvious that fixing those in unit testing would be best. Tune the individual parts before assembling them. It has a much higher probability of producing a well performing system./p h2Realistic Test Scenarios for User Types/h2 pMike talks about performance profiles, and I tend to think about the different ways people will use the system. In a typical web app, you might have 70% of your traffic from anonymous lookers viewing your product catalog or reading articles. Maybe 15% of your traffic is going through the shopping experience and buying something (if you are lucky!). Perhaps you have 10% of your traffic downloading white papers or filling out web forms to register for your newsletters. You probably have a few of your internal employees signed in to modify content or update the product catalog. All of those are hypothetical of course, but they represent realistic user patterns. Your test plan should reflect these. /p pIt is also true that during Christmas the percentage of credit card processing could go up by 100% over normal. Your application may have spikes in certain kinds of traffic at end of year. I spoke with PerfDan at the STP conference last year over lunch. He said the TurboTax Online app could see a million users per day during the second week of April. Hmmm, predictable. My suspicion is that most of them are in a panic too!/p pOne of our clients recently released a new book. Realizing that they needed to load test for the anticipated volume, they started building tests with LoadStorm. What they didn't consider was that most of the traffic would NOT come through their home page to make the purchase. They were sending a large email blast with a link to a specific landing page, and their test wasn't accounting for that. Thus, their original scenario was essentially useless for predicting site performance. Their home page was horribly slow with hundreds of images and dozens of unnecessary Javascript external libraries. The system stressed out at low volume because the test virtual users bottlenecked at the home page. In reality, they were fine because the landing page was simple and loaded quickly without a bunch of crap. We were happy to coach them to improve the home page, but the real point is that traffic patterns are situational, seasonal, and specific to user types./p pMike makes a good suggestion about finding potential performance problems before they bite you in production. He recommends usage modeling, and I agree. More importantly, you should create multiple test plans to reflect each of your usage models. Test with different scenarios together to reflect what could happen under certain circumstances that are likely to happen - even if your actual server logs don't show that happening right now.br / Usage modeling isn't always just about trying to test different worse-case scenarios either. It's also possibly looking at different ways to utilize excess capacity in the off season or testing to see if dynamic provisioning will support certain growth scenarios. You might also try different usage models just so you can build a taxonomy of different load profiles so later on, when you're monitoring production, you can go back and correlate what you're seeing in production to different load patterns during your testing phase. Sometimes that can provide insight into potential problems. 'Oh yea, I remember seeing that before. Last time we saw that we were….' /p pimg src=http://loadstorm.com/sites/loadstorm.com/files/puzzle-pieces-together.jpg class=picture-left alt=pieces fit together /br / h2Load Testing - Putting the Pieces Together/h2 /ppOne of my favorite hobbies is to trace a web transaction all the way through a system. It isn't always easy. Sometimes I get stuck and need to bring in smarter developers than me - I'm the Omega Geek. It's a challenge that I recommend for you. WARNING: it is tough to do. Worth it? Yes. Easy? No./p pIt takes a lot of time and energy investment to walk through the depths of your application code, database queries, stored procedures, UI code, web services, etc. But I tell you that if you ever do this, you will become so enlightened about how all your system components are interacting. You will know what piece of code is calling the next, which methods are heavily used, which queries are unique to a particular button click, and which third party library is referenced yet unused. /p pIt will help you be the best load tester on your team because you will have the best insight into where the bottlenecks can form. You will have a perception into the inner workings that is an outstanding jump start when you begin trying to tune the system performance./p pIf you work with your whole team (developers, architect, product manager, operations, QA), then everyone learns faster. For example, we have an architect that understands the implications of tweaking settings in a message queuing system that is squarely in my area of ignorance. He has shown me ways to employ the messaging technology to get 1000% throughput increase. I wouldn't have figured that out so quickly (or never) if I had not involved him./p pI like Mike's comment about the team concept of breaking down the system into one transaction flow:br / It's almost like practicing a fire drill for a production issue. Teams that can do this analysis well can respond to production issues well. It's almost like a capstone project – tying together all your models, metrics, and logging into one neat package./p pHere are some good monitoring ideas he presents too:br / Look at the profile of the servers while they are processing that transaction. Understand what the impact might be to the end user. Try to become the world's foremost expert on what's going on in the system, for that thin sliver of time, for that small transaction./p h2Some Things Haven't Changed/h2 pSure, web applications are much more complex in 2010 than they were in 1998 when my company started building them. But in some ways, this whole performance testing issue is just like Dr. Hood taught us back in college. He made us follow a single processing thread through the VMS operating system to understand the impact on the VAX and on our application. In many ways it is easier today because the tools for monitoring and testing are better. In other ways it is more challenging because there are so many moving parts to explore./p pThe bottom line in my not-so-humble opinion is that the best performance testing involves a lot of elbow grease. You must dig in and dig in deeply. It is not a trivial exercise, and you will hit roadblocks along the way that seem insurmountable. However, as long as you set your goal properly to fully understand the inner workings of your application, and if you have the tenacity to push through the complexity of interactions of components, then you will become a world-class performance engineer./p pMy last bit of advice: Don't wait to start. Get going on a deep dive now without reading, researching, and studying all the documentation. You can find the answers once you are clear on the questions posed by your system. Dig in and see what wall you hit first. Then start asking people for help. Your colleagues, product vendors, and online forums are great places to help you over that first challenge and on to the next one. Soon you will be amazed at how thoroughly you understand the performance aspects of your application and its environment./p

I just got back from the US after presenting at HP Software Universe 2010. I will save the content of my presentation for another time, so this is a quick overview of what I saw at the Conference and some of the things I learned from the people who attended. Conference Overview HP really know [...]

pimg src=http://loadstorm.com/sites/loadstorm.com/files/the-links.jpg alt=performance testing links class=picture-left /As we begin another week of summer, here are a few helpful links for articles concerning issues of interest to us load and performance testers. We at LoadStorm have been busy the past month with many new customers asking us for help. /p pWe like people, and we like web performance testing. And we really like people that like web performance testing! Our focus is on providing the best value load testing tool. That said, we often get requests from clients that want us to assist them with understanding the best way to go about it. Some want us to coach them as they figure out what to do about poorly performing web applications. In any case, we certainly enjoy working with customers to make their sites run better./p pWhile we know that we aren't a fit for everyone, it has become clear over the past 18 months that we are helping a bunch of people. Thank you to each client that has sent us a testimonial or even referred us to your geek friends. We sincerely appreciate the word of mouth support we have been getting. /p pWhen we say things like, Please let us know if we can help in any way, we mean it. Seriously. So don't hestitate to tell us if you have questions or need some coaching around performance testing. /p pIn the meantime, we offer these links to other writers and vendors that may be useful to you as you tune amp; test your web applications./p pspan /br / span /br / span /br / img src=http://loadstorm.com/sites/loadstorm.com/files/USA.jpg class=picture alt=performance World Cup //span/span/span/p h2Performance of the World Cup Site/h2 pa href=http://blog.dynatrace.com/2010/06/04/hands-on-guide-verifying-fifa-world-cup-web-site-against-performance-best-practices/ target=_blankVerifying FIFA World Cup Web Site against Performance Best Practices/a/p pAndreas Grabner predicts that the World Cup site will fail. The site has poor performance to start with, and Andreas applies some engineering best practices from which all of us can learn. /p blockquotepMy analysis of the FIFA site shows that – once the World Cup starts next week and the site gets really hit by millions of users around the globe – there is a big chance the site will run into performance and scalability issues due to several best practices that my analysis shows the site does not follow. This failure causes load times of the initial page take more than 8 seconds and requires downloads of more than 200 elements. These problems can easily be fixed by following the recommendations I highlight in this blog./p/blockquote pI especially appreciate how Andreas breaks down the Key Performance Indicators (KPI) into groups, then he shows where each metric is a problem for the FIFA site. He makes great suggestions for where to look for improvement./p pHe gives the World Cup site a failing grade for overall performance. In looking at the 4 categories he uses for measurement, none got a good grade./p ul liBrowser Caching: font color=redF/font – 175 images have a short expires header, 4 have a header in the past /liliNetwork: font color=redF/font – 201 Requests in total, 1 Redirect, 1 HTTP 400, duplicated image requests on different domains /liliServer-Side: font color=redC/font - 10 App-Server Requests with a total of 3.6s - analyze server-side processing /liliJavaScript: font color=redD/font - Use CSS Lookups by ID instead of Class Name pspan /br / span /br / span /br / img src=http://loadstorm.com/sites/loadstorm.com/files/up-arrow.jpg class=picture alt=performance moving up //span/span/span/p h2Cloud Does NOT Mean Infinite Scalability/h2 pa href=http://itexpertvoice.com/ad/scale-and-scalability-rethinking-the-most-overused-it-system-selling-point-for-the-cloud-era/ target=_blankScale and Scalability: Rethinking the Most Overused IT System Selling Point for the Cloud Era/abr / Scott Fulton does a nice job dispelling some myths about cloud computing. Application performance is vital to companies today because of the dollars tied to websites, back office systems, and many forms of business automation. New start-ups utilizing cloud infrastructures are quickly coming out of the gate with disruptive technologies that process huge volumes of data for an ever-increasing user base./p pWe geeks have been saying for years, you can't just throw hardware at the problem. Scott says:/p blockquotepIf you believe that a scalable architecture for an information system, by definition, gives you more output in proportion to the resources you throw at it, then you may be thinking a cloud-based deployment could give your existing system “infinite scalability.” Companies that are trying out that theory for the first time are discovering not just that the theory is flawed, but that their systems are flawed… and now they’re calling out for help./p/blockquote pBusiness people are waking up to the fact that scalability is as important to their success as capitalism and democracy. They usually don't know how important scalability is until their system fail. After customers and revenue are lost, then they look into the underlying causes and find their systems don't really have the performance capability they thought. So in typical manager decision making, they use money to solve the problem by re-deploying on a cloud infrastructure to get much more horsepower. Unfortunately, it is NOT that simple. Cloud platforms are not magic. There are thousands of variables in the scalability equation, and the cloud with massive virtualization is only one factor to improve performance./p pSometimes the solution is to re-architect the system. In fact, Twitter has re-architected their system 4 times already. An academic may tell us to plan for growth, think through all the possibilities and produce the most scalable system from the beginning. Yeah, good idea. But it just isn't that easy because there are too many moving parts. Many of the moving parts are on moving platforms that are getting better, faster, cheaper by the month. Scott tells us to do the best we can with what we have now, and then always be looking ahead for ways to improve performance - including throwing away what you have to build a more scalable application architecture./p pBradford Stephens calls today a transitional period where no one truly has scalability figured out. There is no single right answer for evaluating a scalable platform. But he suggests that virtualization and the cloud make it feasible/affordable to rescale by powers of ten rather than multiples of two. That's exciting to me./p pScott says this:/p blockquotepThis is uncharted territory. What we do know is that businesses can no longer afford to develop solutions around the edges of the core problem. They can’t just point to the symbols for their systems’ various ills (latency, load imbalance) and invest in the symbols that represent their solutions (scalability, the cloud, CMS platforms, social network platforms) as tools for postponing the inevitable rethinking of their business models. Throwing your business model at the cloud doesn’t make the symptoms go away; indeed, it magnifies them. Symbols aren’t solutions./p/blockquote pA quote by Sean Leach in this article matches the philosophy of our CTO, Roger Campbell. Namely, get the app into the customers hands quickly and if you need to fix a scale issue, then that's a good problem to worry about when it arrives. Over 90% of web apps never get much traffic anyway, so why waste time until you know it will be an issue? Wasting months upfront analyzing and planning the best scalable architecture could very well translate to you launching nothing...ever./p pa href= target=_blankx/a/p blockquote/blockquote pa href= target=_blankx/a/p blockquote/blockquote pa href= target=_blankx/a/p blockquote/blockquote pa href= target=_blankx/a/p blockquote/blockquote pa href= target=_blankx/a/p blockquote/blockquote pa href= target=_blankx/a/p /li/ul

Velocity 2010 came to an end today. I attended all 3 days – it was a great conference. I did not attend last year, but the crowds this year must have been at least 3 times that of 2008, when I first presented at Velocity. Here are some of my thoughts on the conference.

Performance

Being a performance person, I am naturally biased towards performance topics. So, I’ll cover this first. All of the performance sessions at the conference can be summed up thus :

The biggest performance issue is the time it takes for the browser to load a web page (aka page load times). Here is technique x and trick y and hack z to help you fix this problem.

I learned a lot about how to optimize css, javascript, http headers etc. But I was still disappointed that there was hardly a whisper about how to optimize the server side. The claim is that of the total response time, the server takes tens or at most 100′s of milliseconds where as the client takes several seconds.  So where do you want to focus your energy on ? I can accept that. But that seems to pre-suppose that all web applications have a scalable architecture and have solved their server-side performance and scalability issues. I find that a little hard to believe.

Operations

As expected, the details of how Facebook, Yahoo and twitter run their operations was of great interest to the audience. With so much media now being served, I was surprised to see only one session on optimizing Video serving and even that was not well attended. There was hardly any talk about optimizing infrastructure. I can’t help wondering why web operations wouldn’t be interested in optimizing their infrastructure costs. After all, we’ve been hearing a lot lately about the cost of power, how data centers are going green, more efficient etc. Aren’t these things applicable to the web world as well (not just enterprise IT) ? Even more surprising, a very small portion of the audience said they were deployed on the cloud.

Neil Gunther and I presented a session  on Hidden Scalability Gotchas in Memcached and Friends.

We had a great audience with some folks squatting on the floor in the front and a standing-room only audience in the back. There was tremendous interest in applying the USL Model to accurate data to quantify scalability. If anyone has additional feedback or comments, I would love to hear them.

Tools

I was blown away by the plethora of tools, a good many of which I had never heard of. Firebug with various add-ons (YSlow, PageSpeed) set the trend on browser-side monitoring and now even commercial vendors have versions of their product (available for free !) to do the same. This is great news for developers. If you haven’t heard of HttpWatch, showslow.com, webpagetest, check them out.DynaTrace announced a free end user response time monitoring tool as well.

Products

One real cool product I came across was Strangeloop – this is an appliance that sits in front of your web server and optimizes the response page. It’s amazing that it can do so much javascript optimization resulting in dramatic reduction in latency. I can’t help wondering why browsers don’t do this ? Surely, Mozilla and Google have enough smart engineers to come up with a an optimized javascript interpreter. It will be interesting to watch.

The usual monitoring vendors were all there – Keynote, Gomez (now part of Compuware), Webmetrics, AppDynamics etc.

Misc.

Tuesday was billed as “Workshop” day. However, there really weren’t any workshops – they were all just regular sessions just longer. I guess it’s hard to do workshops with several hundred people in the room. If Velocity really wants to do workshops, they need to have at least a dozen of them scheduled and they need to be longer.

On the whole, the conference was a great success, with sold out crowds, well attended and delivered sessions and lots of new products. Hope I can make it to Velocity 2011.