Feed aggregator

New findings: The median top 100 ecommerce page takes 6.2 seconds to render primary content

Web Performance Today - 2 hours 27 min ago

Every quarter at Radware, we measure and analyze the performance of the top 500 retail websites.* And every quarter, I’ve grown accustomed to the persistence of two trends: pages are growing bigger and, not coincidentally, slower.

But while I expected to see some growth and slowdown in our latest research — released this week in our State of the Union: Ecommerce Page Speed & Web Performance [Summer 2014] — I have to admit that I wasn’t expecting to see this much:

  • The median top 100 ecommerce page now takes 6.2 seconds to render primary content (AKA Time to Interact or TTI). In our Summer 2013 report, the median TTI was 4.9 seconds. In other words, TTI has slowed down by 27% in just one year.
  • The median page has grown by 67% in one year — from 1007 KB in June 2013 to 1677 KB in June 2014.

Our other findings included:

  • Only 14% of the top 100 pages we tested were able to meet user expectations by rendering primary content in 3 seconds or less.
  • 17% of the pages we tested had a TTI of 10 seconds or longer.
  • 43% of sites failed to implement image compression.
  • 66% did not use progressive JPEGs.
Why the dramatic slowdown?

The long answer to this question is as complex as modern web pages themselves. But the short answer is simple: web pages have never been as large and complex as they are today.

The performance challenges that plague modern web pages have been born out of all the great things we can now make pages do: dynamic content, high-resolution images, carousels, custom fonts, responsive design, and third-party scripts that gather sophisticated data about visitors.

But all of this amazing functionality comes with a high performance price tag if it’s not implemented with performance in mind.

Take images, for example. We’re so accustomed to expecting to see high-quality images everywhere on the web that we take them for granted and don’t think about their heavy performance impact. Page size has a direct impact on page speed, and images comprise at least half a typical page’s weight. As such, they represent an extremely fertile ground for optimization.

Yet we found that many leading retailers are not taking advantage of techniques such as image compression and progressive image rendering, which can improve both real and perceived load time. More on this later in this post.

Page complexity and how it affects Time to Interact

To better understand the impact of page size and complexity, we can look to two additional metrics for more insight:

  • Time to First Byte (TTFB) – This is the window of time between when the browser asks the server for content and when it starts to get the first bit back.
  • Start Render Time – This metric indicates when content begins to display in the user’s browser. Start Render Time can be delayed by slow TTFB, as well as a number of other factors, which we’ll discuss after we’ve taken a moment to enjoy this nifty graph:

Slow Time to First Byte is usually a sign that the site has server issues or a latency problem. These problems can be addressed updating your server environment and by using a content delivery network (CDN) to cache page resources closer to end users. Most (78%) of the sites we tested currently use a CDN, and this number has not changed appreciably in recent years. And while our research doesn’t give us visibility into each site’s server environment, it’s safe to assume that the sites we studied are already on top of that. Therefore it’s not a surprise that TTFB has not changed significantly. Site owners are already doing about as much as they can to address TTFB. Throwing more servers at the problem won’t do anything.

But while TTFB has plateaued, Start Render Time has suffered – almost doubling in the past year, from 2.1 seconds in Summer 2013 to 4 seconds today. Increased page complexity is the likely culprit. As we’ve alreasy covered in this post, modern web pages are more complex than ever – in the form of poorly placed style sheets,badly executed JavaScript, and third-party scripts (such as tracking beacons and page analytics) that block the rest of the page from rendering – and this complexity can incur a hefty performance penalty.

Images: The low-hanging fruit of front-end optimization

At least half of a typical page’s weight is comprised of images, yet most of the pages we tested failed to properly implement image optimization techniques that could make their pages significantly faster.

Almost half (43%) of the pages we tested failed to implement image compression, with 8% scoring an A. Image compression is a performance technique that minimizes the size (in bytes) of a graphics file without degrading the quality of the image to an unacceptable level. Reducing an image’s file size has two benefits:

  • reducing the amount of time required for images to be sent over the internet or downloaded, and
  • increasing the number of images that can be stored in the browser cache, thereby improving page render time on repeat visits to the same page.

In last week’s post, I wrote about how progressive image rendering can improve both real and perceived performance. Yet despite the promise of progressive images, we found that two out of three of the pages we tested didn’t use progressive JPEGs, and only 5% scored an A.

These are just two image optimization techniques among many others, such as spriting, lazy loading (when you defer “below the fold” images to load after “above the fold” content), and auto-preloading (which predicts which pages users are likely to visit next based on analyzing overall traffic patterns on your site, and then “preloads” page resources in the visitor’s browser cache). As I mentioned during a talk I gave yesterday at a Shop.org event, unless you’re already implementing all of these best practices, these techniques represent a huge untapped opportunity to optimize your pages.


Even for leading retailers, tackling performance is like trying to take aim at a constantly moving target. As soon as you’ve gotten a bead on one performance issue, a brand-new challenge pops up. Images grow ever richer, custom fonts proliferate, new third-party scripts are introduced, and stylesheets perform increasingly complex tasks. The silver lining here is that the impact of all this complexity can be mitigated with a thoughtful optimization strategy and a commitment to evolving this strategy to continue to meet future demands.

Get the report: State of the Union: Ecommerce Page Speed & Web Performance [Summer 2014]

*Using WebPagetest, we test the home pages of the top 500 retail sites as they would perform in Chrome over a DSL connection. Each URL is tested nine times, and the median result is used in our analysis.

The post New findings: The median top 100 ecommerce page takes 6.2 seconds to render primary content appeared first on Web Performance Today.

The Deadline to Sign up for GTAC 2014 is Jul 28

Google Testing Blog - Tue, 07/22/2014 - 18:53
Posted by Anthony Vallone on behalf of the GTAC Committee

The deadline to sign up for GTAC 2014 is next Monday, July 28th, 2014. There is a great deal of interest to both attend and speak, and we’ve received many outstanding proposals. However, it’s not too late to add yours for consideration. If you would like to speak or attend, be sure to complete the form by Monday.

We will be making regular updates to our site over the next several weeks, and you can find conference details there:

For those that have already signed up to attend or speak, we will contact you directly in mid August.

Categories: Software Testing

How to Approach Application Failures in Production

In my recent article, “Software Quality Metrics for your Continuous Delivery Pipeline – Part III – Logging”, I wrote about the good parts and the not-so-good parts of logging and concluded that logging usually fails to deliver what it is so often mistakenly used for: as a mechanism for analyzing application failures in production. In […]

The post How to Approach Application Failures in Production appeared first on Compuware APM Blog.

Categories: Load & Perf Testing

When to Leverage Commercial Load Testing Services, and When to Go it Alone

LoadImpact - Tue, 07/22/2014 - 03:00

How and where you execute load and performance testing is a decision that depends on a number of factors in your organization and even within the application development team.

It is not a clear cut decision that can be made based on the type of application or the number of users, but must be made in light of organizational preferences, cadence of development, timeline and of course the nature of the application itself and technical expertise currently on staff.

In this post we will provide some context around some of the key decision points that companies of all size should consider when putting together load NS performance testing plans.

This discussion is really an amalgamation of On-Premise versus SaaS/Open-Source versus Commercial Services.

In the load testing space there are commercial offerings that offer both SaaS and on-premise solutions as well as many SaaS only solutions for generating user load.

From an open source perspective, JMeter is the obvious choice (there are other less popular options such as FunkLoad, Gatling, Grinder, SOAPUI, etc). Having said that, let’s look at the advantages and challenges of the open source solution,  JMeter, and contrast that with a cloud-based commercial offering.

Key JMeter Advantages:

  1. 100% Java application so it can be run on any platform (windows, osx, linux) that can run Java.
  2. Ability to test a variety of types of servers – not just front end HTTP servers.  LDAP, JMS, JDBC, SOAP, FTP are some of the more popular services that JMeter can load test out of the box.
  3. Extensible, plug-in architecture. The open source community is very active in development around JMeter plugins and many additional capabilities exist to extend reporting, graphing, server resource monitoring and other feature sets.  Users can write their own plugins if desired as well.  Depending on how much time and effort is spent there is little that JMeter can’t be made to do.
  4. Other than the time to learn the platform there is no software cost of course since it is open source.  This may be of particular value to development teams with limited budget or who have management teams who prefer to spend on in-house expertise versus commercial tools.
  5. It can be easy to point the testing platform at a development server and not have to engage the network or server team to provide external access for test traffic.  It’s worth noting that while this is easier it is also less realistic in terms of real world results.

Key JMeter Disadvantages:

  1. Being that it is open source you do not have an industry vendor to rely upon for support, development or expertise.  This doesn’t mean that JMeter isn’t developed well or that the community isn’t robust – quite the opposite. Depending on the scope of the project and visibility of the application it can be very helpful to have industry expertise available and obligated to assist.  Putting myself in a project manager’s shoes, would I be comfortable telling upper management, “we thoroughly tested the application with an open source tool with assistance from forums and mailing lists?” if there were to be a major scale issue discovered in production?
  2. It’s very easy to end up with test results that aren’t valid.  The results may be highly reliable – but reliably measuring bottlenecks that have nothing to do with the application infrastructure isn’t terribly useful.  Since JMeter can be run right from a desktop workstation, you can quickly run into network and CPU bottlenecks from the testing platform itself – ultimately giving you unrealistic results.
  3. Large scale tests – not in the wheelhouse of JMeter.  Right in the documentation (section 16.2 of best practices) is a warning about limiting numbers of threads.  If a truly large scale test is required you can build a farm of test servers orchestrated by a central controller, but this is getting pretty complicated, requires dedicated hardware and network resources, and still isn’t a realistic real-world scenario anyway.
  4. The biggest disadvantage is inherent in all on-premise tools in this category in that it is not cloud based.  Unless you are developing an in-house application and all users are on the LAN, it does not makes a ton of sense to rely (entirely) on test results from inside your network.  I’m not suggesting they aren’t useful but if users are geographically distributed then testing in that mode should be considered.
  5. Your time: doing everything yourself is a trap many smart folks fall into, and often times at the expense of project deadlines, focus. Your time is valuable and in most cases it could be better spend somewhere else.

This discussion really boils down to if you like to do things yourself or if the project scope and criticality dictate using commercial tools and expertise.

For the purposes of general testing, getting familiar with how load testing works and rough order of magnitude sizing, you can certainly use open source tools on your own – with the caveats mentioned.  If the application is likely to scale significantly or have users geographically distributed, then I do think using a cloud based service is a much more realistic way to test.

In addition to the decision of open source versus commercial tools is if professional consulting services should be engaged.  Testing should be an integral part of the development process and many teams do not have expertise (or time) to develop a comprehensive test plan, script and configure the test, analyse the data and finally sort out remediation strategies on their own.

This is where engaging experts who are 100% focused on testing can provide real tangible value and ensure that your application scales and performs exactly as planned.

A strategy I have personally seen work quite well with a variety of complex technologies is to engage professional services and training at the onset of a project to develop internal capabilities and expertise, allowing the organization to extract maximum value from the commercial product of choice.

I always recommended to my customers to budget for training and service up front with any product purchase instead of trying to shoe-horn it in later, ensuring new capabilities promised by the commercial product are realized and management is satisfied with the product value and vendor relationship.


This post was written by Peter Cannell. Peter has been a sales and engineering professional in the IT industry for over 15 years. His experience spans multiple disciplines including Networking, Security, Virtualization and Applications. He enjoys writing about technology and offering a practical perspective to new technologies and how they can be deployed. Follow Peter on his blog or connect with him on Linkedin.

Don’t miss Peter’s next post, subscribe to the Load Impact blog by clicking the “follow” button below. 

Categories: Load & Perf Testing

How Models Change

DevelopSense - Michael Bolton - Sat, 07/19/2014 - 13:38
Like software products, models change as we test them, gain experience with them, find bugs in them, realize that features are missing. We see opportunities for improving them, and revise them. A product coverage outline, in Rapid Testing parlance, is an artifact (a map, or list, or table…) that identifies the dimensions or elements of […]
Categories: Software Testing

Test Automation Can Be So Much More…

Eric Jacobson's Software Testing Blog - Fri, 07/18/2014 - 13:32

I often hear people describe their automated test approach by naming the tool, framework, harness, technology, test runner, or structure/format.  I’ve described mine the same way.  It’s safe.  It’s simple.  It’s established.  “We use Cucumber”.

Lately, I’ve seen things differently.

Instead of trying to pigeon hole each automated check into a tightly controlled format for an entire project, why not design automated checks for each Story, based on their best fit for that story?

I think this notion comes from my context-driven test schooling.  Here’s an example:

On my current project, we said “let’s write BDD-style automated checks”.  We found it awkward to pigeon-hole many of our checks into Given, When, Then.  After eventually dropping the mandate for BDD-style, I discovered the not-as-natural-language style to be easier to read, more flexible, and quicker to author…for some Stories.  Some Stories are good candidates for data-driven checks authored via Excel.  Some might require manual testing with a mocked product...computer-assisted-exploratory-testing…another use of automation.  Other Stories might test better using non-deterministic automated diffs.

Sandboxing all your automated checks into FitNesse might make test execution easier.  But it might stifle test innovation.

Categories: Software Testing

#UnsexyTech and trying to make it a little sexier

LoadImpact - Thu, 07/17/2014 - 08:54

We think what we do is pretty cool. I mean come on! Performance and load testing, who doesn’t get excited at the idea?!

Well, apparently not everyone. Some have even said performance testing is a bit like selling health insurance: most people know it’s important to have, but you don’t reap the benefits of having it until something unexpected happens.

In any event, we wanted to try and find a way to explain what we do in a more relatable and humorous way. Framing our somewhat “unsexy tech”  in a way that connects back to everyone’s everyday lives.

Well, here is is. With the help of our video producers Adme (props to a job well done), we made this nifty short video to explain what we do and, if possible, make you chuckle a little. Enjoy!



Are you working with unsexy tech? Let us know why you think your tech is super sexy in the comments below.

Categories: Load & Perf Testing

How to extract User Experience by complementing Splunk

Splunk is a great Operational Intelligence solution capable of processing, searching and analyzing masses of machine-generated data from a multitude of disparate sources. By complementing it with an APM solution you can deliver insights that provide value beyond the traditional log analytics Splunk is built upon: True Operational Intelligence with dynaTrace and Splunk for Application Monitoring […]

The post How to extract User Experience by complementing Splunk appeared first on Compuware APM Blog.

Categories: Load & Perf Testing

How to create the illusion of faster web pages (while creating actual happier users)

Web Performance Today - Wed, 07/16/2014 - 17:17

The internet may change, and web pages may grow and evolve, but user expectations are constant. Throughout several studies, these numbers about response times and human perception have been consistent for more than 45 years:

These numbers are hard-wired. We have zero control over them. They’re consistent regardless of the type of device, application, or connection we’re using at any given moment.

These numbers are why our industry has attached itself to the goal of making pages render in 1 second or less — and to the even more audacious goal that pages render in 100 milliseconds or less.

But when you consider how many things have to happen before anything begins to appear in the browser — from DNS lookup and TCP connection to parsing HTML, downloading stylesheets, and executing JavaScript — 1 second can seem impossible. And forget about 100 milliseconds. In our most recent state of the union for ecommerce performance, we found that start render time for the top 500 retailers was 2.9 seconds. In other words, a typical visitor sits and stares at a blank screen for almost 3 seconds before he or she even begins to see something. Not good.

We can do better.

I talk a lot about page bloat, insidious third-party scripts, the challenges of mobile performance, and all the other things that make hitting these goals seem like an impossible feat. But rather than get discouraged, let me point you toward this great quote from Ilya Grigorik in his book High Performance Browser Networking:

“Time is measured objectively but perceived subjectively, and experiences can be engineered to improve perceived performance.”

Today I’m going to talk about some tricks and techniques you can use to manipulate subjective time to your advantage.

1. Progress bars

Never underestimate the power of a progress bar to help improve the user experience on pages that are necessarily slower (e.g. search results page). Progress bars reassure visitors that the page is actually functional and give a sense of how much longer the visitor will need to wait.

A few tips:

  • A progress bar on a page that loads in less than 5 seconds or less will make that page feel slower than it really is. Jakob Nielsen suggests that percent-done progress bars be reserved for pages/applications that take 10 seconds or longer to load.
  • Progress bars that offer the illusion of a left-moving ripple can improve perceived wait time by up to 10%.
  • Progress bars that pulse, and that increase pulsation frequency as the bar progresses, are perceived as being faster.
  • Similarly, progress bars that speed up are considered more satisfying by users. (Not surprising. This harkens back to the research on colonoscopy pain perception we covered a while back.)

Here’s a video that shows some of these principles in action:

More good progress bar best practices here.

2. Load something useful first

This principle seems like a no-brainer, but it’s amazing how many times I analyze pages that load primary content last. In our spring performance SOTU, we found that the median Time to Interact (TTI) for the top 100 retailers was 5.4 seconds. This means that it takes 5.4 seconds for a page’s most important content to render. This represents a failure for both the site owner and the site visitor.

A Nielsen-Norman Group eyetracking study found that a user who is served feature content within the first second of page load spent 20% of his or her time within the feature area, whereas a user who is subjected to an eight-second delay of a page’s feature content spent only 1% of his or her time visually engaging with that area of the page.

In other words, rendering your most important content last is a great way to ensure that your visitors get a sub-par user experience.

3. Defer non-essential content

Further to the point above, deferral is an excellent technique for clearing the decks to ensure that your primary content quickly takes center stage. Help your visitors see the page faster by delaying the loading and rendering of any content that is below the initially visible area (sometimes called “below the fold”). Deferring non-essential content won’t change your bottom-line load time, but it can dramatically improve your perceived load time.

Many script libraries aren’t needed until after a page has finished rendering. Downloading and parsing these scripts can safely be deferred until after the onLoad event. For example, scripts that support interactive user behavior, such as “drag and drop,” can’t possibly be called before the user has even seen the page. The same logic applies to script execution. Defer as much as possible until after onload instead of needlessly holding up the initial rendering of the important visible content on the page.

The script to defer could be your own or, often more importantly, scripts from third parties. Poorly optimized scripts for advertisements, social media widgets, or analytics support can block a page from rendering, sometimes adding precious seconds to load times.

4. Progressive image rendering

Progressive image rendering is predicated on the idea that the faster you can get a complete image in the browser, no matter what the resolution, the better the user experience — as opposed to baseline images, which load line by line and take much longer to render a complete image.

(Image source)

Progressive JPEGs aren’t new. They were widely used in the 1990s but fell out of favor due to performance issues caused by slow connection speeds and crudely rendered JPEGs. Back in the day, watching a progressive image load pixel by pixel was a painful experience. Now that connection speeds have improved and progressive JPEGs are more sophisticated, this technique is feasible again and is returning as a performance best practice. Last year, Google engineer Pat Meenan studied the top 2,000 retail sites and found that progressive JPEGs improved median load time by up to 15%.

Despite Pat’s findings, and despite the argument that rendering images progressively should also yield a better perceived UX, progressive JPEGs have an incredibly low adoption rate: around 7-8%. The reason for this could be the dated prejudice I just mentioned, or it could be the fact that Photoshop’s default save option is for baseline images. (If you’re anti-progressive JPEG, I’d love to know why. Please let me know in the comments.)

Note: Kent Alstad and I will be presenting some cool new research about the impact of progressive image rendering on the user experience at Velocity in New York and Barcelona this fall. I hope to see you there!

5. Auto-preloading (AKA predictive browser caching)

Not to be confused with browser preloading, auto-preloading is a powerful performance technique in which all user paths through a website are observed and recorded. Based on this massive amount of aggregated data, the auto-preloading engine can predict where a user is likely to go based on the page they are currently on and the previous pages in their path. The engine loads the resources for those “next” pages in the user’s browser cache, enabling the page to render up to 70% faster.

(Note that this is necessarily a data-intensive, highly dynamic technique that can only be performed by an automated solution. If you’re interested in learning more, read Kent’s excellent post here.)

6. Pretend to work, even when you don’t

Mark Wilson wrote a great post about a handful of tricks Instagram uses to make their pages appear more responsive. My favourite is the one they call “optimistic actions”. To illustrate: even if your connection is down or slow, if you click on the Instagram “like” button but your connection is slow or down, the UI registers the click and highlights the button. It send the actual data back to server whenever the connection is restored. In this way, the site always appears to be working, even when it’s not.

I’m curious to see other applications of this technique in the wild. If anyone can point me toward more examples, please do.

7. Touch event conversion

This tip applies to mobile. If you’re a mobile user, then you’re probably already aware of the slight delay that occurs between the time you touch your screen and the time it takes for the page to respond. That’s because every time you touch your screen, the device has to translate that touch event into a click event. This translation takes between 300-500 milliseconds per click.

At Radware, we’ve developed a feature called Touch Event Conversion in our FastView solution. It automatically converts click events to touch events, which relieves the mobile browser of the realtime overhead of the click-to-touch translation.

If you want to explore how to do this manually, check out Kyle Peatt’s post on the Mobify blog. He covers some other mobile performance-enhancing tips there as well.

8. Make perceived value match the wait

If long wait times are necessary, ensure that you’re delivering something with a value that’s commensurate with the wait. A good example of this is travel websites. When you’re searching for the best hotel rates, most of us don’t mind waiting several seconds. We rationalize the wait because we assume that the engine is searching a massive repository of awesome travel deals in order to give us the very best results.

If you have experience with other perception-enhancing performance practices, I’d love to hear about them. Leave me a note in the comments.

Related posts:

The post How to create the illusion of faster web pages (while creating actual happier users) appeared first on Web Performance Today.

Are we getting attacked? No, it’s just Google indexing our site

Friday morning at 7:40AM we received the first error from our APMaaS Monitors informing us about our Community Portal being unavailable. It “magically recovered itself” within 20 minutes but just about an hour later was down again. The Potential Root Cause was reported by dynaTrace which captured an Out-of-Memory (OOM) Exception in Confluence’s JVM that […]

The post Are we getting attacked? No, it’s just Google indexing our site appeared first on Compuware APM Blog.

Categories: Load & Perf Testing

You are lying

A few years back I was the manager of an internal service provider. My job was to deliver services to pharmaceutical business units within my company. We bought service externally as well as created our own and sold standards based functionality internally on a no profit basis. My main responsibility was Intel based servers and […]

The post You are lying appeared first on Compuware APM Blog.

Categories: Load & Perf Testing

Very Short Blog Posts (20): More About Testability

DevelopSense - Michael Bolton - Mon, 07/14/2014 - 15:30
A few weeks ago, I posted a Very Short Blog Post on the bare-bones basics of testability. Today, I saw a very good post from Adam Knight talking about telling the testability story. Adam focused, as I did, on intrinsic testability—things in the product itself that it more testable. But testability isn’t just a product […]
Categories: Software Testing

Measuring Coverage at Google

Google Testing Blog - Mon, 07/14/2014 - 13:45
By Marko Ivanković, Google Zürich

Code coverage is a very interesting metric, covered by a large body of research that reaches somewhat contradictory results. Some people think it is an extremely useful metric and that a certain percentage of coverage should be enforced on all code. Some think it is a useful tool to identify areas that need more testing but don’t necessarily trust that covered code is truly well tested. Others yet think that measuring coverage is actively harmful because it provides a false sense of security.

Our team’s mission was to collect coverage related data then develop and champion code coverage practices across Google. We designed an opt-in system where engineers could enable two different types of coverage measurements for their projects: daily and per-commit. With daily coverage, we run all tests for their project, where as with per-commit coverage we run only the tests affected by the commit. The two measurements are independent and many projects opted into both.

While we did experiment with branch, function and statement coverage, we ended up focusing mostly on statement coverage because of its relative simplicity and ease of visualization.

How we measured
Our job was made significantly easier by the wonderful Google build system whose parallelism and flexibility allowed us to simply scale our measurements to Google scale. The build system had integrated various language-specific open source coverage measurement tools like Gcov (C++), Emma / JaCoCo (Java) and Coverage.py (Python), and we provided a central system where teams could sign up for coverage measurement.

For daily whole project coverage measurements, each team was provided with a simple cronjob that would run all tests across the project’s codebase. The results of these runs were available to the teams in a centralized dashboard that displays charts showing coverage over time and allows daily / weekly / quarterly / yearly aggregations and per-language slicing. On this dashboard teams can also compare their project (or projects) with any other project, or Google as a whole.

For per-commit measurement, we hook into the Google code review process (briefly explained in this article) and display the data visually to both the commit author and the reviewers. We display the data on two levels: color coded lines right next to the color coded diff and a total aggregate number for the entire commit.

Displayed above is a screenshot of the code review tool. The green line coloring is the standard diff coloring for added lines. The orange and lighter green coloring on the line numbers is the coverage information. We use light green for covered lines, orange for non-covered lines and white for non-instrumented lines.

It’s important to note that we surface the coverage information before the commit is submitted to the codebase, because this is the time when engineers are most likely to be interested in improving it.

One of the main benefits of working at Google is the scale at which we operate. We have been running the coverage measurement system for some time now and we have collected data for more than 650 different projects, spanning 100,000+ commits. We have a significant amount of data for C++, Java, Python, Go and JavaScript code.

I am happy to say that we can share some preliminary results with you today:

The chart above is the histogram of average values of measured absolute coverage across Google. The median (50th percentile) code coverage is 78%, the 75th percentile 85% and 90th percentile 90%. We believe that these numbers represent a very healthy codebase.

We have also found it very interesting that there are significant differences between languages:


The table above shows the total coverage of all analyzed code for each language, averaged over the past quarter. We believe that the large difference is due to structural, paradigm and best practice differences between languages and the more precise ability to measure coverage in certain languages.

Note that these numbers should not be interpreted as guidelines for a particular language, the aggregation method used is too simple for that. Instead this finding is simply a data point for any future research that analyzes samples from a single programming language.

The feedback from our fellow engineers was overwhelmingly positive. The most loved feature was surfacing the coverage information during code review time. This early surfacing of coverage had a statistically significant impact: our initial analysis suggests that it increased coverage by 10% (averaged across all commits).

Future work
We are aware that there are a few problems with the dataset we collected. In particular, the individual tools we use to measure coverage are not perfect. Large integration tests, end to end tests and UI tests are difficult to instrument, so large parts of code exercised by such tests can be misreported as non-covered.

We are working on improving the tools, but also analyzing the impact of unit tests, integration tests and other types of tests individually.

In addition to languages, we will also investigate other factors that might influence coverage, such as platforms and frameworks, to allow all future research to account for their effect.

We will be publishing more of our findings in the future, so stay tuned.

And if this sounds like something you would like to work on, why not apply on our job site?

Categories: Software Testing

Pushing Up All The Levels On The Mixing Board

Eric Jacobson's Software Testing Blog - Fri, 07/11/2014 - 09:50

…may not be a good way to start testing.

I heard a programmer use this metaphor to describe the testing habits of a tester he had worked with. 

As a tester, taking all test input variables to their extreme, may be an effective way to find bugs.  However, it may not be an effective way to report bugs.  Skilled testers will repeat the same test until they isolate the minimum variable(s) that cause the bug.  Or using this metaphor, they may repeat the same test with all levels on the mixing board pulled down, except the one they are interested in observing.

Once identified, the skilled tester will repeat the test only changing the isolated variable, and accurately predict a pass or fail result.

Categories: Software Testing

Understanding Application Performance on the Network – Part V: Processing Delays

In Part IV, we wrapped up our discussions on bandwidth, congestion and packet loss. In Part V, we examine the four types of processing delays visible on the network, using the request/reply paradigm we outlined in Part I. Server Processing (Between Flows) From the network’s perspective, we allocate the time period between the end of […]

The post Understanding Application Performance on the Network – Part V: Processing Delays appeared first on Compuware APM Blog.

Categories: Load & Perf Testing

“Bug In The Test” vs. “Bug In The Product”

Eric Jacobson's Software Testing Blog - Wed, 07/09/2014 - 12:44

Dear Test Automators,

The next time you discuss automation results, please consider qualifying the context of the word “bug”.

If automation fails, it means one of two things:

  1. There is a bug in the product-under-test.
  2. There is a bug in the automation.

The former is waaaaaay more important than the latter.  Maybe not to you, but certainly for your audience.

Instead of saying,

This automated check failed”,

consider saying,

This automated check failed because of a bug in the product-under-test”.


Instead of saying,

I’m working on a bug”,

consider saying,

I’m working on a bug in the automation”.


Your world is arguably more complex than that of testers who don’t use automation.  You must test twice as many programs (the automation and the product-under-test).  Please consider being precise when you communicate.

Categories: Software Testing

Top Web Performance Tips from Velocity Conf 2014

I attended the Velocity Conference at the Santa Clara Convention Center recently and there were many sessions on how to improve the performance of your web sites on mobile or web, as well as discussions around what companies are doing to get the “above the fold” content within the 1000 ms threshold. Here are snippets […]

The post Top Web Performance Tips from Velocity Conf 2014 appeared first on Compuware APM Blog.

Categories: Load & Perf Testing

Performance & Scalability Test: Pantheon -VS- Other Drupal-Based Platforms

LoadImpact - Tue, 07/08/2014 - 07:22

In this post I will demonstrate how to  setup a Drupal 7 site from scratch using Pantheon. Then, using Load Impact, I will attempt to compare the performance of Pantheon versus other Drupal-based platforms (Aberdeen Cloud platform and VPS).


For those who don’t know Pantheon, it’s a cloud platform for web development using different frameworks such as Drupal or WordPress. It provides development, testing and live environments for easy deployment using git. And since it’s built using containers instead of virtual machines, it’s performance and scalability is considerably better than traditional hosting providers.

Pantheon also provides a platform with preconfigured tools such as Varnish, Redis, Apache Solr, automated backups and application updates, etc.

For this performance test, we are going to need to create a Pantheon user account.

As a registered Pantheon user, we now see the Pantheon control panel:

Now we need to create a site in order to install Drupal and run a performance test.

Steps to creating a Drupal 7 site using Pantheon

1. After registering an account with Pantheon, click “Create a site now” link.

2. Next, provide some information about the site, such as the name and the access URL. Later, you can easily redirect your existing domain if you need it.

Press the “Create Site” button to continue with the process.


3. Create or import a site

You can create a site from scratch using one of the available web application cores for Drupal or WordPress, or you can import an existing one – packing your application files and database dump.

For this test, we are going to start a Drupal 7 site from scratch, but feel free to import your existing application if you want to compare its performance in Pantheon.

Select “Start from scratch” option and one of the available cores – in this case Drupal 7.

Press “Install Drupal 7” button to continue with the site installation.

Pantheon will setup the site with the best suitable configuration for Drupal 7.


4. Use the dashboard and configure your application

Once the site is completely installed you will have access to the site Dashboard where you can configure different aspects of your application, create backups and use the Pantheon tools.

Now it’s time to install Drupal. As you can see, the site installer only copied Drupal 7 files but it didn’t execute Drupal installation.

You have to do it manually, so just press “Visit Development Site” button.

5. Install Drupal

In this case, you are going to install Drupal 7 using a Pantheon profile – a custom Drupal distribution preconfigured with Pantheon modules and custom configuration that enables Drupal caching and provides better performance, and an Apache Solr integration.

Press “Save and continue” button to go to next installation step where you can select default language for Drupal. You can install Drupal 7 in other languages – but that can be done later.

Press “Save and continue” button again and Drupal will install with pre-selected Pantheon configuration.

After the installation you have to configure site details, such as site name or administrator username, password and email.

Once you’ve completed the form details, Drupal should be installed and ready to use.


6. Start using Drupal

To start using Drupal just press “Visit your new site” link and you will access your brand new Drupal 7 website.

7.  Setup your git repository in your local machine

Now it’s time to set up your git repository in your local machine so you can add or modify your Drupal website.

First of all, you need to add your SSH public key into your Pantheon user account. Just go to “Sites & Account” in your user menu.

If you don’t already have a SSH key you have to create it. You can find more information about SSH key generation here: http://helpdesk.getpantheon.com/customer/portal/articles/366938-generate-ssh-keys

Once you have created your SSH key, you should add your Public Key just by pressing the “Add key” button. For more information about SSH keys, visit this page: http://helpdesk.getpantheon.com/customer/portal/articles/361246-loading-ssh-keys

Now you can connect to Pantheon git repository using SSH.

8. Configure the repository in your local machine

To configure the repository in your local machine you should have git previously configured. If you need to install git just follow this guide for all platforms https://help.github.com/articles/set-up-git

First you should copy the path to connect to your server using SSH, because you are going to need it later. You can find it in your project dashboard.

We are going to use Git as connection mode so be sure that it’s selected and copy the URL you find in the textbox. As you can see in the following screenshot:

Go to your terminal and execute the following command from the location where you want to create your project repository folder (Don’t forget to use the URL you previously copied):

$ git clone ssh://codeserver.dev.xxx@codeserver.dev.xxx.drush.in:2222/~/repository.git drupal-loadimpact

If everything goes well, the command will create a folder named drupal-loadimpact with all the files corresponding to Drupal 7 core.

9. Install devel module

Now we are going to install devel module. This Drupal module will help us to creat thousands of nodes in our Drupal website.

A node is a Drupal piece of content with multiple fields and properties, and we are going to use them for our performance test.

You can download the module using wget or drush, but remember to place the module files into sites/all/modules directory in your repository folder.

Then add the new files for commit using the following command:

$ git add sites/all/modules/devel

Commit the changes into your local repository using a descriptive message:

$ git commit -m"Added devel module."

And finally push the changes to the remote repository:

$ git push origin master

You can check that everything went well by checking the commit log in your project dashboard.

You can see the new commit with your descriptive message.

10.  Install devel module by enabling it in Drupal

Select “Modules” menu option from the top menu in your Drupal site.

Enable only Devel and Devel generate modules.


11. Generate content 

Now select “Configuration” menu option from the top menu. And then select “Generate content” option.


In this example, we are going to create 5000 nodes of type Article for our performance test. It should be enough to have a high amount of content in the Drupal database and create different performance tests.

Press “Generate” button and relax because it’s going to take a while. You can take a cup of coffee ;)

…. (coffee break)

So, here you have your 5000 recently created articles. To see them just select “Content” from the top menu.

12. Create page listing all content (for performance testing purposes)

Now we are going to create a page that will list all the content so it will create a high amount of database queries and render a lot information in the page.

For that we will need Views (https://www.drupal.org/project/views) and Chaos Tools Suite (https://www.drupal.org/project/ctools) modules.

You can download them using the same method you used to download Devel module. Don’t forget to place them in sites/all/modules folder to let Drupal to find them and be able to install them.

Once you have downloaded the modules into your local repository, you can add and commit the modules into your local repository as we did before:

$ git add sites/all/modules/views $ git commit -m”Added views module.” $ git add sites/all/modules/ctools $ git commit -m"Added ctools module."

Now push the changes to your remote repository:

$ git push origin master

Once again, you can see your commits in the Commit Log in your project dashboard and verify that everything went well.

To enable the modules, select “Modules” option in your top menu:

Enable Chaos tools module because it’s a dependency of Views module:

Also enable Views and Views UI modules:

Now select “Structure” option form the top menu to start creating your view page.

Then select “Views” option.

On this page you can find a list of preconfigured views. We are going to create our own view page with a list of a specific number of content articles.



13. Create and configure a new View

Select “Add new view” link to start.

Create a new view using the following configuration and press “Continue & edit” button.

This page is the configuration panel for your view. You can add fields, sorting order and filters as well as arguments and other custom features.

We are going to add a few fields to display more content in the page.

Now you can see that there is only Title field available. Press the “Add” button in “Fields” section.

In the next window select Body and Image fields and press “Apply (all displays)” button with default settings for all fields.

Now your “Fields” section should contain the following:

  • Content: Title
  • Content: Body(body)
  • Content: Image(image)

Press “Save” button to save your new view.

14. Visit your new page

Now you should be able to visit your page. The page we created for testing is here: http://dev-drupal-loadimpact.gotpantheon.com/performance-test

You should see a page with 100 article nodes –  which is a normal amount of data for Drupal to load.

It will create a considerable number of database queries and processes to render all the content into the page.

Consider to increase or decrease the number of items in the “Pager” section of your view if you want to test with different amount of data loaded.

You can select different values to adapt the performance test to your real needs.


We have to take into consideration the default configuration that Pantheon profile applies to Drupal.

If you select “Configuration” in the top menu you can find “Performance” menu item:

In this page you can find configuration settings related to Drupal performance. You can see that there are a few settings already enabled.

Page and block caches are enabled for 15 minutes. Also, CSS and JavaScript are aggregation is enable – what packs all CSS and JavaScript files and build them together in order to decrease the number of requests to the server to download all the files.

It’s important to clear caches every time you run a new performance test to be sure that you don’t have cached pages. Just press “Clear all caches” button.

Testing the performance of Drupal-based platforms

Now it’s time for performance testing using Load Impact. Create a user account if you don’t already have one.

After that visit “My tests” to start creating your performance test and press “Start a new test” button. This will execute a rather small test of 50 concurrent users for 5 minutes.

Type your test page URL into the text box and press “Start test” button.

You can also configure a larger test from the “Test configurations” page (e.g. with ramp-up/down; additional IPs, multi-geo load generation; mobile network emulation, server metrics, etc.)

The test will start and you will see how Load Impact is creating requests from different hosts.


Pantheon results

The following are the performance test results for the same Drupal site (using the same configuration and number of content nodes) hosted in Pantheon and Aberdeen Cloud platforms and a Virtual Private Server:

Pantheon results (user load time)

Aberdeen Cloud 

Aberdeen Cloud results (user load time)

VPS (CPU: 1.5 Ghz. – RAM: 1 Gb.)

VPS results (user load time)


You can observe that Pantheon keeps user load time between 1.5 and 3 seconds, meanwhile with Aberdeen Cloud platform and the VPS, the user load time stays between 3 and 4 seconds.

Based solely on these few simple load tests, it seems Pantheon manages to serve Drupal pages at least one second faster than the other tested platforms. 

Now it’s time for you to try different configurations for the Drupal site, like number of content nodes, disable caching or file aggregation and see how it affects performance and scalability.

Read more about load and performance testing with Pantheon: http://helpdesk.getpantheon.com/customer/portal/articles/690128-load-and-performance-testing


This blog post was written by Ruben Teijero. Follow the Load Impact blog (see below) for more great articles by Ruben and many other fantastic developers, testers and operations pros.

Categories: Load & Perf Testing

How to Roll Out Citrix XenDesktop Globally, Without a Hitch

Perf Planet - Tue, 07/08/2014 - 07:00

In addition to Pieter Van Heck, Alasdair Patton (Delivery Architect for Compuware APM) and Mark Revill (Field Technical Services, Compuware APM) also contributed to the authoring of this post.  Windows XP is living on borrowed time: Microsoft officially ended extended support for it on 8 April 2014. It was inevitable, but now many companies and organisations […]

The post How to Roll Out Citrix XenDesktop Globally, Without a Hitch appeared first on Compuware APM Blog.

How to Roll Out Citrix XenDesktop Globally, Without a Hitch

In addition to Pieter Van Heck, Alasdair Patton (Delivery Architect for Compuware APM) and Mark Revill (Field Technical Services, Compuware APM) also contributed to the authoring of this post.  Windows XP is living on borrowed time: Microsoft officially ended extended support for it on 8 April 2014. It was inevitable, but now many companies and organisations […]

The post How to Roll Out Citrix XenDesktop Globally, Without a Hitch appeared first on Compuware APM Blog.

Categories: Load & Perf Testing