Software Testing

Resource Timing practical tips

Steve Souders - Thu, 08/21/2014 - 01:45

The W3C Web Performance Working Group brought us Navigation Timing in 2012 and it’s now available in nearly every major browser. Navigation Timing defines a JavaScript API for measuring the performance of the main page. For example:

// Navigation Timing var t = performance.timing, pageloadtime = t.loadEventStart - t.navigationStart, dns = t.domainLookupEnd - t.domainLookupStart, tcp = t.connectEnd - t.connectStart, ttfb = t.responseStart - t.navigationStart;

Having timing metrics for the main page is great, but to diagnose real world performance issues it’s often necessary to drill down into individual resources. Thus, we have the more recent Resource Timing spec. This JavaScript API provides similar timing information as Navigation Timing but it’s provided for each individual resource. An example is:

// Resource Timing var r0 = performance.getEntriesByType("resource")[0], loadtime = r0.duration, dns = r0.domainLookupEnd - r0.domainLookupStart, tcp = r0.connectEnd - r0.connectStart, ttfb = r0.responseStart - r0.startTime;

As of today, Resource Timing is supported in Chrome, Chrome for Android, Opera, IE10, and IE11. This is likely more than 50% of your traffic so should provide enough data to uncover the slow performing resources in your website.

Using Resource Timing seems straightforward, but when I wrote my first production-quality Resource Timing code I ran into several issues that I want to share. Here are my practical tips for tracking Resource Timing metrics in the real world.

1. Use getEntriesByType(“resource”) instead of getEntries().

You begin using Resource Timing by getting the set of resource timing performance objects for the current page. Many Resource Timing examples use performance.getEntries() to do this, which implies that only resource timing objects are returned by this call. But getEntries() can potentially return four types of timing objects: “resource”, “navigation”, “mark”, and “measure”.

This hasn’t caused problem for developers so far because “resource” is the only entry type in most pages. The “navigation” entry type is part of Navigation Timing 2, which isn’t implemented in any browser AFAIK. The “mark” and “measure” entry types are from the User Timing specification which is available in some browsers but not widely used.

In other words, getEntriesByType("resource") and getEntries() will likely return the same results today. But it’s possible that getEntries()  will soon return a mix of performance object types. It’s best to use performance.getEntriesByType("resource") so you can be sure to only retrieve resource timing objects. (Thanks to Andy Davies for explaining this to me.)

2. Use Nav Timing to measure the main page’s request.

When fetching a web page there is typically a request for the main HTML document. However, that resource is not returned by performance.getEntriesByType("resource"). To get timing information about the main HTML document you need to use the Navigation Timing object (performance.timing).

Although unlikely, this could cause bugs when there are no entries. For example, my earlier Resource Timing example used this code:

performance.getEntriesByType("resource")[0]

If the only resource for a page is the main HTML document, then getEntriesByType("resource") returns an empty array and referencing element [0] results in a JavaScript error. If you don’t have a page with zero subresources then you can test this on http://fast.stevesouders.com/.

3. Beware of issues with secureConnectionStart.

The secureConnectionStart property allows us to measure how long it takes for SSL negotiation. This is important – I often see SSL negotiation times of 500ms or more. There are three possible values for secureConnectionStart:

  • If this attribute is not available then it must be set to undefined.
  • If HTTPS is not used then it must be set to zero.
  • If this attribute is available and HTTPS is used then it must be set to a timestamp.

There are three things to know about secureConnectionStart. First, in Internet Explorer the value of secureConnectionStart is always “undefined” because it’s not available (the value is buried down inside WinINet).

Second, there’s a bug in Chrome that causes secureConnectionStart to be incorrectly set to zero. If a resource is fetched using a pre-existing HTTPS connection then secureConnectionStart is set to zero when it should really be a timestamp. (See bug 404501 for the full details.) To avoid skewed data make sure to check that secureConnectionStart is neither undefined nor zero before measuring the time for SSL negotiation:

var r0 = performance.getEntriesByType("resource")[0]; if ( r0.secureConnectionStart ) { var ssl = r0.connectEnd - r0.secureConnectionStart; }

Third, the spec is misleading with regard to this line: “…if the scheme of the current page is HTTPS, this attribute must return the time immediately before the user agent starts the handshake process…” (emphasis mine). It’s possible for the current page to be HTTP and still contain HTTPS resources for which we want to measure the SSL negotiation. The spec should be changed to “…if the scheme of the resource is HTTPS, this attribute must return the time immediately before the user agent starts the handshake process…”. Fortunately, browsers behave according to the corrected language, in other words, secureConnectionStart is available for HTTPS resources even on HTTP pages.

4. Add “Timing-Allow-Origin” to cross-domain resources.

For privacy reasons there are cross-domain restrictions on getting Resource Timing details. By default, a resource from a domain that differs from the main page’s domain has these properties set to zero:

  • redirectStart
  • redirectEnd
  • domainLookupStart
  • domainLookupEnd
  • connectStart
  • connectEnd
  • secureConnectionStart
  • requestStart
  • responseStart

There are some cases when it’s desirable to measure the performance of cross-domain resources, for example, when a website uses a different domain for its CDN (such as “youtube.com” using “s.ytimg.com”) and for certain 3rd party content (such as “ajax.googleapis.com”). Access to cross-domain timing details is granted if the resource returns the Timing-Allow-Origin response header. This header specifies the list of (main page) origins that are allowed to see the timing details, although in most cases a wildcard (“*”) is used to grant access to all origins. As an example, the Timing-Allow-Origin response header for http://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js is:

Timing-Allow-Origin: *

It’s great when 3rd parties add this response header; it allows website owners to measure and understand the performance of that 3rd party content on their pages. As reported by (and thanks to) Ilya Grigorik, several 3rd parties have added this response header. Here are some example resources that specify “Timing-Allow-Origin: *”:

When measuring Resource Timing it’s important to determine if you have access to the restricted timing properties. You can do that by testing whether any of the restricted properties (listed above) (except for secureConnectionStart) is zero. I always use requestStart. Here’s a better version of the earlier code snippet that checks that the restricted properties are available before calculating the more detailed performance metrics:

// Resource Timing var r0 = performance.getEntriesByType("resource")[0], loadtime = r0.duration; if ( r0.requestStart ) { var dns = r0.domainLookupEnd - r0.domainLookupStart, tcp = r0.connectEnd - r0.connectStart, ttfb = r0.responseStart - r0.startTime; } if ( r0.secureConnectionStart ) { var ssl = r0.connectEnd - r0.secureConnectionStart; }

It’s really important that you do these checks. Otherwise, if it’s assumed you have access to these restricted properties you won’t actually get any errors; you’ll just get bogus data. Since the values are set to zero when access is restricted, things like “domainLookupEnd – domainLookupStart” translate to “0 – 0″ which returns a plausible result (“0″) but is not necessarily the true value of the (in this case) DNS lookup. This results in your metrics having an overabundance of “0″ values which incorrectly skews the aggregate stats to look rosier than they actually are.

5. Understand what zero means.

As mentioned in #4, some Resource Timing properties are set to zero for cross-domain resources that have restricted access. And again, it’s important to check for that state before looking at the detailed properties. But even when the restricted properties are accessible it’s possible that the metrics you calculate will result in a value of zero, and it’s important to understand what that means.

For example, (assuming there are no access restrictions) the values for domainLookupStart and domainLookupEnd are timestamps. The difference of the two values is the time spent doing a DNS resolution for this resource. Typically, there will only be one resource in the page that has a non-zero value for the DNS resolution of a given hostname. That’s because the DNS resolution is cached by the browser; all subsequent requests use that cached DNS resolution. And since DNS resolutions are cached across web pages, it’s possible to have all DNS resolution calculations be zero for an entire page. Bottomline: a zero value for DNS resolution means it was read from cache.

Similarly, the time to establish a TCP connection (“connectEnd – connectStart”) for a given hostname will be zero if it’s a pre-existing TCP connection that is being re-used. Each hostname should have ~6 unique TCP connections that should show up as 6 non-zero TCP connection time measurements, but all subsequent requests on that hostname will use a pre-existing connection and thus have a TCP connection time that is zero. Bottomline: a zero value for TCP connection means a pre-existing TCP connection was re-used.

The same applies to calculating the time for SSL negotiation (“connectEnd – secureConnectionStart”). This might be non-zero for up to 6 resources, but all subsequent resources from the same hostname will likely have a SSL negotiation time of zero because they use a pre-existing HTTPS connection.

Finally, if the duration property is zero this likely means the resource was read from cache.

6. Determine if 304s are being measured.

There’s another bug in Chrome stable (version 36) that has been fixed in version 37. This issue is going away, but since most users are on Chrome stable your current metrics are likely different than they are in reality. Here’s the bug: a cross-origin resource that does have Timing-Allow-Origin on a 200 response will not have that taken into consideration on a 304 response. Thus, 304 responses will have zeroes for all the restricted properties as shown by this test page.

This shouldn’t happen because the Timing-Allow-Origin header from the cached 200 response should be applied (by the browser) to the 304 response. This is what happens in Internet Explorer. (Try the test page in IE 10 or 11 to confirm.) (Thanks to Eric Lawrence for pointing this out.)

This impacts your Chrome Resource Timing results as follows:

  • If you’re checking that the restricted fields are zero (as described in #4), then you’ll skip measuring these 304 responses. That means you’re only measuring 200 responses. But 200 responses are slower than 304 responses, so your Resource Timing aggregate measurements are going to be larger than they are in reality.
  • If you’re not checking that the restricted fields are zero, then you’ll record many zero values which are faster than the 304 response was in reality, and your Resource Timing stats will be too optimistic.

There’s no easy way to avoid these biases, so it’s good news that the bug has been fixed. One thing you might try is to send a Timing-Allow-Origin on your 304 responses. Unfortunately, the popular Apache web server is unable to send this header in 304 responses (see bug 51223). Further evidence of the lack of Timing-Allow-Origin in 304 responses can be found by looking at the 3rd party resources listed in #4. As stated, it’s great that these 3rd parties return that header on 200 responses, but 4 out of 5 of them do not return Timing-Allow-Origin on 304 responses. Until Chrome 37 becomes stable it’s likely that Resource Timing metrics are skewed either too high or too low because of the missing details for the restricted properties. Fortunately, the value for duration is accurate regardless.

7. Look at Boomerang.

If you’re thinking of writing your own Resource Timing code you should first look at the Resource Timing plugin for Boomerang. (The code is on GitHub.) Boomerang is the popular open source RUM package maintained by Philip Tellis. He originally open sourced it when he was at Yahoo! but is now providing ongoing maintenance and enhancements as part of his work at SOASTA for the commercial version (mPulse). The code is clear, pithy, and robust and addresses many of the issues mentioned above.

In conclusion, Navigation Timing and Resource Timing are outstanding new specifications that give website owners much more visibility into the performance of their web pages. Resource Timing is the newer of the two specs, and as such there are still some wrinkles to be ironed out. These tips will help you get the most from your Resource Timing metrics. I encourage you to start tracking these metrics today to understand how your website is performing for the people who matter the most: real users.

 

 

Categories: Software Testing

ISTQB Foundation Level Training in Software Testing (CTFL) - Chicago - Sept 16 - 18, 2014

Randy Rice's Software Testing & Quality - Wed, 08/20/2014 - 10:14
If you are looking for ISTQB Foundation Level Software Testing Training at a reasonable price, in the Chicago area, this is your opportunity!

The class is Sept. 16 - 18, 2014, held in The Regus Offices at One Dearborn St. in downtown Chicago.

The instructor is Thomas Staab, CTFL.

We are only going to hold registrations open for one more week! We will close registrations at 5 p.m. CDT on Wed, Aug 27th.

The class is limited to 10 people. So if you want to get in on this unique opportunity - act fast!

The cost for the class is $2,000 per person, with the exam ($250 value) included in that price. The exam will be given on the final afternoon of the class.

If you bring two people, you can get $500 off your registrations. Just use code "ISTQB916" at check-out
time.

You can register here: https://www.mysoftwaretesting.com/ISTQB_Foundation_Level_Course_in_Software_Testing_p/ctflpub.htm To see the outline: http://riceconsulting.com/home/index.php/ISTQB-Training-for-Software-Tester-Certification/istqb-foundation-level-course-in-software-testing.html
I hope you can make it!
Randy
Categories: Software Testing

Fingered by an Error Message

QA Hates You - Tue, 08/19/2014 - 04:55

Why would a user do that?

None of that stopped 26-year-old Diondre J— of Slidell, who checked into Slidell Memorial Hospital on Aug. 5 under the name of her deceased sister, Delores, Slidell Police Department spokesman Daniel Seuzeneau said Wednesday.

When hospital staff attempted to put the information into the hospital’s database, an error message informed them they might have been treating a dead person. The police were contacted, and Diondre J— was stopped in the hospital parking lot.

It’s good to see someone was on the job testing to see what would happen if you tried to enter a patient’s date of treatment after the patient’s date of death.

Because sometimes a user might do that.

Categories: Software Testing

ISTQB Advanced Test Manager Training - Live Online Sept. 15 - 19

Randy Rice's Software Testing & Quality - Sun, 08/17/2014 - 22:24
I am offering a special live, online version of the ISTQB Advanced Test Manager Training the week of September 15 - 19, 2014. I'm also offering a special "buy 2 get 1 free" offer. Just use coupon code "ISTQB915" at https://www.mysoftwaretesting.com/ISTQB_Advanced_Test_Manager_Course_Public_Course_p/ctaltmpub.htm

Be sure to register as a remote attendee.

I will be the instructor and the class schedule will be from 8:30 a.m. to 4:00 p.m. CDT each day. You will get a complete set of printed course notes if you register at least one week in advance of the class. The sessions will be recorded and posted daily, so if you have miss a portion you can catch up.

I hope to see you there!
Categories: Software Testing

Testing on the Toilet: Web Testing Made Easier: Debug IDs

Google Testing Blog - Tue, 08/12/2014 - 13:01
by Ruslan Khamitov 

This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office.

Adding ID attributes to elements can make it much easier to write tests that interact with the DOM (e.g., WebDriver tests). Consider the following DOM with two buttons that differ only by inner text:
Save buttonEdit button<div class="jfk-button">Save</div><div class="jfk-button">Edit</div>
How would you tell WebDriver to interact with the “Save” button in this case? You have several options. One option is to interact with the button using a CSS selector:
div.jfk-button
However, this approach is not sufficient to identify a particular button, and there is no mechanism to filter by text in CSS. Another option would be to write an XPath, which is generally fragile and discouraged:
//div[@class='jfk-button' and text()='Save']
Your best option is to add unique hierarchical IDs where each widget is passed a base ID that it prepends to the ID of each of its children. The IDs for each button will be:
contact-form.save-button
contact-form.edit-button

In GWT you can accomplish this by overriding onEnsureDebugId()on your widgets. Doing so allows you to create custom logic for applying debug IDs to the sub-elements that make up a custom widget:
@Override protected void onEnsureDebugId(String baseId) {
super.onEnsureDebugId(baseId);
saveButton.ensureDebugId(baseId + ".save-button");
editButton.ensureDebugId(baseId + ".edit-button");
}
Consider another example. Let’s set IDs for repeated UI elements in Angular using ng-repeat. Setting an index can help differentiate between repeated instances of each element:
<tr id="feedback-{{$index}}" class="feedback" ng-repeat="feedback in ctrl.feedbacks" >
In GWT you can do this with ensureDebugId(). Let’s set an ID for each of the table cells:
@UiField FlexTable table;
UIObject.ensureDebugId(table.getCellFormatter().getElement(rowIndex, columnIndex),
baseID + colIndex + "-" + rowIndex);
Take-away: Debug IDs are easy to set and make a huge difference for testing. Please add them early.

Categories: Software Testing

The Role of a Test Architect

Randy Rice's Software Testing & Quality - Fri, 08/08/2014 - 13:14
I get some really good questions by e-mail and this is one of them.

Q: What does a QA Architect do in a team, and what skills are needed for this job?

A; To me, (although different organizations will have different definitions) the test architect is a person who understands testing at a very in-depth level and is responsible for things such as:
  • Designing test frameworks for test automation and testing in general
  • Directing and coordinating the implementation of test automation and other test tools
  • Designing test environments
  • Providing guidance on the selection of the most effective test design techniques in a given situation
  • Providing guidance on technical types of testing such as performance testing and security testing
  • Designing methods for the creation of test data
  • Coordinating testing with release processes
This is a "big picture" person who also understands software development and can work with developers to ensure that the test approaches align with development approaches. So, the test architect should be able to understand various project life cycles. These days, a test architect needs to understand the cloud, SOA and mobile computing.

The terms "architect" in many organizations implies a very deep level of understanding and is a highly respected position. The expectations are pretty high. The architect can provide guidance on things that the test manager may not have technical knowledge about. Yet, the test architect can focus on testing and not have to deal with administrative things that managers deal with.

Follow-up question: "What kind of company is more likely to have such a role? A large company or a smaller company? When you say "directing and coordinating" sounds like communicating across teams, like QA -> dev -> dev ops -> db to get things done."

A: I would say that you would likely find the role in companies that truly understand the role and value of testing. For example, they know that QA does not equal testing. It would be an interesting project to research how many companies in a sample group would have test architect as a defined role. I would tend to think of larger companies being more likely to have a test architect, but I've seen smaller software companies with test architects and larger companies with no one in any type of test leadership or designer role.



Another indication might be those companies that have a more centralized testing function, such as testing center of excellence. I have some misgivings about the COE approach in that they often fail because people see them as little bureaucracies instead of support. Also, they tend to try to "take over the world" in a company in terms of test practice. The lifespan of a testing COE is often less than 2 years from what I have seen. It's good money for testing consultants to come in and establish the COE, but then they leave (or get asked to leave) and the energy/interest goes away.

And...the company would need to see the need for both functional and technical testing. You need a test architect to put those pieces together.

This is not "command and control" but rather design and facilitation. And you are right, the test architect role touches many other areas, tasks and people.


 Follow-up Question: What kind of tools? I'm assuming you're talking more than handy shell scripts. Simulators? Rest clients like postman customizations?
A: Right, the tools are typically high-end commercial tools, but there is a trend toward open source tools and frameworks. One of my friends calls the big commercial tool approach "Big Pharma". The key is that the test architect knows how to define a framework where the tools can work together. This can include the customized homegrown tools as well. Those can be handy.

By the way, the term "framework" can have many meanings. The way I'm using the term here is a structure for test design and test automation that allows functional testers to focus on understanding the application under test and design (and implement) good tests, with the technical testers building and maintaining the technical side of automation.

We also have to expand the view to include not only test automation, but other support tools as well, such as incident management, configuration management and others. For example, there is a great opportunity to use tools to greatly reduce the time needed for test design. Tools such as ACTS (from nist.gov) and Hexawise can be helpful for combinatorial testing, test data generators are needed for test automation and performance testing, and I'm especially keen on model-based testing when the model is used to generate tests (not the manual approach). I think BenderRBT does a great job in designing tests from requirements. Grid Tools has recently introduced a tool called Agile Designer to design tests based on workflow. I'll have more information on that in an upcoming post.

What Does it Take to Become a Test Architect?

I suppose one could take many paths. However, I would not automatically assume someone in an SDET (Software Developer in Test) would be qualified. That's because more than just the technical perspective is needed. The test architect also needs to understand the business processes used in the organization. Personally, I think people at this level, or who aspire to it, would profit by studying Enterprise Architecture. My favorite is the Zachman Enterprise Architecture Framework.

I would look for:

1. Knowledge and understanding at a deep level - Not the superficial knowledge that most people never get past. This means that they know about metrics, code complexity, reliability modeling, performance modeling, security vulnerabilities - all at an advanced to expert level. This is why I encourage people who are on the certification path to go beyond foundation and on to advanced and expert levels of training. This also includes being a continuous learner and reading good books in the testing field and related disciplines. I would start with Boris Beizer's "Software Testing Techniques, 2nd Ed."

2. Meaningful experience - At the risk of just putting an arbitrary number of years as the baseline, I would think at least eight to ten years of solid test design, tool application and perhaps software design and development would be needed. You need a decent portfolio of work to show you have what it takes to work in the role.

3. Great interpersonal skills - The test architect has to negotiate at times and exert influence to get their ideas across. They have to get along with others to function as part of the larger organizational team. Of course, this also includes developers, test managers and development architects. Just because you are a guru doesn't mean you have to be a stubborn and contentious jerk.

4. Objectivity - When choosing between alternative approaches and tools, objectivity is needed.

5. Problem Solving - This requires a creative approach to solving problems and seeing challenges from different angles. It's not uncommon for solutions to be devised where no one has gone before.

I hope this helps raise the awareness of this important role.

Questions or comments? I'm happy to address them!

Randy



Categories: Software Testing

Don’t Bother Indicating “Pass” or “Fail”

Eric Jacobson's Software Testing Blog - Tue, 08/05/2014 - 10:42

This efficiency didn’t occur to me until recently.  I was doing an exploratory test session and documenting my tests via Rapid Reporter.  My normal process had always been to document the test I was about to execute…

TEST: Edit element with unlinked parent

…execute the test.  Then write “PASS” or “FAIL” after it like this…

TEST: Edit element with unlinked parent – PASS

But it occurred to me that if a test appears to fail, I tag said failure as a “Bug”, “Issue”, “Question”, or “Next Time”.  As long as I do that consistently, there is no need to add “PASS” or “FAIL” to the documented tests.  While debriefing about my tests post session, the assumption will be that the test passed unless indicated otherwise.

Even though it felt like going to work without pants, after a few more sessions, it turns out, not resolving to “PASS” or “FAIL” reduces administrative time and causes no ambiguity during test reviews.  Cool!

Wait. It gets better.

On further analysis, resolving all my tests to “PASS” or “FAIL” may have prevented me from actual testing.  It was influencing me to frame everything as a check.  Real testing does not have to result in “PASS” or “FAIL”.  If I didn’t know what was supposed to happen after editing an element with an unlinked parent (as in the above example), well then it didn’t really “PASS” or “FAIL”, right?  However, I may have learned something important nevertheless, which made the test worth doing…I’m rambling.

The bottom line is, maybe you don’t need to indicate “PASS” or “FAIL”.  Try it.

Categories: Software Testing

Live Online Classes for Software Quality

Randy Rice's Software Testing & Quality - Mon, 08/04/2014 - 10:19
Rice Consulting has teamed with Wind Ridge International Consulting to offer the following courses live, online.


These are all short daily online events, spread over a week. Five sessions, 1.5 hours each. 

To learn more about a class, just click on the title or the "More Info" link. To see the specific dates and times, prices, and to register, click on the "Register" link below each description above. 

Building a High Performance Team
A key role of all successful managers is the building and leading a high performance team (HPT) that provides business value to the organization.  The HPT plays a central role in any software organization. Therefore, they need to have a unique combination of well-developed technical, communication, leadership, and people skills in order to be successful. This class focuses on the unique challenges the software organizations faces in building, leading, and retaining a HPT. More Info...

REGISTER HERE

Establishing the Software Quality Organization

This class describes the activities required to establish a successful software quality organization (SQO).  The SQO is comprised of three distinct quality functions, quality assurance, quality control (testing), and configuration management, that work in harmony and complement each other. More Info...

REGISTER HERE

Fundamentals of Process Improvement

There are two ways to make process improvement changes: random or planned.  You can’t start making improvements until you know your exact current state and determine where you want to go.  This class takes the students through a step-by-step proven process for making planned improvements. More Info... 

REGISTER HERE

Introduction to Configuration Management
This class introduces the attendees to the basics of CM as well as a practical process for establishing and maintaining a CM program that meets the corporate goals and information needs. More Info...  

REGISTER HERE


The instructor for these classes is Thomas Staab. Tom is a frequent advisor to numerous companies and government agencies on the topic of technology management, business/technology interface, resource and process optimization, project management, working with the multi-generational/ multi-cultural workforce, successful outsourcing, and maximizing business value and ROI.
Tom specializes in improving processes, managing projects, and maximizing results with the goal of enabling senior management, business and technology leaders to form a cohesive unit and become even more successful corporate and government leaders.
He quickly gains the full and active support from senior leaders down to the front line managers. His mission is to increased business value and produce significant return-on-investment (ROI) for clients.
Categories: Software Testing

Pages