Software Testing

Like Every Michelin Commercial Ever

QA Hates You - Tue, 01/27/2015 - 04:30

Job posting:

Can improved automobile tyres really make the world a better place? Should we trust developers the one to change the tyres?

To be honest, probably better than me: the last time I changed a tire, I cross-threaded two of the lug nuts and then snapped one of them off (with only a lug wrench, sir; I was motivated to remove that bolt). BECAUSE I BREAK THINGS.

Categories: Software Testing

AWS Aurora – The Game Changer

LoadStorm - Mon, 01/26/2015 - 09:30

Last November, Amazon Web Services (AWS) announced a new database service, codenamed Aurora, which appears to be a real challenger to commercial database systems. AWS will offer this service at a very competitive price, which they claim is one-tenth that of leading commercial database solutions. Aurora has a few drawbacks some of which are temporary, but the many benefits far outweigh them.


With a special configuration they’ve come up with that tightly integrates the database and hardware, Aurora delivers over 500,000 SELECTS/sec and 100,000 updates/sec. That is five times higher than MySQL 5.6 running the same benchmark on the same hardware. This new service will utilize a sophisticated redundancy method of six-way replication across three availability zones (AZs), with continuous backups to AWS Simple Storage Service (S3) to maintain 99.999999999% durability. In the event there is a crash, Aurora is design to almost instantaneously recover and continue to serve your application data by performing an asynchronous recovery on parallel threads. Because of the amount of replication, disk segments are easily repaired from other volumes that make up the cluster. This ensures that the repaired segment is current, which avoids data loss and reduces the odds of needing to perform a point-in-time recovery. In the event that a point-in-time recovery is needed, the S3 backup can restore to any point in the retention period up to the last five minutes. Aurora has survivable cache which means the cache is maintained after a database shutdown or restart so there is no need for the cache to “warm up” from normal database use. It also offers a custom feature to input special SQL commands to simulate database failures for testing.

  • Primary instance – Supports read-write workloads, and performs all of the data modifications to the cluster volume. Each Aurora DB cluster has one primary instance.
  • Aurora Replica – Supports only read operations. Each DB cluster can have up to 15 Aurora Replicas in addition to the primary instance, which supports both read and write workloads. Multiple Aurora Replicas distribute the read workload, and by locating Aurora Replicas in separate AZs you can also increase database availability.

Aurora replicas are very fast. In terms of read scaling, Aurora supports up to 15 replicas with minimal impact on the performance of write operations while MySQL supports up to 5 replicas with a noted impact on the performance of write operations. Aurora automatically uses its replicas as failover targets with no data loss while MySQL replicas can have this done manually with potential data loss.

In terms of storage scalability, I asked AWS some questions about how smoothly Aurora will grant additional storage in the event that an unusually large amount of it is being consumed since they’ve stated it will increment 10GB at a time up to a total of 64TB. I wanted to know where the threshold for the autoscaling was at and if it were possible to push data in faster than it could allocate space. According to the response I received from an AWS representative, Aurora begins with an 80GB volume assigned to the instance and allocates 10GB blocks for autoscaling when needed. The instance has a threshold to maintain at least an eighth of the 80GB volume as available space (this is subject to change). This means whenever the volume reaches 10GB of free space or less, the volume is automatically grown by 80GB. This should provide a seamless experience to Aurora customers since it is unlikely you could add data faster than the system can increment the volume. Also, AWS only charges you for the space you’re actually using, so you don’t need to worry about provisioning additional space.

Aurora also uses write quorums to reduce jitter by sending out 6 writes, and only waiting for 4 to come back. This helps to isolate outliers while remaining unaffected by them, and it keeps your database at a low and consistent latency.


For the time being Aurora is free if you can get into the preview which is like a closed beta test. Once it goes live there are a few things to keep in mind when considering the pricing. The cost of each instance is multiplied by each AZ. Storage is $0.10 a month per 1GB used, and IOs are $0.20 per million requests. Backups to S3 are free up to the current storage being actively used by the database, but historical backups that are not in use will have standard S3 rates applied. Another option is to purchase reserved instances which can save you money if your database has a steady volume of traffic. If your database has highly fluctuating traffic then on-demand instances tend to be the best so you only pay for what you need. For full details, please visit their pricing page.


Currently, the smallest Aurora instances start at db.R3.large and scale up from there. This means once the service goes live there will be no support for smaller instances like they offer for other RDS databases. Tech startups and other small business owners may want to use these more inexpensive instances for testing purposes. So if you want to test out the new Aurora database for free you better try apply for access to the preview going on right now. AWS currently does not offer cross region replicas for Aurora, so all of the AZs are located in Virginia. On the other hand, that does mean that latency is very low.

It only supports InnoDB and any tables from other storage engines are automatically converted to InnoDB. Another drawback is that Aurora does not support multiple tablespaces, but rather one global tablespace. This means features such as compressed or dynamic row format are unavailable, and it affects data migration. For more info about migrations, please visit the AWS documentation page.

Temporary Drawbacks

During the preview, Aurora is only available in the AWS North Virginia data center. The AWS console is the only means for accessing Aurora during the preview, but other methods such as CLI or API access may be added later. Another important thing to note is that during preview, Aurora will not offer support for SSL, but they are planning to add this in the near future (probably before the preview is over). Another feature to be added at a later date is the MySQL 5.6 memcached option, which is a simple key-based cache.


All in all, it sounds amazing, and I for one am very excited to see how it will play out once it moves out of the preview phase. Once it is fully released, we may even do a performance experiment to load test a site that relies on an Aurora DB to see how it holds up. If you’re intrigued enough to try and get into the preview, you can sign up for it here.


The post AWS Aurora – The Game Changer appeared first on LoadStorm.

QA Music: An Old New Song

QA Hates You - Mon, 01/26/2015 - 03:22

Howard Jones “New Song”. Which is thirty-one years old, but never mind that.

Full disclosure: The mime in chains following the rock star around inspired me to get into QA.

Categories: Software Testing

Very Short Blog Posts (22): “That wouldn’t be practical”

DevelopSense - Michael Bolton - Sat, 01/24/2015 - 16:46
I have this conversation rather often. A test manager asks, “We’ve got a development project coming up that is expected to take six months. How do I provide an estimate for how long it will take to test it?” My answer would be “Six months.” Testing begins as soon as someone has an idea for […]
Categories: Software Testing

Can load testing help stop hackers?

LoadStorm - Thu, 01/22/2015 - 09:51

In light of the recent Sony hack, security should be on every web developer’s mind. This cyber attack, which is being called “the worst cyber attack in U.S. history” by Sony’s CEO, is a perfect example of why security is something we all need to take seriously. An enormous amount of personal and financial information was revealed for millions of customers.

As we grow increasingly aware of these occurrence, we as developers need to go forward with the mindset that people will be trying to access our data. As the internet and technology permeates throughout physical stores, our information is becoming even more vulnerable to criminals who use online hacking methods against organizations.

What you can do.

There are numerous ways you can be proactive in protecting your website. One practice that is often overlooked is a combination of penetration and stress testing. Stress testing is the practice of determining how well a website functions in deliberately adverse conditions. Penetration testing is actively trying to break down security methods and access forbidden information. Typical actions in this testing test may include:

  • Running several resource-intensive processes (consuming CPU, memory, disk, network) on the web and database servers at the same time
  • Attempting to access and hack into the system and use it as a zombie to spread spam
  • Doubling the baseline number for concurrent attempts to access a single website as a regular user.
  • Attempting to gain cross-account access to gain information when logged into a system

When you evaluate and benchmark your system in this way, you can observe how your system reacts and recovers. As the array of different protocols and applications grow increasingly complex, malicious attacks can quickly bring down a site or exploit a lack of security.

Security breaches are camouflaged.

A previous Sony data breach that jeopardized 77 million users was actually disguised as a DDoS attack, which is an attack characterized with very overwhelming amounts of traffic. According to a recent study from RSA Security and the Ponemon institute, 64 percent of IT professionals in the retail sector have seen an increase in attacks and fraud attempts during abnormally high traffic periods. By testing your site’s response to a simulated attack, this type of security gap can be reduced, which is a proactive step towards protecting your site.

LoadStorm can be used as a tool to determine your site’s breaking point, and possibly, your site’s performance at its most vulnerable point. By simulating an attack in conjunction with a load test, you can now evaluate how network and security devices perform under stress, and isolate and repair flaws. After determining the weak points of your site, get to work implementing a more secure infrastructure. The idea is to close the gap between the attack and the response to the attack.

Readiness is Key.

Many companies make the mistake of launching before they are truly ready. It’s easy to get caught up in launch deadlines or the pressure to conserve time and resources that could be spent on testing. However, with the diverse competition that exists today, many customers will only give you one shot. If your site or your customer’s data has been compromised, don’t be surprised if they leave and do not return. It takes a lot of work to build trust with new users. Don’t lose the value of your hard work on a vulnerable system. As millions of transactions take place on the internet every day, it’s up to us to make sure that our systems are prepared for an attack; that security provisions like network firewalls, flood controls, intrusion detection and prevention, and application firewalls have all been tested thoroughly with realistic simulated traffic.

It’s up to us to ensure that our sites are ready for high traffic and that our data is secure.

The post Can load testing help stop hackers? appeared first on LoadStorm.

Your Web Site’s Undergarments Are Showing

QA Hates You - Thu, 01/22/2015 - 04:29

Keep in mind your invisible meta tags display when a user shares the link in various social media forms:

In this case, I think we can agree it’s a QA fail.

Categories: Software Testing

Waiting for One of Two Things To Appear on a Page in Java Selenium

QA Hates You - Tue, 01/20/2015 - 03:26

I’ve been working on a Java Selenium test automation project, teaching myself Java and Selenium WebDriver along the way.

I ran into a problem that I didn’t see solved on the Internet elsewhere, but my skill at searching StackOverflow is not that good.

The problem is conducting an operation that might yield one of two results, such as success or an error message. Say you’ve got an registration form or something that can succeed if you put the right data in it or will fail if you put the wrong data in it. If the user successfully registers, a welcome page displays. If the user fails, an error message displays. And, given this is the Web, it might take some time for one or the other to display. And the implicit waits didn’t look like they’d handle branching logic.

So here’s what I did:

public int waitForOne(WebDriver driver, Logger log, String lookFor1, String lookFor2){ WebDriverWait wait = new WebDriverWait(driver, 1); for (Integer iterator = 1; iterator < 60; iterator++){ try{ WebElement element = wait.until(ExpectedConditions.elementToBeClickable(; return 1; }catch (Exception e){ } // try 1 try{ WebElement element = wait.until(ExpectedConditions.elementToBeClickable(; return 2; }catch (Exception e){ } //try 2 } //for loop }// waitForOne

You could even create a longer list of events to wait for one of them to occur by passing in an array of strings and then using a For-Each loop to run through the list.

This sample looks for a Web element by its ID, but you could change it to use another By parameter, such as CSS Selector (cssSelector). Or, if you're feeling dangerous, you could pass in the By parameter as a string and parse it in the method to determine whether to use ID, CSS Selector, or a mix therein. But that's outside the simplicity of this example.

Also note that the For loop that limits the waiting for a total of sixty iterations, which in this case will max out at 120 seconds (at 1 second per attempt for 2 items a maximum of 60 times). You could pass the max in as a parameter when calling this method if you want. That's especially important if you're using a list of possible elements to look for. If you're passing in five elements, suddenly you're at a maximum time of five minutes if it times out completely. You might not want your tests to wait that long, especially if you're using the check multiple times per test.

I'm sure there are more elegant solutions for this. Let's hear them. Because, frankly, I'm not very good at searching StackOverflow, and I'd prefer if you'd just correct my foolishness here in the comments.

Categories: Software Testing

QA Music: I Am Machine

QA Hates You - Mon, 01/19/2015 - 02:57

It’s been a while since we’ve had some Three Days’ Grace, so here is “I Am Machine”:

It’s been a while since we’ve had anything, actually. Perhaps I should post something.

Categories: Software Testing

Optimizing for Mobile Traffic

LoadStorm - Thu, 01/15/2015 - 08:01

Every year mobile traffic is growing. This year 41.2% of all cyber monday traffic came from mobile. It’s a trend that every business should start to plan, and develop for.

Options for Mobile Optimized Experiences

Based on figures provided by the mobile experts at Flurry, app usage is increasing while mobile web surfing is decreasing. With the growing trend of using apps over web browsing on mobile devices, it is a good idea to consider developing an app. However, developing your own app is usually very time consuming and expensive. For many companies, developing their own mobile app is simply not an option.

As an alternative, responsive web design is much less expensive to develop and easy to maintain. When deciding on which to use, consider your customer demographics and the type of product or service that you’re providing. Do you have analytic data that tells you how much mobile traffic you have so far? How many of your customers do you think will benefit from one of these options?

According to a study by the Aberdeen Group, responsive design websites showed a 10.9% increase in visitor to buyer conversion rates year-over-year, while non-responsive websites only had a 2.7% increase. So if you’re not ready to invest in developing a full-blown mobile app, the evidence supports responsive web design as a cost effective alternative to a mobile app.

What is responsive design?

For a quick example, click and drag your browser window from full size to very small. You will notice how this website changes based upon the screen size the viewer is using. The goal of responsive web design is to give the customer an experience that is quick and easy to use on whatever size device he or she is using.

Responsive design is a combination of three techniques: fluid grids, flexible images, and media queries. Here is an excellent video tutorial to learn the fundamentals of responsive design:

Responsive Design and Web Performance

During this video tutorial, the presenter mentions that the images in his test site are fairly large, which allows them to be responsive, but being responsive is not enough. If you really want to cater to your mobile audience, you also need to improve performance by considering the impact quality and file size have. Smashing Magazine has a wonderful article to help guide you in making your site both responsive and a great performer over mobile broadband.

Test Your Site

To see how responsive your site is currently, you can visit Matt Kersley’s page.

Or for a more detailed option in testing responsive design, you can use the developer tools in most browsers. This allows you to change the viewing experience so you can select one of many brand name devices, like an iPhone 6 or Galaxy S4, to emulate. To do this in Chrome, you can open your developer tools by right-clicking a page and choosing “inspect element”. Then look to the top left of the tools window for a small icon that looks like a smartphone. Click the icon and the browser view will constrain the viewing area.

This developer tool also shows how easy it is to load test your mobile site using LoadStorm PRO. A recording can be made of the site while using the mobile view, which can then be uploaded and utilized in a load test.

Sidenote for SEO:

Google prefers that developers use responsive design mostly for requiring less resources and is future friendly. However, Google does not rank responsive design higher over other configurations.


The post Optimizing for Mobile Traffic appeared first on LoadStorm.

Bibliographies, Books and Stuff

Alan Page - Wed, 01/14/2015 - 11:22

A while back (no link cause I’m lazy), I mentioned that one thing (among many) missing from How We Test SW at MS was a bibliography of testing books. While we didn’t refer directly to any books when writing the book, the influence of several testing books is definitely there, and worth noting for those who want to study where the craft has come from (which is one way to better understand where it’s going). This post will now serve as the official bibliography of the 2008 edition of How We Test Software at Microsoft. That should get at least one nagger off my back.

But First…

My bookshelf is filled with four different types of books (technically, three – one type is a subset of another.

  1. Business-ish books. I have (and read) a lot of books on leadership, innovation, etc.. I think these are important for all leaders and those who want to influence (definitely not just managers). These books contain the most read and most loved books in my collection – but none of them are included in this list.
  2. Books on thinking. I also read a lot of books about thinking and motivation. Examples include Michalko’s Cracking Creativity and Thinkertoys, Kahneman’s Thinking Fast and Slow, Pink’s Drive, and A Whole New Mind, Weinberg’s General Systems Thinking, and Chabris’s The Invisible Gorilla . I think books like these are a great resource for anyone who does knowledge work. A good way to push my buttons (as many of you know) is to take a nugget from a book like these (the gorilla is, of course, my favorite), and announce to the world, “OMG!!! THIS IZ JUSST LIK TESTING!!!!!!!!” These books aren’t in this blog-iography either.
  3. Software books. The final category is books about software engineering. Note that there are examples of software books that overlap into the above categories.

Please don’t ask where book “Foo” fits into the above categories. The category list above is arbitrary and I just made it up. I may change it later (or by the time I post this).

Category #3 above also includes books on software testing – and after an introduction that is now likely longer than I expect the list of testing books and notes to be, let’s get started. Below is a list of the software testing books that I believe have contributed in some way to my test education (and hence, my own teaching and the writing of HWTSAM), as well as directly and indirectly to many of my peers. In no particular order (other than the order you can find them on the shelf in my office from right to left):

  1. Software Testing Techniques by Boris Beizer. This book has a comprehensive overview of a whole lot of testing techniques. Some of them are good, some don’t apply, and some are complete bullshit. I tell a story about my journey with software testing books where I say, “When I read my first book on software testing, I thought I learned it all. After reading a second book on testing, I was confused. After reading a third testing book, I began to form my own opinions. For me, I don’t think it’s fair to determine where formal models apply and don’t apply until I really understood them – and this book gave me that background.
  2. Software Testing: A Craftsman’s Approach, by Paul Jorgensen. This is another book in the “comprehensive” category, and deals entirely with software testing techniques. Again, it’s an exercise for the reader to figure out the best ways and contexts to apply the techniques (and to filter out the bullshit), but as far as basics of testing goes, it’s one of my favorites. I know that Bj was influenced heavily by Jorgensen’s work, and I think it shows in his chapters on functional techniques.
  3.  A Practitioner’s Guide to Software Test Design, by Lee Copeland. I hate to say this, because Lee occasionally reads this blog, but this is my absolute favorite book on the basics of Test Design. For me, it was so much easier to read than the Jorgensen and Beizer books (and the even Meyers book I’m about to mention), and I think it’s much easier for the reader to take the information in the book and apply it to their own work. I don’t recommend reading just one book on test design (see my comments on the Beizer book), but this is one I think is worth having on your shelf.
  4. The Art of Software Testing by Glenford Myers. This one is sort of the classic of software testing. It too, focuses on functional techniques (and introduces the classic triangle testing problem). Note that this book Ii s generally way overpriced ($130 on Amazon right now)– look for a used copy if you want to own it.
  5. Testing Object-Oriented Systems, by Bob Binder. I probably  mentioned this book in the chapter on model-based testing. This is the book where I first studied MBT, and also the book that got me thinking deeply about test patterns and patterns in general (I was inspired by this book to finally read the GOF book, as well as Alexander). Note that this book can double as a weapon. It’s just under a zillion pages long, so I recommend reading it in chunks rather than in a single three-week sitting.
  6. Software Testing Techniques by Loveland and Miller. In a way, this book could be called How We Test Software at IBM – or, perhaps, Things you can learn from Mainframe Testers. This book came out just a few years before we began writing HWTSAM, and probably subconsciously influenced my decision to agree to think about possibly writing a book about testing at Microsoft. There are a lot of lessons to be learned from this book, and although it’s applicability to my world of testing has waned quite a bit in the last decade, there are definitely some nuggets here for anyone.
  7. This is the placeholder for something obvious I’ve missed. I’ve read dozens of other books on software testing, and unless someone has borrowed something significant without asking, I have to say that those are the most likely books that influenced my world view of testing – and likely what we drew from (at least indirectly) when writing HWTSAM.
    Well… those six books and a combined half a century of practical experience from the authors, and a millennia of experience from the combined contributors.
(potentially) related posts:
  1. Books I’m about to (re)read
  2. It’s a big world (of testing) out there
  3. Happy Birthday HWTSAM
Categories: Software Testing

Testing on the Toilet: Prefer Testing Public APIs Over Implementation-Detail Classes

Google Testing Blog - Wed, 01/14/2015 - 10:35
by Andrew Trenk

This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office.

Does this class need to have tests?
class UserInfoValidator {
public void validate(UserInfo info) {
if (info.getDateOfBirth().isInFuture()) { throw new ValidationException()); }
}Its method has some logic, so it may be good idea to test it. But what if its only user looks like this?
public class UserInfoService {
private UserInfoValidator validator;
public void save(UserInfo info) {
validator.validate(info); // Throw an exception if the value is invalid.
}The answer is: it probably doesn’t need tests, since all paths can be tested through UserInfoService. The key distinction is that the class is an implementation detail, not a public API.

A public API can be called by any number of users, who can pass in any possible combination of inputs to its methods. You want to make sure these are well-tested, which ensures users won’t see issues when they use the API. Examples of public APIs include classes that are used in a different part of a codebase (e.g., a server-side class that’s used by the client-side) and common utility classes that are used throughout a codebase.

An implementation-detail class exists only to support public APIs and is called by a very limited number of users (often only one). These classes can sometimes be tested indirectly by testing the public APIs that use them.

Testing implementation-detail classes is still useful in many cases, such as if the class is complex or if the tests would be difficult to write for the public API class. When you do test them, they often don’t need to be tested in as much depth as a public API, since some inputs may never be passed into their methods (in the above code sample, if UserInfoService ensured that UserInfo were never null, then it wouldn’t be useful to test what happens when null is passed as an argument to UserInfoValidator.validate, since it would never happen).

Implementation-detail classes can sometimes be thought of as private methods that happen to be in a separate class, since you typically don’t want to test private methods directly either. You should also try to restrict the visibility of implementation-detail classes, such as by making them package-private in Java.

Testing implementation-detail classes too often leads to a couple problems:

- Code is harder to maintain since you need to update tests more often, such as when changing a method signature of an implementation-detail class or even when doing a refactoring. If testing is done only through public APIs, these changes wouldn’t affect the tests at all.

- If you test a behavior only through an implementation-detail class, you may get false confidence in your code, since the same code path may not work properly when exercised through the public API. You also have to be more careful when refactoring, since it can be harder to ensure that all the behavior of the public API will be preserved if not all paths are tested through the public API.
Categories: Software Testing

Taking Severity Seriously

DevelopSense - Michael Bolton - Wed, 01/14/2015 - 03:10
There’s a flaw in the way most organizations classify the severity of a bug. Here’s an example from the Elementool Web site (as of 14 January, 2015); I’m sure you’ve seen something like it: Critical: The bug causes a failure of the complete software system, subsystem or a program within the system. High: The bug […]
Categories: Software Testing

WordPress Hosting Providers Study: Web Performance & Scalability

LoadStorm - Mon, 01/05/2015 - 12:14
Click to Zoom


When it comes to web performance, study after study has proven: fast and scalable wins the race. But with thousands of WordPress hosting providers, how do you know which one is fast and scalable?

That is where comes in. Their business is all about helping people identify which hosting provider is the best choice for them. Kevin Ohashi from ReviewSignal has been working with LoadStorm to run a series of load tests on some of the top WordPress hosting providers to determine which is the best for companies who need scalable websites.

Our performance engineers have teamed up with Kevin to analyze the multitude of data and provide this report of the top WordPress hosting providers for web performance. Providers included in this study are: A Small Orange, FlyWheel, GoDaddy, Kinsta, LightningBase, MediaTemple, Nexcess, Pagely, Pantheon, and WebSynthesis. These providers were included in the 2,000 user load test because they didn’t struggle with the first test of 1,000 concurrent users.

This analysis only looks at the final load test of 2,000 concurrent users, but Kevin’s article analyzes the results of both tests and looks at long term up-time reliability. Check out Review Signal’s report of the full study here.


All tests were performed on identical WordPress dummy websites hosted on 10 different hosting services. All sites were tested with the same plugins except in cases where hosts added extra plugins. The websites had identical scripts that included browsing and login. The load tests were run in LoadStorm PRO for 30 minutes with a linear 20 minute ramp up from 500 to 2,000 virtual users and holding at the peak for the for the remaining 10 minutes.


In order to rank the top providers, we have broken our analysis down by the key web performance metrics:

  • Error Rate
  • Average Response Time
  • Peak Response Time
  • Average Page Completion
  • Throughput

To fairly rank the top providers, we ranked each provider by each performance metric at the 20 minute mark in the test, when all sites were under full load of 2,000 users. For each metric, the providers were ranked (1st through 10th) according to their performance and then a point value was assigned to each. Then we determined our final ranking position based on their total score, the sum of all points from all the metrics.

Test Data:

To view the full test results with interactive graphs in LoadStorm PRO, click on each hosting provider below:

Metrics: Error Rate

Error rate is probably the most important metric for businesses wanting to be certain that a website won’t crash under high traffic. High error rates mean one thing: Lost customers.

Surprisingly, we had a 7-way tie for first place with 0% error rates. Overall, this speaks volumes to the scalability of all the websites included in the study. Flywheel started to fail at around 1500 concurrent users and began returning 502 errors, which explains its high error rate.

Average Response Time

Average Response Time is very significant because it directly affects the user experience and perceived load time. This metric measures the time each request takes “round trip” from the browser sending the request to the server, the server processing the request, and then the response from the server back to the browser. The Average Response Time takes into consideration every round trip request/response cycle for that minute interval and calculates the mathematical mean of all response times.

Peak Response Time

This metric also measures the same “round trip” that the Average Response Time does, but instead of averaging the time for all requests, Peak Response Time is simply the single longest (slowest) time for a single request.

Average Page Completion

Average Page Completion Time is a metric that measures the amount of time from the start of the first request to the end of the final request on a page.

In regards to the specific times in this study, the test shows unusually fast Average Page Completion times. After investigating why the pages were loading so quickly, it turns out that some of the pages on the dummy website were very simple with very few requests each. While users with real websites on these providers would expect to see slower average page completion times, the tests are still valid because all providers had the same simple pages.


Throughput is measured by the number of kilobytes per second that is being transferred. This measurement shows how data is flowing back and forth from the server(s). High throughput is a mark of good web performance under load because it shows that there aren’t any bottlenecks blocking and slowing the data transfer. Low throughput, as seen in WebSynthesis, signifies that the server is overwhelmed and is struggling to pass data to and from the server.

Interestingly, GoDaddy pushed triple the amount of data through because their admin screen had more resources being loaded. Which is why the average throughput is so high. Despite the extra data to process, they still had significantly higher average response times than most of the other providers. Anytime a site makes more requests, it slows down performance. Therefore, without so much extra data it is fair to say that GoDaddy could have possibly been faster than all the others.


From the final point tallies, we can see that there are three clear sections.
Top Performers: Pantheon, MediaTemple, GoDaddy, and Kinsta.
Good Performers: Nexcess, LightningBase, A Small Orange, and Pagely.
Fair Performers:: FlyWheel and WebSynthesis.


Overall, most of the providers did surprisingly well under the full load of 2,000 concurrent users. Even though we wanted to rank them in a definitive order, the fact of it is that most providers did not reach failure rates at all in the test. So while we were able to rank them, there were several metrics where the difference between points was negligible (ie: 1 ms average response time difference between GoDaddy and Kinsta) but still calculated in our scores.

Additionally, the test utilized in our report is only part of the full ReviewSignal study. ReviewSignal ran tests at 1,000 users and the providers that crashed were not included in the tests at 2,000. Therefore, all of the providers included in this ranking should be considered great choices for scalable WordPress hosting.

This level of high performance in all 10 providers was unexpected with such a heavy load and we were very impressed by the results.

The post WordPress Hosting Providers Study: Web Performance & Scalability appeared first on LoadStorm.

My year of working sparsely

Alan Page - Mon, 01/05/2015 - 10:31
Warning, non-software post follows.

2014 was an interesting year for me. The year began with the trailing two weeks of a four week visit to Australia. Then, I spent July and August in France, then took another week off when I returned to take care of my dad. Finally, I took the last two weeks of the year off to be with family. With other short vacation days thrown in, I took nearly 13 weeks off in 2014 (if my math is right, that’s a quarter of a year). Most of that time off came from an award I received several years ago that allows me a one-time eight-week vacation (a sabbatical of sorts), and I also used just about all of the five weeks of regular vacation time that Microsoft gives me.

I mentioned the above (in much shorter form) on twitter this week. In the discussion, I mentioned that I’ve been at MS for nearly 19+ years, and it’s a once in a career sort of thing, and Geir Gulbrandsen asked if I’d do an encore after an another 19 years. I answered that I’d likely be in some stage of retirement by then, and that led to a discussion on what retirement is…and it made me want to dump my thoughts into a form longer than 140 characters.

So here goes.

A lot of twitter folks said things like, “If you do what you love, do it forever” or, “I’ll never stop working if I enjoy the job”. I agree with these folks. Even in “retirement”, my dad worked for years just to keep from getting bored. I get bored easily too, and I really like my field of work, so why would I even think about retirement (in any form)?

Because I like things other than working too. I like to travel (actually, I don’t like the travelling part – I like being in other countries). Spending two months in a foreign country (or a month each in two different countries) isn’t something someone can do often when working a full time job. Or, I may write another book someday – but I know from past experience that I didn’t like the schedule I had to keep to work full time while writing. I may take a class. I may want to work in a winery. Who knows. All I know is that at some point in my life (and probably within 10 years or so), I’ll want to work much less than I’m working now. Somewhere in the twitter stream, I said (tweeted) this:

For me, “retirement” isn’t a stoppage of work. It’s a choice to work when I want, on what I want, and as frequently as I want.

— Alan Page (@alanpage) December 31, 2014

I’ve known that I wanted “retirement” options from the time I entered the workforce (as a middle school music teacher when I was 22 years old!). I’ve saved aggressively – maxing out my pre-tax 401k for the last 19 years, and contributing as much as I can afford to post-tax 401k over the past several years. I feed my IRA every year and save whatever else I can. The only debt I’ve had for the last two decades is a mortgage. And, although we went a little crazy with vacations over the last year, we travel cheaply, and save as much as we can while still enjoying life. To be fair, I’ve been successful working for a large company, so my income level enables me to save more than a lot of people, but the point is that I’ve tried to save what I can, so at some point, I have a choice to work – or not, as I choose.

Any sort of retirement for me is still quite a few years off. My kids are in school for another eight years, and will probably go to college after that (I started 529 plans for both of them when they were born, and I expect I’ll have enough saved to cover a large portion of costs for all but the most elite schools). The point is that I think a lot about what I need to do to get myself to a point where I can cover most or all of my expenses without needing to work.

But I’ll still work when I want to.

(potentially) related posts:
  1. Twenty-Ten – my best year ever
  2. 2011 year end roundup
  3. Another year bites the dust
Categories: Software Testing

Introducing Bug Magnet – an exploratory testing helper

The Quest for Software++ - Sat, 01/03/2015 - 09:49

I just compiled a bunch of checklists and notes I often use for exploratory testing into a handy Chrome extension. Bug Magnet provides convenient access to common problematic values and edge cases, so you can keep them handy and access them easily during exploratory testing sessions. Just right-click an input field!

The extension introduces only a tiny overhead per page (<1k), and has no third party library dependencies, works completely passive, so it does not interfere with your web app execution in any way. It works on input fields, text areas, content editable DIVs. Of course, it’s completely opensource (MIT license), so you can easily extend it with your config.

To install the extension, and for more info, head over to

The extension started from a need to scratch my own itch and make common experiments easily accessible, both as test data and as an inspiration for similar ideas. If this sounds useful, propose improvements

What Mythbusters Can Teach Software Testers

Randy Rice's Software Testing & Quality - Fri, 01/02/2015 - 15:12
I hope you have all had a great Christmas and New Year. Normally at this time of year I write a posting about goal setting, reflecting on the past year for lessons learned or something similar, which are still important things to do.

I thought for this year, let's go for "Now for something completely different."

Between all the bowl games and Christmas movies on TV, the Science Channel has been showing a Mythbusters marathon for last couple of weeks. That, my friends, is a lot of episodes - more than I could ever take in. But I have enjoyed watching a few of the episodes.

I often refer to the Mythbusters show when explaining session-based testing in some of my software testing classes and workshops. Mythbusters is a perfect example of how to set-up a test, perform it, then evaluate the results.

However, in watching several episodes back-to-back, some clear trends emerged that I have not previously related to what we do as software testers.


You don't see the written plans, but believe me, they are there. Of course, you have the episode script, but there is much more going on behind the scenes. The crew has to get all the resources in place for the show. Some of the resources get expensive. In one episode I counted eight cars destroyed. Then you have explosives, the permissions to use the explosives, chemicals, the permission to use dangerous chemicals, special cameras and other devices. The list could go on, but the point is that those items, just like the items you need for testing, do not just magically appear. It takes planning.


It is impressive how each of the Mythbusters collaborate and get along. They may disagree on some things, but each one is committed to letting the test tell the story. They ask for input and feedback and it's refreshing to not see the drama that plays out in some of the "reality" shows.

Expert Consultation

When it comes to things beyond their scope of knowledge, the Mythbusters bring in a subject matter expert to evaluate the test results.


The Mythbusters are fanatics which it comes to measuring what they do. Not only do they measure things, but they are careful to take good baseline and control measurements BEFORE they perform the test. If the Mythbusters can measure how much pressure it takes to get candy away from a baby, we can measure what we do in testing.


The Mythbusters allocate certain amounts of time to their tests. That allows them to estimate the time needed for the entire test. The time for each of the tasks varies due to the complexity of the test. Setting the scope of the test is very important. The test is focused on ONE thing, not several. That allows them to stay right on point and on time.

It appeared to me the total test time broke into:
  • Planning - They discuss the urban legend to be busted or confirmed and how they plan to conduct the tests.
  • Set-up - This often takes up the largest percentage of time, depending on the specific test. In some cases, they go to very elaborate means to create the closest conditions possible to the purported occurrence of the original (supposedly) event. In one show, the crew built a jail cell out of cinder blocks, poured concrete in the inner space and reinforced it with rebar. That alone would have taken multiple days.
  • Performance - The tests themselves can go quickly, but the trick is that many of the tests are repeated either to confirm the findings or to get a large enough sample size. In fact, one impressive observation is that they will try a test several different ways. They might start by using just a model, then by using the real thing, such as car, motorcycle, etc.
  • Evaluation - This often goes quickly because the test has been so clearly defined, measured and documented.
  • Tear-down - This is the part we don't see, but in the case of the mock jail cell, I could see it taking a while.
I think a starting point for testers is to measure as a percentage of effort your testing tasks require, based on varying levels of test complexity.

Expected Results

The Mythbusters define what they expect to see either confirmed or busted. In fact, they know what constitutes "confirmed" or "busted" and what the measures must be to support the findings. It's at this point the viewer gets the clear impression that these guys practice science. Personally, I also practice testing as a science, not a guessing game or a diversion. However, the popularity of the show is that they have made science (and testing urban legends) fun and entertaining. I wonder if we could do that in testing?

No Boredom

This is the challenge for testers. We get to blow stuff up also, but not in a physical sense. Let's face it, a system crash is not as exciting as a car crash (Unless the system you are testing operates a car, but we won't go there...). So, how do we make software testing exciting and perhaps even entertaining?

By the way, if you haven't seen my video of testing a mobile phone radar app during a tornado, that was exciting! But, like the Mythbusters say "Don't try this at home."

The best answer I can give for this is to start thinking about testing software like the Mythbusters think about their tests. Create a challenge and build some excitement about confirming or busting ways that the software should work. You may not get to see a literal explosion, but you may make some people's heads almost explode in frustration when they see that really bad defect (I mean showstopper, dead-in-the-water, kind of defect) the day before the release is supposed to go out.

Some companies hold "bug bashes," but in my experience, most of the problems reported are either superficial or not actually a problem. The issues are often misunderstand of how the application should work. In one case, we held a bug bash at a company with customer service reps performing the tests. Over 200 issues were logged over a 4-hour period. 3 of the issues were actually confirmed defects!

Contests are fun, but be careful. Don't make the contest about who finds the most bugs. You will get a lot of bug reports and some of the better testing may be the tests that find few, but important, defects.

Perhaps a contest for the "Bug of the Week" or "Most Humorous Bug" (Hint: error messages are great places to look for humor) might be interesting. I would love to hear your ideas on this. I would also love to hear any things you or your company does to make testing exciting.

How's that for your New Year's challenge?

Now, go bust some of your own software myths!
Categories: Software Testing