Software Testing

Data Testing vs. Functional Testing

Eric Jacobson's Software Testing Blog - Wed, 07/02/2014 - 12:32

So, you’ve got a green thumb.  You’ve been growing houseplants your whole life.  Now try to grow an orchid.  What you’ve learned about houseplants has taught you very little about orchids.

  • Put one in soil and you’ll kill it (orchids grow on rocks or bark). 
  • Orchids need about 20 degrees Fahrenheit difference between day and night.
  • Orchids need wind and humidity to strive.
  • Orchids need indirect sunlight.  Lots of it.  But put them in the sun and they’ll burn.
  • Fading flowers does not mean your orchid is dying (orchids bloom in cycles).

So, you’re a skilled tester.  You’ve been testing functional applications with user interfaces your whole career.  Now try to test a data warehouse.  What you’ve learned about functionality testing has taught you very little about data testing.

  • “Acting like a user”, will not get you far.  Efficient data testing does not involve a UI and depends little on other interfaces.  There are no buttons to click or text boxes to interrogate during a massive data quality investigation.
  • Lack of technical skills will kill you.  Interacting with a DB requires DB Language skills (e.g., TSQL).  Testing millions of lines of data requires coding skills to enlist the help of machine-aided-exploratory-testing.
  • Checking the health of your data warehouse prior to deployments probably requires automated checks.
  • For functional testing, executing shallow tests first to cover breadth, then deep tests later is normally a good approach.  In data testing, the opposite may be true.
  • If you are skilled at writing bug reports with detailed repro steps, this skill may hinder your effectiveness at communicating data warehouse bugs, where repro steps may not be important.
  • If you are used to getting by as a tester, not reading books about the architecture or technology of your system-under-test, you may fail at data warehouse testing.  In order to design valuable tests, a tester will need to study data warehouses until they grok concepts like Inferred Members, Junk Dimensions, Partitioning, Null handling, 3NF, grain, and Rapidly Changing Monster Dimensions.

Testers, let’s respect the differences in the projects we test, and grow our skills accordingly.  Please don’t use a one-size-fits-all approach.

Categories: Software Testing

GTAC is Almost Here!

Google Testing Blog - Tue, 07/01/2014 - 13:28
by The GTAC Committee

GTAC is just around the corner, and we’re all very busy and excited. I know we say this every year, but this is going to be the best GTAC ever! We have updated the GTAC site with important details:

If you are on the attendance list, we’ll see you on April 23rd. If not, check out the Live Stream page where you can watch the conference live and can get involved in Q&A after each talk. Perhaps your team can gather in a conference room and attend remotely.

Categories: Software Testing

GTAC 2013 Wrap-up

Google Testing Blog - Tue, 07/01/2014 - 13:27
by The GTAC Committee

The Google Test Automation Conference (GTAC) was held last week in NYC on April 23rd & 24th. The theme for this year's conference was focused on Mobile and Media. We were fortunate to have a cross section of attendees and presenters from industry and academia. This year’s talks focused on trends we are seeing in industry combined with compelling talks on tools and infrastructure that can have a direct impact on our products. We believe we achieved a conference that was focused for engineers by engineers. GTAC 2013 demonstrated that there is a strong trend toward the emergence of test engineering as a computer science discipline across companies and academia alike.

All of the slides, video recordings, and photos are now available on the GTAC site. Thank you to all the speakers and attendees who made this event spectacular. We are already looking forward to the next GTAC. If you have suggestions for next year’s location or theme, please comment on this post. To receive GTAC updates, subscribe to the Google Testing Blog.

Here are some responses to GTAC 2013:

“My first GTAC, and one of the best conferences of any kind I've ever been to. The talks were consistently great and the chance to interact with so many experts from all over the map was priceless.” - Gareth Bowles, Netflix

“Adding my own thanks as a speaker (and consumer of the material, I learned a lot from the other speakers) -- this was amazingly well run, and had facilities that I've seen many larger conferences not provide. I got everything I wanted from attending and more!” - James Waldrop, Twitter

“This was a wonderful conference. I learned so much in two days and met some great people. Can't wait to get back to Denver and use all this newly acquired knowledge!” - Crystal Preston-Watson, Ping Identity

“GTAC is hands down the smoothest conference/event I've attended. Well done to Google and all involved.” - Alister Scott, ThoughtWorks

“Thanks and compliments for an amazingly brain activity spurring event. I returned very inspired. First day back at work and the first thing I am doing is looking into improving our build automation and speed (1 min is too long. We are not building that much, groovy is dynamic).” - Irina Muchnik, Zynx Health

Categories: Software Testing

ThreadSanitizer: Slaughtering Data Races

Google Testing Blog - Mon, 06/30/2014 - 15:30
by Dmitry Vyukov, Synchronization Lookout, Google, Moscow


I work in the Dynamic Testing Tools team at Google. Our team develops tools like AddressSanitizer, MemorySanitizer and ThreadSanitizer which find various kinds of bugs. In this blog post I want to tell you about ThreadSanitizer, a fast data race detector for C++ and Go programs.

First of all, what is a data race? A data race occurs when two threads access the same variable concurrently, and at least one of the accesses attempts is a write. Most programming languages provide very weak guarantees, or no guarantees at all, for programs with data races. For example, in C++ absolutely any data race renders the behavior of the whole program as completely undefined (yes, it can suddenly format the hard drive). Data races are common in concurrent programs, and they are notoriously hard to debug and localize. A typical manifestation of a data race is when a program occasionally crashes with obscure symptoms, the symptoms are different each time and do not point to any particular place in the source code. Such bugs can take several months of debugging without particular success, since typical debugging techniques do not work. Fortunately, ThreadSanitizer can catch most data races in the blink of an eye. See Chromium issue 15577 for an example of such a data race and issue 18488 for the resolution.

Due to the complex nature of bugs caught by ThreadSanitizer, we don't suggest waiting until product release validation to use the tool. For example, in Google, we've made our tools easily accessible to programmers during development, so that anyone can use the tool for testing if they suspect that new code might introduce a race. For both Chromium and Google internal server codebase, we run unit tests that use the tool continuously. This catches many regressions instantly. The Chromium project has recently started using ThreadSanitizer on ClusterFuzz, a large scale fuzzing system. Finally, some teams also set up periodic end-to-end testing with ThreadSanitizer under a realistic workload, which proves to be extremely valuable. When races are found by the tool, our team has zero tolerance for races and does not consider any race to be benign, as even the most benign races can lead to memory corruption.

Our tools are dynamic (as opposed to static tools). This means that they do not merely "look" at the code and try to surmise where bugs can be; instead they they instrument the binary at build time and then analyze dynamic behavior of the program to catch it red-handed. This approach has its pros and cons. On one hand, the tool does not have any false positives, thus it does not bother a developer with something that is not a bug. On the other hand, in order to catch a bug, the test must expose a bug -- the racing data access attempts must be executed in different threads. This requires writing good multi-threaded tests and makes end-to-end testing especially effective.

As a bonus, ThreadSanitizer finds some other types of bugs: thread leaks, deadlocks, incorrect uses of mutexes, malloc calls in signal handlers, and more. It also natively understands atomic operations and thus can find bugs in lock-free algorithms (see e.g. this bug in the V8 concurrent garbage collector).

The tool is supported by both Clang and GCC compilers (only on Linux/Intel64). Using it is very simple: you just need to add a -fsanitize=thread flag during compilation and linking. For Go programs, you simply need to add a -race flag to the go tool (supported on Linux, Mac and Windows).

Interestingly, after integrating the tool into compilers, we've found some bugs in the compilers themselves. For example, LLVM was illegally widening stores, which can introduce very harmful data races into otherwise correct programs. And GCC was injecting unsafe code for initialization of function static variables. Among our other trophies are more than a thousand bugs in Chromium, Firefox, the Go standard library, WebRTC, OpenSSL, and of course in our internal projects.

So what are you waiting for? You know what to do!
Categories: Software Testing

Software Testing Careers - Should I Stay or Should I Go?

Randy Rice's Software Testing & Quality - Fri, 06/27/2014 - 13:10
At the recent Better Software Conference, I really enjoyed a presentation by James Whittaker in which he gave some keys to succeeding as a tester, or any career, for that matter. The short list is:
1. Ambition
2. Passion
3. Specialization
4. Imitation of mentors
5. Re-invention
6. Clairvoyance - Keep abreast of the industry. Read!
7. Leadership
8. Find your "super powers" (Things you do really, really well)

That got me thinking about this topic again, because my passion is to help people have better lives, not just be better testers. If you are a great tester stuck in a bad company, working for a clueless boss, etc., then you will be miserable. That leads to an unhappy life everywhere outside of work. Believe me, I know.

It also caused me to rethink things I have taught in the past - especially the idea that you can thrive even in a dysfunctional environment.

Some of you reading this are fortunate to be in a great company, working for a great manager and most days are sunny. However, I talk with thousands of testers every year and I think the great majority of them are in frustrating situations.

One of my most popular "lightning talks" is "Ten Proven Ways to Demotivate Your Team." I included those slides as an opener for my keynote at StarEast 2014. Instead of being funny, I think it evoked serious feelings. I guess that's how stand-up comedy works - sometimes the material works and sometimes, it bombs.

After the StarEast session, I had several people pull me aside to ask me how to deal with their dysfunctional workplace issues. So I decided to write here about some ways to survive terrible workplaces and go on to thrive in your career.

I really thought about these points and got feedback from others in the QA and test profession. Their consensus was the same as mine.

1. The main thing is that you must see yourself as a free agent. The idea that a company will take care of you until retirement died about 25 years ago. You have to develop yourself, invest in yourself, and be prepared to work in many situations - like a sports player or a musician can go from team to team or band to band. Unfortunately, corporations have destroyed trust by outsourcing and reorganizations that have devalued people and the value they bring to the company.

2. It's one thing to have a bad manager, but another to be in a company that has a terrible culture. The problem is that even if your manager leaves or gets fired, you are still in the same cesspool as before. No matter how the team structure changes, you are still in a no-win situation. Even if you became the new manager, you will be up against the same people who make the same bad, misguided policies as before.

3. You are responsible for developing your own skills. It's sad, but too many companies don't want to invest in their people when it comes to training or skill building. They are a) tight-fisted with money and b) afraid that people will benefit from the training by leaving the company afterwards. This just shows that management knows the culture is bad, but they don't care enough to fix it. Or, they don't want to take the risk to take a few positive steps, like training.

4. You have to know "when to hold them, know when to fold them." Leaving a job has risks, just as taking a job has risks. Try your best to do your home work by talking to current employees you may have access to. It's not a sign of failure to leave a job. In fact, sometimes that's the only way to get ahead. Just be careful to not appear as a "job hopper." That can be a career killer. When explaining to a prospective employer why you are considering a new position, you want to stay on the positive side. It's risky to be critical of your present situation. You can say that you feel that your skills and talents are being underused and you feel like you can make a greater contribution someplace else. Back when I had a "real job" I took the high road and thanked the employer I was leaving. That paid off.

I think I have a nomination for the worst treatment ever. It happened recently to one of my sons who had just taken a new job and got married. The company gave him an expensive gift - worth about $300. While on his honeymoon, they fired him (He didn't find out until he opened his mail upon returning home). Then, they deducted the cost of the gift from his final paycheck.

There are no perfect answers on this issue. You have to carefully consider the options, but just keep in mind that toxic environments will eventually wear you down. As much as we like to say we can separate work life from personal life, it's practically impossible to do because we are emotional beings.

Do yourself a favor and build a career, not a job.

I hope this helps and I would love to hear your comments.


Categories: Software Testing

I Know It’s Like Training Wheels, But….

QA Hates You - Thu, 06/26/2014 - 03:58

I know this is just a simple trick that marks me as a beginner, but I like to add a comment at the end of a block to indicate what block of code is ending.


} // if button displayed }catch (NoSuchElementException e){ buttonState = "not present"; } // try/catch return buttonState; } // check button } //class


end # until end # wait for browser end # class end

Sure, an IDE will show me, sometimes faintly, what block a bracket closes, but I prefer this clearer indication which is easier to see.

Categories: Software Testing

Who Cares About Test Results?

Eric Jacobson's Software Testing Blog - Wed, 06/25/2014 - 12:28

I think it’s only people who experience bugs.

Sadly, devs, BAs, other testers, stakeholders, QA managers, directors, etc. seldom appear interested in the fruits of our labor.  The big exception is when any of these people experience a bug, downstream of our test efforts.

“Hey, did you test this?  Did it pass?  It’s not working when I try it.”

Despite the disinterest, us testers spend a lot of effort standing up ways to report test results.  Whether it be elaborate pass/fail charts or low-tech information-radiators on public whiteboards, we do our best.  I’ve put lots of energy into coaching my testers to give better test reports but I often second guess this…wondering how beneficial the skill is.

Why isn’t anyone listening?  These are some reasons I can think of:

  • Testers have done such a poor job of communicating test results, in the past, that people don’t find the results valuable.
  • Testers have done such a poor job of testing, that people don’t find the results valuable.
  • People are mainly interested in completing their own work.  They assume all is well with their product until a bug report shows up.
  • Testing is really difficult to summarize.  Testers haven't found an effective way of doing this.
  • Testing is really difficult to summarize.  Potentially interested parties don’t want to take the time to understand the results.
  • People think testers are quality cops instead of quality investigators; People will wait for the cops to knock on their door to deliver bad news.
  • Everyone else did their own testing and already know the results.
  • Test results aren’t important.  They have no apparent bearing on success or failure of a product.
Categories: Software Testing

In Other Words in Other Places

QA Hates You - Wed, 06/25/2014 - 05:15

Now on StickyMinds: Picture Imperfect: Methods for Testing How Your App Handles Images.

It’s a list of dirty tricks but without the snark.

Categories: Software Testing

Once The Users Smell Blood…

Eric Jacobson's Software Testing Blog - Tue, 06/24/2014 - 11:19

We had a relatively disastrous prod deployment last week.  Four bugs, caused by a large refactor, were missed in test.  But here’s the weirder part, along with those four bugs, the users started reporting previously existing functionality as new bugs, and in some cases, convincing us to do emergency patches to change said previously existing functionality.

It seems bugs beget bugs.

Apparently the shock of these initial four bugs created a priming effect, which resulted in overly-critical user perceptions:

“I’ve never noticed that before…must be something else those clowns broke.”

I’ve heard people are more likely to tidy up if they smell a faint scent of cleaning liquid.  Same thing occurs with bugs I guess. 

What’s the lesson here?  Releasing four bugs might be more expensive than fixing four bugs.  It might mean fixing seven and dealing with extra support calls until the priming effect wears off.

Categories: Software Testing

The Secret of My Success

QA Hates You - Tue, 06/24/2014 - 09:18

Haters gonna hate – but it makes them better at their job: Grumpy and negative people are more efficient than happy colleagues:

Everyone hates a hater. They’re the ones who hate the sun because it’s too hot, and the breeze because it’s too cold.

The rest of us, then, can take comfort in the fact that haters may not want to get involved in as many activities as the rest of us.

But in a twist of irony, that grumpy person you know may actually be better at their job since they spend so much time on fewer activities.

It’s not true, of course.

Haters don’t hate other haters.

But the rest could hold true.

Categories: Software Testing

QA Music: Is There Any Hope for QA?

QA Hates You - Mon, 06/23/2014 - 05:27

Devour the Day, “Good Man”:

Categories: Software Testing

LoadStorm v2.0 Now with Transaction Response Timing – What does this mean for you?

LoadStorm - Tue, 06/17/2014 - 12:17

Today, LoadStorm published a press release announcing our new Transaction Response Timing. For many professional performance testers, especially those used to products like HP Loadrunner or SOASTA CloudTest, wrapping timings around logical business processes and related transactions, is a familiar concept. For those of you that aren’t familiar, I’ll explain.

What is Transaction Response Time?

Transaction Response Time represents the time taken for the application to complete a defined transaction or business process.

Why is it important to measure Transaction Response Time?

The objective of a performance test is to ensure that the application is working optimally under load. However, the definition of “optimally” under load may vary with different systems.
By defining an initial acceptable response time, we can benchmark the application if it is performing as anticipated.

The importance of Transaction Response Time is that it gives the project team/application team an idea of how the application is performing in the measurement of time. With this information, they can relate to the users/customers on the expected time when processing request or understanding how their application performed.

What does Transaction Response Time encompass?

The Transaction Response Time encompasses the time taken for the request made to the web server, there after being processed by the Web Server and sent to the Application Server, which in most instances will make a request to the Database Server. All this will then be repeated again in reverse from the Database Server, Application Server, Web Server and back to the user. Take note that the time taken for the request or data in the network transmission is also factored in.

To simplify, the Transaction Response Time is comprised of the following:
Processing time on Web Server
Processing time on Application Server
Processing time on Database Server
Network latency between the servers, and the client

The following diagram illustrates Transaction Response Time.

Transaction Response Time = (t1 + t2 + t3 + t4 + t5 + t6 + t7 + t8 + t9) X 2
Note: X 2 represents factoring of the time taken for the data to return to the client.

How do we measure?

Measurement of the Transaction Response Time begins when the defined transaction makes a request to the application. From here, until the transaction completes before proceeding with the next subsequent request (in terms of transaction), the time is been measured and will stop when the transaction completes.

How can we use Transaction Response Time to analyze performance issues?

Transaction Response Time allows us to identify abnormalities when performance issues surface. This will be represented as a slow response of the transaction, which differs significantly (or slightly) from the average of the Transaction Response Time. With this, we can correlate by using other measurements such as the number of virtual users that are accessing the application at that point in time and the system-related metrics (e.g. CPU Utilization) to identify the root cause.

With all the data that has been collected during the load test, we can correlate the measurements to find trends and bottlenecks between the response time, and the amount of load that was generated.

How is it beneficial to the Project Team?

Using Transaction Response Time, the Project Team can better relate to their users by using transactions as a form of language protocol that their users can comprehend. Users will know that transactions (or business processes) are performing at an acceptable level in terms of time.
Users may be unable to understand the meaning of CPU utilization or Memory usage and thus using the common language of time is ideal to convey performance-related issues.

The post LoadStorm v2.0 Now with Transaction Response Timing – What does this mean for you? appeared first on LoadStorm.

You Can Learn From Others’ Failures

QA Hates You - Tue, 06/17/2014 - 04:27

10 Things You Can Learn From Bad Copy:

We’ve all read copy that makes us cringe. Sometimes it’s hard to put a finger on exactly what it is that makes the copy so bad. Nonetheless, its lack of appeal doesn’t go unnoticed.

Of course, writing is subjective in nature, but there are certain blunders that are universal. While poor writing doesn’t do much to engage the reader or lend authority to its publisher, it can help you gain a better understanding of what is needed to produce quality content.

It’s most applicable to content-heavy Web sites, but some are more broadly applicable to applications in general. Including #8, Grammar Matters:

Obviously, you wouldn’t use poor grammar on purpose. Unfortunately, many don’t know when they’re using poor grammar.

That’s one of the things we’re here for.

(Link via SupaTrey.)

Categories: Software Testing

GTAC 2014: Call for Proposals & Attendance

Google Testing Blog - Mon, 06/16/2014 - 15:57
Posted by Anthony Vallone on behalf of the GTAC Committee

The application process is now open for presentation proposals and attendance for GTAC (Google Test Automation Conference) (see initial announcement) to be held at the Google Kirkland office (near Seattle, WA) on October 28 - 29th, 2014.

GTAC will be streamed live on YouTube again this year, so even if you can’t attend, you’ll be able to watch the conference from your computer.

Presentations are targeted at student, academic, and experienced engineers working on test automation. Full presentations and lightning talks are 45 minutes and 15 minutes respectively. Speakers should be prepared for a question and answer session following their presentation.

For presentation proposals and/or attendance, complete this form. We will be selecting about 300 applicants for the event.

The due date for both presentation and attendance applications is July 28, 2014.

There are no registration fees, and we will send out detailed registration instructions to each invited applicant. Meals will be provided, but speakers and attendees must arrange and pay for their own travel and accommodations.

Categories: Software Testing

QA Music: QA’s Place

QA Hates You - Mon, 06/16/2014 - 04:33

The Pretty Reckless with “Heaven Knows”:

Sometimes it does feel like they want to keep us in our place at the end of the process, ainna?

Categories: Software Testing

An Angry Weasel Hiatus

Alan Page - Sun, 06/15/2014 - 21:24

I mentioned this on twitter a few times, but should probably mention it here for completeness.

I’m taking a break.

I’m taking a break from blogging, twitter, and most of all, my day job. Beyond July 4, 2014, I’ll be absent from the Microsoft campus for a few months– and most likely from social media as well. While I can guarantee my absence from MS, I may share on twitter…we’ll just have to see.

The backstory is that Microsoft offers a an award – a “sabbatical” for employees of a certain career stage and career history. I received my award nine years ago, but haven’t used it yet…but now’s the time – so I’m finally taking ~9 weeks off beginning just after the 4th of July, and concluding in early September.

Now – hanging out at home for two months would be fun…but that’s not going to cut it for this long awaited break. Instead, I’ll be in the south of France (Provence) for most of my break before spending a week in Japan on my way back to reality.

I probably won’t post again until September. I’m sure I have many posts waiting on the other side.


Categories: Software Testing

Forget the walking skeleton – put it on crutches

The Quest for Software++ - Mon, 06/09/2014 - 06:46

TL;DR version: Modern technology and delivery approaches, in particular the push towards continuous delivery, enable us take the walking skeleton technique much further. By building up the UI first and delivering the back-end using continuous delivery without interrupting or surprising users, we can get cut the time to initial value delivery and validation by an order of magnitude!

This is an excerpt from my upcoming book 50 Quick Ideas to Improve your User Stories. Grab the book draft from LeanPub and you’ll get all future updates automatically.

The Walking Skeleton has long been my favourite approach to start a major effort or a legacy transition. Alistair Cockburn provides a canonical definition of this approach in Crystal Clear:

A Walking Skeleton is a tiny implementation of the system that performs a small end-to-end function. It need not use the final architecture, but it should link together the main architectural components. The architecture and the functionality can then evolve in parallel.

Building on that, Dan North formulated the Dancing Skeleton Pattern. In addition to performing a small function on the target architecture, this pattern involves an interface for developers to interact with the environment and experiment. When the users need to perform a more complex function, developers can make the skeleton implementation “dance”.

The walking skeleton sets up the main architectural components, and the dancing skeleton allows experimentation on the target architecture. Both approaches allow teams to ship something quickly, and build on that iteratively. Modern technology and delivery approaches, in particular the push towards continuous delivery, enable us take those approaches even further.

Instead of validating the architecture with a thin slice, we can aim to deliver value with an even thinner slice and build the architecture through iterative delivery later. We can start with a piece that users can interact with, avoid linking together the architectural components, but instead use a simpler and easier back-end. We can then iteratively connect the user interface to the target platform. In the skeleton lingo, don’t worry about making the skeleton walk, put it on crutches and ship out. While users are working with it, build up the muscles and then take away the crutches.

The core idea of the skeleton on crutches is to ship out the user interface early, and plan for iterative build-up of everything below the user interface. The user interaction may change very little, or not at all, while the team improves the back-end through continuous delivery. With a good deployment pipeline, back-ends can be updated and reconfigured without even interrupting user flow. For example, with MindMup we fully multi-version scripts and APIs, so we can deploy a new version of back-end functionality without interrupting any active sessions. Users with open sessions will continue to use the old functionality until they refresh the page, and they can benefit from updated features on their next visit.

Here are some recent examples of skeletons on crutches on the projects I’ve been involved in:

  • A customer report built with Google Analytics events instead of hooking into a complex backend database. It wasn’t as comprehensive or accurate as the final thing, but the first version was done in two hours, and the final version took several weeks to implement because of third party dependencies.
  • A document uploader which saved directly from users’s browsers to Amazon S3 file storage, instead of uploading to the customer database. The skeleton was not as secure or flexible as the final solution, and did not allow fast searching, but was done in a hour.
  • A registration system executed using JotForm, an online forms and workflow system which can take payments. This allowed us to start registering users and charging for access without even having a database.

All these solutions were not nearly as good as the final versions, but they were in production very quickly and allowed users to start getting some value early. Some remained in action for a long time – for example we kept registration through JotForm in production for three months, because it was good enough and there were more important things to do.

Key Benefits

A major benefit of skeletons on crutches is early value delivery. A user segment starts getting some value almost instantly. None of the examples I mentioned earlier were particularly difficult to build, but users got value in hours instead of weeks.

Early validation is an important consequence of early value delivery. Instead of showing something to users and asking them to imagine how the final thing would be fantastic, a skeleton on crutches allows them to actually use something in real work. This allows product managers to get invaluable feedback on features and product ideas early and build that learning into the plans for later versions.

For example, the JotForm registration skeleton allowed us to start charging for access months before the overall solution was ready, and the fact that users paid provided huge validation and encouragement that we’re going in the right direction. We were also able to discover some surprising things people would pay for, and change the focus of our business plans.

How to make this work

Aim to deliver the user interfacing part first, on a simplistic architecture. Try to skip all back-end components that would slow down delivery, even if for half a day. Deliver the user interfacing part as close to the final thing as possible, to avoid future surprises. Iterate over that until you start delivering value. Then replace the back-end with as few changes to the user interfacing part as possible to avoid surprises. If you must change the UI, think about multi-versioning to support uninterrupted usage. Build up capacity, performance, and extend functionality by replacing invisible parts with your target architecture iteratively.

There are many good crutches available for developers online. My favourite ones are:

  • Amazon AWS cloud file stores, database systems and queues enable teams to store user documents and information easily, coordinate simple asynchronous flows and authenticate access to information using pre-signed URLs.
  • JotForm and similar forms processors allow teams to design forms and embed them in web sites. Jotform automatically keeps a record of submissions, it can send e-mail confirmations to users and developers. It also supports simple workflows for things such as payments, adding to a mailing list or sending to a landing page.
  • Stripe is a developer-friendly payment processor, that is easy to integrate and has an option of sending e-mail invoices to customers.
  • Google analytics are well known for web statistics, but they can also collect front-end and back-end events and create relatively complex reports on vast amounts of data. Think of this as a powerful reporting front-end for your data on which you can experiment with reporting structures and requirements.
  • Heroku application servers can run simple tasks, connecting to other crutches, and you can start and shut them down easily on demand. Heroku has lots of third party applications that you can connect easily. They may not do exactly what you need, and they can be quite pricey for large user bases, but Heroku is often a good quick and cheap start for something small.

If you can’t put a skeleton on remote crutches because of security or corporate policy, then consider using simple components onsite. One interesting example of this is a team at a bank I recently worked with. Their corporate procedures required weeks of bureaucracy to deploy an application with access to a production database. The team set up a web reporting application which executed reports on plain text files using Unix command line tools behind the scenes. This allowed them to deploy quickly, but only worked for quite a limited data set. While the users were happily using the reports in the first few weeks, the team obtained access to the production database and replaced the backend, without anyone noticing anything.

HTTP Archive – new stuff!

Steve Souders - Sun, 06/08/2014 - 23:11

The HTTP Archive crawls the world’s top 300K URLs twice each month and records detailed information like the number of HTTP requests, the most popular image formats, and the use of gzip compression. We also crawl the top 5K URLs on real iPhones as part of the HTTP Archive Mobile. In addition to aggregate stats, the HTTP Archive has the same set of data for individual websites plus images and video of the site loading.

I started the project in 2010 and merged it into the Internet Archive in 2011. The data is collected using WebPagetest. The code and data are open source. The hardware, mobile devices, storage, and bandwidth are funded by our generous sponsors:  GoogleMozillaNew RelicO’Reilly MediaEtsyRadwaredynaTrace SoftwareTorbitInstart Logic, and Catchpoint Systems.

For more information about the HTTP Archive see our About page.

New Stuff!

I’ve made a lot of progress on the HTTP Archive in the last two months and want to share the news in this blog post.

A major change was moving the code to Github. It used to be on Google Code but Github is more popular now. There have been several contributors over the years, but I hope the move to Github increases the number of patches contributed to the project.
The HTTP Archive’s trending charts show the average value of various stats over time. These are great for spotting performance regressions and other changes in the Web. But often it’s important to look beyond the average and see more detail about the distribution. As of today all the relevant trending charts have a corresponding histogram. For an example, take a look at the trending chart for Total Transfer Size & Total Requests and its corresponding histograms. I’d appreciate feedback on these histograms. In some cases I wonder if a CDF would be more appropriate.
We now plot the number of TCP connections that were used to load the website. (All desktop stats are gathered using IE9.) Currently the average is 37 connections per page.  
We now record the CDN, if any, for each individual resource. This is currently visible in the CDNs section for individual websites. The determination is based on a reverse lookup of the IP address and a host-to-CDN mapping in WebPagetest.
Custom Metrics

Pat Meenan, the creator of WebPagetest, just added a new feature called custom metrics for gathering additional metrics using JavaScript. The HTTP Archive uses this feature to gather these additional stats:

Average DOM Depth
The complexity of the DOM hierarchy affects JavaScript, CSS, and rendering performance. I first saw average DOM depth as a performance metric in DOM Monster. I think it’s a good stat for tracking DOM complexity. 
Document Height
Web pages are growing in many ways such as total size and use of fonts. I’ve noticed pages also getting wider and taller so HTTP Archive now tracks document height and width. You can see the code here. Document width is more constrained by the viewport of the test agent and the results aren’t that interesting, so I only show document height. 
localStorage & sessionStorage
The use of localStorage and sessionStorage can help performance and offline apps, so the HTTP Archive tracks both of these. Right now the 95th percentile is under 200 characters for both, but watch these charts over the next year. I expect we’ll see some growth.
Iframes are used frequently to contain third party content. This will be another good trend to watch.
The HTTP Archive has tracked the number of external scripts since its inception, but custom metrics allows us to track the total number of SCRIPT tags (inline and external) in the page.
Specifying a doctype affects quirks mode and other behavior of the page. Based on the latest crawl, 14% of websites don’t specify a doctype, and “html” is the most popular doctype at 40%. Here are the top five doctypes.

Doctype Percentage html 40% html -//W3C//DTD XHTML 1.0 Transitional//EN 31% [none] 14% html -//W3C//DTD XHTML 1.0 Strict//EN 8% HTML -//W3C//DTD HTML 4.01 Transitional//EN 3%

Some of these new metrics are not yet available in the HTTP Archive Mobile but we’re working to add those soon. They’re available as histograms currently, but once we have a few months of data I’ll add trending charts, as well.

What’s next?

Big ticket items on the HTTP Archive TODO list include:

  • easier private instance - I estimate there are 20 private instances of HTTP Archive out there today (see here, here, here, here, and here). I applaud these folks because the code and documentation don’t make it easy to setup a private instance. There are thousands of WebPagetest private instances in the world. I feel that anyone running WebPagetest on a regular basis would benefit from storing and viewing the results in HTTP Archive. I’d like to lower the bar to make this happen.
  • 1,000,000 URLs – We’ve increased from 1K URLs at the beginning four years ago to 300K URLs today. I’d like to increase that to 1 million URLs on desktop. I also want to increase the coverage on mobile, but that’s going to probably require switching to emulators.
  • UI overhaul – The UI needs an update, especially the charts.

In the meantime, I encourage you to take a look at the HTTP Archive. Search for your website to see its performance history. If it’s not there (because it’s not in the top 300K) then add your website to the crawl. And if you have your own questions you’d like answered then try using the HTTP Archive dumps that Ilya Grigorik has exported to Google BigQuery and the examples from

Categories: Software Testing

Six ways to make impact maps more effective

The Quest for Software++ - Wed, 06/04/2014 - 14:30

We’ve been working very hard over the last few months to make impact mapping easier and more effective with MindMup. Here are 6 ways we think it’s helping. I’d love to hear your ideas and comments on how we can make it even better.

GTAC 2014 Coming to Seattle/Kirkland in October

Google Testing Blog - Wed, 06/04/2014 - 13:21
Posted by Anthony Vallone on behalf of the GTAC Committee

If you're looking for a place to discuss the latest innovations in test automation, then charge your tablets and pack your gumboots - the eighth GTAC (Google Test Automation Conference) will be held on October 28-29, 2014 at Google Kirkland! The Kirkland office is part of the Seattle/Kirkland campus in beautiful Washington state. This campus forms our third largest engineering office in the USA.

GTAC is a periodic conference hosted by Google, bringing together engineers from industry and academia to discuss advances in test automation and the test engineering computer science field. It’s a great opportunity to present, learn, and challenge modern testing technologies and strategies.

You can browse the presentation abstracts, slides, and videos from last year on the GTAC 2013 page.

Stay tuned to this blog and the GTAC website for application information and opportunities to present at GTAC. Subscribing to this blog is the best way to get notified. We're looking forward to seeing you there!

Categories: Software Testing