Software Testing

QA Music: Sing Me A Song Of Startups

QA Hates You - Mon, 03/21/2016 - 04:39

Linkin Park, “In the End”

Categories: Software Testing

Oracles from the Inside Out, Part 9: Conference as Oracle and as Destination

DevelopSense - Michael Bolton - Thu, 03/17/2016 - 14:57
Over this long series, I’ve described my process of reasoning about problems, using this table: So far, I’ve mostly talked about the role of experience, inference, and reference. However, I’m typically testing for and with clients—product managers, developers, designers, documenters, and so forth. In doing so, I’m trying to establish a shared understanding of the […]
Categories: Software Testing

Test Passes, Sorta

QA Hates You - Thu, 03/17/2016 - 06:32

I found this interesting article on the St. Louis Post-Dispatch Web site:

There are a lot of test articles floating through the Internet in production systems. Why people don’t bother to turn them off after the testing is done, I don’t know.

Bonus points to you if you can spot the issue with the test article itself.

Categories: Software Testing

My Talk On How To Succeed At Being A Consultant Is Online

QA Hates You - Wed, 03/16/2016 - 06:31

If you want to be a successful consultant, you might learn something:

What defects will you log? Whatever defects you like. What is the best methodology for testing? Whatever methodology you like. What’s the best time to start automated testing? Whatever time you like.

Well, “succeed” might be a misnomer. But you’ll certainly be employable.

Categories: Software Testing

Couple Automated Checks With Product Bugs

Eric Jacobson's Software Testing Blog - Mon, 03/14/2016 - 12:16

If you find an escape (i.e., a bug for something marked “Done”), you may want to develop an automated check for it.  In a meeting today, there was a discussion about when the automated check needed to be developed?  Someone asked, “Should we put a task on the product backlog?”.  IMO:

The automated check should be developed when the bug fix is developed.  It should be part of the “Done” criteria for the bug.

Apply the above heuristically.  If your bug gets deffered to a future Sprint, deffer the automated check to that future Sprint.  If your bug gets fixed in the current Sprint, develop your automated check in the current Sprint.

Categories: Software Testing

QA Music – Wolf Out

QA Hates You - Mon, 03/14/2016 - 04:08

In This Moment, “Big Bad Wolf”

If you need more Monday morning wolfery, see also this.

Categories: Software Testing

The most important lesson to improve software delivery

The Quest for Software++ - Sun, 03/13/2016 - 17:00

Richard Tattersall of the parish of St. George-in-the-Fields, liberty of Westminster, gentleman, as he liked to be described, had a lot of interesting claims to his name. In his youth, in mid-18th century Lancashire, he wanted to join the jacobite rebels. After a family intervention, he ran away from home, and ended up becoming a stud-groom in the service of the Duke of Kingston-upon-Hull. A unique business sense led Tattersall to start his own race-horse farm, then establish an auctioning house. Tattersalls Auctioneers’ ended up at Hyde Park Corner, and their clients included the key British nobility and even the King of France. Two and a half centuries later, France is no longer a kingdom, the dukedom of Kingston is extinct, but Tattersalls auctioneers still runs. It is the oldest such institution currently operating in the world, and Europe’s largest. Tattersall was not just a good businessman, he also knew how to entertain. Even the future king George IV was known to frequently visit Tattersall at Highflyer Hall, to enjoy ‘the best port in the land’. In his late years, Tattersall was so universally known and respected throughout the realm, that Charles Dickens wrote about Tattersall ‘no highwayman would molest him, and even a pickpocket returned his handkerchief, with compliments’. It would, then, come to great surprise to the gentleman how often software developers curse his name today.

In an epic twist of irony, people often don’t even know that Tattersall, the person, had no influence on the cause of all that pain. Tattersall had just the right mix of business sense and social skills. At his Hyde Park Corner venue, he reserved two ‘subscription rooms’ for the members of the Jockey Club. The two rooms quickly became the centre of all horse betting in the UK. The Tattersalls Committee, an successor organisation to the informal clubs from the two subscription rooms, had the legal ability to exclude people from the sport or access to racecourses in Britain all the way up to 2008. The rules and regulations they used to settle disputes still more or less govern British horse racing today. And among those rules, there is the famed 4(c), the destroyer of software models. For all the pain it caused me over the years, it’s also responsible for one of the most important lessons in improving software delivery I’ve ever come across.

For teams working on horse racing software, ‘Rule 4’ is the one thing you can always mention to disrupt a meeting, cause everyone to tell you to go to hell, and then spend the rest of the afternoon playing Desktop Tower Defence. It’s the edge case a tester can easily invoke to break almost anything. It’s the reason why people phone in the next day and insist to stay home because of some unforeseen illness. That’s because Rule 4 messes with the one thing developers hate to touch — time.

Horse racing bets are priced either at the time when a bet is placed, or at the time when the race starts. Punters generally prefer the first, called ‘fixed odds’ because they know what they’re expecting. The price and the potential winnings are fixed, hence the name, and that’s the rock solid agreement between a punter and a bookie. Such a premise allows developers to design some nice and elegant settlement models, and then deal with the difficult stuff they like to solve, such as latency, throughput, and performance optimisations. But then Tattersall’s Rule 4 kicks in. It controls what happens in a case when one of the horses doesn’t show up for a race. If, for example, the favourite ends up in a ditch somewhere ten minutes before the race, all the people betting on the second favourite now stand to win a lot more than the bookies expected. Rule 4 allows a bookie to pay out the bets on the other horses as if the favourite never existed. This means going back in time, recalculating the odds for all the other runners, and applying the new price. With weird math, that isn’t exactly completely logical. For each single bet, at the time when that bet was placed. This is where the stomach problems start. Developers realise they need to start saving a lot more information at a time when the bet is placed, mess up with the nice elegant settlement models, break system performance, and a ton of other things. That’s why, generally, once Rule 4 is finally implemented correctly, that piece of code is off-limits. Nobody gets to even view it on a screen, to prevent an accidental Heisenbug.

Rule 4 makes a lot of sense when one of the favourites misses the race, but it’s also applied to Limpy Joe, the horse that almost died in the previous race of old age and boredom. It can lead to weird and pointless complaints about why someone got 5 pence less than expected, and it can cost more in wasted customer support than it protects against fraud. I once worked with a company that tried to save a bit of money on customer servicing, and do something nice for its punters at the same time, by not taking advantage of Rule 4 below a certain threshold. Switching rules on and off wasn’t such a big deal, it turned out that they could actually do it themselves, and everyone was happy. No need for anyone to stay home playing Desktop Tower Defence.

But then, one day, they asked us to change the monthly customer statements, and print out the bet results as if Rule 4 was still applied, then add the deduction back. Not only did this mess with time, but it messed with it twice. It required us to record something that didn’t happen as if it happened, combine it with a ton of other rules that did actually happen, and then flip back the whole thing again. And they asked us to do that not at the point when the bets were recorded or settled, but when the monthly statements were produced. This required keeping a lot more information so we could settle all the other rules backwards and forwards in time, break system performance, and change the one part of the system that nobody wanted to touch. Some of the additional rules were implemented by third parties, so we’d have to chase them to change their code as well. Of course, the whole thing needed to be done as quickly as possible, ideally yesterday. Lots of towers were successfully defended that day.

The next week, on a visit to the call centre, I sat next to an operator who was one of the people insisting we ‘fix the statements’, and watched him deal with a disgruntled punter. The person on the other end of the line heard about the Rule 4 promotion, and called in to complain that he was short-changed. In fact, some other rules produced an odd value for the final payout. The operator was stuck explaining that the amount on the statement is OK, and he had to take the punter through the whole weird math required to settle a bet according to all the other on-going promotions. A few minutes after that call, another one came in. Instead of saving money on customer support, the Rule 4 voiding idea made it worse. It was clear that the interaction of rules was causing the confusion, not Rule 4 itself. I asked about why they singled out that one, and the operator said that they were going to ask for all the other promotions to be pulled out into separate line items as well, just later. They didn’t want to overload us with work, so that Rule 4 could get done quickly.

Observing the problem first-hand, I could see the whole mess, and why they wanted something done about it. But just thinking about the implications on our nice, clean, elegant software models made my stomach turn. Luckily, being next to the one person who suffered the most, and finally understanding what’s going on, I could propose an alternative. What if, instead of actually redoing the numbers to list individual calculation components, we just listed the names of the special promotions applied to a bet? So punters would immediately see that there was more than one thing going on, and that their Rule 4 promotion still applied. That information was already available in the database. The fact that the solution could be deployed in a few days sold it easily. The trick with labels reduced the confusion, and not just for Rule 4, but for all the other promotions they were going to ask about later as well.

That day, I learnt that how important it is to spend time observing people actually using the software that we were building. Sitting next to a user allowed us to together flush out all sorts of weird and wonderful assumptions, and work together on ideas that solve real problems, not just plaster over a huge crack. Together, we discovered insights that nobody could predict. That’s just common sense, so surely everyone is doing it by now, right? Not so much.

With the previous post, I asked people to fill in a quick questionnaire about how frequently they observe people using their software. With slightly more than 700 responses, I can’t really claim any kind of universal statistical relevance for the whole industry. On the other hand, given that you’ve self-selected into a group by reading this, the data should be relevant for teams similar to yours.

Roughly fifty percent of the survey participants said that had no direct interaction with end-users, ever. They’ve never seen an actual user work with their software. Only about 10% teams, across the whole group, actually engage with their users every week.

(click for a larger version)

I asked separately about observing end-users testing future software ideas, pretty much the key aspect of getting any sort of user experience research executed. The numbers are even worse. Only about 6.4% of the respondents said that they do that on a weekly basis. In a fast moving industry such as software today, where bad assumptions and communication problems can cause serious damage, that’s just depressive.

(click for a larger version)

This is both good and bad news. For most people reading this, the bad news is that you’re not benefitting from all the insight that you could easily collect. The good news is that the bar is so low at the moment, that you can easily change your delivery process a bit and be far better than the competition.

If you’re a developer or a tester, figure out an excuse to sit with the actual users for a few hours every week. If you’re managing a team, think about sending the group to observe actual users periodically. I guarantee this will significantly improve the software you deliver. It doesn’t matter if your company already uses a UX specialist agency to deliver the decisions, or if you have someone else already tracking the usage patterns and results. Go and see things for yourself, you will learn stuff you could have never predicted before.

For example, we recently added text notes to MindMup 2.0. This was one of the most requested features on the user forums, so we had a ton of data to start with. Instead of just charging ahead with the ideas collected through initial requests, we spent a week building a rough version, and then invited actual users to try things out. The results were surprising. We got a bunch of assumptions about ordering and exporting wrong, and our users wanted something significantly simpler than what we planned to develop. As a result, we were able to launch that probably a month ahead of the schedule, and make it a lot more intuitive.

Of course, there will be plenty of excuses why spending time with users isn’t possible. Especially if they are not easily available. The survey results broken by type of software delivery clearly show that. Reduced only to software delivered internally, roughly 27% of the participants said that they observe users working with their software at least once a month. This is significantly better than just 16.5% of those working on consumer software. Enterprise B2B, of course, ends up in the last spot with roughly 14%.

Don’t let the fact that your users are remote, or numerous, stop you from talking to them. Even for consumer-oriented products, getting this kind of feedback is easier than it seems. I go to lots of software conferences, and I always try to get a few people to try out MindMup between conference sessions. It doesn’t cost us anything, and most of the people I approach are willing to spend a few minutes helping us out, especially during long boring lunch breaks. If such direct contact isn’t possible for your team, think about remote screen-sharing sessions. It’s not the best way to do user research, and I’d love to have one of those hi-tech rigs that tracks eye movement and blood pressure that ad agencies use, but that’s far beyond my budget. However, even a simple screen share session opens up an incredible amount of insight. The text notes research we did for MindMup 2.0 was done 100% over remote screen sharing, and we had people participating from all over the world. Just get people to think out loud while they are clicking around the screen, and be ready to get surprised.

Everything I Know About Testing I Learned From Zork

QA Hates You - Thu, 03/10/2016 - 07:07

Login Screen
This is a login screen before you can use the application, with an account name, password field, and a Log In! button.
There is bubbly copy and a licensed stock image of a bearded man holding a small boy.

> check copy

The copy is cheery, but not particularly informative. In a stunning turn of events, the words are all spelled correctly, AND they've remembered the serial comma.

> mouseover image

The title and alt text are set for the image and read "Welcome back!"

> type </html> into account name field.

The value displays in the edit box.

> type </html> into password edit box.

The value displays in the edit box.

> click Log In!

A Potentially Malicious Request warning displays! Oh, woe and agony! The site is eaten by a grue.

Categories: Software Testing

How I Became That Guy / The JavaScript Twofer

QA Hates You - Wed, 03/09/2016 - 10:32

I’ve set as a basic test visiting sites under test without JavaScript enabled, without Flash enabled, and without cookies enabled. It was a bit of an exercise for me, because who would visit the Web with JavaScript enabled?

Well, suddenly, that’s me.

I was running an old version of Safari on my iPhone because my client required a lot of backward compatibility testing. I don’t do a lot of processing on my phone, but I do a lot of reading, and suddenly Safari was crashing all the time. It was the ad content JavaScript, and once I turned it off, I could read on my phone again.

I’ve since updated the phone, but I’ve kept the JavaScript off because the pages load faster (if at all) and I’m using less data.

Which means I see all sorts of bad behavior. Not just counting unresponsive forms, but sites completely failing to load because they’re doing something JavaScript before page load with no alternative.

So I’ve set one of my browsers to block JavaScript by default just to see what happens when I hit a new test site. A lot of the time, I encounter something weird and start to log a defect before I realize nothing’s working because I have JavaScript disabled.

So I log a bug to indicate that the page should display a message in this case. That’s the first part of the two-fer.

Then, nine times out of ten, I retest and close that issue and open a new one to capitalize JavaScript correctly. Although I’m pretty clear in the defect, the developers type Javascript. All. The. Time. The JavaScript Disabled message is two bugs in one.

I’ve read memes and jokes on the Internet and Twitter that people who don’t use JavaScript are like vegans and CrossFit people and must tell everyone. Apparently, with this post, I have become that guy.

Oh, and I look forward to the first through one hundredth times I have to log a bug about capitalizing CrossFit correctly.

(Meanwhile, while we’re on the topic of JavaScript, also read Maybe we could tone down the JavaScript.)

Categories: Software Testing

Get Your Automated Checks In Their Face

Eric Jacobson's Software Testing Blog - Thu, 03/03/2016 - 13:03
If a tree falls in the forest and nobody hears it, does it make a sound?  If you have automated checks and nobody knows it, does it make an impact?

To me, the value of any given suite of automated checks depends on its usage...

Categories: Software Testing

Category: Bug

QA Hates You - Thu, 03/03/2016 - 06:04

You know I log every instance of controls/edit boxes/drop-down lists where the lower-cased g gets chopped at the bottom.

Well, except this one in the FogBugz defect tracker itself:

Internet Explorer is the worst offender in this regard, but the screenshot above is from Firefox.

Now you know why Roger Dougherty, single, born in August and living at 1021 Brighton Way, Harrisburg, Oregon always signs up for applications I test.

Categories: Software Testing

Test Planning Is Throwaway, Testing Is Forever

Eric Jacobson's Software Testing Blog - Mon, 02/29/2016 - 16:10

FeatureA will be ready to test soon.  You may want to think about how you will test FeatureA.  Let’s call this activity “Test Planning”.  In Test Planning, you are not actually interacting with the product-under-test.  You are thinking about how you might do it.  Your Test Planning might include, but is not limited to, the following:

  • Make a list of test ideas you can think of.  A Test Idea is the smallest amount of information that can capture the essence of a test.
  • Grok FeatureA:  Analyze the requirements document.  Talk to available people.
  • Interact with the product-under-test before it includes FeatureA.
  • Prepare the test environment data and configurations you will use to test.
  • Note any specific test data you will use.
  • Determine what testing you will need help with (e.g., testing someone else should do).
  • Determine what not to test.
  • Share your test plan with anyone who might care.  At least share the test ideas (first bullet) with the product programmers while they code.
  • If using automation, design the check(s).  Stub them out.

All the above are Test Planning activities.  About four of the above resulted in something you wrote down.  If you wrote them in one place, you have an artifact.  The artifact can be thought of as a Test Plan.  As you begin testing (interacting with the product-under-test), I think you can use the Test Plan one of two ways:

  1. Morph it into “Test Notes” (or “Test Results”).
  2. Refer to it then throw it away.

Either way, we don’t need the Test Plan after the testing.  Just like we don’t need those other above Test Planning activities after the testing.  Plans are more useful before the thing they plan.

Execution is more valuable than a plan.  A goal of a skilled tester is to report on what was learned during testing.  The Test Notes are an excellent way to do this.  Attach the Test Notes to your User Story.  Test Planning is throwaway.

Categories: Software Testing

Is Automated Checking Valuable For Data Warehouses?

Eric Jacobson's Software Testing Blog - Fri, 02/26/2016 - 15:18

My data warehouse team is adopting automated checking.  Along the way, we are discovering some doubters.  Doubters are a good problem.  They challenge us to make sure automation is appropriate.  In an upcoming meeting, we will try to answer the question in this blog post title.

My short answer:  Yes.

My long answer:  See below.

The following are data warehouse (or database) specific:

  • More suited to machines – Machines are better than humans at examining lots of data quickly. 
  • Not mentally stimulating for humans(this is the other side of the above reason) Manual DB testers are hard to find.  Testers tend to like front-ends so they gravitate toward app dev teams.  DB testers need technical skills (e.g., DB dev skills).  People who have them prefer to do DB dev work.
  • Straight forward repeatable automation patterns – For each new dimension table, we normally want the same types of automated checks.  This makes automated check design easier and faster to code.  The entire DW automation suite contains a smaller amount of design patterns than the average appliction.

The following are not limited to data warehouse (or database):

  • Time to market – (Automated checks) help you go faster.  Randy Shoup says it well at 9:55 in this talk.  Writing quick and dirty software leads to technical debt which leads to no time to do it right (“technical debt viscious cycle”).  Writing automated checks as you write software  leads to a solid foundation which leads to confidence which leads to faster and better (“virtuous cycle of quality”)…Randy’s words.
  • Regression checking - In general, machines are better than humans at indicating something changed. 
  • Get the most from your human testing - Free the humans to focus on deep testing of new features, not shallow testing of old features.
  • In case the business ever changes their mind - If you ever have to revist code to make changes or refactor, automated checks will help you do it quicker.  If you think the business will never change their mind, then maybe automation is not as important.
  • Automated checks help document current functionality.
  • Easier to fix problems - Automated checks triggered in a Continuous Integration find problems right after code is checked in.  These problems are usually easier to fix when fresh in a developer’s mind.
Categories: Software Testing

Coming to Dallas/Ft. Worth - Testing Mobile Applications - ASTQB Certification Course - April 19 - 20, 2016

Randy Rice's Software Testing & Quality - Thu, 02/25/2016 - 09:58
I hope you can join me for this special course presentation for the new Certified Mobile Tester (CMT) designation from the American Software Testing Qualifications Board.

We will be in the DFW area (Irving, TX) on the dates of Tuesday, April 19 and Wednesday, April 20 at the Holiday Inn Express - 4235 West Airport Freeway, Irving, TX 75062

Seating is limited, so I recommend registering as soon as possible to get your place. (This class size is limited to 15 people.)

To register:

About the Course and Certification

In the fall of 2015, the ASTQB (of which I am on the board of directors) felt there was a compelling need for testers to have a robust and meaningful certification focused on testing mobile applications. So, we set about writing a syllabus and exam for that certification. I was honored to contribute as a co-author of the syllabus. At the present, this is a certification only offered by the ASTQB.

To learn more about the certification, see the syllabus and sample exam, just go to:

To see the course outline, go to:

There are no pre-requisites for this certification! While we reference concepts from the ISTQB Foundation Level, everything you need to know is taught in this course.

We have exercises to reinforce key concepts and sample exams after each module to give you a taste of what to expect on the actual exam. You can also bring your own mobile device as a way to perform the exercises, although this is not required.

Costs and Logistics

Cost: $1,500 USD per person, plus exam ($150).

You can attend the course without taking the exam. However, we will be offering a live exam at the end of the 2nd day.

There will also be the option to take the exam later electronically, if you desire. However, we need to know your preference 2 weeks in advance.

This class will be streamed live, so if you want to attend virtually, that is possible. There is a $100 discount for virtual attendees. Virtual attendees in the USA will receive a course notebook in advance of the training and will also have access to the e-learning course at no extra cost.

For teams of 3 or more, there is a 10% discount of the course registration fee. The exams are not discounted.

We will not have breakfast items, however, we will have a light lunch (pizza, sandwiches, etc.) brought in each day. Please let us know if you have any dietary needs or requests.

Important Notice for Those Who Plan to Travel to DFW to Attend

Please do not book any non-refundable travel (air fare, hotel, etc.) until we confirm the class. We make every attempt to not cancel a class, but sometimes this is unavoidable. We make the call about 2 - 3 weeks in advance of the class, or earlier, if possible.

We located this class to be close to the DFW airport for the convenience of those who may be traveling in for the class. The hotel runs a free airport shuttle.

You are responsible for making your own hotel reservations.

If you plan to take the exam on Day 2, the exam will take place from 3:30 p.m. to 4:30 p.m.  Please allow adequate time to catch your flight. The hotel is 6.4 miles from the DFW airport.

Other Questions?

Feel free to call our office with any questions or special needs - 405-691-8075.

Categories: Software Testing

QA Music – Better Relationships with Co-Workers

QA Hates You - Mon, 02/22/2016 - 03:10

“The Monster” by Eminem

I’m not friend with the monsters under my bed. I’ve frightened them all away.

Categories: Software Testing

Introducing Claudia.js &ndash; deploy Node.js microservices to AWS easily

The Quest for Software++ - Sun, 02/21/2016 - 16:00

I’m proud to announce the 1.0 release of Claudia.js, a new opensource deployment tool for Javascript developers interested in running microservices in AWS. AWS Lambda and API Gateway offer scalability on demand, zero operations overhead and almost free execution, priced per use, so they are a very compelling way to run server-side code. However they can be tedious to set up, especially for simple scenarios. The runtime is oriented towards executing Java code, so running Node.js functions requires you to iron out quite a few issues, that aren’t exactly well documented. Claudia.js automates and simplifies deployment workflows and error prone tasks, so you can focus on important problems and not have to worry about AWS service quirks. Even better, it sets everything up the way Javascript developers expect, so you’ll feel right at home.

Check out the video below for an example how you can set up and deploy a new API in less than five minutes!

Claudia.js is available from NPM, and the source code is on Github!

Not Only Wireframes, But Yes, Wireframes

QA Hates You - Thu, 02/18/2016 - 10:23

You know, I like to get a look at any and all artifacts as soon as possible to see if I can spot any flaws as early as I can. This includes comps, prototypes, copy, and wireframes, where I hope to catch oversights before they get into the code.

But in addition to looking for oversights, I always wanted to review the documents qua documents, especially if your company is providing wireframes, comps, prototypes, copy, and so on to the client for review. It gives you a chance to catch mistakes, misspellings, improper branding, and inconsistencies before your client can look at them and think, “Ew, these guys can’t spell our name right on the wireframes. What would they do to our Web site?”

Yes, I did review RFP responses and proposals as well.

The Purple One links to this article entitled Wireframes – Should They Really Be Tested? And If So, How?

New trainees came on board and we had a training class to learn software testing concepts. After seeing those enthusiastic faces with their almost blank-slate minds (professionally), I decided to take a detour to my routine training.

After a brief introduction, instead of talking about software testing like I normally do, I threw a question at the fresh minds – ‘Can anyone explain me what a wireframe is? ’

The answer was a pause and thus, we decided to discuss it. And that is how it started – Wireframe/Prototype Testing

This should provide a good argument and overview if you need one.

Categories: Software Testing

Automated Checking Is Very Human

Eric Jacobson's Software Testing Blog - Wed, 02/17/2016 - 15:20

I read A Context-Driven Approach to Automation in Testing at the gym this morning.  I expected the authors to hate on automation but they didn’t.  Bravo.

They contrasted the (popular) perception that automation is cheap and easy because you don’t have to pay the computer, with the (not so popular) perception that automation requires a skilled human to design, code, maintain, and interpret the results of the automation.  That human also wants a paycheck.

Categories: Software Testing

New Software Development Employee Orientation Guide

QA Hates You - Wed, 02/17/2016 - 03:31

You owe it to yourself to make your new co-workers read this: Living in the Age of Software Fuckery: Ten Anti-patterns and Malpractices in Modern Software Development

Well, all except the new managers. They teach this stuff in MBA and MIS programs already. But as a good idea.

Link via iDisposable.

Categories: Software Testing