Stupid Multitasking

Alan Page - Tue, 08/14/2012 - 14:22

Today I saw yet another info-byte on the perils of multi-tasking. I too, think that multitasking is rarely (if ever) as productive as multitaskers believe it is (note: my favorite study compares bong hits with multitasking…ymmv) – but we’ve all seen stuff like this over time.

What’s missing from many of the studies and stories that I’ve seen is a definition of what they mean by multitasking. The study above uses examples (and good ones, like the problem with texting and driving), but then seem to lump all other pseudo-multi-tasking into the same bucket. If, by multi-tasking, you mean texting while driving, writing code while playing a video game, or walking the dog while juggling, the studies are 100% true – but perhaps less so, if you consider multi-tasking to be ordering a pizza on the phone while taking your dog for a walk, or listening to the radio while cooking dinner.

But for most tasks requiring brainpower, we don’t actually do two things at once – humans are largely a single threaded system, and we tend to do one thing at a time quite well – it’s the context switch that kills us. This is especially true in knowledge work where we’re writing / coding / speaking / thinking; and where the context switch of doing something different (checking email, checking a blog post, answering the phone, sending a tweet, etc.) ruins our train of thought and takes us out of “the zone”.

But sometimes, context switches are necessary. My favorite example (as anyone who knows me would guess) is cooking. When I cook any meal of significance, context switches are required. One cannot cook a meal serially. If I make the vegetable, then the appetizer, then the main dish, then the desert, I get to serve everything cold. I need to start with my deadline (dinner time), think through the process needed to make each dish, then determine what needs to be done in what order to accomplish my goal. If I make a marinade that needs to sit for three hours, I better do that first. Anything that requires heat should finish as close to dinner time as possible. I love the orchestration of cooking a great meal, but this task would be impossible (or at least a failure) if I didn’t rely on the context switch.The trick is to plan for the context switch and optimize – do the stuff where you know you need to wait first. Mentally work backwards and set up intermediate check points. Sure, doing only one thing would probably be faster, but it’s not always practical..

I apply the same techniques to my daily work. I look at all of the things I need to do (which is nearly always more than I can do), and do anything with a built-in wait first. For example, I start most days by sending out any requests for information I may need, things I need reviewed, etc. Then I take on bigger tasks. For writing tasks, I use a pomodoro timer – for coding or testing tasks I just go to work (I’ve found that the pomodoro beep ends up being a distraction from these types of tasks– but that’s just me). But regardless of whether it’s a pomodoro timer, or a natural break in flow, I tend to use those breaks to let interruptions decide what’s next. It’s a conscious task switch, but just like in cooking, where I may take a moment to look around the kitchen and see what’s going on (oops – better turn the oven down), if you’re in a role where you need to own multiple pieces of a project, it’s critical to allow yourself to step back, look at the big picture, and either dive back in again, or refocus on another area.

Now, to be fair, when I have a bigger project to take on, I’ll minimize multi-tasking as much as possible. I can think of several occasions where I needed to focus on accomplishing a specific task – even at the expense of potentially more critical tasks. My approach in those situation involves a pomodoro timer, and three-hour blocks of time in the morning and afternoon, broken up by breaks for email and task juggling. But – for the most part, I’ve found much benefit in strategic task switching and planning ahead than in giving up and admitting to stupidity just because your plate is full.

Categories: Software Testing

Don't Claim It If You Only Read It

ABAKAS - Catherine Powell - Tue, 08/14/2012 - 08:44
I've been hiring on behalf of a client lately, looking for a head of QA. One of the trends I noticed as I was reviewing resumes was that a lot of candidates had a list of languages on their resumes: C++, Perl, Ruby, Java, etc. "How cool!", I thought. It's great to see the testing profession attracting people who are capable of writing code but for whatever reason find testing a better fit, Maybe that profession is finally overcoming the stigma of "not good enough for dev" and "tester == manual tester".

And then I started the phone screens.

I've screened 12 people so far who listed languages on their resumes, and so far not one of them has actually been capable of writing code. Nope, that language list? That's the languages they can read. As in, "I can follow along on code written in C++ or Java."

Ouch. Now I'm disappointed in all of them. Here I thought they had some skills they don't have.

I understand what the candidates were going for. They're "technical testers" rather than testers who work purely on the business level and don't understand how software works. I get it. I really do. That such a distinction has meaning is sad but true.

But don't claim languages if you mean you can read them! You're an engineer, darnit! Claim a language if you can write it. There are enough engineers - and enough testers - who can write code that to claim a language without qualifying it means people will think you write it.

If you're trying to say, "hey, I'm a technical tester", then please do so in your resume. Just do so without unintentionally telling people that you're more technical than you are. The right way to say, "I can read but not write C++" is to say: "Languages: C++ (reading), Perl (read/scripting)" or something similar. That way hiring managers don't start with higher expectations than you can meet... And that's good for candidates, too.
Categories: Software Testing

The Problem with TheApp

Alan Page - Mon, 08/13/2012 - 10:58

Last week, I posted a small testing challenge (in short, an application wouldn’t launch).Shmuel Gershon has a wonderful write up of how he found the problem here.

For those too lazy to read, the problem was that the application attempted to read a reg key on launch, and failed to launch (with no diagnostics) if the reg key didn’t exist.

Having done a massive amount of application compatibility testing over the years, this is a fairly common scenario I’ve run across (simplified quite a bit in this case). Typically, it works like this: a team makes an update to an application or operating system, and some application begins to fail in a bizarre way (including failing to launch, as well as functionality or stability changes). After some investigation, we usually discover that the application relied on a registry key, a file, or functionality that no longer exists, or has changed. Half of the time, the application vendor is out of business, or doesn’t have the bandwidth to make an update, so we have to figure out what went wrong and hack do whatever is necessary to ensure the application continues to work.

Some of the time, the investigation requires a bit of reverse engineering, but at least 9 times out of 10, I use tools like sysinternals procmon to see what’s going on. I was happy (but not surprised) to see that Shmuel ended up using this tool to find the issue (and was even happier to see how well the tool led him to the exact issue).

For my job, I do almost as much investigation into why things don’t work as I put into discovering what works (and doesn’t work). To me, the detective work is part of the testing role. I was surprised yesterday, when a tester told me that his job was problem detection only, and that the rest was for someone else. That world of testing would be boring to me – ymmv.

Congrats Shmuel for the great investigation, and for having this valuable tool in his toolbox.


Note: for simplicity, I put the key under HKCU so everyone could access it, and used an obvious key name (MagicValue) – in the real world, it’s not always this way.

Categories: Software Testing

Effective team communication strategies: Structured conversations, product partnerships

SearchSoftwareQuality - Mon, 08/13/2012 - 10:30
Consultants provide insight into logical strategies that aid in team communication in Agile organizations that include structured conversations.

Categories: Software Testing

The Intersection of Boobs and Boobs

QA Hates You - Mon, 08/13/2012 - 04:14

What happens if you don’t check your Web site’s alt text and filenames when you create or upload content to a CMS? Bad, bad things (link safe for work).

Ten women in the St. Louis area have sued their doctor after learning that before-and-after pictures of their breast augmentation surgeries could be found online through a simple search of their names.

. . . .

The photos are widely used as a marketing tool to promote doctors’ work. They are not publicly labeled with names. But if patients’ names aren’t removed from the computerized picture file information, they can be displayed with the images during an Internet search.

You think you might be exempt if you’re just a CMS provider and a Web host? Think again.

In court filings, Koo’s lawyers blame MedNet, saying it failed to maintain Koo’s site in a “competent and professional manner.”

“Whatever Dr. Koo did do or didn’t do,” said one of her lawyers, Jonathan Ries, she had “no intention” of linking pictures and names.

MedNet’s response blames Koo, saying the company did not post, control or influence the content. It also claims legal immunity under the Communications Decency Act, which protects websites from suits over postings by third parties.

Already, you can puzzle out how this happened, or how this could have happened. A civilian, and by that I mean a semi-ignorant user, just uploaded image files from the hard drive without thinking about the filenames, and away you go. The CMS picks up the image name to use as the alt text, and suddenly you’re showing the world JaneDoubleDoeAfter to the world.

What can you do, QA? Remember to check the alt/title text carefully whenever you can, and pay attention to file names and URLs.

Because that stuff doesn’t matter until it does, really expensive-like and attorney-chocked.

UPDATE: And upon further reflection, you know what holds true of the people who made these costly errors? They were all probably badmins.

Categories: Software Testing

Covering all your codebases: A conversation with a Software Engineer in Test

Google Testing Blog - Sat, 08/11/2012 - 19:50
Cross-posted from the Google Student Blog

Today we’re featuring Sabrina Williams, a Software Engineer in Test who joined Google in August 2011. Software Engineers in Test undertake a broad range of challenges on a daily basis, designing and building intelligent systems that can explore various use cases and scenarios for distributed computing infrastructure. Read on to learn more about Sabrina’s path to Google and what she works on now that she’s here!

Tell us about yourself and how you got to Google.
I grew up in rural Prunedale, Calif. and went to Stanford where I double-majored in philosophy and computer science. After college I spent six years as a software engineer at HP, working primarily on printer drivers. I began focusing on testing my last two years there—reading books, looking up information and prototyping test tools in my own time. By the time I left, I’d started a project for an automated test framework that most of our lab used.

I applied for a software engineering role at Google four years ago and didn’t do well in my interviews. Thankfully, a Google recruiter called last year and set me up for software engineer (SWE) interviews again. After a day of talking about testing and mocking for every design question I answered, I was told that there were opportunities for me in SWE and SET. I ended up choosing the SET role after speaking with the SET hiring manager. He said two things that convinced me. First, SETs spend as much time coding as SWEs, and I wanted a role where I could write code. Second, the SETs job is to creatively solve testing problems, which sounded more interesting to me than writing features for a product. This seemed like a really unique and appealing opportunity, so I took it!

So what exactly do SETs do?
SETs are SWEs who are really into testing. We help SWEs design and refactor their code so that it is more testable. We work with test engineers (TEs) to figure out how to automate difficult test cases. We also write harnesses, frameworks and tools to make test automation possible. SETs tend to have the best understanding of how everything plays together (production code, manual tests, automated tests, tools, etc.) and we have to make that information accessible to everyone on the team.

What project do you work on?
I work on the Google Cloud Print team. Our goal is to make it possible to print anywhere from any device. You can use Google Cloud Print to connect home and work printers to the web so that you (and anyone you share your printers with) can access them from your phone, tablet, Chromebook, PC or any other supported web-connected device.

What advice would you give to aspiring SETs?
First, for computer science majors in general: if there’s any other field about which you are passionate, at least minor in it. CS is wonderfully chameleonic in that it can be applied to anything. So if, for example, you love art history, minor in art and you can write software to help restore images of old paintings.

For aspiring SETs, challenge yourself to write tests for all of the code you write for school. If you can get an internship where you have access to a real-world code base, study how that company approaches testing their code. If it’s well-tested, see how they did it. If it’s not well-tested, think about how you would test it. I don’t (personally) know of a CS program that has even a full course based on testing, so you’ll have to teach yourself. Start by looking up buzzwords like “unit test” and “test-driven development.” Look up the different types of tests (unit, integration, component, system, etc.). Find a code coverage tool (if a free/cheap one is available for your language of choice) and see how well you’re covering your code with your tests. Write a tool that will run all of your tests every time you build your code. If all of this sounds like fun...well...we need more people like you! 

If you’re interested in applying for a Software Engineer in Test position, please apply for our general Software Engineer position, then indicate in your resume objective line that you’re interested in the SET role.

Posted by Jessica Safir, University Programs

Categories: Software Testing

The Civilians Are Noticing

QA Hates You - Fri, 08/10/2012 - 07:43

The Atlantic is noticing what we QAssandras have been prophesying forever: Software Runs the World: How Scared Should We Be That So Much of It Is So Bad?

What do most people think of when they think of software? A decade ago, probably Microsoft Word and Excel. Today, it’s more likely to be Gmail, Twitter, or Angry Birds. But the software that does the heavy lifting for the global economy isn’t the apps on your smartphone. It’s the huge, creaky applications that run Walmart’s supply chain or United’s reservation system or a Toyota production line.

And perhaps the most mission-critical of all mission-critical applications are the ones that underpin the securities markets where a large share of the world’s wealth is locked up. Those systems have been in the news a lot recently, and not for good reasons. In March, BATS, an electronic exchange, pulled its IPO because of problems with its own trading systems. During the Facebook IPO in May, NASDAQ was unable to confirm orders for hours. The giant Swiss bank UBS lost more than $350 million that day when its systems kept re-sending buy orders, eventually adding up to 40 million shares that it would later sell at a loss. Then last week Knight Capital — which handled 11 percent of all U. S. stock trading this year — lost $440 million when its systems accidentally bought too much stock that it had to unload at a loss.* (Earlier this year, a bad risk management model was also fingered in JP Morgan’s $N billion trading loss, where N = an ever-escalating digit.)

The underlying problem here is that most software is not very good. Writing good software is hard. There are thousands of opportunities to make mistakes. More importantly, it’s difficult if not impossible to anticipate all the situations that a software program will be faced with, especially when–as was the case for both UBS and Knight–it is interacting with other software programs that are not under your control. It’s difficult to test software properly if you don’t know all the use cases that it’s going to have to support.


Categories: Software Testing

Welcome to the Next Generation of Google Testing

Google Testing Blog - Thu, 08/09/2012 - 23:39
By Anthony Vallone
Wow... it has been a long time since we’ve posted to the blog. This past year has been a whirlwind of change for many test teams as Google has restructured leadership with a focus on products. Now that the dust has settled, our teams are leaner, more focused, and more effective. We have learned quite a bit over the past year about how best to tackle and manage test problems at monumental scale. The next generation of test teams at Google are looking forward to sharing all that we have learned. Stay tuned for a revived Google Testing Blog that will provide deep insight into our latest testing technologies and strategies.
Categories: Software Testing

Making enterprise agility work with cross-functional teams

SearchSoftwareQuality - Thu, 08/09/2012 - 14:48
Enterprise agility works when cross-functional teams are able to self-organize, resolve problems and form Communities of Practice.

Categories: Software Testing

Applying Agile principles to large teams

SearchSoftwareQuality - Thu, 08/09/2012 - 14:38
Agile expert Lisa Crispin discusses the difficulty with large-scale Agile, sharing how Agile principles can be applied to maximize team coordination.

Categories: Software Testing

5 Ways To Clean Your Body’s Engine

Selena Delesie - Wed, 08/08/2012 - 05:15

Just like a car engine, your body works best when it’s clean on the inside. So it should be no surprise that we need to limit (or, gasp! eliminate) consumption of anything that pollutes it, including coffee, alcohol, processed foods,… the list goes on. In a day and age of go-go-go, it’s easy to lose balance and grab something for a quick pick-me-up. Unfortunately that pick-me-up often leaves us worse for the wear.

Despite being health conscious and eating fresh whole foods galore, even I go a little overboard. Sometimes I’ll have too many glasses of wine or chow down on a seemingly delicious bag of chips. When I wake up the next morning though, I feel it!  I’m exhausted, sluggish, sore, bloated, and brain fogged. I may have even put on some unwanted weight.

This is a result of the body trying to get rid of the toxins put into it. We’ll call this a hangover using the definition of “something remaining behind from a former period or state of affairs“.

A one-day binge results in a hangover the next day, but, you don’t need to go overboard in one day to have these effects. While the process is more extreme with a one-day binge, enough bad stuff accumulated over time will still leave you feeling less-than-stellar.



When bad stuff is put into your body, it needs to divert energy from other automatic bodily functions while it tries to clean up the mess you made. Less energy, oxygen and nutrients are supplied to your brain, muscles, and nervous system, as they are instead directed at finding and eradicating the poisons and chemicals.

It doesn’t matter if the source is alcohol, food, or something else. How your body works through the eradication and healing phase is basically the same:

  1. Search for bad stuff
  2. Eliminate bad stuff
  3. Replenish good stuff

I’m sure you have heard of many ways to cure a hangover or detox your body. Many of them are no good, as they just add to the toxic waste already accumulated, and make it harder (if not impossible) for your body to find a healthy, happy balance.

For example:

  • Drink coffee. Bad idea, as it makes you more dehydrated and slows down, or even stops, the search-eliminate-replenish process. And now, your body needs to add that coffee to backlog.
  • Eat greasy food (e.g. chips, fries, pizza, burgers,…). Another bad idea. While the food might soak up any remaining accessible alcohol, your body now has even more toxins to get rid of. You have also increased your level of dehydration.
  • Drink/eat what led to the hangover to begin with. Yes, again a bad idea. You still need to get rid of the toxic waste, and you’ve just put the process on hold all over again.


What Does Help?

There are a number of things you can do that really will help, as they eliminate and/or reduce the symptoms and speed up the healing process. Each approach below is an instinctive craving for our body to get what it needs to clean itself out and heal. Add one approach to your daily routine every week to speed up your body’s recovery and healing process.

Also spend some time connecting with your body and sensing what it asks for, separating this from your emotions, habits, and usual comforts. You will soon hear your body’s requests and start providing it with what it needs to search-eliminate-replenish itself back to a healthy balanced body.


Destinee Weathers

1) Drink lots of water

This should be no surprise! Your body needs water to function, period. Optimally, your body is comprised of 50-60% water. Without adequate water levels, your body automatically slows down different systems, resulting in the awful side effects noted above. It will also be unable to flush toxins out of the body in a speedy manner.

Drink an additional 1-2 litres of water each day to help your body regain proper function and quickly flush out the bad stuff.


2) Do some light exercise

Moving your body helps pump oxygen into your blood and increases blood flow, which in turn allows all parts of your body to operate better. Each part is then able to do it’s job in the search-eliminate process. Don’t push too hard though – you want your body to focus on cleaning itself, not on keeping up with a challenging workout.

Walking, basic yoga, and light body weight exercises are all good choices.


3) Eat fresh whole foods

This means food that has been grown in the ground or on a plant – NOT food from a box, jar, can, bag, or animal. Fresh whole foods are jam-packed with micronutrients, the building blocks your body needs to properly function. Micronutrients are all the vitamins and minerals that your body extracts directly from fresh whole foods to nourish and cleanse your body with. They do not need to be broken down or translated into something else to be effective. Processed foods and animal products are very hard for your body to break down and transform into something useful.

Choose to use your body’s limited energy on nourishing and cleansing itself by eating lots of fruits and vegetables, and small amount of nuts, seeds, and legumes.


4) Sweat

I’m sure the last thing you want to do when you feel unwell is get hot and sweaty, but it’s a surefire way to get toxins out of your body fast. Sweating is the body’s natural way to cool itself down and expel bad stuff.

Do some light exercise, use a sauna for a limited time, or even sit outside on a hot day. Make sure you stay hydrated by drinking even more water to replace the water released through your sweat.


5) Drink fresh juice or a green smoothie

Fresh juices and smoothies are a secret weapon for cleansing your body and improving hangover symptoms.

Freshly pressed fruit and vegetables will quickly nourish your body with loads of vital minerals, vitamins, and water. They are a powerful way to dramatically speed up the search-eliminate-replenish process.

Make your own fresh juice at home, or buy fresh juice at a juice bar. Don’t buy it prepackaged from a store.

Freshly blended smoothies made with leafy greens, fresh fruits, and water will have a similar effect. As they contain all the fibre of the produce, green smoothies are powerful cleansers, that will also provide a high level of nourishment.

Make green smoothies fresh in your blender using only water, leafy greens, and fruit.



Your Turn

With five powerful, effective, and healthy ways to clean your body’s engine, you have no more excuses for not helping your body recover and heal!

What have you done to clean your body’s engine? What has your experience been? What has worked and has not worked?

Share with us!







Categories: Software Testing

The Why Behind He Said So

ABAKAS - Catherine Powell - Tue, 08/07/2012 - 16:30
In software, we ask "why" a lot. "Why" is a seeking question. If we understand the cause or the reason for something then we can make an intelligent decision about changing it or going around it (and let's be honest - we usually ask why because we want to change it or work around it).

The answers to "why" are many and varied, but possibly the worst answer possible is: "Because so-and-so said so."

That's not a reason, that's an excuse. Presumably, whoever said so had a reason for saying so, and that's the actual why. There are two reasons we get into this situation:

  • Because that person is almost always right (or in a position of authority that effectively makes them correct)
  • Because that person is a convenient target for blame
It doesn't have to be this way. "Because so-and-so said so" is a learning opportunity. It's a chance to understand better, both this specific scenario and how so-and-so got to be such an expert. Think of it as a chance to check your work.
"Why is there a limit of 255 virtual machines on this box?" "Because John said so."
Well, John's a really smart guy, and he knows the product really really well, so presumably he has a reason for saying so. So why did he say so? "Because we use a /24 network, which allows 256 hosts, and we reserve one address for the physical box." See? Now you actually know something more about the product than you did before. You also know that there's potentially a problem there (networks typically allow 254 hosts, with .0 and .255 generally reserved). 
Find out why someone said so.... it's illuminating.
Categories: Software Testing

ScriptCover makes Javascript coverage analysis easy

Google Testing Blog - Tue, 08/07/2012 - 10:31
By Ekaterina Kamenskaya, Software Engineer in Test, YouTube

Today we introduce the Javascript coverage analysis tool, ScriptCover. It is a Chrome extension that provides line-by-line Javascript code coverage statistics for web pages in real time without any user modifications required. The results are collected both when the page loads and as users interact with it. The tool reports details about total web page coverage and for each external/internal script, as well as annotated code sources with individually highlighted executed lines.
Short report in Chrome extension’s popup, detailing both overall scores and per-script coverage.

Main features:
  • Report current and previous total Javascript coverage percentages and total number of instrumented code instructions.
  • Report Javascript coverage per individual instruction for each internal and external script.
  • Display detailed reports with annotated Javascript source code.
  • Recalculate coverage statistics while loading the page and on user actions.
Sample of annotated source code from detailed report. First two columns are line number and number of times each instruction has been executed.
Here are the benefits of ScriptCover over other existing tools:
  • Per instructions coverage for external and internal scripts: The tool formats original external and internal Javascript code from ‘<script>’ tags to ideally place one instruction per line and then calculates and displays Javascript coverage statistics. It is useful even when the code is compressed to one line.

  • Dynamic: Users can get updated Javascript coverage statistics while the web page is loading and while interacting with the page.

  • Easy to use: Users with different levels of expertise can install and use the tool to analyse coverage. Additionally, there is no need to write tests, modify the web application’s code, save the inspected web page locally, manually change proxy settings, etc. When the extension is activated in a Chrome browser, users just navigate through web pages and get coverage statistics on the fly.

  • It’s free and open source!
Want to try it out? Install ScriptCover and let us know what you think.

We envision many potential features and improvements for ScriptCover. If you are passionate about code coverage, read our documentation and participate in discussion group. Your contributions to the project’s design, code base and feature requests are welcome!
Categories: Software Testing

Greatest Hits List

QA Hates You - Tue, 08/07/2012 - 03:41

Wikipedia has a rather incomplete list of software bugs that’s amusing. It’s not comprehensive, and it’s not very educational, but it’s a good reminder of what can happen if you aren’t paying attention.

Categories: Software Testing

Does outsourced testing work for Agile teams?

SearchSoftwareQuality - Mon, 08/06/2012 - 12:33
Outsourced testing can be successful for Agile teams under certain circumstances.

Categories: Software Testing

When centralized teams work for Agile enterprises

SearchSoftwareQuality - Mon, 08/06/2012 - 12:23
Success for any team in Agile enterprises requires commitment. Centralized teams may work if they share the same goals.

Categories: Software Testing

Bite-Sized Test Wisdom From RST Class – Part 3

Eric Jacobson's Software Testing Blog - Mon, 08/06/2012 - 10:10

See Part 1 for intro.

  • People don’t make decisions based on numbers, they do so based on feelings (about numbers).
  • Asking for ROI numbers for test automation or social media infrastructure does not make sense because those are not investments, those are expenses.  Value from an automation tool is not quantifiable.  It does not replace a test a human can perform.  It is not even a test.  It is a “check”.
  • Many people say they want a “metric” when what they really want is a “measurement”.  A “metric” allows you to stick a number on an observation.  A “measurement”, per Jerry Weinberg, is anything that allows us to make observations we can rely on.  A measurement is about evaluating the difference between what we have and what we think we have.
  • If someone asks for a metric, you may want to ask them what type of information they want to know (instead of providing them with a metric).
  • When something is presented as a “problem for testing”, try reframing it to “a problem testing can solve”.
  • Requirements are not a thing.  Requirements are not the same as a requirements document.  Requirements are an abstract construct.  It is okay to say the requirements document is in conflict with the requirements.  Don’t ever say “the requirements are incomplete”.  Requirements are not something that can be incomplete.  Requirements are complete before you even know they exist, before anyone attempts to write a requirements document.
  • Skilled testers can accelerate development by revealing requirements.  Who cares what the requirement document says.
  • When testing, don’t get hung up on “completeness”.  Settle for adequate.  Same for requirement documents.  Example: Does your employee manual say “wear pants to work”?  Do you know how to get to your kid’s school without knowing the address?
  • Session-Based Test Management (SBTM) emphasizes conversation over documentation.  It’s better to know where your kid’s school is than to know the address.
  • SBTM requires 4 things:
    • Charter
    • Time-boxed test session
    • Reviewable results
    • Debrief
  • The purpose of a program is to provide value to people.  Maybe testing is more than checking.
  • Quality is more than the absence of bugs.
  • Don’t tell testers to “make sure it works”.  Tell them to “find out where it won’t work.”  (yikes, that does rub against the grain with my We Test To Find Out If Software *Can* Work post, but I still believe each)
  • Maybe when something goes wrong in production, it’s not the beginning of a crisis, it’s the end of an illusion.
Categories: Software Testing

Mind Traps That Aren’t Deadly, But Are Disruptive

QA Hates You - Mon, 08/06/2012 - 03:11

Psychology Today has an interesting article called “Deadly Mind Traps: Simple cognitive errors can have disastrous consequences—unless you know how to watch out for them” that describes a number of, well, cognitive errors that have caused people to make mistakes that lead to their doom.

Although we’re not really working with literal doom on a daily basis in software development (although, increasingly, the errors we introduce, uncover, and/or do not discover in software do lead to literal doom), the same mental errors can lead to problems within software development projects and within the quality assurance within them.

So let’s look at the traps listed in Psychology Today:


Hall [a mountaineer who summitted Mount Everest but died on the descent] fell victim to a simple but insidious cognitive error common to many types of high-pressure undertakings. I call it “redlining.” Anytime we plan a mission that requires us to set a safety parameter, there’s a risk that in the heat of the moment we’ll be tempted to overstep it. Divers see an interesting wreck or coral formation just beyond the maximum limit of their dive tables. Airplane pilots descend through clouds to their minimum safe altitude, fail to see the runway, and decide to go just a little bit lower.

You see a lot of this sort of behavior on timelines, where close to milestones or the end of cycles, someone introduces changes or whatnot. Sometimes you see this when someone wants to use some unproven, misunderstood piece of technology to solve a problem, but the learning curve proves too steep to get things done on time, or someone has to kludge a solution to meet the need at the last minute.

You know how to fix this? Understand and adhere to limits.

The Domino Effect

Similar tragedies play out time and again when people try to rescue companions. A teen jumps from a dangerous waterfall and disappears; his buddies follow, one after the other, until they all drown. A firefighter goes into a burning building to rescue a comrade; another goes in after him, then another.

In each case, the domino effect results from a deep-seated emotion: the need to help others. Altruism offers an evolutionary advantage but can compel us to throw our lives away for little purpose. “In stressful situations, you see a failure in the working memory, which is involved in inhibiting impulses,” says Sian Beilock, a psychology professor at the University of Chicago. “People lose the ability to think about the long-term consequences of their actions.”

You see this in software development in many cases: When something’s going badly, the management throws more people at it, and they do something badly, only less efficiently (see also Parkinson’s Law). Or a company chases features that other companies put into their software or chase some small bit that competitors or the industry press follows.

You see this, too, when too much effort is put into one channel of testing, where one expends a lot of effort in automated testing and continues pursuing that goal to the exclusion of other, more relevant exploratory testing or whatnot.

To keep from being another domino to fall, the article warns you not to leap instinctively into the manure machinery (truly, that’s what it suggests), but that you take a step back and consider alternative actions if you see cascading failure before you. The saying attributed to Einstein, that insanity is doing the same thing over and over again and expecting different results, applies. So does your mother’s “If everyone else jumped off the bridge, would you jump off the bridge, too?”

When you see something fail, don’t try it again. Try it again, differently.

Situational Blindness

As GPS units and satellite navigation apps have flourished over the past few years, there’s been a spate of similar cases, in which travelers follow their devices blindly and wind up getting badly lost. In each case, the underlying mistake is not merely technological but perceptual: the failure to remain aware of one’s environment, what aviation psychologists call situational awareness, or SA. People have always had difficulties maintaining SA, psychologists say, but the proliferation of electronics, and our blind faith that it will keep us safe, has led to an epidemic of absentmindedness.

“A big element in SA is paying attention to cues,” says Jason Kring, president of The Society for Human Performance in Extreme Environments. “If you’re focusing just on that GPS unit, and you see that little icon moving down the road, and say to yourself, OK, I know where I am, technically, that can be a big problem, because you’re not looking at the world passing by your windshield.”

Full situational awareness requires incorporating outside information into a model of your environment, and using that model to predict how the situation might change.

You can lose situational awareness in many ways in software development and testing. You can narrow your focus only to the problems in your niche of the process, your set of features, or what have you. You focus on the trees, and you can’t see the Christmas tree lot.

So to combat this, you need to pay attention to higher level considerations. You need to know as much as you can about the industry, the customers, and your organization as you can, and to factor as much as you can into your decisions as to how to proceed. If you know what your customers, that is, your users, really need and really do over the course of the day, you can weight your testing to cover those needs more than the features your business analyst forced onto the software because your customer’s VP, hired from outside the industry, wanted them.

Double or nothing

The runaway-balloon problem is a manifestation of our irrational assessment of risks and rewards. As Daniel Kahneman and Amos Tversky first pointed out back in 1979, we tend to avoid risk when contemplating potential gains but seek risk to avoid losses. For instance, if you offer people a choice between a certain loss of $1,000 and a 50-50 chance of losing $2,500, the majority will opt for the riskier option, to avoid a definite financial hit. From the perspective of someone dangling 20 feet in the air, the gamble that they might be able to ride the gondola safely back down to the ground seems preferable to a guaranteed pair of broken legs. But in the moment, they can’t factor in the price they’ll pay if they lose.

We see this in software development when someone takes a flier because he or she thinks he or she has nothing to lose. A developer doesn’t like the way the code works or encounters some difficulty with it, so he spends all weekend rewriting several weeks worth of several developers’ work before the Monday client meeting instead of just putting a little rouge on the hog and acting from the insight gleaned at the client meeting. Or a company rushes headlong into a move to a new technology or new cloud strategy against the reluctance of its clients. And so on.

To combat this risky frame of mind, you’ve got to, again, establish limits and procedures to make sure that people know when they can take risks on their own–and when they can’t.

Bending the map

Such errors of overconfidence are due to a phenomenon psychologists call confirmation bias. “When trying to solve a problem or troubleshoot a problem, we get fixated on a specific option or hypothesis,” explains Kring, “and ignore contradictory evidence and other information that could help us make a better decision.”

Confirmation bias is very common in our field. Technological fanboys think that their favored language or framework is the best, and that all evidence points in that direction. The whole sales, marketing, and company magazine teams live just to provide this confirmation bias about how well your organization is doing and how everything is going swimmingly.

To combat confirmation bias, you’ve got to learn to distrust yourself. You’re not that smart. The way you think something is going isn’t the way it is going. Well, maybe it is. You’re not even lucky enough to get it wrong 100% of the time. When you encounter a new piece of evidence or information, don’t just try to fit it into your gestalt or dismiss it. You’d better figure out why that little strange thing happened, and what it can mean for your whole endeavor.

(Link seen in Reader’s Digest, where the things are given in a different order, perhaps to better track Internet content thieves. And, yes, I do read Reader’s Digest. I am an old man.)

Categories: Software Testing

Quick Testing Challenge

Alan Page - Sun, 08/05/2012 - 13:08

Some updates and clarification below.
And a NEW HINT below too

Recently, I was discussing the diagnostic, debugging, and troubleshooting aspects of software testing, and this morning, while playing with Visual Studio 2012, I created a quick little exercise where the solution relies on these skills.

This zip file contains an app (theapp.exe), and the the vc 2012 runtime dll (both for 32-bit Windows) (note, this should be all that is needed, but if you get a load error, let me know in the comments or on twitter).

But even with the runtime, the app won’t launch. I’m sure that many of you will be able to figure out why –but I’m curious to know how you find the answer. When faced with problems like this, I have a favorite method – but I’m curious what other people do.

Two things to add:

  1. This application, does indeed launch. When it runs, you’ll have a generic windows application, and a message box indicating success.
  2. Some of you are probably thinking, “If the app doesn’t even launch, I’ll talk to the developer, since he’ll know what changed.”. But consider this scenario (which is the exact scenario that led me to create this app). You are the tester for Application Awesome. Your application is…Awesome. However, a user tells you that after installing Application Awesome, another app they love (TheApp.exe) fails to launch. WTF happened?
  3. HINTS: Think for a moment about what sorts of things an application may check on launch. Could one of them be failing? How would you (could you) know?
Categories: Software Testing

Let it happen

Alan Page - Sat, 08/04/2012 - 11:17

I come across this frequently enough that I’m sure I’ve blogged about it before, but context dictates that I do it again. The story I hear goes pretty much like this:

My team really needs to improve X, but nobody is taking responsibility for it. The fix is obvious – we need our exec / manager / project owner to mandate that we do X and measure Y & Z to make sure people are doing the right thing.

Here’s a concrete example. I was on a team once where quality was in the toilet. We couldn’t even get a build to run for over a week. The solution was require everyone on the team to perform a list of actions before every check-in. Of course, the list including using flaky tools, and getting sign off on every step, even if the step wasn’t applicable to a particular check-in. Given the extra work required for check-in, developers queued up weeks worth of check-ins for one big check-in rather than go through the steps in the checklist multiple times. Day to day build quality was slightly better, but velocity was way down. While slower velocity is not necessarily a bad thing, the big problem was culture. Someone honestly thought the mandates would create a culture of quality – but the only culture they created was a culture of finding ways to skirt the damn checklist. Team morale sucked, and the product we built sucked too.

To be fair, I’ve seen some successful mandates in my career. The push for improvement in every developer to write secure software at MS worked. But that culture change was also pushed by huge monetary losses.

In general (and in every other case I can think of), mandates don’t work – especially if you are using the mandate to change culture. Yet I see mandates suggested as a solution for changing culture time and time again. I just don’t get it.

The folks at 37 Signals say it best, “You don’t create a culture. Culture Happens.”

So help let it happen. Instead of a mandate, help your team see where you want to go. Make it personal. Appeal to their own pride and values. Show them how change will help them. Find allies who think like you do. Experiment.

Let it happen. Make it happen.

Categories: Software Testing