First, thanks for your thoughts, prayers and concerns after the tornado this week. I helped a friend go through the rubble of her former home on Monday. It was amazing to see her positive attitude. As we arrived at the debris, she said, "Welcome to my humble abode."

While there, she was interviewed by a Brazilian TV crew who also remarked to me how impressed they were with her attitude. Yes, that is inspiring, and many others here are also holding up well even in trying times.


 Also, thanks for those of you responded to the test professional survey last week. I ended up with 100 responses. I need a while longer to write the article (and perhaps white paper) on this, but I thought you might want to see some early results.



I am working on the article and should have it by next week. To me, the early impressions are that (at least to those that responded) the majority of testers see themselves as professionals and it is important to have that view. Management may not see testers as professionals as much, but there is still a significant percentage of managers that do find importance in the view of a test professional as opposed to some other view. I think this has some larger, and very good, implications for those of us in the field of software testing.

More to come soon. Of course, I would love to get your thoughts on these findings.

For those in the U.S., have a great Memorial Day. Remember those who have given their lives for our freedom. Remember those who are rebuilding in Oklahoma.

Thanks as always for being the best at what you do!

Randy

If you've not heard of the term 'Test is dead' then Google [did] it.


Test Is Dead is dead... Long live Test!!

The cat’s out of the bag – I’m popping out of my no-conference bubble, and making an appearance at a testing conference (STAR West in October). The theme of the conference is “Be a Testing Rock star”, and while I think that theme begs for an appearance from Michael Larsen, I’ll do my best to live up to the hype.

I’ll be giving a keynote on testing the Xbox. While I’m certain I’ll have interesting stories and pragmatic examples I can share in a generic fashion, I’m hoping I can share much, much more. I don’t have clearance yet to share too much about the Xbox One, but assuming I can get my proposals in place, it should be a pretty exciting talk to share. By the time STAR rolls around, I’ll have passed my 20 year anniversary working on software, and working on this product has definitely been the highlight of my testing career.

I’m also signed up to give a half-day workshop on testing. Literally – “on testing”. The title of the workshop is Alan Page: On Testing, and I’ll take the time to share some good stories and examples of new testing ideas, but I expect the attendees to drive the content. More details on that as we get closer to the event, but I expect it will be a ton of fun, and especially interesting to those folks who just want to suck up some new and interesting ideas.

More as the conference gets closer.

(potentially) related posts:

1. Twinkle Twinkle I’m back from STAR
2. Happy Birthday HWTSAM
3. Stop Guessing about my STAR presentation

One veteran recommends automating all mobile software tests. Another expert says to focus on planning and automate only where necessary.

I chair an advisory council for a community of senior testers at Microsoft. We have a variety of events ranging from talking heads to open space events to panels to whatever type of event we think is the most different than the previous one.

Yesterday, we had our fifth annual “soapbox” event, a lightning talk-ish event where speakers are encouraged to share a rant or opinion with the rest of the community (five minute limit). Because I’ve been thinking so much about my Tear Down the Wall post, I decided to give the five minute version for my peers.

I began with the story of Lloyd Frink (from this post), and  compared the world of technology in 1979 to the world today. I talked about how “walls” get in the way, and how boxing people into roles (especially roles of gatekeeper and fake-customer) are fruitless at best. I closed by sharing Trish Khoo’s quote from her take on all of this:

It would be somewhat revolutionary, if it weren’t already happening.

This was a point I really wanted to drive home. Many of the senior testers at Microsoft have only been at Microsoft, and few pay any attention to software development outside the Borg, and I think there’s a lot to learn from companies and teams that have successfully blurred and erased lines and boxes from their development teams.

As of today, I’m still employed, so I’ll keep ranting and see what happens.

For MS folks, you can find the talk on //resnet (search for soapbox). The whole event is good, but my talk starts about 19:30 in.

(potentially) related posts:

1. Why?
2. Fall Travel
3. The Ballad of the Senior Tester

In the previous post we presented problems encountered by our client TescaraHats (name changed for commercial reasons), a European market leader in manufacturing customized hats. (...) We argued that while improving search engine ranking is important, you should never forget about the performance and usability in an e-commerce application. In this episode of our e-commerce performance series, we will analyze the impact that the backend performance has on your online sales.

Well…yes.  I would.

The most prolific bug finder on my team is struggling with this question.  The less the team decides to fix her bugs, the less interested she grows in reporting them.  Can you relate?

There is little satisfaction reporting bugs that nobody wants to hear about or fix.  In fact, it can be quite frustrating.  Nevertheless, when our stakeholders choose not to fix certain classes of bugs, they are sending us a message about what is important to them right now.  And as my friend and mentor, Michael Bolton likes to say:

If they decide not to fix my bug, it means one of two things:

• Either I’m not explaining the bug well enough for them to understand its impact,
• or it’s not important enough for them to fix.

So as long as you’re practicing good bug advocacy, it must be the second bullet above.  And IMO, the customer is always right.

Nevertheless, we are testers.  It is our job to report bugs despite adversity.  If we report 10 for every 1 that gets fixed, so be it.  We should not take this personally.  However, we may want to:

• Adjust our testing as we learn more about what our stakeholders really care about.
• Determine a non-traditional method of informing our team/stakeholders our bugs.
• Individual bug reports are expense because they slowly suck everyone’s time as they flow through or sit in the bug repository.  We wouldn’t want to knowingly start filling our bug report repository with bugs that won’t be fixed.
• One approach would be a verbal debrief with the team/stakeholders after testing sessions.  Your testing notes should have enough information to explain the bugs.
• Another approach could be a “supper bug report”; one bug report that lists several bugs.  Any deemed important can get fixed or spun off into separate bug reports if you like.

Complexity is the new reality of web and mobile applications with almost no new release going out without the addition of services and applications spread across many different companies. But the reality of this new interrelationship is still the same: If a third party internet outage or issue occurs,, your brand is the one that [...]

Hi Folks,

Thanks to everyone who has contacted me concerning the tornadoes here today. Thankfully, we were spared, barely.

I was at a client in OKC to attend at 2:00 meeting, which was canceled. It was looking stormy, so I decided to head home. About when I arrived home, the tornado sirens started to sound.

As the tornado approached, I was on the back porch watching it come directly toward us. Just like May 3rd, 1999 all over again. So, I hustled Janet and our two pugs into the car and drove north. We don't have a storm shelter (I think we will soon). The tornado took a turn toward the east, so no damage at our place. But, the loss of life and damage is terrible, especially the children at the school.

(This is a short video I shot from our car. You can see the tornado moving from the right of the screen to the left behind the Homeland store.)

Both our sons and families are fine, although had Ryan (our oldest) and his family not moved 6 years ago, they would have been wiped out and our oldest grandson would have been at the school where the 7 children died.

I don't what it is about Moore. It's like a tornado magnet. It's just really bad here right now. Your prayers are needed.

The people here are tough and resilient. We will rebuild and go on, but the loss of life is the worst part. 51 lives lost as of this writing.

Thanks,

Randy

Update: May 21

Here are some more pictures:

This is the path of the tornado:


















Here is what I was looking at on radar on my iPad when I decided it was time to bug out:

















Here is a pretty dramatic picture after the first tornado. This is a second lowering. This didn't form into a tornado:

Killswitch Engage, "In Due Time":

Software quality assurance is gaining respect as a profession -- but do QA testers have the scripting and security skills the role now requires?

I am working on an article about software testing as a profession. This is a controversial topic to some people and I would like to get your thoughts.

I have created a very short (10-question) survey that will help me write this article. It will only allow 50 respondents, but I would like to invite anyone to participate at:

http://freeonlinesurveys.com/s.asp?sid=prgud68m40uqzws270184
I will share the results and also let you know when the article is out.

Thanks for your help!

Randy

It’s a cliché, I know.  But it really gave me pause when I heard Jeff “Cheezy” Morgan say it during his excellent STAReast track session, “Android Mobile Testing: Right Before Your Eyes”.  He said something like, “instead of looking for bugs, why not focus on preventing them?”.                 

Cheezy demonstrated Acceptance Test Driven Development (ATDD) by giving a live demo, writing Ruby tests via Cucumber, for product code that didn’t exist.  The tests failed until David Shah, Cheezy’s programmer, wrote the product code to make them pass. 

(Actually, the tests never passed, which they later blamed on incompatible Ruby versions…ouch.  But I’ll give these two guys the benefit of the doubt. )

Now back to my blog post title.  I find this mindshift appealing for several reasons, some of which Cheezy pointed out and some of which he did not:

• Per Cheezy’s rough estimate 8/10 bugs involve the UI.  There is tremendous benefit to the programmer knowing about these UI bugs while the programmer is writing the UI initially.  Thus, why not have our testers begin performing exploratory testing before the Story is code complete?
• Programmers are often incentivized to get something code completed so the testers can have it (and so the programmers can work on the next thing).  What if we could convince programmers it’s not code complete until it’s tested?
• Maybe the best time to review a Story is when the team is actually about to start working on it; not at the beginning of a Sprint.  And what do we mean when we say the team is actually about to start working on it?
• First we (Tester, Programmer, Business Analyst) write a bunch of acceptance tests.
• Then, we start writing code as we start executing those tests.
• Yes, this is ATDD, but I don’t think automation is as important as the consultants say.  More on that in a future post.
• Logging bugs is soooooo time consuming and can lead to dysfunction.  The bug reports have to be managed and routed appropriately.  People can’t help but count them and use them as measurements for something…success or failure.  If we are doing bug prevention, we never need to create bug reports.

Okay, I’m starting to bore myself, so I’ll stop.  Next time I want to explore Manual ATDD.

The saying “if it doesn’t exist on the Internet, it doesn’t exist“ is reigning truer every day. Nowadays, it is hard to imagine most businesses without an e-commerce platform, let alone without a web presence at all. Since e-commerce is becoming the new standard, e-commerce performance needs to be at its best. In this blog series, I have come up with several ways to ensure your company's e-commerce performance success, including: avoiding unnecessary network load, reducing number of (internal) HTTP errors, improving backend performance, understanding your clients, ensuring scalability of e-commerce site and finally understanding sales results through conversion rate.

It’s interesting when I go back and look at the number of posts where I talk about what I do, what testing is to me, and how testing is changing. Ever since the Telerik Test Summit (telsum), I’ve been thinking even more about testing and how it fits into software development. When I wrote this post, and added on a bit to it here, I was pondering the same thing. When I (last) blogged about the future of testing, I (likely subconsciously) was thinking about the same thing.

A few things happened recently to make everything really click for me. One, was this article, asking for universities to offer software testing degrees – as a means to “train our testers”. Besides the fact that the author thinks that Universities are vocational schools where we train people (rather than teach people), I think it’s difficult to have a degree in something that the industry can’t define consistently (yes, I know there are test degree programs out there now, and I could make the same argument about those).

Then, just over a week ago at Telerik’s testing summit, we led off the open-space session with a discussion on test design. I shared some of my ideas, including how I use mind-maps to communicate test design. Towards the end of the discussion, Jeff Morgan (@chzy) asked, “Isn’t it just ‘Design’”. What Jeff was implying was that many of the details of test design should be equally at home when thinking of feature design. Rather than have programmers determine how it should work, and then have testers determine how it should be validated and explored, that those tasks could occur (for the most part) simultaneously – e.g. “To implement this feature, I need to read a value from the database, but I also need to make sure I account for and understand performance implications, whether or not the values will be localized, what the error cases are, etc.).” Much (not all) of test design can be considered when designing code – so why don’t we consider test design and code design simultaneously?

After this discussion (and for many, many hours since telsum), I’ve been thinking about programming and testing and how the two roles can work together better – and began to wonder if the test role may be detrimental to software development.

Three things are important to note before you scroll down and leave me hate-comments.

1. I didn’t say test is dead
2. I said “role”, not activity
3. I purposely used the weasel-word “may” – that statement won’t always be true

And now for some much-needed elaboration.
The Wall

There’s a wall between testers and programmers. Programmers “build”, testers “break” – but passing features back and forth between one group and another is wasteful. It takes time and extra communication to traverse the wall. I spent at least the first few years of my career on one side of the wall catching the crappy code programmers threw to me. Then, I found unit-level bugs and threw those bugs back over the wall to the programmers. Then, we played this wasteful game of catch until it was time to ship. Recently, I saw a fairly prominent tester mention that their bread and butter was participating in a game of bad-code and happy-path-bug crap-catch. While I’m happy that there are easy ways to make money in testing, I’d rather tear my eyelids off than do work like this. I like what I do because it’s hard. Finding happy path bugs in crappy software isn’t hard. It’s boring and practically demeaning.

The Neotsys blog latched onto one of my recent posts in their testing roundup. They said that my team at Microsoft has moved to the “whole-team” approach, but that’s not true. We still have testers and programmer roles – and while programmers frequently write test code and testers frequently write product code there’s still a wall. Programmers are still responsible for writing product code, and testers are still responsible for testing – we just don’t have a problem crossing those lines.

We still have a wall. It’s a small wall, but it’s still there.
Tear Down the Wall

If you played with my recent thought experiments of programmers who can test and testers who can program, it’s not much of a reach to picture a team where there are no programmers or testers – just contributors (or developers or engineers, or whatever you want to call them). If you have a hard time imagining that, imagine your current team with no walls between role responsibilities. Many of you may do the same thing you’re doing today, but with the walls gone, I bet you could make better software – faster.

Figure out what needs to get done, and get it done. Leverage the diversity of the team. If someone is a specialist, let them specialize. If they’re not, let them optimize for their own skills.

Tear down the wall.
Titles

On a quick side note, testers worry (far too much) about their titles. This recent blog reminds me of the idiocy of tester titles (and I’ve discussed it here). There will always be a place for people who know testing to be valuable contributors to software development – but perhaps it’s time for all testing titles to go away?
The Future?

I’ve written in the past about where test is going – toward data analysis, non-functional emphasis, etc., but I think I was at least partially wrong. What software teams need in the future is team members who can perform the activities of programming, testing, and analysis – together.

I saw a slide deck recently that stated, “Agile Testers frequently can’t keep up with Programmers”. Good software development can’t happen when you serialize tasks – the team needs to work on this stuff – together.

I’ll always be deeply involved in the activity of software testing, but I don’t know if the role or title exists in my future. After nearly 20 years of being a tester (and likely several more with that title), I’m admittedly going out on a bit of a limb with that statement. Despite my title, I want the walls to go away.

Testing can’t be something that happens after programming is complete any longer. Testers aren’t “breakers” any more – but experts in the testing activity are (or need to be) as critical to making software as anyone else on the team.

Since it’s mostly testers who read this blog I challenge all of us to shed any remnants of tester vs. programmer and figure out how to tear down the wall.

I’ll let you know how my own battle goes.

(potentially) related posts:

1. What is Testing?
2. Activities and Roles
3. Yet another future of testing post (YAFOTP)

With our new service platform and the convergence of dynaTrace PurePath Technology with the Gomez Performance Network, we are proud to offer an APMaaS solution that sets a higher bar for complete user experience management, with end-to-end monitoring technologies that include real-user, synthetic, third-party service monitoring, and business impact analysis. To showcase the capabilities we [...]

So I took This BBC News grammar test, and I got a perfect score, natch:

And a bug, natch.

Come on, kids. One simply must check numbers greater than one digit (or more) to make sure they fit inside your little design elements.

Some days, I feel like the Dowager Countess, sitting here marking pointed remarks as I watch the old order of people caring about this stuff falling apart.

Stone Sour with “Absolute Zero”

Good day!

Soft skills, including influence, leadership and communication, were among the important topics on the agenda at STP Con Spring 2013 in San Diego.

By Andrew Trenk
This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office.

There are typically two ways a unit test can verify that the code under test is working properly: by testing state or by testing interactions. What’s the difference between these?

Testing state means you're verifying that the code under test returns the right results.
public void testSortNumbers() { NumberSorter numberSorter = new NumberSorter(quicksort, bubbleSort); // Verify that the returned list is sorted. It doesn't matter which sorting // algorithm is used, as long as the right result is returned. assertEquals( new ArrayList(1, 2, 3), numberSorter.sortNumbers(new ArrayList(3, 1, 2))); } Testing interactions means you're verifying that the code under test calls certain methods properly.
public void testSortNumbers_quicksortIsUsed() { // Pass in mocks to the class and call the method under test. NumberSorter numberSorter = new NumberSorter(mockQuicksort, mockBubbleSort); numberSorter.sortNumbers(new ArrayList(3, 1, 2)); // Verify that numberSorter.sortNumbers() used quicksort. The test should // fail if mockQuicksort.sort() is never called or if it's called with the // wrong arguments (e.g. if mockBubbleSort is used to sort the numbers). verify(mockQuicksort).sort(new ArrayList(3, 1, 2)); } The second test may result in good code coverage, but it doesn't tell you whether sorting works properly, only that quicksort.sort() was called. Just because a test that uses interactions is passing doesn't mean the code is working properly. This is why in most cases, you want to test state, not interactions.
In general, interactions should be tested when correctness doesn't just depend on what the code's output is, but also how the output is determined. In the above example, you would only want to test interactions in addition to testing state if it's important that quicksort is used (e.g. the method would run too slowly with a different sorting algorithm), otherwise the test using interactions is unnecessary.

What are some other examples of cases where you want to test interactions?
- The code under test calls a method where differences in the number or order of calls would cause undesired behavior, such as side effects (e.g. you only want one email to be sent), latency (e.g. you only want a certain number of disk reads to occur) or multithreading issues (e.g. your code will deadlock if it calls some methods in the wrong order). Testing interactions ensures that your tests will fail if these methods aren't called properly.
- You're testing a UI where the rendering details of the UI are abstracted away from the UI logic (e.g. using MVC or MVP). In tests for your controller/presenter, you only care that a certain method of the view was called, not what was actually rendered, so you can test interactions with the view. Similarly, when testing the view, you can test interactions with the controller/presenter.