Syndicate content
notes and rants about testing and quality from alan page
URL: http://angryweasel.com/blog
Updated: 20 hours 28 min ago

Surface RT and EuroSTAR recap

Sat, 11/10/2012 - 15:50

I’ve been home for a few days from by brief trip to Amsterdam for EuroStar. As I mentioned in a previous blog post (or two), I used my Surface RT as my only device for the trip. Overall, it went quite well. I was able to accomplish most of the things I wanted to, including using my RT to drive my presentation. No glitches in the OS, and no glitches in PowerPoint. Some of my highlights and lowlights include:

Highlights

  • Battery lasts a long, long time.
  • I brought a small Bluetooth keyboard (I’m actually typing on it now), and although the flat keyboard is pretty cool and remarkably usable, a keyboard with keys that go up and down is just much easier to type with for long periods at a time.
  • As I mentioned above, everything just works. Between the office apps, IE10, apps from the marketplace, and the OS itself, I was able to stay in touch, and get some work done.
  • It’s just cool. I used it in both keyboard, and pure tablet mode quite a bit and never had a glitch.

Lowlights

  • I could be a homer and say there were no lowlights, but I’m too honest for that. There are just a few things I didn’t like, or missed by not using a “traditional” laptop
  • The mail app is usable – but that’s all. It seems a bit sluggish to respond (could be the fancy animations), requires that I view in reading pane, and is probably a bit harder than it should be to navigate. I suppose it’s slightly better than mail on a smartphone, but I expected a bit more.
  • There were a few times when it would have helped if I could have accessed some files on the MS corporate network – i.e. there’s no VPN feature in Surface. This is hardly a deal breaker, and I’m probably a bit out of the ordinary for wanting to check out files remotely, but I thought it was worth mentioning.
  • This has been mentioned by others, but typing on your lap is a bit of a pain. For me, the Surface bounces as I type. I didn’t have any problems with the screen angle in any position though.

The Conference

This was my first EuroStar, and I had a fantastic time. The conference begins with a day and a half of tutorials, followed by two and a half days of talks. My talk, “Test Innovation for Everyone” was the opening keynote. I had a good time with the talk, and received some good feedback. . If I ever give the talk again (something I rarely do), I’d clean those bits up a bit. There are a few places where I could have stitched together things a bit smoother, but overall I was happy with it.

Esther Gons drew a picture of my presentation that sums up my topic pretty well.

Esther Gons Drawing of my EuroSTAR prezo

I also met a lot of really cool people – many who I knew of from blogs or twitter, but had never met in person. I also met some folks from Skype there who all seemed pretty sharp. I would try and list the names, but there are way many to list, and honestly, if you saw the list of testing celebs I got to hang out with it would just seem like name dropping anyway.

I had a great time, and hope I get a chance to see Europe and some of the great testers there soon.

Categories: Software Testing

Surface Update

Mon, 11/05/2012 - 02:31

If you read my last post (sorry, no link – I’m not ready to try that yet), you’ll know that I’m attempting to use my Microsoft surface device as my only electronic device on my trip to Amsterdam for Eurostar.

So far, it’s working ok. The cover/keyboard works better than I though tit would – although I seem to have a problem with it registering the space bar a lot. You really do need to be a good touch typer to have confidence with this thing.

My presentation tweaks aren’t quite done, so I’ll have to put this thing through a few more paces to see what I really think. I’m also curious to see how limited the Surface is with no internet connection (I assume I’ll find myself in that situation on this trip, but we’ll see).

Boarding in a few minutes – more updates from Amsterdam.

ugh – first problem is that the wordpress app won’t publish my post…

Continued: Because I can’t post, I’ll continue with my update. My first bad experience has little to do with the surface – it’s that the wordpress app is lame. I won’t describe the depths of the lameness, but I will say that I’ll likely post this directly from my blog rather than the WP app.

I just spent 30 minutes or so tweaking my slides – that was a good experience. Typing is still a bit weird (and I still miss the space bar about every ten words or so), but for formatting, moving slides around, etc. the Surface works remarkably well. I find the screen size and resolution seem to be just about perfect for working on an airplane (I’ll find out if I say the same for hotel rooms and coffee shops in an upcoming post).

More later (when I find Wi-Fi).

Categories: Software Testing

My Surface Challenge

Tue, 10/30/2012 - 23:33

I had a not-so-good week last week.Lot’s of stuff too private for blog sharing, but let’s just say it was a cascade of dark and bad times that nobody should have to go through. One of the bits of bad luck was that the LCD on my laptop broke. Normally, this wouldn’t be so bad. Most of the time (i.e. unless I’m traveling), I log into my laptop via terminal server, so a broken screen has no impact. This is my “work” laptop, and it’s overdue for an upgrade, so normally, everything would line up well.

This is not a normal week. I leave for Amsterdam to speak at Euro STAR on Sunday, and there’s no way to get a replacement laptop through work on time. I looked into getting the screen replaced, but at $500, I didn’t feel like it was cost effective – I thought the same of renting a computer for $120 for the week. I have a strong network at MS, and pinged some of my colleagues on the Windows team for spare hardware, but that well was dry too. I begged admins, IT, and anyone I could find to let me use a laptop for the week, but everything fell through.

So, I figured it was time to buck up and buy a personal laptop. Microsoft is giving me (and every employee) a Surface RT, but we won’t get those until January. I looked around a bit, and felt a little strange paying (at least) $700 for a laptop that I probably wouldn’t use much after this trip (have you figured out yet that I hate spending money?).

In the end, I decided to go all in – to jump on the Windows Surface bandwagon and buy a Surface RT (yes – the same device I’ll get for from MS in a few months). If I like it, it will be nice to have two (if I love it, I’ll give one away!) The challenge I’ve given myself is to see how productive I can be using the Surface on the plane and in my hotel room – and using the Surface as my powerpoint source for my keynote at the conference. It’s going to be a fun few days, and I expect I’ll blog a bit about how it’s going along the way. I only hope that if my experience goes south that the Windows bigwigs aren’t going to get mad.

We’ll see what happens.

Categories: Software Testing

Conference Stuff

Sun, 10/14/2012 - 12:17

Some news about my involvement in upcoming conferences I’m probably overdue to share.

First of all, I’ll be giving a keynote at Euro STAR on November 6th. This is my first trip to Euro STAR, and my first visit to Amsterdam (at least my first visit where I’ll leave the airport). There are a lot of talks I plan to attend, but I’m a bit bummed that I’ll miss the last day of the conference, and will miss a talk by Alexandra Schladebeck on What Agile Teams Can Learn From World of Warcraft. I’ve used this metaphor many times over the years, and I’m excited to see that it’s not just me that sees the connection.

Beyond that, I’m done with conferences for the next eleven months or so while I go back to paying full attention to my family and my day job (but I may give a webinar or two this winter given sufficient interest).

I’m fortunate to have a team and manager who support me attending a few conferences a year, but while they’d probably be supportive of me agreeing to attend more conferences, my day job is much cooler than most, so I’m cutting way back on my travel so I can do even more cool stuff. I have made a commitment to attend STAR West next fall, but other than that, I’m consciously making no commitments (although I would attend a good peer conference if one came up).

Finally- I want to mention that Swiss Testing Day – one of the best conferences I’ve ever attended (both in organization and in content) is coming up. I’m not attending this year, but I do encourage anyone who is, or will be near Zurich on March 13, 2013 to consider attending – I am confident you won’t be disappointed.

Categories: Software Testing

One of my Favorite Bugs

Sat, 10/13/2012 - 07:58

On twitter, @SeanNoxious asked me about my most memorable bug. The answer is way too long for twitter, so I thought I’d document one of my favorites here.

When I first joined Microsoft way back in 1995, I was a tester for networking components on Windows 95. One of the areas I owned was testing networking on Japanese, Chinese, and Korean versions of Windows.The early versions of windows and all of the 9x versions of windows didn’t have Unicode. Character sets were multi-byte on those systems, meaning that characters may be made up of one or two bytes. Alpha-numeric characters were usually single-byte (wide / two byte versions also existed (when a character had both a single-byte and double-byte version, the single-byte version was referred to as the half-width character. There’s a lot more to say about double-byte characters, but I’ll skip ahead to the good part.

The way windows could tell the difference between a double-byte character and a single-byte character was the lead byte. A certain range of characters are designated lead bytes. They always indicate that the lead byte, and the next character in the stream are combined to make a double-byte character. The Japanese character for ten (juu) has a hex value of 0x8F5C. The 8F indicates that the following characters are part of a double-byte character. The interesting thing about this particular character (and all characters with an 5C trail byte) is that 5C is the hex value for a backslash – a character very interesting for network testing on Windows.

There were other characters with interesting trail bytes, but 5C characters were predominant in my testing, and I found a lot of bugs. One day, I was testing copying long files with 5C trail bytes to a Japanese version of Windows NT 3.51 (newly released!), and my test hung. I figured someone else was using the server and rebooted it or was debugging it for some reason, so I walked down to the lab to check it out; and it was sitting at a blue screen. I stared for a minute before doing what every tester does. I rebooted it, and ran back to my office to see if I could do it again.

Sure enough, I had found a serious bug in the NT networking components. One that I knew was tested by people way more experienced than I was, and I was pretty excited.We found other bugs in NT, and plenty of multi-byte bugs in networking, but crashing a remote server has to be one of my favorite finds.

I have another cool backslash story, but I’ll save it for another post.

Categories: Software Testing

The Rise of the Customer

Sat, 09/29/2012 - 10:10

I spoke recently on the topic of “Customer Focused Test Design” (synopsis: customers don’t really care how much functional testing testers do, or how many bugs they find. A test approach that favors scenarios, -ilities, and customer feedback is better; and a bunch of examples for emphasis/proof).

Part of the approach I suggest (and have had some success with), is doing testing of performance, reliability, security, privacy, world readiness, and usability (including accessibility) early in the product cycle. Early, as in don’t bother with functional testing, do the ilities instead. The premise (and my experience) is that testing ilities tests the underlying functionality by default, and that programmers are (in general) doing a better job testing for functionality during code development. For more details, I’ll probably have to revive the whole talk – and that’s not what this post is (intended to be) about.

There’s one –ility that is critical to the customer perception of quality, but it’s not quite like the others – supportability. Customers hate problems, but they love finding solutions to those problems. Online forums, customer connection programs, twitter, facebook, and other online platforms are quickly becoming support forums for a lot of software. Software companies who engage with customers actively are creating happy customers.

Zappos has created a fantastic culture of customer service and have won customers for life (I’m one of them) by actually caring about their customers.

Recently, I moaned on twitter that I couldn’t get my (aisle) seats confirmed for a long flight from Australia to Seattle on a flight booked through Orbitz. Within minutes, @OrbitzCareTeam contacted me on twitter and booked and confirmed my seats. I fly a lot (too much), and I can book my flight with any discount airline– but Orbitz will get my business.

Last month, I visited an Xbox call center, talked to some of the employees there, and listened in on some calls. I was ecstatic to see that nobody works from a script and there are no quotas or other incentives to push people through the system. I listened to several calls where the support folks took their time helping people solve problems step by step (on occasion, converting customer rage into customer appreciativeness). Most interesting was that none of the calls I listened to had anything to do with the Xbox console. Our support people answered questions about entering codes for other manufacturers games, helped customers reset live id passwords, and a variety of other topics related to – but not part of the actual console. Every customer hung up happy – I was blown away.

If you want to be a successful software company, you have to care about customers. In addition to keeping them happy once they have used your software or service, you need to respond to their needs and give them what they need. In The Lean Startup, Ries drives home the point of iteration as a means to obtain validated learning; with the premise that any work that does not provide value to the customer to be wasteful. In other words, listen and learn – often.

Sometimes companies get it wrong. I recently had an issue with an application that was overlaying a decoration on my explorer icons. I found the overlays to be distracting and a detriment to my productivity. When I searched online for a workaround or solution (or sympathy), a representative of the company had this to say (bold mine).

Our thinking was that we want the app to be running in the background, all the time.  We want to to blend into your experience so you almost never have to interact with [the application] to check the status of things.  So we figured icon overlays were a subtle way to do this, while reassuring people that the app was indeed running.

Maybe we can add an option in the future to toggle the overlays, but honestly I wouldn’t hold your breath.  There’s plenty of other cool stuff we want to add first

Note the pronouns in bold. Our thinking…We want…we figured…we want… – these are the statements of Engineering Focused Engineering (note the intentional idiocy of that phrase)

To be clear, I like this product. And on my own, I figured out how to disable the overlays. I also don’t know if the statements above reflect the opinions of one person, or a whole software team…

But the statements above are the statements of an organization that doesn’t give a about their customers. What’s worse, is that this software’s main competition is an application that is praised for its customer focus. In my opinion, it’s unprofessional to approach software development in this way – unless of course you don’t actually want to have customers use your software.

Isn’t it about time to put customers first? Always? I’m all for shifting test design to favor scenarios customers care about, but successful software projects are going to require a customer focus for the lifetime of the product – from concept to development to deployment and beyond.

It’s time for the rise of the customer.

Categories: Software Testing

New Zealand & Australia, 2012

Sat, 09/22/2012 - 13:03

Note: no direct information on software testing or software engineering is in this post. This is purely a trip report (that happens to be related to a trip I took to talk about testing).

As I begin writing this, I’m in the middle of a trip to New Zealand and Australia. The main purpose of the trip was a bit of a speaking tour for SoftEd (for the STANZ and Fusion conferences). I gave a keynote and a workshop in Christchurch, NZ on Monday – repeated the same thing on Tuesday in Auckland, and then hopped across the ditch to Sydney to give a talk on Thursday and a workshop on Friday.

Christchurch

Although this trip marks my first visit to Australia, I spent three weeks in New Zealand almost exactly ten years ago. I arrived Saturday morning so I’d have a bit extra time to acclimate to the new time zone, but I didn’t really need it. Through a combination of horrible movie selections and three seats to myself, I managed to sleep quite a bit – and apparently just the right amount so that morning in Christchurch felt a lot like morning in Seattle. The hotel was nice – as far as hotels based on medieval themes go (this being my first medieval themed hotel, I was quite happy). I went for a run, showered, explored a bit, and then went in search of dinner. Although I spent a few days in Christchurch ten years ago, I wasn’t sure how much I’d remember. I also wasn’t prepared for just how devastating the earthquake in February, 2011 was. Nearly two years later, the bulk of the downtown area is still fenced off and unsafe. I found the hotel where I stayed in 2002 (closed), and a restaurant I remember (also closed). The weird part was how dead the downtown area was on a Saturday night. While I wandered around pondering the destruction, I suddenly realized that it was silent. Other than the flap of a few flags and banners in the strong Christchurch wind, there was nothing – complete stillness in an area that was bustling nearly 24-hours a day during my last trip. I eventually found my way back to the area near the conference hotel – apparently where at least some of the liveliness had moved, and found some food.

I was up early Sunday morning and went for a long run in the park. I met up with fellow conference speaker Elisabeth Hendrickson for a coffee run that evolved into a wonderful walking tour of the area surrounding the hotel. We managed to invite ourselves to a tour of a nature walk and had some great conversations about software engineering, testing, and our shared love of walking miles and miles in foreign cities. Although I’ve met Elisabeth before, one of the big benefits of this trip was getting a chance to get to know her better. She has a sensible holistic approach to software engineering that I really like (and unfortunately, don’t see nearly enough of among people in the testing community).

Monday was a blur – I think I went for a run in the morning, gave a keynote later in the morning, and delivered a workshop in the afternoon. Then we all flew to Auckland, where we checked in, and I decided I preferred sleep much more than dinner.

Auckland

Tuesday was more of the same. I changed my talk a bit – I changed the workshop a bit. Not only do I not like giving the same talk twice – I think I’m incapable of giving the same talk twice. I suppose this keeps things fresh (for me, at least). After the conference a group of us had a wonderful fish dinner at a nearby restaurant.

My flight on Wednesday wasn’t until 6:00pm, so I used the morning and afternoon to explore Auckland. I spent quite a bit of time in Auckland (3-4 days) in 2002, and was eager to re-explore. It’s funny how we remember things. There were restaurants and points of interest I remembered quite well (and went in search of a few of them), but it was a bit of a mind trip to notice places and things that I didn’t remember until I saw them (“Oh wow – I ate at this exact restaurant ten years ago…”)

I made a brief stop at the Microsoft Auckland office during my wandering. I knew I wanted to go to Melbourne while in Australia, but I hadn’t bothered to book a flight or lodging yet. Nor did I have any lodging in Sydney after the conference. I took advantage of some peace and quiet and (apparently rare) high-speed internet and got all of my bookings squared away.

Sydney

The Fusion conference was Thursday and Friday. The opening keynote was Emma Langman, and I really enjoyed her talks (she also gave the closing keynote on Friday). Emma is a Change Magician (cool title, huh?), and knows a ton about organizational change and systems thinking. I applaud SoftEd for inviting her, and think she did a great job capturing the essence of the conference and coming up with relevant topics for the audience.

On top of that, I was a bit intimidated that she attended my Thursday session (on customer-focused test design), but managed to keep it together as I babbled for an hour about test design.

Thursday night, we had a dinner keynote from Dr. Peter Ellyard (Australia’s most prominent futurist). I spoke to him for a few minutes before dinner, where he shared this quote that stuck with me:

Vision without strategy is a daydream. Strategy without vision is a nightmare.

(note that this is a variation of a Japanese proverb that replaces the word ‘strategy’ with ‘action’)

After dinner a few of us hung out and had a few more drinks. I have a vague memory of late-night Karaoke, but the details of the night may be forever lost.

Friday morning came quickly, and I gave my last talk of the conference. All-in-all, it was a fantastic experience, and I am grateful to SoftEd for the opportunity.

Saturday morning, Elisabeth Hendrickson and I met up for a sequel walk and talk event around Sydney, culminating with one of the best cups of coffee I ever had (along with a delicious salmon sandwich). I spent the rest of the weekend doing more wandering around town, getting some laundry done, and catching up on work and reading. I did get in a good run through the botanical gardens, around the opera house, and then up and over the big bridge and back.

Melbourne

On Monday, I left for a quick trip to Melbourne. After a few hours there, I was already bummed that I wasn’t spending more time there. I have to admit, that I’m a bit of a foodie, and good food is everywhere in Melbourne. It’s also a beautiful city with a great mix of new and old architecture.

I went for a nice run along the river Tuesday morning, walked around the city some more, and then met with a Melbourne area testers for a few beers and dinner in the evening (as well as some fun conversations).

After breakfast – and a bit more exploring – I headed back to Sydney in the afternoon. My return trip to Sydney had a few obstacles. The first was that for some reason, my domestic flight to Sydney left from an international terminal. This meant I had to go through some extremely long security and passport control lines. I was particularly ‘lucky’ on this trip and was pulled out of line twice for extra ‘checking’. And then it got interesting. For reasons too long to share here, I’ve always gone by ‘Alan’. My credit cards say ‘Alan’, my Microsoft id says ‘Alan’, my email address is alanpa…I bet most people don’t even know that my first name is ‘Donald’. Because I book my flights with my credit card, my boarding passes often (or nearly always) say ‘Alan Page’. I make sure signatures on my license and passport include my middle name, and this has never, in dozens of international trips, ever been a problem – until this flight. The agent questioned me for ten minutes, trying to get me to prove who I was. I think if I was actually travelling to another country, instead of a few hundred miles north, she wouldn’t have let me leave. But eventually, I made it onto the plane and back to Sydney.

Sydney (part II) & Home Again

My final trip to Sydney was uneventful. I spent most of the afternoon reading in a café, and then had dinner, and got some sleep before heading home.

The flight home was long (SYD->AUK->SFO->SEA), but uneventful. I’m home now, and although I managed to catch a bit of a cold, it’s good to be here and get back to my regular routine. My arrival at home was also nearly perfectly timed with a small fire drill at work, and that is accelerating my return to reality as well.

Overall, I had a fantastic time. I expect to return someday (whether it’s for a conference or for a vacation), and hope to meet many more great testers either way.

Categories: Software Testing

Musings on Test Design

Thu, 08/23/2012 - 11:45

The wikipedia entry on test automation troubles me. It says:

Commonly, test automation involves automating a manual process already in place that uses a formalized testing process.

I can’t decide whether it bothers me because it’s wrong, or because so many people believe the statement is true. In fact, I know the approach in some organizations is to “design” a huge list of manual tests based on written and non-written requirements, and then either try to automate those tests, or pass the tests off to another team (or company) to automate.

This is an awful approach to test design. It’s not holistic. It’s short-sighted. It’s immature, and it’s unprofessional. It’s flat-out crummy testing. I frequently say, “You should automate 100% of the tests that should be automated”. Let me put it another way to be clear:

Some tests can only be run via some sort of test automation.
Some tests can only be done via human interaction.

That part (should be) obvious. Here’s the part I don’t think many people get:

You can’t effectively think about automated testing separately from human testing.

Test Design answers the question, “How are we going to test this?” The answer to that question will help you decide where automation can help (and where it won’t).

Here’s a screen shot of part of the registration form for outlook.com (disclaimer – I have no idea how this was tested).

Let’s look at two different ways of answering the “How will we test this?” question.

The “automator” may look at this and think the following.

  • I’ll build a table of first and last names and use those for test accounts
  • I’ll try every combination of Days, Months, and Years for Birthdate
  • I’ll generate a bunch of different test account names
  • I’ll create a password for each account
  • Once I try all of the combinations, I can sign off on this form
  • (or, they may think, “I wonder what sort of test script the test team will ask me to automate”

The “human” may look at the same form and think this:

  • I’ll want to try short names, long names, and blank names
  • I’ll see if I can find invalid dates (e.g. Feb 29 in a non-leap year)
  • Some characters are invalid for email names – I’ll try to find some of those
  • I’ll make sure the 8-character minimum and case sensitivity is enforced
  • Oh – I’ll try that passwords with foreign characters too.
  • Once I go through all of that, and anything else I discover, I can sign off on this form.

I’ll fire off a disclaimer now, because I’ve probably pissed off both “automators”, and “humans” with the generalizations above. I know there’s overlap. My argument is that there should be more.

In my contrived examples above, the “automator” is answering the question, “What can I automate?”, and the “human is answering the question, “What can I explore or discover?”. Neither is directly answering the question, “how are we going to test this?”.

I could just merge the lists, but that’s not exactly right. Let’s throw away humans and coders for a minute and see if we can use a mind map to get the whole picture together. Here’s what I came up with giving myself a limit of 10 minutes. There’s likely something missing, but it’s a starting point.

Now (and at no time before now), I can begin to think about where automation may help me. My goal isn’t to automate everything, it’s to automate what makes the most sense to automate. Things like submitting the form from multiple machines at once, or cycling through tables of data make sense to automate early. Other items, like checking for non-existent days or checking max length of a name are nice exploratory tasks. And then, there are ideas like foreign characters in the name, or trying RTL languages that may start as an exploratory test, but may lead to ideas for automation.

The point is, and this is worth repeating so often, is that thinking of test automation and “human” testing as separate activities is one of the biggest sins I see in software testing today. It’s not only ineffective, but I think this sort of thinking is a detriment to the advancement of software testing.

In my world, there are no such things as automated testing, exploratory testing, manual testing, etc.

There is only testing.

More musings and rants to follow.

Categories: Software Testing

Orchestrating Test Automation

Mon, 08/20/2012 - 22:18

I’ve been gathering some information on test automation systems recently and will probably have a flurry of posts upcoming on automation and related subjects.

Some observations as I browse what Bing has to tell me about the subject of Test Automation:

  • Wikipedia tells me that test automation “commonly involves automating a manual process already in place”
  • The bulk of the test automation articles are about UI automation
  • There are some reasonably good articles warning about the problems one may run into in test automation projects. None of these articles provide solutions or alternatives.

There’s more in my search results, but those three results alone say a lot about what’s wrong with the way (most?) teams approach test automation.

I expect I’ll write more eventually on all of these points, but my anecdotal experience, along with a few dozen web pages and articles tells me that there’s a lot of confusion in regards to test automation.

You see, test design (including the design of how test automation will execute) is really, really hard. It’s hard to write sustainable and reliable tests, and it’s hard to write trustworthy tests. Double the effort required if those tests involve UI automation. Designing good tests is one of the hardest tasks in software development. That’s worth saying again. Designing good tests is one of the hardest tasks in software development.

Compared to everything else in the equation, the “writing code” part of test automation is easy. Massively easy. Almost brain-dead easy. When I compare writing code to composing music, I talk about the creative and challenging aspect of melody, creating a texture and mood, and figuring out where notes and space between notes help with both. There’s also the rote part of the job – chord voicings, doubling, and other parts of filling in the score. The melody and texture choices are the test design of music composition – filling in the rest of the parts is the coding part. Sure, there’s creativity in a countermelody, as much as there’s creativity in a cool algorithm, but it’s still the rote part of the activity.

I fear that testers are worrying too much about the effort and skills required to automate tests – and not enough about what it takes to design reliable, portable, accurate, and trustworthy tests that actually matter.

Tell me I’m wrong.

Categories: Software Testing

Stupid Multitasking

Tue, 08/14/2012 - 14:22

Today I saw yet another info-byte on the perils of multi-tasking. I too, think that multitasking is rarely (if ever) as productive as multitaskers believe it is (note: my favorite study compares bong hits with multitasking…ymmv) – but we’ve all seen stuff like this over time.

What’s missing from many of the studies and stories that I’ve seen is a definition of what they mean by multitasking. The study above uses examples (and good ones, like the problem with texting and driving), but then seem to lump all other pseudo-multi-tasking into the same bucket. If, by multi-tasking, you mean texting while driving, writing code while playing a video game, or walking the dog while juggling, the studies are 100% true – but perhaps less so, if you consider multi-tasking to be ordering a pizza on the phone while taking your dog for a walk, or listening to the radio while cooking dinner.

But for most tasks requiring brainpower, we don’t actually do two things at once – humans are largely a single threaded system, and we tend to do one thing at a time quite well – it’s the context switch that kills us. This is especially true in knowledge work where we’re writing / coding / speaking / thinking; and where the context switch of doing something different (checking email, checking a blog post, answering the phone, sending a tweet, etc.) ruins our train of thought and takes us out of “the zone”.

But sometimes, context switches are necessary. My favorite example (as anyone who knows me would guess) is cooking. When I cook any meal of significance, context switches are required. One cannot cook a meal serially. If I make the vegetable, then the appetizer, then the main dish, then the desert, I get to serve everything cold. I need to start with my deadline (dinner time), think through the process needed to make each dish, then determine what needs to be done in what order to accomplish my goal. If I make a marinade that needs to sit for three hours, I better do that first. Anything that requires heat should finish as close to dinner time as possible. I love the orchestration of cooking a great meal, but this task would be impossible (or at least a failure) if I didn’t rely on the context switch.The trick is to plan for the context switch and optimize – do the stuff where you know you need to wait first. Mentally work backwards and set up intermediate check points. Sure, doing only one thing would probably be faster, but it’s not always practical..

I apply the same techniques to my daily work. I look at all of the things I need to do (which is nearly always more than I can do), and do anything with a built-in wait first. For example, I start most days by sending out any requests for information I may need, things I need reviewed, etc. Then I take on bigger tasks. For writing tasks, I use a pomodoro timer – for coding or testing tasks I just go to work (I’ve found that the pomodoro beep ends up being a distraction from these types of tasks– but that’s just me). But regardless of whether it’s a pomodoro timer, or a natural break in flow, I tend to use those breaks to let interruptions decide what’s next. It’s a conscious task switch, but just like in cooking, where I may take a moment to look around the kitchen and see what’s going on (oops – better turn the oven down), if you’re in a role where you need to own multiple pieces of a project, it’s critical to allow yourself to step back, look at the big picture, and either dive back in again, or refocus on another area.

Now, to be fair, when I have a bigger project to take on, I’ll minimize multi-tasking as much as possible. I can think of several occasions where I needed to focus on accomplishing a specific task – even at the expense of potentially more critical tasks. My approach in those situation involves a pomodoro timer, and three-hour blocks of time in the morning and afternoon, broken up by breaks for email and task juggling. But – for the most part, I’ve found much benefit in strategic task switching and planning ahead than in giving up and admitting to stupidity just because your plate is full.

Categories: Software Testing

The Problem with TheApp

Mon, 08/13/2012 - 10:58

Last week, I posted a small testing challenge (in short, an application wouldn’t launch).Shmuel Gershon has a wonderful write up of how he found the problem here.

For those too lazy to read, the problem was that the application attempted to read a reg key on launch, and failed to launch (with no diagnostics) if the reg key didn’t exist.

Having done a massive amount of application compatibility testing over the years, this is a fairly common scenario I’ve run across (simplified quite a bit in this case). Typically, it works like this: a team makes an update to an application or operating system, and some application begins to fail in a bizarre way (including failing to launch, as well as functionality or stability changes). After some investigation, we usually discover that the application relied on a registry key, a file, or functionality that no longer exists, or has changed. Half of the time, the application vendor is out of business, or doesn’t have the bandwidth to make an update, so we have to figure out what went wrong and hack do whatever is necessary to ensure the application continues to work.

Some of the time, the investigation requires a bit of reverse engineering, but at least 9 times out of 10, I use tools like sysinternals procmon to see what’s going on. I was happy (but not surprised) to see that Shmuel ended up using this tool to find the issue (and was even happier to see how well the tool led him to the exact issue).

For my job, I do almost as much investigation into why things don’t work as I put into discovering what works (and doesn’t work). To me, the detective work is part of the testing role. I was surprised yesterday, when a tester told me that his job was problem detection only, and that the rest was for someone else. That world of testing would be boring to me – ymmv.

Congrats Shmuel for the great investigation, and for having this valuable tool in his toolbox.

 

Note: for simplicity, I put the key under HKCU so everyone could access it, and used an obvious key name (MagicValue) – in the real world, it’s not always this way.

Categories: Software Testing

Quick Testing Challenge

Sun, 08/05/2012 - 13:08

Some updates and clarification below.
And a NEW HINT below too

Recently, I was discussing the diagnostic, debugging, and troubleshooting aspects of software testing, and this morning, while playing with Visual Studio 2012, I created a quick little exercise where the solution relies on these skills.

This zip file contains an app (theapp.exe), and the the vc 2012 runtime dll (both for 32-bit Windows) (note, this should be all that is needed, but if you get a load error, let me know in the comments or on twitter).

But even with the runtime, the app won’t launch. I’m sure that many of you will be able to figure out why –but I’m curious to know how you find the answer. When faced with problems like this, I have a favorite method – but I’m curious what other people do.

Two things to add:

  1. This application, does indeed launch. When it runs, you’ll have a generic windows application, and a message box indicating success.
  2. Some of you are probably thinking, “If the app doesn’t even launch, I’ll talk to the developer, since he’ll know what changed.”. But consider this scenario (which is the exact scenario that led me to create this app). You are the tester for Application Awesome. Your application is…Awesome. However, a user tells you that after installing Application Awesome, another app they love (TheApp.exe) fails to launch. WTF happened?
  3. HINTS: Think for a moment about what sorts of things an application may check on launch. Could one of them be failing? How would you (could you) know?
Categories: Software Testing

Let it happen

Sat, 08/04/2012 - 11:17

I come across this frequently enough that I’m sure I’ve blogged about it before, but context dictates that I do it again. The story I hear goes pretty much like this:

My team really needs to improve X, but nobody is taking responsibility for it. The fix is obvious – we need our exec / manager / project owner to mandate that we do X and measure Y & Z to make sure people are doing the right thing.

Here’s a concrete example. I was on a team once where quality was in the toilet. We couldn’t even get a build to run for over a week. The solution was require everyone on the team to perform a list of actions before every check-in. Of course, the list including using flaky tools, and getting sign off on every step, even if the step wasn’t applicable to a particular check-in. Given the extra work required for check-in, developers queued up weeks worth of check-ins for one big check-in rather than go through the steps in the checklist multiple times. Day to day build quality was slightly better, but velocity was way down. While slower velocity is not necessarily a bad thing, the big problem was culture. Someone honestly thought the mandates would create a culture of quality – but the only culture they created was a culture of finding ways to skirt the damn checklist. Team morale sucked, and the product we built sucked too.

To be fair, I’ve seen some successful mandates in my career. The push for improvement in every developer to write secure software at MS worked. But that culture change was also pushed by huge monetary losses.

In general (and in every other case I can think of), mandates don’t work – especially if you are using the mandate to change culture. Yet I see mandates suggested as a solution for changing culture time and time again. I just don’t get it.

The folks at 37 Signals say it best, “You don’t create a culture. Culture Happens.”

So help let it happen. Instead of a mandate, help your team see where you want to go. Make it personal. Appeal to their own pride and values. Show them how change will help them. Find allies who think like you do. Experiment.

Let it happen. Make it happen.

Categories: Software Testing

The Easy Part

Tue, 07/31/2012 - 17:13

Recently, I was helping another part of the team with a project. Or at least it ended up that way. There’s a particular bit of what they’re doing where I have some expertise, so I volunteered to take care of a chunk of the work where I thought I could help out.

One thing I’ve learned through experience (and through testers eyes) is to take a look at the big picture of the problem I’m solving before diving in. For example, let’s say someone asked me to change the text on a button from “Query DB” to “Query Database”. What could be a 30 second task is far from it. First, I need to make sure the button is big enough, I need to check to see if there are other buttons in the application that need similar renaming. I need to make sure the documentation (including screen shots) are updated. I probably need to make sure the localization team knows what to do with the update. Of course, I’ll see if there are any tests that look for this particular string on a button, and after I punch them in the face for testing against a hard coded UI string, I’ll make sure they fix it.

In this case, I needed to add functionality ‘A’ to a system. I know functionality ‘A’ pretty well, but in this case, in order to add ‘A’ correctly, I needed to update ‘B’ – and for ‘B’ to work, I needed to refactor huge chunks of ‘C’. I went to the team and told them that I knew what needed to be done, but it was complex (due to A, B, and C), and that while I was willing to do the work, it would take me a few days to a week to implement and test.

Then they asked me my new favorite estimation question. “How long will it take you to do the easy part.” My answer, of course**, was, “It depends. Which part is the easy part?” To be fair, they meant, how long will ‘A’ take (because they had some insight into B & C), but it was still a fun quote.

 

** Alan-ism #17: The answer to any sufficiently complex question is, “It depends”.

Categories: Software Testing

Me, Ranting About Thinking

Mon, 07/23/2012 - 21:38

Joey McAllister (favorite hashtag: #expectpants) recently talked with me (electronically) about critical thinking and learning.

The interview is here in case you’re curious.

Categories: Software Testing