Chris McMahon, who has always impressed me with his words and his wit called me out in his blog.
Apropos of my criticism of “Context Driven Approach to Automation in Testing” (I reviewed version 1.04), I ask you to join me in condemning publicly both the tone and the substance of that paper.
Almost exactly a year ago, I reviewed a draft of the paper, and my name is among those listed as reviewing the paper. The feedback I gave was largely editorial (typos and flow), with a few comments about approach that I’ll repeat here:
The first red flag I called out (and will call out again here) is the phrase that describes test automation as something to “automate testing by automating the user”. This is a shallow view of test automation, but other than comment, I didn’t push hard on it. In hindsight, this was a mistake (much of The A Word touches on this topic).
In regards to the scenario the authors chose for scenario automation, I thought the choice was weird, and asked for more…context and provided some food for thought.
I think you can add even more emphasis to how and why you chose to automate scene creation. Too many times, testers choose to automate because they can automate. The thought process (for me) may be, “The scene feature is pretty important, I’m curious what happens if an author has thousands of scenes. Will it cause performance problems, formatting problems, file load problems, or other issues, etc. There’s no way in hell I want to do this manually, so let me write some code to help me run my experiment”. You may also want to discuss alternate implementation ideas – e.g. creating a macro in Notepad++ to create the text then paste it in, or creating a macro in Word for the same, or using for in a windows console (e.g. for /L %f in (1,1,1000) do echo “##”>>output.txt&echo “Scene %f”>>output.txt ). Using tools for creating and manipulating data could be a whole article.
And that led to my real beef with the paper. It talks about using tools to test – which can be a good thing, but it doesn’t really talk about automation in the way successful teams actually use it.
I think it may be important to talk about purposes of automation and where to apply it – or at least one context of that – as I don’t think that’s discussed enough (but I’ve made a note to myself to write a blog on this very subject). At a BFC (big company) like Microsoft, we write a lot of shared / distributed automation – automated tests that we need to run on a lot of different hardware/platform configurations in some sort of lab setting (tools like SauceLabs or BrowserStack are helpful here). Web apps are famous for this [problem].
I also commented:
The other kind of automation (which you cover in your article) is what I sometimes call exploratory automation (or more often, just “Testing”). This is where we get a test idea, and want to write some quick automation to help learn about the product. While we may turn this sort of test into something that’s distributed / shared someday, its primary purpose is to help me answer questions and learn. There’s (another) story in HWTSAM where I described a case of this. I wrote really ugly brute force automation in C (using things like FindWindow and SendMessage(LBUTTON_DOWN…) to simulate opening and closing a connection to a remote host many times (the only thing this app did). It found a nice memory leak that may not have been found otherwise (or at least not as quickly).
All of this feedback fed my uber-point, which was that while the article talked about test automation, the examples really just talked about using somewhat random tools to help the authors explore or test some software. There was nothing about strategy, or about more typical use of test automation. I asked about it in a comment:
I wonder why you don’t use the word “Tools” in the title – e.g. “A CDT approach to tools and automation in testing” or something like that.
…because the paper is about, as I said above, using non-standard tools to help test. Sure, it’s automation in a sense, but nothing in the paper reflects the way test automation is used successfully in thousands of successful products.
All that said, I do not support this paper as a description of good test automation, and I think it’s an inappropriate method for anyone to learn about how to write automation. Chris requested that the authors remove the paper; while I support this, and do believe that the paper can cause more harm than good, there’s so much bad advice on the internet about creating software, that removing this one piece of bad advice will hardly make a dent.
I did not realize my name was listed as a reviewer, and although I did (as admitted above) review this paper, I do not want my name associated with it, and will request that the authors remove my name.(potentially) related posts:
There’s a light at the end of the oh-my-work-is-so-crazy train, and I look forward to ranting more often both here and on twitter.
But first, a few minor anniversaries to acknowledge. Monday was my 21-year anniversary at Microsoft. It’s not a nice even number like 20, but it’s weird to think that people born on the day I started (full-time) at Microsoft, can now drink in the U.S. While I doubt I’ll make it to 25, I doubted that I’d make it to 20…or 10, so this is definitely an area where I’m bad at estimating.
Meanwhile, the ABTesting Podcast just hit episode #40. That’s another milestone I never thought I’d hit, but Brent and I keep finding things to talk about (or new ways to talk about the same things). We should hit the 50-episode milestone (by my already established as poor estimates) before the end of the calendar year. I’m thinking of inviting Satya to be a guest, but I don’t think he’ll show up.
On the announcement front, I’m speaking at Test Bash Philadelphia in November. I’ll be talking about “Testing without Testers and other stupid ideas that sometimes work”. This is an evolution of a talk I’ve been giving recently, but I’m preparing something extra special for test bash that should inspire, as well as cause some great conversations to happen.(potentially) related posts:
I’m at no loss for blog material, but have been short on time (that’s not going to change, so I’ll need to tweak priorities). But…I wanted to write something a bit different from normal in case anyone else ever needs to solve this specific problem (or if anyone else knows that this problem already has an even better solution).
Our team uses a tool called Istanbul to measure code coverage. It generates a report that looks sort of like this (minus the privacy scribbling).
For those who don’t know me, I feel compelled to once again share that I think Code Coverage is a wonderful tool, but a horrible metric. Driving coverage numbers up purely for the sake of getting a higher number is idiotic and irresponsible. However, the value of discovering untested and unreachable code is invaluable, and dismissing the tool entirely can be worse than using the measurements incorrectly.The Missing Piece
Istanbul shows all up coverage for our web app (about 600 files in 300 or so directories). What I wanted to do, was to break down coverage by feature team as well. The “elegant” solution would be to create a map of files to features, then add code to the Istanbul reporter to add the feature team to each file / directory, and then modify the table output to include the ability to filter by team (or create separate reports by team).
I don’t have time for the elegant solution (but here’s where someone can tell me if it already exists).The (or “My”) Solution
This seems like a job for Excel, so first, I looked to see if Istanbul had CSV as a reporter format (it doesn’t). It does, however output json and xml, so I figured a quick and dirty solution was possible.
The first thing I did was assign a team owner to each code directory. I pulled the list of directories from the Istanbul report (I copied from the html, but I could have pulled from the xml as well), and then used excel to create a CSV file with file and owner. I could figure out a team owner for over 90% of the files from the name (thanks to reasonable naming conventions!), and I used git log to discover the rest. I ended up with a format that looked like this:
Then it was a matter of parsing the coverage xml created by Istanbul and making a new CSV with the data I cared about (directory, coverage percentage, statements, and statements hit). The latter two are critical, because I would need to recalculate coverage per team.
There was a time (like my first 20+ years in software) where a batch file was my answer for almost anything, but lately – and especially in this case – a bit of powershell was the right tool for the job.
The pseudo code was pretty much:
- Load the xml file into a PS object
- Walk the xml nodes to get the coverage data for a node
- Load a map file from a csv
- Use the map and node information to create a new csv
Hacky, yet effective.
I posted the whole script on github here.(potentially) related posts:
I was thinking about this on the way to work today, and thought I’d try to spit out a quick blog post before I got side-tracked again.
I’ve been very fortunate to have had success with organizational change with teams at Microsoft. Whether it’s getting programmers to run integration tests before check-in, or helping a team get to a daily zero-bug bar, my leadership style is the same. I believe that people will do things that they think are valuable. In fact, this quote from Eisenhower (which is, admittedly, overused) aligns tightly with my style.
Leadership is the art of getting someone else to do something you want done because [s]he wants to do it.
I talk with people to understand what their concerns and motivations are. I communicate plans and strategies to the team. Often, I “plant seeds” – for example, I may mention to a manager a few of the benefits of keeping engineering debt low and give a few examples. No judgement or decree – just an idea to put in their head. Later, I may mention that it would may be a good idea to keep pri 1 bug counts at zero, and maybe overall bugs below some arbitrary number. Often, a few weeks later, I’ll see that manager’s team with zero pri 1 bugs. Or, I’ll mention in a meeting that I’d like to get the whole team down to zero bugs, and I generally have support from everywhere I planted a seed.
The big advantage of this style of change management (in my experience) is that the team owns the change, and accept it as part of the way they work. The disadvantage, is that it takes time. To me, that time investment is worth it.
There’s a faster approach, but I don’t like it – yet I see it used often. It probably has a better name, but I’ll call it the do-it-because-I-said-so style of leadership. Eisenhower also said that leadership doesn’t come from barking orders or insisting on action (paraphrase because I’m too lazy to look it up). To me, leadership isn’t about your ideas, it’s about working with others and building your tribe. Too many so-called leaders think that leadership is being the loudest voice, or being the one that makes mandates to an organization. That’s not leadership to me. That’s being a dick.
That said, there’s a middle ground there, that I see often enough to respect, but not often enough to completely understand. I know some leaders who are able to make explicit mandates and have their team rally around them immediately. They don’t do this often, and I think it helps. They are humble and I think this helps. They have a relationship with their followers – and this helps too. Maybe the answer is that they’ve waited until they’re a real leader (rather than a self-proclaimed chest-thumper), and waited until circumstances were necessary before making a mandate.
What kind of leader do you want to be?(potentially) related posts: