Alan Page

Subscribe to Alan Page feed
notes and rants about software and software quality
Updated: 11 hours 55 min ago

Filling a gap in Istanbul coverage

Tue, 04/19/2016 - 17:38

I’m at no loss for blog material, but have been short on time (that’s not going to change, so I’ll need to tweak priorities). But…I wanted to write something a bit different  from normal in case anyone else ever needs to solve this specific problem (or if anyone else knows that this problem already has an even better solution).

Our team uses a tool called Istanbul to measure code coverage. It generates a report that looks sort of like this (minus the privacy scribbling).

For those who don’t know me, I feel compelled to once again share that I think Code Coverage is a wonderful tool, but a horrible metric. Driving coverage numbers up purely for the sake of getting a higher number is idiotic and irresponsible. However, the value of discovering untested and unreachable code is invaluable, and dismissing the tool entirely can be worse than using the measurements incorrectly.

The Missing Piece

Istanbul shows all up coverage for our web app (about 600 files in 300 or so directories). What I wanted to do, was to break down coverage by feature team as well. The “elegant” solution would be to create a map of files to features, then add code to the Istanbul reporter to add the feature team to each file / directory, and then modify the table output to include the ability to filter by team (or create separate reports by team).

I don’t have time for the elegant solution (but here’s where someone can tell me if it already exists).

The (or “My”) Solution

This seems like a job for Excel, so first, I looked to see if Istanbul had CSV as a reporter format (it doesn’t). It does, however output json and xml, so I figured a quick and dirty solution was possible.

The first thing I did was assign a team owner to each code directory. I pulled the list of directories from the Istanbul report (I copied from the html, but I could have pulled from the xml as well), and then used excel to create a CSV file with file and owner. I could figure out a team owner for over 90% of the files from the name (thanks to reasonable naming conventions!), and I used git log to discover the rest. I ended up with a format that looked like this:

app.command-foo,SomeTeam
app.components.a,SomeTeam
app.components.b,AnotherTeam
app.components.c,SomeTeam
app.components.d,SomeOtherTeam

Then it was a matter of parsing the coverage xml created by Istanbul and making a new CSV with the data I cared about (directory, coverage percentage, statements, and statements hit). The latter two are critical, because I would need to recalculate coverage per team.

There was a time (like my first 20+ years in software) where a batch file was my answer for almost anything, but lately – and especially in this case – a bit of powershell was the right tool for the job.

The pseudo code was pretty much:

  • Load the xml file into a PS object
  • Walk the xml nodes to get the coverage data for a node
  • Load a map file from a csv
  • Use the map and node information to create a new csv

Hacky, yet effective.

I posted the whole script on github here.

(potentially) related posts:
  1. Scaling Code Coverage
  2. An Approach to Code Coverage
  3. Filling A Hole
Categories: Software Testing

Do it *my* way, or do it *our* way

Thu, 04/14/2016 - 13:27

I was thinking about this on the way to work today, and thought I’d try to spit out a quick blog post before I got side-tracked again.

I’ve been very fortunate to have had success with organizational change with teams at Microsoft. Whether it’s getting programmers to run integration tests before check-in, or helping a team get to a daily zero-bug bar, my leadership style is the same. I believe that people will do things that they think are valuable. In fact, this quote from Eisenhower (which is, admittedly, overused) aligns tightly with my style.

Leadership is the art of getting someone else to do something you want done because [s]he wants to do it.

I talk with people to understand what their concerns and motivations are. I communicate plans and strategies to the team. Often, I “plant seeds” – for example, I may mention to a manager a few of the benefits of keeping engineering debt low and give a few examples. No judgement or decree – just an idea to put in their head. Later, I may mention that it would may be a good idea to keep pri 1 bug counts at zero, and maybe overall bugs below some arbitrary number. Often, a few weeks later, I’ll see that manager’s team with zero pri 1 bugs. Or, I’ll mention in a meeting that I’d like to get the whole team down to zero bugs, and I generally have support from everywhere I planted a seed.

The big advantage of this style of change management (in my experience) is that the team owns the change, and accept it as part of the way they work. The disadvantage, is that it takes time. To me, that time investment is worth it.

There’s a faster approach, but I don’t like it – yet I see it used often. It probably has a better name, but I’ll call it the do-it-because-I-said-so style of leadership. Eisenhower also said that leadership doesn’t come from barking orders or insisting on action (paraphrase because I’m too lazy to look it up). To me, leadership isn’t about your ideas, it’s about working with others and building your tribe. Too many so-called leaders think that leadership is being the loudest voice, or being the one that makes mandates to an organization. That’s not leadership to me. That’s being a dick.

That said, there’s a middle ground there, that I see often enough to respect, but not often enough to completely understand. I know some leaders who are able to make explicit mandates and have their team rally around them immediately. They don’t do this often, and I think it helps. They are humble and I think this helps. They have a relationship with their followers – and this helps too. Maybe the answer is that they’ve waited until they’re a real leader (rather than a self-proclaimed chest-thumper), and waited until circumstances were necessary before making a mandate.

What kind of leader do you want to be?

(potentially) related posts:
  1. Stuff About Leadership
  2. The Ballad of the Senior Tester
  3. What’s Your Super Power?
Categories: Software Testing

Madness, Math and (half) Marathons – and Medium!

Mon, 01/25/2016 - 14:02

I decided that in the rare occurrences where I post non-software articles, that I’d use a blog on Medium and post my attempts at story-telling along with the rest of the world.

My first post is here.

(potentially) related posts:
  1. Settling on Quality?
  2. Writing About Testing
  3. Walls on the Soapbox
Categories: Software Testing

Intelligence and Insight

Wed, 01/13/2016 - 16:32

I have an after-work event tonight, and rather than leave my car in the garage overnight, I ran to work. Since I’ve moved to downtown Bellevue, I’ve done this a few times – and given that I’m running another half-marathon in 10 days, it was a great opportunity for a long training run before I begin to taper my mileage down a bit leading up to the race.

App Issues

I’ve been a long-time user of a running companion app called Runtastic. It does the usual stuff of tracking mileage, route, pace, and giving voice updates at user specified intervals. I find it especially valuable when training or racing, because I usually have very specific pace goals. Having a voice tell me how long the last mile/half-mile/km took me let’s me know if I’m running too fast (and burning out) or too slow (and putting my goals in jeopardy). Granted, I have a pretty good internal clock, and usually run my paces pretty well withouth “help”, but the feedback is really useful to me.

Today, I took off from home, started spacing out, and before long, I was a mile or two from home…when I noticed that I had not received any voice feedback yet. I knew exactly what happened (because it’s happened before). When you start the app, it gives you a 15 second timer before the tracking actually starts, along with the ability to add time to the delay up to two minutes or so. I LOVE this feature, because I can give myself time to put my phone into my running belt, walk twenty yards and curse myself for having such a painful hobby before actually exerting any physical energy.

Unfortunately, at least 25% of the time I use the app the countdown fails. But I never know it failed until too late.

What happens is that the countdown stops at 1 second. I hear the voice prompt count down 5-4-3-2-1, and I think it’s tracking, but it’s stuck on one second.

Today was extra painful not only because I wanted to see how I was doing on “race pace”, but after I discoverd it failed, I restarted the app, set the countdown again…and it “hung” on one second again.

Grrrr…

Now, it’s a bug – that’s for sure. Some testers I know would automatically assume that every user in the world was hitting this bug and that public shaming of the company would be the next course of action. I, however, realize that context plays a role in many (every?) part of software engineering, and that given the value I get from the product (and my very amateur level of running) that while this is a painfully annoying bug, it’s not the end of the world.

Intelligence?

Given that I wasn’t concerned at all with pace for the remaining 5+ miles of my commute, my mind began to wander. What follows is a completely made up story of how a Runtastic engineer may discover this issue without me, or anyone else, reporting it.

Pointy Haired Runtastic Manager: Hey super-smart employee (sse). Our default time for delay before a run is 15 seconds. Can you look at the data and see if our estimate for a default delay length is in the ballpark of what people actually use? Someone on the train told me that they though 30 seconds would be a lot better. I don’t think they’re right, but I want to make a decision based on data.

Super Smart Employee (sse): Sure boss. That data is pretty easy to pull. I’ll take a look!

What SSE is about to do at this point is gather Business Intelligence. They want to use data to make a business decision.

SSE looks at hundreds of thousands of activities from the past six months and sees that nearly 60% of the people just use the default 15 seconds. She quickly generates a scatter graph showing that shows the outliers and prepares it for her boss. Before sending, she realized that she wants to exclude the instances where people cancel the activity completely before the countdown completes (phone calls, cold feet (literally, and metaphorically), and a variety of other reasons could cause this). She filters the data and starts to send the report…but – while she’s there, she notices something…interesting. First, the number of “cancelled” activities seem high to her (over 15%). She flips the filter to look only at cancelled activities and things get weirder. Of the 15% of cancelled activities, 90% are cancelled at exactly 1 second.

That’s too weird to be true.

Insight

SSE looks at every activity where the timer was “killed” at one second. Often, those users started another activity within 10-15 minutes.

Or, maybe they all had the same model of phone.

Or maybe they were all running Spotify at the same time.

Or something. Remember. This story is completely made up.

The point is that SSE quickly went from gathering BI to using discovery and insight to find a pretty cool bug. Using data!

Aside

I told a story at a conference recently about a team I worked on that used an offshore vendor team to run through a large number of applications for app compatability testing. We asked them to take notes and to send a report, but not to bother filing bug reports.

HuWhat?

Yeah – we had sufficient telemetry and monitoring that we knew about all the bugs and glitches (and had collected call stacks and other helpful information) already. Many, in fact, that the test team didn’t (or couldn’t) notice. Entering the bugs would have been a waste of time. In the rare cases where something weird happened that we didn’t track, we immediately added the appropriate instrumentation to track that class of failure in the future.

I expect that for most of you, my world isn’t your world. But in my world, data driven engineering is critical.

Epilogue?

Since I don’t know how made up my made up story really is, I’m going to report it to Runtastic anyway. I can’t predict the future (or anything else), but I hope the reply to my complaint is, “Yeah – we already knew about that. From the data”.

(potentially) related posts:
  1. PBKAC?
  2. The Goal and the Path
  3. Riffing on the Quadrants
Categories: Software Testing