QA Music: You Can’t Take The Sky From Me
You’ve got to find something at your core that’s inviolable if you’re going to make a career of QA.
Because they’ll take everything else from you. One way or another.
JavaOne 2012: Improving ALM workflow with automation in Oracle PaaS
Advice for test managers: How to develop, coach and lead your team
Translating business requirements and understanding team roles
Why user interface testing is important
Bid old test rules goodbye: New mind-set needed to test mobile apps
Self-Assessments from Becoming an Influential Test Team Leader Tutorial at StarWest 2012
I promised the people in my Tuesday tutorial at StarWest 2012 the Word versions of the two self-assessments: One on the people issues in testing and the other on building core competencies in testing. Here they are!
Core competency self-assessment
Feel free to modify for your own purposes. Thanks to everyone who attended my tutorial. It was a great time!
Consider a complex and rich web app. Under the hood, it is probably a maze of servers, each performing a different task and most talking to each other. Any user action navigates this server maze on its round-trip from the user to the datastores and back. A lot of Google’s web apps are like this including GMail and Google+. So how do we write end-to-end tests for them?
The “End-To-End” Test
An end-to-end test in the Google testing world is a test that exercises the entire server stack from a user request to response. Here is a simplified view of the System Under Test (SUT) that an end-to-end test would assert. Note that the frontend server in the SUT connects to a third backend which this particular user request does not need.
One of the challenges to writing a fast and reliable end-to-end test for such a system is avoiding network access. Tests involving network access are slower than their counterparts that only access local resources, and accessing external servers might lead to flakiness due to lack of determinism or unavailability of the external servers.
One of the tricks we use at Google to design end-to-end tests is Hermetic Servers.
What is a Hermetic Server? The short definition would be a “server in a box”. If you can start up the entire server on a single machine that has no network connection AND the server works as expected, you have a hermetic server! This is a special case of the more general “hermetic” concept which applies to an isolated system not necessarily on a single machine.
Why is it useful to have a hermetic server? Because if your entire SUT is composed of hermetic servers, it could all be started on a single machine for testing; no network connection necessary! The single machine could be a physical or virtual machine.
Designing Hermetic Servers
The process for building a hermetic server starts early in the design phase of any new server. Some things we watch out for:
- All connections to other servers are injected into the server at runtime using a suitable form of dependency injection such as commandline flags or Guice.
- All required static files are bundled in the server binary.
- If the server talks to a datastore, make sure the datastore can be faked with data files or in-memory implementations.
- Make sure those connection points which our test won’t exercise have appropriate fakes or mocks to verify this non-interaction.
- Provide modules to easily populate datastores with test data.
- Provide logging modules that can help trace the request/response path as it passes through the SUT.
Using Hermetic Servers in tests
Let’s take the SUT shown earlier and assume all the servers in it are hermetic servers. Here is how an end-to-end test for the same user request would look:
The end-to-end test does the following steps:
- starts the entire SUT as shown in the diagram on a single machine
- makes requests to the server via the test client
- validates responses from the server
This end-to-end test is more reliable because it uses no network connection. It is faster because everything it needs is available in-memory or in the local hard disk. We run such tests on our continuous builds, so they run at each changelist affecting any of the servers in the SUT. If the test fails, the logging module helps track where the failure occurred in the SUT.
We use hermetic servers in a lot of end-to-end tests. Some common cases include
- Startup tests for servers using Guice to verify that there are no Guice errors on startup.
- API tests for backend servers.
- Micro-benchmark performance tests.
- UI and API tests for frontend servers.
Hermetic servers do have some limitations. They will increase your test’s runtime since you have to start the entire SUT each time you run the end-to-end test. If your test runs with limited resources such as memory and CPU, hermetic servers might push your test over those limits as the server interactions grow in complexity. The dataset size you can use in the in-memory datastores will be much smaller than production datastores.
Hermetic servers are a great testing tool. Like all other tools, they need to be used thoughtfully where appropriate.
Disciplined Agile Delivery: Agile process framework for the enterprise
So we're splitting the teams. Going forward, we will have two functionally equivalent teams. First order of business: a team name!
Sounds simple, right? Nope, not so much. Let's back up for a second.
This isn't the first client I've had who's split teams. The last client that split wound up with three teams needing names. And then the fun started. One team suggested greek alphabet as a theme. They would, of course, be Team Alpha. The second team would be Team Beta. The third team would be Team Omega. Team Beta was particularly unamused. Another team suggested a theme of 80's TV shows. They would be Miami Vice (cool cars!). The second team would be Growing Pains. The third team would be the Golden Girls (this team, incidentally had no women). Neither team was amused. The third team suggested a simple numbering system. They would be team 1. Someone else would be team 2. And the last team would be team 3. The squabbling continued.
It took about three weeks to land on colors as a relatively neutral theme. Granted, this was an extreme example of a group that collectively contained some pretty fragile egos and a highly developed sense of hierarchy.
The underlying point is interesting, though. It's useful to give teams names. After all, we have to refer to them somehow. It's also fun to let teams pick their own names.
Make sure those names are neutral and don't imply positioning or relative importance in any way. Neutral things like colors, places, video games, and others are a good idea. And make sure every team gets to pick their own name.
One day, I'll get to be on Team Neutral. Until then, you can find me on Team has-a-neutral-name.
Mobile device testing: Placing testers in the field, not just the lab
SaaS application security: Risks and best practices
Cloud security in the ALM process
How to set priorities: A software project team exercise
The Unleveling Wind
In a stunning turn of events, flat corporate structures might not be the optimal solution:
Disenchanted with the corporate world, many entrepreneurs strive to keep their organizations as flat as possible. But a recent study suggests that, in some cases, office hierarchies help employees get more done. And that too many dominant personalities could jam up the works.
The study, conducted by researchers from the business schools at Columbia University and Northwestern and the University of Queensland, Australia, found that when there are tasks that require teamwork, people get more done when there are defined leaders and followers.
I once worked at a place that moved from a hierarchy to a flatter structure. Although this sort of structure is often promoted as egalitarian and allowing everyone to take a stake in the outcome, the result is often less Utopian. Sometimes, when everyone takes a stake, nobody takes responsibility, and when things go wrong the blame–if blame is due–is diffused. It can always be somebody else’s fault.
In other situations, flat hierarchy projects end up with the people who think it’s most important making the most effort and carrying along other extraneous people who hope to share in the success of the project or at least appear busy enough to continue drawing a paycheck.
Frankly, I favor an open communication hierarchy where members of the hierarchy have good relationships not only with their peers in the org chart but also with members of other levels of the hierarchy, mostly but not limited to people above and below you directly.
In that case, you know how to escalate things when things need to be escalated. In a team of equals, you get a chasing-the-rabbit-around-the-racetrack sort of effect, where someone can pass you off to someone else until you eventually find someone who wants to fix it or someone who cannot think of someone else to pass you to.
Requirements shifting with enterprization of mobile apps
QA Music: Getting into the Mood
Let’s get in the mood for another week. “Ready to Rumble” by Mystikal
The Rise of the Customer
I spoke recently on the topic of “Customer Focused Test Design” (synopsis: customers don’t really care how much functional testing testers do, or how many bugs they find. A test approach that favors scenarios, -ilities, and customer feedback is better; and a bunch of examples for emphasis/proof).
Part of the approach I suggest (and have had some success with), is doing testing of performance, reliability, security, privacy, world readiness, and usability (including accessibility) early in the product cycle. Early, as in don’t bother with functional testing, do the ilities instead. The premise (and my experience) is that testing ilities tests the underlying functionality by default, and that programmers are (in general) doing a better job testing for functionality during code development. For more details, I’ll probably have to revive the whole talk – and that’s not what this post is (intended to be) about.
There’s one –ility that is critical to the customer perception of quality, but it’s not quite like the others – supportability. Customers hate problems, but they love finding solutions to those problems. Online forums, customer connection programs, twitter, facebook, and other online platforms are quickly becoming support forums for a lot of software. Software companies who engage with customers actively are creating happy customers.
Zappos has created a fantastic culture of customer service and have won customers for life (I’m one of them) by actually caring about their customers.
Recently, I moaned on twitter that I couldn’t get my (aisle) seats confirmed for a long flight from Australia to Seattle on a flight booked through Orbitz. Within minutes, @OrbitzCareTeam contacted me on twitter and booked and confirmed my seats. I fly a lot (too much), and I can book my flight with any discount airline– but Orbitz will get my business.
Last month, I visited an Xbox call center, talked to some of the employees there, and listened in on some calls. I was ecstatic to see that nobody works from a script and there are no quotas or other incentives to push people through the system. I listened to several calls where the support folks took their time helping people solve problems step by step (on occasion, converting customer rage into customer appreciativeness). Most interesting was that none of the calls I listened to had anything to do with the Xbox console. Our support people answered questions about entering codes for other manufacturers games, helped customers reset live id passwords, and a variety of other topics related to – but not part of the actual console. Every customer hung up happy – I was blown away.
If you want to be a successful software company, you have to care about customers. In addition to keeping them happy once they have used your software or service, you need to respond to their needs and give them what they need. In The Lean Startup, Ries drives home the point of iteration as a means to obtain validated learning; with the premise that any work that does not provide value to the customer to be wasteful. In other words, listen and learn – often.
Sometimes companies get it wrong. I recently had an issue with an application that was overlaying a decoration on my explorer icons. I found the overlays to be distracting and a detriment to my productivity. When I searched online for a workaround or solution (or sympathy), a representative of the company had this to say (bold mine).
Our thinking was that we want the app to be running in the background, all the time. We want to to blend into your experience so you almost never have to interact with [the application] to check the status of things. So we figured icon overlays were a subtle way to do this, while reassuring people that the app was indeed running.
Maybe we can add an option in the future to toggle the overlays, but honestly I wouldn’t hold your breath. There’s plenty of other cool stuff we want to add first
Note the pronouns in bold. Our thinking…We want…we figured…we want… – these are the statements of Engineering Focused Engineering (note the intentional idiocy of that phrase)
To be clear, I like this product. And on my own, I figured out how to disable the overlays. I also don’t know if the statements above reflect the opinions of one person, or a whole software team…
But the statements above are the statements of an organization that doesn’t give a about their customers. What’s worse, is that this software’s main competition is an application that is praised for its customer focus. In my opinion, it’s unprofessional to approach software development in this way – unless of course you don’t actually want to have customers use your software.
Isn’t it about time to put customers first? Always? I’m all for shifting test design to favor scenarios customers care about, but successful software projects are going to require a customer focus for the lifetime of the product – from concept to development to deployment and beyond.
It’s time for the rise of the customer.
Bring your own device driving changes in mobile QA
When That SaaS You Use Goes Night-Night
So I’m having a devil of a time finding just one crushable c-crown black fedora with a 4″ crown and 2 or 2 1/4″ brim in large or size 7 1/4. The ones from Zappos are all contemporary, short brim hats that make the wearer look like Jimmy Olsen.
So, I went through my options and tried to implement Desperation Plan B, Acquire Wool Felt To Make Own Hat.
So I thought about Joann Fabrics, which might have felt, or wool that I can felt:
Down at the bottom, it has a link to a store locator, wherein I could find out how early I can go into the store since I’m not sleeping so well without my hat (note: you actually can clutch a crushable felt fedora to your chest while you sleep, and it will snap back into shape when it’s time to slip into the rain).
So I click the store locator link, and:
The company apparently uses a third party bit of software that has gone night-night, and not in the cool queen with a sword way.
Which brings to mind a pretty good question: What do you do if your third party applications stop working? Do you have a back-up plan for an important Web site feature like the store locator? Do you just hope to wait it out and hope your vendor has not just been shut down by an overzealous government prosecutor in New Zealand? How fast do you act? Do you hide the link immediately while you frantically call your vendor’s customer support line, whose technician has left that particular cell phone in the club last night?
These are important questions to ask when choosing a third party piece of software, especially one sold as a service. It’s a bad thing to have to decide very suddenly.
And thank you for your concern about my ongoing hat situation. I’m back to Desperation Plan A: Hanging Out In Blues Bars and Hoping To Win A Cool Hat in a Poker Game or Steal One.