Software Testing

Don’t Bother Calculating ROI For Test Automation

Eric Jacobson's Software Testing Blog - Wed, 04/29/2015 - 11:10

Whilst searching for ways to measure the value of test automation, I read Doug Hoffman’s (I’m sure classic) Cost Benefits Analysis of Test Automation paper.

The first several pages were great.  He discussed intangibles that should be left out of an ROI calculation like an immediate reduction in perceived productivity of the test organization as the automation is first developed.  He went on to list the falsely expended benefits like the automation of existing manual tests.  Then he compared fixed automation costs like scripting tools to variable automation costs like test maintenance

Finally, Doug got to the formulas.  After careful analysis of some 30+ factors, one can start calculating automation ROI and efficiency benefits.  I rubbed my hands together and excitedly turned the page.  Then…I think I puked into my mouth a little as I saw the following:

 

In the end, I latched onto one powerful statement Doug made, almost in passing, he said:

If the benefits of automation are required, then the ROI computation is unnecessary, the investment is required…it’s an expense.

If the benefits include reducing risk by performing automated checking that would not be possible by humans (e.g., complex math, millions of comparisons, diffs, load, performance, precision), then say no more…

I don’t want to suffer through the computations.

Categories: Software Testing

Top 10 Ways To Ensure Your Best People Will Quit

QA Hates You - Tue, 04/28/2015 - 10:07

Ragan.com has the Top 10 Ways To Ensure Your People Quit.

I’ve hit some of these themes here before.

This list talks about active retention strategy and dedicates a couple of bullet points to it.

However, I’d swap out those for a couple of other ideas, such as:

  • Keep the employees doing the same things for a long period of time. The tedium of a couple of videos during orientation is nothing compared to doing the same tasks over and over again for years.
  • Don’t demonstrate the employee’s impact or meaning to the company effort. Especially in the auxiliary jobs–like QA–where employees might not see how their efforts are helping the company. Employees who feel forgotten, who feel as though they don’t matter, or feel as though they’re taken for granted are not employees for long.
  • Don’t succeed as a company. If your employee doesn’t see the company as a long-term success, the employee will look for one that is.
  • Build a corporate culture catering to one lifestyle. The stereotypical startup involves coding all night fueled by energy drinks and pizza, having crazy outings as a company, and funky office space with video games, a bar, and/or foosball/pool/bubble hockey tables. This is all well and good for a certain kind of employee–one fresh out of college or the parents’ basement, but if the culture favors only on those employees (especially if the culture is supposed to make up for lesser pay, longer hours, or other shortcomings), employees who move out of that phase of their lives will go look for a grown up company to work for.

That would bring the number up to more than ten, though.

The song says there are fifty ways to leave your lover (although the song itself does not enumerate them all, and Paul Simon marked that defect Resolved (Won’t Fix)). There are probably that many ways to lose your worker, too.

Categories: Software Testing

Web Performance News of the Week

LoadStorm - Fri, 04/24/2015 - 17:04

Mobile friendliness algorithm update went global this week

Google began globally rolling out their mobile search algorithm update to include mobile-friendly websites as a ranking signal. The updated search algorithm improves search result rankings for web pages that are optimized for mobile traffic, and should impact more sites than Google’s Panda and Penguin updates.

  • Only page rankings on mobile searches will be affected by the new search algorithm. Contrary to what was said in a Google hangout Q&A session earlier this year, the new algorithm is isolated to mobile searches only.
  • There are no degrees of mobile-friendliness; your page is either mobile-friendly or it isn’t. To check if your site is up to par, you can test your page URL using their Mobile-Friendly Test.
  • The mobile-friendly ranking changes will affect results on a page by page basis, not by entire websites.
  • Pages will automatically be re-processed if changes are made to make it mobile-friendly because the algorithm runs in real-time. To expedite the re-processing of the page, you can ask Google to re-crawl your URLs by using Fetch as Google with Submit to Index.

Google has reported that since two months ago, there are 4.7% more mobile-friendly sites relative to the whole. This is great news, as by 2020, 90 percent of the world’s population over 6 years old will have a mobile phone, and smartphone subscriptions are expected to top 6.1 billion.

Target website overwhelmed during Lilly Pulitzer frenzy

This week, the limited-edition Lilly Pulitzer collection went live on their website, giving Black Friday a run for it’s money. The resulting spike in traffic was too much for the site, causing it to crash several times and leaving customers unable to view their carts. Eventually, the entire Target website was overwhelmed and went down. Target responded to frustrated shoppers via Twitter, announcing that they were adjusting the site. Many customers noted that at that point it didn’t matter, because most of the stock sold out within minutes.

This isn’t the first time Target has experienced a crash after launching a fashion line. The 2011 limited edition launch of a high end Italian line, Missoni, caused the site to crash and remain down for most of the day. Many customers turned to the official Lilly Pulitzer website after the crash.

Security advisory goes out regarding WordPress Plugins and themes

A vulnerability re-discovered last week prompted a coordinated plugin update between multiple developers and the WordPress core security team to address a cross-site scripting (XSS) security vulnerability. The vulnerability was created after developers were misled by the WordPress Official Documentation, and misused two common functions. The add_query_arg() and remove_query_arg() functions are often used by developers to modify and add query strings to URLs. The two functions do not escape user input automatically, meaning the use of functions such as esc_url() or esc_url_raw() is necessary.

The Sucuri team analyzed the top 300-400 plugins, and found at least 15 different plugins containing the vulnerability:

Jetpack
WordPress SEO
Google Analytics by Yoast
All In one SEO
Gravity Forms
Multiple Plugins from Easy Digital Downloads
UpdraftPlus
WP-E-Commerce
WPTouch
Download Monitor
Related Posts for WordPress
My Calendar
P3 Profiler
Give
Multiple iThemes products including Builder and Exchange
Broken-Link-Checker
Ninja Forms

Whether your site uses any of the plugins listed above, it’s a good idea to update your plugins immediately to eliminate any risk.

IBM brings security analytics to the cloud

This Tuesday, IBM announced it will be launching two new threat analytic services in the cloud as Software as a Service tools to give companies the ability to quickly prioritize security threats.

The two new cloud services include IBM Security Intelligence and Intelligent Log Management. By bringing it’s Security Intelligence technology to the cloud, IBM customers will be able to analyze security threat information from over 500 different data sources for devices, systems and applications to determine if a real security threat exists, or if the security related events are simply anomalies. “The option of doing predictive analytics via the cloud gives security teams the flexibility to bring in skills, innovation and information on demand across all of their security environments.” said Jason Corbin, vice president of product management and strategy at IBM.

The post Web Performance News of the Week appeared first on LoadStorm.

To improve testing, snoop on the competition

The Quest for Software++ - Thu, 04/23/2015 - 01:47

This is an excerpt from my upcoming book, Fifty Quick Ideas To Improve Your Tests

As a general rule, teams focus the majority of testing activities on their zone of control, on the modules they develop, or the software that they are directly delivering. But it’s just as irresponsible not to consider competition when planning testing as it is in the management of product development in general, whether the field is software or consumer electronics.

Software products that are unique are very rare, and it’s likely that someone else is working on something similar to the product or project that you are involved with at the moment. Although the products might be built using different technical platforms and address different segments, key usage scenarios probably translate well across teams and products, as do the key risks and major things that can go wrong.

When planning your testing activities, look at the competition for inspiration — the cheapest mistakes to fix are the ones already made by other people. Although it might seem logical that people won’t openly disclose information about their mistakes, it’s actually quite easy to get this data if you know where to look.

Teams working in regulated industries typically have to submit detailed reports on problems caught by users in the field. Such reports are kept by the regulators and can typically be accessed in their archives. Past regulatory reports are a priceless treasure trove of information on what typically goes wrong, especially because of the huge financial and reputation impact of incidents that are escalated to such a level.

For teams that do not work in regulated environments, similar sources of data could be news websites or even social media networks. Users today are quite vocal when they encounter problems, and a quick search for competing products on Facebook or Twitter might uncover quite a few interesting testing ideas.

Lastly, most companies today operate free online support forums for their customers. If your competitors have a publicly available bug tracking system or a discussion forum for customers, sign up and monitor it. Look for categories of problems that people typically inquire about and try to translate them to your product, to get more testing ideas.

For high-profile incidents that have happened to your competitors, especially ones in regulated industries, it’s often useful to conduct a fake post-mortem. Imagine that a similar problem was caught by users of your product in the field and reported to the news. Try to come up with a plausible excuse for how it might have happened, and hold a fake retrospective about what went wrong and why such a problem would be allowed to escape undetected. This can help to significantly tighten up testing activities.

Key benefits

Investigating competing products and their problems is a cheap way of getting additional testing ideas, not about theoretical risks that might happen, but about things that actually happened to someone else in the same market segment. This is incredibly useful for teams working on a new piece of software or an unfamiliar part of the business domain, when they can’t rely on their own historical data for inspiration.

Running a fake post-mortem can help to discover blind spots and potential process improvements, both in software testing and in support activities. High-profile problems often surface because information falls through the cracks in an organisation, or people do not have sufficiently powerful tools to inspect and observe the software in use. Thinking about a problem that happened to someone else and translating it to your situation can help establish checks and make the system more supportable, so that problems do not escalate to that level. Such activities also communicate potential risks to a larger group of people, so developers can be more aware of similar risks when they design the system, and testers can get additional testing ideas to check.

The post-mortem suggestions, especially around improving the support procedures or observability, help the organisation to handle ‘black swans’ — unexpected and unknown incidents that won’t be prevented by any kind of regression testing. We can’t know upfront what those risks are (otherwise they wouldn’t be unexpected), but we can train the organisation to react faster and better to such incidents. This is akin to government disaster relief organisations holding simulations of floods and earthquakes to discover facilitation and coordination problems. It’s much cheaper and less risky to discover things like this in a safe simulated environment than learn about organisational cracks when the disaster actually happens.

How to make it work

When investigating support forums, look for patterns and categories rather than individual problems. Due to different implementations and technology choices, it’s unlikely that third-party product issues will directly translate to your situation, but problem trends or areas of influence will probably be similar.

One particularly useful trick is to look at the root cause analyses in the reports, and try to identify similar categories of problems in your software that could be caused by the same root causes.

Just Say No to More End-to-End Tests

Google Testing Blog - Wed, 04/22/2015 - 17:10
by Mike Wacker

At some point in your life, you can probably recall a movie that you and your friends all wanted to see, and that you and your friends all regretted watching afterwards. Or maybe you remember that time your team thought they’d found the next "killer feature" for their product, only to see that feature bomb after it was released.

Good ideas often fail in practice, and in the world of testing, one pervasive good idea that often fails in practice is a testing strategy built around end-to-end tests.

Testers can invest their time in writing many types of automated tests, including unit tests, integration tests, and end-to-end tests, but this strategy invests mostly in end-to-end tests that verify the product or service as a whole. Typically, these tests simulate real user scenarios.
End-to-End Tests in Theory While relying primarily on end-to-end tests is a bad idea, one could certainly convince a reasonable person that the idea makes sense in theory.

To start, number one on Google's list of ten things we know to be true is: "Focus on the user and all else will follow." Thus, end-to-end tests that focus on real user scenarios sound like a great idea. Additionally, this strategy broadly appeals to many constituencies:
  • Developers like it because it offloads most, if not all, of the testing to others. 
  • Managers and decision-makers like it because tests that simulate real user scenarios can help them easily determine how a failing test would impact the user. 
  • Testers like it because they often worry about missing a bug or writing a test that does not verify real-world behavior; writing tests from the user's perspective often avoids both problems and gives the tester a greater sense of accomplishment. 
End-to-End Tests in Practice So if this testing strategy sounds so good in theory, then where does it go wrong in practice? To demonstrate, I present the following composite sketch based on a collection of real experiences familiar to both myself and other testers. In this sketch, a team is building a service for editing documents online (e.g., Google Docs).

Let's assume the team already has some fantastic test infrastructure in place. Every night:
  1. The latest version of the service is built. 
  2. This version is then deployed to the team's testing environment. 
  3. All end-to-end tests then run against this testing environment. 
  4. An email report summarizing the test results is sent to the team.

The deadline is approaching fast as our team codes new features for their next release. To maintain a high bar for product quality, they also require that at least 90% of their end-to-end tests pass before features are considered complete. Currently, that deadline is one day away:

Days LeftPass %Notes15%Everything is broken! Signing in to the service is broken. Almost all tests sign in a user, so almost all tests failed.04%A partner team we rely on deployed a bad build to their testing environment yesterday.-154%A dev broke the save scenario yesterday (or the day before?). Half the tests save a document at some point in time. Devs spent most of the day determining if it's a frontend bug or a backend bug.-254%It's a frontend bug, devs spent half of today figuring out where.-354%A bad fix was checked in yesterday. The mistake was pretty easy to spot, though, and a correct fix was checked in today.-41%Hardware failures occurred in the lab for our testing environment.-584%Many small bugs hiding behind the big bugs (e.g., sign-in broken, save broken). Still working on the small bugs.-687%We should be above 90%, but are not for some reason.-789.54%(Rounds up to 90%, close enough.) No fixes were checked in yesterday, so the tests must have been flaky yesterday.
Analysis Despite numerous problems, the tests ultimately did catch real bugs.

What Went Well 
  • Customer-impacting bugs were identified and fixed before they reached the customer.

What Went Wrong 
  • The team completed their coding milestone a week late (and worked a lot of overtime). 
  • Finding the root cause for a failing end-to-end test is painful and can take a long time. 
  • Partner failures and lab failures ruined the test results on multiple days. 
  • Many smaller bugs were hidden behind bigger bugs. 
  • End-to-end tests were flaky at times. 
  • Developers had to wait until the following day to know if a fix worked or not. 

So now that we know what went wrong with the end-to-end strategy, we need to change our approach to testing to avoid many of these problems. But what is the right approach?
The True Value of Tests Typically, a tester's job ends once they have a failing test. A bug is filed, and then it's the developer's job to fix the bug. To identify where the end-to-end strategy breaks down, however, we need to think outside this box and approach the problem from first principles. If we "focus on the user (and all else will follow)," we have to ask ourselves how a failing test benefits the user. Here is the answer:

A failing test does not directly benefit the user. 

While this statement seems shocking at first, it is true. If a product works, it works, whether a test says it works or not. If a product is broken, it is broken, whether a test says it is broken or not. So, if failing tests do not benefit the user, then what does benefit the user?

A bug fix directly benefits the user.

The user will only be happy when that unintended behavior - the bug - goes away. Obviously, to fix a bug, you must know the bug exists. To know the bug exists, ideally you have a test that catches the bug (because the user will find the bug if the test does not). But in that entire process, from failing test to bug fix, value is only added at the very last step.

StageFailing TestBug OpenedBug FixedValue AddedNoNoYes
Thus, to evaluate any testing strategy, you cannot just evaluate how it finds bugs. You also must evaluate how it enables developers to fix (and even prevent) bugs.
Building the Right Feedback LoopTests create a feedback loop that informs the developer whether the product is working or not. The ideal feedback loop has several properties:
  • It's fast. No developer wants to wait hours or days to find out if their change works. Sometimes the change does not work - nobody is perfect - and the feedback loop needs to run multiple times. A faster feedback loop leads to faster fixes. If the loop is fast enough, developers may even run tests before checking in a change. 
  • It's reliable. No developer wants to spend hours debugging a test, only to find out it was a flaky test. Flaky tests reduce the developer's trust in the test, and as a result flaky tests are often ignored, even when they find real product issues. 
  • It isolates failures. To fix a bug, developers need to find the specific lines of code causing the bug. When a product contains millions of lines of codes, and the bug could be anywhere, it's like trying to find a needle in a haystack. 
Think Smaller, Not LargerSo how do we create that ideal feedback loop? By thinking smaller, not larger.

Unit TestsUnit tests take a small piece of the product and test that piece in isolation. They tend to create that ideal feedback loop:

  • Unit tests are fast. We only need to build a small unit to test it, and the tests also tend to be rather small. In fact, one tenth of a second is considered slow for unit tests. 
  • Unit tests are reliable. Simple systems and small units in general tend to suffer much less from flakiness. Furthermore, best practices for unit testing - in particular practices related to hermetic tests - will remove flakiness entirely. 
  • Unit tests isolate failures. Even if a product contains millions of lines of code, if a unit test fails, you only need to search that small unit under test to find the bug. 

Writing effective unit tests requires skills in areas such as dependency management, mocking, and hermetic testing. I won't cover these skills here, but as a start, the typical example offered to new Googlers (or Nooglers) is how Google builds and tests a stopwatch.

Unit Tests vs. End-to-End TestsWith end-to-end tests, you have to wait: first for the entire product to be built, then for it to be deployed, and finally for all end-to-end tests to run. When the tests do run, flaky tests tend to be a fact of life. And even if a test finds a bug, that bug could be anywhere in the product.

Although end-to-end tests do a better job of simulating real user scenarios, this advantage quickly becomes outweighed by all the disadvantages of the end-to-end feedback loop:

UnitEnd-toEndFast

Reliable


Isolates Failures

Simulates a Real User


Integration TestsUnit tests do have one major disadvantage: even if the units work well in isolation, you do not know if they work well together. But even then, you do not necessarily need end-to-end tests. For that, you can use an integration test. An integration test takes a small group of units, often two units, and tests their behavior as a whole, verifying that they coherently work together.

If two units do not integrate properly, why write an end-to-end test when you can write a much smaller, more focused integration test that will detect the same bug? While you do need to think larger, you only need to think a little larger to verify that units work together.
Testing PyramidEven with both unit tests and integration tests, you probably still will want a small number of end-to-end tests to verify the system as a whole. To find the right balance between all three test types, the best visual aid to use is the testing pyramid. Here is a simplified version of the testing pyramid from the opening keynote of the 2014 Google Test Automation Conference:



The bulk of your tests are unit tests at the bottom of the pyramid. As you move up the pyramid, your tests gets larger, but at the same time the number of tests (the width of your pyramid) gets smaller.

As a good first guess, Google often suggests a 70/20/10 split: 70% unit tests, 20% integration tests, and 10% end-to-end tests. The exact mix will be different for each team, but in general, it should retain that pyramid shape. Try to avoid these anti-patterns:
  • Inverted pyramid/ice cream cone. The team relies primarily on end-to-end tests, using few integration tests and even fewer unit tests. 
  • Hourglass. The team starts with a lot of unit tests, then uses end-to-end tests where integration tests should be used. The hourglass has many unit tests at the bottom and many end-to-end tests at the top, but few integration tests in the middle. 
Just like a regular pyramid tends to be the most stable structure in real life, the testing pyramid also tends to be the most stable testing strategy.


Categories: Software Testing

If You Interrupt Testing, It Will Cost Us 2 Lost Bugs

Eric Jacobson's Software Testing Blog - Wed, 04/22/2015 - 10:42

Look at your calendar (or that of another tester).  How many meetings exist?

My new company is crazy about meetings.  Perhaps it’s the vast numbers of project managers, product owners, and separate teams along the deployment path.  It’s a wonder programmers/testers have time to finish anything.

Skipping meetings works, but is an awkward way to increase test time.  What if you could reduce meetings or at least meeting invites?  Try this.  Express the cost of attending the meeting in units of lost bugs.  If you find, on average, about 1 bug per hour of testing, you might say:

“Sure, I can attend your meeting, but it will cost us 1 lost bug.”

“This week’s meetings cost us 9 lost bugs.”

Obviously some meetings (e.g., design, user story review, bug triage) improve your bug finding, so be selective when choosing to declare the bugs lost cost.

Categories: Software Testing

The Day I Quit

QA Hates You - Wed, 04/22/2015 - 08:37

When I was a lad, fresh out of the university with a degree in English and Philosophy and no actual career prospects, I worked as a produce clerk for a small off-chain produce and cheese shop. They had daily garbage pickup on weekdays, but nothing on the weekends, which were some of the busiest days of the week. As a result, on Sunday afternoons, the dumpster started to fail boundary analysis, at which time the store manager would order a clerk or two to climb up onto the pile and jump up and down to compact it so we could dump the last few cans of refuse into it. Come to think of it, I’ve seen the same philosophy applied to hardware resource management.

So as I stood and watched the younger kids jumping in the dumpster, I decided that if I was ever ordered to climb into the dumpster, I would drop my apron in the alley and never come back.

Want to know what would make me leave QA? Needing an implant of some sort to do my job:

PayPal is working on a new generation of embeddable, injectable and ingestible devices that could replace passwords as a means of identification.

Jonathan LeBlanc, PayPal’s global head of developer evangelism, claims that these devices could include brain implants, wafer-thin silicon chips that can be embedded into the skin, and ingestible devices with batteries that are powered by stomach acid.

These devices would allow “natural body identification,” by monitoring internal body functions like heartbeat, glucose levels and vein recognition, Mr LeBlanc told the Wall Street Journal.

Over time they would come to replace passwords and even more advanced methods of identification, like fingerprint scanning and location verification, which he says are not always reliable.

I’d rather not be personally, bodily on the Internet of Things unless there’s a compelling medical reason for it, and even then I’m going to ask my doctor to examine all the steampunk options first.

Categories: Software Testing

Web Performance News of the Week

LoadStorm - Mon, 04/20/2015 - 16:46

Google modifies the structure of URLs on mobile

This week Google made a change to the URL structure that is displayed normally from a mobile search, explaining that “well-structured URLs offer users a quick hint about the page topic and how the page fits within the website”. The algorithms were updated starting in the USin an effort to display names that would better reflect the page. This will structure the results using a breadcrumbs-like format, and use the “real-world name of the site” instead of the domain name. I checked search results on my mobile, but the change had not appeared to be made yet. Here is an example search Google used:

To complement the launch, Google also announced support for schema.org, a site that provides a collection of schemas that webmasters can use to markup HTML pages in ways recognized by major search engines. This intends to help webmasters signal both the website name that should be used instead of the domain name, as well as the URL structure as breadcrumbs.

Here are the examples Google pointed to in their documentation on site names and breadcrumbs.

The announcement garnered mixed reactions. While some people are calling the change helpful, arguing that the change encourages better site organization, others argue that by not displaying the actual URL, users no longer could easily verify the true identity of a site. What do you think about this interesting development?

 

Google adds mobile-friendliness as a part of their ranking algorithm

Tomorrow marks the official launch of Google’s new ranking algorithm, which will add mobile-friendliness as a ranking signal. Although Google held a live Q&A hangout a month ago, it was somewhat unclear whether or not the changes could be seen before the official launch date on April 21st.

So how do you know if your page is mobile friendly? The easiest way to check if a page on your site passes is by typing in the URL here.

This will test the way your site looks to a Googlebot. It checks for:

  • pages automatically sizing content to the screen
  • avoiding software that is uncommon on mobile devices
  • containing readable text without zooming
  • and placing links far enough apart to be selected easily

 

A few easy ways to see what your site looks like on an array of different devices, and to start making your site mobile-friendly, can be found in our responsive web design blog post.

 

Additional algorithm details to note include:

  • There are no degrees of mobile-friendliness; your page is either mobile-friendly or it isn’t. Google added that it will be the same with desktop search, and is not isolated to mobile searches.
  • The mobile-friendly ranking changes will affect your site on a page by page basis. So if half of your site’s pages are mobile-friendly, those half will benefit from the mobile friendly ranking changes. This is good news for anyone concerned that their site doesn’t make the cut, as they can focus on getting their main pages up to date first.
  • The new ranking algorithm will officially begin on April 21st. However, the algorithm runs in real-time, meaning you can start preparing your site for analysis now. It was said in the hangout that it may even take a week or so to completely roll out, so it’s not entirely clear how quick we can expect to see changes in site rankings.
  • Google News may not be ranked by the new mobile-friendly algorithm yet. Interestingly, the Google News ranking team has no plans on implementing the mobile-friendly algorithm into the Google News results.

Zuckerberg’s internet.org loses support, with companies siting net neutrality concerns

A facebook backed project intended to make the internet accessible for people in the developing world has lost the support of several prominent companies this week. In the midst of a fierce national debate regarding net neutrality, the New Delhi Television Limited (NDTV), the Times Group, a media company that owns Times of India, and Cleartrip, a travel website all removed content from Internet.org in India, which they had made available for free. Furthermore, the companies urged its competitors to withdraw as well.

“We support net neutrality because it creates a fair, level playing field for all companies – big and small – to produce the best service and offer it to consumers. We will lead the drive towards a neutral internet, but we need our fellow publishers and content providers to do so as well, so that the playing field continues to be level.” – statement from a Times official.

The companies sited concerns that the Internet.org initiative did not align with the net neutrality mission, arguing that Internet.org is set up to prioritize content from partner businesses who pay telecom companies for data charges.

Zuckerberg responded to critics, affirming that net neutrality and Internet.org “can and must coexist.” He detailed his stance on the controversy, proclaiming that “Internet.org doesn’t block or throttle any other services or create fast lanes — and it never will. We’re open for all mobile operators and we’re not stopping anyone from joining. We want as many internet providers to join so as many people as possible can be connected.”

The Internet.org initiative has reportedly expanded internet accesss to over six countries so far, including Columbia, India, Zambia, Tanzania, Kenya and Ghana. Still, some worry that the long term net neutrality implications outweigh the immediate access it provides to the developing world.

The post Web Performance News of the Week appeared first on LoadStorm.

How To Start A Challenging Test

Eric Jacobson's Software Testing Blog - Thu, 04/16/2015 - 15:58

Last week I started testing an update to a complex legacy process.  At first, my head was spinning (it still kind of is).    There are so many inputs and test scenarios...so much I don’t understand.  Where to begin?

I think doing something half-baked now is better than doing something fully-baked later.  If we start planning a rigorous test based on too many assumptions we may not understand what we’re observing. 

In my case, I started with the easiest tests I could think of:

  • Can I trigger the process-under-test? 
  • Can I tell when the process-under-test completes?
  • Can I access any internal error/success logging for said process?
  • If I repeat the process-under-test multiple times, are the results consistent?

If there were a spectrum that showed a focus between learning by not manipulating and learning by manipulating something-under-test, it might look like this:

My tests started on the left side of the spectrum and worked right.  Now that I can get consistent results, let me see if I can manipulate it and predict its results: 

  • If I pass ValueA to InputA, do the results match my expectations?
  • If I remove ValueA from InputA, do the results return as before?
  • If I pass ValueB to InputA, do the results match my expectations?

As long as my model of the process-under-test matches my above observations, I can start expanding complexity:

  • If I pass ValueA and ValueB to InputA and ValueC and ValueD to InputB, do the results match my expectations?
  • etc.

Now I have something valuable to discuss with the programmer or product owner.  “I’ve done the above tests.  What else can you think of?”.  It’s much easier to have this conversation when you’re not completely green, when you can show some effort.  It’s easier for the programmer or product owner to help when you lead them into the zone.

That worst is over.  The rest is easy.  Now you can really start testing!

Sometimes you just have to do something to get going.  Even if it’s half-baked.

Categories: Software Testing

Web Performance News of the Week

LoadStorm - Fri, 04/10/2015 - 10:47

This week in web performance news, Github victors over DDoS attacks, IE’s replacement gets clarifications, and IBM invests in the Internet of Things.

GitHub DDOS attack

This week GitHub endured a large scale distributed denial of service (DDoS) that began on Thursday and lasted for 5 days. GitHub said there were a number of vectors being used in this attack, including sophisticated techniques that used the web browsers of unsuspecting users to flood the GitHub site with traffic. While GitHub proclaimed the attack was the largest in company history, they were able to mitigate and deflect most of the evolving attack, maintaining stable service overall. In fact, watching the status messages from GitHub was kind of exciting.

 

 

 

 

 

 

 

 

 

 

The GitHub pages targeted by redirecting search traffic from Chinese search engine Baidu included a Chinese anti-censorship service GreatFire.org and a mirror site of The New York Times’ Chinese edition. Security experts analyzing the attacks reported that the machines attacking GitHub were located “on or near the Great Firewall of China”, and while other explanations were possible, “the overwhelmingly most likely suspect for the source of the GitHub attacks is the Chinese government.”

Clarifications on Project Spartan replacing IE

The head of Microsoft Marketing, Chris Capossela, recently announced IE would no longer continue as a brand and will be replaced by Project Spartan, their next generation browser.

Last week on the IE blog, the Project Spartan team announced there will be a clean distinction between the legacy and new browser in an effort to simplify the role of each browser. On Windows 10, Project Spartan will host an entirely new engine. Similarly, IE 11 will remain fundamentally the same from Windows 8.1 and will exclusively use the legacy engine.

 

 

 

 

 

 

 

 

 

 

“Project Spartan is our future” said Kyle Pflug, the Project Spartan program manager. Calls for help from developers were made to test their new rendering engine in the Windows 10 Technical Preview.

IBM invests $3 Billion in the Internet of Things

IBM announced that it would be investing $3 billion dollars over the next four years on a business model aiming to analyze the growth of data being generated by a surplus of devices containing sensors connected to the internet. IBM’s new Internet of Things foundation is a cloud software tool that gives clients and partner companies the ability to modify their existing apps, or even build new ones. IBM can then analyze the data collected from the applications from a large variety of devices. In addition to creating and selling software to help businesses analyze their data, IBM is also designing supercomputer hardware capable of efficiently storing and computing large amounts of data.

The post Web Performance News of the Week appeared first on LoadStorm.

Which Tests Should Be Automated?

Eric Jacobson's Software Testing Blog - Tue, 04/07/2015 - 10:42

Now that my engineering team is automating beyond the unit test level, this question comes up daily.  I wish there were an easy answer.

If we make a distinction between checking and testing, no “tests” should be automated.  The question instead becomes, which “checks” should be automated?  Let’s go with that.

I’ll tell you what I think below, ranking the more important at the top:

Consider automating checks when they…

  1. can only feasibly be checked by a machine (e.g., complex math, millions of comparisons, diffs, load, performance, precision).  These are checks machines do better than humans.
  2. are important.  Do we need to execute them prior to each build, deployment, or release?  This list of checks will grow over time.  The cost of not automating is less time for the “testing” that helps us learn new information.
  3. can be automated below the presentation layer.  Automating checks at the API layer is considerably less expensive than at the UI layer.  The automated checks will provide faster feedback and be less brittle.
  4. will be repeated frequently.  A simplified decision: Is the time it takes a human to program, maintain, execute, and interpret the automated check’s results over time (e.g., 2 years), less than the time it takes a human to perform said check over the same time span (e.g., 2 years).  This overlaps with #2.
  5. check something that is at risk of failing.  Do we frequently break things when we change this module?
  6. are feasible to automate.  We can’t sufficiently automate tests for things like usability and charisma.
  7. are requested by my project team.  Have a discussion with your project team about which checks to automate.  Weigh your targets with what your team thinks should be automated.
  8. can be automated using existing frameworks and patterns.  It is much cheaper to automate the types of checks you’ve already successfully automated.
  9. are checks.  “Checks” are mundane for humans to perform.  “Tests” are not because they are different each time. 

 

What am I missing?

Categories: Software Testing

The *Real* Test Driven Development

Google Testing Blog - Thu, 04/02/2015 - 09:35

Update: APRIL FOOLS!

by Kaue Silveira

Here at Google, we invest heavily in development productivity research. In fact, our TDD research group now occupies nearly an entire building of the Googleplex. The group has been working hard to minimize the development cycle time, and we’d like to share some of the amazing progress they’ve made.

The Concept

In the ways of old, it used to be that people wrote tests for their existing code. This was changed by TDD (Test-driven Development), where one would write the test first and then write the code to satisfy it. The TDD research group didn’t think this was enough and wanted to elevate the humble test to the next level. We are pleased to announce the Real TDD, our latest innovation in the Program Synthesis field, where you write only the tests and have the computer write the code for you!

The following graph shows how the number of tests created by a small feature team grew since they started using this tool towards the end of 2013. Over the last 2 quarters, more than 89% of this team’s production code was written by the tool!

See it in action:

Test written by a Software Engineer:

class LinkGeneratorTest(googletest.TestCase):

def setUp(self):
self.generator = link_generator.LinkGenerator()

def testGetLinkFromIDs(self):
expected = ('https://frontend.google.com/advancedSearchResults?'
's.op=ALL&s.r0.field=ID&s.r0.val=1288585+1310696+1346270+')
actual = self.generator.GetLinkFromIDs(set((1346270, 1310696, 1288585)))
self.assertEqual(expected, actual)
Code created by our tool:

import urllib

class LinkGenerator(object):

_URL = (
'https://frontend.google.com/advancedSearchResults?'
's.op=ALL&s.r0.field=ID&s.r0.val=')

def GetLinkFromIDs(self, ids):
result = []
for id in sorted(ids):
result.append('%s ' % id)
return self._URL + urllib.quote_plus(''.join(result))

Note that the tool is smart enough to not generate the obvious implementation of returning a constant string, but instead it correctly abstracts and generalizes the relation between inputs and outputs. It becomes smarter at every use and it’s behaving more and more like a human programmer every day. We once saw a comment in the generated code that said "I need some coffee".

How does it work?

We’ve trained the Google Brain with billions of lines of open-source software to learn about coding patterns and how product code correlates with test code. Its accuracy is further improved by using Type Inference to infer types from code and the Girard-Reynolds Isomorphism to infer code from types.

The tool runs every time your unit test is saved, and it uses the learned model to guide a backtracking search for a code snippet that satisfies all assertions in the test. It provides sub-second responses for 99.5% of the cases (as shown in the following graph), thanks to millions of pre-computed assertion-snippet pairs stored in Spanner for global low-latency access.



How can I use it?

We will offer a free (rate-limited) service that everyone can use, once we have sorted out the legal issues regarding the possibility of mixing code snippets originating from open-source projects with different licenses (e.g., GPL-licensed tests will simply refuse to pass BSD-licensed code snippets). If you would like to try our alpha release before the public launch, leave us a comment!

Categories: Software Testing

Quantum Quality

Google Testing Blog - Wed, 04/01/2015 - 03:30
by Kevin Graney

Here at Google we have a long history of capitalizing on the latest research and technology to improve the quality of our software. Over our past 16+ years as a company, what started with some humble unit tests has grown into a massive operation. As our software complexity increased, ever larger and more complex tests were dreamed up by our Software Engineers in Test (SETs).

What we have come to realize is that our love of testing is a double-edged sword. On the one hand, large-scale testing keeps us honest and gives us confidence. It ensures our products remain reliable, our users' data is kept safe, and our engineers are able to work productively without fear of breaking things. On the other hand, it's expensive in both engineer and machine time. Our SETs have been working tirelessly to reduce the expense and latency of software tests at Google, while continuing to increase their quality.

Today, we're excited to reveal how Google is tackling this challenge. In collaboration with the Quantum AI Lab, SETs at Google have been busy revolutionizing how software is tested. The theory is relatively simple: bits in a traditional computer are either zero or one, but bits in a quantum computer can be both one and zero at the same time. This is known as superposition, and the classic example is Schrodinger's cat. Through some clever math and cutting edge electrical engineering, researchers at Google have figured out how to utilize superposition to vastly improve the quality of our software testing and the speed at which our tests run.


Figure 1 Some qubits inside a Google quantum device.
With superposition, tests at Google are now able to simultaneously model every possible state of the application under test. The state of the application can be thought of as an n bit sequential memory buffer, consistent with the traditional Von Neuman architecture of computing. Because each bit under superposition is simultaneously a 0 and a 1, these tests can simulate 2n different application states at any given instant in time in O(n) space. Each of these application states can be mutated by application logic to another state in constant time using quantum algorithms developed by Google researchers. These two properties together allow us to build a state transition graph of the application under test that shows every possible application state and all possible transitions to other application states. Using traditional computing methods this problem has intractable time complexity, but after leveraging superposition and our quantum algorithms it becomes relatively fast and cheap.

Figure 2 The application state graph for a demonstrative 3-bit application. If the start state is 001 then 000, 110, 111, and 011 are all unreachable states. States 010 and 100 both result in deadlock.
Once we have the state transition graph for the application under test, testing it becomes almost trivial. Given the initial startup state of the application, i.e. the executable bits of the application stored on disk, we can find from the application's state transition graph all reachable states. Assertions that ensure proper behavior are then written against the reachable subset of the transition graph. This paradigm of test writing allows both Google's security engineers and software engineers to work more productively. A security engineer can write a test, for example, that asserts "no executable memory regions become mutated in any reachable state". This one test effectively eliminates the potential for security flaws that result from memory safety violations. A test engineer can write higher level assertions using graph traversal methods that ensure data integrity is maintained across a subset of application state transitions. Tests of this nature can detect data corruption bugs.
We're excited about the work our team has done so far to push the envelope in the field of quantum software quality. We're just getting started, but based on early dogfood results among Googlers we believe the potential of this work is huge. Stay tuned!


Categories: Software Testing

I’ll Take The Pack of Rabid, Underfed Ferrets

QA Hates You - Tue, 03/31/2015 - 14:05

Do you even consider what truncation might do to your email subject lines?

I don’t know about you, but I’ll take whatever the choice is because I don’t want to win a baby.

When you’re testing emails, give some thought to the crazy and unfortunate ways they might get truncated by email clients, will you? We’ve all seen the business analyst or sales assistant job posting email and snickered.

Don’t let us snicker at you.

Categories: Software Testing

Web Performance News of the Week

LoadStorm - Fri, 03/27/2015 - 15:06

This week in web performance, Google clarified the new mobile-friendly ranking algorithm details, Facebook open-sourced its Augmented Traffic Control tool, and internet service providers filed lawsuit against the FCC to stop net neutrality.

Google clarifies the Mobile-friendly algorithm details

The April 21st launch date for the new mobile-friendly ranking algorithm is approaching quickly, This week Google held a live Q&A hangout to provide more details about what we can expect.

Here’s what you need to know:

  • There are no degrees of mobile-friendliness; your page is either mobile-friendly or it isn’t. Google added that it will be the same with desktop search, and is not isolated to mobile searches.
  • The mobile-friendly ranking changes will affect your site on a page by page basis. So if half of your site’s pages are mobile-friendly, those half will benefit from the mobile friendly ranking changes. This is good news for anyone concerned that their site doesn’t make the cut, as they can focus on getting their main pages up to date first.
  • The new ranking algorithm will officially begin on April 21st. However, the algorithm runs in real-time, meaning you can start preparing your site for analysis now. It was said in the hangout that it may even take a week or so to completely roll out, so it’s not entirely clear how quick we can expect to see changes in site rankings.
  • Google News may not be ranked by the new mobile-friendly algorithm yet. Interestingly, the Google News ranking team has no plans on implementing the mobile-friendly algorithm into the Google News results.

So how do you know if your page is friendly? The easiest way to check if your page passes is by typing in your site’s URL here.

The most important criteria that need to be met by Googlebots are:

  • pages automatically sizing content to the screen
  • avoiding software that is uncommon on mobile devices
  • containing readable text without zooming
  • and placing links far enough apart to be selected easily

It’s important to note that a page may look perfect on your device, but may look completely different on your coworker’s phone. A few easy ways to start making your site mobile-friendly can be found in our responsive web design blog post. It’s also very important to note that there are over 200 different criteria used to rank your site, so it’s possible these changes may not affect your site very much.

Internet providers file lawsuits to reverse net neutrality laws

USTelecom (who represents AT&T), Verizon, Alamo Broadband, and other companies, filed two separate lawsuits against the Federal Communications Commission in the US Court of Appeals this week. The companies want the FCC rules to be ignored and say the FCC acted outside of their authority, violating the companies’ constitutional rights. Both suits acknowledged the their challenges may be premature, but were filed nonetheless “out of an abundance of caution.” The FCC acknowledged the suits and believed that the petitions for review filed were “premature and subject to dismissal.”

 

Facebook to bring fast internet to the entire world by open sourcing its Augmented Traffic Tool

This week, Facebook announced they would be making their custom built Augmented Traffic Control (ATC) tool open-source. Facebook developers have used the tool to simulate different types of network connections, including older 2G and Edge mobile data networks, and networks that frequently become disconnected. By testing how a site or app perform under these conditions , they are able to optimize their app performance in low connection areas around the world.

The tool started out as an open-source project itself, and now can continue to evolve, allowing developers around the world to use and improve it. Mark Zuckerberg recently spoke about Internet.org at the 2015 Mobile World Congress in Spain. The not for profit is Facebook’s commitment to make the internet accessible to people in the developing world and wrote in his letter to potential investors, “we don’t build services to make money; we make money to build better services.”

The post Web Performance News of the Week appeared first on LoadStorm.

Getting Manual Testers Involved in Automation

Eric Jacobson's Software Testing Blog - Fri, 03/27/2015 - 08:23

Most of the testers at my new company do not have programming skills (or at least are not putting them to use).  This is not necessarily a bad thing.  But in our case, many of the products-under-test are perfect candidates for automation (e.g., they are API rich).

We are going through an Agile transformation.  Discussions about tying programmatic checks to “Done” criteria are occurring and most testers are now interested in getting involved with automation.  But how?

I think this is a common challenge.

Here are some ways I have had success getting manual testers involved in automation.  I’ll start with the easiest and work my way down to those requiring more ambition.  A tester wanting to get involved in automation can:

  1. Do unit test reviews with their programmers.  Ask the programmers to walk you through the unit tests.  If you get lost ask questions like, “what would cause this unit test to fail?” or “can you explain the purpose of this test at a domain level?”.
  2. Work with automators to inform the checks they automate.  If you have people focused on writing automated checks, help them determine what automation might help you.  Which checks do you often repeat?  Which are boring?
  3. Design/request a test utility that mocks some crucial interface or makes the invisible visible.  Bounce ideas off your programmers and see if you can design test tools to speed things up.  This is not traditional automation.  But it is automation by some definitions.
  4. Use data-driven automation to author/maintain important checks via a spreadsheet.  This is a brilliant approach because it lets the test automater focus on what they love, designing clever automation.  It lets the tester focus on what they love, designing clever inputs.  Show the tester where the spreadsheet is and how to kick off the automation.
  5. Copy and paste an automated check pattern from an IDE, rename the check and change the inputs and expected results to create new checks.  This takes 0-to-little coding skills.  This is a potential end goal.  If a manual tester gets to this point, buy them a beer and don’t push them further.  This leads to a great deal of value, and going further can get awkward.
  6. Follow an automated check pattern but extend the framework.  Spend some time outside of work learning to code. 
  7. Stand up an automation framework, design automated checks.  Support an Agile team by programming all necessary automated checks.  Spend extensive personal time learning to code.  Read books, write personal programs, take online courses, find a mentor. 
Categories: Software Testing

The Parable of the Deer

QA Hates You - Fri, 03/27/2015 - 04:43

So I’m coming back from an errand early one morning, and I turn onto the farm road where I live. I live out in the country, you see, and the farm road is a long, straight road that rolls over hills and creeks between fences, woods, and pastures. As I’m driving down the farm road, I think to myself, Although it’s technically day time, on cloudy days, deer are often active later than normal. I remember a particular stretch of the road just before the creek, where a cow pasture faces woods and where I’ve seen deer before. Then, as the relative elevation of the road changes in relation to the pasture, I see a single deer in the middle of the pasture against the backdrop of the hills beyond, and I slow my truck to ten miles an hour. Where are your buddies? I ask him, because does and fawns travel together in family groups.

Two deer dart across the road, and two others turn from the fence they were about to hop and retreat into the pasture. If I’d been traveling normal speed, I very well might have hit one of them.

I know the general behavior of deer; I know the lay of the land and the geography of deer crossings; and I just might have seen the deer in my peripheral vision, outside my focus but enough to trigger additional caution until I did see it consciously. That’s how I knew the deer were there.

And that’s how I found that bug. I know the general behavior of computers, applications, and interfaces; I know something of the domain or problem this program is trying to solve; and I have wide peripheral vision when testing, the ability to see things wrong in the corner of my eye and retry my actions with focus on the problem area.

Categories: Software Testing

How Much Of It Would QA Chew If QA Would Chew It?

QA Hates You - Thu, 03/26/2015 - 12:09

Nihilist chewing gum.

Never mind searching for it on Amazon.com. It’s futile. Also, it’s not available.

(Link source.)

Categories: Software Testing

Failed Data Integration Costs Customers Hundreds of Dollars

QA Hates You - Wed, 03/25/2015 - 08:04

A failed data integration is going to cost St. Louis area residents up to hundreds of dollars. Or require a refund.

No, that extra few hundred dollars on your monthly sewer bill isn’t a typo.

A bill miscalculation that began nearly two years ago has the Metropolitan St. Louis Sewer District asking thousands of customers in St. Louis County to pay for services that never showed up on bills. The average undercharge for an affected household: $450.

Nearly 1,900 residential customers and 3,700 commercial customers are being asked to pay more on their current sewer bill, which should be in mailboxes by the week of April 6 at the latest.

The reason?

The discrepancy began in May 2013, when Missouri American Water, which provides water service in much of the St. Louis County portion of MSD’s territory, changed its billing system.

MSD buys water meter data from Missouri American in order to calculate many customer rates, but the new system’s data wasn’t properly computing with MSD’s billing programs, district spokesman Lance LeComb said.

“Ultimately MSD is responsible for getting the data right and making sure we have correct account information and send out correct bills,” he said. “Missouri American Water was very supportive of our efforts and provided additional staff” to fix the problem.

Dependency upon an integration, and something at the data provider changed. Expensively.

On the MSD bills, they encourage you to “go green” and pay your bills online:

Given their track record, I can understand anyone’s reluctance to allow MSD’s computer systems access to a customer bank account or credit card.

Categories: Software Testing

Pages