Selenium's Sweet Spot: Preventing Catastrophes

I'm proud of the automated tests I built at Sulia. They run in two minutes. They prevent big bugs from reaching production. The false positive rate is about 1%. In over a decade of working on big teams and on small ones, I haven't seen a more effective end-to-end testing regime. It didn't start that way. Nine months ago, our 3-person team had no UI testing at all. Along the way, we've learned some lessons beyond the obvious "use CSS selectors" stuff.

Lesson One: Selenium isn't always the best way to assure quality

The biggest lesson of all is the title of this post. Selenium's sweet spot is preventing catastrophes. As I discussed in an earlier post, the goal of test automation is to reduce costs throughout your product development lifecycle. The goal isn't to increase coverage. And it certainly isn't to increase coverage using a particular technology, such as Selenium. There are tons of tools for lowering development costs; manual black-box testing, unit/functional testing, and Selenium testing are among them. And some particularly effective cost reduction tools don't involve testing at all. Examples include continuous deployment, automated monitoring, and pager rotations.

As with just about anything else, Selenium testing is most cost effective when you don't overdo it. Catching catastrophic breakages is easy and cheap. It's harder and considerably more expensive to catch nitpick bugs with Selenium. The optimal point of Selenium coverage will vary from product to product. It'll be different for off the shelf apps and for apps on the public internet. But that optimal point will not involve writing a Selenium test to catch every potential bug.

Some months back, I attended a meetup on Selenium testing. One company that was new to Selenium boasted that they had written over 600 Selenium tests. Etsy, an acknowledged leader in engineering processes, boasted that they had written only seven. Etsy's presenter was proud of their quality and their uptime. His point was that Selenium was only a part of their testing strategy and that they used it to the point at which it was most cost-effective.

Lesson Two: Product development teams should own test development (at least initially)

Test development and product development are inherently linked. Product developers are the first to know when they've built something that needs to be tested. They're also the most qualified to assess the lowest cost way to test a feature. When product developers are writing the tests, they have a natural incentive to write testable code. When test developers are building the product, they the requisite knowledge to avoid writing brittle tests that depend on string matching or unstable xpaths.

Teams also need to be aware of a perverse organizational incentive. As I said in Lesson One, the goal of test automation is to reduce development costs. But when a separate test authoring team exists, it's natural to assume that the team is successful if it writes more tests. I'm not attacking the integrity of test developers here. Every test developer I've worked with has been a conscientious professional. But aligning a team around the non-goal of test development distorts their incentives and leads to a waste of effort.

I came from an organization that had separate product and test development teams to an organization that had a one-man team (me) for developing and testing our UI. I'm responsible for developing features that work and for supporting them when they don't work. For every feature, I decide the mix of unit, functional, and end-to-end tests that are most cost effective. The natural alignment of my incentives is a key reason that our test suites work as well as they do.

Lesson Three: Don't worry about cross-browser Selenium testing

One of the fantastic things about Selenium is that it works across browsers. But one of the awful things about browsers is that they aren't all terribly stable. Stabilizing your tests on Firefox, multiple IE versions, Chrome, and Safari is a ton of work. I tried and I gave up.

At Sulia, I run a core set of tests against multiple browsers but I run most of my tests against Chrome, the fastest and most stable browser. I'd rather have a stable set of tests that I can develop cheaply than a fully comprehensive set that's a lot more expensive for me to build and to maintain.

Back to the post's title: Selenium's sweet spot is preventing catastrophes. To prevent quirks, I use modern cross-browser frameworks, including CSS grids and cross-browser libraries like jQuery. These steps don't eliminate the probability of IE-only bugs. But they do reduce that probability to the point that it isn't worth my time to develop a lot of Selenium tests for IE. As I mentioned in Lesson One, I use other techniques, including manual testing and automatic monitoring. Nearly every cross-browser issue that sneaked past my tests and into production would have been undetectable by an automatic framework.

Lesson Four: Stay sane by skipping tests 

One aggravating problem with end-to-end tests is that they sometimes fail due to bizarre environmental problems. I've found that troubleshooting those problems is usually a waste of time. At Sulia, some of our tests require us to authenticate with twitter. Any twitter hiccups (which are shockingly frequent) during a test will hose that test .

When a test encounters a condition that's due to a transient environmental problem, I skip the test. It counts neither as a pass nor a failure. If there are no failures and if the skip count is reasonably small, then I am comfortable pushing code to production.

Lesson Five: Outsource your test environment

It's hard to build a working test environment. You need virtual machine images for multiple browsers, and you need to spin them up in parallel to make your test run go quickly. Until you reach a massive scale, it's foolish to hire a team to do that work for you in house. I outsource it to SauceLabs, the best vendor I've worked with for anything. For each test failure, I can view videos and screen shots that speed up troubleshooting dramatically. They also let me run tests in parallel. We run 32 Sulia tests at one time. It allows us to run our suite in under two minutes.

Remember that it's about cost

I've found Selenium to be an invaluable part of a testing and deployment process. But it's only one part. Find the most cost-effective ways to use it. Don't get carried away. And use the best tools and processes that you can find to increase your team's efficiency.

Comments

[Paul Baarn]

Hi,

Nice post. Very insightful. I'm just starting out with Selenium so it's great to see your lessons. I hope I will be able to put them to use.

Best, Paul