Stop thinking about test coverage and start writing good tests

The problem with test coverage

The Salesforce platform enforces a minimum 75% code coverage for code to be promoted to production. This is of course a very good way to encourage developers to test code extensively. Well, at least in principle.

Getting away with testing. I've worked with many developers that see unit testing as a painful task you need to do with the least possible effort, in order to get away with coverage requirements and get the code out. Tests are often left at the end of development, and seen as a burden rather than something valuable. This is very bad for quality, and sooner or later, it backfires.

High test coverage ≠ robust code. Some people seem to fundamentally misunderstand what code coverage is, and what it tells as a code metric. Code coverage is a measure of how many lines, blocks, arcs are executed by your test methods. It is simply a risk indicator, that does not tell you if your code is robust. In fact, code coverage is pretty easy to trick.

Risks of unstructured testing. If you only worry about coverage and approach testing without a clear direction, your project can (and probably will) be damaged in multiple ways:

  • quality issues to emerge after development
  • subsequent code changes to cause regressions as the complexity grows
  • too much time/money spent to maintain (fix) tests as the solution grows

Being pragmatic and finding the right trade-off is fundamental. Developers should know with great precision what to test and how to test it.

Good testing

Why should you aspire to good testing? And what makes tests good? Let's start from the foundations.

Benefits of good testing

Good testing is what ultimately makes you agile.

A tough client once told me: «Scrum is not what makes you agile. Good testing is!». My initial reaction was probably having a laugh, but retrospectively I must admit he was deeply right. Good testing has some very tangible benefits.

  • Coding with confidence. Good testing allows developers to code with confidence and make sure, at any point in time, that what they are building works, and works well. This generally translates in a growing sense of investment and pride for the team that will build up with time, as the solution complexity grows.
  • Everybody knows precisely how the code is supposed to work. Good unit tests are a form of testable comments - they allow developers to keep the focus on what they build and state how they should behave, without the need of extensively documenting it.
  • Enable continuous change. Testing allows to add and remove features or refactor the existing code whenever needed, ensuring stability is not affected.
  • Enable continuous delivery. The high degree of automation, repeatability and isolation allows good tests to be continuously executed and checked throughout the development lifecycle, giving confidence to IT and enabling faster delivery.
  • Save money in the long-term. Fixing bugs late is expensive. Good tests allow to spot bugs early in development, which is the most cost-effective way to address them.

Turns out all these things are actually at the very heart of agility.

Properties of Good Tests

So the question is: what makes a test good? Hunt and Thomas wrote good tests are A-TRIP, which is an acronym for the following five properties.

Automatic. Unit tests need to run automatically. This involves both running the tests and checking the results. Whatever initialisation is required by the test to run should also be fully automated. The good news here is that the Force.com platform handles this in a number of ways (hint: Apex Testing Framework, Tooling API) so, as long as you know your tools, you have this covered.

Thorough. Good unit tests must be thorough and should test and check everything that is likely to break. The bad news is that thoroughness doesn't come out of the box. It's not built-in the platform, it's entirely up to architects and developers. This is potentially huge: we need to be pragmatic to go in depth, whilst avoid over testing.

Repeatable. You need to be able to repeat the same tests over and over, every time any change is introduced in the code, and tests should always product the same result. Again, the platform allows to do this in many ways (hint: Tooling API, Force.com Migration Tool), so let's not worry too much about repeatability for the moment.

Independent. Tests should be independent from one another, meaning that the order in which they are executed must be irrelevant. Also they should not depend on data, network connectivity, and other environmental factors. This is sometimes referred to as isolation. Developers need to be very aware of this, as the platform allows to write tests that don't have this characteristic. There are best practices that need to be followed to ensure tests are independent (hint: @SeeAllData, Mock Callouts.)

Professional. Last but not least, good tests are professional. Tests make no exception to the rest of the code: whatever coding principle applies, it should also be used for unit tests (KISS, loose coupling, DRY, naming and styling conventions, etc.). Tests must be short, concise and readable.

Writing Good Tests on Force.com

So as we have seen the Apex testing framework is a very good foundation for writing good tests and takes care of some key characteristics of A-TRIP tests (Automation, Repeatability). In order to make sure our tests are good we need to cover the following:

In the next parts of this series we'll look at how each one of these characteristics of good tests can be implemented on the Force.com platform.

Edit: Check out the second post of this series: how to write thorough unit tests