How Much Testing Is Too Much?

You know how painful it is to work with badly tested code. Every time you fix a bug, you create five more. And when things do work, you never really know if it was designed that way, or just worked coincidentally.

On the other hand, you just wrote what seems like 200 tests to ship one tiny feature. You constantly have to redesign already-working code to hit 100% test coverage. You can’t shake the feeling that your best-tested code is somehow getting less readable. And worst of all, you’re starting to get burned out on your app.

There must be a middle ground. So how much testing is the right amount?

It’d be great if there was a nice round number you could use as a rule: twice as many lines of test code as app code, maybe, or 95% test coverage. But even saying “95% test coverage” is ambiguous.

Coverage can be an indicator of well-tested code, but it’s not a guarantee of well-tested code. I’ve had 100%-covered apps that had more bugs than apps with 85% coverage.

So, the right amount of testing can’t be about a number. Instead, it’s about something fuzzier, and harder to define. It’s about testing efficiently.

Efficient testing

Testing efficiently is all about getting the most benefit for the least amount of work. Sounds great, doesn’t it?

But there’s a lot that goes into testing more efficiently. So it helps to think about three things in particular: size, isolation, and focus.

Size

Integration tests are awesome. They mirror a path an actual person takes through your app. They test all of your code, working together, the same way it’s used in the real world.

But integration tests are slow. They can be long and messy. And if you want to thoroughly test one small part of your system, they add a lot of overhead.

Unit tests are smaller. They run faster. They’re easy to think about, since you only need to keep one tiny part of your system in your head while you write them.

But they can also be fake. Just because something works inside a unit test doesn’t mean it’ll also work in the real world. (Especially if you’re doing a lot of mocking).

So how do you balance those?

Since unit tests are fast and easy to write, it doesn’t cost much to have a lot of them. So they’re a great place to test things like edge cases and complicated logic.

Once you have a bunch of well-tested pieces of your system, you still have to fill in the gaps. You have to test how those parts interact, and the full journeys someone could take through your app. But because most of your edge cases and logic are tested by your unit tests, you only need a few of these more complicated, slower integration tests.

You’ll hear this idea called the “Test Pyramid.” It’s a few integration tests, sitting on top of a base of many unit tests. And if you want to learn more about it, take a look at the third chapter of my book, Practicing Rails.

Isolation

Still, if your system is complicated, it might take what feels like an infinite number of tests to cover every situation you might run into. This can be a sign that you need to rethink your app’s design. It means that parts of your system depend too closely on each other.

Say you have an object that could be in one of a few different states:

case user.type
when :admin
  message = admin_message
when :user
  message = user_message
when :author
  message = author_message
else
  message = anonymous_message
end

if user.preferred_notification_method = :email
  send_email(message)
elsif user.preferred_notification_method = :text
  send_text_message(message)
else
  queue_notification(message) 
end

If you wanted to test every possible path here, you’d have 12 different situations to test:

  1. User is an admin, preferred_notification_method is email
  2. User is an admin, preferred_notification_method is text
  3. User is an admin, preferred_notification_method is neither
  4. User is a user, preferred_notification_method is email
  5. User is a user, preferred_notification_method is text
  6. User is a user, preferred_notification_method is neither
  7. User is an author, preferred_notification_method is email
  8. User is an author, preferred_notification_method is text
  9. User is an author, preferred_notification_method is neither
  10. User is anonymous, preferred_notification_method is email
  11. User is anonymous, preferred_notification_method is text
  12. User is anonymous, preferred_notification_method is neither

There are so many cases to test because “sending a message based on a notification method” and “generating a message based on the type of a user” are tied together. You might be able to squeeze by with fewer, but it’s not obvious – and it’s just asking for bugs.

But what if you broke them apart?

message = get_message_based_on_user_type(user.type)

send_notification(message, user.preferred_notification_method)

Now you can test each part separately.

For the first part, you can test that the right message is returned for each type of user.

For the second part, you can test that a given message is sent correctly based on the value of preferred_notification_method.

And finally, you can test that the parent method will pass the message returned from do_stuff_based_on_user_type along to send_email_or_text. So now, you have 8 states to test:

  1. User is an admin
  2. User is a user
  3. User is an author
  4. User is anonymous
  5. preferred_notification_method is email
  6. preferred_notification_method is text
  7. preferred_notification_method is neither
  8. and one test for the parent method

Here, you save four tests by breaking code apart so you can test it separately. In the second example, it’s a lot more obvious that you can get by with fewer tests. And you can imagine how as you add more states, splitting your code up becomes an even better idea.

It takes time and practice before you’ll find the best balance between isolation and readability. But if you break your dependencies in the right place, you can get by with a lot fewer tests.

Focus

Your app should be well-tested. But that doesn’t mean every part of your app deserves the same amount of attention on its tests.

Even if you do aim for 100% test coverage, you still won’t test everything. You probably won’t test every line of text in your views, for instance, or that you’re polling for updates every five seconds instead of ten.

That’s where focus comes in. Writing fewer, more useful tests. And making a conscious decision where you can best spend the time you have.

Focus is another thing that’s hard to get right. These are a few questions I ask myself that help me concentrate on the most important tests:

  • How interconnected is this with the rest of my app? If it breaks, how many other pieces will go down with it?

  • How likely is it that this will change naturally? If my tests fail, will it be because of a bug, or because someone updated some text in the UI?

  • What’s the impact of this breaking? Am I going to charge someone’s credit card twice, or is it just going to end up with some missing text?

  • How often is this part used? Is it critical to the app’s behavior, or is it an about page buried somewhere in the footer?

You shouldn’t only test the important parts. But you’ll have an app that feels higher quality if you spend your testing time well.

If you try to test every single possible path someone could take through your app, you’ll never ship. TDD helps, but it won’t solve all of your testing problems.

Of course, that doesn’t mean you shouldn’t test at all.

You can use the test pyramid to keep your tests small. You can isolate and break dependencies to turn m * n test cases into m + n. And you can prioritize, so you can spend more time testing the most important parts of your app.

So, how much do you test? Do you consider any of these ideas as you build out your app? And how do you know which parts of your app to focus on? Leave a comment and tell me all about it!

Did you like this article? You should read these:

Comments