Are you over-testing your mobile app?

Over the last few weeks, I’ve been thinking a lot about mobile app quality, and in particular, how much testing is too much. Whilst quality per se is a good thing, setting the quality bar too high could have a negative impact on your ability to drive business value from your mobile app.

Think of your home security setup. If you went out and left your doors open, there’s a good chance that someone would break in and steal your stuff. So you probably have locks on your doors and windows, and an alarm that you set when you go out. If you had some really expensive stuff in your house, say £1m in cash, a couple of Picassos and some priceless jewellery, you’d need to up the ante. You could add in internal door locks, a more robust alarm system, maybe even round-the-clock security guards. You have a lot to lose, so you need to do more to protect it. But now lets say you don’t have all that expensive stuff, but you still have all those extra security measures in place. You’re spending more on home security than you need to, both in terms of money, and the interference with how you go about your business. But it’s hard to recognise that you’re over-secure. You’re getting the desired outcomes, i.e. no break-ins, and it might feel risky to relax your security in case something happens.

It’s the same with over-testing your mobile app. It’s important to be aware of because it’s hard to spot the signs of over-testing, making it a silent threat.

In this article I’ll cover:

• What do I mean by quality.
• Why quality is important for mobile apps.
• The cost of setting your quality bar too high.
• Balance the trade-offs to set your quality bar.
• Why the cost of quality in mobile is unavoidably higher than other channels.
• How to tip the scales in your favour, and reduce your quality costs.
• How to set your quality bar at the right level for your unique circumstances.

Throughout the article I’m going to use some simple graphs as illustration. Please note, these are simple schematics, and are based on experience and gut feel, rather than hard data.

What do I mean by quality?

Here’s a simple graph, showing quality increasing from 0 to 100. What does quality actually mean here? I’m going to be intentionally vague, because quality can mean different things to different people in different situations. But let’s say that a quality of 100 means that you have complete confidence that everything in your app works exactly as intended, all of the time. At 0, you have no idea what works and what doesn’t, and you are doing nothing to prevent bugs and incidents.

You probably recognise intuitively that some quality is better than none. I’d agree with you there. But you might also think that you’ll always gain by increasing your quality, all the way up to 100. You’d be wrong with that assumption.

To maximise the value you can drive from your mobile app, your quality bar needs to be set at some point higher than 0, but lower than 100. Let’s start by looking at the impact of setting your bar too low.

Why is quality important for mobile apps?

When your quality bar is set too low, you will be incurring costs relating to bugs and incidents. There are lots of sources of these costs, but here are some examples:

• Loss of revenue if people can’t use your service or make payments.

• Damage to your brand reputation caused by outages or unreliability.

• The cost of dealing with a high volume of customer support issues.

• Opportunity cost of not releasing new features, because you’re dealing with bugs and incidents.

• The cost of development work of fixing things that you weren’t expecting.

• Managing extra releases, to fix urgent incidents.

To go back to my home security analogy, bugs and incidents are the equivalent of someone breaking in and stealing your stuff.

The lower your quality bar, the more likely you are to face a higher volume of more serious bugs and incidents. And the more serious a bug or incident is, the more it is likely to cost your business.

Increasing your quality bar from 0 will give you big gains at first. You’ll avoid the most serious issues, and you’ll escape from the chaos of constant firefighting. As you continue to raise the bar, your costs will continue to decrease, but by less and less. As your bar gets towards 100, you’ll be finding more obscure bugs, that affect fewer people, and that might never even actually affect anyone. Your quality and confidence continue to increase, but the incremental gains to your business value become vanishingly small. But is that a problem? Yes. Let’s take a look at the cost of setting your quality bar too high.

The cost of setting your quality bar too high

We’ve just seen the effects on cost of increasing your quality bar. But we haven’t yet taken into account the cost of achieving that quality bar.

To improve your quality, you need to test. Testing costs money. Again, there are many sources of costs here, some of them are:

• Time and effort spent on manual testing.

• Writing and maintaining test cases.

• Setting up and maintaining test data.

• Writing and maintaining automated tests.

• Investigating and fixing flakey UI tests.

• Paying for 3rd party tools and test services.

• Purchasing test devices, or paying for access to device farms.

To go back to the home security analogy again, testing is the equivalent of your locks and security alarm.

Increasing your quality from 0 will give you a big impact for a small cost. For example, you could pay an unskilled work-experience student to do some untargeted testing, and that would likely discover a whole raft of bugs that you could fix. But as you increase your quality towards 100, your costs will increase more and more relative to the gains you make. You’ll exhaust all those easy bugs, so making more improvements will require increasing levels of expertise, time, and cost. There’s a trade-off between the cost of bugs and incidents, and the cost of testing.

If your quality bar is too high, you are over-testing. This is a particular problem because it isn’t easily visible to the business. If you under-test, your quality is too low, and that’s pretty visible. Anyone using the app, or monitoring customer support issues, will see the impact. Over-testing is more insidious. It often feels like the right thing to do, but you can’t easily observe the impact of it.

Balance the trade-offs to set your quality bar

We’ve seen that as your quality goes down, the cost of bugs and incidents goes up. And conversely, as your quality goes up, the cost of testing goes up. It’s a see-saw effect. The best place for you to set your quality bar is where the total cost is minimised.

Exactly where this point is depends on your particular context. For example an app that sells high value goods or services will have a higher cost of bugs, so its quality bar would be higher.

On the other hand, an app with a very small user base might have a lower cost of bugs, so its quality bar would be lower.

When you see the trade-off you’re making with quality, it’s clear that you can over-test your app. If you are spending more on testing that you are gaining from the extra quality that brings, you’re over-testing. You’d be able to drive more business value with your app if you dialled back your testing effort.

Why the cost of quality in mobile is unavoidably higher than other channels

At this point I’d like to take a slight detour. If you’re used to building for platforms other than mobile, you might be frustrated that quality seems harder on mobile. Maybe you struggle to get the same level of quality. Or if you do, it seems more expensive. That’s not because you or your team are doing something wrong, it’s because there are some fundamental differences that make quality on mobile more expensive.

The cost of bugs and incidents on mobile is higher because:

• You can’t roll back releases, you can only fix forwards.

• The review and release process takes time (days, or sometimes even weeks or months).

• Updates aren’t guaranteed, so some users might never receive a bug fix.

The cost of testing on mobile is higher because:

• You are usually supporting two platforms (iOS and Android), and often two code bases. These need testing separately.

• You have to support different form factors (phone and tablet), different screen shapes and sizes, and different flavours of operating system (particularly on Android).

• UI tests are slow and flakey, even more so on mobile than other platforms.

While the overall cost of quality on mobile is higher, it isn’t fixed. You can tip the scales in your favour.

How to tip the scales in your favour, and reduce quality costs

Quality on mobile is more expensive. And setting your quality bar is all about balancing the trade-offs between the cost of bugs and incidents, and the cost of testing. However these costs aren’t set in stone. There are things you can do to decrease the cost of bugs and incidents. There are also things you can do to decrease the cost of testing. This will both decrease your overall costs, and change the level at which you should set your quality bar.

Some of the ways you can decrease the cost of bugs and incidents are:

• Set up your architecture so you can fix bugs quickly.

• Optimise development workflows and processes so you can make changes and release them quickly.

• Build or improve CI/CD and automation capabilities so you can release quickly and cheaply.

• Decouple platforms so you can release iOS and Android independently.

• Use a platform like Code Push to allow you to make small changes without releasing a new app version through the stores.

• Use feature flags and kill switches so you can recover quickly if things don’t work as expected.

• Use development feature switches so your code is always in a state that can be released.

• Release more frequently so changes are in smaller batches and there is less to go wrong.

These all make it cheaper to recover from any bugs or incidents that do occur. This decreases your overall cost, and makes your optimal quality bar lower.

One the other side, some of the ways you can decrease the cost of testing are:

• Set up your architecture to make it easy to add automated tests.

• Focus testing on critical paths, i.e. high value / high traffic areas of your app.

• Reduce testing on low value / low traffic areas of your app.

• Apply the principles of the ‘testing pyramid’ to pick appropriate forms of testing, e.g. lots of cheap unit tests, sparing use of more expensive testing such as UI tests, and limited use of the most expensive manual and regression testing.

• Explore different kinds of tests that give similar results for less cost, e.g. use snapshot testing so you can reduce manual testing, and use contract testing so you can reduce end to end testing.

• Use test specialists as coaches for the whole team, rather than having them focus on executing manual tests.

• Use AI tools to generate automated tests more cheaply.

• Reduce test bottlenecks and dependencies by adopting a cross-functional approach, where everyone in the team can test.

• Document regression tests and processes so they are easier to execute.

• Invest in making automated tests easier to write and maintain.

These all make it cheaper to test. This decreases your overall cost, and makes your optimal quality bar higher, because it costs you less to test more.

How to set your quality bar at the right level for your unique circumstances

Where you set your quality bar is important, because it directly impacts the business value you can drive from your mobile apps. You have to manage the trade-off between the costs of bugs and incidents, and the cost of testing needed to prevent those bugs and incidents.

In practice, it’s hard to do. Imagine you buy a coffee. You know how much you paid, and you get immediate feedback on how good the coffee was. Did it taste nice? Did you get enough coffee? If you don’t like the trade-off, you can adjust in the future, e.g., buy coffee from a different place next time. But when you’re balancing the cost of bugs and incidents with the cost of testing, it’s much harder. The costs of both are subjective and hard to measure, and you don’t get immediate feedback. You apply a level of testing, then at some point in the future you pay the price for bugs and incidents. It’s not an exact science, and you need to develop a feel for it.

The cost of under-testing is relatively easy to gauge. You have an incident, you feel the impact, and that triggers you to increase your quality if needed. But the cost of over-testing is much harder to recognise. Everyone is working hard, the quality is high and bugs and incidents are rare. That’s good, right? Well, as we’ve seen, not necessarily.

Some signs that you might be over-testing are:

• Development feels slow.

• There is a long lag between making a change and being able to release to end users.

• The bulk of your development time is spent writing and / or maintaining tests.

• You run lengthy manual regression tests before before releases.

• You are more focused on code coverage stats than the value impact of the tests you create.

• You aim for the highest coverage of UI tests that you can.

• You don’t have a ‘testing pyramid’ strategy for applying tests.

• You don’t differentiate between high value / high impact areas of the app and low value / low impact areas.

• You rely heavily on manual testing.

• Bugs and incidents are very rare.

Experience is key here, because all of the above individually can be symptoms of other problems, and it helps to have insight into how testing is applied across a range of apps. However if you recognise a handful or more of these within your app, over-testing could be your prime suspect.

The exact approach you should take to reducing over-testing depends on your very specific context, but if you’re looking for ways to get started, I’d recommend to start by picking some of the tactics listed above to reduce your costs of testing, such as:

• Explore different kinds of tests that give similar results for less cost, e.g. use snapshot testing so you can reduce manual testing, and use contract testing so you can reduce end to end testing.

• Set up your architecture to make it easy to add automated tests.

This will take some effort up front, but once in place will decrease your cost of testing, and increase your optimal quality bar. If you were over-testing, at the very least you’ll now be over-testing less.

Next, pick some of the tactics to reduce your costs of bugs and incidents, such as:

• Optimise development workflows and processes so you can make changes and release them quickly.

• Build or improve CI/CD and automation capabilities so you can release quickly and cheaply.

Setting yourself up to recover quickly from bugs and incidents reduces the risk of impact, and lets you bring your optimal quality bar down with more confidence. Now you’re in a good position to start dialling back your level of testing in a controlled way, for example by:

• Focus testing on critical paths, i.e. high value / high traffic areas of your app and reduce testing on low value / low traffic areas of your app.

• Apply the principles of the testing pyramid to pick appropriate forms of testing, e.g. lots of cheap unit tests, sparing use of more expensive testing such as UI tests, and limited use of the most expensive manual and regression testing.

Summary

In this article, I’ve covered what ‘quality’ means for an app, and the costs involved when you set the quality bar either too low, or too high, and how you balance the trade-offs of those costs. I’ve talked about how the cost of quality for mobile is unavoidably higher than for other systems, but what you can do to reduce those costs. Finally I looked at how you can recognise the signs of over-testing, and steps to take if you are over-testing your app.

The one thing I’d like you to take away is that it is possible to over-test your app, and that whilst the impact on your business value will be significant, it’s really hard to spot. Remember my home security analogy - don’t spend money on round-the-clock security guards that get in the way of your life when all you need is locks on your doors and windows, and a simple alarm.