If you’re a Flutter developer, you may have written widget tests before. But have you unleashed their full power?
Judging by the Flutter developers I have worked with the last couple of years, I would venture it’s a “no”… Not because of lack of skills, I have worked with many excellent developers, but because best practices regarding testing take a little longer to establish when a framework is still “new”.
The common approach
Widget tests are often written as a unit test for a widget. The widget is launched, all the data passed to the widget is mocked, any state management used by the widget is mocked, and there you have it, a unit test for a widget.
For example, you may have a widget that has a couple of parameters: a string
title and a
onTap function. Your “unit” widget test would test the function is called when the title is tapped. That’s great if you develop a UI library, but let’s face it, most bugs do not reside at that level, most bugs reside in the interactions between widgets and/or (async) data.
Unleash the full power of widget tests
Have you ever maintained a UI test suite with 1000+ tests running on a device? Well, in my last job as a native Android developer, we had 1200 UI tests (Espresso). They were a life saver, absolutely, but they took 16 minutes on 28 shards on one virtual device on Firebase test lab, at a cost of $5. Not only it slows down CI and it is costly, but also, it meant we could never run the whole suite on one of our devices. If you work on an isolated feature, that’s fine, but what about a large refactoring job that potentially affects a lot of feature? Painful. Very painful.
With Flutter, we are extremely lucky that we can write UI tests that do not need a device to run. Widget tests can cover 90% of those tests you’d normally run on a device for native code. They run at a fraction of the time and without complicated and costly CI integration with a device test lab.
How? Just launch the main app widget. Simple as that.
What, launching the whole app each time? Yep.
Are you sure? Yep.
Remember, those tests run fast. It doesn’t matter that you need 5 taps to navigate to the screen you actually want to test. It will happen super fast.
What do you mock in those tests then? Only what lives outside Flutter. That’s the json sent by the server, and anything sent other to your Flutter code over a native channel.
Before diving further, let’s agree on a terminology for the rest of this article. I will call this kind of widget test a “full widget tree” test. There may be a better word for it, and feel free to suggest it. Until then, this is what I call it 🙂
Why bother with “full widget tree” tests? What aren’t “unit” widget tests enough?
“Full widget tree” tests enable you to:
- refactor your code safely. They test your app functionality as experienced by the user, they won’t break if you merge two widgets, or change their parameters, or refactor how you handle state management.
- test your navigation. A lot of bugs happen between widgets.
- document how your widget should work. No need to track down 5 layers to find out if error code
INVALID_GIFT_VOUCHERis handled, you can check your widget tests and see if you have a test for it.
- increase your test coverage super quickly, as they exercise a lot of code.
How to gain full power from “full widget tree” tests?
A few tips:
- test for what the users see, not implementation details. So, generally, aim to find widgets by a string or icon, not a key or a widget type. There are exceptions, but most of your tests should test for strings and icons.
- spend time setting up helper methods for writing widget tests. Eg no one wants to tap then pump a widget, it’s not concise, create a helper method for that.
- spend time setting up a way to easily mock a specific API call with a specific json file. Also set up a way to mock different status codes from server, including error codes your server may send, as well as IO exception.
- spend time setting up helper methods for common initial scenarios that rely on some data saved on device, such as a logged in user.
When to use a “unit” widget test?
I’m not against “unit” widget tests. Really, I’m not.
They are great for:
- your own library of UI widgets you use throughout your app, or that other people may use in their app.
- in app form data validation. Though you can do that in a full widget tree test too, there is no harm launching only the form widget for those tests.
- a widget you specifically want to test on a different screen size, but you don’t want to test rest of app on that screen size (changing screen size will affect need for scrolling etc).
Should you test everything?
Ultimately, maybe. But not yet.
Things to prioritise:
- error scenarios
- core paths in your app
- core information on the core screens of your app, including being able to scroll to it if information is further down
- strings should be in team’s main language (for test readability)
- pick a common screen size
- if your app uses timezones, please test those. A lot. They are a major source of bugs (I have work on 3 such apps, timezones are hard to reason with, no matter the programming language).
Then, you can add:
- loading states
- non core screens
- less common data scenarios
- strings in other languages supported by app
- analytics. Eg tap a button, does your app call your analytics logger?
- error reporting. Eg throw some gibbering json at a screen, does your app call your error reporting logger?
When to write “full widget tree” tests?
As soon as you can. From the start if you’re lucky enough to work on a new app, or as soon as you join a project if it’s existing codebase.
Whatever you do, do not refactor a legacy codebase without such tests. Please, just don’t. Most unit tests and “unit” widget tests will need to be refactored and be therefore completely useless if you undertake a non trivial refactoring.
But you know what, it’s not so much work. OK, you need to set up a few things at first, but having such tests mean you can bypass writing some unit tests.
If you use Blocs and you’re not developing a library, the only Blocs you need unit tests for are those used by many widgets, such as connectivity status or user login changes.
You can also bypass unit tests for repositories, and for your json parsing, as all those layers will be exercised in your “full widget tree” tests.
There is no need to test that your data source code throws
InvalidGiftVoucherExceptionwhen you have a “full widget tree” test that can simulate a server response with status code
403and error code
INVALID_GIFT_VOUCHER. After all, the user doesn’t care that your code throws
InvalidGiftVoucherException, what matters is that the app displays
Sorry, we do not recognise this gift voucher code.
This is why those tests increase your test code coverage quickly, you basically exercise all the layers.
I know, I know, it runs counter productive to most of the advice we hear as developers. We learn to layer our code and separate concerns, and we often assume that if we unit test everything, then everything will work fine. Except, it does not. Most bugs happen when two things that go together don’t work together properly.
What to use integration tests for?
For full integration. With the device AND with the server. Eg don’t mock the json, actually make the server calls to your staging server for your core screens.
When in doubt, let the principle of “user experience” guide you.
Most apps are UI code. The rest of the code is there to put the right thing on the UI so the user can see it and to process the actions a user takes through the UI. An app launches from an entry screen on a device. A Flutter app maintains a widget tree, and those can get quite complex. Therefore, for most apps, we need tests that mimic the real conditions as closely as possible, while also keeping in mind the other goals of clean code, such as easy to maintain code and fast feedback loops (hence widget tests rather than on device tests). This is why I think we all should write more “full widget tree” tests 🙂