The analytics checklist: what to measure and look for

When you launch an app or update, you often hope to get new users and/or more revenue. Google Play Developer Console keeps track of those metrics for you, but whether you see numbers you like or dislike, you don’t know much about what drives them.

Analytics, which I’m using in the general sense of collecting data on usage and errors, can help you understand both the pain points and the users of your app, allowing you to focus on removing roadblocks and serving your users’ needs better. This will, in turn, increase your users’ satisfaction, leading to higher ratings and new users, as well as higher revenue per user.

Google Analytics, specifically, comes with an Android SDK and the web interface is already known to many business users, making it a great tool to implement your analytics strategy. There is a whole section on the android developer website taking you around its features. It’s a great product that gives you rich “out of the box” data on your users.

Putting aside the implementation details for a minute, let’s focus on what to measure and look for…

Identify functional problems

  • Crashes – If you don’t already, regularly check the reported Exceptions and ANRs in Google Play Developer Console.

  • Caught Exceptions – Some exceptions are expected but you should still know about them, because they may indicate a latent issue with your app.

  • Communication with servers – Background syncing is vital to delivering a fast, smooth, efficient app. How long do your server calls take? What percentage of your calls end in an IO Exception or a response time of more than 10 seconds? What percentage of your users are on wifi, 4G, 3G and 2G? How often do specific exceptions related to your local caching and server sync happen, for example SQLExceptions?

  • Location, location, location – For exceptions that you catch, do they happen specifically for a given time zone or language? Android is the most popular mobile platform in the world today so you can’t assume that your users live in your country and speak your language.

  • Search – Do users search for what you expect them to? Do they make spelling mistakes in their queries that appropriate suggestions could minimise?

Identify UI problems

As a minimum, you should track which elements your users interact with, how much time is spent on each Activity and the flow between the Activities, to identify the following problems.

  • Activity UI hierarchy – Are the icons in your ActionBar really necessary? Should you move a button higher or lower up the screen (particularly relevant in long scrollable screens)?

  • UX flow – If many users go through screen A -> B -> C -> D, would it make sense to give them a way to do screen A -> D?

In addition, you may consider specific tracking to help you discover the following potential problems.

  • Slow Activity initialisation – Are you loading some data in onCreate that should be moved off the main thread?

  • Scrolling – If you have long screens, do you know how much your users scroll them? Are you loading data fast enough for your users?

Understand your users

The data collected to understand UI problems will also help you understand your users.

  • Undiscovered features – You can assess this by looking at the combination of click data with user reviews.

  • Unused features –  You can assess this by looking at the combination of click data with time spent on a specific activity.

The next step up from basic understanding is to segment users by groups, for instance heavy users, median users, and infrequent users. For each segment, the basic questions to ask yourself are:

  • How frequently do your users use the app?

  • Where do they spend time?

  • What are the most common screen flows?

  • Are some features undiscovered?

  • Are some features unused?

  • How do your users interact with your push notifications?

The core principle in creating a rich, meaningful experience for your users is to deliver the right information at the right time. Once you understand your users, you can consider targeted strategies around advertising, content, and personalization. For example, you may

  • Focus on a single segment and advertise your app as such.

  • Provide contextual help on your home screen when certain actions are done, a tactic often employed by games in the early levels of play.

  • Personalise your home screen, for example, by adapting the placement of different sections based on the type of user.

Tips for creating and maintaining robust analytics in your app

  • Write down your assumptions – Use your detailed roadmap, epics or simple long term goals to surface assumptions you have made about your users.

  • Assign a risk to each assumption – How much do your long term goals rely on this assumption being true?

  • Consider how you could determine the validity of each assumption – Starting with your riskiest assumption, consider which data you could gather in your app to help you confirm or invalidate each assumption.

  • Place yourself in your shoes six months from now – Imagine that, by then, your app is completely bug-free and has enough users to justify you investing in another round of heavy development. What would be your various options and what would you need to know to decide which one is the best?

  • Implement analytics as part of the feature – It’s too easy to rush a feature out of the door and forget analytics. However, how will you be able to measure the impact of your new feature?

  • Iterate your analytics – As you gain knowledge about your current users, you will no doubt ask yourself more questions about them. Just like you iterate your app, also iterate your analytics.

Privacy of user data

Yes, collecting data is useful to serve your users better but do not forget about privacy for your users, so anonymise the data you collect. In particular, make sure you comply with section 4.3 of the Developer Distribution Agreement.

Some data such as user location is particularly sensitive, so you may want to split your data collection into two groups. The first group wouldn’t require user consent, it would use anonymized IPs and would be general things such as general user segmentation, click events and screens. The second group would require user consent, it would not use anonymized IPs and may include things such as user location.

Tips for implementation

  • Define an interface – For maximum flexibility, create an interface for your analytics methods, a general implementation class (that you call from your code), and an implementation class for the provider you have selected to use. This way, should you decide to change between analytics providers, you will simply add an implementation class for the new provider, and change the code in your general implementation class to use this provider instead.

  • Define enums for screens and events – Each enum would contain the full data sent to the analytics service for the screen or event respectively, making the analytics code in your application very concise and easier to maintain.

  • Development setup – Analytics need to be tested too, so use a different analytics id for your dev builds. Once you have implemented the analytics in your app, make sure your business team has access to the dev analytics data to ensure they can understand it.

  • A/B testing – A way to do it is to apply the modulo function to the current time in milliseconds of the system to determine the user testing group id, then save it in a shared preferences field. This should be done in the onCreate method of your Application class, only once. For example, if you want to send 90% of your users to group with id 1 and 10% of your users to group with id 2, you would do something like this: int userTestingGroupId = System.currentTimeMillis()%10 != 0? 1:2;

Don’t forget to…

  • Do in person usability testing – This can be done with a prototype of your app. In person usability testing is your first port of call to identify UI problems.

  • Do manual and automated testing – App testing is your first port of call to identify functional problems.

  • Use the alpha/beta publishing channels on Google Play Developer Console – It’s best to catch both functional and UI problems before all your users experience them. You may consider segmenting your alpha/beta channels to match a specific user segment, eg only heavy users can be alpha testers and only moderate users can be beta testers.

2 thoughts on “The analytics checklist: what to measure and look for”

Leave a Reply

Your email address will not be published. Required fields are marked *