Essentials of Battery Drain Testing — Part I: Model-based Approach

The introduction of Battery in Settings in both Android and iOS in the past few years has empowered billions of smartphone users to identify the battery-draining “culprit” apps on their phones. These excessive battery draining apps are now subject to users taking action and uninstalling them to extend their battery life. As such, battery drain testing has become an increasingly important step in the QA process of mobile app development and is being adopted by major industry players such as Facebook, Pandora, and others.

What is battery drain testing?

We define battery drain testing as a special type of scenario-based performance test that informs the tester about the phone’s battery drain attributed to a mobile app’s runtime. For example, a 1-minute music playback session of Spotify running in the foreground. Or, 3-minute browsing and reading session in Google News. There are many things that can influence the battery drain. Some of which are app version, OS version, handset model, user interactions and app behavior during the test scenario. There are several external conditions and internal device conditions that will also influence battery drain such as ambient temperature and caching.

In this blog, we discuss the limitations of the conventional battery drain testing approach and introduce our power model-based battery drain testing methodology. We discuss and experimentally show several advantages of the power model-based methodology over the conventional approach.

Conventional Battery Drain Testing

One way of performing battery drain testing is to physically connect a power monitor to the battery leads of a device and observe the power draw readings during a test scenario. Another approach is to observe the overall battery level drop of a device over an extended period of time. But these conventional approaches have several drawbacks discussed below. They are also summarized in Table 1.

Using a power monitor can be costly and is not scalable. Power monitors are expensive equipment, and thus equipping a device lab with power monitors for each phone quickly adds up in hardware cost.

Using the battery level drop is inexpensive as it does not need extra hardware. But this approach can be time-consuming as modern smartphones come equipped with high-capacity batteries (e.g., Pixel 3 battery is 2915 mAh). For example, it takes 29 minutes for battery level to drop from 90% to 89% when streaming music from Pandora 6.6 on a Nexus 6 with screen off.

Perhaps most importantly, these conventional approaches to performing battery drain testing cannot differentiate the battery drain attributed to the app under test from other apps and system services and do not give insights in terms of what in the code is causing the battery drain. Developers are left clueless on how to deal with the detected battery drain problem.

Table 1. Comparison of conventional and power model-based battery drain testing methodologies.

The basics of battery drain testing by means of power model-based energy profiling

Eagle Tester developed by Mobile Enerlytics employs an advanced power model-based energy profiling approach to overcome the limitations of the conventional ways of performing battery drain testing. The key technical components of our advanced energy profiling are as follows:

• Power modeling — Derive accurate power models for every power-hungry phone component. e.g., CPU, GPU, Network (LTE Radio, Wifi Radio), GPS, OLED Screen, Media Decoder, and DRM Hardware.

• Power estimation — Capture system log events that serve as power model triggers at test scenario runtime and use them along with the individual component power models to estimate the instantaneous power draw of every phone component.

Figure 1 illustrates the two phases of model-based energy profiling: Power modeling and Power estimation.

A power model for each phone component captures the correlation between the events that trigger the usage of the component and the measured component power draw. In the model derivation phase, microbenchmarks and/or real apps are run on the phone to exercise the hardware to induce the power trigger events. The triggers associated with each component are captured along with the instantaneous power measured using an external power meter and the built-in power sensor (available in most modern smartphones). Next, we isolate the power used by each phone component then derive the power model by extracting the correlation between the triggers and the components’ power draw.

Once a model is derived, it can be used to estimate the power draw of that phone component during the runtime of any app. A power meter or power sensor is no longer needed. This is done by collecting the triggers during the runtime of an app and then in post-processing applying the power model to the collected triggers to produce the estimated power draw.

In summary, our power modeling approach to energy profiling allows us to accurately calculate the energy drain of every phone component used by the app under test.

Battery draining testing using such a power model-based energy profiling methodology is:

• inexpensive — it does not require any external hardware such as power monitor.
• fast — the duration of battery drain testing scenarios can now be short. For example, we no longer need to endure the 29-minute wait while streaming music with Pandora waiting for 1% total battery drop. The testing scenario can be as short as just 1 minute since Eagle Tester has microsecond level granularity.
• fine-resolution — it can isolate the energy drain of the app under test from other apps and system services by attributing each trigger to their respective process invocation source.
• insightful — energy drain is further broken down by app threads which inform developers to zoom into the relevant code at the root of detected energy problems.

A closer look at the power model-based approach in action compared to the conventional approaches

To quantify the power model-based energy profiling approach we performed some battery drain testing on the popular video streaming app Netflix. We wrote about this in detail in a previous blog post. To summarize, the test scenario simply comprised of launching Netflix and then putting the app into the background by pressing the device home key.

Test resolution: Test results of the conventional approach are typically pass/fail based on a single battery level number, e.g., battery level drop or power monitor reading within the testing interval. More importantly, the reported battery level drop or power draw is that of the whole device. In contrast, Eagle Tester gives an accurate breakdown of the battery drain of the app under test within the test scenario duration along with every other background app and system service running on the test device. Figure 2 below shows a sunburst chart of the whole phone energy breakdown. We see that Netflix running in the background only accounts for 35.6% of the whole-phone battery drain during the test scenario.

Actionable insights: The conventional approach to battery drain testing suffers from the lack of insights into what caused a test failure. A common form of battery drain testing is around detecting battery drain deviations of new app releases compared to a baseline, usually the previous app release. The conventional approach may be able to detect such a deviation, although with some uncertainty due to the resolution limitation discussed above. However, the battery level drop of the whole phone does not give a developer any insights on what to do with the detected battery drain deviation. In contrast, Eagle Tester’s fine-resolution energy profiling capability accurately breaks down the battery drain by different app hardware components, processes and app threads, which exposes the source of the battery drain deviation. This allows the developer to zoom into the relevant part of the app source code to pinpoint and optimize the root cause of the detected battery drain.

Figure 3 shows the Eagle Tester battery drain testing results of Netflix versions 6.1.0 to 7.8.0 while sitting idle in the background. The output highlights that the versions 6.3.0 to 6.20.1 consumed significantly higher battery than Netflix version 6.1.0.

Further, Figure 4 shows a timeline of the total current drawn of Netflix and the current drawn by individual threads in a buggy version of the Netflix app (v6.4.0). We see that the Netflix app’s current drain is dominated by ProcessManager thread which draws a relatively constant current of 300mAh.

Read more in our previous blog about how we helped the Netflix team diagnose this pervasive battery draining bug that plagued many versions of the Netflix app.

Takeaway

In this blog, we discussed the major drawbacks of conventional battery drain testing and introduced the power model-based energy profiling methodology which overcomes the limitations of the conventional approaches.

Model-based battery drain testing faces a few new technical challenges. These include how to achieve high accuracy and repeatability of test results and how to support the diverse ecosystem of handset models in the smart-phone market. We will take a deep dive into each of these topics in our next posts.

Is app battery drain testing important to the growth and success of your business? Tell us your current practices. Leave your thoughts in the comments section below — we would love to hear from you! We also invite you to try out Eagle Tester for free. Don’t hesitate to contact us.