Implementation of Android Test Orchestrator

Introduction

We have more than 200 automation test cases (clubbing similar 4-5 scenarios together) written for our application which takes around 1 to 1.5 hours to run completely. Recently we were facing an issue of OOM (Out Of Memory) error while running test cases. We tried fixing it by multiple solutions like increasing the memory of RAM, dividing the test cases by module and run them, etc. But then we got to know about Android Test Orchestrator. Android test orchestrator provides bettor support to solve the issue of OOM to the instrumentation tests. In this article, I am going to explain why it is, how to implement it and what issue it solves.

As you see in the android official page (https://developer.android.com/training/testing/junit-runner.html#using-android-test-orchestrator), it provides multiple ways to add android test orchestrator in your project. Here, I am going to provide a way to enable it from the Gradle.

Why it is?

If you are working with and android espresso, you might facing the following problems

1. OOM (Out Of Memory) error.
2. App crashes.
3. Test overlaps.

OOM (Out Of Memory) error

Recently we were facing an issue for OOM (Out Of Memory) error while running all test suit randomly. This was leading to the app crash and complete test suit would stops. To fix this, we planned to implement Android Test Orchestrator. Android Test orchestrator provides a flag to clear all the package data after the completion of every test case.

testInstrumentationRunnerArguments clearPackageData: 'true'

Basically, when there is a memory leak in your application, heap size of the device continuously increases when your test cases are running and after some point it may cause an OOM error.  When you add test orchestrator with the above flag, it removes all the shared state from your device’s CPU and memory after each test.

App crashes

If there is an app crash in the application all the test suit gets stop without any test assertion failure report. Detecting app crash looks good but sometimes it will be a headache, because we have to again re-run complete the test suit. Because if you are running 500 test case and app gets crash on 236 all the test suit will stop and we have to again start from the first test.

As Android Test Orchestrator creates a separate instance of the instrumentation. So, if there is an app crash then this would crash only for current instrumentation and other tests would not be affected.

Test overlaps

Test overlaps occurs when there is an asynchronous call. Idling Resources provides a good support to Handle the asynchronous calls but it does not provide a support for every test case. Handling asynchronous calls is hard task to stable the UI tests.

Suppose we have two tests A and B. I starts an asynchronous operation in test A and test A completed with success and my asynchronous action is still in progress but my second test B start. If this asynchronous call gets completed while running test B, it will surely affect the test B.

Android Test Orchestrator starts a new instance of the app with each test and closes the one from the previous test. Therefore the overlap is solved automatically. The cleanup after each test is simplified.

How to Implement Android Test Orchestrator?

There are three ways to enable android test orchestrator

1. Enable from Gradle
2. Enable from Android Studio
3. Enable from Command Line

We are going to enable the android test orchestrator from the gradle and to complete the implementation you have to follow the following steps:

1. Add the following lines to your gradle file

   android {  
    defaultConfig {
          ...
          testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
          testInstrumentationRunnerArguments clearPackageData: 'true'
   }
   testOptions {
    execution 'ANDROID_TEST_ORCHESTRATOR'
    }
   } 
   
   dependencies {
   androidTestImplementation 'com.android.support.test:runner:1.0.2'
   androidTestUtil 'com.android.support.test:orchestrator:1.0.2' 
   }
   
   

   Note: If you referred android official documentation, you will notice there are androidx dependencies. If you are adding androidx dependencies please make sure that all application dependencies should be of androidx type.

2.   Run Android Test Orchestrator by executing the following command from the command line:

   ./gradlew connectedCheck

   Note: If you are using latest gradle version then please update your project level gradle like:

   dependencies {    
    classpath 'com.android.tools.build:gradle:3.4.1' 
   }

   and run the above command. If your gradle version is old you may found the error ‘No Test Found’. Please update your gradle.

   While updating your gradle you may also found you should have to update the gradle version from the file ‘gradle-wrapper.properties’. Don’t stop after updating the gradle files please update your android studio also else you might found errors related to the kotlin dependencies.

   Now you are ready to execute your complete test suit.

References:

1. https://developer.android.com/training/testing/junit-runner.html#using-android-test-orchestrator
2. https://medium.com/stepstone-tech/android-test-orchestrator-unmasked-83b8879928fa