Unit Tests

Unit test coverage report with Travis CI for Robolectric on Android

2017-03-19 Android, Continuous Integration, DSP2017, Gradle, Unit Tests No comments

Introduction

Some time ago, I’ve written an article about Test coverage report for Android application. It got some interest (many comments below article and many visits according to Google Analytics), so I decided to refresh this topic. Previously, I’ve written instrumentation unit tests, which needed to be executed on a real device or an emulator. It’s a good approach when you want to test functionalities strongly connected with the device. E.g. when you want to test operations on a real SQLite database or something like that. Nevertheless, this approach has huge disadvantages. It’s hard to run tests on the Continous Integration server because we need to have the emulator or device up & connected all the time and also tests need to interact properly with the device to get passed what is not so easy. In most cases, mocking part of the application’s behavior is enough. In that case, we can easily run tests on a CI server and have deterministic test results. In order to do that, we can use Robolectric.

Gradle configuration

First, we have to add appropriate dependency to jacoco-android plugin in our top-level build.gradle file:

buildscript {
  repositories {
    jcenter()
  }
  dependencies {
    classpath 'com.android.tools.build:gradle:2.3.0'
    classpath 'com.dicedmelon.gradle:jacoco-android:0.1.1'
  }
}

Next, we need to add appropriate test dependencies in another build.gradle file for our app or library.

dependencies {
  testCompile 'junit:junit:4.12'
  testCompile 'com.google.truth:truth:0.32'
  testCompile 'org.robolectric:robolectric:3.1.2'
  testCompile 'org.mockito:mockito-core:2.7.17'
}

I’ve added also dependencies to JUnit, Truth and Mockito library, which are used in my tests.

We also need to add appropriate plugins:

apply plugin: 'jacoco'
apply plugin: 'jacoco-android'

To avoid ignoring our tests by the coverage report, we need to configure the following settings:

android {
  testOptions {
    unitTests.all {
      jacoco {
        includeNoLocationClasses = true
      }
    }
  }
}

Next, we need to configure report output:

jacocoAndroidUnitTestReport {
  csv.enabled false
  html.enabled true
  xml.enabled true
}

Travis CI configuration

We are done with Gradle configuration. I’m assuming we have Travis CI build configured. If you don’t know, how to do this, visit travis-ci.org and enable builds for your project. It’ pretty easy. Now, we should visit codecov.io website, register there (e.g. with GitHub account) and add our project. After that, we need to add the following items to our .travis.yml file:

after_success:
  - bash <(curl -s https://codecov.io/bash)

script:
  - ./gradlew clean build test jacocoTestReport check

Here we are performing clean, build an application, running unit tests, generating test coverage report with Jacoco and performing check (Lint, FindBugs, PMD & CheckStyle).

Writing unit tests with Robolectric

Next we can place our tests in src/test/ directory.
Sample unit test can look as follows:

In my case, I also needed to create src/test/resources/robolectric.properties file with the following content:

sdk=23

because Robolectric didn’t work with the Android SDK newer than 23. Moreover, I also needed to use Robolectric v. 3.1.2, because I had problems with running tests and generating coverage report with the latest version of the Robolectric.

Summary

When we have everything configured, we can push our tests to the GitHub repository, Travis CI build will be triggered and we can beautiful test coverage report, which can help to improve our tests.

I’ve applied approach described in this article in ReactiveNetwork open-source library. If want to see the complete solution, take a look at the source code of this project.

Test coverage report for Android application

2015-06-03 Android, Continuous Integration, Unit Tests 59 comments

For a long time I wasn’t able to find the right way to generate test coverage reports for Android instrumentation unit tests created with AndroidJUnit4 runner. In the past I’ve found many solutions, but none of them was working or they were compatible only with Robolectric, which I currently don’t use. Other developers also stopped using it for the new projects. E.g. Sqlbrite from Square is using AndroidJUnit4 runner as my projects do. Recently, I’ve found an interesting discussion on Reddit, where we can find information about enabling test coverage very easily without additional plugins, scripts and many lines of configuration. Moreover, Android SDK currently has built-in support for Emma Test Coverage, what we can read in official documentation.

Basically, the only thing we need to do, is to apply jacoco-android plugin in build.gradle file:

apply plugin: 'jacoco-android'

and then set testCoverageEnabled parameter to true as follows:

android {
   buildTypes {
      debug {
         testCoverageEnabled = true
      }
   }
}

In order to create test coverage report, we need Android device or emulator connected to the computer, because connectedCheck task needs to be executed before creating the report.

Next, we can execute the following Gradle task from CLI:

./gradlew createDebugCoverageReport

Task will analyze code of our project in /src/main/java/ directory and unit tests placed in /src/androidTest/java/ directory.
After executing this task, we can find test coverage report in the following directory of the module:

/build/outputs/reports/coverage/debug/

When we open index.html file, we can see visual report from test coverage, which can be viewed in a web browser.
Moreover, in the same directory, we can find report.xml file, which can used for test coverage analysis on continuous integration server.
Besides mentioned files, Gradle will generate coverage.ec file in the following directory:

/build/outputs/code-coverage/connected/

In some cases, we may need this file. E.g. if chosen Jenkins plugin or another tool will need it in order to generate test coverage report properly.

Below, you can see exemplary test coverage report for Prefser, which is an open-source Android library.

prefser_test_coverage_report_03.06.2015

This report was generated with JaCoCo code coverage library.

After analyzing test coverage report, I’ve added a few new unit tests, slightly modified the project and managed to obtain 100% test coverage.

prefser_test_coverage_04.06.2015

In order to publish our report on Jenkins CI, we can use test coverage plugins, but I am not sure about their stability. Another solution is HTML Publisher plugin. We can add post-build action and publish our test coverage report generated by default as HTML website inside Jenkins job. In my opinion, it’s quite convenient solution, because it is easy to set up and allows us to navigate to the code, browse uncovered lines, methods and branches.

Finally, with this clean and fast solution we can easily monitor test coverage of our Android projects, find bottlenecks and increase overall quality of the applications and libraries.

Static Code Analysis in Android project

2015-03-26 Android, Continuous Integration, Static Code Analysis, Unit Tests No comments

Overview

I’ve recently read great article telling how to improve quality and syntax of your Android code by Vincent Brison. It presents how to use tools like Checkstyle, Findbugs, PMD, and Android Lint for Static Code Analysis in Android project. Static Code Analysis allows us to keep consistency of the source code style and helps us to find errors, warnings, possible bugs, bottlenecks and ways to improve the projects. I highly recommend to use tools like those mentioned.

Example

Visit repository at: https://github.com/vincentbrison/vb-android-app-quality.
The most important files are:

We can run analysis by executing the following command:

./gradlew check

Besides that, we can run each command like checkstyle, findbugs, pmd or lint separately if we want to.
In addition, when we build the project with command:

./gradlew build

check command will be executed as well, because of the following line in quality.gradle file:

check.dependsOn 'checkstyle', 'findbugs', 'pmd', 'lint'

After that, we can find output with our reports in directory: app/build/reports/. Of course, we can customize output directory and names of the report files. In default configuration, check command with generate reports in HTML format. Only output from the checkstyle operation will be generated in XML format. We can customize findbugs, pmd and lint, to generate XML reports instead of HTML reports in quality.gradle file.

Sensitivity of the build

We can customize rules of the Static Code Analysis tools in restrictive and non-restrictive way.
In findbugs and pmd task, we can set ignoreFailures attribute as follows:

ignoreFailures = false

When it’s set to false, tasks will only generate reports, but when we set it to true, build will fail, when at least one error will be found.
In lint task, we can customize attribute abortOnError as follows:

abortOnError true

It will literally abort the build, when any lint error occur. Warnings will be ignored. When we want to ignore errors, we can set this attribute to false.

Static Code Analysis on CI server

When we want to perform Static Code Analysis on CI server, we can do it easily from the command line interface. If we want to automatically collect and analyze reports generated by mentioned tools, we should generate them in XML format instead of HTML, so they can be parsed by appropriate plugins. Mentioned tools can be easily integrated with Jenkins CI server.
We need the following plugins:

After installing those plugins, we can define post-build actions in our Jenkins Job, provide paths to the appropriate XML reports and then we can view results of the reports and trend graphs directly from web panel of our Jenkins server.

Exemplary trend graphs looks as follows:

lint_checkstyle_findbugs_pmd_trends

If we have unit tests in our project, we can also generate test reports and monitor test trend.
In the chart below, failed build don’t have generated test reports that’s why chart may look strange. In addition, number of tests was decreased, because they were migrated from JUnit3 to JUnit4. In JUnit3 for some reason each test case is treated as an additional unit test. JUnit4 report shows real number of tests.

test_trend

In every build, we can browse generated reports and their details.

android_build_report

I really recommend to use Static Code Analysis tools and reporting plugins on CI in the projects, because they will increase quality of the code really fast and will allow to eliminate many errors before shipping the application.

Mockito on Android

2015-03-15 Android, Unit Tests No comments

Overview

When we write unit tests for an Android application, sometimes there’s a necessity to mock some classes or interfaces. Instead of writing implementations dedicated for unit tests by hand, we can use Mockito library. We can read article about Mockito on Android on Square’s blog. Mockito can make our tests cleaner and better. It can be used both in classic Java projects and Android projects. It’s one of the most popular mocking framework in Java and if you’re a Java developer who writes unit tests and tries to apply TDD approach, you should be familiar with it.

Sample configuration

When we want to use Mockito on Android, we have to remember to add dependency to dexmaker 1.0 and dexmaker-mockito 1.0.

Sample test configuration in build.gradle file can look as follows (I’ve added more comments to clarify optional doubts):

dependencies {
    // our project dependencies go here...

    androidTestCompile 'com.android.support.test:testing-support-lib:0.1' // Android testing support library
    androidTestCompile('com.google.truth:truth:0.25') { // Google's library for assertions (not required by Mockito)
        exclude group: 'junit' // Android has JUnit built in
    }
    androidTestCompile 'com.google.dexmaker:dexmaker:1.0' // required by Mockito
    androidTestCompile 'com.google.dexmaker:dexmaker-mockito:1.0' // required by Mockito
    androidTestCompile 'org.mockito:mockito-core:1.9.5'
}

We should also add information about tesInstrumentationRunner to build.gradle file when we are using it.
When we have problems with running our tests, we should configure packagingOptions properly.

android {
    defaultConfig {
        ...
        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
    }
    packagingOptions {
        exclude 'LICENSE.txt'
        exclude 'META-INF/LICENSE.txt'
    }
}

Remember

Quotation from Mockito website:

  • Do not mock types you don’t own
  • Don’t mock value objects
  • Don’t mock everything
  • Show love with your tests!

References

Unit Testing Support for Android Studio 1.1

2015-02-15 Android, Unit Tests No comments

As official Android website says:

Version 1.1 of Android Studio and the Android gradle plugin brings support for unit testing your code. This feature is still considered experimental, we encourage you to try it and file any bugs you find in our bug tracker.

It’s great news, because now we can write Unit Tests with JUnit 4.12 without any “hacks” or tricky configuration, which had to be done earlier.
We can execute two kinds of tests:

  • Unit Tests – pure Java tests without Android dependencies written in JUnit, which can be run in Android Studio or CLI via Gradle without emulator or attached device. These tests have to be located in: /src/test/java/ directory.
  • Android Instrumentation Tests – tests for code, which have dependencies to Android Context or Android API and have to be executed on Android emulator or physical device like phone or tablet. These tests have to be located in: /src/androidTest/java/ directory.

Read more at: http://tools.android.com/tech-docs/unit-testing-support
and check Gradle Plugin User Guide: http://tools.android.com/tech-docs/new-build-system/user-guide

You can see examples of pure Java tests written in JUnit for Android library in one of my projects at: https://github.com/pwittchen/kirai/
(check library/src/test/java/ directory).