JUnit in a Nutshell: Test Runners

Home  >>  JUnit  >>  JUnit in a Nutshell: Test Runners

JUnit in a Nutshell: Test Runners

On September 3, 2014, Posted by , In JUnit, By ,,, , With 4 Comments

The fourth chapter of my multi-part tutorial about JUnit testing essentials explains the purpose of the tool’s exchangable test runners architecture and introduces some of the available implementations. The ongoing example enlarges upon the subject by going through the different possibilities of writting parameterized tests.

Since I have already published an introduction to JUnit Rules, I decided to skip the announced sections on that topic. Instead, I spend the latter a minor update.

Test Runners Architecture

Don’t be afraid to give up the good to go for the great.John D. Rockefeller

In the previous posts we have learned to use some of the xUnit testing patterns [MES] with JUnit. Those concepts are well supported by the default behavior of the tool’s runtime. But sometimes there is a need to vary or supplement the latter for particular test types or objectives.

Consider for example integration tests, that often need to be run in specific environments. Or imagine a set of test cases comprising the specification of a subsystem, which should be composed for common test execution.

JUnit supports the usage of various types of test processors for this purpose. Thus it delegates at runtime test class instantiation, test execution and result reporting to such processors, which have to be sub types of org.junit.Runner.

A test case can specify its expected runner type with the @RunWith annotation. If no type is specified the runtime chooses BlockJUnit4ClassRunner as default. Which is responsible that each test runs with a fresh test instance and invokes lifecycle methods like implicit setup or teardown handlers (see also the chapter about Test Structure).

@RunWith( FooRunner.class )
public class BarTest {

The code snippet shows how the imaginary FooRunner is specified as test processor for the also imaginary BarTest.

Usually there is no need to write custom test runners. But in case you have to, Michael Scharhag has written a good explanation of the JUnit’s runner architecture recently.

It seems that usage of special test runners is straight forward, so let us have a look at a few:

Suite and Categories

Probably one of the best known processors is the Suite. It allows to run collections of tests and/or other suites in a hierarchically or thematically structured way. Note that the specifying class itself has usually no body implementation . It is annotated with a list of test classes, that get executed by running the suite:

@RunWith(Suite.class)
@SuiteClasses( { 
  NumberRangeCounterTest.class,
  // list of test cases and other suites
} )
public class AllUnitTests {}

However the structuring capabilities of suites are somewhat limited. Because of this JUnit 4.8 introduced the lesser known Categories concept. This makes it possible to define custom category types like unit-, integration- and acceptance tests for example. To assign a test case or a method to one of those categories the Category annotation is provided:

// definition of the available categories
public interface Unit {}
public interface Integration {}
public interface Acceptance {}

// category assignment of a test case
@Category(Unit.class)
public class NumberRangeCounterTest {
  [...]
}

// suite definition that runs tests
// of the category 'Unit' only
@RunWith(Categories.class)
@IncludeCategory(Unit.class)
@SuiteClasses( { 
  NumberRangeCounterTest.class,
  // list of test cases and other suites
} )
public class AllUnitTests {}

With Categories annotated classes define suites that run only those tests of the class list, that match the specified categories. Specification is done via include and/or exclude annotations. Note that categories can be used in Maven or Gradle builds without defining particular suite classes (see the Categories section of the JUnit documentation).

For more information on categories: John Ferguson Smart’s has written a detailled explanation about Grouping tests using JUnit categories.

Since maintenance of the suite class list and category annotations is often considered somewhat tedious, you might prefer categorizing via test postfix names à la FooUnitTest instead of FooTest. This allows to filter categories on type-scope at runtime.

But this filtering is not supported by JUnit itself, why you may need a special runner that collects the available matching tests dynamically. A library that provides an appropriate implementation is Johannes Link‘s ClasspathSuite. If you happen to work with integration tests in OSGi environment Rüdiger‘s BundleTestSuite does something similar for bundles.

See also  JUnit in a Nutshell: Test Isolation

After these first impressions of how test runners can be used for test bundling let us continue the tutorial’s example with something more exciting.

Parameterized Tests

The example used throughout this tutorial is about writing a simple number range counter, which delivers a certain amount of consecutive integers, starting from a given value. Additionally a counter depends on a storage type for preserving its current state. For more information please refer to the previous chapters.

Now assume that our NumberRangeCounter, which is initialized by constructor parameters, should be provided as API. So we may consider it reasonable, that instance creation checks the validity of the given parameters.

We could specify the appropriate corner cases, which should be acknowledged with IllegalArgumentExceptions, by a single test each. Using the Clean JUnit Throwable-Tests with Java 8 Lambdas approach, such a test verifying that the storage parameter must not be null might look like this:

  @Test
  public void testConstructorWithNullAsStorage() {
    Throwable actual = thrown( () -> new NumberRangeCounter( null, 0, 0 ) );
    
    assertTrue( actual instanceof IllegalArgumentException );
    assertEquals( NumberRangeCounter.ERR_PARAM_STORAGE_MISSING,
                  actual.getMessage() );
  }
Note that I stick with the JUnit build-in functionality for verification. I will cover the pro and cons of particular matcher libraries (Hamcrest, AssertJ) in a separate post.

To keep the post in scope, I also skip the discussion, whether a NPE would be better than the IAE.

In case we have to cover a lot of corner cases of that kind, the approach above might lead to a lot of very similar tests. JUnit offers the Parameterized runner implementation to reduce such redundancy. The idea is to provide various data records for the common test structure.

To do so a public static method annotated with @Parameters is used to create the data records as a collection of object arrays. Furthermore, the test case needs a public constructor with arguments, that match the data types provided by the records.

The parameterized processor runs a given test for each record supplied by the parameters method. This means for each combination of test and record a new instance of the test class is created. The constructor parameters get stored as fields and can be accessed by the tests for setup, exercise, and verification:


@RunWith( Parameterized.class )
public class NumberRangeCounterTest {
  
  private final String message;
  private final CounterStorage storage;
  private final int lowerBound;
  private final int range;
  
  @Parameters
  public static Collection<Object[]> data() {
    CounterStorage dummy = mock( CounterStorage.class );
    return Arrays.asList( new Object[][] { 
      { NumberRangeCounter.ERR_PARAM_STORAGE_MISSING, null, 0, 0 }, 
      { NumberRangeCounter.ERR_LOWER_BOUND_NEGATIVE, dummy, -1, 0 },
       [...] // further data goes here... 
    } );
  }
  
  public NumberRangeCounterTest(
    String message, CounterStorage storage, int lowerBound, int range )
  {
    this.message = message;
    this.storage = storage;
    this.lowerBound = lowerBound;
    this.range = range;
  }
  
  @Test
  public void testConstructorParamValidation() {
    Throwable actual = thrown( () -> 
      new NumberRangeCounter( storage, lowerBound, range ) );
    
    assertTrue( actual instanceof IllegalArgumentException );
    assertEquals( message, actual.getMessage() );
  }

  [...]
}

While the example surely reduces test redundancy it is at least debatable with respect to readability. In the end, this often depends on the amount of tests and the structure of the particular test data. But it is definitively unfortunate that tests, which do not use any record values, will be executed multiple times, too.

Because of this parameterized tests are often kept in separate test cases, which usually feels more like a workaround than a proper solution. Hence, a wise guy came up with the idea to provide a test processor that circumvents the described problems.

Update 2015/10/16: Recent versions of JUnit make the parameterized constructor obsolete by providing the @Parameter annotation for field injection instead. See https://github.com/junit-team/junit/wiki/Parameterized-tests#using-parameter-for-field-injection-instead-of-constructor for more information.

 Testing with JUnit

Testing with JUnit Book

Testing with JUnit is one of the most valuable skills a Java developer can learn. No matter what your specific background, whether you’re simply interested in building up a safety net to reduce regressions of your desktop application or in improving your server-side reliability based on robust and reusable components, unit testing is the way to go.

See also  What are Mockito Extra Interfaces?

Frank has written a book that gives a profound entry point in the essentials of testing with JUnit and prepares you for test-related daily work challenges.

  Get It Now! 

JUnitParams

The library JUnitParams provides the types JUnitParamsRunner and @Parameter. The param annotation specifies the data records for a given test. Note the difference to the JUnit annotation with the same simple name. The latter marks a method that provides the data records!

The test scenario above could be rewritten with JUnitParams as shown in the following snippet:

@RunWith( JUnitParamsRunner.class )
public class NumberRangeCounterTest {
  
  public static Object data() {
    CounterStorage dummy = mock( CounterStorage.class );
    return $( $( ERR_PARAM_STORAGE_MISSING, null, 0, 0 ),
              $( ERR_LOWER_BOUND_NEGATIVE, dummy, -1, 0 ) );  
  }
  
  @Test
  @Parameters( method = "data" )
  public void testConstructorParamValidation(
    String message, CounterStorage storage, int lowerBound, int range ) 
  {
    Throwable actual = thrown( () -> 
      new NumberRangeCounter( storage, lowerBound, range ) );
    
    assertTrue( actual instanceof IllegalArgumentException );
    assertEquals( message, actual.getMessage() );
  }
  
  [...]
}

While this is certainly more compact and looks cleaner on first glance, a few constructs need further explanation. The $(...) method is defined in JUnitParamsRunner (static import) and is a shortcut for creating arrays of objects. Once accustomed to it, data definition gets more readable.

The $ shortcut is used in the method data to create a nested array of objects as return value. Although the runner expects a nested data array at runtime, it is able to handle a simple object type as a return value.

The test itself has an additional @Parameters annotation. The annotation’s method declaration refers to the data provider used to supply the test with the declared parameters. The method name is resolved at runtime via reflection. This is the down-side of the solution, as it is not compile-time safe.

But there are other use case scenarios where you can specify data provider classes or implicit values, which, therefore, do not suffer from that trade-off. For more information please have a look at the library’s quick start guide for example.

Another huge advantage is, that now only those tests run against data records that use the @Parameters annotation. Standard tests are executed only once. This, in turn, means that the parameterized tests can be kept in the unit’s default test case.

Test Runners: junit-params-test

Wrap Up

The sections above outlined the sense and purpose of JUnit’s exchangeable test runners architecture. It introduced suite and categories to show the basic usage and carried on with an example of how test runners can ease the task of writing data record related tests.

For a list of additional test runners the pages Test runners and Custom Runners at junit.org might be a good starting point. And if you wonder what the Theories runner of the title picture is all about, you might have a look at Florian Waibels post JUnit – the Difference between Practice and @Theory.

Next time on JUnit in a Nutshell, I finally will cover the various types of assertions available to verify test results. So stay tuned and do not forget to share the knowledge with one of the social media buttons below ;-)

Chapter Navigation
 Prev | Table of Content | Next 

References

[MES] xUnit Test Patterns, Gerard Meszaros, 2007

Frank Appel
Follow me
Latest posts by Frank Appel (see all)

4 Comments so far:

  1. […] fundamentals of test driven development on JUnit with proper test structure, test isolation, and test runners. Learn how to get up and running with plenty of examples and best […]

  2. […] test runner called JUnitPlatform can be used to run new tests as part of a JUnit 4 run. You will find it in […]