JUnit in a Nutshell: Test Runners
The fourth chapter of my multi-part tutorial about JUnit testing essentials explains the purpose of the tool’s exchangable test runners architecture and introduces some of the available implementations. The ongoing example enlarges upon the subject by going through the different possibilities of writting parameterized tests.
Since I have already published an introduction to JUnit Rules, I decided to skip the announced sections on that topic. Instead, I spend the latter a minor update.
Test Runners Architecture
Don’t be afraid to give up the good to go for the great.
In the previous posts we have learned to use some of the xUnit testing patterns [MES] with JUnit. Those concepts are well supported by the default behavior of the tool’s runtime. But sometimes there is a need to vary or supplement the latter for particular test types or objectives.
Consider for example integration tests, that often need to be run in specific environments. Or imagine a set of test cases comprising the specification of a subsystem, which should be composed for common test execution.
JUnit supports the usage of various types of test processors for this purpose. Thus it delegates at runtime test class instantiation, test execution and result reporting to such processors, which have to be sub types of org.junit.Runner
.
A test case can specify its expected runner type with the @RunWith
annotation. If no type is specified the runtime chooses BlockJUnit4ClassRunner
as default. Which is responsible that each test runs with a fresh test instance and invokes lifecycle methods like implicit setup or teardown handlers (see also the chapter about Test Structure).
@RunWith( FooRunner.class ) public class BarTest {
The code snippet shows how the imaginary FooRunner
is specified as test processor for the also imaginary BarTest
.
It seems that usage of special test runners is straight forward, so let us have a look at a few:
Suite and Categories
Probably one of the best known processors is the Suite
. It allows to run collections of tests and/or other suites in a hierarchically or thematically structured way. Note that the specifying class itself has usually no body implementation . It is annotated with a list of test classes, that get executed by running the suite:
@RunWith(Suite.class) @SuiteClasses( { NumberRangeCounterTest.class, // list of test cases and other suites } ) public class AllUnitTests {}
However the structuring capabilities of suites are somewhat limited. Because of this JUnit 4.8 introduced the lesser known Categories
concept. This makes it possible to define custom category types like unit-, integration- and acceptance tests for example. To assign a test case or a method to one of those categories the Category
annotation is provided:
// definition of the available categories public interface Unit {} public interface Integration {} public interface Acceptance {} // category assignment of a test case @Category(Unit.class) public class NumberRangeCounterTest { [...] } // suite definition that runs tests // of the category 'Unit' only @RunWith(Categories.class) @IncludeCategory(Unit.class) @SuiteClasses( { NumberRangeCounterTest.class, // list of test cases and other suites } ) public class AllUnitTests {}
With Categories
annotated classes define suites that run only those tests of the class list, that match the specified categories. Specification is done via include and/or exclude annotations. Note that categories can be used in Maven or Gradle builds without defining particular suite classes (see the Categories section of the JUnit documentation).
Since maintenance of the suite class list and category annotations is often considered somewhat tedious, you might prefer categorizing via test postfix names à la FooUnitTest instead of FooTest. This allows to filter categories on type-scope at runtime.
But this filtering is not supported by JUnit itself, why you may need a special runner that collects the available matching tests dynamically. A library that provides an appropriate implementation is Johannes Link‘s ClasspathSuite
. If you happen to work with integration tests in OSGi environment Rüdiger‘s BundleTestSuite
does something similar for bundles.
After these first impressions of how test runners can be used for test bundling let us continue the tutorial’s example with something more exciting.
Parameterized Tests
The example used throughout this tutorial is about writing a simple number range counter, which delivers a certain amount of consecutive integers, starting from a given value. Additionally a counter depends on a storage type for preserving its current state. For more information please refer to the previous chapters.
Now assume that our NumberRangeCounter
, which is initialized by constructor parameters, should be provided as API. So we may consider it reasonable, that instance creation checks the validity of the given parameters.
We could specify the appropriate corner cases, which should be acknowledged with IllegalArgumentException
s, by a single test each. Using the Clean JUnit Throwable-Tests with Java 8 Lambdas approach, such a test verifying that the storage parameter must not be null might look like this:
@Test public void testConstructorWithNullAsStorage() { Throwable actual = thrown( () -> new NumberRangeCounter( null, 0, 0 ) ); assertTrue( actual instanceof IllegalArgumentException ); assertEquals( NumberRangeCounter.ERR_PARAM_STORAGE_MISSING, actual.getMessage() ); }
To keep the post in scope, I also skip the discussion, whether a NPE would be better than the IAE.
In case we have to cover a lot of corner cases of that kind, the approach above might lead to a lot of very similar tests. JUnit offers the Parameterized
runner implementation to reduce such redundancy. The idea is to provide various data records for the common test structure.
To do so a public static method annotated with @Parameters
is used to create the data records as a collection of object arrays. Furthermore, the test case needs a public constructor with arguments, that match the data types provided by the records.
The parameterized processor runs a given test for each record supplied by the parameters method. This means for each combination of test and record a new instance of the test class is created. The constructor parameters get stored as fields and can be accessed by the tests for setup, exercise, and verification:
@RunWith( Parameterized.class ) public class NumberRangeCounterTest { private final String message; private final CounterStorage storage; private final int lowerBound; private final int range; @Parameters public static Collection<Object[]> data() { CounterStorage dummy = mock( CounterStorage.class ); return Arrays.asList( new Object[][] { { NumberRangeCounter.ERR_PARAM_STORAGE_MISSING, null, 0, 0 }, { NumberRangeCounter.ERR_LOWER_BOUND_NEGATIVE, dummy, -1, 0 }, [...] // further data goes here... } ); } public NumberRangeCounterTest( String message, CounterStorage storage, int lowerBound, int range ) { this.message = message; this.storage = storage; this.lowerBound = lowerBound; this.range = range; } @Test public void testConstructorParamValidation() { Throwable actual = thrown( () -> new NumberRangeCounter( storage, lowerBound, range ) ); assertTrue( actual instanceof IllegalArgumentException ); assertEquals( message, actual.getMessage() ); } [...] }
While the example surely reduces test redundancy it is at least debatable with respect to readability. In the end, this often depends on the amount of tests and the structure of the particular test data. But it is definitively unfortunate that tests, which do not use any record values, will be executed multiple times, too.
Because of this parameterized tests are often kept in separate test cases, which usually feels more like a workaround than a proper solution. Hence, a wise guy came up with the idea to provide a test processor that circumvents the described problems.
@Parameter
annotation for field injection instead. See https://github.com/junit-team/junit/wiki/Parameterized-tests#using-parameter-for-field-injection-instead-of-constructor for more information.Testing with JUnit
Testing with JUnit is one of the most valuable skills a Java developer can learn. No matter what your specific background, whether you’re simply interested in building up a safety net to reduce regressions of your desktop application or in improving your server-side reliability based on robust and reusable components, unit testing is the way to go.
Frank has written a book that gives a profound entry point in the essentials of testing with JUnit and prepares you for test-related daily work challenges.
JUnitParams
The library JUnitParams provides the types JUnitParamsRunner
and @Parameter
. The param annotation specifies the data records for a given test. Note the difference to the JUnit annotation with the same simple name. The latter marks a method that provides the data records!
The test scenario above could be rewritten with JUnitParams as shown in the following snippet:
@RunWith( JUnitParamsRunner.class ) public class NumberRangeCounterTest { public static Object data() { CounterStorage dummy = mock( CounterStorage.class ); return $( $( ERR_PARAM_STORAGE_MISSING, null, 0, 0 ), $( ERR_LOWER_BOUND_NEGATIVE, dummy, -1, 0 ) ); } @Test @Parameters( method = "data" ) public void testConstructorParamValidation( String message, CounterStorage storage, int lowerBound, int range ) { Throwable actual = thrown( () -> new NumberRangeCounter( storage, lowerBound, range ) ); assertTrue( actual instanceof IllegalArgumentException ); assertEquals( message, actual.getMessage() ); } [...] }
While this is certainly more compact and looks cleaner on first glance, a few constructs need further explanation. The $(...)
method is defined in JUnitParamsRunner
(static import) and is a shortcut for creating arrays of objects. Once accustomed to it, data definition gets more readable.
The $
shortcut is used in the method data
to create a nested array of objects as return value. Although the runner expects a nested data array at runtime, it is able to handle a simple object type as a return value.
The test itself has an additional @Parameters
annotation. The annotation’s method declaration refers to the data provider used to supply the test with the declared parameters. The method name is resolved at runtime via reflection. This is the down-side of the solution, as it is not compile-time safe.
But there are other use case scenarios where you can specify data provider classes or implicit values, which, therefore, do not suffer from that trade-off. For more information please have a look at the library’s quick start guide for example.
Another huge advantage is, that now only those tests run against data records that use the @Parameters
annotation. Standard tests are executed only once. This, in turn, means that the parameterized tests can be kept in the unit’s default test case.
Wrap Up
The sections above outlined the sense and purpose of JUnit’s exchangeable test runners architecture. It introduced suite and categories to show the basic usage and carried on with an example of how test runners can ease the task of writing data record related tests.
For a list of additional test runners the pages Test runners and Custom Runners at junit.org might be a good starting point. And if you wonder what the Theories
runner of the title picture is all about, you might have a look at Florian Waibels post JUnit – the Difference between Practice and @Theory.
Next time on JUnit in a Nutshell, I finally will cover the various types of assertions available to verify test results. So stay tuned and do not forget to share the knowledge with one of the social media buttons below ;-)
References
[MES] xUnit Test Patterns, Gerard Meszaros, 2007
- Xmas Clean Sheet Update (0.9) - 21. December 2021
- Clean Sheet Service Update (0.8) - 23. May 2020
- Clean Sheet Service Update (0.7) - 24. April 2020
[…] fundamentals of test driven development on JUnit with proper test structure, test isolation, and test runners. Learn how to get up and running with plenty of examples and best […]
[…] http://www.codeaffine.com/2014/09/03/junit-nutshell-test-runners/ […]
thank you for your tutorial.
I have read some source code from junit.
http://www.lookatsrc.com/source/class:org.junit.runner.Runner
http://www.lookatsrc.com/source/org/junit/Before.java
[…] test runner called JUnitPlatform can be used to run new tests as part of a JUnit 4 run. You will find it in […]