Running JUnit Tests Repeatedly Without Loops
Recently I came across a problem where I had to write tests for a method that calculates randomly distributed values within a certain range of possibilities 1. More precisely if you assume a signature that looks like
interface RandomRangeValueCalculator {
long calculateRangeValue( long center, long radius );
}
a test might verify the following2:
public class RandomRangeValueCalculatorImplTest {
@Test
public void testCalculateRangeValue() {
long center = [...];
long radius = [...];
RangeValueCalculator calculator = [...];
long actual = calculator.calculateRangeValue( center, radius );
assertTrue( center + radius >= actual );
assertTrue( center - radius <= actual );
}
}
However calculating range values for the same center and radius multiple times will return different results (at least most of the time). Therefore the solution seemed somewhat brittle in a sense that a poor implementation might easily produce intermittent failures. On the other hand, I did not want to descend into the depths of actually measuring the value distribution. The latter (random, Gaussian or the like) was provided by a collaborator and its proper usage was already confirmed by additional tests.
It occurred to me that a more pragmatic solution could be to actually run the test above automatically over and over again to make it more ‘significant’. Of course, the easiest way to achieve this would be to put the test’s content into a loop and go on living.
Testing with JUnit
Testing with JUnit is one of the most valuable skills a Java developer can learn. No matter what your specific background, whether you’re simply interested in building up a safety net to reduce regressions of your desktop application or in improving your server-side reliability based on robust and reusable components, unit testing is the way to go.
Frank has written a book that gives a profound entry point in the essentials of testing with JUnit and prepares you for test-related daily work challenges.
But for a start, it looked somewhat wrong having the asserts within a loop and mixing two aspects into one test run. And even more important the covered problem domain required more tests of the kind. So given the intent of reducing redundancy I remembered my post about JUnit-Rules and implemented a simple repeat rule3. With this rule in place, the test above could be gently amended to:
public class RandomRangeValueCalculatorImplTest {
@Rule
public RepeatRule repeatRule = new RepeatRule();
@Test
@Repeat( times = 10000 )
public void testCalculateRangeValue() {
long center = [...];
long radius = [...];
RangeValueCalculator calculator = [...];
long actual= calculator.calculateRangeValue( center, radius );
assertTrue( center + radius >= actual );
assertTrue( center - radius <= actual );
}
}
I think it is quite easy to understand that the testCalculateRangeValue method will be executed 10000 times while running the test case. The following snippet shows the implementation of the RepeatRule, which is straightforward:
public class RepeatRule implements TestRule {
@Retention( RetentionPolicy.RUNTIME )
@Target( {
java.lang.annotation.ElementType.METHOD
} )
public @interface Repeat {
public abstract int times();
}
private static class RepeatStatement extends Statement {
private final int times;
private final Statement statement;
private RepeatStatement( int times, Statement statement ) {
this.times = times;
this.statement = statement;
}
@Override
public void evaluate() throws Throwable {
for( int i = 0; i < times; i++ ) {
statement.evaluate();
}
}
}
@Override
public Statement apply(
Statement statement, Description description )
{
Statement result = statement;
Repeat repeat = description.getAnnotation( Repeat.class );
if( repeat != null ) {
int times = repeat.times();
result = new RepeatStatement( times, statement );
}
return result;
}
}
So far the RepeatRule serves it purpose and the system functionalities based on the mentioned implementation is working like a charm. Nevertheless sometimes someone misses the forest for the trees and so I thought it might be a good idea to share this solution to see what other people come up with.
Update 2014/05/09: Due to a request I have created a github gist to provide the rule code as download:
https://gist.github.com/fappel/8bcb2aea4b39ff9cfb6e
- Actually, this was only one part of the problem domain but I consider it as a sufficient motivation for this post. ↩
- Formalistically spoken: f(n,m)∈{e|e≥n-m∧e≤n+m}, for all e,n,m ∈ ℕ ↩
- A short google search only came up with a similar solution available in Spring, which was not available in my set of libraries. ↩
With a specialization in the Java language and a keen focus on quality-oriented agile methodologies, he is a strong advocate for test-driven practices.
To Frank, the software development process is akin to craftsmanship, founded on a harmonious blend of knowledge and daily work experience.
fappel@codeaffine.com
- Xmas Clean Sheet Update (0.9) - 21. December 2021
- Clean Sheet Service Update (0.8) - 23. May 2020
- Clean Sheet Service Update (0.7) - 24. April 2020
it’s awesome article, thank you. But your button Google+ doesn’t work somehow.
Regards
Walid
Thanks for the feedback – but I can’t reproduce a problem. What browser are you using?
Generating random numbers is a separate concern from doing something with them so it might be a good idea for any implementation of RandomRangeValueCalculator to take a dependency for an implementation of a random value generator.
This would allow you to pass in a mock random value generator that generates a repeatable series of values allowing you to test the specific boundary cases you are interested in, making your tests precisely predictable and repeatable.
Arnold, I completely agree with your thorough explanation and that is what I meant by the short sentence ‘The latter […] was provided by a collaborator and its proper usage was already confirmed by additional tests’. So you caught the somewhat weak point in my motivation ;-)
However working with mocks I consider it desirable to also have kind of integration tests with real collaborator implementations (if possible with reasonable effort). I have observed that “corner cases” might change over time unreflected by tests with mocks. Using mocks reminds me of crash test dummies for cars, where the car manufacturers at some point realized, that they were building very save cars for the dummies – but not for human beings… So mocks are very helpful but they are not the real thing.
With problems like the given example in the post running tests repeatedly might be a simple way to achieve such ‘real world’ test scenarios.
You mention a similar construct available in Spring, but you link generally to the Spring Source site rather than their particular solution. My quick Google search isn’t netting me results – what are you searching exactly to find the Spring solution? Thanks in advance!
Good question – if only I could remember…
Well, after a quick google search I think I was refering to
http://static.springsource.org/spring/docs/2.5.6/reference/testing.html; 8.3.5. Common annotations; @Repeat
Maybe this helps ;-)
If we use this annotation will it also repeatedly run the methods with the @Before and @After annotation as well or do we need to make sure that the resting happens within the method??
Hi Jonathan, this is a good question. The execution of the RepeatRule is based on
Statement.evaluate(). The later ensures that @Before and @After annotated methods will be excecuted as well. However it does not create a new instance of the Testcase for each repetition of a method annotated with @Repeat. This means you should initialize your fixture within @Before annotated methods and not using an implicit constructor approach. To make the later more understandable have a look at the following Testcase:public class FooTest {
@Rule public RepeatRule rule = new RepeatRule();
public FooTest() {
System.out.println();
System.out.println( “create”);
}
@Before
public void setUp() {
System.out.println( “before” );
}
@After
public void tearDown() {
System.out.println( “after” );
}
@Test
@Repeat(times=4)
public void testRepeatedly() {
System.out.println( “execute”);
}
@Test
@Repeat(times=4)
public void testRepeatedly2() {
System.out.println( “execute2”);
}
}
The output created running the tests will be:
create
before
execute
after
before
execute
after
before
execute
after
before
execute
after
create
before
execute2
after
before
execute2
after
before
execute2
after
before
execute2
after
So hopefully this answers your question :-)
If we use Rule to repeat the test, the testStarted and the testFinished of the RunListeners is called only once even if the statement.evaluate is called twice. Please shed some more light on this. If the @Before and @After can be called by the framework, then why not the testStarted and testFinished?
RunListeners#testStarted and#testFinished are called by ParentRunner#runLeaf(Statement, Description, RunNotifier). If you have a look at this the notification of the runlisteners callback methods takes place in a try-catch block around Statement#evaluate(). In our case that statement is the RepeatRule statement which executes the ‘real’ test multiple times (including @Before and @After annotated methods). However notifications are only fired once. Which might be a bit unsatisfactory if you want to get informed about any repetition.
Unfortunately I think there is not much you can do, at most writing your own runner for particular cases. But I don’t know if the latter is really a good idea or worthwhile the effort. Keep also in mind that the repeat rule was meant that a test fails if one repetition fails and the test is successful if all repetitions are successful. This means the implementation works from the point of the RunListeners as expected.
The gist does not contain the following code:
@Retention( RetentionPolicy.RUNTIME )
@Target( {
java.lang.annotation.ElementType.METHOD
} )
public @interface Repeat {
public abstract int times();
}
Renan, thanks for the hint. I updated the gist accordingly.
Hello Frank,
Well, I have just copied the Repeated Rule and added it to my Integration Test that I m running with Arquillian. When I put times = 5 . It gets executed 25 times instead of 5. It seems like its multiplying the times number. 5×5 . Because If I put times =t 4 times , it gets executed 16 times .
Any idea what is going on ?
Hi Saif, this sounds strange indeed. But if you have a look at line 19 of the RepeatRule gist code, there is no nested loop whatsoever. And doing a short test I could not reproduce the problem you have described. Do you have a simple test example to reproduce the behavior? Frank
Hi, I’ve bundled such a repetition rule in a tiny Maven dependency with built-in concurrency support.
Hope it helps.
https://github.com/cbismuth/junit-repeat-rule
Cool, thx Chrstophe. This surely eases the integration of such a repetition rule.
Hi, Christophe!
Can I ask you?
public class JustTest {
@Rule
public RepeatRule repeatRule = new RepeatRule();
@Test
@Repeat( times = 1 )
public void testCalculateRangeValue() {
assertTrue( 8==9 );
assertNotNull( null );
}
}
Why test passed?
nice one, works like a charm, thanks
Thx, glad it does :-)
Added credits to you Franck on my GitHub repo, sorry for having forgotten to do so.
https://github.com/cbismuth/junit-repeat-rule/commit/c05130db352ea9d618811de26326db201c627dce
No Problem – better late than never, thx Christophe ;-)
public class JustTest {
@Rule
public RepeatRule repeatRule = new RepeatRule();
@Test
@Repeat( times = 1 )
public void testCalculateRangeValue() {
assertTrue( 8==9 );
assertNotNull( null );
}
}
Why test passed?
Thanks Serzh for checking out the RepeatRule. I just checked the gist against your example, but it worked as expected. Meaning, both asserts fail.
Can I ask you TestClass full example with imports & dependencies?
This mine. And asserts pass.
Thanks.
import org.junit.Rule;
import org.junit.Test;
import repeat.Repeat;
import repeat.RepeatRule;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
public class JustTest {
@Rule
public RepeatRule repeatRule = new RepeatRule();
@Test
@Repeat( times = 1 )
public void testCalculateRangeValue() {
assertTrue( 8==9 );
assertNotNull( null );
}
}
junit
junit
4.12
com.github.cbismuth
junit-repeat-rule
1.1.1
Sorry, that was not your implementation.
[…] For a more useful implementation based on an annotation that define which test method has to been executed how often have a look at this blog: http://www.codeaffine.com/2013/04/10/running-junit-tests-repeatedly-without-loops/ […]