A JUnit Rule to Conditionally Ignore Tests

Home  >>  JUnit  >>  A JUnit Rule to Conditionally Ignore Tests

A JUnit Rule to Conditionally Ignore Tests

On November 18, 2013, Posted by , In JUnit, By ,, , With 24 Comments

I always believed that using @Ignore to deactivate tests is a bad idea. Except, maybe as one way to put tests that fail intermittently into quarantine to attend to them later (as Martin Fowler describes it here). This bears the danger that the test suite decays as more and more tests keep getting ignored and forgotten. Therefore, you should have a policy in place to ensure that tests aren’t quarantined for too long. Well, so I thought until recently when we ran into this:

In a project that Frank and I work on, we ran into an SWT issue described here. On non-Windows platforms, asserting whether an SWT widget has got the input focus does not work with automated tests.

We decided to ignore focus-related tests on non-Windows platforms for now. Though our build server runs on Linux, we found it safe enough as both our development environments run on Windows.

In JUnit, assumptions are the means to skip tests that aren’t meaningful under the given condition. Expressed that way, our test would look like this:

public void testFocus() {
  assumeTrue( isRunningOnWindows() );
  // ...
}

But we didn’t want the test code mingled with conditions if it should be executed at all. The code that decides whether the test is ignored should be separated from the test code itself.

This led us to create a ConditionalIgnore annotation and a corresponding rule to hook it into the JUnit runtime. The thing is simple and best explained with an example:

public class SomeTest {
  @Rule
  public ConditionalIgnoreRule rule = new ConditionalIgnoreRule();

  @Test
  @ConditionalIgnore( condition = NotRunningOnWindows.class )
  public void testFocus() {
    // ...
  }
}

public class NotRunningOnWindows implements IgnoreCondition {
  public boolean isSatisfied() {
    return !System.getProperty( "os.name" ).startsWith( "Windows" );
  }
}

The ConditionalIgnore annotation requires a ‘condition’ property that points to a class that implements IgnoreContition. At runtime, an instance of the IgnoreCondition implementation is created and its isSatisfied() method decides whether the test is ignored (returns true) or not (returns false). Finally, there is an IgnoreConditionRule that hooks the annotations into the JUnit runtime.

See also  What the Heck Is Mutation Testing?

If the IgnoreCondition implementation decides to ignore a test case, an AssumptionViolatedException is thrown. Therefore, the ConditionalIgnore annotation has the same effect as if an Assume condition would return false. With a slight difference that we consider an advantage: @Before and @After methods are not executed for ignored tests.

The source code of the rule and its related classes can be found here:
https://gist.github.com/rherrmann/7447571.

The remaining issue with Assume is that it affects the test statistics. If an Assume condition is found to be false, it treats the test as having passed even though it hasn’t run. To overcome that, you’d have to provide your own runner that handles AssumptionViolatedException the way you want.

Even though I just wrote about ignoring tests in length, I am still convinced that tests at best should not be ignored and if so only in exceptional cases.

 Testing with JUnit

Testing with JUnit Book

Testing with JUnit is one of the most valuable skills a Java developer can learn. No matter what your specific background, whether you’re simply interested in building up a safety net to reduce regressions of your desktop application or in improving your server-side reliability based on robust and reusable components, unit testing is the way to go.

Frank has written a book that gives a profound entry point in the essentials of testing with JUnit and prepares you for test-related daily work challenges.

  Get It Now! 

Rüdiger Herrmann
Follow me

24 Comments so far:

  1. Tomek says:

    Looks like a good example of JUnit rule usage! Thanks for sharing!


    Tomek Kaczanowski
    http://practicalunittesting.com

  2. Vijaya says:

    By Using the above sample, if a testcase is ignored, will that appear in Skipped one, if so, how to make sure that test case is ignored and does not appear in Skipped one

    • Rüdiger Herrmann says:

      Vijaya,

      if you refer to Maven ‘skipped’ tests, then yes, the maven test runner lists ignored tests as skipped. But as I said above, how assumption violations are handled is up to the test runner. Does that answer your question?

      Rüdiger

  3. Matt Morrissette says:

    The above example can cause an error using the referenced gist on GitHub for the ConditionalIgnoreRule if your class that extends IgnoreCondition is declared inside your test case.

    Please use the gist at:
    https://gist.github.com/yinzara/9980184

    • Rüdiger Herrmann says:

      Good catch, Matt. Thank you for this hint. I have also updated the original gist so that readers who don’t follow down to the comments benefit from your enhancement.

  4. Sebastian Millies says:

    Thanks for the post. Your approach does makt the test code cleaner.

    One idea, not tested: With java 8, shouldn’t it be possible to simplify things quite a bit and leave out the IgnoreConditionCreator, passing a lambda directly in @ConditionalIgnore? The interface IgnoreCondition could just trivially extend BooleanSupplier.

  5. Sebastian Millies says:

    Well, I guess I have to take that back because functional interfaces are not allowed as annotation method return types, making it impossible to pass a lambda as an annotation parameter. Too bad.

    • Sebastian Millies says:

      However, it is still possible to have better static type safety and simpler reflection code by requiring IgnoreCondition to be implemented by enum constants. See my fork of your gist at https://gist.github.com/smillies/2ca19e1faa903f08f47e

      • Rüdiger Herrmann says:

        Thanks Sebastian for sharing your idea and pardon the delay, I was on vacation.

        I can’t help but your approach seems to complicate things for clients, or am I missing something? In your example, clients have to specify the enum-class and a string that denotes the enum-value (which might break during rename refactorings) whereas the original approach only requires a class to be specified.

  6. Olegs Klujs says:

    Thanks Herrmann,

    Just what I was searching for.
    Made my day!

  7. raho says:

    Hi,
    isn’t there an error in line 42 of ConditionalIgnoreRule.java?
    When the condition’s isSatisfied() returns true, an IgnoreStatement will be created. This is the opposite as mentioned in your comment above “isSatisfied() method decides whether the test is ignored (returns true) or not (returns false)”
    Regards, Rainer

    • Rüdiger Herrmann says:

      Hi Rainer,

      maybe it is because it’s a litle late, but I am afraid, I can’t see the error. The code returns an IgnoreStatement when isSatisfied() returns true and causes the test to be ignored. Isn’t that in line with the description that “the method decides whether the test is ignored (isSatisfied() returns true)”?

      Best
      Rüdiger

  8. raho says:

    Hi Rüdiger,
    you are right – it was my fault, ’cause I had a testcase already marked with @Ignore and wondered why the @ConditionalIgnore does not work!
    Sorry and Regards,
    Rainer

  9. Clyde says:

    Hi Rüdiger,

    I’m running into an issue on Chrome, Firefox, and IE browsers. The problem is that if a test Class has multiple test cases some using @IgnoreCondition and some don’t. If the test case with @IgnoreCondition runs first then all the tests follow for that Class won’t run. I noticed that if the non ignore test runs first then it is ok.

    ex.
    public class MyTest {
    @ConditionalIgnore (condition = IgnorePlatforms.class)
    public void test1() {
    }
    public void test2() {
    }
    }

    Do you have any insight on how to fix this?

    Thanks,
    Clyde

    • Rüdiger Herrmann says:

      Hi Clyde,

      I am afraid, with the given information I can’t really tell as to why those tests won’t run.
      What do you mean with ‘running on’ Chrome, FF, etc? What means ‘won’t run’ exactly, are those tests entirely missing in the tests test report/JUnit view? Can you provide a self-contained test class that reproduces the behavior? Did you try to annotate the tests that should always run with an ‘ignore never’ condition?

      HTH
      Rüdiger

      • Clyde says:

        Hi Rüdiger,

        Thanks for the response and sorry I wasn’t clear. The issue is that if a test case with @IgnoreCondition annotation runs first, the test run stops and won’t run the remaining test cases in the same test class. This is reproducible on Chrome, FF, and IE browsers.

        If I run a test class that has 2 test cases and the first test that runs has @IgnoreCondition like above, this is result I got:
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 1

        As compare to:
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 1

        Side note: I’m using this annotation in a Serenity automation Framework (https://github.com/serenity-bdd/serenity-core), I will try to simplify my project to see if there’s any luck.

        I haven’t heard of ‘ignore never’ annotation, do you have a reference?

        Regards,
        Clyde

        • Rüdiger Herrmann says:

          Hi Clyde,

          as I haven’t seen this before, it may be related to Serenity. The readme there says that it instruments the test code, which may be the reason. But I’m only guessing here.

          The ‘ignore never’ conditional ignore would be a workaround. Based on that you said, that tests annotated with @IgnoreCondition are run and those without do not run, the fix would be to annotate all tests with @IgnoreCondition. Those tests that should always run use a condition whose isSatistied() method always returns false.

          Best,
          Rüdiger

  10. Paresh says:

    Can I pass multiple conditions in @ConditionalIgnore ?

    I have situation where I need to skip test on particular environment and region
    I’m looking for something like below where it can accepts two conditions
    @ConditionalIgnore(condition = condition1.class , condition2.class )

    • Rüdiger Herrmann says:

      In its current state, the annotation allows for a single condition only. But you can certainly extend it to evaluate multiple conditions.

      However, I would rather use a composite condition, for example

      class CompositeContition implements IgnoreCondition {
      @Override
      public boolean isSatisfied() {
      return new Condition1().isSatisfied() && new Condition1().isSatisfied();
      }
      }

      Allowing for multiple conditions also leaves the open question to the reader as to how the conditions are combined (and / or / xor even?). With the latter approach, you can choose a meaningful name and – in doubt – step into the code of the condition.

      • Paresh says:

        Make sense.
        Currently I have very few conditions however I’m expecting more conditions to be added.
        Is there any way that I can specify all conditions in one class and use is as per need instead of creating class for each combination

        • Rüdiger Herrmann says:

          Annotation member types are restricted and I don’t see how to specify something dynamic. Method references aren’t allowed yet either so that conditions could be written more compact.

          However, you can group conditions in an umbrella class and thus have them in one place, but still, each combination would require a class of its own.

          I think your best bet is to wait, as you said that currently there are few combinations. If the number of combination grows you can still extend the IgnoreCondition to allow for multiple conditions.

  11. Melanie says:

    Is this rule packaged into a maven project ?

    • Frank Appel says:

      Hi Melanie, I’m sorry but it isn’t. So, if you want to use it you’ll have to copy the code of the gist into your project code.