如何立即重新运行失败的JUnit测试？

Question

有没有办法让JUnit规则或类似的东西给每个失败的测试提供第二次机会,只需尝试再次运行它.

背景:我有一大堆用JUnit编写的Selenium2-WebDriver测试.由于非常激进的时间(点击后只有很短的等待时间),一些测试(100个中的1个,总是不同的测试)可能会失败,因为服务器有时响应速度稍慢.但我不能让等待时间太久以至于它肯定足够长,因为那时测试将永远持续下去.) - 所以我认为这个用例即使需要一秒钟,测试也是绿色的是可以接受的尝试.

当然,最好有3个中的2个(重复3次失败的测试,如果其中两个测试是正确的,则将它们视为正确),但这将是未来的改进.

Answer 1

您可以使用TestRule执行此操作.这将为您提供所需的灵活性.TestRule允许您在测试周围插入逻辑,因此您将实现重试循环:

public class RetryTest {
    public class Retry implements TestRule {
        private int retryCount;

        public Retry(int retryCount) {
            this.retryCount = retryCount;
        }

        public Statement apply(Statement base, Description description) {
            return statement(base, description);
        }

        private Statement statement(final Statement base, final Description description) {
            return new Statement() {
                @Override
                public void evaluate() throws Throwable {
                    Throwable caughtThrowable = null;

                    // implement retry logic here
                    for (int i = 0; i < retryCount; i++) {
                        try {
                            base.evaluate();
                            return;
                        } catch (Throwable t) {
                            caughtThrowable = t;
                            System.err.println(description.getDisplayName() + ": run " + (i+1) + " failed");
                        }
                    }
                    System.err.println(description.getDisplayName() + ": giving up after " + retryCount + " failures");
                    throw caughtThrowable;
                }
            };
        }
    }

    @Rule
    public Retry retry = new Retry(3);

    @Test
    public void test1() {
    }

    @Test
    public void test2() {
        Object o = null;
        o.equals("foo");
    }
}

a的核心TestRule是base.evaluate(),它调用你的测试方法.因此,围绕此调用,您将进行重试循环.如果在您的测试方法中抛出异常(断言失败实际上是一个AssertionError),那么测试失败,您将重试.

还有一件事可能有用.您可能只想将此重试逻辑应用于一组测试,在这种情况下,您可以在方法的特定注释的测试上方添加到Retry类中.Description包含该方法的注释列表.有关这方面的更多信息,请参阅我的答案如何在每个JUnit @Test方法之前单独运行一些代码,而不使用@RunWith和AOP？.

使用自定义TestRunner

这是CKuck的建议,你可以定义自己的Runner.您需要扩展BlockJUnit4ClassRunner并覆盖runChild().有关更多信息,请参阅我对如何在套件中定义JUnit方法规则的回答？.此答案详细说明了如何定义如何为Suite中的每个方法运行代码,您必须为其定义自己的Runner.

Answer 2

至于我编写自定义转轮更灵活的解决方案.上面发布的解决方案(带代码示例)有两个缺点:

1. 如果它在@BeforeClass阶段失败,它将不会重试测试;
2. 它计算测试的运行方式有点不同(当你有3次重试时,你将收到测试运行:4,成功1可能会令人困惑);

这就是为什么我更喜欢写自定义跑步者的方法.自定义运行程序的代码可能如下:

import org.junit.Ignore;
import org.junit.internal.AssumptionViolatedException;
import org.junit.internal.runners.model.EachTestNotifier;
import org.junit.runner.Description;
import org.junit.runner.notification.RunNotifier;
import org.junit.runner.notification.StoppedByUserException;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.Statement;


public class RetryRunner extends BlockJUnit4ClassRunner {

    private final int retryCount = 100;
    private int failedAttempts = 0;

    public RetryRunner(Class<?> klass) throws InitializationError {
        super(klass);
    }    


    @Override
    public void run(final RunNotifier notifier) {
        EachTestNotifier testNotifier = new EachTestNotifier(notifier,
                getDescription());
        Statement statement = classBlock(notifier);
        try {

            statement.evaluate();
        } catch (AssumptionViolatedException e) {
            testNotifier.fireTestIgnored();
        } catch (StoppedByUserException e) {
            throw e;
        } catch (Throwable e) {
            retry(testNotifier, statement, e);
        }
    }

    @Override
    protected void runChild(final FrameworkMethod method, RunNotifier notifier) {
        Description description = describeChild(method);
        if (method.getAnnotation(Ignore.class) != null) {
            notifier.fireTestIgnored(description);
        } else {
            runTestUnit(methodBlock(method), description, notifier);
        }
    }

    /**
     * Runs a {@link Statement} that represents a leaf (aka atomic) test.
     */
    protected final void runTestUnit(Statement statement, Description description,
            RunNotifier notifier) {
        EachTestNotifier eachNotifier = new EachTestNotifier(notifier, description);
        eachNotifier.fireTestStarted();
        try {
            statement.evaluate();
        } catch (AssumptionViolatedException e) {
            eachNotifier.addFailedAssumption(e);
        } catch (Throwable e) {
            retry(eachNotifier, statement, e);
        } finally {
            eachNotifier.fireTestFinished();
        }
    }

    public void retry(EachTestNotifier notifier, Statement statement, Throwable currentThrowable) {
        Throwable caughtThrowable = currentThrowable;
        while (retryCount > failedAttempts) {
            try {
                statement.evaluate();
                return;
            } catch (Throwable t) {
                failedAttempts++;
                caughtThrowable = t;
            }
        }
        notifier.addFailure(caughtThrowable);
    }
}

Answer 3

现在有一个更好的选择.如果您正在使用maven插件,例如:surfire或failsefe,则可以选择添加参数rerunFailingTestsCount SurFire Api.这些东西在以下票证中实现:Jira Ticket.在这种情况下,您不需要编写自定义代码和插件自动修改测试结果报告.
我看到这种方法只有一个缺点:如果某些测试失败,则在课前/课后阶段测试将不会重新运行.

Answer 4

建议的评论是根据这篇文章写的，并添加了一些内容。

在这里，如果您的 jUnit 项目中的某个测试用例得到“失败”或“错误”结果，则该测试用例将再重新运行一次。在这里我们总共设置了 3 次获得成功结果的机会。

所以，我们需要创建规则类和添加“@rule”通知到您的测试类。

如果您不想为每个测试类编写相同的“@Rule”通知，您可以将其添加到您的抽象 SetProperty 类（如果有）并从中扩展。

规则类：

import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;

public class RetryRule implements TestRule {
    private int retryCount;

    public RetryRule (int retryCount) {
        this.retryCount = retryCount;
    }

    public Statement apply(Statement base, Description description) {
        return statement(base, description);
    }

    private Statement statement(final Statement base, final Description description) {
        return new Statement() {
            @Override
            public void evaluate() throws Throwable {
                Throwable caughtThrowable = null;

                // implement retry logic here
                for (int i = 0; i < retryCount; i++) {
                    try {
                        base.evaluate();
                        return;
                    } catch (Throwable t) {
                        caughtThrowable = t;
                        //  System.out.println(": run " + (i+1) + " failed");
                        System.err.println(description.getDisplayName() + ": run " + (i + 1) + " failed.");
                    }
                }
                System.err.println(description.getDisplayName() + ": giving up after " + retryCount + " failures.");
                throw caughtThrowable;
            }
        };
    }
}

测试类：

import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.MatcherAssert.assertThat;

/**
 * Created by ONUR BASKIRT on 27.03.2016.
 */
public class RetryRuleTest {

    static WebDriver driver;
    final private String URL = "http://www.swtestacademy.com";

    @BeforeClass
    public static void setupTest(){
        driver = new FirefoxDriver();
    }

    //Add this notification to your Test Class 
    @Rule
    public RetryRule retryRule = new RetryRule(3);

    @Test
    public void getURLExample() {
        //Go to www.swtestacademy.com
        driver.get(URL);

        //Check title is correct
        assertThat(driver.getTitle(), is("WRONG TITLE"));
    }
}

Answer 5

你必须自己编写org.junit.runner.Runner并注释你的测试@RunWith(YourRunner.class).