使用cucumber-jvm重新尝试失败的黄瓜测试

Question

我有一个Cucumber-JVM,JUnit,Selenium设置.我通过RunSmokeTests.java在Eclipse中使用JUnit来启动运行.我还设置了一个maven配置文件来从命令行运行测试,将来可能还有Jenkins.

当测试运行时,其中一些可能会失败,主要是由于应用程序花费的时间超过预期.然后我必须重新运行这些场景.目前我通过手动将@rerun标签附加到失败然后运行的标签来运行它们RunReruns.java,这类似于RunSmokeTest.java带@rerun标签.

随着自动化测试数量的增加,标记测试并开始运行并清除标签非常耗时.有没有使用Cucumber-JVM重新运行失败测试的自动化方法？

RunSmokeTests.java

package testGlueClasses;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@Cucumber.Options(features = "src/test/java", strict = true, format = {
        "html:target/CucumberReport", "json:target/JSON/Cucumber.json",
        "FrameworkCore.CustomTestReporter" }, tags = { "@SmokeTest" }, glue = {
        "FrameworkCore", "MyApp.Utils", "MyApp.StepDefinitions" })
public class RunSmokeTests {

} 

Maven片段:

    <profile>
        <id>smoke</id>
        <properties>
            <include.tests>
                **/RunSmokeTests.java
            </include.tests>
        </properties>
    </profile>

Answer 1

我提出了另一种解决方案,使用maven和黄瓜重新运行失败的测试.

1)使用a记录测试失败 RunNotifier

public class RerunningCucumber extends Cucumber {

    private final String className;

    @SuppressWarnings("rawtypes")
    public RerunningCucumber(Class clazz) throws InitializationError, IOException {
        super(clazz);
        className = clazz.getSimpleName();
    }


    @Override
    public void run(RunNotifier notifier) {
        notifier.addListener(new RunListener(){

            public void testFailure(Failure failure) throws Exception {

                Throwable error = failure.getException();
                if (error instanceof AssertionError){
                    //Nothing. This is a normal failure. Continue
                    return;
                }

                //No! A wild exception has appeared!
                //Let's run this test again.
                RerunningCucumber.addFile(className);
            }

        });
        super.run(notifier);
    }


    private static final String filename = "target/rerun.properties";
    private static final Set<String> addedClasses = new HashSet<String>();
    public static synchronized void addFile(String className) throws IOException{
        //First find the file

        if (addedClasses.contains(className)){
            return;
        }

        File file = new File(filename);
        if (!file.exists()){
            //Need to create the file
            PrintWriter writer = new PrintWriter(file, "UTF-8");
            writer.print("retryclasses=**/"+className+".class");
            writer.close();
        }
        else {
            PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter(file, true)));
            out.print(",**/"+className+".class");
            out.close();
        }

        addedClasses.add(className);
    }
}

2)使用自定义类作为黄瓜测试的跑步者.

这将运行测试,并且每当发生故障时,将失败的类输出到文件.诀窍是保持功能简短并创建大量测试类以避免重复测试.

@RunWith(RerunningCucumber.class)
@CucumberOptions(features = {"classpath:features/testFeature.feature}, format = {
        "html:target/cucumber-html-report/testFeature.html",
        "json:target/cucumber-json-report/testFeature.json"},
        tags = {"@testFeature"})

public class RunTestFeature {
}

3)Rerun向maven 添加配置文件.

这样做有三件事:1)它将失败的类加载到内存中,2)清除失败的类属性文件,3)仅重新运行从属性文件加载的失败的测试:

    <profile>
        <id>retry</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>properties-maven-plugin</artifactId>
                    <version>1.0-alpha-2</version>
                    <executions>
                        <!-- Associate the read-project-properties goal with the initialize 
                            phase, to read the properties file. -->
                        <execution>
                            <phase>pre-clean</phase>
                            <goals>
                                <goal>read-project-properties</goal>
                            </goals>
                            <configuration>
                                <files>
                                    <file>target/rerun.properties</file>
                                </files>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-clean-plugin</artifactId>
                    <version>2.6.1</version>
                    <configuration>
                        <filesets>
                            <fileset>
                                <directory>target</directory>
                                <includes>
                                    <include>rerun.properties</include>
                                </includes>
                            </fileset>
                        </filesets>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-antrun-plugin</artifactId>
                    <version>1.6</version>
                    <executions>
                        <execution>
                            <phase>compile</phase>
                            <goals>
                                <goal>run</goal>
                            </goals>
                            <configuration>
                                <target>
                                    <echo>Retrying the following classes: "${retryclasses}"</echo>
                                </target>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>2.17</version>
                    <configuration>
                        <includes>
                            <include>${retryclasses}</include>
                        </includes>
                        <testFailureIgnore>true</testFailureIgnore>
                    </configuration>
                    <executions>
                        <execution>
                            <phase>test</phase>
                            <goals>
                                <goal>test</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>
    </profile>

4)用法

首次试运行:

mvn clean test

下一个测试运行:

mvn clean test -Pretry
mvn clean test -Pretry
mvn clean test -Pretry
...

您可以根据需要重复多次,直到没有错误.

Answer 2

我手头没有可执行的示例，但您也可以在 jvm 上执行此操作。有一个RerunFormatter写入文本文件，列出失败场景的文件和行号：

@CucumberOptions(format = {"rerun:target/rerun.txt"})

您应该能够通过在该文件前面添加前缀来指定该文件作为另一个测试类的输入@：

@CucumberOptions(features = {"@target/rerun.txt"})