Azu*_*ron 8 scala maven scalatest apache-spark
我正在尝试使用Maven上的scalatest和spark-testing -base进行集成测试Spark.Spark作业读入CSV文件,验证结果,并将数据插入数据库.我试图通过放入已知格式的文件并查看它们是否以及如何失败来测试验证.这个特殊的测试只是确保验证通过.不幸的是,scalatest无法找到我的测试.
相关的插件:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<skipTests>true</skipTests>
</configuration>
</plugin>
<!-- enable scalatest -->
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<version>1.0</version>
<configuration>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<wildcardSuites>com.cainc.data.etl.schema.proficiency</wildcardSuites>
</configuration>
<executions>
<execution>
<id>test</id>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
Run Code Online (Sandbox Code Playgroud)
这是测试类:
class ProficiencySchemaITest extends FlatSpec with Matchers with SharedSparkContext with BeforeAndAfter {
private var schemaStrategy: SchemaStrategy = _
private var dataReader: DataFrameReader = _
before {
val sqlContext = new SQLContext(sc)
import sqlContext._
import sqlContext.implicits._
val dataInReader = sqlContext.read.format("com.databricks.spark.csv")
.option("header", "true")
.option("nullValue", "")
schemaStrategy = SchemaStrategyChooser("dim_state_test_proficiency")
dataReader = schemaStrategy.applySchema(dataInReader)
}
"Proficiency Validation" should "pass with the CSV file proficiency-valid.csv" in {
val dataIn = dataReader.load("src/test/resources/proficiency-valid.csv")
val valid: Try[DataFrame] = Try(schemaStrategy.validateCsv(dataIn))
valid match {
case Success(v) => ()
case Failure(e) => fail("Validation failed on what should have been a clean file: ", e)
}
}
}
Run Code Online (Sandbox Code Playgroud)
当我运行时mvn test,它找不到任何测试并输出此消息:
[INFO] --- scalatest-maven-plugin:1.0:test (test) @ load-csv-into-db ---
[36mDiscovery starting.[0m
[36mDiscovery completed in 54 milliseconds.[0m
[36mRun starting. Expected test count is: 0[0m
[32mDiscoverySuite:[0m
[36mRun completed in 133 milliseconds.[0m
[36mTotal number of tests run: 0[0m
[36mSuites: completed 1, aborted 0[0m
[36mTests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0[0m
[33mNo tests were executed.[0m
Run Code Online (Sandbox Code Playgroud)
更新
使用:
<suites>com.cainc.data.etl.schema.proficiency.ProficiencySchemaITest</suites>
Run Code Online (Sandbox Code Playgroud)
代替:
<wildcardSuites>com.cainc.data.etl.schema.proficiency</wildcardSuites>
Run Code Online (Sandbox Code Playgroud)
我可以让一个Test运行.显然,这并不理想.它可能会破坏wildcardSuites; 我打算在GitHub上打开一张票,看看会发生什么.
| 归档时间: |
|
| 查看次数: |
2348 次 |
| 最近记录: |