使用Cabal运行时,Haskell QuickCheck测试无法正常运行

Question

我正在开发的Cabal包中没有任何意义,我将问题归结为以下示例:

我有以下简单的测试模块:

module Main where
import Test.QuickCheck (quickCheck)
main = quickCheck False

以下Cabal文件位于同一目录中:

name:                project
version:             0.1.0.0
cabal-version:       >= 1.10
build-type:          Simple

executable project
  main-is:             Main.hs
  build-depends:       base, QuickCheck
  default-language:    Haskell2010

test-suite project-test
  type:                exitcode-stdio-1.0
  main-is:             Main.hs
  build-depends:       base, QuickCheck
  default-language:    Haskell2010

目录中唯一的其他文件是dist(由Cabal在构建时创建)和cabal沙箱文件.可执行文件和测试套件都引用Main.hs,因此我希望在运行"cabal run"时获得与运行"cabal test"时相同的测试结果.然而,显然情况并非如此.

"cabal run"给出:

Preprocessing executable 'project' for project-0.1.0.0...
Running project...
*** Failed! Falsifiable (after 1 test):

这是有道理的,因为属性测试'quickCheck False'应该失败.正如预期的那样,这与我在ghci中运行main时得到的结果相同.

然而,"阴谋测试"给出了:

Test suite project-test: RUNNING...
Test suite project-test: PASS
Test suite logged to: dist/test/project-0.1.0.0-project-test.log
1 of 1 test suites (1 of 1 test cases) passed.

为什么"cabal测试"通过测试用例,而"cabal run"按预期失败？

Answer 1

quickCheck不退出程序,因此,它不设置退出代码.毕竟,您可以进行多项quickCheck测试,这些测试应该相互独立:

main = do
    quickCheck prop1 -- should this exit if the test fails?
    quickCheck prop2 -- should this exit if the test fails?
    quickCheck prop3 -- should this exit if the test fails?

但是,如果你a)在你的一个测试没有通过时退出,或者b)记住是否没有通过一个测试,然后使用正确的代码退出,你可以轻松解决这个问题.

仅使用QuickCheck,但不使用其他库

一旦测试失败,退出失败

对于这一点,你只需使用quickCheckResult,您可以查询isSuccess从Test.QuickCheck.Test.如果要使用当前 quickCheck定义,可以使用限定包含来与特殊实现交换默认实现:

import           Control.Monad          (when)    
import           System.Exit            (exitFailure)
import           Test.QuickCheck hiding (quickCheck, quickCheckWith)
import qualified Test.QuickCheck.Test as Q

quickCheckWith :: Testable prop => Args -> prop -> IO ()
quickCheckWith args p = do
      success <- fmap Q.isSuccess $ quickCheckWithResult args p 
      when (not success) $ exitFailure -- exit if the result is a failure

quickCheck :: Testable prop => prop -> IO ()
quickCheck p = quickCheckWith stdArgs p

你可能应该使用另一个名字,特别是如果其他人在同一个项目上工作.checkOrExit这似乎是合情合理的.

返回失败代码但运行所有测试

这基本上是相同的,但运行所有测试.您必须再次使用quickCheckResult或quickCheckWithResult:

import           Control.Monad          (when)    
import           System.Exit            (exitFailure)
import           Test.QuickCheck        (quickCheckResult)
import           Test.QuickCheck.Test   (isSuccess)

main :: IO ()
main = do
  let tests = [ quickCheckResult prop1
              , quickCheckResult prop2
              , quickCheckResult prop3
              , quickCheckResult prop4
              ]
  success <- fmap (all isSuccess) . sequence $ tests
  when (not success) $ exitFailure

使用其他测试库

虽然quickCheck非常适合进行财产检查,但它并不能提供完整的测试框架.这就是其他完整的框架,如tasty或hspec派上用场.他们可以相应地Testable a检查并检查QuickCheck的结果.使用示例hspec:

module Main where
import Test.Hspec
import Test.QuickCheck (property)

main = hspec $ do
  describe "<method you would like to test>" $ 
    it "<property/assumption you would like to test>" $ 
       property $ False -- quickCheck

这给出了(更详细)输出

<method you would like to test>
  - <property/assumption you would like to test> FAILED [1]

1) <method you would like to test> <property/assumption you would like to test>
Falsifiable (after 1 test):
   [remark: the used values would be here, but `False` is a boolean]

Randomized with seed 2077617428

Finished in 0.0019 seconds
1 example, 1 failure

此外,它会以错误代码退出,因此您cabal test将正确识别此失败的测试.这些测试框架还具有其他功能,超出了本答案的范围.

TL; DR

QuickCheck不会退出您的程序,但Cabal 只检查退出代码以确定测试是否已通过.由于任何定期结束的Haskell程序返回零(也就是没有错误),您需要使用exitFailure(或类似的东西),或者exitFailure在幕后使用的框架.

请注意,这仅适用于测试type: exitcode-stdio-1.0.