当断言失败时继续Python的unittest

Question

编辑:切换到一个更好的例子,并澄清为什么这是一个真正的问题.

我想在Python中编写单元测试,在断言失败时继续执行,这样我就可以在单个测试中看到多个失败.例如:

class Car(object):
  def __init__(self, make, model):
    self.make = make
    self.model = make  # Copy and paste error: should be model.
    self.has_seats = True
    self.wheel_count = 3  # Typo: should be 4.

class CarTest(unittest.TestCase):
  def test_init(self):
    make = "Ford"
    model = "Model T"
    car = Car(make=make, model=model)
    self.assertEqual(car.make, make)
    self.assertEqual(car.model, model)  # Failure!
    self.assertTrue(car.has_seats)
    self.assertEqual(car.wheel_count, 4)  # Failure!

在这里,测试的目的是确保Car __init__正确设置其字段.我可以将它分解为四种方法(这通常是一个好主意),但在这种情况下,我认为将它作为测试单个概念的单个方法("对象被正确初始化")更具可读性.

如果我们假设这里最好不分解方法,那么我有一个新问题:我无法立即看到所有错误.当我修复model错误并重新运行测试时,会wheel_count出现错误.当我第一次运行测试时,它可以节省我看到两个错误的时间.

为了比较,Google的C++单元测试框架区分了非致命EXPECT_*断言和致命ASSERT_*断言:

断言成对出现,测试相同的东西但对当前函数有不同的影响.ASSERT_*版本在失败时会生成致命的故障,并中止当前的功能.EXPECT_*版本生成非致命故障,不会中止当前功能.通常EXPECT_*是首选,因为它们允许在测试中报告多个失败.但是,如果在有问题的断言失败时继续没有意义,则应使用ASSERT_*.

有没有办法EXPECT_*在Python中获得类似行为unittest？如果没有unittest,那么是否有另一个支持这种行为的Python单元测试框架？

---

顺便说一下,我很好奇有多少现实测试可能会从非致命断言中受益,所以我看了一些代码示例(编辑2014-08-19使用searchcode代替Google Code Search,RIP).在第一页中随机选择的10个结果中,所有结果都包含在同一测试方法中进行多个独立断言的测试.所有人都将受益于非致命的断言.

Answer 1

使用非致命断言的另一种方法是捕获断言异常并将异常存储在列表中.然后声明该列表是空的,作为tearDown的一部分.

import unittest

class Car(object):
  def __init__(self, make, model):
    self.make = make
    self.model = make  # Copy and paste error: should be model.
    self.has_seats = True
    self.wheel_count = 3  # Typo: should be 4.

class CarTest(unittest.TestCase):
  def setUp(self):
    self.verificationErrors = []

  def tearDown(self):
    self.assertEqual([], self.verificationErrors)

  def test_init(self):
    make = "Ford"
    model = "Model T"
    car = Car(make=make, model=model)
    try: self.assertEqual(car.make, make)
    except AssertionError, e: self.verificationErrors.append(str(e))
    try: self.assertEqual(car.model, model)  # Failure!
    except AssertionError, e: self.verificationErrors.append(str(e))
    try: self.assertTrue(car.has_seats)
    except AssertionError, e: self.verificationErrors.append(str(e))
    try: self.assertEqual(car.wheel_count, 4)  # Failure!
    except AssertionError, e: self.verificationErrors.append(str(e))

if __name__ == "__main__":
    unittest.main()

Answer 2

一个选项是立即将所有值断言为元组.

例如:

class CarTest(unittest.TestCase):
  def test_init(self):
    make = "Ford"
    model = "Model T"
    car = Car(make=make, model=model)
    self.assertEqual(
            (car.make, car.model, car.has_seats, car.wheel_count),
            (make, model, True, 4))

此测试的输出将是:

======================================================================
FAIL: test_init (test.CarTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\temp\py_mult_assert\test.py", line 17, in test_init
    (make, model, True, 4))
AssertionError: Tuples differ: ('Ford', 'Ford', True, 3) != ('Ford', 'Model T', True, 4)

First differing element 1:
Ford
Model T

- ('Ford', 'Ford', True, 3)
?           ^ -          ^

+ ('Ford', 'Model T', True, 4)
?           ^  ++++         ^

这表明模型和车轮计数都不正确.

Answer 3

从 Python 3.4 开始，您还可以使用子测试：

def test_init(self):
    make = "Ford"
    model = "Model T"
    car = Car(make=make, model=model)
    with self.subTest(msg='Car.make check'):
        self.assertEqual(car.make, make)
    with self.subTest(msg='Car.model check'):
        self.assertEqual(car.model, model)
    with self.subTest(msg='Car.has_seats check'):
        self.assertTrue(car.has_seats)
    with self.subTest(msg='Car.wheel_count check'):
        self.assertEqual(car.wheel_count, 4)

（msg参数用于更容易地确定哪个测试失败。）

输出：

======================================================================
FAIL: test_init (__main__.CarTest) [Car.model check]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 23, in test_init
    self.assertEqual(car.model, model)
AssertionError: 'Ford' != 'Model T'
- Ford
+ Model T


======================================================================
FAIL: test_init (__main__.CarTest) [Car.wheel_count check]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 27, in test_init
    self.assertEqual(car.wheel_count, 4)
AssertionError: 3 != 4

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=2)

Answer 4

您可能想要做的是派生,unittest.TestCase因为那是断言失败时抛出的类.您将不得不重新设计您TestCase的不投掷(可能会保留失败列表).重新构建东西可能会导致您必须解决的其他问题.例如,您可能最终需要派生TestSuite以进行更改以支持对您所做的更改TestCase.

Answer 5

在单个单元测试中具有多个断言被认为是反模式.单个单元测试预计只测试一件事.也许你的测试太多了.考虑将此测试分成多个测试.这样您就可以正确命名每个测试.

但有时候,可以同时检查多个内容.例如,当您断言同一对象的属性时.在这种情况下,您实际上断言该对象是否正确.一种方法是编写一个自定义帮助器方法,该方法知道如何在该对象上断言.您可以编写该方法,使其显示所有失败的属性,或者例如在断言失败时显示预期对象的完整状态和实际对象的完整状态.

Answer 6

PyPI 中有一个名为的软断言包softest可以满足您的要求。它的工作原理是收集故障、组合异常和堆栈跟踪数据，并将其全部报告为常规unittest输出的一部分。

例如，这段代码：

import softest

class ExampleTest(softest.TestCase):
    def test_example(self):
        # be sure to pass the assert method object, not a call to it
        self.soft_assert(self.assertEqual, 'Worf', 'wharf', 'Klingon is not ship receptacle')
        # self.soft_assert(self.assertEqual('Worf', 'wharf', 'Klingon is not ship receptacle')) # will not work as desired
        self.soft_assert(self.assertTrue, True)
        self.soft_assert(self.assertTrue, False)

        self.assert_all()

if __name__ == '__main__':
    softest.main()

...产生以下控制台输出：

======================================================================
FAIL: "test_example" (ExampleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\...\softest_test.py", line 14, in test_example
    self.assert_all()
  File "C:\...\softest\case.py", line 138, in assert_all
    self.fail(''.join(failure_output))
AssertionError: ++++ soft assert failure details follow below ++++

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The following 2 failures were found in "test_example" (ExampleTest):
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Failure 1 ("test_example" method)
+--------------------------------------------------------------------+
Traceback (most recent call last):
  File "C:\...\softest_test.py", line 10, in test_example
    self.soft_assert(self.assertEqual, 'Worf', 'wharf', 'Klingon is not ship receptacle')
  File "C:\...\softest\case.py", line 84, in soft_assert
    assert_method(*arguments, **keywords)
  File "C:\...\Python\Python36-32\lib\unittest\case.py", line 829, in assertEqual
    assertion_func(first, second, msg=msg)
  File "C:\...\Python\Python36-32\lib\unittest\case.py", line 1203, in assertMultiLineEqual
    self.fail(self._formatMessage(msg, standardMsg))
  File "C:\...\Python\Python36-32\lib\unittest\case.py", line 670, in fail
    raise self.failureException(msg)
AssertionError: 'Worf' != 'wharf'
- Worf
+ wharf
 : Klingon is not ship receptacle

+--------------------------------------------------------------------+
Failure 2 ("test_example" method)
+--------------------------------------------------------------------+
Traceback (most recent call last):
  File "C:\...\softest_test.py", line 12, in test_example
    self.soft_assert(self.assertTrue, False)
  File "C:\...\softest\case.py", line 84, in soft_assert
    assert_method(*arguments, **keywords)
  File "C:\...\Python\Python36-32\lib\unittest\case.py", line 682, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1)

注意：我创建并维护softest。

Answer 7

每个断言都用一个单独的方法.

class MathTest(unittest.TestCase):
  def test_addition1(self):
    self.assertEqual(1 + 0, 1)

  def test_addition2(self):
    self.assertEqual(1 + 1, 3)

  def test_addition3(self):
    self.assertEqual(1 + (-1), 0)

  def test_addition4(self):
    self.assertEqaul(-1 + (-1), -1)