Flutter 黄金图像测试 diff 阈值

Question

我正在利用黄金图像测试来测试颤动中的简单无状态小部件。每隔几次测试运行，我都会将以下错误打印到控制台：

\n

\xe2\x95\x90\xe2\x95\x90\xe2\x95\xa1 EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK \xe2\x95\x9e\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\xe2\x95\x90\nThe following TestFailure object was thrown running a test (but after the test had completed):\n  Expected: one widget whose rasterized image matches golden image\n"goldens/mywidget.png"\n  Actual: _WidgetTypeFinder:<zero widgets with type "MyWidget" (ignoring offstage widgets)>\n   Which: Golden "goldens/mywidget.png": Pixel test failed, 0.04% diff detected.\n          Failure feedback can be found at\n/home/cirrus/project/test/widgets/failures\n\nWhen the exception was thrown, this was the stack:\n

\n

这令人愤怒有两个原因：

\n

\n
1. 没有任何变化，差异似乎是子像素渲染差异。这种情况不应该发生，因此黄金测试变得毫无用处。
\n
2. 即使出现上述消息，测试也会通过。所以 flutter 认为有一个例外，但甚至没有通过它的测试。
\n

\n

有没有办法设置差异阈值，以便忽略差异低于 5% 的测试，以避免我的测试输出过于冗长和嘈杂？

\n

值得注意的是，这个问题并不局限于我们的 CI 服务器，它也发生在我们的本地计算机上，因此同一台计算机会根据一天中的不同时间以不同的方式呈现测试，这似乎很奇怪。

\n

Answer 1

我遇到了同样的问题，并正在寻找类似的解决方案。在讨论了我创建的Github 问题后，我学会了一种设置阈值的方法。

首先，在测试文件夹中的某个位置创建一个文件（我将其放在test/utils），例如local_file_comparator_with_threshold.dart：

    import 'dart:typed_data';
    
    import 'package:flutter/foundation.dart';
    import 'package:flutter_test/flutter_test.dart';
    
    /// Works just like [LocalFileComparator] but includes a [threshold] that, when
    /// exceeded, marks the test as a failure.
    class LocalFileComparatorWithThreshold extends LocalFileComparator {
      /// Threshold above which tests will be marked as failing.
      /// Ranges from 0 to 1, both inclusive.
      final double threshold;
    
      LocalFileComparatorWithThreshold(Uri testFile, this.threshold)
          : assert(threshold >= 0 && threshold <= 1),
            super(testFile);
    
      /// Copy of [LocalFileComparator]'s [compare] method, except for the fact that
      /// it checks if the [ComparisonResult.diffPercent] is not greater than
      /// [threshold] to decide whether this test is successful or a failure.
      @override
      Future<bool> compare(Uint8List imageBytes, Uri golden) async {
        final result = await GoldenFileComparator.compareLists(
          imageBytes,
          await getGoldenBytes(golden),
        );
    
        if (!result.passed && result.diffPercent <= threshold) {
          debugPrint(
            'A difference of ${result.diffPercent * 100}% was found, but it is '
            'acceptable since it is not greater than the threshold of '
            '${threshold * 100}%',
          );
    
          return true;
        }
    
        if (!result.passed) {
          final error = await generateFailureOutput(result, golden, basedir);
          throw FlutterError(error);
        }
        return result.passed;
      }
    }

然后，您可以覆盖您的测试配置flutter_test_config.dart（您需要放入项目的test文件夹中，如果还没有，请创建一个）：

import 'dart:async';

import 'package:flutter_test/flutter_test.dart';

import 'utils/local_file_comparator_with_threshold.dart';

/// Customise your threshold here
/// For example, the error threshold here is 0.5%
/// Golden tests will pass if the pixel difference is equal to or below 0.5%
const _kGoldenTestsThreshold = 0.5 / 100;

Future<void> testExecutable(FutureOr<void> Function() testMain) async {
  if (goldenFileComparator is LocalFileComparator) {
    final testUrl = (goldenFileComparator as LocalFileComparator).basedir;

    goldenFileComparator = LocalFileComparatorWithThreshold(
      // flutter_test's LocalFileComparator expects the test's URI to be passed
      // as an argument, but it only uses it to parse the baseDir in order to
      // obtain the directory where the golden tests will be placed.
      // As such, we use the default `testUrl`, which is only the `baseDir` and
      // append a generically named `test.dart` so that the `baseDir` is
      // properly extracted.
      Uri.parse('$testUrl/test.dart'),
      _kGoldenTestsThreshold,
    );
  } else {
    throw Exception(
      'Expected `goldenFileComparator` to be of type `LocalFileComparator`, '
      'but it is of type `${goldenFileComparator.runtimeType}`',
    );
  }

  await testMain();
}

完成此设置后，每当您在项目中运行黄金测试时，当像素差异小于或等于您在 中设置的阈值时，它们就会通过flutter_test_config.dart。