在 CI 上的 UI 测试不稳定

Question

更新：

VNC 进入构建机器（不做任何更改）以某种方式修复了这个......奇怪

---

语境

我们使用 Mac EC2 实例 CI 集群来运行 UI 测试。集群设置为每晚回收。这意味着新集群是在早上配置的。我们注意到测试在一天中变得越来越不稳定。

• 构建 312 是早上的第一个构建。
• 红色是失败的测试，橙色是静音测试，黄色是片状测试，灰色是未知状态
• 提交只是触发构建的空白自动提交。测试内容相同。
• 注意：Bluepill 配置为每台机器运行一个模拟器（无并发）

日志

片状测试或失败通常是由于“等待某些资源空闲”并达到超时，或某些网络请求完成。失败或不稳定的测试也有一个共同点，即它们通常与视频有关

这是几个示例日志

等待应用空闲

Timed out waiting for app to idle. 
The following idling resources are busy.
1. GREYAppStateTracker:
Waiting for network requests to finish. By default, EarlGrey tracks all network requests. To change this behavior, refer to GREYConfiguration.
<__NSCFLocalDataTask:0x7fcfcfd87450, URL:"https://v.myorgimg.com/videos/mc/hls/e7/01/ca/e701caebac12684056eb699f316fc411_360w.m3u8"> => Waiting for network requests to finish. By default, EarlGrey tracks all network requests. To change this behavior, refer to GREYConfiguration.
(
    0   MyorgDevelopmentEG2             0x000000010ebf3205 -[GREYAppStateTrackerObject setState:] + 69
    1   MyorgDevelopmentEG2             0x000000010ebf22b3 __133-[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDeallocationTracker:orExternalAppStateTrackerObject:]_block_invoke + 931
    2   MyorgDevelopmentEG2             0x000000010ebf18f0 -[GREYAppStateTracker grey_performBlockInCriticalSection:] + 224
    3   MyorgDevelopmentEG2             0x000000010ebf1e34 -[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDeallocationTracker:orExternalAppStateTrackerObject:] + 1076
    4   MyorgDevelopmentEG2             0x000000010ebf0ac6 -[GREYAppStateTracker trackState:forObject:] + 102
    5   MyorgDevelopmentEG2             0x000000010ebca867 -[__NSCFLocalDataTask_GREYApp grey_track] + 135
    6   MyorgDevelopmentEG2             0x000000010ebcac55 -[__NSCFLocalDataTask_GREYApp greyswizzled_resume] + 181
    7   MediaToolbox                        0x00007fff2b617285 figHttpRequestSetupNSURLSessionTask + 1380
    8   MediaToolbox                        0x00007fff2b616bb6 _FigHTTPRequestCreateWithNSURLSession + 2923
    9   MediaToolbox                        0x00007fff2b64aaaf figHTTPRequestSessionNSCreateHTTPRequest + 199
    10  MediaToolbox                        0x00007fff2b71f5b1 segPumpCreateHTTPRequest + 944
    11  MediaToolbox                        0x00007fff2b721de6 segPumpSendIndexFileRequest + 1703
    12  MediaToolbox                        0x00007fff2b7431d5 segPumpRequestIndexForStream + 753
    13  MediaToolbox                        0x00007fff2b7421ee segPumpSetAlternateForStream + 3145
    14  MediaToolbox                        0x00007fff2b73e9e5 segPumpSetCurrentAlternate + 9898
    15  MediaToolbox                        0x00007fff2b791d6a ProduceStreamingAssetProperty + 661
    16  MediaToolbox                        0x00007fff2b78e07c URLAssetPropertyWorkFunction + 872
    17  libdispatch.dylib                   0x00007fff201078df _dispatch_client_callout + 8
    18  libdispatch.dylib                   0x00007fff2010de15 _dispatch_lane_serial_drain + 715
    19  libdispatch.dylib                   0x00007fff2010e9c3 _dispatch_lane_invoke + 455
    20  libdispatch.dylib                   0x00007fff20117a8e _dispatch_root_queue_drain + 350
    21  libdispatch.dylib                   0x00007fff2011786c _dispatch_worker_thread + 222
    22  libsystem_pthread.dylib             0x00007fff60342950 _pthread_start + 224
    23  libsystem_pthread.dylib             0x00007fff6033e47b thread_start + 15
)

并等待网络请求完成

9. GREYAppStateTracker:
Waiting for network requests to finish. By default, EarlGrey tracks all network requests. To change this behavior, refer to GREYConfiguration.
Waiting for UIView's draw/layout pass to complete. A draw/layout pass normally completes in the next runloop drain.
<ASCollectionView: 0x7feca21ad000> => Waiting for UIView's draw/layout pass to complete. A draw/layout pass normally completes in the next runloop drain.
(
    0   MyorgDevelopmentEG2             0x000000010bb2e335 -[GREYAppStateTrackerObject setState:] + 69
    1   MyorgDevelopmentEG2             0x000000010bb2d3e3 __133-[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDeallocationTracker:orExternalAppStateTrackerObject:]_block_invoke + 931
    2   MyorgDevelopmentEG2             0x000000010bb2ca20 -[GREYAppStateTracker grey_performBlockInCriticalSection:] + 224
    3   MyorgDevelopmentEG2             0x000000010bb2cf64 -[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDeallocationTracker:orExternalAppStateTrackerObject:] + 1076
    4   MyorgDevelopmentEG2             0x000000010bb2bbf6 -[GREYAppStateTracker trackState:forObject:] + 102
    5   MyorgDevelopmentEG2             0x000000010bb02b0d -[UIView(GREYApp) greyswizzled_setNeedsLayout] + 93
    6   UIKitCore                           0x00007fff24ba5675 -[UIScrollView setNeedsLayout] + 79
    7   UIKitCore                           0x00007fff23dc0390 -[UICollectionView _updateAnimationDidStop:finished:context:] + 2236
    8   UIKitCore                           0x00007fff23dbf9e4 __102-[UICollectionView _updateWithItems:tentativelyForReordering:propertyAnimator:collectionViewAnimator:]_block_invoke.2110 + 77
    9   MyorgDevelopmentEG2             0x000000010bb2fdef __60-[GREYDispatchQueueTracker grey_dispatchAsyncCallWithBlock:]_block_invoke + 47
    10  libdispatch.dylib                   0x00007fff2010670d _dispatch_call_block_and_release + 12
    11  libdispatch.dylib                   0x00007fff201078df _dispatch_client_callout + 8
    12  libdispatch.dylib                   0x00007fff20114a27 _dispatch_main_queue_callback_4CF + 1045
    13  CoreFoundation                      0x00007fff203908f8 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 9
    14  CoreFoundation                      0x00007fff2038b169 __CFRunLoopRun + 2781
    15  CoreFoundation                      0x00007fff2038a1a7 CFRunLoopRunSpecific + 567
    16  GraphicsServices                    0x00007fff2b874d85 GSEventRunModal + 139
    17  UIKitCore                           0x00007fff246c14df -[UIApplication _run] + 912
    18  UIKitCore                           0x00007fff246c639c UIApplicationMain + 101
    19  MyorgDevelopmentEG2             0x0000000108f175de main + 190
    20  libdyld.dylib                       0x00007fff2025abbd start + 1
    21  ???                                 0x0000000000000005 0x0 + 5
)
<_ASDisplayLayer: 0x600007996580> => Waiting for UIView's draw/layout pass to complete. A draw/layout pass normally completes in the next runloop drain.

假设

这让我认为有一些孤立的子进程或守护进程仍然占有资源。或者没有正确清理工件。

我尝试过的事情

• 关闭，擦除，删除构建之间的所有模拟器（无帮助）

    xcrun simctl shutdown all 
    xcrun simctl erase all
    xcrun simctl delete all
    osascript -e 'tell application "iOS Simulator" to quit'
    osascript -e 'tell application "Simulator" to quit'

• 在构建之间终止 CoreSimulator 相关服务（无帮助）

    launchctl remove com.apple.CoreSimulator.CoreSimulatorService || true
    launchctl remove com.apple.CoreSimulator.SimLaunchHost-x86 || true
    launchctl remove com.apple.CoreSimulator.SimulatorTrampoline || true
    launchctl remove com.apple.CoreSimulator.SimLaunchHost-arm64 || true

• 在构建之间删除模拟器设备、日志和缓存。这实际上有点帮助。片状最初消失了，但在几次构建后很快又回来了，而机器仍在构建之间执行清洁......不知道为什么。

    echo "Delete all simulator devices, cache, and temp"
    rm -rf ~/Library/Developer/CoreSimulator

    echo "Delete Simulator Logs"
    rm -rf ~/Library/Logs/CoreSimulator

• 在 EarlGrey 上增加动画超时、交互超时
• 将请求挂起的 URL 列入黑名单。这确实通过忽略症状而不修复实际问题来提高测试套件的稳定性，所以我希望我可以避免这种情况。

还有其他人遇到同样的问题吗？感谢有人可以对此有所了解或提出尝试建议。

注意：我们使用Bluepill和EarlGrey2在 Buildkite CI 管道上运行 UI 测试（iOS 14.5、Xcode 12.5）。

Answer 1

您是否尝试过运行：

sudo systemsetup -setcomputersleep never