KIF:如何自动运行/压力测试iOS应用程序以查找罕见UI错误的原因？

Question

注意:我为标题添加了kif,仅用于搜索索引目的,考虑到大部分答案都是为了讨论它

我正在寻找类似于selenium for iOS的东西,基本上是一个测试自动化/单元测试框架,它可以多次运行某个UI场景,直到它崩溃,这将帮助我缩小很少发生的UI错误的原因并随机.

(顺便说一句,我已经NSLogged了数据源/表交互的每一行代码,并花了几个小时分析潜在的原因..但没有找到任何结论..再次这个bug很少发生).

我查看了iOS中的一些单元测试框架,但它们看起来很多.我不确定要选哪个.此外,我对selenium的引用基于猜想,因为我曾与过去在大型Web项目中使用Selenium的QA人员合作过(我假设iOS必须有类似的东西).

现在我是一个从事iOS项目的单人团队,我将不得不戴上QA帽子来解决这个问题.

我遇到了一个经典的错误,当UITableView中插入的实际行数与数据源委托返回的行数之间存在差异时会发生这种错误.这是错误消息:

*** Assertion failure in -[UITableView
 _endCellAnimationsWithContext:] Exception in insertRows: Invalid
 update: invalid number of rows in section 0.

The number of rows contained in an existing section after the update (2) must be equal to
 the number of rows contained in that section before the update (2),
 plus or minus the number of rows inserted or deleted from that section
 (1 inserted, 0 deleted) and plus or minus the number of rows moved
 into or out of that section (0 moved in, 0 moved out).

我点击一个UITableViewCell将我带入另一个UITableView.有时它有效

有时(很少)它没有(有上述错误):

Answer 1

更新： ..我在分隔符后的底部添加了有关 KIF 2.0 的示例代码..对于那些对 KIF 比我面临的具体问题更感兴趣的人：

经过一些研究和实验......我将我的选择范围缩小到两个测试自动化库： Frank和KIF。我最终决定借用KIFCucumber的Gherkin语法来描述我的单元测试。

KIF我选择（而不是Frank）的原因是它KIF是 100% 基于 obj-c，而不是像 那样使用 ruby Frank​​。因此设置更简单，并且更适合我狭窄的测试用例要求。话虽这么说，我承认Frank如果我的应用程序更复杂（即使用来自多个服务器的输入等）会更有用。您可以观看这个精彩演示的最后一个季度，以了解有关 KIF、Frank 和其他自动化测试框架（包括 Apple 自己的UI Automation）的优缺点的更多信息。

使用 KIF 后，我发现了导致上述错误的 bug，并且我可以 100% 地使用 KIF 重现它！这种情况很少发生的原因是因为只有当我非常快地点击屏幕时才会发生这种情况..并且由于 KIF 自动化了这些步骤..它以令人难以置信的快速度完成这些步骤..这暴露了该错误:)。

因此，以下是我用于测试的代码示例。这只是为了让您快速了解 KIF（和 Gherkin）可以为您做什么：

在一个文件中，我指定要运行的场景：

- (void)initializeScenarios;
{
    [self addScenario:[KIFTestScenario scenarioToCompleteSignInAndLoadInbox]];
    [self addScenario:[KIFTestScenario scenarioToFillAttachmentsWithData]];
    [self addScenario:[KIFTestScenario scenarioToViewAndLoadFileBucket]];
    [self addScenario:[KIFTestScenario scenarioToViewAndLoadFileBucketSubView]];
}

每个场景都映射到步骤（要了解更多有关 Gherkin 语法和基于测试驱动程序开发的行为驱动开发的信息，我强烈建议阅读这本关于Cucumber的优秀书籍）：

/* @given the application is at a fresh state
   @and   the user already has an imap email account with a valid username/pwd

   @then  the user can successfully log in
   @and   the inbox view will be loaded
   @and   the inbox will get loaded with the latest batch of emails in the user inbox
 */
+ (id)scenarioToCompleteSignInAndLoadInbox
{
    KIFTestScenario *scenario = 
      [KIFTestScenario scenarioWithDescription:@"Test that a user 
                                                 can successfully log in."];
    [scenario addStepsFromArray:[KIFTestStep stepsCompleteSignInAndLoadInbox]];

    return scenario;
}


/* @given that the user is already signed in
   @and   the user has already downloaded their folders 

   @then  the user can click on the folders view
   @and   the user can click on the 'attachments' remote folder
   @and   the latest batch from the 'attachments' remote folder will download
 */
+ (id)scenarioToFillAttachmentsWithData {
    KIFTestScenario* scenario = 
      [KIFTestScenario scenarioWithDescription:@"Test that we can view the 
                                                 attachments folder and fill 
                                                 it with data."];
    [scenario addStepsFromArray:[KIFTestStep stepsToFillAttachmentsWithData]];
    return scenario;

}

/* @given that the user is already signed in
   @and   the user has already downloaded their folders
   @and   the user has already downloaded attachments

   @then  the user can click on inbox menu button
   @and   the user can click on folder list menu button
   @and   the user can click on the file bucket icon (on the account list view)
   @and   the data for the file bucket is fetched from the dbase
   @and   the file bucket view displayes the attachments
 */
+ (id)scenarioToViewAndLoadFileBucket {
    KIFTestScenario *scenario = 
       [KIFTestScenario scenarioWithDescription:@"Test that a user can successfully 
                                                  view and load 
                                                  file bucket parent view"];
    [scenario addStepsFromArray:[KIFTestStep stepsToViewAndLoadFileBucketPage]];

    return scenario;
}

/* @given that the user is already signed in
   @and   the user has already downloaded their folders
   @and   the user has already downloaded attachments
   @and   the user has already opened file bucket view 

   @then  the user can click on a random row in the file bucket view table
   @and   the subview will retrieve data from the dbase pertaining to that row
   @and   the subview will display the data in the uitableview
 */
+ (id)scenarioToViewAndLoadFileBucketSubView {
    KIFTestScenario *scenario = 
       [KIFTestScenario scenarioWithDescription:@"Test that a user can successfully
                                                  view and load filet
                                                  bucket sub view"];
    [scenario addStepsFromArray:[KIFTestStep stepsToViewAndLoadFileBucketSubPage]];
    return scenario;   
}

并使用 KIF 的 UI 自动化方法定义步骤（这只是一个示例）：

// this step assumes there is an attachment folder that contains emails with attachments
+ (NSArray *)stepsToFillAttachmentsWithData {

    NSMutableArray* steps = [@[] mutableCopy];

    [steps addObject:
        [KIFTestStep stepToTapViewWithAccessibilityLabel:@"InboxMenuButton"]];

    NSIndexPath* indexPath = 
        [NSIndexPath indexPathForRow:remoteAttachmentFolderNumber inSection:0];
    KIFTestStep* tapAttachmentRowStep = 
        [KIFTestStep stepToTapRowInTableViewWithAccessibilityLabel:
                                     @"attachments" atIndexPath:indexPath];

    [steps addObject:[KIFTestStep stepToWaitForNotificationName:
         (NSString *)kBeganSyncingOlderEmails object:nil           
                          whileExecutingStep:tapAttachmentRowStep]];

    [steps addObject:tapAttachmentRowStep];

    [steps addObject:
        [KIFTestStep stepToWaitForViewWithAccessibilityLabel:@"attachments"]];

    KIFTestStep *fillingInboxStep = 
        [KIFTestStep stepToWaitForNotificationName:
                                 (NSString *)kOldMailBatchDelivered object:nil];

    [fillingInboxStep setTimeout:kSpecialTimeoutForLongTests];
    [steps addObject:fillingInboxStep];

    return steps;
}

KIF 2.0 示例代码： KIF 2.0 使用 Xcode 5 的全新测试导航器.. 这比 KIF 1.0 所做的有了巨大的改进.. 现在您的测试感觉比过去更加有机和自然..（即它是真实的）时间..而不是创建将来运行的场景等）..您甚至可以使用播放按钮等来测试每个场景..您应该尝试一下。

以下是一些示例（再次使用小黄瓜语法）：

#import <KIF/KIF.h>
#import "KIFUITestActor+EXAdditions.h"
#import "KIFUITestActor+UserRegistration.h"

@interface LoginTests : KIFTestCase

@end
@implementation LoginTests

- (void)testReset {
    [tester flushDbase];
    [tester reset];
}

/* @given that the app is in a fresh clean state
 @and   that no one has ever registered with the server

 @then  the user can register their themselves with the server
 @and   immediately start with the rider's map
 @and   their location on the map shows
 */

- (void)testRegistration
{
    [tester flushDbase];
    [tester reset];
    [tester singleUserRegistration];
    [tester showUserCurrentLocationOnMap];
}

/* @given that the user has already registered with the server
   @and the user is not currently logged in

 @then  the user can login using their user name and password
 @and   immediately start with the rider's map
 @and   their location on the map shows
 */
- (void)testSuccessfulLogin
{
    [tester reset];
    [tester login];
    [tester showUserCurrentLocationOnMap];
}

/* @given that the user has already registered 
   @and that the user is already logged in before app launch

 @then the user starts on the map view with the location visible
 @and the button prompts them to set pick up location
 */
- (void)testStartOfApplication {
    [tester showUserCurrentLocationOnMap];
    [tester showsPickUpButton];
}
@end

这是类别文件中一些测试用例的实现：

- (void)reset
{
    [self runBlock:^KIFTestStepResult(NSError **error) {
        BOOL successfulReset = YES;

        // Do the actual reset for your app. Set successfulReset = NO if it fails.
        AppDelegate* appDelegate = [[UIApplication sharedApplication] delegate];
        [appDelegate resetApp];

        KIFTestCondition(successfulReset, error, @"Failed to reset some part of the application.");

        return KIFTestStepResultSuccess;
    }];
}

- (void)flushDbase {
    [self runBlock:^KIFTestStepResult(NSError **error){
        NSURL *url = [NSURL URLWithString:@"http://randomdomain.com/flush_db"];
        NSURLRequest *request = [NSURLRequest requestWithURL:url];
        NSError *connectionError = nil;

        BOOL databaseFlushSucceeded = YES;

        NSURLResponse *response;
        NSData *resultData = [NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&connectionError];
        if (!resultData) {
            databaseFlushSucceeded = NO;
            KIFTestCondition(databaseFlushSucceeded, error, @"failed to connect to server!");
        }

        if (connectionError) {
            databaseFlushSucceeded = NO;
            KIFTestCondition(databaseFlushSucceeded, error, [NSString stringWithFormat:@"connection failed. Error: %@", [connectionError localizedDescription]]);
        }

        return KIFTestStepResultSuccess;
    }];
}


- (void)navigateToLoginPage
{
    [self tapViewWithAccessibilityLabel:@"login email"];
}

- (void)returnToLoggedOutHomeScreen
{
    [self tapViewWithAccessibilityLabel:@"Logout"];
    [self tapViewWithAccessibilityLabel:@"Logout"]; // Dismiss alert.
}