为什么AVSampleBufferDisplayLayer停止显示从AVCaptureVideoDataOutput的委托获取的CMSampleBuffers？

Question

我想用AVSampleBufferDisplayLayer显示一些CMSampleBuffer，但是在显示第一个样本后冻结。

我从AVCaptureVideoDataOutputSampleBuffer委托获取了samplebuffers：

-(void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    CFRetain(sampleBuffer);
    [self imageToBuffer:sampleBuffer];
    CFRelease(sampleBuffer);
}

将它们放入向量

-(void) imageToBuffer: (CMSampleBufferRef )source{
//buffers is defined as: std::vector<CMSampleBufferRef> buffers;
        CMSampleBufferRef newRef;
        CMSampleBufferCreateCopy(kCFAllocatorDefault, source, &newRef);
        buffers.push_back(newRef);
}

然后尝试通过AVSampleBufferDisplayLayer（在另一个ViewController中）显示它们

AVSampleBufferDisplayLayer * displayLayer = [[AVSampleBufferDisplayLayer alloc] init];

    displayLayer.bounds = self.view.bounds;
    displayLayer.position = CGPointMake(CGRectGetMidX(self.displayOnMe.bounds), CGRectGetMidY(self.displayOnMe.bounds));
    displayLayer.videoGravity = AVLayerVideoGravityResizeAspectFill;
    displayLayer.backgroundColor = [[UIColor greenColor] CGColor];

    [self.view.layer addSublayer:displayLayer];
    self.view.autoresizingMask = UIViewAutoresizingFlexibleWidth | UIViewAutoresizingFlexibleHeight;

    dispatch_queue_t queue = dispatch_queue_create("My queue", DISPATCH_QUEUE_SERIAL);
    [displayLayer setNeedsDisplay];
    [displayLayer requestMediaDataWhenReadyOnQueue:queue
                                        usingBlock:^{
                                            while ([displayLayer isReadyForMoreMediaData]) {

                                                if (samplesKey < buffers.size()) {
                                                    CMSampleBufferRef buf = buffers[samplesKey];
                                                    [displayLayer enqueueSampleBuffer:buffers[samplesKey]];
                                                    samplesKey++;

                                                }else
                                                {
                                                    [displayLayer stopRequestingMediaData];
                                                    break;
                                                }
                                            }

                                        }];

但它显示第一个样本随后冻结，并且不执行任何操作。

我的视频数据输出设置如下：

//set up our output
self.videoDataOutput = [[AVCaptureVideoDataOutput alloc] init];
dispatch_queue_t queue = dispatch_queue_create("VideoQueue", DISPATCH_QUEUE_SERIAL);
[_videoDataOutput setSampleBufferDelegate:self queue:queue];
[_videoDataOutput setVideoSettings:[NSDictionary dictionaryWithObjectsAndKeys:
                                                [NSNumber numberWithInt:kCVPixelFormatType_32BGRA],(id)kCVPixelBufferPixelFormatTypeKey,
                                                nil]]; 

Answer 1

我在同一上下文中遇到了这个问题，试图从 AVCaptureVideoDataOutput 获取输出并将其显示在 AVSampleDisplay 层中。

如果您的帧按显示顺序出现，则修复非常简单，只需在 CMSampleBufferRef 上设置立即显示标志即可。

获取委托返回的样本缓冲区，然后...

CFArrayRef attachments = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, YES);
CFMutableDictionaryRef dict = (CFMutableDictionaryRef)CFArrayGetValueAtIndex(attachments, 0);

CFDictionarySetValue(dict, kCMSampleAttachmentKey_DisplayImmediately, kCFBooleanTrue);

如果您的帧按编码器顺序（而不是显示顺序）出现，则 CMSampleBuffer 上的时间戳需要偏零并重新标记，以便第一个帧时间戳等于时间 0。

 double pts = CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(sampleBuffer));

 // ptsStart is equal to the first frames presentationTimeStamp so playback starts from time 0.
 CMTime presentationTimeStamp = CMTimeMake((pts-ptsStart)*1000000,1000000);

 CMSampleBufferSetOutputPresentationTimeStamp(sampleBuffer, presentationTimeStamp);

更新：

当我使用零偏差方法时，我遇到了一些视频仍然无法流畅播放的情况，我进行了进一步调查。正确答案似乎是使用您打算播放的第一帧中的 PTS 。

我的答案在这里，但我也会在这里发布。

设置 AVSampleBufferDisplayLayer 渲染样本缓冲区的速率

时基需要设置为您要解码的第一帧的演示时间戳 (pts)。我通过从所有后续 pts 中减去初始 pts 并将时基设置为 0，将第一帧的 pts 索引为 0。无论出于何种原因，这不适用于某些视频。

你想要这样的东西（在调用解码之前调用）：

CMTimebaseRef controlTimebase;
CMTimebaseCreateWithMasterClock( CFAllocatorGetDefault(), CMClockGetHostTimeClock(), &controlTimebase );

displayLayer.controlTimebase = controlTimebase;

// Set the timebase to the initial pts here
CMTimebaseSetTime(displayLayer.controlTimebase, CMTimeMake(ptsInitial, 1));
CMTimebaseSetRate(displayLayer.controlTimebase, 1.0);

为 CMSampleBuffer 设置 PTS...

CMSampleBufferSetOutputPresentationTimeStamp(sampleBuffer, presentationTimeStamp);

也许确保没有设置立即显示....

CFDictionarySetValue(dict, kCMSampleAttachmentKey_DisplayImmediately, kCFBooleanFalse);

这在 WWDC 2014 Session 513 中有非常简要的介绍。