从CMBlockBuffer中提取h264

Question

我正在使用Apple VideoTool Box(iOS)来压缩设备相机捕获的原始帧.

我的回调是使用包含CMBlockBuffer的CMSampleBufferRef对象调用的.

CMBlockBuffer对象包含H264基本流,但我没有找到任何方法来获取指向基本流的指针.

当我打印到控制台时,我得到了CMSampleBufferRef对象:

(lldb) po blockBufferRef
CMBlockBuffer 0x1701193e0 totalDataLength: 4264 retainCount: 1 allocator: 0x1957c2c80 subBlockCapacity: 2
 [0] 4264 bytes @ offset 128 Buffer Reference:
    CMBlockBuffer 0x170119350 totalDataLength: 4632 retainCount: 1 allocator: 0x1957c2c80 subBlockCapacity: 2
     [0] 4632 bytes @ offset 0 Memory Block 0x10295c000, 4632 bytes (custom V=0 A=0x0 F=0x18498bb44 R=0x0)

似乎我设法获得指针的CMBlockBuffer对象是另一个无法访问的CMBlockBuferRef(4632字节).

任何人都可以发布如何访问H264 elemantry流？

谢谢!

Answer 1

我一直在努力解决这个问题已经有一段时间了,终于把事情搞清楚了.

该功能CMBlockBufferGetDataPointer使您可以访问所需的所有数据,但是有一些不太明显的事情需要将它转换为基本流.

AVCC与附件B格式

CMBlockBuffer中的数据以AVCC格式存储,而基本流通常遵循附件B规范(这里是两种格式的优秀概述).在AVCC格式中,4个第一个字节包含NAL单元的长度(H264包的另一个字).您需要将此标头替换为4字节起始码:0x00 0x00 0x00 0x01,它用作附件B基本流中NAL单元之间的分隔符(3字节版本0x00 0x00 0x01也可正常工作).

单个CMBlockBuffer中的多个NAL单元

下一个不太明显的事情是单个CMBlockBuffer有时会包含多个NAL单元.Apple似乎在每个I-Frame NAL单元(也称为IDR)中添加了一个包含元数据的附加NAL单元(SEI).这可能是您在单个CMBlockBuffer对象中看到多个缓冲区的原因.但是,该CMBlockBufferGetDataPointer函数为您提供了一个可以访问所有数据的指针.话虽如此,多个NAL单元的存在使AVCC头的转换复杂化.现在你实际上必须读取AVCC头中包含的长度值以找到下一个NAL单元,并继续转换头,直到你到达缓冲区的末尾.

Big-Endian vs Little-Endian

下一个不太明显的事情是AVCC标题以Big-Endian格式存储,而iOS本身就是Little-Endian.因此,当您读取AVCC头中包含的长度值时,CFSwapInt32BigToHost首先将其传递给函数.

SPS和PPS NAL单位

最后不太明显的是CMBlockBuffer中的数据不包含参数NAL单元SPS和PPS,其包含解码器的配置参数,例如配置文件,级别,分辨率,帧速率.它们作为元数据存储在样本缓冲区的格式描述中,可以通过该函数访问CMVideoFormatDescriptionGetH264ParameterSetAtIndex.请注意,您必须在发送之前将起始代码添加到这些NAL单元.SPS和PPS NAL单元不必与每个新帧一起发送.解码器只需要读取它们一次,但通常会定期重新发送它们,例如在每个新的I帧NAL单元之前.

代码示例

下面是考虑所有这些因素的代码示例.

static void videoFrameFinishedEncoding(void *outputCallbackRefCon,
                                       void *sourceFrameRefCon,
                                       OSStatus status,
                                       VTEncodeInfoFlags infoFlags,
                                       CMSampleBufferRef sampleBuffer) {
    // Check if there were any errors encoding
    if (status != noErr) {
        NSLog(@"Error encoding video, err=%lld", (int64_t)status);
        return;
    }

    // In this example we will use a NSMutableData object to store the
    // elementary stream.
    NSMutableData *elementaryStream = [NSMutableData data];


    // Find out if the sample buffer contains an I-Frame.
    // If so we will write the SPS and PPS NAL units to the elementary stream.
    BOOL isIFrame = NO;
    CFArrayRef attachmentsArray = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, 0);
    if (CFArrayGetCount(attachmentsArray)) {
        CFBooleanRef notSync;
        CFDictionaryRef dict = CFArrayGetValueAtIndex(attachmentsArray, 0);
        BOOL keyExists = CFDictionaryGetValueIfPresent(dict,
                                                       kCMSampleAttachmentKey_NotSync,
                                                       (const void **)&notSync);
        // An I-Frame is a sync frame
        isIFrame = !keyExists || !CFBooleanGetValue(notSync);
    }

    // This is the start code that we will write to
    // the elementary stream before every NAL unit
    static const size_t startCodeLength = 4;
    static const uint8_t startCode[] = {0x00, 0x00, 0x00, 0x01};

    // Write the SPS and PPS NAL units to the elementary stream before every I-Frame
    if (isIFrame) {
        CMFormatDescriptionRef description = CMSampleBufferGetFormatDescription(sampleBuffer);

        // Find out how many parameter sets there are
        size_t numberOfParameterSets;
        CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description,
                                                           0, NULL, NULL,
                                                           &numberOfParameterSets,
                                                           NULL);

        // Write each parameter set to the elementary stream
        for (int i = 0; i < numberOfParameterSets; i++) {
            const uint8_t *parameterSetPointer;
            size_t parameterSetLength;
            CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description,
                                                               i,
                                                               &parameterSetPointer,
                                                               &parameterSetLength,
                                                               NULL, NULL);

            // Write the parameter set to the elementary stream
            [elementaryStream appendBytes:startCode length:startCodeLength];
            [elementaryStream appendBytes:parameterSetPointer length:parameterSetLength];
        }
    }

    // Get a pointer to the raw AVCC NAL unit data in the sample buffer
    size_t blockBufferLength;
    uint8_t *bufferDataPointer = NULL;
    CMBlockBufferGetDataPointer(CMSampleBufferGetDataBuffer(sampleBuffer),
                                0,
                                NULL,
                                &blockBufferLength,
                                (char **)&bufferDataPointer);

    // Loop through all the NAL units in the block buffer
    // and write them to the elementary stream with
    // start codes instead of AVCC length headers
    size_t bufferOffset = 0;
    static const int AVCCHeaderLength = 4;
    while (bufferOffset < blockBufferLength - AVCCHeaderLength) {
        // Read the NAL unit length
        uint32_t NALUnitLength = 0;
        memcpy(&NALUnitLength, bufferDataPointer + bufferOffset, AVCCHeaderLength);
        // Convert the length value from Big-endian to Little-endian
        NALUnitLength = CFSwapInt32BigToHost(NALUnitLength);
        // Write start code to the elementary stream
        [elementaryStream appendBytes:startCode length:startCodeLength];
        // Write the NAL unit without the AVCC length header to the elementary stream
        [elementaryStream appendBytes:bufferDataPointer + bufferOffset + AVCCHeaderLength
                               length:NALUnitLength];
        // Move to the next NAL unit in the block buffer
        bufferOffset += AVCCHeaderLength + NALUnitLength;
    }
}   

Answer 2

感谢安东的出色回答！我正在为那些有兴趣直接在他们基于 Swift 的项目中使用这里讨论的概念的人提供一个简单的 Swift 端口解决方案。

public func didEncodeFrame(frame: CMSampleBuffer)
{
    print ("Received encoded frame in delegate...")

    //----AVCC to Elem stream-----//
    var elementaryStream = NSMutableData()

    //1. check if CMBuffer had I-frame
    var isIFrame:Bool = false
    let attachmentsArray:CFArray = CMSampleBufferGetSampleAttachmentsArray(frame, false)!
    //check how many attachments
    if ( CFArrayGetCount(attachmentsArray) > 0 ) {
        let dict = CFArrayGetValueAtIndex(attachmentsArray, 0)
        let dictRef:CFDictionaryRef = unsafeBitCast(dict, CFDictionaryRef.self)
        //get value
        let value = CFDictionaryGetValue(dictRef, unsafeBitCast(kCMSampleAttachmentKey_NotSync, UnsafePointer<Void>.self))
        if ( value != nil ){
            print ("IFrame found...")
            isIFrame = true
        }
    }

    //2. define the start code
    let nStartCodeLength:size_t = 4
    let nStartCode:[UInt8] = [0x00, 0x00, 0x00, 0x01]

    //3. write the SPS and PPS before I-frame
    if ( isIFrame == true ){
        let description:CMFormatDescriptionRef = CMSampleBufferGetFormatDescription(frame)!
        //how many params
        var numParams:size_t = 0
        CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description, 0, nil, nil, &numParams, nil)

        //write each param-set to elementary stream
        print("Write param to elementaryStream ", numParams)
        for i in 0..<numParams {
            var parameterSetPointer:UnsafePointer<UInt8> = nil
            var parameterSetLength:size_t = 0
            CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description, i, &parameterSetPointer, &parameterSetLength, nil, nil)
            elementaryStream.appendBytes(nStartCode, length: nStartCodeLength)
            elementaryStream.appendBytes(parameterSetPointer, length: unsafeBitCast(parameterSetLength, Int.self))
        }
    }

    //4. Get a pointer to the raw AVCC NAL unit data in the sample buffer
    var blockBufferLength:size_t = 0
    var bufferDataPointer: UnsafeMutablePointer<Int8> = nil
    CMBlockBufferGetDataPointer(CMSampleBufferGetDataBuffer(frame)!, 0, nil, &blockBufferLength, &bufferDataPointer)
    print ("Block length = ", blockBufferLength)

    //5. Loop through all the NAL units in the block buffer
    var bufferOffset:size_t = 0
    let AVCCHeaderLength:Int = 4
    while (bufferOffset < (blockBufferLength - AVCCHeaderLength) ) {
        // Read the NAL unit length
        var NALUnitLength:UInt32 =  0
        memcpy(&NALUnitLength, bufferDataPointer + bufferOffset, AVCCHeaderLength)
        //Big-Endian to Little-Endian
        NALUnitLength = CFSwapInt32(NALUnitLength)
        if ( NALUnitLength > 0 ){
            print ( "NALUnitLen = ", NALUnitLength)
            // Write start code to the elementary stream
            elementaryStream.appendBytes(nStartCode, length: nStartCodeLength)
            // Write the NAL unit without the AVCC length header to the elementary stream
            elementaryStream.appendBytes(bufferDataPointer + bufferOffset + AVCCHeaderLength, length: Int(NALUnitLength))
            // Move to the next NAL unit in the block buffer
            bufferOffset += AVCCHeaderLength + size_t(NALUnitLength);
            print("Moving to next NALU...")
        }
    }
    print("Read completed...")
}