如何处理GPUImage图像缓冲区,以便它们可以用于像Tokbox这样的东西？

Question

我正在使用OpenTok并用我自己的包含GPUImage的子类版本替换了他们的Publisher.我的目标是添加过滤器.

应用程序构建并运行,但崩溃在这里:

   func willOutputSampleBuffer(sampleBuffer: CMSampleBuffer!) {
        let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
        CVPixelBufferLockBaseAddress(imageBuffer!, 0)
        videoFrame?.clearPlanes()
        for var i = 0 ; i < CVPixelBufferGetPlaneCount(imageBuffer!); i++ {
            print(i)
            videoFrame?.planes.addPointer(CVPixelBufferGetBaseAddressOfPlane(imageBuffer!, i))
        }
        videoFrame?.orientation = OTVideoOrientation.Left
        videoCaptureConsumer.consumeFrame(videoFrame) //comment this out to stop app from crashing. Otherwise, it crashes here.
        CVPixelBufferUnlockBaseAddress(imageBuffer!, 0)
    }

如果我评论该行,我就可以运行应用程序而不会崩溃.事实上,我看到过滤器正确应用,但它正在闪烁.Nothings发布到Opentok.

我的整个代码库都可以下载.单击此处查看特定文件:这是该类的特定文件.它实际上很容易运行 - 只需在运行之前进行pod安装.

经过检查,可能videoCaptureConsumer是没有初始化.协议参考

我不知道我的代码意味着什么.我直接从这个客观的C文件中翻译了它:Tokbox的示例项目

Answer 1

我分析了你的Swift项目和Objective-C项目。我发现这两者都不起作用。

在这篇文章中，我想进行第一次更新，并展示一个如何将 GPU 图像过滤器与 OpenTok 结合使用的真正有效的演示。

您的 GPUImage 文件管理器实现无法与 OpenTok 配合使用的原因

#1 多目标规范

let sepia = GPUImageSepiaFilter()
videoCamera?.addTarget(sepia)
sepia.addTarget(self.view)        
videoCamera?.addTarget(self.view) // <-- This is wrong and produces the flickering
videoCamera?.startCameraCapture()

两个源尝试渲染到同一视图中。使事物闪烁...

第一部分已解决。下一步：为什么没有向 OpenTok 发布任何内容？为了找到原因，我决定从“工作”入手Objective-C版本开始。

#2 - Objective-C 原始代码库

原始Objective-C版本没有预期的功能。GPUImageVideoCamera向订阅者的发布OpenTok是有效的，但不涉及过滤。这就是您的核心要求。关键是，由于不同的图像格式和如何进行异步编程的机制不同，添加过滤器并不像人们想象的那么简单。

所以原因#2，为什么你的代码没有按预期工作：你的移植工作的参考代码库不正确。它不允许在发布 - 订阅者管道之间放置 GPU 过滤器。

一个可行的 Objective-C 实现

我修改了Objective-C版本。目前的结果如下所示：

[![在此处输入图像描述][1]][1]

它运行顺利。

最后步骤

这是自定义发布者的完整代码Tok。TokBoxGPUImagePublisher它基本上是来自[ https://github.com/JayTokBox/TokBoxGPUImage/blob/master/TokBoxGPUImage/ViewController.m][2]的原始代码( ) ，并进行了以下显着修改：

OTVideoFrame 使用新格式进行实例化

    ...
    format = [[OTVideoFormat alloc] init];
    format.pixelFormat = OTPixelFormatARGB;
    format.bytesPerRow = [@[@(imageWidth * 4)] mutableCopy];
    format.imageWidth = imageWidth;
    format.imageHeight = imageHeight;
    videoFrame = [[OTVideoFrame alloc] initWithFormat: format];
    ...

替换WillOutputSampleBuffer回调机制

仅当直接来自的样本缓冲区GPUImageVideoCamera准备就绪而不是来自您的自定义过滤器时，才会触发此回调。GPUImageFilters不提供这样的回调/委托机制。GPUImageRawDataOutput这就是为什么我们在中间放置一个并要求它提供准备好的图像。该管道在方法中实现initCapture，如下所示：

    videoCamera = [[GPUImageVideoCamera alloc] initWithSessionPreset:AVCaptureSessionPreset640x480 cameraPosition:AVCaptureDevicePositionBack];

    videoCamera.outputImageOrientation = UIInterfaceOrientationPortrait;
    sepiaImageFilter = [[GPUImageSepiaFilter alloc] init];
    [videoCamera addTarget:sepiaImageFilter];
    // Create rawOut
    CGSize size = CGSizeMake(imageWidth, imageHeight);
    rawOut = [[GPUImageRawDataOutput alloc] initWithImageSize:size resultsInBGRAFormat:YES];

    // Filter into rawOut
    [sepiaImageFilter addTarget:rawOut];
    // Handle filtered images
    // We need a weak reference here to avoid a strong reference cycle.
    __weak GPUImageRawDataOutput* weakRawOut = self->rawOut;
    __weak OTVideoFrame* weakVideoFrame = self->videoFrame;
    __weak id<OTVideoCaptureConsumer> weakVideoCaptureConsumer = self.videoCaptureConsumer;
    //
    [rawOut setNewFrameAvailableBlock:^{
        [weakRawOut lockFramebufferForReading];
        // GLubyte is an uint8_t
        GLubyte* outputBytes = [weakRawOut rawBytesForImage];


        // About the video formats used by OTVideoFrame
        // --------------------------------------------
        // Both YUV video formats (i420, NV12) have the (for us) following important properties:
        //
        //  - Two planes
        //  - 8 bit Y plane
        //  - 8 bit 2x2 subsampled U and V planes (1/4 the pixels of the Y plane)
        //      --> 12 bits per pixel
        //
        // Further reading: www.fourcc.org/yuv.php
        //
        [weakVideoFrame clearPlanes];
        [weakVideoFrame.planes addPointer: outputBytes];
        [weakVideoCaptureConsumer consumeFrame: weakVideoFrame];
        [weakRawOut unlockFramebufferAfterReading];
    }];
    [videoCamera addTarget:self.view];
    [videoCamera startCameraCapture];

整个代码（真正重要的是initCapture）

//

//  TokBoxGPUImagePublisher.m

//  TokBoxGPUImage

//

//  Created by Jaideep Shah on 9/5/14.

//  Copyright (c) 2014 Jaideep Shah. All rights reserved.

//



#import "TokBoxGPUImagePublisher.h"

#import "GPUImage.h"

static size_t imageHeight = 480;

static size_t imageWidth = 640;





@interface TokBoxGPUImagePublisher() <GPUImageVideoCameraDelegate, OTVideoCapture> {

    GPUImageVideoCamera *videoCamera;

    GPUImageSepiaFilter *sepiaImageFilter;

    OTVideoFrame* videoFrame;

    GPUImageRawDataOutput* rawOut;

    OTVideoFormat* format;

}



@end



@implementation TokBoxGPUImagePublisher



@synthesize videoCaptureConsumer ;  // In OTVideoCapture protocol



- (id)initWithDelegate:(id<OTPublisherDelegate>)delegate name:(NSString*)name

{

    self = [super initWithDelegate:delegate name:name];

    if (self)

    {

        self.view = [[GPUImageView alloc] initWithFrame:CGRectMake(0, 0, 1, 1)];

        [self setVideoCapture:self];

        format = [[OTVideoFormat alloc] init];
        format.pixelFormat = OTPixelFormatARGB;
        format.bytesPerRow = [@[@(imageWidth * 4)] mutableCopy];
        format.imageWidth = imageWidth;
        format.imageHeight = imageHeight;
        videoFrame = [[OTVideoFrame alloc] initWithFormat: format];
    }

    return self;

}

#pragma mark GPUImageVideoCameraDelegate



- (void)willOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer

{

    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);

    CVPixelBufferLockBaseAddress(imageBuffer, 0);



    [videoFrame clearPlanes];



    for (int i = 0; i < CVPixelBufferGetPlaneCount(imageBuffer); i++) {

        [videoFrame.planes addPointer:CVPixelBufferGetBaseAddressOfPlane(imageBuffer, i)];

    }

    videoFrame.orientation = OTVideoOrientationLeft;



    [self.videoCaptureConsumer consumeFrame:videoFrame];



    CVPixelBufferUnlockBaseAddress(imageBuffer, 0);

}

#pragma mark OTVideoCapture



- (void) initCapture {
    videoCamera = [[GPUImageVideoCamera alloc] initWithSessionPreset:AVCaptureSessionPreset640x480

                                                      cameraPosition:AVCaptureDevicePositionBack];

    videoCamera.outputImageOrientation = UIInterfaceOrientationPortrait;
    sepiaImageFilter = [[GPUImageSepiaFilter alloc] init];
    [videoCamera addTarget:sepiaImageFilter];
    // Create rawOut
    CGSize size = CGSizeMake(imageWidth, imageHeight);
    rawOut = [[GPUImageRawDataOutput alloc] initWithImageSize:size resultsInBGRAFormat:YES];

    // Filter into rawOut
    [sepiaImageFilter addTarget:rawOut];
    // Handle filtered images
    // We need a weak reference here to avoid a strong reference cycle.
    __weak GPUImageRawDataOutput* weakRawOut = self->rawOut;
    __weak OTVideoFrame* weakVideoFrame = self->videoFrame;
    __weak id<OTVideoCaptureConsumer> weakVideoCaptureConsumer = self.videoCaptureConsumer;
    //
    [rawOut setNewFrameAvailableBlock:^{
        [weakRawOut lockFramebufferForReading];
        // GLubyte is an uint8_t
        GLubyte* outputBytes = [weakRawOut rawBytesForImage];


        // About the video formats used by OTVideoFrame
        // --------------------------------------------
        // Both YUV video formats (i420, NV12) have the (for us) following important properties:
        //
        //  - Two planes
        //  - 8 bit Y plane
        //  - 8 bit 2x2 subsampled U and V planes (1/4 the pixels of the Y plane)
        //      --> 12 bits per pixel
        //
        // Further reading: www.fourcc.org/yuv.php
        //
        [weakVideoFrame clearPlanes];
        [weakVideoFrame.planes addPointer: outputBytes];
        [weakVideoCaptureConsumer consumeFrame: weakVideoFrame];
        [weakRawOut unlockFramebufferAfterReading];
    }];
    [videoCamera addTarget:self.view];
    [videoCamera startCameraCapture];
}



- (void)releaseCapture

{

    videoCamera.delegate = nil;

    videoCamera = nil;

}

- (int32_t) startCapture {

    return 0;

}



- (int32_t) stopCapture {

    return 0;

}



- (BOOL) isCaptureStarted {

    return YES;

}

- (int32_t)captureSettings:(OTVideoFormat*)videoFormat {

    videoFormat.pixelFormat = OTPixelFormatNV12;

    videoFormat.imageWidth = imageWidth;

    videoFormat.imageHeight = imageHeight;

    return 0;

}

@end