从AVPlayerItemVideoOutput到openGL纹理的最佳路径

Question

一直在试图找出从AVFoundation视频到openGLTexture的当前最佳路径,我找到的大部分内容都与iOS有关,我似乎无法在OSX中使其运行良好.

首先,这是我设置videoOutput的方式:

NSDictionary *pbOptions = [NSDictionary dictionaryWithObjectsAndKeys:
                          [NSNumber numberWithInt:kCVPixelFormatType_422YpCbCr8], kCVPixelBufferPixelFormatTypeKey,
                          [NSDictionary dictionary], kCVPixelBufferIOSurfacePropertiesKey,
                                   nil];
        self.playeroutput   = [[AVPlayerItemVideoOutput alloc] initWithPixelBufferAttributes:pbOptions];
        self.playeroutput.suppressesPlayerRendering = YES;

我正在尝试三种不同的解决方案,其中只有一种似乎能够始终如一地工作,但不确定它是最快的.一个人工作一点点,然后分裂跳跃到整个地方的框架,一个只是产生黑色.

首先,使用glTexImage2D的工作解决方案

- (BOOL) renderWithCVPixelBufferForTime: (NSTimeInterval) time
{
    CMTime vTime = [self.playeroutput itemTimeForHostTime:CACurrentMediaTime()];

    if ([self.playeroutput hasNewPixelBufferForItemTime:vTime]) {
        if (_cvPixelBufferRef) {
            CVPixelBufferUnlockBaseAddress(_cvPixelBufferRef, kCVPixelBufferLock_ReadOnly);
            CVPixelBufferRelease(_cvPixelBufferRef);
        }
        _cvPixelBufferRef = [self.playeroutput copyPixelBufferForItemTime:vTime itemTimeForDisplay:NULL];

        CVPixelBufferLockBaseAddress(_cvPixelBufferRef, kCVPixelBufferLock_ReadOnly);
        GLsizei       texWidth    = CVPixelBufferGetWidth(_cvPixelBufferRef);
        GLsizei       texHeight   = CVPixelBufferGetHeight(_cvPixelBufferRef);
        GLvoid *baseAddress = CVPixelBufferGetBaseAddress(_cvPixelBufferRef);


        glBindTexture(GL_TEXTURE_RECTANGLE_ARB, self.textureName);
        glTexParameteri(GL_TEXTURE_RECTANGLE_ARB, GL_TEXTURE_STORAGE_HINT_APPLE , GL_STORAGE_CACHED_APPLE);
        glTexImage2D(GL_TEXTURE_RECTANGLE_ARB, 0, GL_RGB, texWidth, texHeight, 0, GL_YCBCR_422_APPLE, GL_UNSIGNED_SHORT_8_8_APPLE, baseAddress);

        glBindTexture(GL_TEXTURE_RECTANGLE_ARB, 0);
    } 
    return YES;
} 

这种方法花费了大部分时间来锁定像素缓冲区的基地址,但是文档说如果从GPU访问数据并且可能会影响性能,则不需要它.我无法弄清楚如何在没有锁定的情况下获得纹理.

接下来,使用iOSurface的几乎正常工作的解决方案,这有点工作,然后变得非常小问题,就好像ioSurfaces从以前的帧使用:

- (BOOL) renderWithIOSurfaceForTime:(NSTimeInterval) time {
    CMTime vTime = [self.playeroutput itemTimeForHostTime:CACurrentMediaTime()];

    if ([self.playeroutput hasNewPixelBufferForItemTime:vTime]) {

         CVPixelBufferRef pb = [self.playeroutput copyPixelBufferForItemTime:vTime itemTimeForDisplay:NULL];
         IOSurfaceRef newSurface = CVPixelBufferGetIOSurface(pb);
         if (_surfaceRef != newSurface) {
            IOSurfaceDecrementUseCount(_surfaceRef);
            _surfaceRef = newSurface;
            IOSurfaceIncrementUseCount(_surfaceRef);
            GLsizei       texWidth = (int) IOSurfaceGetWidth(_surfaceRef);
            GLsizei       texHeight= (int) IOSurfaceGetHeight(_surfaceRef);
            size_t        rowbytes = CVPixelBufferGetBytesPerRow(_cvPixelBufferRef);

            glBindTexture(GL_TEXTURE_RECTANGLE_ARB, self.textureName);
            CGLTexImageIOSurface2D(cgl_ctx, GL_TEXTURE_RECTANGLE_ARB, GL_RGB8, texWidth, texHeight, GL_YCBCR_422_APPLE, GL_UNSIGNED_SHORT_8_8_APPLE, _surfaceRef, 0);
            glBindTexture(GL_TEXTURE_RECTANGLE_ARB, 0);
        }
        CVPixelBufferRelease(pb);   
    }
    return YES;
}

如果可行的话,这似乎是最好的解决方案.我有另一个从ioSurfaces创建纹理的过程,它工作得很好,同时也非常快.

最后似乎推荐用于iOS的一个是使用CVOpenGLTextureCache,osx中的实现看起来略有不同,我无法让它呈现除黑色之外的任何东西,而且它似乎比第一个解决方案更慢......

- (BOOL) renderByCVOpenGLTextureCacheForTime:(NSTimeInterval) time
{
    CMTime vTime = [self.playeroutput itemTimeForHostTime:CACurrentMediaTime()];

    if ([self.playeroutput hasNewPixelBufferForItemTime:vTime]) {
        _cvPixelBufferRef = [self.playeroutput copyPixelBufferForItemTime:vTime itemTimeForDisplay:NULL];
        if (!_textureCacheRef) {
            CVReturn error = CVOpenGLTextureCacheCreate(kCFAllocatorDefault, NULL, cgl_ctx, CGLGetPixelFormat(cgl_ctx), NULL, &_textureCacheRef);
            if (error) {
                NSLog(@"Texture cache create failed");
            }
        }

        CVReturn error = CVOpenGLTextureCacheCreateTextureFromImage(kCFAllocatorDefault, _textureCacheRef, _cvPixelBufferRef, NULL, &_textureRef);
        if (error) {
            NSLog(@"Failed to copy video texture");
        }


        CVPixelBufferRelease(_cvPixelBufferRef);

        _textureName = CVOpenGLTextureGetName(_textureRef);

    }
    return YES;
}

可能我没有正确设置,osx中的纹理缓存没有文档.

我发现最好在渲染周期之间保留cvpixelbufferref,据我所知,纹理上传可以与CGLTexImage2d异步运行,我很满意,其他几个对象可以同时渲染,cglflushDrawable是最终在最终绘制纹理时调用.

我找到的用于视频的大多数苹果示例都与iOS相关,并将纹理分成两部分以在着色器中重新组合,例如在此示例中https://developer.apple.com/library/ios/samplecode/GLCameraRipple/ Listings/GLCameraRipple_main_m.html#// apple_ref/doc/uid/DTS40011222-GLCameraRipple_main_m-DontLinkElementID_11 由于纹理缓存在iOS中具有不同的实现,我无法直接调整代码.

所以任何指针都会很棒,看起来像是重要的功能,但是我发现关于av基础和osx上的opengl的信息似乎非常消极.

更新:使用计数更新了ioSurface代码,工作时间稍长,但最终仍然出现故障.

Answer 1

我开始了同样的旅程，对 OpenGL 的了解与我对养羊业的了解一样多，但确实注意到您的 pbOptions 不包含 kCVPixelBufferOpenGLCompatibilityKey

NSDictionary *pbOptions = [NSDictionary dictionaryWithObjectsAndKeys:
    [NSNumber numberWithInt:kCVPixelFormatType_422YpCbCr8], kCVPixelBufferPixelFormatTypeKey,
    [NSDictionary dictionary], kCVPixelBufferIOSurfacePropertiesKey,
    [NSNumber numberWithBool:YES], kCVPixelBufferOpenGLCompatibilityKey, nil];

我将像素缓冲区请求为 kCVPixelFormatType_32BGRA 而不是组件，这对我适用于 _currentSurface (IOSurfaceRef)、textureName (GLuint)、_sourceWidth (int) 和 _sourceHeight (int) 的局部变量

IOSurfaceRef newSurface = CVPixelBufferGetIOSurface(pixelBuffer);
if (_currentSurface != newSurface) {
    CGLContextObj  cgl_ctx = (CGLContextObj)[[self openGLContext] CGLContextObj];
    [[self openGLContext] makeCurrentContext];

    IOSurfaceDecrementUseCount(_currentSurface);
    _currentSurface = newSurface;
    IOSurfaceIncrementUseCount(_currentSurface);
    GLsizei texWidth = (int) IOSurfaceGetWidth(_currentSurface);
    GLsizei texHeight = (int) IOSurfaceGetHeight(_currentSurface);

    if (_sourceWidth == 0 && _sourceHeight == 0) {
        // used during drawing of texture
        _sourceWidth = texWidth;
        _sourceHeight = texHeight;
    }

    if (!_textureName) {
        GLuint name;
        glGenTextures(1, &name);
        _textureName = name;
    }

    glBindTexture(GL_TEXTURE_RECTANGLE_ARB, _textureName);
    CGLTexImageIOSurface2D(cgl_ctx, GL_TEXTURE_RECTANGLE_ARB, GL_RGBA, texWidth, texHeight, GL_BGRA, GL_UNSIGNED_INT_8_8_8_8_REV, self.currentSurface, 0);        
    glBindTexture(GL_TEXTURE_RECTANGLE_ARB, 0);
}