在iOS中用白色像素替换部分像素缓冲区

Question

我正在使用iPhone相机捕捉实时视频并将像素缓冲区馈送到进行某些对象识别的网络.以下是相关代码:(我不会发布用于设置AVCaptureSession 等的代码,因为这是非常标准的.)

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
    CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);

    OSType sourcePixelFormat = CVPixelBufferGetPixelFormatType( pixelBuffer );
    int doReverseChannels;
    if ( kCVPixelFormatType_32ARGB == sourcePixelFormat ) {
        doReverseChannels = 1;
    } else if ( kCVPixelFormatType_32BGRA == sourcePixelFormat ) {
        doReverseChannels = 0;
    } else {
        assert(false);
    }

    const int sourceRowBytes = (int)CVPixelBufferGetBytesPerRow( pixelBuffer );
    const int width = (int)CVPixelBufferGetWidth( pixelBuffer );
    const int fullHeight = (int)CVPixelBufferGetHeight( pixelBuffer );
    CVPixelBufferLockBaseAddress( pixelBuffer, 0 );
    unsigned char* sourceBaseAddr = CVPixelBufferGetBaseAddress( pixelBuffer );
    int height;
    unsigned char* sourceStartAddr;
    if (fullHeight <= width) {
        height = fullHeight;
        sourceStartAddr = sourceBaseAddr;
    } else {
        height = width;
        const int marginY = ((fullHeight - width) / 2);
        sourceStartAddr = (sourceBaseAddr + (marginY * sourceRowBytes));
    }
}

然后,网络需要sourceStartAddr,width,height,sourceRowBytes和doReverseChannels作为输入.

我的问题如下:用所有白色'像素'替换或删除部分图像数据的最简单和/或最有效的方法是什么？是否有可能直接覆盖像素缓冲区数据的e部分,如果是,如何？

我对这个像素缓冲区的工作方式只有一个非常基本的了解,所以如果我在这里遗漏了一些非常基本的东西,我会道歉.我在Stackoverflow上找到的与我最密切相关的问题就是这个问题,其中a EAGLContext用于向视频帧添加文本.虽然这实际上适用于我的目标,只需要替换单个图像,但我认为如果应用于每个视频帧,这一步将会破坏性能,我想知道是否有其他方法.任何帮助在这里将不胜感激.

Answer 1

这是一种在CVPixelBufferRef不使用Core Graphics或OpenGL等其他库的情况下操作a的简单方法:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
    CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);

    const int kBytesPerPixel = 4;
    CVPixelBufferLockBaseAddress( pixelBuffer, 0 );
    int bufferWidth = (int)CVPixelBufferGetWidth( pixelBuffer );
    int bufferHeight = (int)CVPixelBufferGetHeight( pixelBuffer );
    size_t bytesPerRow = CVPixelBufferGetBytesPerRow( pixelBuffer );
    uint8_t *baseAddress = CVPixelBufferGetBaseAddress( pixelBuffer );

    for ( int row = 0; row < bufferHeight; row++ )
    {
        uint8_t *pixel = baseAddress + row * bytesPerRow;
        for ( int column = 0; column < bufferWidth; column++ )
        {
            if ((row < 100) && (column < 100) {
                pixel[0] = 255; // BGRA, Blue value
                pixel[1] = 255; // Green value
                pixel[2] = 255; // Red value
            }
            pixel += kBytesPerPixel;
        }
    }

    CVPixelBufferUnlockBaseAddress( pixelBuffer, 0 );

    // Do whatever needs to be done with the pixel buffer
}

这将覆盖图像中带有白色像素的100 x 100像素的左上角补丁.

我在这个名为RosyWriter的 Apple Developer Example中找到了这个解决方案.

有点惊讶我在这里没有得到任何答案,考虑到这是多么容易.希望这有助于某人.

Answer 2

使用 Swift 实现更新它。

        CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: 0))
        let bufferWidth = Int(CVPixelBufferGetWidth(pixelBuffer))
        let bufferHeight = Int(CVPixelBufferGetHeight(pixelBuffer))
        let bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer)

        guard let baseAddress = CVPixelBufferGetBaseAddress(pixelBuffer) else {
                return
        }

        for row in 0..<bufferHeight {
            var pixel = baseAddress + row * bytesPerRow
            for col in 0..<bufferWidth {
                let blue = pixel
                blue.storeBytes(of: 255, as: UInt8.self)

                let red = pixel + 1
                red.storeBytes(of: 255, as: UInt8.self)

                let green = pixel + 2
                green.storeBytes(of: 255, as: UInt8.self)
             
                let alpha = pixel + 3
                alpha.storeBytes(of: 255, as: UInt8.self)
                
                pixel += 4;
            }
        }

        CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: 0))

由于baseAddressgiveUnsafeMutableRawPointer不支持下标，因此您必须storeBytes改用。这基本上是与上述 Objective-C 版本的唯一主要区别。