Android Camera.PreviewCallback调度(使用OpenGL和OpenCV)

Question

我正在开发一个应用程序,需要使用相机输入和实时结果显示进行大量图像处理.我决定使用OpenGL和OpenCV以及Android的普通相机API.到目前为止,它已经成为一个多线程的噩梦,不幸的是,由于缺少onPreviewFrame()回调的文档,我感到非常有限.

我从文档中了解到,在使用Camera.open()获取摄像头的线程上调用onPreviewFrame().令我困惑的是这个回调是如何安排的 - 它似乎是固定的帧率.我当前的体系结构依赖于onPreviewFrame()回调来启动图像处理/显示周期,当我阻止相机回调线程太长时间似乎陷入死锁,所以我怀疑回调在调度方面是不灵活的.我想减慢帧速率以测试它,但我的设备不支持这个.

我开始使用http://maninara.blogspot.ca/2012/09/render-camera-preview-using-opengl-es.html上的代码.此代码不是很平行,它只是为了准确显示相机返回的数据.根据我的需要,我调整了代码来绘制位图,并使用专用线程将相机数据缓冲到另一个专用的重型图像处理线程(所有OpenGL线程之外).

这是我的代码(简化):

CameraSurfaceRenderer.java

class CameraSurfaceRenderer implements GLSurfaceView.Renderer, SurfaceTexture.OnFrameAvailableListener,
    Camera.PreviewCallback
{

static int[]                surfaceTexPtr;

static CameraSurfaceView    cameraSurfaceView;
static FloatBuffer          pVertex;
static FloatBuffer          pTexCoord;
static int                  hProgramPointer;

static Camera               camera;
static SurfaceTexture       surfaceTexture;

static Bitmap               procBitmap;
static int[]                procBitmapPtr;

static boolean              updateSurfaceTex = false;

static ConditionVariable    previewFrameLock;
static ConditionVariable    bitmapDrawLock;

// MarkerFinder extends CameraImgProc
static MarkerFinder         markerFinder = new MarkerFinder();
static Thread               previewCallbackThread;

static
{
    previewFrameLock = new ConditionVariable();
    previewFrameLock.open();

    bitmapDrawLock = new ConditionVariable();
    bitmapDrawLock.open();
}

CameraSurfaceRenderer(Context context, CameraSurfaceView view)
{
    rendererContext = context;
    cameraSurfaceView = view;

    // … // Load pVertex and pTexCoord vertex buffers
}

public void close()
{
    // … // This code usually doesn’t have the chance to get called
}

@Override
public void onSurfaceCreated(GL10 unused, EGLConfig config)
{
// .. // Initialize a texture object for the bitmap data

    surfaceTexPtr = new int[1];
    surfaceTexture = new SurfaceTexture(surfaceTexPtr[0]);
    surfaceTexture.setOnFrameAvailableListener(this);

    //Initialize camera on its own thread so preview frame callbacks are processed in parallel
    previewCallbackThread = new Thread()
    {
        @Override
        public void run()
        {
            try {
                camera = Camera.open();
            } catch (RuntimeException e) {
                // … // Bitch to the user through a Toast on the UI thread
            }
            assert camera != null;
            //Callback set on CameraSurfaceRenderer class, but executed on worker thread
            camera.setPreviewCallback(CameraSurfaceRenderer.this);
            try {
                camera.setPreviewTexture(surfaceTexture);
            } catch (IOException e) {
                Log.e(Const.TAG, "Unable to set preview texture");
            }

            Looper.prepare();
            Looper.loop();
        }
    };
    previewCallbackThread.start();

   // … // More OpenGL initialization stuff
}

@Override
public void onDrawFrame(GL10 unused)
{
    GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);

    synchronized (this)
    {
        surfaceTexture.updateTexImage();
    }

// Binds bitmap data to texture
    bindBitmap(procBitmap);

// … // Acquire shader program ttributes, render
    GLES20.glFlush();
}

@Override
public synchronized void onFrameAvailable(SurfaceTexture surfaceTexture)
{
    cameraSurfaceView.requestRender();
}

@Override
public void onPreviewFrame(byte[] data, Camera camera)
{
    Bitmap bitmap = markerFinder.exchangeRawDataForProcessedImg(data, null, camera);

    // … // Check for null bitmap

    previewFrameLock.block();

    procBitmap = bitmap;

    previewFrameLock.close();
    bitmapDrawLock.open();
}

void bindBitmap(Bitmap bitmap)
{
    GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
    GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, procBitmapPtr[0]);

    bitmapDrawLock.block();

    if (bitmap != null && !bitmap.isRecycled())
    {
        GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, bitmap, 0);
        bitmap.recycle();
    }

    bitmapDrawLock.close();
    previewFrameLock.open();
}

@Override
public void onSurfaceChanged(GL10 unused, int width, int height)
{
    GLES20.glViewport(0, 0, width, height);

    // … // Set camera parameters

    camera.startPreview();
}

void deleteTexture()
{
    GLES20.glDeleteTextures(1, surfaceTexPtr, 0);
}
}

CameraImgProc.java(抽象类)

public abstract class CameraImgProc
{
CameraImgProcThread  thread = new CameraImgProcThread();
Handler              handler;
ConditionVariable    bufferSwapLock = new ConditionVariable(true);
Runnable             processTask = new Runnable()
{
    @Override
    public void run()
    {
        imgProcBitmap = processImg(lastWidth, lastHeight, cameraDataBuffer, imgProcBitmap);
        bufferSwapLock.open();
    }
};

int lastWidth    = 0;
int lastHeight   = 0;

Mat cameraDataBuffer;
Bitmap imgProcBitmap;

public CameraImgProc()
{
    thread.start();
    handler = thread.getHandler();
}

protected abstract Bitmap allocateBitmapBuffer(int width, int height);

public final Bitmap exchangeRawDataForProcessedImg(byte[] data, Bitmap dirtyBuffer, Camera camera)
{
    Camera.Parameters parameters = camera.getParameters();
    Camera.Size size = parameters.getPreviewSize();

    // Wait for worker thread to finish processing image
    bufferSwapLock.block();
    bufferSwapLock.close();

    Bitmap freshBuffer = imgProcBitmap;
    imgProcBitmap = dirtyBuffer;

    // Reallocate buffers if size changes to avoid overflow
    assert size != null;
    if (lastWidth != size.width || lastHeight != size.height)
    {
        lastHeight  = size.height;
        lastWidth   = size.width;

        if (cameraDataBuffer != null) cameraDataBuffer.release();
        //YUV format requires 1.5 times as much information in vertical direction
        cameraDataBuffer = new Mat((lastHeight * 3) / 2, lastWidth, CvType.CV_8UC1);

        imgProcBitmap = allocateBitmapBuffer(lastWidth, lastHeight);
        // Buffers had to be resized, therefore no processed data to return

        cameraDataBuffer.put(0, 0, data);

        handler.post(processTask);
        return null;
    }

    // If program did not pass a buffer
    if (imgProcBitmap == null)
        imgProcBitmap = allocateBitmapBuffer(lastWidth, lastHeight);

    // Exchange data
    cameraDataBuffer.put(0, 0, data);

    // Give img processing task to worker thread
    handler.post(processTask);

    return freshBuffer;
}

protected abstract Bitmap processImg(int width, int height, Mat cameraData, Bitmap dirtyBuffer);

class CameraImgProcThread extends Thread
{
    volatile Handler handler;

    @Override
    public void run()
    {
        Looper.prepare();
        handler = new Handler();
        Looper.loop();
    }

    Handler getHandler()
    {
        //noinspection StatementWithEmptyBody
        while (handler == null)
        {
            try {
                Thread.currentThread();
                Thread.sleep(5);
            } catch (Exception e) {
                //Do nothing
            }
        };
        return handler;
    }
}
}

我想要一个健壮的应用程序,无论CameraImgProc.processImg()函数完成多长时间.不幸的是,当相机帧以固定速率馈入时唯一可能的解决方案是在图像处理尚未完成时丢帧,否则我将很快出现缓冲区溢出.

我的问题如下:

有没有办法按需减慢Camera.PreviewCallback频率？

是否有现有的Android API用于从相机获取按需帧数？

我可以参考这个问题的现有解决方案吗？

Answer 1

在使用Camera.open()获取摄像头的线程上调用onPreviewFrame()

这是一个常见的误解.此描述中缺少的关键词是"事件".要将相机回调安排到非UI线程,您需要和"事件线程",HandlerThread的同义词.请参阅我在SO的其他地方的解释和示例.好吧,使用通常的线程来打开相机,就像你的代码一样,并不是没用的,因为这个调用本身在某些设备上可能需要几百毫秒,但事件线程要好得多.

现在让我来解答你的问题:不,你无法控制相机回调的时间表.

如果要接收1 FPS或更低的回调,可以使用setOneShotPreviewCallback().您的milage可能会有所不同,这取决于设备,但我建议使用setPreviewCallbackWithBuffer,如果您想更频繁地检查相机,只需从onPreviewFrame()返回.这些无效回调的性能很小.

请注意,即使将回调卸载到后台线程,它们也会阻塞:如果处理预览帧需要200毫秒,相机将等待.因此,我通常将byte []发送到一个工作线程,并快速释放回调线程.我不建议通过在阻塞模式下处理它们来减慢预览回调的流量,因为在释放线程之后,下一个回调将传递具有未定义时间戳的帧.也许这将是一个新鲜的,或者它可能是一个缓冲前一段时间.