Camera2 将图像从 ImageReader 传递到 MediaRecorder

Question

我正在尝试创建一个CameraCaptureSession具有四个输出的 Camera2：

1. 屏幕预览（SurfaceView，最高 1080p）
2. 照片拍摄（ImageReader，最多 8k 照片）
3. 视频捕捉（MediaRecorder/ MediaCodec，最多 4k 视频）
4. 帧处理（ImageReader最多 4k 视频帧）

不幸的是，Camera2 不支持同时连接所有这四个输出（表面），因此我必须做出妥协。

对我来说最合乎逻辑的妥协是将两个视频捕获管道合并为一个，以便帧处理输出（#4，ImageReader）将帧重定向到视频捕获输出（#3，MediaRecorder）。

如何从 ImageReader 写入图像：

val imageReader = ImageReader.newInstance(4000, 2256, ImageFormat.YUV_420_888, 3)
imageReader.setOnImageAvailableListener({ reader ->
  val image = reader.acquireNextImage() ?: return@setOnImageAvailableListener
  callback.onVideoFrameCaptured(image)
}, queue.handler)

val captureSession = device.createCaptureSession(.., imageReader.surface)

..进入Surface从MediaRecorder？

val surface = MediaCodec.createPersistentInputSurface()
val recorder = MediaRecorder(context)
..
recorder.setInputSurface(surface)

---

我想我可能需要一个带有直通着色器的 OpenGL 管道 - 但我不知道如何从ImageReader到ImageOpenGL 纹理，所以这里的任何帮助将不胜感激。

---

我尝试过的：我研究了 HardwareBuffer API，特别是

auto clientBuffer = eglGetNativeClientBufferANDROID(hardwareBuffer);
...
auto image = eglCreateImageKHR(display,
                               EGL_NO_CONTEXT,
                               EGL_NATIVE_BUFFER_ANDROID,
                               clientBuffer,
                               attribs);
...
glEGLImageTargetTexture2DOES(GR_GL_TEXTURE_EXTERNAL, image);

我认为这可能可行，但它需要 API 级别 28。所以我仍然需要 API 级别 23 及以上的解决方案。该image.getPlanes()函数返回 3 个ByteBufferYUV 数据，但不知道如何从那里创建 OpenGL 纹理。

Answer 1

我（有点）想通了！我找到了ImageWriterAPI，这正是我要从头开始重建的内容 - 从图像到表面的直通管道。

所以现在我将相机帧流式传输到ImageReader，使用 调用帧处理器Image，然后Image使用 作为中间人将其传递MediaRecorder给ImageWriter:)

val size = config.getOutputSizes(ImageFormat.PRIVATE).max()

// Video Recorder Surface. We need to stream Frames here if we are recording.
val surface = recordingSession.surface
val imageWriter = ImageWriter.newInstance(surface,
                                          VIDEO_OUTPUT_BUFFER_SIZE)

// Image Reader Surface. We stream Frames here for Frame Processor or Recording.
val flags = HardwareBuffer.USAGE_GPU_SAMPLED_IMAGE or HardwareBuffer.USAGE_VIDEO_ENCODE
val imageReader = ImageReader.newInstance(size.width,
                                          size.height,
                                          ImageFormat.PRIVATE,
                                          VIDEO_OUTPUT_BUFFER_SIZE,
                                          flags)

imageReader.setOnImageAvailableListener({ reader ->
  val image = reader.acquireNextImage() ?: return
  image.timestamp = System.nanoTime()

  // Call JS Frame Processor
  frameProcessor?.call(image)
                                         
  // If recording, write to Video File             
  if (isRecording) {
    imageWriter.queueInputImage(image)
  }

  image.close()
}, CameraQueues.videoQueue)

// Camera only streams frames into one single Surface
cameraSession.configure(.., imageReader.surface)

我现在唯一的问题是，录制的视频有时在录制约 3 秒后会出现约 1 秒长的停顿，我不知道为什么。也许我应该使用MediaCodec而不是MediaRecorder. 也许我应该使用不同的ImageFormat. 也许我应该调查生成的.mp4文件，看看出了什么问题。也许我应该修复图像时间戳。我不知道。

另外，logcat 会收到这样的垃圾邮件：

2023-08-17 11:38:17.977  3780-3899  GraphicBufferSource     com.mrousavy.camera.example          W  released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.021  3780-3899  GraphicBufferSource     com.mrousavy.camera.example          W  released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.050  3780-3899  GraphicBufferSource     com.mrousavy.camera.example          W  released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.082  3780-3899  GraphicBufferSource     com.mrousavy.camera.example          W  released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.113  3780-3899  GraphicBufferSource     com.mrousavy.camera.example          W  released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.146  3780-3899  GraphicBufferSource     com.mrousavy.camera.example          W  released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.179  3780-3899  GraphicBufferSource     com.mrousavy.camera.example          W  released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]

但是嘿 - 它会录制视频。这是一个好的开始。