如何从 MP4 中逐帧获取？（媒体编解码器）

Question

实际上我正在使用 OpenGL，我想将我所有的纹理都放在 MP4 中以压缩它们。

然后我需要从我的 Android 上的 MP4 中获取它

我需要以某种方式解码 MP4 并根据请求逐帧获取。

我找到了这个 MediaCodec

https://developer.android.com/reference/android/media/MediaCodec

和这个 MediaMetadataRetriever

https://developer.android.com/reference/android/media/MediaMetadataRetriever

但是我没有看到如何逐帧请求的方法......

如果有人用过MP4，请给我一个去哪里的方法。

PS我正在使用本机方式（JNI），所以怎么做都无所谓..Java或本机，但我需要找到方法。

编辑1

我制作了某种电影（只有一个 3d 模型），所以我每 32 毫秒更改一次几何和纹理。因此，在我看来，将 mp4 用于 tex 似乎是合理的，因为每个新帧（32 毫秒）都与原始帧非常相似......

现在我为一个模型使用 400 帧。对于几何，我使用 .mtr，对于 tex，我使用 .pkm（因为它针对 android 进行了优化），所以我有大约 350 个 .mtr 文件（因为有些文件包含子索引）和 400 个 .pkm 文件......

这就是为什么我要为 tex 使用 mp4 的原因。因为一个 mp4 比 400 .pkm 小得多

编辑2

请看一下 Edit1

实际上，我需要知道的只是可以MP4按帧读取的Android API ？也许某种getNextFrame()方法？

像这样的东西

MP4Player player = new MP4Player(PATH_TO_MY_MP4_FILE);

void readMP4(){
   Bitmap b;

   while(player.hasNext()){
      b = player.getNextFrame();

      ///.... my code here ...///
   }
}

编辑3

我在 Java 上做了这样的实现

public static void read(@NonNull final Context iC, @NonNull final String iPath)
{
    long time;

    int fileCount = 0;

    //Create a new Media Player
    MediaPlayer mp = MediaPlayer.create(iC, Uri.parse(iPath));
    time = mp.getDuration() * 1000;

    Log.e("TAG", String.format("TIME :: %s", time));

    MediaMetadataRetriever mRetriever = new MediaMetadataRetriever();
    mRetriever.setDataSource(iPath);

    long a = System.nanoTime();

    //frame rate 10.03/sec, 1/10.03 = in microseconds 99700
    for (int i = 99700 ; i <= time ; i = i + 99700)
    {
        Bitmap b = mRetriever.getFrameAtTime(i, MediaMetadataRetriever.OPTION_CLOSEST_SYNC);

        if (b == null)
        {
            Log.e("TAG", String.format("BITMAP STATE :: %s", "null"));
        }
        else
        {
            fileCount++;
        }

        long curTime = System.nanoTime();
        Log.e("TAG", String.format("EXECUTION TIME :: %s", curTime - a));
        a = curTime;
    }

    Log.e("TAG", String.format("COUNT :: %s", fileCount));
}

和这里执行时间

  E/TAG: EXECUTION TIME :: 267982039
  E/TAG: EXECUTION TIME :: 222928769
  E/TAG: EXECUTION TIME :: 289899461
  E/TAG: EXECUTION TIME :: 138265423
  E/TAG: EXECUTION TIME :: 127312577
  E/TAG: EXECUTION TIME :: 251179654
  E/TAG: EXECUTION TIME :: 133996500
  E/TAG: EXECUTION TIME :: 289730345
  E/TAG: EXECUTION TIME :: 132158270
  E/TAG: EXECUTION TIME :: 270951461
  E/TAG: EXECUTION TIME :: 116520808
  E/TAG: EXECUTION TIME :: 209071269
  E/TAG: EXECUTION TIME :: 149697230
  E/TAG: EXECUTION TIME :: 138347269

这次以纳秒为单位 == +/- 200 毫秒......它非常缓慢......我需要大约 30 毫秒的帧。

所以，我认为这个方法是在CPU上执行的，所以请问是否有在GPU上执行的方法？

编辑4

我发现有MediaCodec课

https://developer.android.com/reference/android/media/MediaCodec

我也在这里发现了类似的问题MediaCodec 从视频中获取所有帧

我知道有一种方法可以按字节读取，但不能按帧读取...

所以，仍然有问题 - 是否有办法mp4按帧读取视频？

Answer 1

The solution would look something like the ExtractMpegFramesTest, in which MediaCodec is used to generate "external" textures from video frames. In the test code, the frames are rendered to an off-screen pbuffer and then saved as PNG. You would just render them directly.

There are a few problems with this:

1. MPEG video isn't designed to work well as a random-access database. A common GOP (group of pictures) structure has one "key frame" (essentially a JPEG image) followed by 14 delta frames, which just hold the difference from the previous decoded frame. So if you want frame N, you may have to decode frames N-14 through N-1 first. Not a problem if you're always moving forward (playing a movie onto a texture) or you only store key frames (at which point you've invented a clumsy database of JPEG images).
2. As mentioned in comments and answers, you're likely to get some visual artifacts. How bad these look depends on the material and your compression rate. Since you're generating the frames, you may be able to reduce this by ensuring that, whenever there's a big change, the first frame is always a key frame.
3. The firmware that MediaCodec interfaces with may want several frames before it starts producing output, even if you start at a key frame. Seeking around in a stream has a latency cost. See e.g. this post. (Ever wonder why DVRs have smooth fast-forward, but not smooth fast-backward?)
4. MediaCodec frames passed through SurfaceTexture become "external" textures. These have some limitations vs. normal textures -- performance may be worse, can't use as color buffer in an FBO, etc. If you're just rendering it once per frame at 30fps this shouldn't matter.
5. MediaMetadataRetriever's getFrameAtTime() method has less-than-desirable performance for the reasons noted above. You're unlikely to get better results by writing it yourself, although you can save a bit of time by skipping the step where it creates a Bitmap object. Also, you passed OPTION_CLOSEST_SYNC in, but that will only produce the results you want if all your frames are sync frames (again, clumsy database of JPEG images). You need to use OPTION_CLOSEST.

如果您只是想在纹理上播放电影（或者您的问题可以减少到那个程度），Grafika有一些示例。一个可能相关的是 TextureFromCamera，它在可以缩放和旋转的 GLES 矩形上呈现相机视频流。您可以使用其他演示之一中的 MP4 播放代码替换相机输入。如果您只是向前玩，这会很好用，但如果您想跳过或向后走，就会遇到麻烦。

您所描述的问题听起来与 2D 游戏开发人员处理的问题非常相似。做他们所做的可能是最好的方法。