使用Android的MediaCodec和MediaMuxer复用AAC音频

Question

我正在修改一个Android Framework示例,将MediaCodec生成的基本AAC流打包成一个独立的.mp4文件.我正在使用MediaMuxer包含由MediaCodec实例生成的一个AAC轨道的单个实例.

但是,我总是在调用时始终收到错误消息mMediaMuxer.writeSampleData(trackIndex, encodedData, bufferInfo):

E/MPEG4Writer?timestampUs 0 < lastTimestampUs XXXXX for Audio track

当我将原始输入数据排队时,mCodec.queueInputBuffer(...)我提供0作为每个框架示例的时间戳值(我也尝试使用具有相同结果的单调增加的时间戳值.我已成功将原始相机帧编码为h264/mp4文件同样的方法).

查看完整的来源

最相关的片段:

private static void testEncoder(String componentName, MediaFormat format, Context c) {
    int trackIndex = 0;
    boolean mMuxerStarted = false;
    File f = FileUtils.createTempFileInRootAppStorage(c, "aac_test_" + new Date().getTime() + ".mp4");
    MediaCodec codec = MediaCodec.createByCodecName(componentName);

    try {
        codec.configure(
                format,
                null /* surface */,
                null /* crypto */,
                MediaCodec.CONFIGURE_FLAG_ENCODE);
    } catch (IllegalStateException e) {
        Log.e(TAG, "codec '" + componentName + "' failed configuration.");

    }

    codec.start();

    try {
        mMediaMuxer = new MediaMuxer(f.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
    } catch (IOException ioe) {
        throw new RuntimeException("MediaMuxer creation failed", ioe);
    }

    ByteBuffer[] codecInputBuffers = codec.getInputBuffers();
    ByteBuffer[] codecOutputBuffers = codec.getOutputBuffers();

    int numBytesSubmitted = 0;
    boolean doneSubmittingInput = false;
    int numBytesDequeued = 0;

    while (true) {
        int index;

        if (!doneSubmittingInput) {
            index = codec.dequeueInputBuffer(kTimeoutUs /* timeoutUs */);

            if (index != MediaCodec.INFO_TRY_AGAIN_LATER) {
                if (numBytesSubmitted >= kNumInputBytes) {
                    Log.i(TAG, "queueing EOS to inputBuffer");
                    codec.queueInputBuffer(
                            index,
                            0 /* offset */,
                            0 /* size */,
                            0 /* timeUs */,
                            MediaCodec.BUFFER_FLAG_END_OF_STREAM);

                    if (VERBOSE) {
                        Log.d(TAG, "queued input EOS.");
                    }

                    doneSubmittingInput = true;
                } else {
                    int size = queueInputBuffer(
                            codec, codecInputBuffers, index);

                    numBytesSubmitted += size;

                    if (VERBOSE) {
                        Log.d(TAG, "queued " + size + " bytes of input data.");
                    }
                }
            }
        }

        MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
        index = codec.dequeueOutputBuffer(info, kTimeoutUs /* timeoutUs */);

        if (index == MediaCodec.INFO_TRY_AGAIN_LATER) {
        } else if (index == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
            MediaFormat newFormat = codec.getOutputFormat();
            trackIndex = mMediaMuxer.addTrack(newFormat);
            mMediaMuxer.start();
            mMuxerStarted = true;
        } else if (index == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
            codecOutputBuffers = codec.getOutputBuffers();
        } else {
            // Write to muxer
            ByteBuffer encodedData = codecOutputBuffers[index];
            if (encodedData == null) {
                throw new RuntimeException("encoderOutputBuffer " + index +
                        " was null");
            }

            if ((info.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
                // The codec config data was pulled out and fed to the muxer when we got
                // the INFO_OUTPUT_FORMAT_CHANGED status.  Ignore it.
                if (VERBOSE) Log.d(TAG, "ignoring BUFFER_FLAG_CODEC_CONFIG");
                info.size = 0;
            }

            if (info.size != 0) {
                if (!mMuxerStarted) {
                    throw new RuntimeException("muxer hasn't started");
                }

                // adjust the ByteBuffer values to match BufferInfo (not needed?)
                encodedData.position(info.offset);
                encodedData.limit(info.offset + info.size);

                mMediaMuxer.writeSampleData(trackIndex, encodedData, info);
                if (VERBOSE) Log.d(TAG, "sent " + info.size + " audio bytes to muxer with pts " + info.presentationTimeUs);
            }

            codec.releaseOutputBuffer(index, false);

            // End write to muxer
            numBytesDequeued += info.size;

            if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
                if (VERBOSE) {
                    Log.d(TAG, "dequeued output EOS.");
                }
                break;
            }

            if (VERBOSE) {
                Log.d(TAG, "dequeued " + info.size + " bytes of output data.");
            }
        }
    }

    if (VERBOSE) {
        Log.d(TAG, "queued a total of " + numBytesSubmitted + "bytes, "
                + "dequeued " + numBytesDequeued + " bytes.");
    }

    int sampleRate = format.getInteger(MediaFormat.KEY_SAMPLE_RATE);
    int channelCount = format.getInteger(MediaFormat.KEY_CHANNEL_COUNT);
    int inBitrate = sampleRate * channelCount * 16;  // bit/sec
    int outBitrate = format.getInteger(MediaFormat.KEY_BIT_RATE);

    float desiredRatio = (float)outBitrate / (float)inBitrate;
    float actualRatio = (float)numBytesDequeued / (float)numBytesSubmitted;

    if (actualRatio < 0.9 * desiredRatio || actualRatio > 1.1 * desiredRatio) {
        Log.w(TAG, "desiredRatio = " + desiredRatio
                + ", actualRatio = " + actualRatio);
    }


    codec.release();
    mMediaMuxer.stop();
    mMediaMuxer.release();
    codec = null;
}

更新:我发现我认为存在根本症状MediaCodec.:

我发presentationTimeUs=1000来queueInputBuffer(...),但收到info.presentationTimeUs= 33219呼叫后MediaCodec.dequeueOutputBuffer(info, timeoutUs).fadden留下了与此行为相关的有用评论.

Answer 1

感谢fadden的帮助,我在Github上有一个概念验证音频编码器和视频+音频编码器.综上所述:

将AudioRecord样本发送到MediaCodec+ MediaMuxer包装器.使用系统时间audioRecord.read(...)足够好作为音频时间戳,只要您经常轮询以避免填充AudioRecord的内部缓冲区(以避免在您调用读取的时间和AudioRecord记录样本的时间之间漂移).太糟糕的AudioRecord没有直接传达时间戳......

// Setup AudioRecord
while (isRecording) {
    audioPresentationTimeNs = System.nanoTime();
    audioRecord.read(dataBuffer, 0, samplesPerFrame);
    hwEncoder.offerAudioEncoder(dataBuffer.clone(), audioPresentationTimeNs);
}

请注意,AudioRecord 仅保证支持16位PCM采样,但MediaCodec.queueInputBuffer输入为byte[].传递一个byte[]到audioRecord.read(dataBuffer,...)会截断分离的16个采样到8位给你.

我发现以这种方式进行轮询仍然偶尔会产生timestampUs XXX < lastTimestampUs XXX for Audio track错误,所以我提供了一些逻辑来跟踪bufferInfo.presentationTimeUs报告,mediaCodec.dequeueOutputBuffer(bufferInfo, timeoutMs)并在调用之前根据需要进行调整mediaMuxer.writeSampleData(trackIndex, encodedData, bufferInfo).