Problem setting video frame rate using AVAssetWriter/AVAssetReader

Question

Situation:

I am trying to export video with some parameters like video bit rate, audio bit rate, frame rate, changing video resolution, etc. Note that I am letting the user set the video frame rate in fractions; like user can set the video frame rate say, 23.98.

I use AVAssetWriter and AVAssetReader for this operation. I use AVAssetWriterInputPixelBufferAdaptor for writing the sample buffers.

Everything else works just fine, except: the video frame rate.

What I have tried:

1. Setting the AVAssetWriter.movieTimeScale as suggested here. Which does change the video frame rate, but also make the video sluggish. (gist here)

---

2. Setting AVVideoExpectedSourceFrameRateKey. Which does not help. (gist here)

---

3. Setting AVAssetWriterInput.mediaTimeScale. Again, it changes the video frame rate, but makes the video sluggish like AVAssetWriter.movieTimeScale does. The video shows different frame at some point and sometimes it sticks and resumes again. (gist here)

---

4. Using AVAssetReaderVideoCompositionOutput and setting AVMutableVideoComposition.frameDuration; just like SDAVAssetExportSession does. Ironically with SDAVAssetExportSession code, the video is being exported just at the right frame rate what I want, but it just does not work in my code. gist here

I am not sure why it won't work with my code. The issue with this approach is it always returns nil from AVAssetReaderVideoCompositionOutput.copyNextSampleBuffer().

---

5. Manually changing the frames timestamp with CMSampleTimingInfo, as suggested here Something like:

var sampleTimingInfo = CMSampleTimingInfo()
var sampleBufferToWrite: CMSampleBuffer?

CMSampleBufferGetSampleTimingInfo(vBuffer, at: 0, timingInfoOut: &sampleTimingInfo)

sampleTimingInfo.duration = CMTimeMake(value: 100, timescale: Int32(videoConfig.videoFrameRate * 100))

sampleTimingInfo.presentationTimeStamp = CMTimeAdd(previousPresentationTimeStamp, sampleTimingInfo.duration)

previousPresentationTimeStamp = sampleTimingInfo.presentationTimeStamp

let status = CMSampleBufferCreateCopyWithNewTiming(allocator: kCFAllocatorDefault, sampleBuffer: vBuffer,sampleTimingEntryCount: 1, sampleTimingArray: &sampleTimingInfo, sampleBufferOut: &sampleBufferToWrite)

With this approach, I do get the frame rate set just right, but it increases the video duration (as mentioned in the comment of that question’s answer). I think at some point I may have to discard some frames (if the target frame rate is lower; I need to lower the frame rate in most of the cases).

If I know that if I want 30fps, and my current frame rate is 60fps, its simple to discard every second frame and setting the SampleBuffer time accordingly.

If I go with this approach(i.e. setting 23.98 fps), how do I decide which frame to discard and if the target frame rate is higher, which frame to duplicate? Reminder: the frame rate could be in fractions.

---

Answer 1

这是选择框架的想法。假设源视频的fps为F，目标fps为TF。比率=TF/F

初始化一个等于-rate的变量n，每次添加rate，当n的整数部分发生变化时，选择该帧。

e.g. rate = 0.3
          n: -0.3 0 0.3 0.6 0.9 1.2 1.5 1.8 2.1
                  ^              ^           ^
frame index:      0  1   2   3   4   5   6   7
select 0 4 7

float rate = 0.39999f; // TF/F 
float n =  -rate; // to make sure first frame will be selected
for (int i = 0; i < 100; ++i, n += rate) { // i stands for frame index, take a video with 100 frames as an example
    int m = floor(n);
    int tmp = n+rate;
    // if rate > 1.0 repeat i
    // if rate < 1.0 some of the frames will be dropped
    for (int j = 0; m+j < tmp; ++j) {
        // Use this frame
        printf("%d ", i);
    }
}