自定义AVVideoCompositing类无法按预期工作

Question

我正在尝试将CIFilter应用于AVAsset,然后在应用过滤器的情况下保存它.我这样做的方法是使用AVAssetExportSessionwith videoCompositionset设置为AVMutableVideoComposition具有自定义AVVideoCompositing类的对象.

我还将instructions我的AVMutableVideoComposition对象设置为自定义组合指令类(符合AVMutableVideoCompositionInstruction).这个类传递了一个跟踪ID,以及一些其他不重要的变量.

不幸的是,我遇到了一个问题 - startVideoCompositionRequest:我的自定义视频合成器类(符合AVVideoCompositing)中的函数没有被正确调用.

当我将passthroughTrackID自定义指令类的变量设置为轨道ID时,startVideoCompositionRequest(request)我的函数AVVideoCompositing不会被调用.

然而,当我没有设置passthroughTrackID我的自定义指令类的变量,将startVideoCompositionRequest(request) 被调用,但没有正确-打印request.sourceTrackIDs结果空数组,并request.sourceFrameByTrackID(trackID)导致零值.

我发现有趣的是,cancelAllPendingVideoCompositionRequests:在尝试使用过滤器导出视频时,该函数总是被调用两次.startVideoCompositionRequest:在startVideoCompositionRequest:未调用的情况下,它要么被调用一次,要么被调用一次,或者连续两次被调用.

我创建了三个用于导出带过滤器的视频的类.这是实用程序类,它基本上只包含一个export函数并调用所有必需的代码

class VideoFilterExport{

    let asset: AVAsset
    init(asset: AVAsset){
        self.asset = asset
    }

    func export(toURL url: NSURL, callback: (url: NSURL?) -> Void){
        guard let track: AVAssetTrack = self.asset.tracksWithMediaType(AVMediaTypeVideo).first else{callback(url: nil); return}

        let composition = AVMutableComposition()
        let compositionTrack = composition.addMutableTrackWithMediaType(AVMediaTypeVideo, preferredTrackID: kCMPersistentTrackID_Invalid)

        do{
            try compositionTrack.insertTimeRange(track.timeRange, ofTrack: track, atTime: kCMTimeZero)
        }
        catch _{callback(url: nil); return}

        let videoComposition = AVMutableVideoComposition(propertiesOfAsset: composition)
        videoComposition.customVideoCompositorClass = VideoFilterCompositor.self
        videoComposition.frameDuration = CMTimeMake(1, 30)
        videoComposition.renderSize = compositionTrack.naturalSize

        let instruction = VideoFilterCompositionInstruction(trackID: compositionTrack.trackID)
        instruction.timeRange = CMTimeRangeMake(kCMTimeZero, self.asset.duration)
        videoComposition.instructions = [instruction]

        let session: AVAssetExportSession = AVAssetExportSession(asset: composition, presetName: AVAssetExportPresetMediumQuality)!
        session.videoComposition = videoComposition
        session.outputURL = url
        session.outputFileType = AVFileTypeMPEG4

        session.exportAsynchronouslyWithCompletionHandler(){
            callback(url: url)
        }
    }
}

这是另外两个类 - 我将它们放在一个代码块中,以使这篇文章更短

// Video Filter Composition Instruction Class - from what I gather,
// AVVideoCompositionInstruction is used only to pass values to
// the AVVideoCompositing class

class VideoFilterCompositionInstruction : AVMutableVideoCompositionInstruction{

    let trackID: CMPersistentTrackID
    let filters: ImageFilterGroup
    let context: CIContext


    // When I leave this line as-is, startVideoCompositionRequest: isn't called.
    // When commented out, startVideoCompositionRequest(request) is called, but there
    // are no valid CVPixelBuffers provided by request.sourceFrameByTrackID(below value)
    override var passthroughTrackID: CMPersistentTrackID{get{return self.trackID}}
    override var requiredSourceTrackIDs: [NSValue]{get{return []}}
    override var containsTweening: Bool{get{return false}}


    init(trackID: CMPersistentTrackID, filters: ImageFilterGroup, context: CIContext){
        self.trackID = trackID
        self.filters = filters
        self.context = context

        super.init()

        //self.timeRange = timeRange
        self.enablePostProcessing = true
    }

    required init?(coder aDecoder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }

}


// My custom AVVideoCompositing class. This is where the problem lies -
// although I don't know if this is the root of the problem

class VideoFilterCompositor : NSObject, AVVideoCompositing{

    var requiredPixelBufferAttributesForRenderContext: [String : AnyObject] = [
        kCVPixelBufferPixelFormatTypeKey as String : NSNumber(unsignedInt: kCVPixelFormatType_32BGRA), // The video is in 32 BGRA
        kCVPixelBufferOpenGLESCompatibilityKey as String : NSNumber(bool: true),
        kCVPixelBufferOpenGLCompatibilityKey as String : NSNumber(bool: true)
    ]
    var sourcePixelBufferAttributes: [String : AnyObject]? = [
        kCVPixelBufferPixelFormatTypeKey as String : NSNumber(unsignedInt: kCVPixelFormatType_32BGRA),
        kCVPixelBufferOpenGLESCompatibilityKey as String : NSNumber(bool: true),
        kCVPixelBufferOpenGLCompatibilityKey as String : NSNumber(bool: true)
    ]

    let renderQueue = dispatch_queue_create("co.getblix.videofiltercompositor.renderingqueue", DISPATCH_QUEUE_SERIAL)

    override init(){
        super.init()
    }

    func startVideoCompositionRequest(request: AVAsynchronousVideoCompositionRequest){
       // This code block is never executed when the
       // passthroughTrackID variable is in the above class  

        autoreleasepool(){
            dispatch_async(self.renderQueue){
                guard let instruction = request.videoCompositionInstruction as? VideoFilterCompositionInstruction else{
                    request.finishWithError(NSError(domain: "getblix.co", code: 760, userInfo: nil))
                    return
                }
                guard let pixels = request.sourceFrameByTrackID(instruction.passthroughTrackID) else{
                    // This code block is executed when I comment out the
                    // passthroughTrackID variable in the above class            

                    request.finishWithError(NSError(domain: "getblix.co", code: 761, userInfo: nil))
                    return
                }
                // I have not been able to get the code to reach this point
                // This function is either not called, or the guard
                // statement above executes

                let image = CIImage(CVPixelBuffer: pixels)
                let filtered: CIImage = //apply the filter here

                let width = CVPixelBufferGetWidth(pixels)
                let height = CVPixelBufferGetHeight(pixels)
                let format = CVPixelBufferGetPixelFormatType(pixels)

                var newBuffer: CVPixelBuffer?
                CVPixelBufferCreate(kCFAllocatorDefault, width, height, format, nil, &newBuffer)

                if let buffer = newBuffer{
                    instruction.context.render(filtered, toCVPixelBuffer: buffer)
                    request.finishWithComposedVideoFrame(buffer)
                }
                else{
                    request.finishWithComposedVideoFrame(pixels)
                }
            }
        }
    }

    func renderContextChanged(newRenderContext: AVVideoCompositionRenderContext){
        // I don't have any code in this block
    }

    // This is interesting - this is called twice,
    // Once before startVideoCompositionRequest is called,
    // And once after. In the case when startVideoCompositionRequest
    // Is not called, this is simply called twice in a row
    func cancelAllPendingVideoCompositionRequests(){
        dispatch_barrier_async(self.renderQueue){
            print("Cancelled")
        }
    }
}

我一直在寻找Apple的AVCustomEdit示例项目,以获得相关指导,但我似乎无法找到为什么会发生这种情况.

我怎样才能使request.sourceFrameByTrackID:函数正确调用,并CVPixelBuffer为每个帧提供有效？

Answer 1

事实证明,requiredSourceTrackIDs自定义AVVideoCompositionInstruction类(VideoFilterCompositionInstruction在问题中)中的变量必须设置为包含轨道ID的数组

override var requiredSourceTrackIDs: [NSValue]{
  get{
    return [
      NSNumber(value: Int(self.trackID))
    ]
  }
}

所以最终的自定义组合指令类是

class VideoFilterCompositionInstruction : AVMutableVideoCompositionInstruction{
    let trackID: CMPersistentTrackID
    let filters: [CIFilter]
    let context: CIContext

    override var passthroughTrackID: CMPersistentTrackID{get{return self.trackID}}
    override var requiredSourceTrackIDs: [NSValue]{get{return [NSNumber(value: Int(self.trackID))]}}
    override var containsTweening: Bool{get{return false}}

    init(trackID: CMPersistentTrackID, filters: [CIFilter], context: CIContext){
        self.trackID = trackID
        self.filters = filters
        self.context = context

        super.init()

        self.enablePostProcessing = true
    }

    required init?(coder aDecoder: NSCoder){
        fatalError("init(coder:) has not been implemented")
    }
}

此实用程序的所有代码也在GitHub上

Answer 2

正如您所指出的那样,passthroughTrackID返回要过滤的轨道并不是正确的方法 - 您需要返回要过滤的轨道requiredSourceTrackIDs.(看起来,一旦你这样做,如果你也从它那里返回它并不重要passthroughTrackID.)回答剩下的问题,为什么它以这种方式工作......

对于文档passthroughTrackID和requiredSourceTrackIDs肯定不是苹果有史以来最清晰的写作.(提交一个关于它的错误,它们可能会有所改进.)但如果仔细观察前者的描述,就会有一个提示(强调增加)......

如果在指令的持续时间内,视频合成结果是源帧之一,则此属性返回相应的轨道ID.合成器将不会在指令持续时间内运行,而是使用正确的源帧.

因此,passthroughTrackID 只有当您创建的指令类在没有处理的情况下通过单个轨道时才使用.

如果您计划执行任何图像处理,即使它只是一个没有合成的轨道,也可以指定该轨道requiredSourceTrackIDs.