如何从AVCapturePhoto生成具有正确方向的UIImage？

Question

我正在调用AVFoundation的委托方法来处理照片捕获,但是我很难将它生成的AVCapturePhoto转换为具有正确方向的UIImage.虽然下面的例程是成功的,但我总是得到一个面向右侧的UIImage(UIImage.imageOrientation = 3).使用UIImage(数据:图像)并尝试首先使用photo.cgImageRepresentation()时,我无法提供方向？takeRetainedValue()也没有帮助.请协助.

在此处,图像方向至关重要,因为生成的图像将被输入Vision框架工作流程.

func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {
    // capture image finished
    print("Image captured.")
    if let imageData = photo.fileDataRepresentation() {
        if let uiImage = UIImage(data: imageData){
            // do stuff to UIImage
        }
    }
}

更新1: 阅读Apple的Photo Capture编程指南(iOS11过时),我确实找到了一件我做错的事情:

1. 在每次捕获调用(self.capturePhotoOutput.capturePhoto)时,必须与PhotoOutput对象建立连接并更新其方向以匹配拍摄照片时设备的方向.为此,我创建了UIDeviceOrientation的扩展,并在我创建的snapPhoto()函数上使用它来调用捕获例程并等待执行didFinishProcessingPhoto委托方法.我添加了代码的快照,因为这里的代码示例分隔符似乎没有正确显示它们.

更新2 链接到GitHub上的完整项目:https://github.com/agu3rra/Out-Loud

Answer 1

最后更新: 我使用该应用程序进行了一些实验并得出以下结论:

1. kCGImagePropertyOrientation似乎不会影响应用程序内捕获图像的方向,只有在每次要调用capturePhoto方法时更新photoOutput连接时,它才会因设备方向而异.所以:

   func snapPhoto() {
       // prepare and initiate image capture routine
   
       // if I leave the next 4 lines commented, the intented orientation of the image on display will be 6 (right top) - kCGImagePropertyOrientation
       let deviceOrientation = UIDevice.current.orientation // retrieve current orientation from the device
       guard let photoOutputConnection = capturePhotoOutput.connection(with: AVMediaType.video) else {fatalError("Unable to establish input>output connection")}// setup a connection that manages input > output
       guard let videoOrientation = deviceOrientation.getAVCaptureVideoOrientationFromDevice() else {return}
       photoOutputConnection.videoOrientation = videoOrientation // update photo's output connection to match device's orientation
   
       let photoSettings = AVCapturePhotoSettings()
       photoSettings.isAutoStillImageStabilizationEnabled = true
       photoSettings.isHighResolutionPhotoEnabled = true
       photoSettings.flashMode = .auto
       self.capturePhotoOutput.capturePhoto(with: photoSettings, delegate: self) // trigger image capture. It appears to work only if the capture session is running.
   }
   

2. 在调试器上查看生成的图像向我展示了它们是如何生成的,因此我可以推断出所需的旋转(UIImageOrientation)以使其直立显示.换句话说:更新UIImageOrientation会告诉您应该如何旋转图像,以便您以正确的方向查看图像.所以我来到下表:

3. 我不得不将我的UIDeviceOrientation扩展更新为一个相当不直观的形式:

   extension UIDeviceOrientation {
       func getUIImageOrientationFromDevice() -> UIImageOrientation {
           // return CGImagePropertyOrientation based on Device Orientation
           // This extented function has been determined based on experimentation with how an UIImage gets displayed.
           switch self {
           case UIDeviceOrientation.portrait, .faceUp: return UIImageOrientation.right
           case UIDeviceOrientation.portraitUpsideDown, .faceDown: return UIImageOrientation.left
           case UIDeviceOrientation.landscapeLeft: return UIImageOrientation.up // this is the base orientation
           case UIDeviceOrientation.landscapeRight: return UIImageOrientation.down
           case UIDeviceOrientation.unknown: return UIImageOrientation.up
           }
       }
   }
   

4. 这就是我的最终委托方法现在的样子.它以预期的方向显示图像.

   func photoOutput(_ output: AVCapturePhotoOutput,
                                    didFinishProcessingPhoto photo: AVCapturePhoto,
                                    error: Error?)
   {
       // capture image finished
       print("Image captured.")
   
       let photoMetadata = photo.metadata
       // Returns corresponting NSCFNumber. It seems to specify the origin of the image
       //                print("Metadata orientation: ",photoMetadata["Orientation"])
   
       // Returns corresponting NSCFNumber. It seems to specify the origin of the image
       print("Metadata orientation with key: ",photoMetadata[String(kCGImagePropertyOrientation)] as Any)
   
       guard let imageData = photo.fileDataRepresentation() else {
           print("Error while generating image from photo capture data.");
           self.lastPhoto = nil; self.controller.goToProcessing();
           return
   
       }
   
       guard let uiImage = UIImage(data: imageData) else {
           print("Unable to generate UIImage from image data.");
           self.lastPhoto = nil; self.controller.goToProcessing();
           return
   
       }
   
       // generate a corresponding CGImage
       guard let cgImage = uiImage.cgImage else {
           print("Error generating CGImage");self.lastPhoto=nil;return
   
       }
   
       guard let deviceOrientationOnCapture = self.deviceOrientationOnCapture else {
           print("Error retrieving orientation on capture");self.lastPhoto=nil;
           return
   
       }
   
       self.lastPhoto = UIImage(cgImage: cgImage, scale: 1.0, orientation: deviceOrientationOnCapture.getUIImageOrientationFromDevice())
   
       print(self.lastPhoto)
       print("UIImage generated. Orientation:(self.lastPhoto.imageOrientation.rawValue)")
       self.controller.goToProcessing()
   }
   
   
   func photoOutput(_ output: AVCapturePhotoOutput, 
                      willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) 
                      {
       print("Just about to take a photo.")
       // get device orientation on capture
       self.deviceOrientationOnCapture = UIDevice.current.orientation
       print("Device orientation: \(self.deviceOrientationOnCapture.rawValue)")
   }
   

Answer 2

我已经成功地做到了这一点：

func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {

        let cgImage = photo.cgImageRepresentation()!.takeRetainedValue()
        let orientation = photo.metadata[kCGImagePropertyOrientation as String] as! NSNumber
        let uiOrientation = UIImage.Orientation(rawValue: orientation.intValue)!
        let image = UIImage(cgImage: cgImage, scale: 1, orientation: uiOrientation)

}

它基于苹果在其文档中提到的内容：

每次访问此方法时，AVCapturePhoto 都会生成一个新的 CGImageRef。当由压缩容器（例如 HEIC）支持时，CGImageRepresentation 会根据需要进行延迟解码。当由未压缩格式（例如 BGRA）支持时，它会被复制到单独的后备缓冲区中，其生命周期与 AVCapturePhoto 的生命周期无关。对于 12 兆像素的图像，BGRA CGImage 代表每次调用约 48 兆字节。如果您只想使用 CGImage 进行屏幕渲染，请改用 PreviewCGImageRepresentation。请注意，CGImageRef 的物理旋转与主图像的物理旋转相匹配。尚未应用 Exif 方向。如果您希望在使用 UIImage 时应用旋转，可以通过查询照片的元数据 [kCGImagePropertyOrientation] 值，并将其作为方向参数传递给 +[UIImage imageWithCGImage:scale:orientation:] 来实现。RAW 图像始终返回 nil 的 CGImageRepresentation。如果您希望从 RAW 图像制作 CGImageRef，请使用 CoreImage 框架中的 CIRAWFilter。

Answer 3

为了创建具有正确方向的图像，我们需要UIImage.Orientation在初始化图像时输入正确的方向。

最好使用CGImagePropertyOrientation从 photoOutput 委托返回的 来获取拍摄照片时相机会话所处的准确方向。UIImage.Orientation这里唯一的问题是，虽然和之间的枚举值CGImagePropertyOrientation相同，但原始值却不同。苹果建议使用一个简单的映射来解决这个问题。

https://developer.apple.com/documentation/imageio/cgimagepropertyorientation

这是我的实现：

AVCapturePhotoCaptureDelegate

func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {
        if let _ = error {
            // Handle Error
        } else if let cgImageRepresentation = photo.cgImageRepresentation(),
            let orientationInt = photo.metadata[String(kCGImagePropertyOrientation)] as? UInt32,
            let imageOrientation = UIImage.Orientation.orientation(fromCGOrientationRaw: orientationInt) {

            // Create image with proper orientation
            let cgImage = cgImageRepresentation.takeUnretainedValue()
            let image = UIImage(cgImage: cgImage,
                                scale: 1,
                                orientation: imageOrientation)
        }
    }

映射扩展

extension UIImage.Orientation {

    init(_ cgOrientation: CGImagePropertyOrientation) {
        // we need to map with enum values becuase raw values do not match
        switch cgOrientation {
        case .up: self = .up
        case .upMirrored: self = .upMirrored
        case .down: self = .down
        case .downMirrored: self = .downMirrored
        case .left: self = .left
        case .leftMirrored: self = .leftMirrored
        case .right: self = .right
        case .rightMirrored: self = .rightMirrored
        }
    }


    /// Returns a UIImage.Orientation based on the matching cgOrientation raw value
    static func orientation(fromCGOrientationRaw cgOrientationRaw: UInt32) -> UIImage.Orientation? {
        var orientation: UIImage.Orientation?
        if let cgOrientation = CGImagePropertyOrientation(rawValue: cgOrientationRaw) {
            orientation = UIImage.Orientation(cgOrientation)
        } else {
            orientation = nil // only hit if improper cgOrientation is passed
        }
        return orientation
    }
}

Answer 4

在AVCapturePhotoI\xe2\x80\x99m 中，你肯定会找到一个metadata也称为CGImageProperties.
\n在其中您将找到用于方向的 EXIF 字典，下一步就是获取方向并根据该方向创建图像。
\n我没有使用经验AVCapturePhotoOutput，但我有一些使用旧方法。
\n注意 EXIF 字典在 UIImageOrientation 中的映射不同。
\n这是我很久以前写的一篇文章，但主要原理仍然有效。
\n这个问题将向您指出一些实现，它也很旧，我很确定在最新版本中他们发布了更简单的 API，但它仍然会引导您解决这个问题。

\n