如何在arkit上添加黑白过滤器(swift4)

Question

我想做的就是采用基本的arkit视图并将其转换为黑白视图.现在基本视图是正常的,我不知道如何添加过滤器.理想情况下,在截取屏幕截图时,黑白过滤器会添加到屏幕截图中.

import UIKit
import SceneKit
import ARKit

class ViewController: UIViewController, ARSCNViewDelegate {

    @IBOutlet var sceneView: ARSCNView!

    override func viewDidLoad() {
        super.viewDidLoad()
        sceneView.delegate = self
        sceneView.showsStatistics = true
    }

    override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)
        let configuration = ARWorldTrackingConfiguration()
        sceneView.session.run(configuration)
    }

    override func viewWillDisappear(_ animated: Bool) {
        super.viewWillDisappear(animated)
        sceneView.session.pause()
    }

    @IBAction func changeTextColour(){
        let snapShot = self.augmentedRealityView.snapshot()
        UIImageWriteToSavedPhotosAlbum(snapShot, self, #selector(image(_:didFinishSavingWithError:contextInfo:)), nil)
    }
}

Answer 1

如果要实时应用过滤器,最好的方法就是使用SCNTechnique.技术用于后处理,并允许我们SCNView在几个过程中呈现内容 - 正是我们需要的(首先渲染场景,然后对其应用效果).

这是示例项目.

Plist设置

首先,我们需要在.plist文件中描述一种技术.

这plist是我想出的一个截图(为了更好的可视化):

这是它的来源:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>sequence</key>
    <array>
        <string>apply_filter</string>
    </array>
    <key>passes</key>
    <dict>
        <key>apply_filter</key>
        <dict>
            <key>metalVertexShader</key>
            <string>scene_filter_vertex</string>
            <key>metalFragmentShader</key>
            <string>scene_filter_fragment</string>
            <key>draw</key>
            <string>DRAW_QUAD</string>
            <key>inputs</key>
            <dict>
                <key>scene</key>
                <string>COLOR</string>
            </dict>
            <key>outputs</key>
            <dict>
                <key>color</key>
                <string>COLOR</string>
            </dict>
        </dict>
    </dict>
</dict>

SCNTechniques 的主题是广泛的,我只会快速介绍我们手头的案例.为了真正了解他们的能力,我建议您阅读Apple关于技术的综合文档.

技术说明

passes是一个字典,包含您想要SCNTechnique执行的过程的描述.

sequence 是一个数组,指定使用其键执行这些传递的顺序.

你没有在这里指定主渲染通道(意思是在不应用SCNTechniques的情况下渲染的任何东西) - 它是隐含的,并且它的结果颜色可以使用COLOR常量访问(稍微多一点).

因此,我们要做的唯一"额外"传递(除了主要传递)将是apply_filter将颜色转换为黑白(它可以命名为任何你想要的,只需确保它具有相同的键passes和sequence).

现在来说明apply_filter传球本身.

渲染传递说明

metalVertexShader和metalFragmentShader- Metal将用于绘图的着色器函数的名称.

draw定义传递将要呈现的内容.DRAW_QUAD代表:

仅渲染覆盖视图整个边界的矩形.使用此选项绘制处理先前传递输出的图像缓冲区的传递.

粗略地说,这意味着我们将使用渲染过程渲染一个简单的"图像".

inputs指定我们将能够在着色器中使用的输入资源.正如我之前所说,COLOR指的是主渲染通道提供的颜色数据.

outputs指定输出.它可以是color,depth或者stencil,但我们只需要一个color输出.COLOR值意味着我们简单地说,将"直接"渲染到屏幕上(而不是渲染到中间目标中).

金属着色器

创建.metal包含以下内容的文件:

#include <metal_stdlib>
using namespace metal;
#include <SceneKit/scn_metal>

struct VertexInput {
    float4 position [[ attribute(SCNVertexSemanticPosition) ]];
    float2 texcoord [[ attribute(SCNVertexSemanticTexcoord0) ]];
};

struct VertexOut {
    float4 position [[position]];
    float2 texcoord;
};

// metalVertexShader
vertex VertexOut scene_filter_vertex(VertexInput in [[stage_in]])
{
    VertexOut out;
    out.position = in.position;
    out.texcoord = float2((in.position.x + 1.0) * 0.5 , (in.position.y + 1.0) * -0.5);
    return out;
}

// metalFragmentShader
fragment half4 scene_filter_fragment(VertexOut vert [[stage_in]],
                                    texture2d<half, access::sample> scene [[texture(0)]])
{
    constexpr sampler samp = sampler(coord::normalized, address::repeat, filter::nearest);
    constexpr half3 weights = half3(0.2126, 0.7152, 0.0722);

    half4 color = scene.sample(samp, vert.texcoord);
    color.rgb = half3(dot(color.rgb, weights));

    return color;
}

请注意,片段和顶点着色器的函数名称应plist与传递描述符中的文件中指定的名称相同.

要更好地了解内容VertexInput和VertexOut结构的含义,请参阅SCNProgram文档.

给定的顶点函数可以在任何DRAW_QUAD渲染过程中使用.它基本上为我们提供了屏幕空间的标准化坐标(可以vert.texcoord在片段着色器中访问).

片段功能是所有"魔法"发生的地方.在那里,你可以操纵你从主要传递中获得的纹理.使用此设置,您可以实现大量过滤器/效果等.

在我们的例子中,我使用了基本的去饱和度(零饱和度)公式来获得黑白颜色.

Swift设置

现在,我们终于可以在ARKit/中使用所有这些了SceneKit.

let plistName = "SceneFilterTechnique" // the name of the plist you've created

guard let url = Bundle.main.url(forResource: plistName, withExtension: "plist") else {
    fatalError("\(plistName).plist does not exist in the main bundle")
}

guard let dictionary = NSDictionary(contentsOf: url) as? [String: Any] else {
    fatalError("Failed to parse \(plistName).plist as a dictionary")
}

guard let technique = SCNTechnique(dictionary: dictionary) else {
    fatalError("Failed to initialize a technique using \(plistName).plist")
}

而只是将其设置为technique的ARSCNView.

sceneView.technique = technique

而已.现在整个场景将以灰度渲染,包括拍摄快照时.

Answer 2

过滤ARSCNView快照：如果要创建黑白屏幕，ARSCNView可以执行以下操作，该操作返回UIImageGrayScale并由此augmentedRealityView引用一个ARSCNView：

/// Converts A UIImage To A High Contrast GrayScaleImage
///
/// - Returns: UIImage
func highContrastBlackAndWhiteFilter() -> UIImage?
{
    //1. Convert It To A CIIamge
    guard let convertedImage = CIImage(image: self) else { return nil }

    //2. Set The Filter Parameters
    let filterParameters = [kCIInputBrightnessKey: 0.0,
                            kCIInputContrastKey:   1.1,
                            kCIInputSaturationKey: 0.0]

    //3. Apply The Basic Filter To The Image
    let imageToFilter = convertedImage.applyingFilter("CIColorControls", parameters: filterParameters)

    //4. Set The Exposure
    let exposure =  [kCIInputEVKey: NSNumber(value: 0.7)]

    //5. Process The Image With The Exposure Setting
    let processedImage = imageToFilter.applyingFilter("CIExposureAdjust", parameters: exposure)

    //6. Create A CG GrayScale Image
    guard let grayScaleImage = CIContext().createCGImage(processedImage, from: processedImage.extent) else { return nil }

    return UIImage(cgImage: grayScaleImage, scale: self.scale, orientation: self.imageOrientation)
}

因此，使用此示例如下所示：

 override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {

    //1. Create A UIImageView Dynamically
    let imageViewResult = UIImageView(frame: CGRect(x: 0, y: 0, width: self.view.bounds.width, height: self.view.bounds.height))
    self.view.addSubview(imageViewResult)

    //2. Create The Snapshot & Get The Black & White Image
    guard let snapShotImage = self.augmentedRealityView.snapshot().highContrastBlackAndWhiteFilter() else { return }
    imageViewResult.image = snapShotImage

    //3. Remove The ImageView After A Delay Of 5 Seconds
    DispatchQueue.main.asyncAfter(deadline: .now() + 5) {
        imageViewResult.removeFromSuperview()
    }

}

这将产生如下结果：

为了使您的代码可重用，您还可以创建一个extensionUIImage：

//------------------------
//MARK: UIImage Extensions
//------------------------

extension UIImage
{

    /// Converts A UIImage To A High Contrast GrayScaleImage
    ///
    /// - Returns: UIImage
    func highContrastBlackAndWhiteFilter() -> UIImage?
    {
        //1. Convert It To A CIIamge
        guard let convertedImage = CIImage(image: self) else { return nil }

        //2. Set The Filter Parameters
        let filterParameters = [kCIInputBrightnessKey: 0.0,
                                kCIInputContrastKey:   1.1,
                                kCIInputSaturationKey: 0.0]

        //3. Apply The Basic Filter To The Image
        let imageToFilter = convertedImage.applyingFilter("CIColorControls", parameters: filterParameters)

        //4. Set The Exposure
        let exposure =  [kCIInputEVKey: NSNumber(value: 0.7)]

        //5. Process The Image With The Exposure Setting
        let processedImage = imageToFilter.applyingFilter("CIExposureAdjust", parameters: exposure)

        //6. Create A CG GrayScale Image
        guard let grayScaleImage = CIContext().createCGImage(processedImage, from: processedImage.extent) else { return nil }

        return UIImage(cgImage: grayScaleImage, scale: self.scale, orientation: self.imageOrientation)
    }

}

然后您可以轻松使用它，如下所示：

guard let snapShotImage = self.augmentedRealityView.snapshot().highContrastBlackAndWhiteFilter() else { return }

记住，您应该将扩展名放在class declaration例如：

extension UIImage{

}

class ViewController: UIViewController, ARSCNViewDelegate {

}

因此，根据您的问题中提供的代码，您将获得以下内容：

/// Creates A Black & White ScreenShot & Saves It To The Photo Album
@IBAction func changeTextColour(){

    //1. Create A Snapshot
    guard let snapShotImage = self.augmentedRealityView.snapshot().highContrastBlackAndWhiteFilter() else { return }

    //2. Save It The Photos Album
    UIImageWriteToSavedPhotosAlbum(snapShotImage, self, #selector(image(_:didFinishSavingWithError:contextInfo:)), nil)

}

///Calback To Check Whether The Image Has Been Saved
@objc func image(_ image: UIImage, didFinishSavingWithError error: Error?, contextInfo: UnsafeRawPointer) {

    if let error = error {
        print("Error Saving ARKit Scene \(error)")
    } else {
        print("ARKit Scene Successfully Saved")
    }
}

黑白实时渲染： 使用diviaki在此处给出的出色答案，我还可以使用以下方法获得整个摄像机源，以黑白渲染：

1号 注册ARSessionDelegate像这样：

 augmentedRealitySession.delegate = self

2号 然后在下面的委托回调中添加以下内容：

 //-----------------------
 //MARK: ARSessionDelegate
 //-----------------------

 extension ViewController: ARSessionDelegate{

 func session(_ session: ARSession, didUpdate frame: ARFrame) {

        /*
        Full Credit To /sf/ask/3214382181/
        */

        //1. Convert The Current Frame To Black & White
        guard let currentBackgroundFrameImage = augmentedRealityView.session.currentFrame?.capturedImage,
              let pixelBufferAddressOfPlane = CVPixelBufferGetBaseAddressOfPlane(currentBackgroundFrameImage, 1) else { return }

        let x: size_t = CVPixelBufferGetWidthOfPlane(currentBackgroundFrameImage, 1)
        let y: size_t = CVPixelBufferGetHeightOfPlane(currentBackgroundFrameImage, 1)
        memset(pixelBufferAddressOfPlane, 128, Int(x * y) * 2)

      }

 }

可以成功渲染黑白摄像机的供稿：

黑白SCNScene的过滤元素：

就像@Confused正确地说的那样，如果您决定希望cameraFeed彩色，但您的内容为AR Experience黑白，则可以将过滤器直接应用到SCNNode使用它的filters属性，这很简单：

一组核心图像滤镜，将应用于节点的渲染内容。

举例来说，假设我们SCNNodes使用a 动态创建了3 ，Sphere Geometry我们可以CoreImageFilter像这样直接将a 应用于这些对象：

/// Creates 3 Objects And Adds Them To The Scene (Rendering Them In GrayScale)
func createObjects(){

    //1. Create An Array Of UIColors To Set As The Geometry Colours
    let colours = [UIColor.red, UIColor.green, UIColor.yellow]

    //2. Create An Array Of The X Positions Of The Nodes
    let xPositions: [CGFloat] = [-0.3, 0, 0.3]

    //3. Create The Nodes & Add Them To The Scene
    for i in 0 ..< 3{

        let sphereNode = SCNNode()
        let sphereGeometry = SCNSphere(radius: 0.1)
        sphereGeometry.firstMaterial?.diffuse.contents = colours[i]
        sphereNode.geometry = sphereGeometry
        sphereNode.position = SCNVector3( xPositions[i], 0, -1.5)
        augmentedRealityView.scene.rootNode.addChildNode(sphereNode)

        //a. Create A Black & White Filter
        guard let blackAndWhiteFilter = CIFilter(name: "CIColorControls", withInputParameters: [kCIInputSaturationKey:0.0]) else { return }
        blackAndWhiteFilter.name = "bw"
        sphereNode.filters = [blackAndWhiteFilter]
        sphereNode.setValue(CIFilter(), forKeyPath: "bw")
    }

}

这将产生如下结果：

有关这些过滤器的完整列表，请参考以下内容：CoreImage过滤器参考

示例项目：这是一个完整的示例项目，您可以自己下载和浏览。

希望能帮助到你...