操作 AudioBuffer.mData 以显示音频可视化

Question

我正在尝试实时处理音频数据，以便可以根据麦克风输入的声音显示屏幕频谱分析仪/可视化。我正在使用 AVFoundationAVCaptureAudioDataOutputSampleBufferDelegate来捕获音频数据，这会触发 delgate 函数captureOutput。函数如下：

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {

    autoreleasepool {

        guard captureOutput != nil,
            sampleBuffer != nil,
            connection != nil,
            CMSampleBufferDataIsReady(sampleBuffer) else { return }

        //Check this is AUDIO (and not VIDEO) being received
        if (connection.audioChannels.count > 0)
        {
            //Determine number of frames in buffer
            var numFrames = CMSampleBufferGetNumSamples(sampleBuffer)

            //Get AudioBufferList
            var audioBufferList = AudioBufferList(mNumberBuffers: 1, mBuffers: AudioBuffer(mNumberChannels: 0, mDataByteSize: 0, mData: nil))
            var blockBuffer: CMBlockBuffer?
          CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer, nil, &audioBufferList, MemoryLayout<AudioBufferList>.size, nil, nil, UInt32(kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment), &blockBuffer)

            let audioBuffers = UnsafeBufferPointer<AudioBuffer>(start: &audioBufferList.mBuffers, count: Int(audioBufferList.mNumberBuffers))

            for audioBuffer in audioBuffers {
                let data = Data(bytes: audioBuffer.mData!, count: Int(audioBuffer.mDataByteSize))

                let i16array = data.withUnsafeBytes {
                    UnsafeBufferPointer<Int16>(start: $0, count: data.count/2).map(Int16.init(bigEndian:))
                }

                for dataItem in i16array
                {
                    print(dataItem)
                }

            }

        }
    }
}

上面的代码Int16按预期打印类型的正数和负数，但需要帮助将这些原始数字转换为有意义的数据，例如可视化工具的功率和分贝。

Answer 1

我走在正确的轨道上...感谢RobertHarvey对我的问题的评论 - 需要使用加速框架的 FFT 计算函数来实现频谱分析仪。但即使在我使用这些函数之前，您也需要将原始数据转换为 ofArray类型Float，因为许多函数需要Float数组。

首先，我们将原始数据加载到一个Data对象中：

//Read data from AudioBuffer into a variable
let data = Data(bytes: audioBuffer.mData!, count: Int(audioBuffer.mDataByteSize))

我喜欢将Data对象视为 1 字节大小的信息块（每个 8 位）的“列表”，但是如果我检查样本中的帧数以及对象的总大小（以字节Data为单位），它们不匹配：

//Get number of frames in sample and total size of Data
var numFrames = CMSampleBufferGetNumSamples(sampleBuffer) //= 1024 frames in my case
var dataSize = audioBuffer.mDataByteSize //= 2048 bytes in my case

我的数据的总大小（以字节为单位）是我的CMSampleBuffer. 这意味着每帧音频的长度为 2 个字节。为了有意义地读取数据，我需要将我的Data对象（1 字节块的“列表”）转换为 2 字节块的数组。Int16包含 16 位（或 2 个字节 - 正是我们所需要的），所以让我们创建一个Arrayof Int16：

//Convert to Int16 array
let samples = data.withUnsafeBytes {
    UnsafeBufferPointer<Int16>(start: $0, count: data.count / MemoryLayout<Int16>.size)
}

现在我们有了Arrayof Int16，我们可以将其转换为Arrayof Float：

//Convert to Float Array
let factor = Float(Int16.max)
var floats: [Float] = Array(repeating: 0.0, count: samples.count)
for i in 0..<samples.count {
    floats[i] = Float(samples[i]) / factor
}

现在我们有了Float数组，现在可以使用 Accelerate Framework 的复杂数学将原始Float值转换为有意义的值，例如幅度、分贝等。链接到文档：

Apple 的加速框架

快速傅立叶变换 (FFT)

我发现苹果的文档相当繁琐。幸运的是，我在网上找到了一个非常好的例子，我可以根据我的需要重新调整它的用途，称为TempiFFT。实施如下：

//Initiate FFT
let fft = TempiFFT(withSize: numFrames, sampleRate: 44100.0)
fft.windowType = TempiFFTWindowType.hanning

//Pass array of Floats
fft.fftForward(floats)

//I only want to display 20 bands on my analyzer
fft.calculateLinearBands(minFrequency: 0, maxFrequency: fft.nyquistFrequency, numberOfBands: 20)

//Then use a loop to iterate through the bands in your spectrum analyzer
var magnitudeArr = [Float](repeating: Float(0), count: 20)
var magnitudeDBArr = [Float](repeating: Float(0), count: 20)
for i in 0..<20
{
    var magnitudeArr[i] = fft.magnitudeAtBand(i)
    var magnitudeDB = TempiFFT.toDB(fft.magnitudeAtBand(i))
    //..I didn't, but you could perform drawing functions here...
}

其他有用的参考：

将数据转换为 Int16 数组

将 Int16 数组转换为 Float 数组