录制音频并将数据传递到iOS 8/9上的UIWebView(JavascriptCore)

Question

我们有一个应用程序,主要是一个基于javascript的Web应用程序的UIWebView.我们遇到的要求是能够向用户播放音频然后记录用户,回放该记录以进行确认,然后将音频发送到服务器.这适用于Chrome,Android和其他平台,因为该功能内置于浏览器中.不需要本机代码.

遗憾的是,iOS(iOS 8/9)Web视图缺乏录制音频的能力.

我们尝试的第一个解决方法是使用AudioQueue录制音频并将数据(LinearPCM 16bit)传递给JS AudioNode,以便Web应用程序可以像处理其他平台一样处理iOS音频.这就达到了我们可以将音频传递给JS的程度,但应用程序最终会因内存访问错误而崩溃,或者javascript方面无法跟上发送的数据.

下一个想法是将录音保存到文件并将部分音频数据发送到JS以获得视觉反馈,这是在录制期间显示的基本音频可视化器.

线性PCM签名为16bit时,音频记录并播放到WAVE文件.JS可视化工具是我们陷入困境的地方.期待线性PCM无符号8位,所以我添加了一个可能错误的转换步骤.我尝试了几种不同的方式,主要是在网上找到的,但是在我们进入转换步骤之前,没有找到一种可以让我认为其他错误或缺失的方法.

由于我不知道问题究竟是什么或在哪里,我将转储下面的代码用于音频录制和播放类.欢迎任何建议解决或绕过某种方式解决此问题.

我的一个想法是使用不同的格式标记以不同的格式(CAF)进行记录.查看生成的值,非带符号的16位整数甚至接近最大值.我很少看到+/- 1000以上的任何东西.是因为AudioStreamPacketDescription中的kLinearPCMFormatFlagIsPacked标志？删除该标志会因为格式无效而导致音频文件无法创建.也许切换到CAF会起作用,但我们需要在将音频发送回服务器之前转换为WAVE.

或者我从签名的16位到无符号8位的转换是错误的？我也试过了比特换档和铸造.唯一的区别是,通过此转换,所有音频值都被压缩到125到130之间.位移和转换将其更改为0-5和250-255.这并没有真正解决JS方面的任何问题.

下一步是,不是将数据传递给JS,而是通过FFT函数运行它,并生成JS直接用于音频可视化器的值.在去那个方向之前,我宁愿弄清楚我是否做了一些明显错误的事情.

AQRecorder.h - 编辑:将更新的音频格式更新为LinearPCM 32bit Float.

#ifndef AQRecorder_h  
#define AQRecorder_h  
#import <AudioToolbox/AudioToolbox.h>  
#define NUM_BUFFERS 3  
#define AUDIO_DATA_TYPE_FORMAT float  
#define JS_AUDIO_DATA_SIZE 32  
@interface AQRecorder : NSObject {  
    AudioStreamBasicDescription  mDataFormat;  
    AudioQueueRef                mQueue;  
    AudioQueueBufferRef          mBuffers[ NUM_BUFFERS ];  
    AudioFileID                  mAudioFile;  
    UInt32                       bufferByteSize;  
    SInt64                       mCurrentPacket;  
    bool                         mIsRunning;  
}  
- (void)setupAudioFormat;  
- (void)startRecording;  
- (void)stopRecording;  
- (void)processSamplesForJS:(UInt32)audioDataBytesCapacity audioData:(void *)audioData;  
- (Boolean)isRunning;  
@end  
#endif 

AQRecorder.m - 编辑:将更新的音频格式更新为LinearPCM 32bit Float.在processSamplesForJS中添加了FFT步骤,而不是直接发送音频数据.

#import <AVFoundation/AVFoundation.h>  
#import "AQRecorder.h"  
#import "JSMonitor.h"  
@implementation AQRecorder  
void AudioQueueCallback(void * inUserData,   
                        AudioQueueRef inAQ,  
                        AudioQueueBufferRef inBuffer,  
                        const AudioTimeStamp * inStartTime,  
                        UInt32 inNumberPacketDescriptions,  
                        const AudioStreamPacketDescription* inPacketDescs)  
{  

    AQRecorder *aqr = (__bridge AQRecorder *)inUserData;  
    if ( [aqr isRunning] )  
    {  
        if ( inNumberPacketDescriptions > 0 )  
        {  
            AudioFileWritePackets(aqr->mAudioFile, FALSE, inBuffer->mAudioDataByteSize, inPacketDescs, aqr->mCurrentPacket, &inNumberPacketDescriptions, inBuffer->mAudioData);  
            aqr->mCurrentPacket += inNumberPacketDescriptions;  
            [aqr processSamplesForJS:inBuffer->mAudioDataBytesCapacity audioData:inBuffer->mAudioData];  
        }  

        AudioQueueEnqueueBuffer(inAQ, inBuffer, 0, NULL);  
    }  
}  
- (void)debugDataFormat  
{  
    NSLog(@"format=%i, sampleRate=%f, channels=%i, flags=%i, BPC=%i, BPF=%i", mDataFormat.mFormatID, mDataFormat.mSampleRate, (unsigned int)mDataFormat.mChannelsPerFrame, mDataFormat.mFormatFlags, mDataFormat.mBitsPerChannel, mDataFormat.mBytesPerFrame);  
}  
- (void)setupAudioFormat  
{  
    memset(&mDataFormat, 0, sizeof(mDataFormat));  

    mDataFormat.mSampleRate = 44100.;  
    mDataFormat.mChannelsPerFrame = 1;  
    mDataFormat.mFormatID = kAudioFormatLinearPCM;  
    mDataFormat.mFormatFlags = kLinearPCMFormatFlagIsFloat | kLinearPCMFormatFlagIsPacked;  

    int sampleSize = sizeof(AUDIO_DATA_TYPE_FORMAT);  
    mDataFormat.mBitsPerChannel = 32;
    mDataFormat.mBytesPerPacket = mDataFormat.mBytesPerFrame = (mDataFormat.mBitsPerChannel / 8) * mDataFormat.mChannelsPerFrame;
    mDataFormat.mFramesPerPacket = 1;
    mDataFormat.mReserved = 0;  

    [self debugDataFormat];  
}  
- (void)startRecording/  
{  
    [self setupAudioFormat];  

    mCurrentPacket = 0;  

    NSString *recordFile = [NSTemporaryDirectory() stringByAppendingPathComponent: @"AudioFile.wav"];  
    CFURLRef url = CFURLCreateWithString(kCFAllocatorDefault, (CFStringRef)recordFile, NULL);;  
    OSStatus *stat =  
    AudioFileCreateWithURL(url, kAudioFileWAVEType, &mDataFormat, kAudioFileFlags_EraseFile, &mAudioFile);  
    NSError *error = [NSError errorWithDomain:NSOSStatusErrorDomain code:stat userInfo:nil];  
    NSLog(@"AudioFileCreateWithURL OSStatus :: %@", error);  
    CFRelease(url);  

    bufferByteSize = 896 * mDataFormat.mBytesPerFrame;  
    AudioQueueNewInput(&mDataFormat, AudioQueueCallback, (__bridge void *)(self), NULL, NULL, 0, &mQueue);  
    for ( int i = 0; i < NUM_BUFFERS; i++ )  
    {  
        AudioQueueAllocateBuffer(mQueue, bufferByteSize, &mBuffers[i]);  
        AudioQueueEnqueueBuffer(mQueue, mBuffers[i], 0, NULL);  
    }  
    mIsRunning = true;  
    AudioQueueStart(mQueue, NULL);  
}  
- (void)stopRecording  
{  
     mIsRunning = false;  
    AudioQueueStop(mQueue, false);  
    AudioQueueDispose(mQueue, false);  
    AudioFileClose(mAudioFile);  
}  
- (void)processSamplesForJS:(UInt32)audioDataBytesCapacity audioData:(void *)audioData  
{  
    int sampleCount = audioDataBytesCapacity / sizeof(AUDIO_DATA_TYPE_FORMAT);  
    AUDIO_DATA_TYPE_FORMAT *samples = (AUDIO_DATA_TYPE_FORMAT*)audioData;  

    NSMutableArray *audioDataBuffer = [[NSMutableArray alloc] initWithCapacity:JS_AUDIO_DATA_SIZE];

    // FFT stuff taken mostly from Apples aurioTouch example
    const Float32 kAdjust0DB = 1.5849e-13;

    int bufferFrames = sampleCount;
    int bufferlog2 = round(log2(bufferFrames));
    float fftNormFactor = (1.0/(2*bufferFrames));
    FFTSetup fftSetup = vDSP_create_fftsetup(bufferlog2, kFFTRadix2);

    Float32 *outReal = (Float32*) malloc((bufferFrames / 2)*sizeof(Float32));
    Float32 *outImaginary = (Float32*) malloc((bufferFrames / 2)*sizeof(Float32));
    COMPLEX_SPLIT mDspSplitComplex = { .realp = outReal, .imagp = outImaginary };

    Float32 *outFFTData = (Float32*) malloc((bufferFrames / 2)*sizeof(Float32));

    //Generate a split complex vector from the real data
    vDSP_ctoz((COMPLEX *)samples, 2, &mDspSplitComplex, 1, bufferFrames / 2);

    //Take the fft and scale appropriately
    vDSP_fft_zrip(fftSetup, &mDspSplitComplex, 1, bufferlog2, kFFTDirection_Forward);
    vDSP_vsmul(mDspSplitComplex.realp, 1, &fftNormFactor, mDspSplitComplex.realp, 1, bufferFrames / 2);
    vDSP_vsmul(mDspSplitComplex.imagp, 1, &fftNormFactor, mDspSplitComplex.imagp, 1, bufferFrames / 2);

    //Zero out the nyquist value
    mDspSplitComplex.imagp[0] = 0.0;

    //Convert the fft data to dB
    vDSP_zvmags(&mDspSplitComplex, 1, outFFTData, 1, bufferFrames / 2);

    //In order to avoid taking log10 of zero, an adjusting factor is added in to make the minimum value equal -128dB
    vDSP_vsadd(outFFTData, 1, &kAdjust0DB, outFFTData, 1, bufferFrames / 2);
    Float32 one = 1;
    vDSP_vdbcon(outFFTData, 1, &one, outFFTData, 1, bufferFrames / 2, 0);

    // Average out FFT dB values
    int grpSize = (bufferFrames / 2) / 32;
    int c = 1;
    Float32 avg = 0;
    int d = 1;
    for ( int i = 1; i < bufferFrames / 2; i++ )
    {
        if ( outFFTData[ i ] != outFFTData[ i ] || outFFTData[ i ] == INFINITY )
        { // NAN / INFINITE check
            c++;
        }
        else
        {
            avg += outFFTData[ i ];
            d++;
            //NSLog(@"db = %f, avg = %f", outFFTData[ i ], avg);

            if ( ++c >= grpSize )
            {
                uint8_t u = (uint8_t)((avg / d) + 128); //dB values seem to range from -128 to 0.
                NSLog(@"%i = %i (%f)", i, u, avg);
                [audioDataBuffer addObject:[NSNumber numberWithUnsignedInt:u]];
                avg = 0;
                c = 0;
                d = 1;
            }
        }
    } 

    [[JSMonitor shared] passAudioDataToJavascriptBridge:audioDataBuffer];  
}  
- (Boolean)isRunning  
{  
    return mIsRunning;  
}  
@end 

音频播放和录音控制器类Audio.h

#ifndef Audio_h  
#define Audio_h  
#import <AVFoundation/AVFoundation.h>  
#import "AQRecorder.h"  
@interface Audio : NSObject <AVAudioPlayerDelegate> {  
    AQRecorder* recorder;  
    AVAudioPlayer* player;  
    bool mIsSetup;  
    bool mIsRecording;  
    bool mIsPlaying;  
}  
- (void)setupAudio;  
- (void)startRecording;  
- (void)stopRecording;  
- (void)startPlaying;  
- (void)stopPlaying;  
- (Boolean)isRecording;  
- (Boolean)isPlaying;  
- (NSString *) getAudioDataBase64String;  
@end  
#endif 

Audio.m

#import "Audio.h"  
#import <AudioToolbox/AudioToolbox.h>  
#import "JSMonitor.h"  
@implementation Audio  
- (void)setupAudio  
{  
    NSLog(@"Audio->setupAudio");  
    AVAudioSession *session = [AVAudioSession sharedInstance];  
    NSError * error;  
    [session setCategory:AVAudioSessionCategoryPlayAndRecord error:&error];  
    [session setActive:YES error:nil];  

    recorder = [[AQRecorder alloc] init];  

    mIsSetup = YES;  
}  
- (void)startRecording  
{  
    NSLog(@"Audio->startRecording");  
    if ( !mIsSetup )  
    {  
        [self setupAudio];  
    }  

    if ( mIsRecording ) {  
        return;  
    }  

    if ( [recorder isRunning] == NO )  
    {  
        [recorder startRecording];  
    }  

    mIsRecording = [recorder isRunning];  
}  
- (void)stopRecording  
{  
    NSLog(@"Audio->stopRecording");  
    [recorder stopRecording];  
    mIsRecording = [recorder isRunning];  

    [[JSMonitor shared] sendAudioInputStoppedEvent];  
}  
- (void)startPlaying  
{  
    if ( mIsPlaying )  
    {  
        return;  
    }  

    mIsPlaying = YES;  
    NSLog(@"Audio->startPlaying");  
    NSError* error = nil;  
    NSString *recordFile = [NSTemporaryDirectory() stringByAppendingPathComponent: @"AudioFile.wav"];  
    player = [[AVAudioPlayer alloc] initWithContentsOfURL:[NSURL fileURLWithPath:recordFile] error:&error];  

    if ( error )  
    {  
        NSLog(@"AVAudioPlayer failed :: %@", error);  
    }  

    player.delegate = self;  
    [player play];  
}  
- (void)stopPlaying  
{  
    NSLog(@"Audio->stopPlaying");  
    [player stop];  
    mIsPlaying = NO;  
    [[JSMonitor shared] sendAudioPlaybackCompleteEvent];  
}  
- (NSString *) getAudioDataBase64String  
{  
    NSString *recordFile = [NSTemporaryDirectory() stringByAppendingPathComponent: @"AudioFile.wav"];  

    NSError* error = nil;  
    NSData *fileData = [NSData dataWithContentsOfFile:recordFile options: 0 error: &error];  
    if ( fileData == nil )  
    {  
        NSLog(@"Failed to read file, error %@", error);  
        return @"DATAENCODINGFAILED";  
    }  
    else  
    {  
        return [fileData base64EncodedStringWithOptions:0];  
    }  
}  
- (Boolean)isRecording { return mIsRecording; }  
- (Boolean)isPlaying { return mIsPlaying; }  

- (void)audioPlayerDidFinishPlaying:(AVAudioPlayer *)player successfully:(BOOL)flag  
{  
    NSLog(@"Audio->audioPlayerDidFinishPlaying: %i", flag);  
    mIsPlaying = NO;  
    [[JSMonitor shared] sendAudioPlaybackCompleteEvent];  
}  
- (void)audioPlayerDecodeErrorDidOccur:(AVAudioPlayer *)player error:(NSError *)error  
{  
    NSLog(@"Audio->audioPlayerDecodeErrorDidOccur: %@", error.localizedFailureReason);  
    mIsPlaying = NO;  
    [[JSMonitor shared] sendAudioPlaybackCompleteEvent];  
}  
@end 

JSMonitor类是UIWebView javascriptcore和本机代码之间的桥梁.我不包括它,因为除了在这些类和JSCore之间传递数据/调用之外,它对音频没有任何作用.

编辑

音频格式已更改为LinearPCM Float 32bit.它不是发送音频数据,而是通过FFT函数发送,而dB值被平均并发送.

Answer 1

Core Audio 使用起来很痛苦。幸运的是，AVFoundation 提供了 AVAudioRecorder录制视频的功能，并且还允许您访问平均和峰值音频功率，您可以将其发送回 JavaScript 以更新 UI 可视化工具。来自文档：

AVAudioRecorder 类的实例（称为录音机）在您的应用程序中提供音频录制功能。使用录音机您可以：

• 录音直到用户停止录音
• 记录指定持续时间
• 暂停和恢复录音
• 获取可用于提供电平计量的输入音频电平数据

这个堆栈溢出问题有一个如何使用的示例AVAudioRecorder。