如何使用Java Sound中的音频样本数据？

Question

这个问题通常被问到是另一个问题的一部分,但事实证明答案很长.我决定在这里回答它,所以我可以在其他地方链接到它.

虽然我不知道Java此时可以为我们生成音频样本的方式,但如果将来发生变化,这可能是一个适合它的地方.我知道JavaFX有一些这样的东西,例如AudioSpectrumListener,但仍然不能直接访问样本.

---

我正在javax.sound.sampled用于播放和/或录制,但我想对音频做些什么.

也许我想直观地展示它或以某种方式处理它.

如何使用Java Sound访问音频样本数据？

也可以看看:

• Java声音教程(官方)
• Java声音资源(非官方)

Answer 1

嗯,最简单的答案是,目前Java无法为程序员生成样本数据.

这句话来自官方教程:

有两种方法可以应用信号处理:

• 您可以使用混音器或其组件线支持的任何处理,查询Control对象,然后根据用户需要设置控件.混音器和线路支持的典型控制包括增益,声像和混响控制.

• 如果混音器或其线路不提供您需要的处理类型,您的程序可以直接在音频字节上操作,根据需要进行操作.

本页更详细地讨论了第一种技术,因为第二种技术没有特殊的API.

Playback with javax.sound.sampled largely acts as a bridge between the file and the audio device. The bytes are read in from the file and sent off.

Don't assume the bytes are meaningful audio samples! Unless you happen to have an 8-bit AIFF file, they aren't. (On the other hand, if the samples are definitely 8-bit signed, you can do arithmetic with them. Using 8-bit is one way to avoid the complexity described here, if you're just playing around.)

So instead, I'll enumerate the types of AudioFormat.Encoding and describe how to decode them yourself. This answer will not cover how to encode them, but it's included in the complete code example at the bottom. Encoding is mostly just the decoding process in reverse.

This is a long answer but I wanted to give a thorough overview.

A Little About Digital Audio

Generally when digital audio is explained, we're referring to Linear Pulse-Code Modulation (LPCM).

A continuous sound wave is sampled at regular intervals and the amplitudes are quantized to integers of some scale.

Shown here is a sine wave sampled and quantized to 4-bit:

(Notice that the most positive value in two's complement representation is 1 less than the most negative value. This is a minor detail to be aware of. For example if you're clipping audio and forget this, the positive clips will overflow.)

When we have audio on the computer, we have an array of these samples. A sample array is what we want to turn the byte array in to.

要解码PCM样本,我们并不关心采样率或通道数,所以我不会在这里说太多.通道通常是交错的,因此如果我们有一个数组,它们将被存储如下:

Index 0: Sample 0 (Left Channel)
Index 1: Sample 0 (Right Channel)
Index 2: Sample 1 (Left Channel)
Index 3: Sample 1 (Right Channel)
Index 4: Sample 2 (Left Channel)
Index 5: Sample 2 (Right Channel)
...

换句话说,对于立体声,阵列中的样本只是在左右之间交替.

一些假设

所有代码示例都将采用以下声明:

• byte[] bytes;的byte阵列,从所述读出AudioInputStream.
• float[] samples; 我们要填充的输出样本数组.
• float sample; 我们目前正在研究的样本.
• long temp; 用于一般操作的临时值.
• int i;中相应的位置byte,其中当前样本的数据开始阵列.

我们将float[]数组中的所有样本标准化为范围-1f <= sample <= 1f.我见过的所有浮点音频都是这样的,非常方便.

如果我们的源音频不是那样(例如整数样本),我们可以使用以下方法自己标准化它们:

sample = sample / fullScale(bitsPerSample);

哪里fullScale是2个^{bitsPerSample - 1},即Math.pow(2, bitsPerSample-1).

如何将byte数组强制转换为有意义的数据？

该byte数组包含分割的样本帧,并且全部在一行中.这实际上是非常简单的,除了名为endianness的东西,它是byte每个样本包中s 的排序.

Here's a diagram. This sample (packed in to a byte array) holds the decimal value 9999:

  24-bit sample as big-endian:

 bytes[i]   bytes[i + 1] bytes[i + 2]
 ????????     ????????     ????????
 00000000     00100111     00001111

 24-bit sample as little-endian:

 bytes[i]   bytes[i + 1] bytes[i + 2]
 ????????     ????????     ????????
 00001111     00100111     00000000

They hold the same binary values; however, the byte orders are reversed.

• In big-endian, the more significant bytes come before the less significant bytes.
• In little-endian, the less significant bytes come before the more significant bytes.

WAV files are stored in little-endian order and AIFF files are stored in big-endian order. Endianness can be obtained from AudioFormat.isBigEndian.

To concatenate the bytes and put them in to our long temp variable, we:

1. Bitwise AND each byte with the mask 0xFF (which is 0b1111_1111) to avoid sign-extension when the byte is automatically promoted. (char, byte and short are promoted to int when arithmetic is performed on them.) See also What does value & 0xff do in Java?
2. Bit shift each byte in to position.
3. Bitwise OR the bytes together.

Here's a 24-bit example:

long temp;
if (isBigEndian) {
    temp = (
          ((bytes[i    ] & 0xffL) << 16)
        | ((bytes[i + 1] & 0xffL) <<  8)
        |  (bytes[i + 2] & 0xffL)
    );
} else {
    temp = (
           (bytes[i    ] & 0xffL)
        | ((bytes[i + 1] & 0xffL) <<  8)
        | ((bytes[i + 2] & 0xffL) << 16)
    );
}

Notice that the shift order is reversed based on endianness.

This can also be generalized to a loop, which can be seen in the full code at the bottom of this answer. (See the unpackAnyBit and packAnyBit methods.)

Now that we have the bytes concatenated together, we can take a few more steps to turn them in to a sample. The next steps depend on the actual encoding.

How do I decode Encoding.PCM_SIGNED?

The two's complement sign must be extended. This means that if the most significant bit (MSB) is set to 1, we fill all the bits above it with 1s. The arithmetic right-shift (>>) will do the filling for us automatically if the sign bit is set, so I usually do it this way:

int bitsToExtend = Long.SIZE - bitsPerSample;
float sample = (temp << bitsToExtend) >> bitsToExtend.

(Where Long.SIZE is 64. If our temp variable wasn't a long, we'd use something else. If we used e.g. int temp instead, we'd use 32.)

To understand how this works, here's a diagram of sign-extending 8-bit to 16-bit:

 11111111 is the byte value -1, but the upper bits of the short are 0.
 Shift the byte's MSB in to the MSB position of the short.

 0000 0000 1111 1111
 <<                8
 ???????????????????
 1111 1111 0000 0000

 Shift it back and the right-shift fills all the upper bits with 1s.
 We now have the short value of -1.

 1111 1111 0000 0000
 >>                8
 ???????????????????
 1111 1111 1111 1111

Positive values (that had a 0 in the MSB) are left unchanged. This is a nice property of the arithmetic right-shift.

Then normalize the sample, as described in Some Assumptions.

You might not need to write explicit sign-extension if your code is simple

Java does sign-extension automatically when converting from one integral type to a larger type, for example byte to int. If you know that your input and output format are always signed, you can use the automatic sign-extension while concatenating bytes in the earlier step.

Recall from the section above (How do I coerce the byte array in to meaningful data?) that we used b & 0xFF to prevent sign-extension from occurring. If you just remove the & 0xFF from the highest byte, sign-extension will happen automatically.

For example, the following decodes signed, big-endian, 16-bit samples:

for (int i = 0; i < bytes.length; i++) {
    int sample = (bytes[i] << 8) // high byte is sign-extended
               | (bytes[i + 1] & 0xFF); // low byte is not
    // ...
}

How do I decode Encoding.PCM_UNSIGNED?

We turn it in to a signed number. Unsigned samples are simply offset so that, for example:

• An unsigned value of 0 corresponds to the most negative signed value.
• An unsigned value of 2^{bitsPerSample - 1} corresponds to the signed value of 0.
• An unsigned value of 2^{bitsPerSample} corresponds to the most positive signed value.

So this turns out to be pretty simple. Just subtract the offset:

float sample = temp - fullScale(bitsPerSample);

Then normalize the sample, as described in Some Assumptions.

How do I decode Encoding.PCM_FLOAT?

This is new since Java 7.

In practice, floating-point PCM is typically either IEEE 32-bit or IEEE 64-bit and already normalized to the range of ±1.0. The samples can be obtained with the utility methods Float#intBitsToFloat and Double#longBitsToDouble.

// IEEE 32-bit
float sample = Float.intBitsToFloat((int) temp);

// IEEE 64-bit
double sampleAsDouble = Double.longBitsToDouble(temp);
float sample = (float) sampleAsDouble; // or just use double for arithmetic

How do I decode Encoding.ULAW and Encoding.ALAW?

These are companding compression codecs that are more common in telephones and such. They're supported by javax.sound.sampled I assume because they're used by Sun's Au format. (However, it's not limited to just this type of container. For example, WAV can contain these encodings.)

You can conceptualize A-law and μ-law like they're a floating-point format. These are PCM formats but the range of values is non-linear.

There are two ways to decode them. I'll show the way which uses the mathematical formula. You can also decode them by manipulating the binary directly which is described in this blog post but it's more esoteric-looking.

For both, the compressed data is 8-bit. Standardly A-law is 13-bit when decoded and μ-law is 14-bit when decoded; however, applying the formula yields a range of ±1.0.

Before you can apply the formula, there are three things to do:

1. Some of the bits are standardly inverted for storage due to reasons involving data integrity.
2. They're stored as sign and magnitude (rather than two's complement).
3. The formula also expects a range of ±1.0, so the 8-bit value has to be scaled.

For μ-law all the bits are inverted, so:

temp ^= 0xffL; // 0xff == 0b1111_1111

(Note that we can't use ~, because we don't want to invert the high bits of the long.)

For A-law, every other bit is inverted, so:

temp ^= 0x55L; // 0x55 == 0b0101_0101

(XOR can be used to do inversion. See How do you set, clear and toggle a bit?)

To convert from sign and magnitude to two's complement, we:

1. Check to see if the sign bit was set.
2. If so, clear the sign bit and negate the number.

// 0x80 == 0b1000_0000
if ((temp & 0x80L) != 0) {
    temp ^= 0x80L;
    temp = -temp;
}

Then scale the encoded numbers, the same way as described in Some Assumptions:

sample = temp / fullScale(8);

Now we can apply the expansion.

然后,转换为Java的μ律公式为:

sample = (float) (
    signum(sample)
        *
    (1.0 / 255.0)
        *
    (pow(256.0, abs(sample)) - 1.0)
);

那么翻译成Java的A律公式就是:

float signum = signum(sample);
sample = abs(sample);

if (sample < (1.0 / (1.0 + log(87.7)))) {
    sample = (float) (
        sample * ((1.0 + log(87.7)) / 87.7)
    );
} else {
    sample = (float) (
        exp((sample * (1.0 + log(87.7))) - 1.0) / 87.7
    );
}

sample = signum * sample;

这是SimpleAudioConversion该类的完整示例代码.

package mcve.audio;

import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioFormat.Encoding;

import static java.lang.Math.*;

/**
 * <p>Performs simple audio format conversion.</p>
 *
 * <p>Example usage:</p>
 *
 * <pre>{@code  AudioInputStream ais = ... ;
 * SourceDataLine  line = ... ;
 * AudioFormat      fmt = ... ;
 *
 * // do setup
 *
 * for (int blen = 0; (blen = ais.read(bytes)) > -1;) {
 *     int slen;
 *     slen = SimpleAudioConversion.decode(bytes, samples, blen, fmt);
 *
 *     // do something with samples
 *
 *     blen = SimpleAudioConversion.encode(samples, bytes, slen, fmt);
 *     line.write(bytes, 0, blen);
 * }}</pre>
 *
 * @author Radiodef
 * @see <a href="http://stackoverflow.com/a/26824664/2891664">Overview on Stack Overflow</a>
 */
public final class SimpleAudioConversion {
    private SimpleAudioConversion() {}

    /**
     * Converts from a byte array to an audio sample float array.
     *
     * @param bytes   the byte array, filled by the AudioInputStream
     * @param samples an array to fill up with audio samples
     * @param blen    the return value of AudioInputStream.read
     * @param fmt     the source AudioFormat
     *
     * @return the number of valid audio samples converted
     *
     * @throws NullPointerException if bytes, samples or fmt is null
     * @throws ArrayIndexOutOfBoundsException
     *         if bytes.length is less than blen or
     *         if samples.length is less than blen / bytesPerSample(fmt.getSampleSizeInBits())
     */
    public static int decode(byte[]      bytes,
                             float[]     samples,
                             int         blen,
                             AudioFormat fmt) {
        int   bitsPerSample = fmt.getSampleSizeInBits();
        int  bytesPerSample = bytesPerSample(bitsPerSample);
        boolean isBigEndian = fmt.isBigEndian();
        Encoding   encoding = fmt.getEncoding();
        double    fullScale = fullScale(bitsPerSample);

        int i = 0;
        int s = 0;
        while (i < blen) {
            long temp = unpackBits(bytes, i, isBigEndian, bytesPerSample);
            float sample = 0f;

            if (encoding == Encoding.PCM_SIGNED) {
                temp = extendSign(temp, bitsPerSample);
                sample = (float) (temp / fullScale);

            } else if (encoding == Encoding.PCM_UNSIGNED) {
                temp = unsignedToSigned(temp, bitsPerSample);
                sample = (float) (temp / fullScale);

            } else if (encoding == Encoding.PCM_FLOAT) {
                if (bitsPerSample == 32) {
                    sample = Float.intBitsToFloat((int) temp);
                } else if (bitsPerSample == 64) {
                    sample = (float) Double.longBitsToDouble(temp);
                }
            } else if (encoding == Encoding.ULAW) {
                sample = bitsToMuLaw(temp);

            } else if (encoding == Encoding.ALAW) {
                sample = bitsToALaw(temp);
            }

            samples[s] = sample;

            i += bytesPerSample;
            s++;
        }

        return s;
    }

    /**
     * Converts from an audio sample float array to a byte array.
     *
     * @param samples an array of audio samples to encode
     * @param bytes   an array to fill up with bytes
     * @param slen    the return value of the decode method
     * @param fmt     the destination AudioFormat
     *
     * @return the number of valid bytes converted
     *
     * @throws NullPointerException if samples, bytes or fmt is null
     * @throws ArrayIndexOutOfBoundsException
     *         if samples.length is less than slen or
     *         if bytes.length is less than slen * bytesPerSample(fmt.getSampleSizeInBits())
     */
    public static int encode(float[]     samples,
                             byte[]      bytes,
                             int         slen,
                             AudioFormat fmt) {
        int   bitsPerSample = fmt.getSampleSizeInBits();
        int  bytesPerSample = bytesPerSample(bitsPerSample);
        boolean isBigEndian = fmt.isBigEndian();
        Encoding   encoding = fmt.getEncoding();
        double    fullScale = fullScale(bitsPerSample);

        int i = 0;
        int s = 0;
        while (s < slen) {
            float sample = samples[s];
            long temp = 0L;

            if (encoding == Encoding.PCM_SIGNED) {
                temp = (long) (sample * fullScale);

            } else if (encoding == Encoding.PCM_UNSIGNED) {
                temp = (long) (sample * fullScale);
                temp = signedToUnsigned(temp, bitsPerSample);

            } else if (encoding == Encoding.PCM_FLOAT) {
                if (bitsPerSample == 32) {
                    temp = Float.floatToRawIntBits(sample);
                } else if (bitsPerSample == 64) {
                    temp = Double.doubleToRawLongBits(sample);
                }
            } else if (encoding == Encoding.ULAW) {
                temp = muLawToBits(sample);

            } else if (encoding == Encoding.ALAW) {
                temp = aLawToBits(sample);
            }

            packBits(bytes, i, temp, isBigEndian, bytesPerSample);

            i += bytesPerSample;
            s++;
        }

        return i;
    }

    /**
     * Computes the block-aligned bytes per sample of the audio format,
     * using Math.ceil(bitsPerSample / 8.0).
     * <p>
     * Round towards the ceiling because formats that allow bit depths
     * in non-integral multiples of 8 typically pad up to the nearest
     * integral multiple of 8. So for example, a 31-bit AIFF file will
     * actually store 32-bit blocks.
     *
     * @param  bitsPerSample the return value of AudioFormat.getSampleSizeInBits
     * @return The block-aligned bytes per sample of the audio format.
     */
    public static int bytesPerSample(int bitsPerSample) {
        return (int) ceil(bitsPerSample / 8.0); // optimization: ((bitsPerSample + 7) >>> 3)
    }

    /**
     * Computes the largest magnitude representable by the audio format,
     * using Math.pow(2.0, bitsPerSample - 1). Note that for two's complement
     * audio, the largest positive value is one less than the return value of
     * this method.
     * <p>
     * The result is returned as a double because in the case that
     * bitsPerSample is 64, a long would overflow.
     *
     * @param bitsPerSample the return value of AudioFormat.getBitsPerSample
     * @return the largest magnitude representable by the audio format
     */
    public static double fullScale(int bitsPerSample) {
        return pow(2.0, bitsPerSample - 1); // optimization: (1L << (bitsPerSample - 1))
    }

    private static long unpackBits(byte[]  bytes,
                                   int     i,
                                   boolean isBigEndian,
                                   int     bytesPerSample) {
        switch (bytesPerSample) {
            case  1: return unpack8Bit(bytes, i);
            case  2: return unpack16Bit(bytes, i, isBigEndian);
            case  3: return unpack24Bit(bytes, i, isBigEndian);
            default: return unpackAnyBit(bytes, i, isBigEndian, bytesPerSample);
        }
    }

    private static long unpack8Bit(byte[] bytes, int i) {
        return bytes[i] & 0xffL;
    }

    private static long unpack16Bit(byte[]  bytes,
                                    int     i,
                                    boolean isBigEndian) {
        if (isBigEndian) {
            return (
                  ((bytes[i    ] & 0xffL) << 8)
                |  (bytes[i + 1] & 0xffL)
            );
        } else {
            return (
                   (bytes[i    ] & 0xffL)
                | ((bytes[i + 1] & 0xffL) << 8)
            );
        }
    }

    private static long unpack24Bit(byte[]  bytes,
                                    int     i,
                                    boolean isBigEndian) {
        if (isBigEndian) {
            return (
                  ((bytes[i    ] & 0xffL) << 16)
                | ((bytes[i + 1] & 0xffL) <<  8)
                |  (bytes[i + 2] & 0xffL)
            );
        } else {
            return (
                   (bytes[i    ] & 0xffL)
                | ((bytes[i + 1] & 0xffL) <<  8)
                | ((bytes[i + 2] & 0xffL) << 16)
            );
        }
    }

    private static long unpackAnyBit(byte[]  bytes,
                                     int     i,
                                     boolean isBigEndian,
                                     int     bytesPerSample) {
        long temp = 0;

        if (isBigEndian) {
            for (int b = 0; b < bytesPerSample; b++) {
                temp |= (bytes[i + b] & 0xffL) << (
                    8 * (bytesPerSample - b - 1)
                );
            }
        } else {
            for (int b = 0; b < bytesPerSample; b++) {
                temp |= (bytes[i + b] & 0xffL) << (8 * b);
            }
        }

        return temp;
    }

    private static void packBits(byte[]  bytes,
                                 int     i,
                                 long    temp,
                                 boolean isBigEndian,
                                 int     bytesPerSample) {
        switch (bytesPerSample) {
            case  1: pack8Bit(bytes, i, temp);
                     break;
            case  2: pack16Bit(bytes, i, temp, isBigEndian);
                     break;
            case  3: pack24Bit(bytes, i, temp, isBigEndian);
                     break;
            default: packAnyBit(bytes, i, temp, isBigEndian, bytesPerSample);
                     break;
        }
    }

    private static void pack8Bit(byte[] bytes, int i, long temp) {
        bytes[i] = (byte) (temp & 0xffL);
    }

    private static void pack16Bit(byte[]  bytes,
                                  int     i,
                                  long    temp,
                                  boolean isBigEndian) {
        if (isBigEndian) {
            bytes[i    ] = (byte) ((temp >>> 8) & 0xffL);
            bytes[i + 1] = (byte) ( temp        & 0xffL);
        } else {
            bytes[i    ] = (byte) ( temp        & 0xffL);
            bytes[i + 1] = (byte) ((temp >>> 8) & 0xffL);
        }
    }

    private static void pack24Bit(byte[]  bytes,
                                  int     i,
                                  long    temp,
                                  boolean isBigEndian) {
        if (isBigEndian) {
            bytes[i    ] = (byte) ((temp >>> 16) & 0xffL);
            bytes[i + 1] = (byte) ((temp >>>  8) & 0xffL);
            bytes[i + 2] = (byte) ( temp         & 0xffL);
        } else {
            bytes[i    ] = (byte) ( temp         & 0xffL);
            bytes[i + 1] = (byte) ((temp >>>  8) & 0xffL);
            bytes[i + 2] = (byte) ((temp >>> 16) & 0xffL);
        }
    }

    private static void packAnyBit(byte[]  bytes,
                                   int     i,
                                   long    temp,
                                   boolean isBigEndian,
                                   int     bytesPerSample) {
        if (isBigEndian) {
            for (int b = 0; b < bytesPerSample; b++) {
                bytes[i + b] = (byte) (
                    (temp >>> (8 * (bytesPerSample - b - 1))) & 0xffL
                );
            }
        } else {
            for (int b = 0; b < bytesPerSample; b++) {
                bytes[i + b] = (byte) ((temp >>> (8 * b)) & 0xffL);
            }
        }
    }

    private static long extendSign(long temp, int bitsPerSample) {
        int bitsToExtend = Long.SIZE - bitsPerSample;
        return (temp << bitsToExtend) >> bitsToExtend;
    }

    private static long unsignedToSigned(long temp, int bitsPerSample) {
        return temp - (long) fullScale(bitsPerSample);
    }

    private static long signedToUnsigned(long temp, int bitsPerSample) {
        return temp + (long) fullScale(bitsPerSample);
    }

    // mu-law constant
    private static final double MU = 255.0;
    // A-law constant
    private static final double A = 87.7;
    // natural logarithm of A
    private static final double LN_A = log(A);

    private static float bitsToMuLaw(long temp) {
        temp ^= 0xffL;
        if ((temp & 0x80L) != 0) {
            temp = -(temp ^ 0x80L);
        }

        float sample = (float) (temp / fullScale(8));

        return (float) (
            signum(sample)
                *
            (1.0 / MU)
                *
            (pow(1.0 + MU, abs(sample)) - 1.0)
        );
    }

    private static long muLawToBits(float sample) {
        double sign = signum(sample);
        sample = abs(sample);

        sample = (float) (
            sign * (log(1.0 + (MU * sample)) / log(1.0 + MU))
        );

        long temp = (long) (sample * fullScale(8));

        if (temp < 0) {
            temp = -temp ^ 0x80L;
        }

        return temp ^ 0xffL;
    }

    private static float bitsToALaw(long temp) {
        temp ^= 0x55L;
        if ((temp & 0x80L) != 0) {
            temp = -(temp ^ 0x80L);
        }

        float sample = (float) (temp / fullScale(8));

        float sign = signum(sample);
        sample = abs(sample);

        if (sample < (1.0 / (1.0 + LN_A))) {
            sample = (float) (sample * ((1.0 + LN_A) / A));
        } else {
            sample = (float) (exp((sample * (1.0 + LN_A)) - 1.0) / A);
        }

        return sign * sample;
    }

    private static long aLawToBits(float sample) {
        double sign = signum(sample);
        sample = abs(sample);

        if (sample < (1.0 / A)) {
            sample = (float) ((A * sample) / (1.0 + LN_A));
        } else {
            sample = (float) ((1.0 + log(A * sample)) / (1.0 + LN_A));
        }

        sample *= sign;

        long temp = (long) (sample * fullScale(8));

        if (temp < 0) {
            temp = -temp ^ 0x80L;
        }

        return temp ^ 0x55L;
    }
}