一旦UTF-8编码,如何截断一个java字符串以适应给定的字节数？

Question

我如何截断一个java,String以便我知道一旦它是UTF-8编码它将适合给定数量的字节存储？

Answer 1

这是一个简单的循环,它计算UTF-8表示的大小,并在超出时截断:

public static String truncateWhenUTF8(String s, int maxBytes) {
    int b = 0;
    for (int i = 0; i < s.length(); i++) {
        char c = s.charAt(i);

        // ranges from http://en.wikipedia.org/wiki/UTF-8
        int skip = 0;
        int more;
        if (c <= 0x007f) {
            more = 1;
        }
        else if (c <= 0x07FF) {
            more = 2;
        } else if (c <= 0xd7ff) {
            more = 3;
        } else if (c <= 0xDFFF) {
            // surrogate area, consume next char as well
            more = 4;
            skip = 1;
        } else {
            more = 3;
        }

        if (b + more > maxBytes) {
            return s.substring(0, i);
        }
        b += more;
        i += skip;
    }
    return s;
}

这会处理输入字符串中出现的代理项对.Java的UTF-8编码器(正确)将代理对输出为单个4字节序列而不是两个3字节序列,因此truncateWhenUTF8()将返回最长的截断字符串.如果忽略实现中的代理对,则截断的字符串可能会短于它们所需的短路.

我没有对该代码进行过大量测试,但这里有一些初步测试:

private static void test(String s, int maxBytes, int expectedBytes) {
    String result = truncateWhenUTF8(s, maxBytes);
    byte[] utf8 = result.getBytes(Charset.forName("UTF-8"));
    if (utf8.length > maxBytes) {
        System.out.println("BAD: our truncation of " + s + " was too big");
    }
    if (utf8.length != expectedBytes) {
        System.out.println("BAD: expected " + expectedBytes + " got " + utf8.length);
    }
    System.out.println(s + " truncated to " + result);
}

public static void main(String[] args) {
    test("abcd", 0, 0);
    test("abcd", 1, 1);
    test("abcd", 2, 2);
    test("abcd", 3, 3);
    test("abcd", 4, 4);
    test("abcd", 5, 4);

    test("a\u0080b", 0, 0);
    test("a\u0080b", 1, 1);
    test("a\u0080b", 2, 1);
    test("a\u0080b", 3, 3);
    test("a\u0080b", 4, 4);
    test("a\u0080b", 5, 4);

    test("a\u0800b", 0, 0);
    test("a\u0800b", 1, 1);
    test("a\u0800b", 2, 1);
    test("a\u0800b", 3, 1);
    test("a\u0800b", 4, 4);
    test("a\u0800b", 5, 5);
    test("a\u0800b", 6, 5);

    // surrogate pairs
    test("\uD834\uDD1E", 0, 0);
    test("\uD834\uDD1E", 1, 0);
    test("\uD834\uDD1E", 2, 0);
    test("\uD834\uDD1E", 3, 0);
    test("\uD834\uDD1E", 4, 4);
    test("\uD834\uDD1E", 5, 4);

}

更新了修改后的代码示例,它现在处理代理对.

Answer 2

您应该使用CharsetEncoder,getBytes()尽可能多的简单+副本可以将UTF-8字符切成两半.

像这样的东西:

public static int truncateUtf8(String input, byte[] output) {

    ByteBuffer outBuf = ByteBuffer.wrap(output);
    CharBuffer inBuf = CharBuffer.wrap(input.toCharArray());

    Charset utf8 = Charset.forName("UTF-8");
    utf8.newEncoder().encode(inBuf, outBuf, true);
    System.out.println("encoded " + inBuf.position() + " chars of " + input.length() + ", result: " + outBuf.position() + " bytes");
    return outBuf.position();
}

Answer 3

这是我提出的,它使用标准的Java API,因此应该安全并兼容所有unicode古怪和代理对等.解决方案来自http://www.jroller.com/holy/entry/truncating_utf_string_to_the与检查添加为null并在字符串比maxBytes更少的字节时避免解码.

/**
 * Truncates a string to the number of characters that fit in X bytes avoiding multi byte characters being cut in
 * half at the cut off point. Also handles surrogate pairs where 2 characters in the string is actually one literal
 * character.
 *
 * Based on: http://www.jroller.com/holy/entry/truncating_utf_string_to_the
 */
public static String truncateToFitUtf8ByteLength(String s, int maxBytes) {
    if (s == null) {
        return null;
    }
    Charset charset = Charset.forName("UTF-8");
    CharsetDecoder decoder = charset.newDecoder();
    byte[] sba = s.getBytes(charset);
    if (sba.length <= maxBytes) {
        return s;
    }
    // Ensure truncation by having byte buffer = maxBytes
    ByteBuffer bb = ByteBuffer.wrap(sba, 0, maxBytes);
    CharBuffer cb = CharBuffer.allocate(maxBytes);
    // Ignore an incomplete character
    decoder.onMalformedInput(CodingErrorAction.IGNORE)
    decoder.decode(bb, cb, true);
    decoder.flush(cb);
    return new String(cb.array(), 0, cb.position());
}

Answer 4

UTF-8编码具有一个简洁的特性,允许您查看字节集中的位置.

检查您想要的字符限制流.

• 如果它的高位为0,则它​​是一个单字节的字符,只需用0替换它就可以了.
• 如果它的高位为1,那么下一位是,那么你就是一个多字节字符的开头,所以只需将该字节设置为0就可以了.
• 如果高位为1但下一位为0,那么你就在一个字符的中间,沿着缓冲区返回,直到你找到一个高位为2或更多1的字节,然后用0.

示例:如果您的流是:31 33 31 C1 A3 32 33 00,您可以将字符串设置为1,2,3,5,6或7个字节长,但不能设置为4,因为这会将C1置于C1之后,是多字节char的开头.

Answer 5

你可以使用 -new String( data.getBytes("UTF-8") , 0, maxLen, "UTF-8");