Java Unicode字符串长度

Question

我正在努力获取unicode字符串的数量,并尝试了各种选项.看起来像一个小问题,但在很大程度上受到了打击.

在这里,我试图获得字符串str1的长度.我得到它6.但实际上它是3.将光标移动到字符串"குமார்"也显示为3个字符.

基本上我想测量长度并打印每个角色.喜欢"கு","மா","ர்".

 public class one {
    public static void main(String[] args) {
            String str1 = new String("??????");
            System.out.print(str1.length());
    }
}

PS:这是泰米尔语.

Answer 1

找到了解决问题的方法.

基于这个SO答案,我创建了一个程序,使用正则表达式字符类来搜索可能具有可选修饰符的字母.它将您的字符串拆分为单个(必要时组合)字符并将它们放入列表中:

import java.util.*;
import java.lang.*;
import java.util.regex.*;

class Main
{
    public static void main (String[] args)
    {
        String s="??????";
        List<String> characters=new ArrayList<String>();
        Pattern pat = Pattern.compile("\\p{L}\\p{M}*");
        Matcher matcher = pat.matcher(s);
        while (matcher.find()) {
            characters.add(matcher.group());            
        }

        // Test if we have the right characters and length
        System.out.println(characters);
        System.out.println("String length: " + characters.size());

    }
}

其中\\p{L}表示Unicode字母,\\p{M}表示Unicode标记.

代码段的输出是:

??
??
??
String length: 3

有关正常运行的演示,请访问https://ideone.com/Apkapn

编辑

我现在用http://en.wikipedia.org/wiki/Tamil_script中的表格中的所有有效泰米尔语字母检查我的正则表达式.我发现使用当前的正则表达式我们没有正确捕获所有字母(Grantha复合表中最后一行中的每个字母都被分成两个字母),所以我将我的正则表达式改进为以下解决方案:

Pattern pat = Pattern.compile("\u0B95\u0BCD\u0BB7\\p{M}?|\\p{L}\\p{M}?");

使用此模式而不是上述模式,您应该能够将句子分成每个有效的泰米尔语字母(只要维基百科的表格完整).

我用来检查的代码如下:

String s = "???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????";
List<String> characters = new ArrayList<String>();
Pattern pat = Pattern.compile("\u0B95\u0BCD\u0BB7\\p{M}?|\\p{L}\\p{M}?");
Matcher matcher = pat.matcher(s);
while (matcher.find()) {
    characters.add(matcher.group());
}

System.out.println(characters);
System.out.println(characters.size() == 325);

Answer 2

看看Normalizer类.有可能是您的问题的原因的解释.在Unicode中,您可以通过多种方式对字符进行编码,例如Á:

  U+00C1    LATIN CAPITAL LETTER A WITH ACUTE

要么

  U+0041    LATIN CAPITAL LETTER A
  U+0301    COMBINING ACUTE ACCENT

您可以尝试使用Normalizer将字符串转换为组合形式,然后迭代字符.

编辑:根据上面@halex建议的文章,在Java中尝试:

    String str = new String("??????");

    ArrayList<String> characters = new ArrayList<String>();
    str = Normalizer.normalize(str, Form.NFC);
    StringBuilder charBuffer = new StringBuilder();
    for (int i = 0; i < str.length(); i++) {
        int codePoint = str.codePointAt(i);
        int category = Character.getType(codePoint);
        if (charBuffer.length() > 0
                && category != Character.NON_SPACING_MARK
                && category != Character.COMBINING_SPACING_MARK
                && category != Character.CONTROL
                && category != Character.OTHER_SYMBOL) {
            characters.add(charBuffer.toString());
            charBuffer.delete(0, charBuffer.length());
        }
        charBuffer.appendCodePoint(codePoint);
    }
    if (charBuffer.length() > 0) {
        characters.add(charBuffer.toString());
    }
    System.out.println(characters);

我得到的结果是[??, ??, ??].如果它不适用于所有字符串,请尝试使用if块中的其他Unicode字符类别进行fiddeling .

Answer 3

这原来是真的丑....我已经调试您的字符串,它包含以下字符(和它们的十六进制位置):

க0
x0b95ு
0x0bc1
ம0
x0baeா
0x0bbe ர0 x0bb0் 0x0bcd

所以泰米尔语显然使用类似变音符的序列来获取所有不幸被视为单独实体的字符.

这不是UTF-8/UTF-16的问题,正如其他答案所暗示的那样,它是泰米尔语的Unicode编码所固有的.

建议的规范化器不起作用,似乎tamil由Unicode"专家"设计,以明确使用无法规范化的组合序列.AARGH.

我的下一个想法是不计算字符,而是字形,字符的可视化表示.

String str1 = new String(Normalizer.normalize("??????", Normalizer.Form.NFC ));

Font display = new Font("SansSerif",Font.PLAIN,12);
GlyphVector vec = display.createGlyphVector(new FontRenderContext(new AffineTransform(),false, false),str1);

System.out.println(vec.getNumGlyphs());
for (int i=0; i<str1.length(); i++)
        System.out.printf("%s %s %s %n",str1.charAt(i),Integer.toHexString((int) str1.charAt(i)),vec.getGlyphVisualBounds(i).getBounds2D().toString());

结果:

கb95[x = 0.0,y = -6.0,w = 7.0,h = 6.0]
ுbc1[x = 8.0,y = -6.0,w = 7.0,h = 4.0]
மbae [x = 17.0,y = - 6.0,w = 6.0,h = 6.0]
ாbbe [x = 23.0,y = -6.0,w = 5.0,h = 6.0]
ரbb0[x = 30.0,y = -6.0,w = 4.0,h = 8.0]
்bcd [x = 31.0,y = -9.0,w = 1.0,h = 2.0]

由于字形相交,您需要使用Java字符类型函数,就像在其他解决方案中一样.

解:

我正在使用此链接:http://www.venkatarangan.com/blog/content/binary/Counting%20Letters%20in%20an%20Unicode%20String.pdf

public static int getTamilStringLength(String tamil) {
    int dependentCharacterLength = 0;
    for (int index = 0; index < tamil.length(); index++) {
        char code = tamil.charAt(index);
        if (code == 0xB82)
            dependentCharacterLength++;
        else if (code >= 0x0BBE && code <= 0x0BC8)
            dependentCharacterLength++;
        else if (code >= 0x0BCA && code <= 0x0BD7)
            dependentCharacterLength++;
    }
    return tamil.length() - dependentCharacterLength;
  }

您需要排除组合字符并相应地计算它们.