std :: string和UTF-8编码的unicode

Question

如果我理解得很好,可以使用string和wstring来存储UTF-8文本.

• With char, ASCII characters take a single byte, some chinese characters take 3 or 4, etc. Which means that str[3] doesn't necessarily point to the 4th character.

• With wchar_t same thing, but the minimal amount of bytes used per characters is always 2 (instead of 1 for char), and a 3 or 4 byte wide character will take 2 wchar_t.

Right ?

So, what if I want to use string::find_first_of() or string::compare(), etc with such a weirdly encoded string ? Will it work ? Does the string class handle the fact that characters have a variable size ? Or should I only use them as dummy feature-less byte arrays, in which case I'd rather go for a wchar_t[] buffer.

If std::string doesn't handle that, second question: are there libraries providing string classes that could handle that UTF-8 encoding so that str[3] actually points to the 3rd character (which would be a byte array from length 1 to 4) ?

Answer 1

你在谈论Unicode.Unicode使用32位来表示字符.然而,由于这会浪费内存,因此存在更紧凑的编码.UTF-8就是这样一种编码.它假定您使用字节单位并将Unicode字符映射到1,2,3或4个字节.UTF-16是另一种使用单词作为单位并将Unicode字符映射到1或2个字(2或4个字节)的UTF-16.您可以同时使用string和wchar_t进行编码.对于英文文本/数字,UTF-8往往更紧凑.

无论使用哪种编码和类型(比较),有些东西都会起作用.但是,所有需要理解一个角色的功能都将被破坏.即第5个字符并不总是底层数组中的第5个字符.它可能看起来像是在使用某些示例,但它最终会破坏.string :: compare可以工作,但不希望按字母顺序排序.这取决于语言.string :: find_first_of适用于某些但不是全部.长字符串可能只是因为它们很长而较短,而较短字符串可能会被字符对齐混淆并产生非常难以发现的错误.

最好的办法是找到一个为你处理它的库,并忽略下面的类型(除非你有充分的理由选择其中一个).