通过字形而不是字符枚举字符串

Question

字符串通常按字符枚举.但是,特别是在使用Unicode和非英语语言时,有时我需要通过字形来枚举字符串.也就是说,组合标记和变音符号应该与它们修改的基本字符保持一致.在.Net中执行此操作的最佳方法是什么？

使用案例:计算一系列IPA单词中不同的语音.

1. 简化定义:字形和声音之间存在一对一的关系.
2. 逼真的定义:特殊的"字母"字符也应包含在基本字符中(例如pʰ),有些声音可以用连杆(k͡p)连接的两个符号表示.

Answer 1

简化方案

该TextElementEnumerator是非常有用和有效的:

private static List<SoundCount> CountSounds(IEnumerable<string> words)
{
    Dictionary<string, SoundCount> soundCounts = new Dictionary<string, SoundCount>();

    foreach (var word in words)
    {
        TextElementEnumerator graphemeEnumerator = StringInfo.GetTextElementEnumerator(word);
        while (graphemeEnumerator.MoveNext())
        {
            string grapheme = graphemeEnumerator.GetTextElement();

            SoundCount count;
            if (!soundCounts.TryGetValue(grapheme, out count))
            {
                count = new SoundCount() { Sound = grapheme };
                soundCounts.Add(grapheme, count);
            }
            count.Count++;
        }
    }

    return new List<SoundCount>(soundCounts.Values);
}

您也可以使用正则表达式执行此操作:(从文档中,TextElementEnumerator处理下面的表达式没有的一些情况,特别是补充字符,但这些非常罕见,在任何情况下我的应用程序都不需要.)

private static List<SoundCount> CountSoundsRegex(IEnumerable<string> words)
{
    var soundCounts = new Dictionary<string, SoundCount>();
    var graphemeExpression = new Regex(@"\P{M}\p{M}*");

    foreach (var word in words)
    {
        Match graphemeMatch = graphemeExpression.Match(word);
        while (graphemeMatch.Success)
        {
            string grapheme = graphemeMatch.Value;

            SoundCount count;
            if (!soundCounts.TryGetValue(grapheme, out count))
            {
                count = new SoundCount() { Sound = grapheme };
                soundCounts.Add(grapheme, count);
            }
            count.Count++;

            graphemeMatch = graphemeMatch.NextMatch();
        }
    }

    return new List<SoundCount>(soundCounts.Values);
}

性能:在我的测试中,我发现TextElementEnumerator的速度是正则表达式的4倍.

现实场景

不幸的是,没有办法"调整"TextElementEnumerator枚举的方式,因此该类在现实场景中没有用处.

一种解决方案是调整我们的正则表达式:

[\P{M}\P{Lm}]      # Match a character that is NOT a character intended to be combined with another character or a special character that is used like a letter
(?:                # Start a group for the combining characters:
  (?:                # Start a group for tied characters:
    [\u035C\u0361]      # Match an under- or over- tie bar...
    \P{M}\p{M}*         # ...followed by another grapheme (in the simplified sense)
  )                  # (End the tied characters group)
  |\p{M}             # OR a character intended to be combined with another character
  |\p{Lm}            # OR a special character that is used like a letter
)*                 # Match the combining characters group zero or more times.

我们也可以使用CharUnicodeInfo.GetUnicodeCategory创建我们自己的IEnumerator <string>来重新获得我们的性能,但这似乎对我来说太多了,需要额外的代码来维护.(还有其他人想要去吗？)Regex是为此而制作的.