为什么单词字符(\ w)在使用locale pragma下不匹配？

Question

当我use locale,我的语言环境(et_EE.UTF-8)中的一些字符与之匹配时\w,我没有看到任何理由.

除了ASCII之外,爱沙尼亚还使用了六个字符:

õäöüšž

在我下面的测试脚本中,我使用$string了三个额外的特殊字符ð??(不属于爱沙尼亚字母).

use feature 'say';
use POSIX qw( locale_h );

{
  use utf8;
  my  $string = "õäöüšž ð??";
  binmode STDOUT, ":encoding(UTF-8)";
  say "nothing";
  say 'LOCALE: ', setlocale(LC_CTYPE), ' ', setlocale(LC_COLLATE);
  say 'UC: ', uc( $string );
  say 'SORT: ', sort( split(//, $string) );
  say $string =~ m/\w/g;
  say $string =~ m/\p{Word}/g;
  say '';
}

{
  use utf8;
  use locale;
  binmode STDOUT, ":encoding(UTF-8)";
  my  $string = "õäöüšž ð??";
  say "locale";
  say 'LOCALE: ', setlocale(LC_CTYPE), ' ', setlocale(LC_COLLATE);
  say 'UC: ', uc( $string );
  say 'SORT: ', sort( split(//, $string) );
  say $string =~ m/\w/g;
  say $string =~ m/\p{Word}/g;
  say '';
}

{
  use utf8::all;
  my  $string = "õäöüšž ð??";
  say "utf8::all";
  say 'LOCALE: ', setlocale(LC_CTYPE), ' ', setlocale(LC_COLLATE);
  say 'UC: ', uc( $string );
  say 'SORT: ', sort( split(//, $string) );
  say $string =~ m/\w/g;
  say $string =~ m/\p{Word}/g;
  say '';
}

{
  use utf8::all;
  use locale;
  my  $string = "õäöüšž ð??";
  say "utf8::all + locale";
  say 'LOCALE: ', setlocale(LC_CTYPE), ' ', setlocale(LC_COLLATE);
  say 'UC: ', uc( $string );
  say 'SORT: ', sort( split(//, $string) );
  say $string =~ m/\w/g;
  say $string =~ m/\p{Word}/g;
  say '';
}

我试过Perl 5.10.1和5.14.2并且都给了我这样的输出:

nothing
LOCALE: et_EE.UTF-8 et_EE.UTF-8
UC: ÕÄÖÜŠŽ Ð??
SORT:  äðõöü?šž?
õäöüšžð??
õäöüšžð??

locale
LOCALE: et_EE.UTF-8 et_EE.UTF-8
UC: ÕÄÖÜŠŽ Ð??
SORT:  ð?šžõäöü?
šž??
õäöüšžð??

utf8::all
LOCALE: et_EE.UTF-8 et_EE.UTF-8
UC: ÕÄÖÜŠŽ Ð??
SORT:  äðõöü?šž?
õäöüšžð??
õäöüšžð??

utf8::all + locale
LOCALE: et_EE.UTF-8 et_EE.UTF-8
UC: ÕÄÖÜŠŽ Ð??
SORT:  ð?šžõäöü?
šž??
õäöüšžð??

什么不是我想象的？

• 主要问题:在use locale我希望\w匹配我的所有六个字符,但结果šž??是相当奇怪的.为何如此匹配？从perlrecharclass我读到:

对于高于255的代码点...\w与此范围内的\ p {Word}匹配相同....对于低于256的代码点...如果区域设置规则生效...\w匹配平台的本机下划线字符以及区域设置认为是字母数字的任何内容.

因此,\w匹配高于255的字符,但不匹配"无论语言环境认为是字母数字".为什么？同时在语言环境下排序工作正常(并且没有语言环境),结果ð?šžõäöü?是正确的顺序,这表明正确表示了正确的字符.AFAIU,如果不知道它们"无论语言环境认为是字母数字",它就无法正常工作.要么？

• 我认为setlocale只在locale-pragma下给出结果.我如何测试哪个区域设置对范围有效？
• 我没想到在每个测试用例中所有字符都是大写的.AFAIU uc并且lc应该依赖于语言环境.在第一种情况下,我认为他们都会低调,但使用现场我等前六个字符是上限,而其他人没有.只有我等待所有字符大写的情况,才是第三.我看到我错过了一些重要的事情.哎呀,现在我从lc文档中找到:"否则,如果EXPR设置了UTF-8标志:Unicode语义用于案例更改." UTF-8标志总是设置在我的上面$string,因此在写入时得到了答案.

使用locale排序和\p{Word}进行匹配是可以接受的,但我仍然会用一些提示:为什么\w如我所料不工作？

Answer 1

请不要使用破碎的use localepragma.

请,请使用Unicode::Collate::Localelocale collat​​ion.它使用CLDR规则,并且完全可移植,并且不依赖于狡猾的破坏POSIX语言环境,而这些语言环境根本无法正常工作.

如果按代码点排序,则会产生废话,但如果使用使用Unicode::Collate::Locale爱沙尼亚语言环境构造的对象进行排序,则可以得到合理的结果:

Codepoint sort:  äðõöü?šž?
Estonian  sort:  ð?šžõäöü?

此外,当您执行此原始代码点排序时,您会受到规范化问题的严重影响.考虑:

NFC/NFD sort by codepoint is DIFFERENT
NFC Codepoint sort:  äðõöü?šž?
NFD Codepoint sort:  a?o?o?s?u?z?ð??

NFC/NFD sort in estonian  is SAME
NFC Estonian  sort:  ð?šžõäöü?
NFD Estonian  sort:  ð?s?z?o?a?o?u??

以下是制作所有这些的演示程序.

#!/usr/bin/env perl
#
# et-demo - show how to handle Estonian collation correctly
#
# Tom Christinansen <tchrist@perl.com>
# Fri Feb 22 19:27:51 MST 2013

use v5.14;
use utf8;
use strict;
use warnings;
use warnings FATAL => "utf8";
use open qw(:std :utf8);

use Unicode::Normalize;
use Unicode::Collate::Locale;

main();
exit();

sub graphemes(_) {
    my($str) = @_;
    my @graphs = $str =~ /\X/g;
    return @graphs;
}

sub same_diff($$) {
    my($s1, $s2) = @_;
    no locale;

    if (NFC($s1) eq NFC($s2)) {
        return "SAME";
    } else {
        return "DIFFERENT";
    }
}

sub stringy {
    return join("" => @_);
}

sub cp_sort {
    no locale;
    return sort @_;
}

sub et_sort {
    state $collator = # we want Estonian here:
        Unicode::Collate::Locale->new(locale => "et");
    return $collator->sort(@_);
}

sub main {
    my $orig = "õäöüšž ð??";

    say "    Codepoint sort: ", cp_sort(graphemes($orig));
    say "    Estonian  sort: ", et_sort(graphemes($orig));

    my $nfc = NFC($orig);
    my $nfc_cp_sort = stringy cp_sort(graphemes($nfc));
    my $nfc_et_sort = stringy et_sort(graphemes($nfc));

    my $nfd = NFD($orig);
    my $nfd_cp_sort = stringy cp_sort(graphemes($nfd));
    my $nfd_et_sort = stringy et_sort(graphemes($nfd));

    say "NFC/NFD sort by codepoint is ",
        same_diff($nfc_cp_sort, $nfd_cp_sort);

    say "NFC Codepoint sort: ", $nfc_cp_sort;
    say "NFD Codepoint sort: ", $nfd_cp_sort;

    say "NFC/NFD sort in estonian  is ",
        same_diff($nfc_et_sort, $nfd_et_sort);

    say "NFC Estonian  sort: ", $nfc_et_sort;
    say "NFD Estonian  sort: ", $nfd_et_sort;

}

这就是你应该如何处理区域设置整理.有关大量示例,请参阅此答案.