提升精神齐轨道线并解析unicode

Question

我想跟踪unicode字符串的输入位置和输入行.

对于位置,我存储一个迭代器,以便std::distance在所需的位置开始和使用.只要输入不是unicode,这就可以正常工作.使用unicode符号,位置会移动,即ä在输入流中占用两个空格,并且位置关闭1.因此,我切换到了boost::u8_to_u32_iterator,这样可以正常工作.

对于我使用的线boost::spirit::line_pos_iterator也很好用.

我的问题是将两个概念结合起来使用行迭代器和unicode迭代器.允许在unicode字符串上使用pos和line的另一种解决方案当然也是受欢迎的.

这是unicode解析器的一个小例子; 如上所述我想另外包装迭代器,boost::spirit::line_pos_iterator但甚至不编译.

#define BOOST_SPIRIT_USE_PHOENIX_V3
#define BOOST_SPIRIT_UNICODE
#include <boost/regex/pending/unicode_iterator.hpp>

#include <boost/fusion/adapted/struct.hpp>
#include <boost/spirit/include/phoenix.hpp>

namespace phx = boost::phoenix;

#include <boost/spirit/include/qi.hpp>

namespace qi = boost::spirit::qi;

#include <boost/spirit/repository/include/qi_iter_pos.hpp>
#include <boost/spirit/include/support_line_pos_iterator.hpp>

#include <iostream>
#include <string>

//==============================================================================
std::string to_utf8(const std::u32string& input) {
  return std::string(
      boost::u32_to_u8_iterator<std::u32string::const_iterator>(input.begin()),
      boost::u32_to_u8_iterator<std::u32string::const_iterator>(input.end()));
}

//==============================================================================
int main() {
  std::string input(u8"Hallo äöüß");

  typedef boost::u8_to_u32_iterator<std::string::const_iterator> iterator_type;

  iterator_type first(input.begin()), last(input.end());

  qi::rule<iterator_type, std::u32string()> string_u32 = *(qi::char_ - qi::eoi);

  qi::rule<iterator_type, std::string()> string =
      string_u32[qi::_val = phx::bind(&to_utf8, qi::_1)];

  qi::rule<iterator_type, std::string()> rule = string;

  std::string ast;
  bool result = qi::parse(first, last, rule, ast);
  if (result) {
    result = first == last;
  }

  if (result) {
    std::cout << "Parsed: " << ast << std::endl;
  } else {
    std::cout << "Failure" << std::endl;
  }
}

Answer 1

更新 演示在Coliru上添加了Live

当你试图包裹iterator_type一个时,我看到同样的问题line_pos_iterator.

经过一番思考后,我不太清楚是什么导致它(可能通过在迭代器适配器中包装u8_to_u32转换迭代器适配器来解决这个问题boost::spirit::multi_pass<>,但是......听起来很笨重,我甚至都没试过).

相反,我认为断行的本质是它(主要是？)charset不可知.因此,您可以line_pos_iterator在编码转换之前首先包装源迭代器.

这确实编译.当然,那么你将获得源迭代器的位置信息,而不是'逻辑字符' ^[1].

让我在下面展示一个示例.它将空格分隔的单词解析为字符串向量.显示位置信息的最简单方法是使用iterator_ranges 的向量而不是仅仅strings.我曾经qi::raw[]暴露过迭代器^[2].

因此,在成功解析后,我遍历匹配的范围并打印其位置信息.首先,我打印报告的实际位置line_pos_iterators.请记住,这些是"原始"字节偏移,因为源迭代器是面向字节的.

接下来,我get_current_line和u8_to_u32转换进行一点跳舞,将行内的偏移转换为(更多)逻辑计数.你会看到例如的范围

注意我目前假设范围不会跨越线边界(对于这种语法也是如此).否则,需要提取和转换2行.我现在这样做的方式相当昂贵.考虑通过例如使用Boost String Algorithm的find_all设施进行优化.您可以构建一个行尾列表,并用于std::lower_bound更有效地定位当前行.

注意有可能是用的实现问题get_line_start和get_current_line; 如果您注意到这样的事情,那么您可以尝试在[spirit-general]用户列表中找到10行补丁

不用多说,代码和输出:

#define BOOST_SPIRIT_USE_PHOENIX_V3
#define BOOST_SPIRIT_UNICODE
#include <boost/regex/pending/unicode_iterator.hpp>
#include <boost/fusion/adapted/struct.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <boost/phoenix/function/adapt_function.hpp>

namespace phx = boost::phoenix;

#include <boost/spirit/include/qi.hpp>

namespace qi       = boost::spirit::qi;
namespace encoding = boost::spirit::unicode;

#include <boost/spirit/repository/include/qi_iter_pos.hpp>
#include <boost/spirit/include/support_line_pos_iterator.hpp>

#include <iostream>
#include <string>

//==============================================================================
std::string to_utf8(const std::u32string& input) {
  return std::string(
      boost::u32_to_u8_iterator<std::u32string::const_iterator>(input.begin()),
      boost::u32_to_u8_iterator<std::u32string::const_iterator>(input.end()));
}

BOOST_PHOENIX_ADAPT_FUNCTION(std::string, to_utf8_, to_utf8, 1)

//==============================================================================
int main() {
    std::string input(u8"Hallo äöüß\n¡Bye! ????");

    typedef boost::spirit::line_pos_iterator<std::string::const_iterator> source_iterator;

    typedef boost::u8_to_u32_iterator<source_iterator> iterator_type;

    source_iterator soi(input.begin()), 
                    eoi(input.end());
    iterator_type   first(soi), 
                    last(eoi);

    qi::rule<iterator_type, std::u32string()> string_u32 = +encoding::graph;
    qi::rule<iterator_type, std::string()>    string     = string_u32 [qi::_val = to_utf8_(qi::_1)];

    std::vector<boost::iterator_range<iterator_type> > ast;
    // note the trick with `raw` to expose the iterators
    bool result = qi::phrase_parse(first, last, *qi::raw[ string ], encoding::space, ast);

    if (result) {
        for (auto const& range : ast)
        {
            source_iterator 
                base_b(range.begin().base()), 
                base_e(range.end().base());
            auto lbound = get_line_start(soi, base_b);

            // RAW access to the base iterators:
            std::cout << "Fragment: '" << std::string(base_b, base_e) << "'\t" 
                << "raw: L" << get_line(base_b) << ":" << get_column(lbound, base_b, /*tabs:*/4)
                <<     "-L" << get_line(base_e) << ":" << get_column(lbound, base_e, /*tabs:*/4);

            // "cooked" access:
            auto line = get_current_line(lbound, base_b, eoi);
            // std::cout << "Line: '" << line << "'\n";

            // iterator_type is an alias for u8_to_u32_iterator<...>
            size_t cur_pos = 0, start_pos = 0, end_pos = 0;
            for(iterator_type it = line.begin(), _eol = line.end(); ; ++it, ++cur_pos)
            {
                if (it.base() == base_b) start_pos = cur_pos;
                if (it.base() == base_e) end_pos   = cur_pos;

                if (it == _eol)
                    break;
            }
            std::cout << "\t// in u32 code _units_: positions " << start_pos << "-" << end_pos << "\n";
        }
        std::cout << "\n";
    } else {
        std::cout << "Failure" << std::endl;
    }

    if (first!=last)
    {
        std::cout << "Remaining: '" << std::string(first, last) << "'\n";
    }
}

输出:

clang++ -std=c++11 -Os main.cpp && ./a.out
Fragment: 'Hallo'   raw: L1:1-L1:6  // in u32 code _units_: positions 0-5
Fragment: 'äöüß'    raw: L1:7-L1:15 // in u32 code _units_: positions 6-10
Fragment: '¡Bye!'   raw: L2:2-L2:8  // in u32 code _units_: positions 1-6
Fragment: '????'    raw: L2:9-L2:21 // in u32 code _units_: positions 7-11

^[1]我认为在这种背景下对于角色的定义没有一个有用的定义.有字节,代码单元,代码点,字形集群,可能更多.只需说iterator(std::string::const_iterator)处理字节(因为它是charset /编码不知道).在u32string中,您可以/几乎/假设单个位置大致是一个代码点(尽管我认为(？)对于> L2 UNICODE支持,您仍然必须支持从多个代码单元组合的代码点).

^[2]这意味着当前属性转换和语义动作是多余的,但你会得到:)