使用Boost :: spirit编写的解析器的性能问题

Question

使用Boost :: spirit编写的解析器的性能问题

Mar*_*ler 4 c++ parsing boost boost-spirit-qi c++11

我想解析一个看起来像这样的文件(类似FASTA的文本格式):

    >InfoHeader
    "Some text sequence that has a line break after every 80 characters"
    >InfoHeader
    "Some text sequence that has a line break after every 80 characters"
    ...

Run Code Online (Sandbox Code Playgroud)

例如:

    >gi|31563518|ref|NP_852610.1| microtubule-associated proteins 1A/1B light chain 3A isoform b [Homo sapiens]
    MKMRFFSSPCGKAAVDPADRCKEVQQIRDQHPSKIPVIIERYKGEKQLPVLDKTKFLVPDHVNMSELVKI
    IRRRLQLNPTQAFFLLVNQHSMVSVSTPIADIYEQEKDEDGFLYMVYASQETFGFIRENE

Run Code Online (Sandbox Code Playgroud)

我用boost :: spirit为此编写了一个解析器.解析器正确地存储标题行和下面的文本序列,std::vector< std::pair< string, string >>但是对于较大的文件(对于100MB文件为17秒)需要很长时间.作为比较,我编写了一个没有boost :: spirit(只是STL函数)的程序,只需复制一个100MB文件的每一行std::vector.整个过程不到一秒钟.用于比较的"程序"不符合目的,但我不认为解析器应该花更长的时间......

我知道有很多其他的FASTA解析器,但我很好奇为什么我的代码很慢.

.hpp文件:

#include <boost/filesystem/path.hpp>

namespace fs = boost::filesystem;


class FastaReader {

public:
    typedef std::vector< std::pair<std::string, std::string> > fastaVector;

private:
    fastaVector fV;
    fs::path file;  

public:
    FastaReader(const fs::path & f);
    ~FastaReader();

    const fs::path & getFile() const;
    const fastaVector::const_iterator getBeginIterator() const;
    const fastaVector::const_iterator getEndIterator() const;   

private:
    void parse();

};

Run Code Online (Sandbox Code Playgroud)

和.cpp文件:

#include <iomanip>
#include <boost/date_time/posix_time/posix_time.hpp>
#include <boost/filesystem/fstream.hpp>
#include <boost/filesystem/operations.hpp>
#include <boost/filesystem/path.hpp>
#include <boost/spirit/include/classic_position_iterator.hpp>
#include <boost/spirit/include/phoenix_bind.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_fusion.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <boost/spirit/include/qi.hpp>
#include "fastaReader.hpp"


using namespace std;

namespace fs = boost::filesystem;
namespace qi = boost::spirit::qi;
namespace pt = boost::posix_time;

template <typename Iterator, typename Skipper>
struct FastaGrammar : qi::grammar<Iterator, FastaReader::fastaVector(), qi::locals<string>, Skipper> {
    qi::rule<Iterator> infoLineStart;
    qi::rule<Iterator> inputEnd;
    qi::rule<Iterator> lineEnd;
    qi::rule<Iterator, string(), Skipper> infoLine;
    qi::rule<Iterator, string(), Skipper> seqLine;
    qi::rule<Iterator, FastaReader::fastaVector(), qi::locals<string>, Skipper> fasta;


    FastaGrammar() : FastaGrammar::base_type(fasta, "fasta") {
        using boost::spirit::standard::char_;
        using boost::phoenix::bind;
        using qi::eoi;
        using qi::eol;
        using qi::lexeme;
        using qi::_1;
        using qi::_val;
        using namespace qi::labels;

        infoLineStart = char_('>');
        inputEnd = eoi;

        /* grammar */       
        infoLine = lexeme[*(char_ - eol)];
        seqLine = *(char_ - infoLineStart);

        fasta = *(infoLineStart > infoLine[_a = _1] 
            > seqLine[bind(&FastaGrammar::addValue, _val, _a, _1)]
            )
            > inputEnd
        ;

        infoLineStart.name(">");
        infoLine.name("sequence identifier");
        seqLine.name("sequence");

    }

    static void addValue(FastaReader::fastaVector & fa, const string & info, const string & seq) {
        fa.push_back(make_pair(info, seq));
    }
};


FastaReader::FastaReader(const fs::path & f) {
    this->file = f; 
    this->parse();
}


FastaReader::~FastaReader() {}


const fs::path & FastaReader::getFile() const {
    return this->file;
}


const FastaReader::fastaVector::const_iterator FastaReader::getBeginIterator() const {
    return this->fV.cbegin();
}


const FastaReader::fastaVector::const_iterator FastaReader::getEndIterator() const {
    return this->fV.cend();
}


void FastaReader::parse() {
    if ( this->file.empty() ) throw string("FastaReader: No file specified.");
    if ( ! fs::is_regular_file(this->file) ) throw (string("FastaReader: File not found: ") + this->file.string());

    typedef boost::spirit::istream_iterator iterator_type;
    typedef boost::spirit::classic::position_iterator2<iterator_type> pos_iterator_type;
    typedef FastaGrammar<pos_iterator_type, boost::spirit::ascii::space_type> fastaGr;

    fs::ifstream fin(this->file);
    if ( ! fin.is_open() ) {
        throw (string("FastaReader: Access denied: ") + this->file.string());
    }

    fin.unsetf(ios::skipws);

    iterator_type begin(fin);
    iterator_type end;

    pos_iterator_type pos_begin(begin, end, this->file.string());
    pos_iterator_type pos_end;

    fastaGr fG;
    try {
        std::cerr << "Measuring: Parsing." << std::endl;
        const pt::ptime startMeasurement = pt::microsec_clock::universal_time();

        qi::phrase_parse(pos_begin, pos_end, fG, boost::spirit::ascii::space, this->fV);

        const pt::ptime endMeasurement = pt::microsec_clock::universal_time();
        pt::time_duration duration (endMeasurement - startMeasurement);
        std::cerr << duration <<  std::endl;
    } catch (std::string str) {
        cerr << "error message: " << str << endl;
    }   
}

Run Code Online (Sandbox Code Playgroud)

因此语法执行以下操作:它查找">"符号,然后存储所有后续字符,直到检测到EOL.在EOL之后,文本序列在检测到">"符号时开始和结束.然后通过调用将两个字符串(标题行和文本序列)存储在std :: vector中FastaReader::addValue().

我使用带有-O2和-std = c ++ 11标志的g ++版本4.8.2编译了我的程序.

那么我的代码中的性能问题在哪里呢？

Answer 1

seh*_*ehe 8

_{上一页:第3步:MOAR更快,零复制}
_{返回步骤1.清理+分析}

第4步:删除位置迭代器

由于你没有使用它,我们可以删除有状态迭代器,这可能会抑制很多优化(并且在探查器输出中间接可见)

Live On Coliru

#define BOOST_SPIRIT_USE_PHOENIX_V3
#include <boost/filesystem/path.hpp>
#include <boost/utility/string_ref.hpp>
#include <boost/iostreams/device/mapped_file.hpp>
namespace io = boost::iostreams;
namespace fs = boost::filesystem;


class FastaReader {

public:
    typedef std::pair<boost::string_ref, boost::string_ref> Entry;
    typedef std::vector<Entry> Data;

private:
    Data fV;
    fs::path file;  

public:
    FastaReader(const fs::path & f);
    ~FastaReader();

    const fs::path & getFile() const;
    const Data::const_iterator begin() const;
    const Data::const_iterator end() const;   

private:
    io::mapped_file_source mmap;
    void parse();

};

#include <iomanip>
#include <boost/date_time/posix_time/posix_time.hpp>
#include <boost/filesystem/fstream.hpp>
#include <boost/filesystem/operations.hpp>
#include <boost/filesystem/path.hpp>

#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <boost/fusion/adapted/std_pair.hpp>
//#include "fastaReader.hpp"

#include <boost/iostreams/device/mapped_file.hpp>

using namespace std;

namespace fs = boost::filesystem;
namespace qi = boost::spirit::qi;
namespace pt = boost::posix_time;
namespace io = boost::iostreams;

namespace boost { namespace spirit { namespace traits {
    template <typename It>
    struct assign_to_attribute_from_iterators<boost::string_ref, It, void> {
        static void call(It f, It l, boost::string_ref& attr) { attr = boost::string_ref { f, size_t(std::distance(f,l)) }; }
    };
} } }

template <typename Iterator>
struct FastaGrammar : qi::grammar<Iterator, FastaReader::Data()> {

    FastaGrammar() : FastaGrammar::base_type(fasta) {
        using namespace qi;
        using boost::phoenix::construct;
        using boost::phoenix::begin;
        using boost::phoenix::size;

        entry = ('>' >> raw[ *~char_('\n') ] >> '\n' >> raw[ *~char_('>') ]);
        fasta = *entry >> *eol >> eoi ;

        BOOST_SPIRIT_DEBUG_NODES((fasta)(entry));
    }
  private:
    qi::rule<Iterator, FastaReader::Data()>  fasta;
    qi::rule<Iterator, FastaReader::Entry()> entry;
};

FastaReader::FastaReader(const fs::path & f) : file(f), mmap(file.c_str()) {
    parse();
}

FastaReader::~FastaReader() {}

const fs::path & FastaReader::getFile() const {
    return this->file;
}


const FastaReader::Data::const_iterator FastaReader::begin() const {
    return this->fV.cbegin();
}


const FastaReader::Data::const_iterator FastaReader::end() const {
    return this->fV.cend();
}

void FastaReader::parse() {
    if (this->file.empty())                throw std::runtime_error("FastaReader: No file specified.");
    if (! fs::is_regular_file(this->file)) throw std::runtime_error(string("FastaReader: File not found: ") + this->file.string());

    typedef char const*                  iterator_type;
    typedef FastaGrammar<iterator_type>  fastaGr;

    static const fastaGr fG{};
    try {
        std::cerr << "Measuring: Parsing." << std::endl;
        const pt::ptime startMeasurement = pt::microsec_clock::universal_time();

        iterator_type first(mmap.data()), last(mmap.end());
        qi::phrase_parse(first, last, fG, boost::spirit::ascii::space, this->fV);

        const pt::ptime endMeasurement = pt::microsec_clock::universal_time();
        pt::time_duration duration (endMeasurement - startMeasurement);
        std::cerr << duration <<  std::endl;
    } catch (std::exception const& e) {
        cerr << "error message: " << e.what() << endl;
    }   
}

int main() {
    FastaReader reader("input.txt");

    for (auto& e : reader) std::cout << '>' << e.first << '\n' << e.second << "\n\n";
}

Run Code Online (Sandbox Code Playgroud)

现在它快了74.8倍.

$ time ./test | head -n4
Measuring: Parsing.
00:00:00.194432

Run Code Online (Sandbox Code Playgroud)

Answer 2

seh*_*ehe 6

_{下一步:第2步.更快mmap}

步骤1.清理+分析

你应该避免它们引入类型擦除的许多规则.

如果你的输入是理智的,你可以不用船长(无论如何,线路末端很重要,所以跳过它们是没有意义的).

使用融合自适应而不是助手来构建新对:

这不是最佳的,但更清洁:

$ ./test1
Measuring: Parsing.
00:00:22.681605

Run Code Online (Sandbox Code Playgroud)

通过减少运动部件和间接性来提高效率:

Live On Coliru

#include <boost/filesystem/path.hpp>

namespace fs = boost::filesystem;

class FastaReader {    
public:
    typedef std::pair<std::string, std::string> Entry;
    typedef std::vector<Entry> Data;

private:
    Data fV;
    fs::path file;  

public:
    FastaReader(const fs::path & f);
    ~FastaReader();

    const fs::path & getFile() const;
    const Data::const_iterator begin() const;
    const Data::const_iterator end() const;   

private:
    void parse();    
};

#include <iomanip>
#include <boost/date_time/posix_time/posix_time.hpp>
#include <boost/filesystem/fstream.hpp>
#include <boost/filesystem/operations.hpp>
#include <boost/filesystem/path.hpp>

#include <boost/spirit/include/classic_position_iterator.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/fusion/adapted/std_pair.hpp>
//#include "fastaReader.hpp"

using namespace std;

namespace fs = boost::filesystem;
namespace qi = boost::spirit::qi;
namespace pt = boost::posix_time;

template <typename Iterator>
struct FastaGrammar : qi::grammar<Iterator, FastaReader::Data()> {
    qi::rule<Iterator, FastaReader::Data()> fasta;

    FastaGrammar() : FastaGrammar::base_type(fasta) {
        using namespace qi;

        fasta = *('>' >> *~char_('\n') >> '\n' 
                      >> *~char_('>')) 
                >> *eol
                >> eoi
                ;

        BOOST_SPIRIT_DEBUG_NODES((fasta));
    }
};


FastaReader::FastaReader(const fs::path & f) : file(f) {
    parse();
}

FastaReader::~FastaReader() {}

const fs::path & FastaReader::getFile() const {
    return this->file;
}

const FastaReader::Data::const_iterator FastaReader::begin() const {
    return this->fV.cbegin();
}

const FastaReader::Data::const_iterator FastaReader::end() const {
    return this->fV.cend();
}

void FastaReader::parse() {
    if (this->file.empty())                throw std::runtime_error("FastaReader: No file specified.");
    if (! fs::is_regular_file(this->file)) throw std::runtime_error(string("FastaReader: File not found: ") + this->file.string());

    typedef boost::spirit::istream_iterator                           iterator_type;
    typedef boost::spirit::classic::position_iterator2<iterator_type> pos_iterator_type;
    typedef FastaGrammar<pos_iterator_type>                           fastaGr;

    fs::ifstream fin(this->file);
    if (!fin) {
        throw std::runtime_error(string("FastaReader: Access denied: ") + this->file.string());
    }

    static const fastaGr fG{};
    try {
        std::cerr << "Measuring: Parsing." << std::endl;
        const pt::ptime startMeasurement = pt::microsec_clock::universal_time();

        pos_iterator_type first(iterator_type{fin >> std::noskipws}, {}, file.string());
        qi::phrase_parse<pos_iterator_type>(first, {}, fG, boost::spirit::ascii::space, this->fV);

        const pt::ptime endMeasurement = pt::microsec_clock::universal_time();
        pt::time_duration duration (endMeasurement - startMeasurement);
        std::cerr << duration <<  std::endl;
    } catch (std::exception const& e) {
        cerr << "error message: " << e.what() << endl;
    }   
}

int main() {
    std::ios::sync_with_stdio(false);

    FastaReader reader("input.txt");

    //for (auto& e : reader) std::cout << '>' << e.first << '\n' << e.second << "\n\n";
}

Run Code Online (Sandbox Code Playgroud)

这仍然很慢.让我们看看需要这么长时间:

在此输入图像描述

这很漂亮,但很难告诉我们我们需要知道什么.然而,这确实是:前N时代的消费者

在此输入图像描述

所以大部分时间都花在了istream迭代和多次传递适配器上.您可以争辩说多通道适配器可以通过偶尔刷新一次(每一行？)进行优化,但实际上,我们宁愿不依赖于(流)缓冲区上的整个流和运算符.

所以,我虽然让我们使用映射文件:

下一步:第2步.更快mmap

Answer 3

seh*_*ehe 5

_{上一页:步骤1.清理+分析}
_{下一步:第3步:MOAR更快,零复制}

第2步.更快 `mmap`

Live On Coliru

#include <boost/filesystem/path.hpp>

namespace fs = boost::filesystem;


class FastaReader {

public:
    typedef std::pair<std::string, std::string> Entry;
    typedef std::vector<Entry> Data;

private:
    Data fV;
    fs::path file;  

public:
    FastaReader(const fs::path & f);
    ~FastaReader();

    const fs::path & getFile() const;
    const Data::const_iterator begin() const;
    const Data::const_iterator end() const;   

private:
    void parse();

};

#include <iomanip>
#include <boost/date_time/posix_time/posix_time.hpp>
#include <boost/filesystem/fstream.hpp>
#include <boost/filesystem/operations.hpp>
#include <boost/filesystem/path.hpp>

#include <boost/spirit/include/classic_position_iterator.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/fusion/adapted/std_pair.hpp>
//#include "fastaReader.hpp"

#include <boost/iostreams/device/mapped_file.hpp>

using namespace std;

namespace fs = boost::filesystem;
namespace qi = boost::spirit::qi;
namespace pt = boost::posix_time;
namespace io = boost::iostreams;

template <typename Iterator>
struct FastaGrammar : qi::grammar<Iterator, FastaReader::Data()> {
    qi::rule<Iterator, FastaReader::Data()> fasta;

    FastaGrammar() : FastaGrammar::base_type(fasta) {
        using namespace qi;

        fasta = *('>' >> *~char_('\n') >> '\n' 
                      >> *~char_('>')) 
                >> *eol
                >> eoi
                ;

        BOOST_SPIRIT_DEBUG_NODES((fasta));
    }
};


FastaReader::FastaReader(const fs::path & f) : file(f) {
    parse();
}

FastaReader::~FastaReader() {}

const fs::path & FastaReader::getFile() const {
    return this->file;
}


const FastaReader::Data::const_iterator FastaReader::begin() const {
    return this->fV.cbegin();
}


const FastaReader::Data::const_iterator FastaReader::end() const {
    return this->fV.cend();
}

void FastaReader::parse() {
    if (this->file.empty())                throw std::runtime_error("FastaReader: No file specified.");
    if (! fs::is_regular_file(this->file)) throw std::runtime_error(string("FastaReader: File not found: ") + this->file.string());

    typedef char const*                                               iterator_type;
    typedef boost::spirit::classic::position_iterator2<iterator_type> pos_iterator_type;
    typedef FastaGrammar<pos_iterator_type>                           fastaGr;

    io::mapped_file_source mmap(file.c_str());

    static const fastaGr fG{};
    try {
        std::cerr << "Measuring: Parsing." << std::endl;
        const pt::ptime startMeasurement = pt::microsec_clock::universal_time();

        pos_iterator_type first(iterator_type{mmap.data()}, iterator_type{mmap.end()}, file.string());
        qi::phrase_parse<pos_iterator_type>(first, {}, fG, boost::spirit::ascii::space, this->fV);

        const pt::ptime endMeasurement = pt::microsec_clock::universal_time();
        pt::time_duration duration (endMeasurement - startMeasurement);
        std::cerr << duration <<  std::endl;
    } catch (std::exception const& e) {
        cerr << "error message: " << e.what() << endl;
    }   
}

int main() {
    FastaReader reader("input.txt");

    //for (auto& e : reader) std::cout << '>' << e.first << '\n' << e.second << "\n\n";
}

Run Code Online (Sandbox Code Playgroud)

事实上,在我的系统上,它大约快3倍(输入为229 MiB):

$ ./mapped_file_source
Measuring: Parsing.
00:00:07.385787

Run Code Online (Sandbox Code Playgroud)

下一步:第3步:MOAR更快,零复制

Answer 4

seh*_*ehe 5

_{上一页:步骤2.mmap} _下一步_更快
_{:步骤4:删除位置迭代器}

第3步:MOAR更快,零复制

我们避免分配!如果我们将文件映射移动到FastaReader类中,我们可以直接指向映射中的数据,而不是一直复制字符串.

使用boost :: string_ref,如下所述:C++:快速读取映射文件到你可以做的矩阵

Live On Coliru

#define BOOST_SPIRIT_USE_PHOENIX_V3
#include <boost/filesystem/path.hpp>
#include <boost/utility/string_ref.hpp>
#include <boost/iostreams/device/mapped_file.hpp>
namespace io = boost::iostreams;
namespace fs = boost::filesystem;

class FastaReader {

public:
    typedef std::pair<boost::string_ref, boost::string_ref> Entry;
    typedef std::vector<Entry> Data;

private:
    Data fV;
    fs::path file;  

public:
    FastaReader(const fs::path & f);
    ~FastaReader();

    const fs::path & getFile() const;
    const Data::const_iterator begin() const;
    const Data::const_iterator end() const;   

private:
    io::mapped_file_source mmap;
    void parse();

};

#include <iomanip>
#include <boost/date_time/posix_time/posix_time.hpp>
#include <boost/filesystem/fstream.hpp>
#include <boost/filesystem/operations.hpp>
#include <boost/filesystem/path.hpp>

#include <boost/spirit/include/classic_position_iterator.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>
#include <boost/fusion/adapted/std_pair.hpp>
//#include "fastaReader.hpp"

#include <boost/iostreams/device/mapped_file.hpp>

using namespace std;

namespace fs = boost::filesystem;
namespace qi = boost::spirit::qi;
namespace pt = boost::posix_time;
namespace io = boost::iostreams;

namespace boost { namespace spirit { namespace traits {
    template <typename It>
    struct assign_to_attribute_from_iterators<boost::string_ref, It, void> {
        static void call(It f, It l, boost::string_ref& attr) { attr = boost::string_ref { f.base(), size_t(std::distance(f.base(),l.base())) }; }
    };
} } }

template <typename Iterator>
struct FastaGrammar : qi::grammar<Iterator, FastaReader::Data()> {

    FastaGrammar() : FastaGrammar::base_type(fasta) {
        using namespace qi;
        using boost::phoenix::construct;
        using boost::phoenix::begin;
        using boost::phoenix::size;

        entry = ('>' >> raw[ *~char_('\n') ] >> '\n' >> raw[ *~char_('>') ]);
        fasta = *entry >> *eol >> eoi ;

        BOOST_SPIRIT_DEBUG_NODES((fasta)(entry));
    }
  private:
    qi::rule<Iterator, FastaReader::Data()>  fasta;
    qi::rule<Iterator, FastaReader::Entry()> entry;
};

FastaReader::FastaReader(const fs::path & f) : file(f), mmap(file.c_str()) {
    parse();
}

FastaReader::~FastaReader() {}

const fs::path & FastaReader::getFile() const {
    return this->file;
}


const FastaReader::Data::const_iterator FastaReader::begin() const {
    return this->fV.cbegin();
}


const FastaReader::Data::const_iterator FastaReader::end() const {
    return this->fV.cend();
}

void FastaReader::parse() {
    if (this->file.empty())                throw std::runtime_error("FastaReader: No file specified.");
    if (! fs::is_regular_file(this->file)) throw std::runtime_error(string("FastaReader: File not found: ") + this->file.string());

    typedef char const*                                               iterator_type;
    typedef boost::spirit::classic::position_iterator2<iterator_type> pos_iterator_type;
    typedef FastaGrammar<pos_iterator_type>                           fastaGr;

    static const fastaGr fG{};
    try {
        std::cerr << "Measuring: Parsing." << std::endl;
        const pt::ptime startMeasurement = pt::microsec_clock::universal_time();

        pos_iterator_type first(iterator_type{mmap.data()}, iterator_type{mmap.end()}, file.string());
        qi::phrase_parse<pos_iterator_type>(first, {}, fG, boost::spirit::ascii::space, this->fV);

        const pt::ptime endMeasurement = pt::microsec_clock::universal_time();
        pt::time_duration duration (endMeasurement - startMeasurement);
        std::cerr << duration <<  std::endl;
    } catch (std::exception const& e) {
        cerr << "error message: " << e.what() << endl;
    }   
}

int main() {
    FastaReader reader("input.txt");

    for (auto& e : reader) std::cout << '>' << e.first << '\n' << e.second << "\n\n";
}

Run Code Online (Sandbox Code Playgroud)

这确实已经快了4.8倍:

$ ./test3 | head -n4
Measuring: Parsing.
00:00:04.577123
>gi|31563518|ref|NP_852610.1| microtubule-associated proteins 1A/1B light chain 3A isoform b [Homo sapiens]
MKMRFFSSPCGKAAVDPADRCKEVQQIRDQHPSKIPVIIERYKGEKQLPVLDKTKFLVPDHVNMSELVKI
IRRRLQLNPTQAFFLLVNQHSMVSVSTPIADIYEQEKDEDGFLYMVYASQETFGFIRENE

Run Code Online (Sandbox Code Playgroud)

下一步:第4步:删除位置迭代器

归档时间：	10 年，9 月前
查看次数：	789 次
最近记录：	10 年，9 月前

使用Boost :: spirit编写的解析器的性能问题

第4步:删除位置迭代器

步骤1.清理+分析

第2步.更快 mmap

第3步:MOAR更快,零复制

第2步.更快 `mmap`