如何创建一个类似std :: vector的类,它可以二进制读/写巨大的块？

Question

问题

我有这个旧的pre-stl C++代码,我想将其转换为std C++ 11而不会降低效率.

using T = unsigned;  // but can be any POD
FILE* fp = fopen( outfile.c_str(), "r" );
T* x = new T[big_n];
fread( x, sizeof(T), big_n, fp );
delete[] x;
fclose( fp );

请注意,big_n非常大 - 就像数百万条记录一样大,所以任何低效率都会发生.

以前的解决方案

在上一个问题的答案中,我接受了这个解决方案:

std::vector<T> x(big_n);
fread(x.data(), sizeof(T), big_n, fp);

问题和解决方案

之前的解决方案有效,但构造函数实际上调用了T的默认构造函数big_n次.当big_n真的很大时(这完全不必要,因为我要从磁盘中fread()整个块),这非常慢.FWIW,在我的一个文件的测试用例中,它需要3秒而不是200ms.

所以我尝试使用它:

std::vector<T> x;
x.reserve( big_n );
fread(x.data(), sizeof(T), big_n, fp);

这似乎有效,但后来我遇到了size()返回0而不是big_n的问题.

如何在不损失太多效率的情况下纠正此问题？

附录

我只是注意到std::vector<>可以采用自定义分配器.可以使用那种形式的构造函数来解决我的问题吗？我现在正在研究这种方法.

什么对我有用

除了jrok的简单阵列解决方案之外,我还研究了下面的Ali的自定义分配器解决方案.我决定采用jrock的解决方案,因为它易于理解/降低维护.

我想出的工作代码如下:

#include <vector>
#include <set>
#include <memory>
#include <fstream>
#include <iostream>
#include <cassert>

struct Foo
{
        int m_i;
        Foo() { }
        Foo( int i ) : m_i( i ) { }
        bool operator==( Foo const& rhs ) const { return m_i==rhs.m_i; }
        bool operator!=( Foo const& rhs ) const { return m_i!=rhs.m_i; }
        friend std::ostream& operator<<( std::ostream& os, Foo const& rhs )
        { os << rhs.m_i; }
};


// DESIGN NOTES  /*{{{*/
//
// LIMITATION  T must be a POD so we can fread/fwrite quickly
//
// WHY DO WE NEED THIS CLASS?
//
// We want to write a large number of small PODs to disk and read them back without
//   1. spurious calls to default constructors by std::vector
//   2. writing to disk a gazillion times
//
// SOLUTION
//   A hybrid class containing a std::vector<> for adding new items and a
//   std::unique_ptr<T[]> for fast persistence.  From the user's POV, it looks
//   like a std::vector<>.
//
// Algorithm
//   1. add new items into:
//      std::vector<T>        m_v;
//   2. when writing to disk, write out m_v as a chunk
//   3. when reading from disk, read into m_chunk (m_v will start empty again)
//   4. m_chunk and m_v combined will represent all the data
/*}}}*/

template<typename T>
class vector_chunk
{
// STATE  /*{{{*/
        size_t                m_n_in_chunk;
        std::unique_ptr<T[]>  m_chunk;
        std::vector<T>        m_v;
/*}}}*/

// CONSTRUCTOR, INITIALIZATION  /*{{{*/
public:
        vector_chunk() : m_n_in_chunk( 0 ) { }
/*}}}*/

// EQUALITY /*{{{*/
        public:
                bool operator==( vector_chunk const& rhs ) const
                {
                        if ( rhs.size()!=size() )
                                return false;

                        for( size_t i=0; i<size(); ++i )
                                if ( operator[]( i )!=rhs[i] )
                                        return false;

                        return true;
                }
/*}}}*/

// OSTREAM /*{{{*/
        public:
                friend std::ostream& operator<<( std::ostream& os, vector_chunk const& rhs )
                {
                        for( size_t i=0; i<rhs.m_n_in_chunk; ++i )
                                os << rhs.m_chunk[i] << "\n";
                        for( T const& t : rhs.m_v )
                                os << rhs.t << "\n";
                }
/*}}}*/
// BINARY I/O  /*{{{*/
public:
        void write_as_binary( std::ostream& os ) const
        {
                // write everything out
                size_t const  n_total = size();
                os.write( reinterpret_cast<const char*>( &n_total ), sizeof( n_total ));
                os.write( reinterpret_cast<const char*>( &m_chunk[0] ), m_n_in_chunk * sizeof( T ));
                os.write( reinterpret_cast<const char*>( m_v.data() ), m_v.size() * sizeof( T ));
        }
        void read_as_binary(  std::istream& is )
        {
                // only read into m_chunk, clear m_v
                is.read( reinterpret_cast<char*>( &m_n_in_chunk ), sizeof( m_n_in_chunk ));
                m_chunk.reset( new T[ m_n_in_chunk ] );
                is.read( reinterpret_cast<char*>( &m_chunk[0] ), m_n_in_chunk * sizeof( T ));
                m_v.clear();
        }
/*}}}*/

// DELEGATION to std::vector<T>  /*{{{*/
public:
        size_t size() const                 { return m_n_in_chunk + m_v.size(); }
        void push_back( T const& value )    { m_v.push_back( value ); }
        void push_back( T&&      value )    { m_v.push_back( value ); }
        template< class... Args >
        void emplace_back( Args&&... args ) { m_v.emplace_back( args... ); }
        typename std::vector<T>::const_reference
        operator[]( size_t pos ) const
        { return ((pos < m_n_in_chunk) ? m_chunk[ pos ] : m_v[ pos - m_n_in_chunk]); }

        typename std::vector<T>::reference
        operator[]( size_t pos )
        { return ((pos < m_n_in_chunk) ? m_chunk[ pos ] : m_v[ pos - m_n_in_chunk]); }
/*}}}*/
};

int main()
{
        size_t const n = 10;
        vector_chunk<Foo>  v, w;
        for( int i=0; i<n; ++i )
                v.emplace_back( Foo{ i } );

        std::filebuf                   ofb, ifb;
        std::unique_ptr<std::ostream>  osp;
        std::unique_ptr<std::istream>  isp;

        ofb.open( "/tmp/junk.bin", (std::ios::out | std::ios::binary));
        osp.reset( new std::ostream( &ofb ));
        v.write_as_binary( *osp );
        ofb.close();

        ifb.open( "/tmp/junk.bin", (std::ios::in | std::ios::binary));
        isp.reset( new std::istream( &ifb ));
        w.read_as_binary(  *isp );
        ifb.close();

        assert( v==w );
}

Answer 1

使用vector::reserve()然后写入vector::data()是一个肮脏的黑客和未定义的行为.请不要这样做.

解决此问题的方法是使用自定义分配器,例如本答案中的自定义分配器.我刚测试过它,与clang 3.5 trunk一起工作正常但不能用gcc 4.7.2编译.

虽然,正如其他人已经指出的那样,unique_ptr<T[]>将满足您的需求.