从文件加载大型2D int数组的最快方法是什么？

Question

我正在从文件中加载一个2D数组,它是15,000,000*3整数(最终将是40,000,000*3).现在,我dataInputStream.readInt()用来顺序读取整数.大约需要15秒.我可以使它显着(至少3倍)更快或者这个速度和我一样快吗？

Answer 1

将文件映射到内存中!

Java 7代码:

FileChannel channel = FileChannel.open(Paths.get("/path/to/file"), 
    StandardOpenOption.READ);
ByteBuffer buf = channel.map(0, channel.size(),
    FileChannel.MapMode.READ_ONLY);

// use buf

有关详细信息,请参见此处

如果您使用Java 6,则必须:

RandomAccessFile file = new RandomAccessFile("/path/to/file", "r");
FileChannel channel = file.getChannel();
// same thing to obtain buf

.asIntBuffer()如果需要,您甚至可以在缓冲区上使用.当您需要阅读时,您只能阅读实际需要阅读的内容.而且它不会影响您的堆.

Answer 2

Yes, you can. From benchmark of 13 different ways of reading files:

If you have to pick the fastest approach, it would be one of these:

• FileChannel with a MappedByteBuffer and array reads.
• FileChannel with a direct ByteBuffer and array reads.
• FileChannel with a wrapped array ByteBuffer and direct array access.

For the best Java read performance, there are 4 things to remember:

• Minimize I/O operations by reading an array at a time, not a byte at a time. An 8 KB array is a good size (that's why it's a default value for BufferedInputStream).
• 通过一次获取数据数组来最小化方法调用,而不是一次获取一个字节.使用数组索引来获取数组中的字节数.
• 如果不需要线程安全,请最小化线程同步锁.对线程安全类进行较少的方法调用,或者使用类似FileChannel和的非线程安全类MappedByteBuffer.
• 最大限度地减少JVM/OS,内部缓冲区和应用程序阵列之间的数据复制.用于FileChannel内存映射,或直接或包装数组ByteBuffer.