Java8 Stream batch processing to avoid OutOfMemory

Question

I'm having something like:

    List<Data> dataList = stepts.stream()
        .flatMap(step -> step.getPartialDataList().stream())
        .collect(Collectors.toList());

So I'm combining into dataList multiple lists from every step.

My problem is that dataList might run into OutOfMemoryError. Any suggestions on how I can batch the dataList and save the batches into db?

My primitive idea is to:

    for (Step step : steps) {
        List<Data> partialDataList = step.getPartialDataList();

        if (dataList.size() + partialDataList.size() <= MAXIMUM_SIZE) {
            dataList.addAll(partialDataList);
        } else {
            saveIntoDb(dataList);
            dataList = new ArrayList<>();
        }
    }

PS: I know there is this post, but the difference is that I might not be able to store whole data in memory.

LE: getPartialDataList metod is more like createPartialDataList()

Answer 1

If your concern is OutOfMemoryError you probably shouldn't create additional intermediate data structures like lists or streams before saving to the database.

Since the Step.getPartialDataList() already returns List<Data> the data is already in the memory, unless you have your own List implementation. You just need to use JDBC batch insert:

PreparedStatement ps = c.prepareStatement("INSERT INTO data VALUES (?, ?, ...)");
for (Step step : steps) {
    for (Data data : step.getPartialDataList()) {
        ps.setString(1, ...);
        ps.setString(2, ...);
        ...
        ps.addBatch();
    }   
}
ps.executeBatch();

There is no need to chunk into smaller batches prematurely with dataList. First see what your database and JDBC driver are supporting before doing premature optimizations.

请注意，对于大多数数据库，正确的插入大量数据的方法是外部实用程序，而不是JDBC，例如PostgreSQLCOPY。