Neo4j正在耗尽内存/ GC错误

Question

我的学士论文的一部分是将.nt文件(海龟格式)加载到neo4j数据库中,在阅读它并寻找可能的实现后,我发现OpenRDF的SailRepositoryConnection实现应该像我需要的那样工作,但我遇到了沉重的OOM/GC问题.我使用的库: - 蓝图2.5 - OpenRDF Sesame 2.7.8 - Neo4j 2.0.3

我使用的系统: - Fedora 19 64位 - 4 Gb Ram - Java版本1.7.0_51

我的代码如下:

public void loadFile() throws SailException {
        //load file
        Neo4j2Graph neo4jGraph = new Neo4j2Graph(this.DB_DIRECTORY);
        BatchGraph<TransactionalGraph> neo = new BatchGraph<TransactionalGraph>(neo4jGraph, VertexIDType.NUMBER, 1);

    GraphSail sail = new GraphSail( neo4jGraph );
    sail.initialize();

    SailRepositoryConnection connection;
    try { 
        connection = new SailRepository( sail ).getConnection();

        URL url = f.toURI().toURL(); // getClass().getResource( this.f.getCanonicalPath() );
        // System.out.println( "Loading " + url + ": " );

        connection.add(url, null , RDFFormat.NTRIPLES);

        connection.commit();
        connection.close();
    } catch ( Exception e ) {
        e.printStackTrace(System.out);
    }

    // System.out.print( "Done." );

    sail.shutDown();
    neo4jGraph.shutdown();
    neo.shutdown();
}

使用非常小的.nt文件(如4800三联),它工作得很好.但是尝试加载包含180万个三元组的.nt文件时会发生以下错误:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java:53)
    at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:199)
    at org.apache.lucene.search.Searcher.createNormalizedWeight(Searcher.java:168)
    at org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:664)
    at org.apache.lucene.search.Query.weight(Query.java:103)
    at org.neo4j.index.impl.lucene.Hits.<init>(Hits.java:101)
    at org.neo4j.index.impl.lucene.LuceneIndex.search(LuceneIndex.java:427)
    at org.neo4j.index.impl.lucene.LuceneIndex.query(LuceneIndex.java:314)
    at org.neo4j.index.impl.lucene.LuceneIndex.get(LuceneIndex.java:229)
    at org.neo4j.kernel.impl.coreapi.AbstractAutoIndexerImpl$IndexWrapper.get(AbstractAutoIndexerImpl.java:176)
    at com.tinkerpop.blueprints.impls.neo4j2.Neo4j2Graph.getVertices(Neo4j2Graph.java:369)
    at com.tinkerpop.blueprints.oupls.sail.GraphSail$DataStore.findVertex(GraphSail.java:297)
    at com.tinkerpop.blueprints.oupls.sail.GraphSailConnection.getOrCreateVertex(GraphSailConnection.java:241)
    at com.tinkerpop.blueprints.oupls.sail.GraphSailConnection.addStatementInternal(GraphSailConnection.java:208)
    at com.tinkerpop.blueprints.oupls.sail.GraphSailConnection.addStatementInternal(GraphSailConnection.java:165)
    at org.openrdf.sail.helpers.SailConnectionBase.addStatement(SailConnectionBase.java:471)
    at org.openrdf.repository.sail.SailRepositoryConnection.addWithoutCommit(SailRepositoryConnection.java:281)
    at org.openrdf.repository.base.RepositoryConnectionBase.add(RepositoryConnectionBase.java:469)
    at org.openrdf.repository.util.RDFInserter.handleStatement(RDFInserter.java:207)
    at org.openrdf.rio.ntriples.NTriplesParser.parseTriple(NTriplesParser.java:319)
    at org.openrdf.rio.ntriples.NTriplesParser.parse(NTriplesParser.java:193)
    at org.openrdf.rio.ntriples.NTriplesParser.parse(NTriplesParser.java:132)
    at org.openrdf.repository.util.RDFLoader.loadInputStreamOrReader(RDFLoader.java:325)
[Full GC 963967K->963934K(963968K), 5.8010570 secs]
    at org.openrdf.repository.util.RDFLoader.load(RDFLoader.java:222)
    at org.openrdf.repository.util.RDFLoader.load(RDFLoader.java:180)
    at org.openrdf.repository.base.RepositoryConnectionBase.add(RepositoryConnectionBase.java:253)
    at src4neo2graph.QnL2.loadFile(QnL2.java:181)
    at main.Main.main(Main.java:52)

我也尝试在另一个系统上运行4Gb但发生了同样的错误.所以我尝试使用一些命令进行垃圾收集:(没有特别按顺序排列,但我在某个时候尝试了所有这些命令)

Xms2G -Xmx2G -XX:PermSize = 256M -XX:MaxPermSize = 256M -XX:+ UseConcMarkSweepGC -verbose:gc -XX:+ PrintGCTimeStamps -XX:+ PrintGCDetails -XX:MaxHeapFreeRatio = 95

但这也没有帮助.

我已经搜索了很多,这似乎是每个人用来将.nt文件加载到Neo4j数据库中的实现,就像我写的那样,只需少量三元组即可.所以基本上:我的实现有问题吗？还有更好的吗？我能做些什么(除了购买更多内存(就像我说我在另一个系统上测试更多内容))来完成这项工作？

提前致谢.

Answer 1

你不应该真正将文件分成几块; 只计算添加的语句和每个第n个语句后的commit().有一个便利类,SailLoader(源码),在blueprints-graph-sail中为你做这个,例如

public void loadFile() throws Exception {
    File f = new File("/tmp/sp2bench-1000000.nt.gz");

    Neo4j2Graph neo4jGraph = new Neo4j2Graph(this.DB_DIRECTORY);
    try {
        GraphSail sail = new GraphSail(neo4jGraph);
        sail.initialize();

        try {
            SailLoader loader = new SailLoader(sail);
            loader.setBufferSize(1000);  // this is the default
            loader.setVerbose(true);
            loader.load(f);
        } finally {
            sail.shutDown();
        }
    } finally {
        neo4jGraph.shutdown();
    }
}