如何修复java OutOfMemoryError:DataImportHandler的Java堆空间？

Question

我正在尝试将大型数据集(4100万条记录)导入新的Solr索引.我已经设置了核心,它工作,我插入了一些测试文档,他们工作.我已经设置了如下的data-config.xml,然后我开始全导入.大约12个小时后!导入失败.

文档大小可能会变得非常大,错误可能是因为大文档(或字段)还是因为数据量大量进入DataImportHandler？

我怎样才能让这个令人沮丧的导入任务正常工作!？!

我已经在下面添加了tomcat错误日志.

如果有任何我错过的信息,请告诉我!

日志:

Jun 1, 2011 5:47:55 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO: Creating a connection for entity results with URL: jdbc:sqlserver://myserver;databaseName=mydb;responseBuffering=adaptive;selectMethod=cursor
Jun 1, 2011 5:47:56 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO: Time taken for getConnection(): 1185
Jun 1, 2011 5:48:02 PM org.apache.solr.core.SolrCore execute
INFO: [results] webapp=/solr path=/dataimport params={command=full-import} status=0 QTime=0
...
Jun 2, 2011 5:16:32 AM org.apache.solr.common.SolrException log
SEVERE: Full Import failed:org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space
    at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:664)
    at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:267)
    at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:186)
    at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:353)
    at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:411)
    at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:392)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.lang.StringCoding$StringDecoder.decode(Unknown Source)
    at java.lang.StringCoding.decode(Unknown Source)
    at java.lang.String.<init>(Unknown Source)
    at java.lang.String.<init>(Unknown Source)
    at com.microsoft.sqlserver.jdbc.DDC.convertStreamToObject(DDC.java:419)
    at com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValue(dtv.java:1974)
    at com.microsoft.sqlserver.jdbc.DTV.getValue(dtv.java:175)
    at com.microsoft.sqlserver.jdbc.Column.getValue(Column.java:113)
    at com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:1982)
    at com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:1967)
    at com.microsoft.sqlserver.jdbc.SQLServerResultSet.getObject(SQLServerResultSet.java:2256)
    at com.microsoft.sqlserver.jdbc.SQLServerResultSet.getObject(SQLServerResultSet.java:2265)
    at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:286)
    at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$700(JdbcDataSource.java:228)
    at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:266)
    at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:260)
    at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:78)
    at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
    at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:238)
    at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:591)
    ... 5 more

Jun 2, 2011 5:16:32 AM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: start rollback
Jun 2, 2011 5:16:44 AM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: end_rollback

数据-config.xml文件:

<dataConfig> 
  <dataSource type="JdbcDataSource" 
        driver="com.microsoft.sqlserver.jdbc.SQLServerDriver" 
        url="jdbc:sqlserver://myserver;databaseName=mydb;responseBuffering=adaptive;selectMethod=cursor"   
        user="sa" 
        password="password"/> 
  <document> 
    <entity name="results" query="SELECT fielda, fieldb, fieldc FROM mydb.[dbo].mytable WITH (NOLOCK)"> 
      <field column="fielda" name="fielda"/><field column="fieldb" name="fieldb"/><field column="fieldc" name="fieldc"/> 
    </entity> 
  </document> 
</dataConfig> 

solrconfig.xml片段:

<indexDefaults>
    <useCompoundFile>false</useCompoundFile>
    <mergeFactor>25</mergeFactor>
    <ramBufferSizeMB>128</ramBufferSizeMB>
    <maxFieldLength>100000</maxFieldLength>
    <writeLockTimeout>10000</writeLockTimeout>
    <commitLockTimeout>10000</commitLockTimeout>
  </indexDefaults>
  <mainIndex>
    <useCompoundFile>false</useCompoundFile>
    <ramBufferSizeMB>128</ramBufferSizeMB>
    <mergeFactor>25</mergeFactor>
     <infoStream file="INFOSTREAM.txt">true</infoStream>
  </mainIndex>

Java配置设置:init mem 128mb,最大512mb

环境:solr 3.1 tomcat 7.0.12 windows server 2008 java:v6 update 25(build 1.6.0_25-b06)(数据来自:sql 2008 r2)

/admin/stats.jsp - DataImportHandler
    Status : IDLE
    Documents Processed : 2503083
    Requests made to DataSource : 1
    Rows Fetched : 2503083
    Documents Deleted : 0
    Documents Skipped : 0
    Total Documents Processed : 0
    Total Requests made to DataSource : 0
    Total Rows Fetched : 0
    Total Documents Deleted : 0
    Total Documents Skipped : 0
    handlerStart : 1306759913518
    requests : 9
    errors : 0 

编辑:我目前正在运行一个SQL查询来找出最大的单个记录的字段长度,因为我认为这可能是异常的原因.此外,使用jconsole再次运行导入以监视堆使用情况.

编辑:读取solr性能因子页面.将maxFieldLength更改为1000000并更改ramBufferSizeMB = 256.现在进行另一次导入运行(yay ...)

Answer 1

Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.lang.StringCoding$StringDecoder.decode(Unknown Source)
    at java.lang.StringCoding.decode(Unknown Source)
    at java.lang.String.<init>(Unknown Source)
    at java.lang.String.<init>(Unknown Source)
    at com.microsoft.sqlserver.jdbc.DDC.convertStreamToObject(DDC.java:419)

很明显,MS JDBC驱动程序已经用完ram了.许多JDBC驱动程序可以默认在内存中一次获取所有结果.因此,看看是否可以调整或考虑使用开源JTDS驱动程序,无论如何通常表现更好

我不相信maxfieldlength会帮助你 - 这将影响Lucene截断多少,但不会影响最初转移多少.另一个选择是一次只传输一个选择,比如一百万,使用TOP和ROWNUMBER等进行传呼.