NIFI：尽管增加了堆大小，但出现内存不足错误

Question

我遇到了一个问题，NIFI服务只是拒绝启动。我监视了nifi-app.log，发现它是OutOfMemory错误。我增加了bootstap.conf文件中的内存。但是错误仍然存​​在。

附件为nifi-app.log

    2017-12-19 17:08:37,133 ERROR [ActiveMQ InactivityMonitor ReadCheckTimer] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[ActiveMQ InactivityMonitor ReadCheckTimer,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,134 ERROR [Provenance Maintenance Thread-2] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Provenance Maintenance Thread-2,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,134 ERROR [logback-2] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[logback-2,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,133 ERROR [FileSystemRepository Workers Thread-2] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[FileSystemRepository Workers Thread-2,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,135 ERROR [pool-15-thread-1] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[pool-15-thread-1,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,133 ERROR [Flow Service Tasks Thread-1] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Flow Service Tasks Thread-1,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:39,496 ERROR [Provenance Maintenance Thread-2] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:39,496 ERROR [logback-2] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:39,496 ERROR [FileSystemRepository Workers Thread-2] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:38,482 ERROR [Flow Service Tasks Thread-1] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:38,482 ERROR [Event-Driven Process Thread-1] org.apache.nifi.engine.FlowEngine A flow controller task execution stopped abnormally
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:38,483 ERROR [FileSystemRepository Workers Thread-4] org.apache.nifi.engine.FlowEngine A flow controller task execution stopped abnormally
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:38,482 ERROR [pool-15-thread-1] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:39,497 ERROR [ActiveMQ InactivityMonitor ReadCheckTimer] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:40,509 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5351d0ef checkpointed with 2358 Records and 0 Swap Files in 114969 milliseconds (Stop-the-world time = 2361 milliseconds, Clear Edit Logs time = 936 millis), max Transaction ID 7622
2017-12-19 17:10:41,433 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5351d0ef checkpointed with 2358 Records and 0 Swap Files in 922 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID 7622

我还尝试删除流文件和内容存储库，以免它可能已损坏。但是，仍然没有变化。

我能理解的是，这是一个线程错误，但是我找不到它的解决方法。

Answer 1

尝试关闭，删除存储库，并在nifi.properties中设置nifi.flowcontroller.autoResumeState = false，然后开始备份。

这样可以确保在启动NiFi时，整个流程都将停止，这样您的流程就不会再带来更多数据并导致内存不足错误，那么您可以启动流程的一部分以查看问题出在哪里。

OOM的典型原因是获取一个大流文件并将其拆分为数百万个小流文件，例如，将一千万行的CSV放入SplitText中，行数为1。

另一个原因可能是使用将流文件的所有内容读入内存并让该处理器命中数GB的流文件的处理器，或者是使用ExtractText将大量内容提取到属性中。