我试图通过doctrine2在mysql中插入大约20000条记录
foreach ($getIds as $id){// 20000 Ids
$twNode = new TwNode();
$twNode->setTwId($id);
$twNode->addAcFriend($this->twAccount);
$this->twAccount->addFollower($twNode);
$this->em->persist($twNode);
$this->em->persist($this->twAccount);
$this->em->flush();
$this->output->write('W');
$followersWrite++;
}
}
但是这个错误很愉快,我怎样才能避免这个允许的内存错误？
PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 72 bytes) in /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php on line 62
PHP Stack trace:
PHP 1. {main}() /Users/whitebear/httproot/myapp/app/console:0
PHP 2. Symfony\Component\Console\Application->run() /Users/whitebear/httproot/myapp/app/console:22
PHP 3. Symfony\Bundle\FrameworkBundle\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:106
PHP 4. Symfony\Component\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:83
PHP 5. Symfony\Component\Console\Command\Command->run() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:200
PHP 6. Acme\TopBundle\Command\twCrawlerCommand->execute() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php:242
PHP 7. Acme\TopBundle\Command\twCrawlerCommand->initialRun() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:430
PHP 8. EntityManager52ee6dcbdb2bb_546a8d27f194334ee012bfe64f629947b07e4919\__CG__\Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:284
PHP 9. Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/app/cache/dev/jms_diextra/doctrine/EntityManager_52ee6dcbdb2bb.php:402
PHP 10. Doctrine\ORM\UnitOfWork->commit() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php:389
PHP 11. Doctrine\ORM\UnitOfWork->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:335
PHP 12. Doctrine\ORM\Persisters\BasicEntityPersister->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:952
PHP 13. Doctrine\DBAL\Statement->execute() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/BasicEntityPersister.php:279
PHP 14. Doctrine\DBAL\Logging\LoggerChain->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php:161
PHP 15. Symfony\Bridge\Doctrine\Logger\DbalLogger->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Logging/LoggerChain.php:54
PHP 16. Symfony\Bridge\Doctrine\Logger\DbalLogger->log() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:80
PHP 17. Monolog\Logger->debug() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:102
PHP 18. Monolog\Logger->addRecord() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:423
PHP 19. Monolog\Handler\AbstractProcessingHandler->handle() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:244
PHP 20. Monolog\Formatter\LineFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php:35
PHP 21. Monolog\Formatter\NormalizerFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:43
PHP 22. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:40
PHP 23. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88
PHP 24. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:70
PHP 25. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88
Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 72 bytes) in /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php on line 62
Call Stack:
0.0002 229968 1. {main}() /Users/whitebear/httproot/myapp/app/console:0
0.0126 2063376 2. Symfony\Component\Console\Application->run() /Users/whitebear/httproot/myapp/app/console:22
0.0144 2256968 3. Symfony\Bundle\FrameworkBundle\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:106
0.1503 11751600 4. Symfony\Component\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:83
0.1518 11753384 5. Symfony\Component\Console\Command\Command->run() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:200
0.1521 11758040 6. Acme\TopBundle\Command\twCrawlerCommand->execute() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php:242
0.2409 20673112 7. Acme\TopBundle\Command\twCrawlerCommand->initialRun() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:430
101.5883 235069104 8. EntityManager52ee6dcbdb2bb_546a8d27f194334ee012bfe64f629947b07e4919\__CG__\Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:284
101.5883 235069200 9. Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/app/cache/dev/jms_diextra/doctrine/EntityManager_52ee6dcbdb2bb.php:402
101.5883 235069280 10. Doctrine\ORM\UnitOfWork->commit() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php:389
107.0744 259211408 11. Doctrine\ORM\UnitOfWork->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:335
107.1128 259277872 12. Doctrine\ORM\Persisters\BasicEntityPersister->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:952
109.1604 267877360 13. Doctrine\DBAL\Statement->execute() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/BasicEntityPersister.php:279
109.1604 267877408 14. Doctrine\DBAL\Logging\LoggerChain->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php:161
109.1604 267877408 15. Symfony\Bridge\Doctrine\Logger\DbalLogger->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Logging/LoggerChain.php:54
109.1604 267877592 16. Symfony\Bridge\Doctrine\Logger\DbalLogger->log() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:80
109.1604 267877592 17. Monolog\Logger->debug() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:102
109.1604 267877640 18. Monolog\Logger->addRecord() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:423
109.1604 267878968 19. Monolog\Handler\AbstractProcessingHandler->handle() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:244
109.1604 267878968 20. Monolog\Formatter\LineFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php:35
109.1604 267878968 21. Monolog\Formatter\NormalizerFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:43
109.1604 267878968 22. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:40
109.1604 267878968 23. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88
109.1605 267881872 24. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:70
109.1605 267881872 25. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88
感谢你的评论,我明白它与$ this-> em的内存相关
所以,我把代码设置为每隔5000次取消$ this-> em;
if ($k > 1000){
$this->em->flush();
unset($this->em);
$this->em = $this->getContainer()->get('doctrine')->getManager();
}
但仍然徒劳,我想我应该取消$ this-> getContainer() - > get('doctrine') - > getManager();
可能吗？或者我是对的？
您可能想要阅读Doctrine 2 Batch Processing.他们正在使用这种模式:
// $em instanceof EntityManager
$batchSize = 20;
for ($i=1; $i<=10000; ++$i) {
$obj = new MyEntity;
$obj->setFoo('...');
// ... set more data
$em->persist($obj);
if (($i % $batchSize) == 0) {
$em->flush();
$em->clear();
}
}
$em->flush();
据我所知 - 在持久化之后,doctrine将所有映射对象保存在身份映射的内存中.因此,每个新对象的内存需求都在不断增长.如果你执行clear(),它将分离所有托管实体,如果你没有引用它们,它们将被垃圾收集.
注意:它将分离您正在使用的所有其他实体.或者您只能分离那些插入的对象$em->detach($twNode); // detach from Doctrine, so that it can be GC'd immediately
编辑:在for循环后添加flush()以确保保存所有实体,即使实体计数不能被批量大小整除,如评论中所建议的那样