Foreach()和每个()内存耗尽,分块无法正常工作

Question

我正在编写一个工匠控制台命令,它循环遍历表中的所有记录并重新生成该表上的字段.

该字段是a hash并且生成为md5()特定字符串的一个.

最初我的代码看起来像这样:

// Get all recipes
$recipes = Recipe::all();

$hashProgress = $this->output->createProgressBar(count($recipes));

// Loop over each recipe and generate a new hash for it
foreach ($recipes as $recipe)
{
    $hashString = '';

    $hashString .= $recipe->field1;
    $hashString .= $recipe->field2;
    $hashString .= $recipe->field3;
    $hashString .= $recipe->field4;
    $hashString .= $recipe->field5;
    $hashString .= $recipe->field6;
    $hashString .= $recipe->field7;

    $extras1Total = $recipe->extras1->sum('amount');
    $hashString .= $recipe->extras1->reduce(function ($str, $item) use ($extras1Total) {
        return $str . $item->name . ($extras1Total == 0 ? $item->amount : ($item->amount / $extras1Total * 100));
    }, '');

    $extras2Total = $recipe->extras2->sum('amount');
    $hashString .= $recipe->extras2->reduce(function ($str, $item) use ($extras2Total) {
        return $str . $item->name . ($extras2Total == 0 ? $item->amount : ($item->amount / $extras2Total * 100));
    }, '');

    $extras3Total = $recipe->extras3->sum('amount');
    $hashString .= $recipe->extras3->reduce(function ($str, $item) use ($extras3Total) {
        return $str . $item->name . ($extras3Total == 0 ? $item->amount : ($item->amount / $extras3Total * 100));
    }, '');

    $extras4Total = $recipe->extras4->sum('amount');
    $hashString .= $recipe->extras4->reduce(function ($str, $item) use ($extras4Total) {
        return $str . $item->name . ($extras4Total == 0 ? $item->amount : ($item->amount / $extras4Total * 100));
    }, '');

    $recipe->update([
        'hash' => md5($hashString),
    ]);

    $hashProgress->advance();
}

$hashProgress->finish();
$this->info(' Recipe hashes regenerated.');

在获得大约10,000个28,000条记录之后,它将因内存耗尽错误而死亡:

PHP致命错误:允许的内存大小为268435456字节耗尽(尝试分配4096字节)

我认为chunk这可能会有所帮助:

// Get all recipes
$recipes = Recipe::all();

$hashProgress = $this->output->createProgressBar(count($recipes));

// Loop over each recipe and generate a new hash for it
foreach ($recipes->chunk(1000) as $chunk)
{
    foreach ($chunk as $recipe)
    {
        $hashString = '';

        $hashString .= $recipe->field1;
        $hashString .= $recipe->field2;
        $hashString .= $recipe->field3;
        $hashString .= $recipe->field4;
        $hashString .= $recipe->field5;
        $hashString .= $recipe->field6;
        $hashString .= $recipe->field7;

        $extras1Total = $recipe->extras1->sum('amount');
        $hashString .= $recipe->extras1->reduce(function ($str, $item) use ($extras1Total) {
            return $str . $item->name . ($extras1Total == 0 ? $item->amount : ($item->amount / $extras1Total * 100));
        }, '');

        $extras2Total = $recipe->extras2->sum('amount');
        $hashString .= $recipe->extras2->reduce(function ($str, $item) use ($extras2Total) {
            return $str . $item->name . ($extras2Total == 0 ? $item->amount : ($item->amount / $extras2Total * 100));
        }, '');

        $extras3Total = $recipe->extras3->sum('amount');
        $hashString .= $recipe->extras3->reduce(function ($str, $item) use ($extras3Total) {
            return $str . $item->name . ($extras3Total == 0 ? $item->amount : ($item->amount / $extras3Total * 100));
        }, '');

        $extras4Total = $recipe->extras4->sum('amount');
        $hashString .= $recipe->extras4->reduce(function ($str, $item) use ($extras4Total) {
            return $str . $item->name . ($extras4Total == 0 ? $item->amount : ($item->amount / $extras4Total * 100));
        }, '');

        $recipe->update([
            'hash' => md5($hashString),
        ]);

        $hashProgress->advance();
    }
}

$hashProgress->finish();
$this->info(' Recipe hashes regenerated.');

但我仍然得到内存耗尽错误.

如何在不增加内存限制的情况下遍历所有这些记录并实现我的目标？

Answer 1

您"分块"的方式实际上比初始代码消耗更多内存.

你在做什么越来越所有记录一次,将它们存储在$recipes,然后通过调用分块的结果chunk()的结果集.

相反,您需要chunk()在基础Recipe模型的查询构建器上调用具有相同名称的方法,并按块生成哈希块:

Recipe::chunk(1000, function ($recipies) {
    // Hash generation logic here
});

这样,你就消除了一个巨大的$recipes变量,我确信这是一个瓶颈.根据可用内存,您可能需要稍微调整块大小以避免内存耗尽.

另外,我想尝试生成散列的时候,而不是留下的痕迹,以使用更少的变量$extras1Total,extras2Total,...变量.所有这些都可以替换$total为一遍又一遍地重写.这是微优化.

PS如果数据库写入压力很大(总共28k很少见),您可能需要考虑在一个(或几个)go(s)中进行最终更新,而不是每个记录执行一次.