如何使用laravel将Mysql（大）表导出为CSV？

Question

您好，我正在尝试使用 laravel 5 框架导出大表（1Millions 行到 CSV 文件）。

粘贴在下面的代码无法正常工作，因为它分配了太多内存。所有结果都存储在内存中，直到我完成写入文件（php exec）

起初我尝试过（非常低效）

1. 从数据库中获取结果

$mytable = 'table';
                $filePath = storage_path().'/csv/test.csv';

                $mins =  Carbon::now()->subMinutes(10000);

                // set the fetch Assoc mode temoporarly
                DB::setFetchMode(PDO::FETCH_ASSOC);

                $results = DB::table("$mytable")->where('ts_activity', '>=', $mins)->get();

                // revert the fetch Class mode 
                DB::setFetchMode(PDO::FETCH_CLASS);

2. 将结果写入文件

if(count($results)>0){
                   $headerLine = implode(',',array_keys($results[0]));

                    dump($headerLine);

                    if(!file_put_contents($filePath, $headerLine.PHP_EOL)){
                        throw new Exception("Cannot write FILE: {$filePath}");                    
                    }

                    foreach($results as $row){   
                        foreach($row as $k=>$v){
                            if($row[$k] == '0000-00-00 00:00:00'){$row[$k] = '';}
                            $row[$k] = trim(str_replace(array(',', "\n", "\r"), ' ', $row[$k]));
                        }

                        //ADDs content to file
                        if(!file_put_contents($filePath, implode(',', $row).PHP_EOL, FILE_APPEND)){
                            throw new Exception("Cannot write FILE: {$filePath}");                    
                        }
                    }

                    //check file size created                
                    $fileSize = filesize($filePath);
                    $reportString   = "Created file at: $filePath of ($fileSize)";
                    //report
                    print($reportString);

                    RETURN TRUE;                            
                }

然后我尝试直接使用 mysqli，一切都很好，因为每一行都被提取到内存中，然后在每个循环中写入文件。

 //conection: 
                $link = mysqli_connect(env('DB_HOST'),env('DB_USERNAME'),env('DB_PASSWORD'),env('DB_DATABASE')) or die("Error " . mysqli_error($link)); 

                //consultation: 

                $query = "SELECT * FROM $mytable WHERE ts_activity>='$mins';" or die("Error in the consult.." . mysqli_error($link)); 

                //execute the query. 

                $result = mysqli_query($link, $query); 
                $count = 0;
                while($row = $result->fetch_assoc()) { 
                    if($count++==0){
                        $headerLine = implode(',', array_keys($row));                        
                        dump($headerLine);
                        if(!file_put_contents($filePath, $headerLine.PHP_EOL)){
                            throw new Exception("Cannot write FILE: {$filePath}");                    
                        }
                        continue;
                    }

                    foreach($row as $k=>$v){
                        if($row[$k] == '0000-00-00 00:00:00'){$row[$k] = '';}
                        $row[$k] = trim(str_replace(array(',', "\n", "\r"), ' ', $row[$k]));
                    }

                    //ADDs content to file
                    if(!file_put_contents($filePath, implode(',', $row).PHP_EOL, FILE_APPEND)){
                        throw new Exception("Cannot write FILE: {$filePath}");                    
                    }
                }

如何使用 Laravel 5 类 (DB) 获得相同的效率？ 谢谢

Answer 1

你可能想试试这个。这将很明显地将您的输出流式传输到文件中。如果这不是您想要的，我认为下一个最好的方法是存储作业并稍后处理，然后在完成后通知/发送给用户。希望这可以帮助。

public function export($ids = array())
{
    $headers = array(
        'Content-Type'        => 'text/csv',
        'Cache-Control'       => 'must-revalidate, post-check=0, pre-check=0',
        'Content-Disposition' => 'attachment; filename=test.csv',
        'Expires'             => '0',
        'Pragma'              => 'public',
    );

    $response = new StreamedResponse(function() use($ids){
        // Open output stream
        $handle = fopen('php://output', 'w');

        // Add CSV headers
        fputcsv($handle, [
            "Id",
            "Name",
        ]);

        User::whereIn('id', $ids)           
            ->chunk(500, function($users) use($handle) {
            foreach ($users as $user) {
                // Add a new row with data
                fputcsv($handle, [
                    $user->id,
                    $user->name,
                ]);
            }
        });

        // Close the output stream
        fclose($handle);
    }, 200, $headers);

    return $response->send();
}