我如何创建一个巨大的json文件

Question

我想创建一个包含数据库中大量记录的大文件.此文件由另一个进程使用.

使用xml时,我不必将所有内容加载到内存中,只需使用XML :: Writer即可

使用JSON时,我们通常会创建一个perl数据结构,并使用to_json函数来转储结果.

这意味着我必须将所有内容加载到内存中.

有没有办法避免它？

JSON适合大文件吗？

Answer 1

只需使用JSON :: Streaming :: Writer

描述

Most JSON libraries work in terms of in-memory data structures. In Perl, JSON 
serializers often expect to be provided with a HASH or ARRAY ref containing 
all of the data you want to serialize.

This library allows you to generate syntactically-correct JSON without first 
assembling your complete data structure in memory. This allows large structures 
to be returned without requiring those structures to be memory-resident, and 
also allows parts of the output to be made available to a streaming-capable 
JSON parser while the rest of the output is being generated, which may 
improve performance of JSON-based network protocols.

概要

my $jsonw = JSON::Streaming::Writer->for_stream($fh)
$jsonw->start_object();
$jsonw->add_simple_property("someName" => "someValue");
$jsonw->add_simple_property("someNumber" => 5);
$jsonw->start_property("someObject");
$jsonw->start_object();
$jsonw->add_simple_property("someOtherName" => "someOtherValue");
$jsonw->add_simple_property("someOtherNumber" => 6);
$jsonw->end_object();
$jsonw->end_property();
$jsonw->start_property("someArray");
$jsonw->start_array();
$jsonw->add_simple_item("anotherStringValue");
$jsonw->add_simple_item(10);
$jsonw->start_object();
# No items; this object is empty
$jsonw->end_object();
$jsonw->end_array();

还有JSON :: Streaming :: Reader :)