Node.js - 使用 createReadStream 和 createWriteStream 时内存使用率较高

Question

我正在使用节点测试流，并设置一个程序来读取大文件并使用流再次写入它。问题是运行程序时，节点的内存使用量高达 1.3 GB，这正是正在读取的文件的大小。就像它不会流式传输它，它会缓冲它并一次性写入它，或者垃圾收集器不会破坏chunk内存中的变量。这是程序：

const {  createReadStream, createWriteStream } = require('fs');

const readStream = createReadStream('../movie.mp4', {
    highWaterMark: 10000
});
const writeStream = createWriteStream('./copy.mp4', {
    highWaterMark: 10000
});

readStream.on('data', function (chunk) {
    writeStream.write(chunk);
})

readStream.on('end', function () {
    console.log("reading done");
    writeStream.end();
});

writeStream.on('close', function () {
    console.log("Writing done.");
})

奇怪的是，如果我通过管道传输这些流，它会按预期工作，并且内存使用量不会超过 20 MB。像这样：

const {  createReadStream, createWriteStream } = require('fs');

const readStream = createReadStream('../movie.mp4', {
    highWaterMark: 10000
});
const writeStream = createWriteStream('./copy.mp4', {
    highWaterMark: 10000
});

readStream.pipe(writeStream);

什么会导致这种行为？

节点版本：v14.15.4

Answer 1

嗯，我发现了问题。有一种情况称为背压。就我而言，发生这种情况是因为读取流比写入流快得多。发生这种情况是因为将readStream读取的数据缓冲在内存中，直到将writeMemory其写入。因此，解决方案是我们readStream暂时暂停，直到writeStream完成写入，然后我们向其提供更多数据块。这是正确的程序：

const {  createReadStream, createWriteStream } = require('fs');

const readStream = createReadStream('../movie.mp4', {
    highWaterMark: 10000
});

const writeStream = createWriteStream('./copy.mp4', {
    highWaterMark: 10000
});


readStream.on('data', function (chunk) {
    // according to docs the value of result variable is: 
    // Returns: <boolean> false if the stream wishes for the calling code to wait for the 'drain' event to be emitted before continuing to write additional data; otherwise true.
    const result = writeStream.write(chunk);

    if(!result) {
        console.log("BACKPRESSURE");
        readStream.pause();
    }
});

writeStream.on('drain', () => {
    console.log("DREAINED");
    readStream.resume();
});

readStream.on('end', function () {
    console.log("reading done");
    writeStream.end();
});

writeStream.on('close', function () {
    console.log("Writing done.");
})

drain有关事件的文档在这里。