如何使用Express功能为Firebase执行HTTP文件上传(multer,busboy)

Question

我正在尝试将文件上传到云功能,使用Express处理那里的请求,但我没有成功.我创建了一个本地工作的版本:

服务器端js

const express = require('express');
const cors = require('cors');
const fileUpload = require('express-fileupload');

const app = express();
app.use(fileUpload());
app.use(cors());

app.post('/upload', (req, res) => {
    res.send('files: ' + Object.keys(req.files).join(', '));
});

客户端js

const formData = new FormData();
Array.from(this.$refs.fileSelect.files).forEach((file, index) => {
    formData.append('sample' + index, file, 'sample');
});

axios.post(
    url,
    formData, 
    {
        headers: { 'Content-Type': 'multipart/form-data' },
    }
);

当部署到云函数时,这个完全相同的代码似乎会中断,其中req.files未定义.有谁知道这里发生了什么？

编辑 我也有一个使用multer,在本地工作得很好,但一旦上传到云功能,这让我得到一个空数组(相同的客户端代码):

const app = express();
const upload = multer();
app.use(cors());

app.post('/upload', upload.any(), (req, res) => {
    res.send(JSON.stringify(req.files));
});

Answer 1

云功能设置确实发生了重大变化,引发了此问题.它与中间件的工作方式有关,它适用于用于提供HTTPS功能的所有Express应用程序(包括默认应用程序).基本上,Cloud Functions将解析请求的主体并决定如何处理它,将主体的原始内容保留在Buffer中req.rawBody.您可以使用它来直接解析您的多部分内容,但不能使用中间件(如multer).

相反,您可以使用名为busboy的模块直接处理原始主体内容.它可以接受rawBody缓冲区,并使用它找到的文件回拨给你.下面是一些示例代码,它将迭代所有上传的内容,将它们保存为文件,然后删除它们.你显然想要做一些更有用的事情.

const path = require('path');
const os = require('os');
const fs = require('fs');
const Busboy = require('busboy');

exports.upload = functions.https.onRequest((req, res) => {
    if (req.method === 'POST') {
        const busboy = new Busboy({ headers: req.headers });
        // This object will accumulate all the uploaded files, keyed by their name
        const uploads = {}

        // This callback will be invoked for each file uploaded
        busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
            console.log(`File [${fieldname}] filename: ${filename}, encoding: ${encoding}, mimetype: ${mimetype}`);
            // Note that os.tmpdir() is an in-memory file system, so should only 
            // be used for files small enough to fit in memory.
            const filepath = path.join(os.tmpdir(), fieldname);
            uploads[fieldname] = { file: filepath }
            console.log(`Saving '${fieldname}' to ${filepath}`);
            file.pipe(fs.createWriteStream(filepath));
        });

        // This callback will be invoked after all uploaded files are saved.
        busboy.on('finish', () => {
            for (const name in uploads) {
                const upload = uploads[name];
                const file = upload.file;
                res.write(`${file}\n`);
                fs.unlinkSync(file);
            }
            res.end();
        });

        // The raw bytes of the upload will be in req.rawBody.  Send it to busboy, and get
        // a callback when it's finished.
        busboy.end(req.rawBody);
    } else {
        // Client error - only support POST
        res.status(405).end();
    }
})

请记住,保存到临时空间的文件占用内存,因此它们的大小应限制为总共10MB.对于较大的文件,您应该将这些文件上载到云存储并使用存储触发器处理它们.

另请注意,Cloud Functions添加的默认中间件选择目前尚未通过本地模拟器添加firebase serve.因此,在这种情况下,此示例将不起作用(rawBody将不可用).

该团队正在努力更新文档,以便更清楚地了解与标准Express应用程序不同的HTTPS请求中发生的所有事情.

Answer 2

我能够结合Brian和Doug的回应.这是我的中间件,它最终会模仿multer中的req.files,因此不会对其余代码进行重大更改.

module.exports = (path, app) => {
app.use(bodyParser.json())
app.use(bodyParser.urlencoded({ extended: true }))
app.use((req, res, next) => {
    if(req.rawBody === undefined && req.method === 'POST' && req.headers['content-type'].startsWith('multipart/form-data')){
        getRawBody(req, {
            length: req.headers['content-length'],
            limit: '10mb',
            encoding: contentType.parse(req).parameters.charset
        }, function(err, string){
            if (err) return next(err)
            req.rawBody = string
            next()
        })
    } else {
        next()
    }
})

app.use((req, res, next) => {
    if (req.method === 'POST' && req.headers['content-type'].startsWith('multipart/form-data')) {
        const busboy = new Busboy({ headers: req.headers })
        let fileBuffer = new Buffer('')
        req.files = {
            file: []
        }

        busboy.on('field', (fieldname, value) => {
            req.body[fieldname] = value
        })

        busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
            file.on('data', (data) => {
                fileBuffer = Buffer.concat([fileBuffer, data])
            })

            file.on('end', () => {
                const file_object = {
                    fieldname,
                    'originalname': filename,
                    encoding,
                    mimetype,
                    buffer: fileBuffer
                }

                req.files.file.push(file_object)
            })
        })

        busboy.on('finish', () => {
            next()
        })


        busboy.end(req.rawBody)
        req.pipe(busboy)
    } else {
        next()
    }
})}

Answer 3

感谢上面的答案,我为此构建了一个npm模块(github)

它适用于谷歌云功能,只需安装它(npm install --save express-multipart-file-parser)并使用它像这样:

const fileMiddleware = require('express-multipart-file-parser')

...
app.use(fileMiddleware)
...

app.post('/file', (req, res) => {
  const {
    fieldname,
    filename,
    encoding,
    mimetype,
    buffer,
  } = req.files[0]
  ...
})

Answer 4

几天我一直在遭受同样的问题,事实证明firebase团队已经将multipart/form-data的原始数据放入了他们的中间件req.body.如果您尝试使用console.log(req.body.toString())在使用multer处理您的请求之前,您将看到您的数据.当multer创建一个新的req.body对象,它覆盖了生成的req,数据消失了,我们所能得到的只是一个空的req.body.希望firebase团队能够很快纠正这个问题.

Answer 5

要添加到Cloud Function团队的官方答案中，您可以通过执行以下操作在本地模拟此行为（显然，将中间件添加到比他们发布的busboy代码更高的位置）

const getRawBody = require('raw-body');
const contentType = require('content-type');

app.use(function(req, res, next){
    if(req.rawBody === undefined && req.method === 'POST' && req.headers['content-type'] !== undefined && req.headers['content-type'].startsWith('multipart/form-data')){
        getRawBody(req, {
            length: req.headers['content-length'],
            limit: '10mb',
            encoding: contentType.parse(req).parameters.charset
        }, function(err, string){
            if (err) return next(err);
            req.rawBody = string;
            next();
        });
    }
    else{
        next();
    }
});

Answer 6

云函数request在进一步传递对象之前对其进行预处理。因此，原始multer中间件不起作用。此外，使用busboy级别太低，您需要自己处理所有事情，这并不理想。相反，您可以使用分叉版本的 multer中间件来处理multipart/form-data云函数。

这是你可以做的。

1. 安装前叉

npm install --save emadalam/multer#master

2. 使用startProcessing配置自定义处理req.rawBody添加的云功能。

const express = require('express')
const multer = require('multer')

const SIZE_LIMIT = 10 * 1024 * 1024 // 10MB
const app = express()

const multipartFormDataParser = multer({
  storage: multer.memoryStorage(),
  // increase size limit if needed
  limits: {fieldSize: SIZE_LIMIT},
  // support firebase cloud functions
  // the multipart form-data request object is pre-processed by the cloud functions
  // currently the `multer` library doesn't natively support this behaviour
  // as such, a custom fork is maintained to enable this by adding `startProcessing`
  // https://github.com/emadalam/multer
  startProcessing(req, busboy) {
    req.rawBody ? busboy.end(req.rawBody) : req.pipe(busboy)
  },
})

app.post('/some_route', multipartFormDataParser.any(), function (req, res, next) {
  // req.files is array of uploaded files
  // req.body will contain the text fields
})