发送多个 HTTP 请求

Question

我需要编写使用项目 ID 查找有关项目的信息的程序。

\n

该 API 一次仅获取一项，因此我只能对每项执行一次查询。API 仅限于五个同时请求。任何额外的结果都会给出 HTTP 429 错误。

\n

如果有一个 JavaScript 对象，其中包含所有带有 ID\xe2\x80\x99s 的项目

\n

如何检索所有给定 ID 的信息而不触发同时请求限制，并且不对已看到的项目 ID 执行不必要的查询。

\n

import fetch from "node-fetch";\n\nlet itemObject = [\n  { itemName: "", itemID: "" },\n  { itemName: "", itemID: "" },\n  { itemName: "", itemID: "" },\n  { itemName: "", itemID: "" },\n];\n\nasync function sendIDRequests() {\n  try {\n    const response = await fetch("https://url/items/:ID", {\n      headers: {\n        Authorization: "",\n      },\n    });\n    if (!response.ok) {\n      throw new Error(`${response.status} ${response.statusText}`);\n    }\n    response\n      .text()\n      .then((res) => console.log(res))\n      .catch((err) => {\n        throw new Error(err);\n      });\n  } catch (error) {\n    console.error(error);\n  }\n}\n\nsendRequests()\n

\n

Answer 1

为此，我想到了两种方法。批处理和滑动窗口方法。批处理可能更容易，但使用滑动窗口将是更有效的实现。

使用 Promise.all() 进行批处理

此方法会创建指定数量的批量请求batchSize，并且只有在一批中的所有请求完成后，才会发出下一批请求。

您需要在此处添加一些错误处理，以防请求失败。

import fetch from "node-fetch";

// list of items that you might want to use to compose your URL (not actually used here)
let itemObject = [
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
    { itemName: "", itemID: "" },
];


(async () => {
    // number of concurrent requests in one batch
    const batchSize = 4;
    // request counter
    let curReq = 0;
    // as long as there are items in the list continue to form batches
    while (curReq < itemObject.length) {
        // a batch is either limited by the batch size or it is smaller than the batch size when there are less items required
        const end = itemObject.length < curReq + batchSize ? itemObject.length: curReq + batchSize;
        // we know the number of concurrent request so reserve memory for this
        const concurrentReq = new Array(batchSize);
        // issue one request for each item in the batch
        for (let index = curReq; index < end; index++) {
            concurrentReq.push(fetch("https://postman-echo.com/get"))
            console.log(`sending request ${curReq}...`)
            curReq++;
        }
        // wait until all promises are done or one promise is rejected
        await Promise.all(concurrentReq);
        console.log(`requests ${curReq - batchSize}-${curReq} done.`)
    }
})();

预期结果：

sending request 0...
sending request 1...
sending request 2...
sending request 3...
requests 0-4 done.
sending request 4...
sending request 5...
sending request 6...
sending request 7...
requests 4-8 done.
sending request 8...
sending request 9...
sending request 10...
sending request 11...
requests 8-12 done.

带信号量的滑动窗口

此方法使用滑动窗口，并在完成另一个请求后立即调度新请求，同时始终保持请求计数低于或等于n任何一次的最大并发请求数。你需要实现这个的是一个Semaphore。

JavaScript 中有一个名为async-mutex的库。

下面是使用该库向 Postman Echo API 同时发送 2 个请求的示例程序。在信号量允许的情况下，永远不会有更多的请求同时运行（在您的情况下，限制为 5，这里是 2）。

sending request 0...
sending request 1...
sending request 2...
sending request 3...
requests 0-4 done.
sending request 4...
sending request 5...
sending request 6...
sending request 7...
requests 4-8 done.
sending request 8...
sending request 9...
sending request 10...
sending request 11...
requests 8-12 done.

预期输出（顺序可能有所不同）：

sending request 0...
sending request 1...
request 1 done...
sending request 2...
request 2 done...
sending request 3...
request 3 done...
sending request 4...
request 0 done...
sending request 5...
request 4 done...
sending request 6...
request 5 done...
sending request 7...
request 6 done...
sending request 8...
request 7 done...
sending request 9...
request 8 done...
sending request 10...
request 9 done...
sending request 11...
request 10 done...
request 11 done...