Node中并行/异步的多个分页GET API调用

Question

我正在调用bitbucket API来获取repo中的所有文件.我已达到这样的程度,我可以获取repo中所有文件夹的列表,并对repo中的所有根文件夹进行第一次API调用,并获取所有文件夹的前1000个文件列表.

但问题是bitbucket api一次只能给我每个文件夹1000个文件.

我需要附加一个查询参数&start = nextPageStart并再次进行调用,直到它为null并且每个API的isLastPage为true.如何使用以下代码实现这一目标？

我从第一次调用api获得nextPageStart.请参阅下面的API响应.

下面是我到目前为止的代码.

任何帮助或指导表示赞赏.

来自每个文件夹调用的各个API的响应.

{
    "values": [
        "/src/js/abc.js",
        "/src/js/efg.js",
        "/src/js/ffg.js",
        ...
    ],
    "size": 1000,
    "isLastPage": false,
    "start": 0,
    "limit": 1000,
    "nextPageStart": 1000
}

函数,我进行异步调用以获取文件列表

export function getFilesList() {
  const foldersURL: any[] = [];
  getFoldersFromRepo().then((response) => {
    const values = response.values;
    values.forEach((value: any) => {
    //creating API URL for each folder in the repo
      const URL = 'https://bitbucket.abc.com/stash/rest/api/latest/projects/'
                   + value.project.key + '/repos/' + value.slug + '/files?limit=1000';
      foldersURL.push(URL);
        });
    return foldersURL;
      }).then((res) => {
    // console.log('Calling all the URLS in parallel');
    async.map(res, (link, callback) => {
       const options = {
         url: link,
         auth: {
           password: 'password',
           username: 'username',
         },
       };
       request(options, (error, response, body) => {

      // TODO: How do I make the get call again so that i can paginate and append the response to the body till the last page.

         callback(error, body);
       });
     }, (err, results) => {
       console.log('In err, results function');
       if (err) {
         return console.log(err);
       }
       //Consolidated results after all API calls.
       console.log('results', results);
     });
  })
   .catch((error) => error);
}

Answer 1

我能够通过回调创建一个函数.

export function getFilesList() {
  const foldersURL: any[] = [];
  getFoldersFromRepo().then((response) => {
    const values = response.values;
    values.forEach((value: any) => {
    //creating API URL for each folder in the repo
      const URL = 'https://bitbucket.abc.com/stash/rest/api/latest/projects/'
                   + value.project.key + '/repos/' + value.slug + '/files?limit=1000';
      foldersURL.push(URL);
        });
    return foldersURL;
      }).then((res) => {
    // console.log('Calling all the URLS in parallel');
    async.map(res, (link, callback) => {
       const options = {
         url: link,
         auth: {
           password: 'password',
           username: 'username',
         },
       };
      const myarray = [];
// This function will consolidate response till the last Page per API.
      consolidatePaginatedResponse(options, link, myarray, callback);
     }, (err, results) => {
       console.log('In err, results function');
       if (err) {
         return console.log(err);
       }
       //Consolidated results after all API calls.
       console.log('results', results);
     });
  })
   .catch((error) => error);
}

function consolidatePaginatedResponse(options, link, myarray, callback) {
  request(options, (error, response, body) => {
    const content = JSON.parse(body);
    content.link = options.url;
    myarray.push(content);
    if (content.isLastPage === false) {
      options.url = link + '&start=' + content.nextPageStart;
      consolidatePaginatedResponse(options, link, myarray, callback);
    } else {
// Final response after consolidation per API
      callback(error, JSON.stringify(myarray));
    }
  });
}