我试图爬行几个网页来检查损坏的链接并将链接的结果写入 json 文件,但是,在第一个文件完成后,应用程序崩溃,没有弹出错误...
我使用 Puppeteer 进行爬行,使用 Bluebird 来同时运行每个链接,使用 fs 来写入文件。
我尝试过什么:
const express = require('express');
const router = express.Router();
const puppeteer = require('puppeteer');
const bluebird = require("bluebird");
const fs = require('fs');
router.get('/', function(req, res, next) {
(async () => {
// Our (multiple) URLs.
const urls = ['https://www.testing.com/allergy-test/', 'https://www.testing.com/genetic-testing/'];
const withBrowser = async (fn) => {
const browser = await puppeteer.launch();
try {
return await fn(browser);
} finally {
await browser.close();
} …Run Code Online (Sandbox Code Playgroud)