Yield Request调用在scrapy的递归方法中产生奇怪的结果

Question

我试图在一天之内使用Python和Scrapy从所有国家的所有机场取消所有出发和到达.

这个着名站点(飞行雷达)使用的JSON数据库需要在一个机场的出发或到达> 100时逐页查询.我还根据查询的实际UTC计算时间戳.

我尝试使用此层次结构创建数据库:

country 1
 - airport 1
    - departures
      - page 1
      - page ...
    - arrivals
      - page 1
      - page ...
- airport 2
    - departures
      - page 1
      - page ...
    - arrivals
      - page 
      - page ...
...

我使用两种方法按页面计算时间戳和网址查询:

def compute_timestamp(self):
    from datetime import datetime, date
    import calendar
    # +/- 24 heures
    d = date(2017, 4, 27)
    timestamp = calendar.timegm(d.timetuple())
    return timestamp

def build_api_call(self,code,page,timestamp):
    return 'https://api.flightradar24.com/common/v1/airport.json?code={code}&plugin\[\]=&plugin-setting\[schedule\]\[mode\]=&plugin-setting\[schedule\]\[timestamp\]={timestamp}&page={page}&limit=100&token='.format(
        code=code, page=page, timestamp=timestamp)

我将结果存储到CountryItem包含许多AirportItem机场的结果中.我item.py是:

class CountryItem(scrapy.Item):
    name = scrapy.Field()
    link = scrapy.Field()
    num_airports = scrapy.Field()
    airports = scrapy.Field()
    other_url= scrapy.Field()
    last_updated = scrapy.Field(serializer=str)

class AirportItem(scrapy.Item):
    name = scrapy.Field()
    code_little = scrapy.Field()
    code_total = scrapy.Field()
    lat = scrapy.Field()
    lon = scrapy.Field()
    link = scrapy.Field()
    departures = scrapy.Field()
    arrivals = scrapy.Field()

我的主要解析为所有国家建立了一个国家项目(例如我在这里限制以色列).接下来,我为每个国家收益a scrapy.Request刮去机场.

###################################
# MAIN PARSE
####################################
def parse(self, response):
    count_country = 0
    countries = []
    for country in response.xpath('//a[@data-country]'):
        item = CountryItem()
        url =  country.xpath('./@href').extract()
        name = country.xpath('./@title').extract()
        item['link'] = url[0]
        item['name'] = name[0]
        item['airports'] = []
        count_country += 1
        if name[0] == "Israel":
            countries.append(item)
            self.logger.info("Country name : %s with link %s" , item['name'] , item['link'])
            yield scrapy.Request(url[0],meta={'my_country_item':item}, callback=self.parse_airports)

这种方法为每个机场提取信息,并且还要求每个机场scrapy.request使用机场网址来刮取出发和到达:

  ###################################
# PARSE EACH AIRPORT
####################################
def parse_airports(self, response):
    item = response.meta['my_country_item']
    item['airports'] = []

    for airport in response.xpath('//a[@data-iata]'):
        url = airport.xpath('./@href').extract()
        iata = airport.xpath('./@data-iata').extract()
        iatabis = airport.xpath('./small/text()').extract()
        name = ''.join(airport.xpath('./text()').extract()).strip()
        lat = airport.xpath("./@data-lat").extract()
        lon = airport.xpath("./@data-lon").extract()
        iAirport = AirportItem()
        iAirport['name'] = self.clean_html(name)
        iAirport['link'] = url[0]
        iAirport['lat'] = lat[0]
        iAirport['lon'] = lon[0]
        iAirport['code_little'] = iata[0]
        iAirport['code_total'] = iatabis[0]

        item['airports'].append(iAirport)

    urls = []
    for airport in item['airports']:
        json_url = self.build_api_call(airport['code_little'], 1, self.compute_timestamp())
        urls.append(json_url)
    if not urls:
        return item

    # start with first url
    next_url = urls.pop()
    return scrapy.Request(next_url, self.parse_schedule, meta={'airport_item': item, 'airport_urls': urls, 'i': 0})

使用递归方法,parse_schedule我将每个机场添加到国家/地区项目.在这一点上,SO成员已经帮助了我.

###################################
# PARSE EACH AIRPORT OF COUNTRY
###################################
def parse_schedule(self, response):
        """we want to loop this continuously to build every departure and arrivals requests"""
        item = response.meta['airport_item']
        i = response.meta['i']
        urls = response.meta['airport_urls']

        urls_departures, urls_arrivals = self.compute_urls_by_page(response, item['airports'][i]['name'], item['airports'][i]['code_little'])

        print("urls_departures = ", len(urls_departures))
        print("urls_arrivals = ", len(urls_arrivals))

        ## YIELD NOT CALLED
        yield scrapy.Request(response.url, self.parse_departures_page, meta={'airport_item': item, 'page_urls': urls_departures, 'i':0 , 'p': 0}, dont_filter=True)

        # now do next schedule items
        if not urls:
            yield item
            return
        url = urls.pop()

        yield scrapy.Request(url, self.parse_schedule, meta={'airport_item': item, 'airport_urls': urls, 'i': i + 1})

该self.compute_urls_by_page方法计算正确的URL以检索一个机场的所有出发和到达.

###################################
# PARSE EACH DEPARTURES / ARRIVALS
###################################
def parse_departures_page(self, response):
    item = response.meta['airport_item']
    p = response.meta['p']
    i = response.meta['i']
    page_urls = response.meta['page_urls']

    print("PAGE URL = ", page_urls)

    if not page_urls:
        yield item
        return
    page_url = page_urls.pop()

    print("GET PAGE FOR  ", item['airports'][i]['name'], ">> ", p)

    jsonload = json.loads(response.body_as_unicode())
    json_expression = jmespath.compile("result.response.airport.pluginData.schedule.departures.data")
    item['airports'][i]['departures'] = json_expression.search(jsonload)

    yield scrapy.Request(page_url, self.parse_departures_page, meta={'airport_item': item, 'page_urls': page_urls, 'i': i, 'p': p + 1})

接下来,parse_schedule通常调用self.parse_departure_page递归方法产生奇怪结果的第一个产量.Scrapy打电话给这个方法,但我只收集一个机场的离境页面,我不明白为什么......我的请求中可能有订单错误或收到源代码,所以也许你可以帮我找出答案.

完整的代码在GitHub上https://github.com/IDEES-Rouen/Flight-Scrapping/tree/master/flight/flight_project

您可以使用scrapy cawl airports命令运行它.

更新1:

我试着单独回答这个问题yield from,没有成功,因为你可以看到答案底部...所以,如果你有一个想法？

Answer 1

是的,我终于找到了答案在这里的SO ...

当您使用递归时yield,您需要使用yield from.这里简化了一个例子:

airport_list = ["airport1", "airport2", "airport3", "airport4"]

def parse_page_departure(airport, next_url, page_urls):

    print(airport, " / ", next_url)


    if not page_urls:
        return

    next_url = page_urls.pop()

    yield from parse_page_departure(airport, next_url, page_urls)

###################################
# PARSE EACH AIRPORT OF COUNTRY
###################################
def parse_schedule(next_airport, airport_list):

    ## GET EACH DEPARTURE PAGE

    departures_list = ["p1", "p2", "p3", "p4"]

    next_departure_url = departures_list.pop()
    yield parse_page_departure(next_airport,next_departure_url, departures_list)

    if not airport_list:
        print("no new airport")
        return

    next_airport_url = airport_list.pop()

    yield from parse_schedule(next_airport_url, airport_list)

next_airport_url = airport_list.pop()
result = parse_schedule(next_airport_url, airport_list)

for i in result:
    print(i)
    for d in i:
        print(d)

更新,不要使用真正的程序:

我试着在这里用真实的程序重现相同的yield from模式,但我使用它时出错,不明白为什么......scrapy.Request

这里的python回溯:

Traceback (most recent call last):
  File "/home/reyman/.pyenv/versions/venv352/lib/python3.5/site-packages/scrapy/utils/defer.py", line 102, in iter_errback
    yield next(it)
  File "/home/reyman/.pyenv/versions/venv352/lib/python3.5/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
    for x in result:
  File "/home/reyman/.pyenv/versions/venv352/lib/python3.5/site-packages/scrapy/spidermiddlewares/referer.py", line 339, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/home/reyman/.pyenv/versions/venv352/lib/python3.5/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/reyman/.pyenv/versions/venv352/lib/python3.5/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/reyman/Projets/Flight-Scrapping/flight/flight_project/spiders/AirportsSpider.py", line 209, in parse_schedule
    yield from scrapy.Request(url, self.parse_schedule, meta={'airport_item': item, 'airport_urls': urls, 'i': i + 1})
TypeError: 'Request' object is not iterable
2017-06-27 17:40:50 [scrapy.core.engine] INFO: Closing spider (finished)
2017-06-27 17:40:50 [scrapy.statscollectors] INFO: Dumping Scrapy stats:

Answer 2

评论: ... 不完全清楚 ... 你调用 AirportData(response, 1) ... 这里还有一个小错误： self.pprint(schedule)

我曾经class AirportData实施过（限制为 2 页和 2 个航班）。
更新了我的代码，删除class AirportData并添加了class Page.
现在应该填充所有依赖项。

这不是打字错误，self.pprint(...是class AirportsSpider Method用于漂亮打印对象，如末尾显示的输出。我已经增强class Schedule以显示基本用法。

评论：您的回答中的 AirportData 是什么？

编辑：class AirportData删除。
如# ENDPOINT所述Page object，飞行数据的 a 拆分为page.arrivals和page.departures。（仅限 2 页和 2 个航班）

Page = [Flight 1, Flight 1, ... Flight n] 
schedule.airport['arrivals'] == [Page 1, Page 2, ..., Page n]
schedule.airport['departures'] == [Page 1, Page 2, ..., Page n]

评论：...我们有多个页面，其中包含多个出发/到达。

是的，在第一次回答时，我没有任何api json进一步的回应。
现在我得到了来自 的响应，api json但没有反映给定的timestamp，从current date. 在api params寻找罕见，有你的链接描述？

不过，请考虑这种简化的方法：

# Page 对象持有一页到达/离开航班数据

class Page(object):
    def __init__(self, title, schedule):
        # schedule includes ['arrivals'] or ['departures]
        self.current = schedule['page']['current']
        self.total = schedule['page']['total']

        self.header = '{}:page:{} item:{}'.format(title, schedule['page'], schedule['item'])
        self.flight = []
        for data in schedule['data']:
            self.flight.append(data['flight'])

    def __iter__(self):
        yield from self.flight

# 调度对象持有一个机场所有页面

class Schedule(object):
    def __init__(self):
        self.country = None
        self.airport = None

    def __str__(self):
        arrivals = self.airport['arrivals'][0]
        departures = self.airport['departures'][0]
        return '{}\n\t{}\n\t\t{}\n\t\t\t{}\n\t\t{}\n\t\t\t{}'. \
            format(self.country['name'],
                   self.airport['name'],
                   arrivals.header,
                   arrivals.flight[0]['airline']['name'],
                   departures.header,
                   departures.flight[0]['airline']['name'], )

# 解析每个国家的机场

def parse_schedule(self, response):
    meta = response.meta

    if 'airport' in meta:
        # First call from parse_airports
        schedule = Schedule()
        schedule.country = response.meta['country']
        schedule.airport = response.meta['airport']
    else:
        schedule = response.meta['schedule']

    data = json.loads(response.body_as_unicode())
    airport = data['result']['response']['airport']

    schedule.airport['arrivals'].append(Page('Arrivals', airport['pluginData']['schedule']['arrivals']))
    schedule.airport['departures'].append(Page('Departures', airport['pluginData']['schedule']['departures']))

    page = schedule.airport['departures'][-1]
    if page.current < page.total:
        json_url = self.build_api_call(schedule.airport['code_little'], page.current + 1, self.compute_timestamp())
        yield scrapy.Request(json_url, meta={'schedule': schedule}, callback=self.parse_schedule)
    else:
        # ENDPOINT Schedule object holding one Airport.
        # schedule.airport['arrivals'] and schedule.airport['departures'] ==
        #   List of Page with List of Flight Data
        print(schedule)

# 解析每个机场

def parse_airports(self, response):
    country = response.meta['country']

    for airport in response.xpath('//a[@data-iata]'):
        name = ''.join(airport.xpath('./text()').extract()[0]).strip()

        if 'Charles' in name:
            meta = response.meta
            meta['airport'] = AirportItem()
            meta['airport']['name'] = name
            meta['airport']['link'] = airport.xpath('./@href').extract()[0]
            meta['airport']['lat'] = airport.xpath("./@data-lat").extract()[0]
            meta['airport']['lon'] = airport.xpath("./@data-lon").extract()[0]
            meta['airport']['code_little'] = airport.xpath('./@data-iata').extract()[0]
            meta['airport']['code_total'] = airport.xpath('./small/text()').extract()[0]

            json_url = self.build_api_call(meta['airport']['code_little'], 1, self.compute_timestamp())
            yield scrapy.Request(json_url, meta=meta, callback=self.parse_schedule)

# 主要解析

注：response.xpath('//a[@data-country]')返回的所有国家的两倍！

def parse(self, response):
    for a_country in response.xpath('//a[@data-country]'):
            name = a_country.xpath('./@title').extract()[0]
            if name == "France":
                country = CountryItem()
                country['name'] = name
                country['link'] = a_country.xpath('./@href').extract()[0]

                yield scrapy.Request(country['link'],
                                     meta={'country': country},
                                     callback=self.parse_airports)

Qutput：缩短为2页和每页2 个航班

France
    Paris Charles de Gaulle Airport
        Departures:(page=(1, 1, 7)) 2017-07-02 21:28:00 page:{'current': 1, 'total': 7} item:{'current': 100, 'limit': 100, 'total': 696}
            21:30 PM    AF1558  Newcastle Airport (NCL) Air France ARJ  Estimated dep 21:30
            21:30 PM    VY8833  Seville San Pablo Airport (SVQ) Vueling 320 Estimated dep 21:30
            ... (omitted for brevity)
        Departures:(page=(2, 2, 7)) 2017-07-02 21:28:00 page:{'current': 2, 'total': 7} item:{'current': 100, 'limit': 100, 'total': 696}
            07:30 AM    AF1680  London Heathrow Airport (LHR)   Air France 789  Scheduled
            07:30 AM    SN3628  Brussels Airport (BRU)  Brussels Airlines 733   Scheduled
            ... (omitted for brevity)
        Arrivals:(page=(1, 1, 7)) 2017-07-02 21:28:00 page:{'current': 1, 'total': 7} item:{'current': 100, 'limit': 100, 'total': 693}
            16:30 PM    LY325   Tel Aviv Ben Gurion International Airport (TLV) El Al Israel Airlines B739  Estimated 21:29
            18:30 PM    AY877   Helsinki Vantaa Airport (HEL)   Finnair E190    Landed 21:21
            ... (omitted for brevity)
        Arrivals:(page=(2, 2, 7)) 2017-07-02 21:28:00 page:{'current': 2, 'total': 7} item:{'current': 100, 'limit': 100, 'total': 693}
            00:15 AM    AF982   Douala International Airport (DLA)  Air France 772  Scheduled
            23:15 PM    AA44    New York John F. Kennedy International Airport (JFK)    American Airlines B763  Scheduled
            ... (omitted for brevity)

用 Python 测试：3.4.2 - Scrapy 1.4.0