相关疑难解决方法(0)

我正在尝试获取此网址的内容" https://itunes.apple.com/in/genre/ios-business/id6000?mt=8 "并显示此错误

import requests
from lxml import html

page = requests.get("https://itunes.apple.com/in/genre/ios-business/id6000?mt=8")
tree = html.fromstring(page.text)

flist = []
plist = []
for i in range(0, 100):
    app = tree.xpath("//div[@class='column first']/ul/li/a/@href")
    ap = app[0]
    page1 = requests.get(ap)

代码是

Traceback (most recent call last):
  File "/home/preetham/Desktop/eg.py", line 17, in <module>
    page1 = requests.get(ap)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, …

我不确定为什么,但我的脚本一旦到达第9页就会停止爬行.没有错误,例外或警告,所以我有点不知所措.

有人可以帮帮我吗？

PS 这是完整的脚本,以防任何人想要自己测试它!

def initiate_crawl():
    def refresh_page(url):
        ff = create_webdriver_instance()
        ff.get(url)
        ff.find_element(By.XPATH, '//*[@id="FilterItemView_sortOrder_dropdown"]/div/span[2]/span/span/span/span').click()
        ff.find_element(By.XPATH, '//a[contains(text(), "Discount - High to Low")]').click()
        items = WebDriverWait(ff, 15).until(
            EC.visibility_of_all_elements_located((By.XPATH, '//div[contains(@id, "100_dealView_")]'))
        )
        print(len(items))
        for count, item in enumerate(items):
            slashed_price = item.find_elements(By.XPATH, './/span[contains(@class, "a-text-strike")]')
            active_deals = item.find_elements(By.XPATH, './/*[contains(text(), "Add to Cart")]')
            if len(slashed_price) > 0 and len(active_deals) > 0:
                product_title = item.find_element(By.ID, 'dealTitle').text
                if product_title not in already_scraped_product_titles:
                    already_scraped_product_titles.append(product_title)
                    url = ff.current_url
                    ff.quit()
                    refresh_page(url)
                    break
            if count+1 is len(items):
                try:
                    next_button = …

有一个测试,像:

 import //needed imports

 public class TestClass{
    WebDriver driver;

    @Before
    public void setUp() {
       //some code
    }

    @Test
    public void test1() {
       //some code, including init of driver (geckodriver)
    }

   //@After
   // public void tearDown() {
   //  driver.quit();
   //}
}

所以,我使用firefox实例,使用geckodriver并成功运行我的测试.但是我希望不要在每次运行后关闭firefox窗口,因为我只想分析我拥有的东西,并在测试运行后修复任何需要的东西(我将在稍后的unComment driver.quit()).同时,每次调用而不关闭驱动程序会导致我的PC上的RAM过度影响(并且无关紧要 - 我是否在测试后手动关闭浏览器):

所以,问题是:是否有任何方法可以关闭"geckodriver"的进程(更确切地说 - 执行smth,这将关闭taskmgr中的geckodriver.exe进程),但是在测试完成后不会关闭浏览器？例如,在测试本身添加一些方法,无论如何......这不会影响我的工作/测试本身,我只想添加一些优化.

我正在尝试用 selenium 抓取网站的几页并使用结果，但是当我运行该函数两次时

[WinError 10061] No connection could be made because the target machine actively refused it'

第二个函数调用出现错误。这是我的方法：

import os
import re
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup as soup

opts = webdriver.ChromeOptions()
opts.binary_location = os.environ.get('GOOGLE_CHROME_BIN', None)
opts.add_argument("--headless")
opts.add_argument("--disable-dev-shm-usage")
opts.add_argument("--no-sandbox")
browser = webdriver.Chrome(executable_path="CHROME_DRIVER PATH", options=opts)

lst =[]
def search(st):
    for i in range(1,3):
        url = "https://gogoanime.so/anime-list.html?page=" + str(i)
        browser.get(url)
        req = browser.page_source
        sou = soup(req, "html.parser")
        title = sou.find('ul', class_ …

我尝试编写一个简单的脚本，每小时检查一次网站，并在发现有空时向我发送电子邮件。

我认为每隔一小时执行此操作不会引发任何问题，但出现以下错误：

MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=60745): Max retries exceeded with url: /session/900f45d6c8c800f2a8ebcf43daa05b69/url (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa42c261c10>: Failed to establish a new connection: [Errno 61] Connection refused'))

这是我的代码：

import schedule
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from notification import *
#script i have to send an email (works fine)

PATH = "mypath"
# i have the path where there drivers are

chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--window-size=1920x1080")
# to not open the browser

driver = webdriver.Chrome(options=chrome_options, executable_path=PATH) …

因此，我希望遍历 URL 数组并打开不同的 URL 以使用 Selenium 进行网页抓取。问题是，一旦我点击第二个 browser.get(url)，我就会收到“URL 超出最大重试次数”和“无法建立连接，因为目标计算机主动拒绝它”。

编辑：添加了其余的代码，尽管它只是 BeautifulSoup 的东西。

from bs4 import BeautifulSoup
import time
from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options
import json

chrome_options = Options()  
chromedriver = webdriver.Chrome(executable_path='C:/Users/andre/Downloads/chromedriver_win32/chromedriver.exe', options=chrome_options)
urlArr = ['https://link1', 'https://link2', '...']

for url in urlArr:
   with chromedriver as browser:
      browser.get(url)
      time.sleep(5)
      # Click a button
      chromedriver.find_elements_by_tag_name('a')[7].click()

      chromedriver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
      time.sleep(2)
      for i in range (0, 2):
         chromedriver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
         time.sleep(5)

      html = browser.page_source
      page_soup = BeautifulSoup(html, 'html.parser')
      boxes = …