如何在 Colaboratory Google 上使用 Selenium？

Question

我从网上抓取了很多信息，我希望它可以在云上运行。所以我想使用 colaboratory，但它变成了错误

WebDriverException                        Traceback (most recent call last)
<ipython-input-35-abcc3b93dfa7> in <module>()
     20 options.add_argument("--start-maximized");
     21 options.add_argument("--headless");
---> 22 driver = webdriver.Chrome('chromedriver', chrome_options=options)
     23 
     24 book  = cd + "/target.xlsx"

/usr/local/lib/python3.6/dist-packages/selenium/webdriver/chrome/webdriver.py in __init__(self, executable_path, port, options, service_args, desired_capabilities, service_log_path, chrome_options, keep_alive)
     71             service_args=service_args,
     72             log_path=service_log_path)
---> 73         self.service.start()
     74 
     75         try:

/usr/local/lib/python3.6/dist-packages/selenium/webdriver/common/service.py in start(self)
     96         count = 0
     97         while True:
---> 98             self.assert_process_still_running()
     99             if self.is_connectable():
    100                 break

/usr/local/lib/python3.6/dist-packages/selenium/webdriver/common/service.py in assert_process_still_running(self)
    109             raise WebDriverException(
    110                 'Service %s unexpectedly exited. Status code was: %s'
--> 111                 % (self.path, return_code)
    112             )
    113 

WebDriverException: Message: Service chromedriver unexpectedly exited. Status code was: -6

我阅读了文章，它说这是有效的。 我们如何在 colab.research.google.com 中使用 Selenium Webdriver？ 但实际上不是。

任何想法表示赞赏。

我的选择是

options = Options()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--disable-gpu')
driver = webdriver.Chrome('chromedriver', chrome_options=options)

? 这最后一句话出错了

WebDriverException: Message: Service chromedriver unexpectedly exited. Status code was: -6

============================================ 我的整个图表

!sudo apt install unzip
!wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
!unzip chromedriver_linux64.zip -d /usr/bin/
from google.colab import drive
drive.mount('/content/drive')
!pip install selenium
!pip install openpyxl

然后，python脚本是

cd = "drive/My Drive/doc/????/????/scrap/*"
import os, subprocess
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
import selenium
import bs4
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from bs4 import BeautifulSoup
import openpyxl
import time, re, csv, urllib.parse
options = Options()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--disable-gpu')
driver = webdriver.Chrome('chromedriver', chrome_options=options)

Answer 1

# install chromium, its driver, and selenium
!apt update
!apt install chromium-chromedriver
!pip install selenium
# set options to be headless, ..
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
# open it, go to a website, and get results
wd = webdriver.Chrome('chromedriver',options=options)
wd.get("https://www.website.com")
print(wd.page_source)  # results

我把这一切都包装成一个图书馆

!pip install kora
from kora.selenium import wd

Answer 2

我认为这段代码会起作用：

\n

!sudo apt install unzip\n!wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip\n#!unzip chromedriver_linux64.zip -d /usr/bin/\nfrom google.colab import drive\n!pip install selenium\n!pip install openpyxl\n!apt-get update\n!apt-get install -y unzip xvfb libxi6 libgconf-2-4\n!apt-get install default-jdk \ncd = "drive/My Drive/doc/\xe6\xa5\xad\xe5\x8b\x99\xe8\xb3\x87\xe6\x96\x99/\xe3\x82\xa4\xe3\x83\xbc\xe3\x82\xb3\xe3\x83\xac/scrape/*"\nimport os, subprocess\nimport sys\nsys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')\nimport selenium\nimport bs4\nfrom selenium import webdriver\nfrom selenium.webdriver.chrome.options import Options\nfrom selenium.webdriver.support.ui import WebDriverWait\nfrom selenium.webdriver.support import expected_conditions as EC\nfrom selenium.webdriver.common.by import By\nfrom selenium.common.exceptions import TimeoutException\nfrom bs4 import BeautifulSoup\nimport openpyxl\nimport time, re, csv, urllib.parse\noptions = Options()\noptions.add_argument('--headless')\noptions.add_argument('--no-sandbox')\noptions.add_argument('--disable-dev-shm-usage')\noptions.add_argument('--disable-gpu')\ndriver = webdriver.Chrome('chromedriver', chrome_options=options)\n

\n