Nohup在Ubuntu EC2中运行Silenium Webscraper

Question

我有一个使用硒的网络刮板，我想在注销后在后台事件中在Ubuntu EC2上运行，所以我尝试使用nohup。我当前的代码是：

webscrape.py：

from pyvirtualdisplay import Display
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0
from selenium.webdriver.support import expected_conditions as EC 

def main():

    display = Display(visible=0, size=(800, 600))
    display.start() #starts vitual display

    driver = webdriver.Firefox()

    ...do the webscraping...

    driver.close()
    display.stop()

if __name__ == "__main__": main()

当我登录EC2并python webscrape.py正常运行时。但是，当我nohup python webscrape.py注销时，它停止了。在nohup.out日志中，出现以下错误：

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Cruz/Scripts/WebScrape/google_brand_web_scraper.py", line 175, in <module>
    if __name__ == "__main__": main()
  File "/usr/local/lib/python2.7/dist-packages/Cruz/Scripts/WebScrape/google_brand_web_scraper.py", line 120, in main
    website = GoogleBrandWebsiteScraper().brand_url_pull_from_google(i,driver) # get website for a brand
  File "/usr/local/lib/python2.7/dist-packages/Cruz/Scripts/WebScrape/google_brand_web_scraper.py", line 34, in brand_url_pull_from_google
    s = BeautifulSoup(driver.page_source)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 436, in page_source
    return self.execute(Command.GET_PAGE_SOURCE)['value']
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 171, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 349, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 379, in _request
    self._conn.request(method, parsed_url.path, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 772, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 571, in create_connection
    raise err
socket.error: [Errno 111] Connection refused

因此，很明显，我注销后会有些混乱。任何提示表示赞赏。

Answer 1

您可能想尝试一下screen。我对nohup不熟悉，无法弄清楚您在那里遇到的问题，但是screen应该可以解决。

1. 您将运行screen以创建一个可以在其中进行工作的新终端。
2. 运行你的代码
3. 点击Ctrl+，a然后d从该终端分离（它将继续在后台运行）。
4. 运行screen -r将重新连接到该终端。

当您与终端“分离”时，可以与系统断开连接，该分离的终端将继续运行。因此，在步骤3和4之间，您可以断开连接。