无法获取正确的 div 以从表中获取数据

Question

我试图从https://gmatclub.com/forum/decision-tracker.html抓取数据经过大量的点击和试验，我仍然无法确定如何从表格中获取数据？

import requests
from bs4 import BeautifulSoup
url = "https://gmatclub.com/forum/decision-tracker.html"
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')
container = soup.find('div', attrs = {'class' : 'mainPage'})
print(container)

Answer 1

如果你想练习，看看Developer Toos -> Network -> XHR并抓住更新端点：

https://gmatclub.com/api/schools/v1/forum/app-tracker-latest-updates?limit=50&year=all

并使用它来获取当前数据。

就是这样：

https://gmatclub.com/api/schools/v1/forum/app-tracker-latest-updates?limit=50&year=all

这将输出一个字典列表，它们实际上是您的表。然后您可以从中访问任何密钥。

{'id': '194901', 'user_id': '273781', 'applicant_type': 'regular', 'round_id': '4236', 'status_id': '9', 'school_id': '5', 'school_title': 'Booth', 'program_id': '11', 'program_type': '1', 'date': '2021-05-24 23:56:46', 'seconds_ago': '511', 'country': None, 'state': None, 'gmat_quant': None, 'gmat_verbal': None, 'gmat_total': None, 'gmat_modified': None, 'gre_quant': None, 'gre_verbal': None, 'gre_total': None, 'gre_modified_time': None, 'ea_quant': None, 'ea_verbal': None, 'ea_ir': None, 'ea_total': None, 'ea_modified_time': None, 'cat_india_percentile': None, 'cat_india_total': None, 'cat_india_modified_time': None, 'industry': None, 'we': None, 'gpa': None, 'accepted_via': 'phone', 'scholarship': '1', 'user_colour': '', 'truncate_username': '0', 'user_name': 'binhtbc'}

或者您可以将响应转储到pandas dataframe. 例如：

df = pd.DataFrame(endpoint["statistics"])
print(df.head(10))

输出：

       id  user_id applicant_type  ... user_colour truncate_username    user_name
0  194901   273781        regular  ...                             0      binhtbc
1  183152   643532        regular  ...                             0         AG23
2  194061     None        regular  ...      2a2a2a                 0      private
3  192923  1034549        regular  ...                             0  RicardoLima
4  193383  1034549        regular  ...                             0  RicardoLima
5  194900  1130431        regular  ...                          None          VFA
6  177937   876400        regular  ...      F87431                 0   icanhazmba
7  194899  1128750        regular  ...                          None     Amanda29
8  194898  1128002        regular  ...                          None      Raydiaz
9  193974  1021516        regular  ...                             0    Kurathore

而且，如果您愿意，请将其另存为.csv文件：

    df.to_csv("your_table_data.csv", index=False)