如何在迭代 pandas 数据框时显示进度条

Question

我正在尝试迭代包含近一百万个条目的 Pandas 数据框。我正在使用 for 循环来迭代它们。以下面的代码为例

import pandas as pd 
import os 
from requests_html import HTMLSession
from tqdm import tqdm
import time


df = pd.read_csv(os.getcwd()+'/test-urls.csv')
df = df.drop('Unnamed: 0', axis=1 )

new_df = pd.DataFrame(columns = ['pid', 'orig_url', 'hosted_url'])
refused_df = pd.DataFrame(columns = ['pid', 'refused_url'])

tic = time.time()

for idx, row in df.iterrows():

    img_id = row['pid']
    url = row['image_url']

    #Let's do scrapping 
    session = HTMLSession()
    r  = session.get(url)
    r.html.render(sleep=1, keep_page=True, scrolldown=1)

    count = 0 
    link_vals =  r.html.find('.zoomable')

    if len(link_vals) != 0 : 
        attrs = link_vals[0].attrs
        # print(attrs['src'])  
        embed_link = attrs['src']

    else: 
        while count <=7:
            link_vals =  r.html.find('.zoomable')
             count += 1
        else:
             print('Link refused connection for 7 tries. Adding URL to Refused URLs Data Frame')
            ref_val = [img_id,URL]
            len_ref = len(refused_df)
            refused_df.loc[len_ref] = ref_val
            print('Refused URL added')
            continue
    print('Got 1 link')

#Append scraped data to new_df
    len_df = len(new_df)
    append_value = [img_id,url, embed_link]
    new_df.loc[len_df] = append_value

我想知道如何使用进度条来添加进度的视觉表示。我将不胜感激任何帮助。如果您需要任何说明，请告诉我。

Answer 1

您可以尝试一下TQDM

from tqdm import tqdm
for idx, row in tqdm(df.iterrows()):
      do something

这主要用于命令行进度条。如果您正在寻找更多 GUI，还有其他解决方案。我想到了PySimpleGUI，但肯定有点复杂。