在不同的 CSV 文件中保存迭代

Question

对于当前项目，我计划运行以下脚本的多次迭代，并将结果保存在不同的 CSV 文件中，每次迭代都有一个新文件（CSV 部分位于脚本的末尾）。

给定的代码当前在终端中显示相关结果，而它只创建空的 CSV 文件。我花了几天时间想出如何解决这种情况，但无法找到解决方案。有没有人可以帮忙？

注意：我已根据用户建议更新了代码，而原始问题/挑战仍然存在。

import string
import json
import csv

import pandas as pd
import datetime
from dateutil.relativedelta import *

import numpy as np
import matplotlib.pyplot as plt


# Loading and reading dataset
file = open("Glassdoor_A.json", "r")
data = json.load(file)
df = pd.json_normalize(data)
df['Date'] = pd.to_datetime(df['Date'])


# Allocate periods for individual CSV file names
periods = pd.period_range('2009Q1','2018Q4',freq='Q')
ts = pd.Series(np.random.randn(40), periods)
type(ts.index)
intervals = ts.index


# Create individual empty files with headers
for i in intervals:
    name = 'Glassdoor_A_' + 'Text Main_' + str(i)
    with open(name+'.csv', 'w', newline='') as file:
        writer = csv.writer(file)


        # Create an empty dictionary
        d = dict()


        # Filtering by date
        start_date = pd.to_datetime('2009-01-01')
        end_date = pd.to_datetime('2009-03-31')
        last_end_date = pd.to_datetime('2017-12-31')
        mnthBeg = pd.offsets.MonthBegin(3)
        mnthEnd = pd.offsets.MonthEnd(3)
        while end_date <= last_end_date:
            filtered_dates = df[df.Date.between(start_date, end_date)]
            n = len(filtered_dates.index)
            print(f'Date range: {start_date.strftime("%Y-%m-%d")} - {end_date.strftime("%Y-%m-%d")},  {n} rows.')
            if n > 0:
                print(filtered_dates)
            start_date += mnthBeg
            end_date += mnthEnd


            # Processing Text Main section
            for index, row in filtered_dates.iterrows():
                line = row['Text Main']

                # Remove the leading spaces and newline character
                line = line.split(' ')
                line = [val.strip() for val in line]

                # Convert the characters in line to
                # lowercase to avoid case mismatch
                line = [val.lower() for val in line]

                # Remove the punctuation marks from the line
                line = [val.translate(val.maketrans("", "", string.punctuation)) for val in line]
                print(line)
                # Split the line into words
                # words = [val.split(" ") for val in line]
                # print(words)
                # Iterate over each word in line
                for word in line:
                    # Check if the word is already in dictionary
                    if word in d.keys():
                        # Increment count of word by 1
                        d[word] = d[word] + 1
                    else:
                        # Add the word to dictionary with count 1
                        d[word] = 1

                        print(d)


        # Print the contents of dictionary
        for key in list(d.keys()):
            print(key, ":", d[key])

            # Count the total number of words
            total = sum(d.values())
            percent = d[key] / total

            print(d[key], total, percent)


            # Save as CSV file
            while end_date <= last_end_date:

                for index, row in filtered_dates.iterrows():

                    for i in data:
                        name = 'Glassdoor_A_' + str(i)
                        with open(name+'.csv', 'a', newline='') as file:
                            writer.writerow(["Word", "Occurrences", "Percentage"])
                            writer.writerows([key, d[key], percent] for key in list(d.keys()))

Answer 1

编写写入 CSV 文件的内部循环：

# Create individual file names
for i in data:
    name = 'Glassdoor_A_' + str(i)

    # Save output in CSV file
    with open(name+'.csv', 'w', newline='') as file:
        ...

? 为外循环的每次迭代执行for index, row in filtered_dates.iterrows():。所以每次迭代都会覆盖以前创建的文件。尝试使用 mode as 'a'(append) 并在这两个循环之外使用空数据写入标头。

无需深入了解您正在计算和写出的内容，将数据附加到输出文件的方法是：

1. 创建仅包含脚本开头的标题的文件。
2. 最后一个内部循环应该以追加模式写入文件。

因此，在脚本的开头，添加：

data = json.load(file)
# Create individual empty files with headers
for i in data:
    name = 'Glassdoor_A_' + str(i)
    with open(name+'.csv', 'w', newline='') as file:
        writer = csv.writer(file)  # you probably don't need to use the csv module for the first part
        writer.writerow(["Text Main Words", "Text Main Occurrences"])
        # nothing else here for now

然后在脚本的末尾，对于写出数据的最内层循环，请执行以下操作：

while end_date <= last_end_date:
    ...
    for index, row in filtered_dates.iterrows():
        ...
        for i in data:
            name = 'Glassdoor_A_' + str(i)
            with open(name+'.csv', 'a', newline='') as file:  # note the 'append' mode
                writer = csv.writer(file)
                writer.writerows([occurrence])

_{顺便说一句，最后一行writer.writerows([occurrence])可能应该是writer.writerows(list(occurrence))如果occurrence还不是元组列表或每个内部列表中有两个元素的列表列表。}