使用python脚本从CSV文件中删除重复行并更新此CSV文件

Question

我有一个myfile.csv像行的行

first, second, third
1, 2, 3
a, b, c
1, 2, 3

等等.

我不明白如何删除重复的行myfile.csv.

一个条件,我们无法保存新文件,我们需要更新myfile.csv.
为了运行后脚本myfile.csv看起来像

first, second, third
a, b, c
1, 2, 3

因此,新数据不会保存到需要更新的新文件中myfile.csv.
非常感谢你.

Answer 1

您可以遍历数据并过滤列表以仅包含唯一值:

import csv
with open('filename.csv') as f:
  data = list(csv.reader(f))
  new_data = [a for i, a in enumerate(data) if a not in data[:i]]
  with open('filename.csv', 'w') as t:
     write = csv.writer(t)
     write.writerows(new_data)

Answer 2

简单而简短的pandas模块：

import pandas as pd

df = pd.read_csv('myfile.csv')
df.drop_duplicates(inplace=True)
df.to_csv('myfile.csv', index=False)

https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop_duplicates.html https： //pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_csv.html