Python重复删除

Question

我有一个关于在Python中删除重复项的问题.我已经阅读了很多帖子但还没有解决它.我有以下csv文件:

编辑

输入:

ID, Source, 1.A, 1.B, 1.C, 1.D
1, ESPN, 5,7,,,M
1, NY Times,,10,12,W
1, ESPN, 10,,Q,,M

输出应该是:

ID, Source, 1.A, 1.B, 1.C, 1.D, duplicate_flag
1, ESPN, 5,7,,,M, duplicate
1, NY Times,,10,12,W, duplicate
1, ESPN, 10,,Q,,M, duplicate 
1, NY Times, 5 (or 10 doesn't matter which one),7, 10, 12, W, not_duplicate

换句话说,如果ID相同,则从具有"NY Times"源的行中获取值,如果具有"NY Times"的行具有空白值并且来自"ESPN"源的重复行具有该单元格的值,从"ESPN"源获取行的值.对于输出,将原始两行标记为重复并创建第三行.

为了进一步澄清,因为我需要在具有不同列标题的许多不同csv文件上运行此脚本,所以我不能执行以下操作:

    def main():
        with open(input_csv, "rb") as infile:
            input_fields = ("ID", "Source", "1.A", "1.B", "1.C", "1.D")
            reader = csv.DictReader(infile, fieldnames = input_fields)
            with open(output_csv, "wb") as outfile:
                output_fields = ("ID", "Source", "1.A", "1.B", "1.C", "1.D", "d_flag")
                writer = csv.DictWriter(outfile, fieldnames = output_fields)
                writer.writerow(dict((h,h) for h in output_fields))
                next(reader)
                first_row = next(reader)
                for next_row in reader:
                    #stuff

因为我希望程序在前两列上运行,而与表中的其他列无关.换句话说,"ID"和"Source"将出现在每个输入文件中,但其余列将根据文件而变化.

非常感谢您提供的任何帮助!仅供参考,"来源"只能是:纽约时报,ESPN或华尔街日报,重复的优先顺序是:如果可用,请选择纽约时报,否则选择ESPN,否则选择华尔街日报.这适用于每个输入文件.