从字典列表中删除重复项

Question

如果嵌套字典前面没有键，我目前能够删除重复项。我使用此函数的字典列表的一个示例是：

 [{'asndb_prefix': '164.39.xxx.0/17',
  'cidr': '164.39.xxx.0/17',
  'cymru_asn': 'XXX',
  'cymru_country': 'GB',
  'cymru_owner': 'XXX , GB',
  'cymru_prefix': '164.39.xxx.0/17',
  'ips': ['164.39.xxx.xxx'],
  'network_id': '164.39.xxx.xxx/24',},
 {'asndb_prefix': '54.192.xxx.xxx/16',
  'cidr': '54.192.0.0/16',
  'cymru_asn': '16509',
  'cymru_country': 'US',
  'cymru_owner': 'AMAZON-02 - Amazon.com, Inc., US',
  'cymru_prefix': '54.192.144.0/22',
  'ips': ['54.192.xxx.xxx', '54.192.xxx.xxx'],
  'network_id': '54.192.xxx.xxx/24',
  }]

def remove_dict_duplicates(list_of_dicts):
    """
    "" Remove duplicates in dict 
    """
    list_of_dicts = [dict(t) for t in set([tuple(d.items()) for d in list_of_dicts])]
    # remove the {} before and after - not sure why these are placed as 
    # the first and last element 
    return list_of_dicts[1:-1]

但是，我希望能够根据该字典中关联的键和所有值删除重复项。因此，如果里面有相同的密钥具有不同的值，我不想删除它，但如果有完整的副本，则删除它。

    [{'50.16.xxx.0/24': {'asndb_prefix': '50.16.0.0/16',
   'cidr': '50.16.0.0/14',
   'cymru_asn': 'xxxx',
   'cymru_country': 'US',
   'cymru_owner': 'AMAZON-AES - Amazon.com, Inc., US',
   'cymru_prefix': '50.16.0.0/16',
   'ip': '50.16.221.xxx',
   'network_id': '50.16.xxx.0/24',
   'pyasn_asn': xxxx,
   'whois_asn': 'xxxx'}},
   // This would be removed
   {'50.16.xxx.0/24': {'asndb_prefix': '50.16.0.0/16',
   'cidr': '50.16.0.0/14',
   'cymru_asn': 'xxxxx',
   'cymru_country': 'US',
   'cymru_owner': 'AMAZON-AES - Amazon.com, Inc., US',
   'cymru_prefix': '50.16.0.0/16',
   'ip': '50.16.221.xxx',
   'network_id': '50.16.xxx.0/24',
   'pyasn_asn': xxxx,
   'whois_asn': 'xxxx'}},
   // This would NOT be removed
   {'50.16.xxx.0/24': {'asndb_prefix': '50.999.0.0/16',
   'cidr': '50.999.0.0/14',
   'cymru_asn': 'xxxx',
   'cymru_country': 'US',
   'cymru_owner': 'AMAZON-AES - Amazon.com, Inc., US',
   'cymru_prefix': '50.16.0.0/16',
   'ip': '50.16.221.xxx',
   'network_id': '50.16.xxx.0/24',
   'pyasn_asn': xxxx,
   'whois_asn': 'xxxx'}}]

我该怎么做呢？谢谢。

Answer 1

要从字典列表中删除重复项：

list_of_unique_dicts = []
for dict_ in list_of_dicts:
    if dict_ not in list_of_unique_dicts:
        list_of_unique_dicts.append(dict_)