我是否通过这种重复数据删除功能重新发明了轮子？

Question

我正在寻找一种set()类似于重复删除列表的方法,除了原始列表中的项目不可清除(它们是dicts).

我花了一段时间寻找足够的东西,最后我写了这个小功能:

def deduplicate_list(lst, key):
    output = []
    keys = []
    for i in lst:
        if not i[key] in keys:
            output.append(i)
            keys.append(i[key])

    return output

如果a key是正确给出的并且是a string,则此函数可以很好地完成其工作.毋庸置疑,如果我了解一个允许相同功能的内置或标准库模块,我很乐意放弃我的小程序,转而采用更标准,更健壮的选择.

你知道这样的实施吗？

- 注意

从这个答案找到以下单行,

[dict(t) for t in set([tuple(d.items()) for d in l])]

聪明,不会工作,因为我必须使用项目作为嵌套dicts.

- 例子

为清楚起见,以下是使用此类例程的示例:

with_duplicates = [
    {
        "type": "users",
        "attributes": {
            "first-name": "John",
            "email": "john.smith@gmail.com",
            "last-name": "Smith",
            "handle": "jsmith"
        },
        "id": "1234"
    },
    {
        "type": "users",
        "attributes": {
            "first-name": "John",
            "email": "john.smith@gmail.com",
            "last-name": "Smith",
            "handle": "jsmith"
        },
        "id": "1234"
    }
]

without_duplicates = deduplicate_list(with_duplicates, key='id')

Answer 1

这个答案将有助于解决一个更通用的问题 - 查找唯一元素不是通过单个属性（id在您的情况下），而是如果任何嵌套属性不同

以下代码将返回唯一元素的索引列表

import copy

def make_hash(o):

  """
  Makes a hash from a dictionary, list, tuple or set to any level, that contains
  only other hashable types (including any lists, tuples, sets, and
  dictionaries).
  """

  if isinstance(o, (set, tuple, list)):

    return tuple([make_hash(e) for e in o])    

  elif not isinstance(o, dict):

    return hash(o)

  new_o = copy.deepcopy(o)
  for k, v in new_o.items():
    new_o[k] = make_hash(v)

  return hash(tuple(frozenset(sorted(new_o.items()))))

l = [
    {
        "type": "users",
        "attributes": {
            "first-name": "John",
            "email": "john.smith@gmail.com",
            "last-name": "Smith",
            "handle": "jsmith"
        },
        "id": "1234"
    },
    {
        "type": "users",
        "attributes": {
            "first-name": "AAA",
            "email": "aaa.aaah@gmail.com",
            "last-name": "XXX",
            "handle": "jsmith"
        },
        "id": "1234"
    },
    {
        "type": "users",
        "attributes": {
            "first-name": "John",
            "email": "john.smith@gmail.com",
            "last-name": "Smith",
            "handle": "jsmith"
        },
        "id": "1234"
    },
]

# get indicies of unique elements
In [254]: list({make_hash(x):i for i,x in enumerate(l)}.values())
Out[254]: [1, 2]