有没有一种方法可以将CSV列转换为层次关系？

Question

我有700万份生物多样性记录的csv，其中分类学级别为列。例如：

RecordID,kingdom,phylum,class,order,family,genus,species
1,Animalia,Chordata,Mammalia,Primates,Hominidae,Homo,Homo sapiens
2,Animalia,Chordata,Mammalia,Carnivora,Canidae,Canis,Canis
3,Plantae,nan,Magnoliopsida,Brassicales,Brassicaceae,Arabidopsis,Arabidopsis thaliana
4,Plantae,nan,Magnoliopsida,Fabales,Fabaceae,Phaseoulus,Phaseolus vulgaris

我想在D3中创建一个可视化文件，但是数据格式必须是网络，其中每个列的不同值都是上一个特定值的列的子级。我需要从csv转到类似这样的内容：

{
  name: 'Animalia',
  children: [{
    name: 'Chordata',
    children: [{
      name: 'Mammalia',
      children: [{
        name: 'Primates',
        children: 'Hominidae'
      }, {
        name: 'Carnivora',
        children: 'Canidae'
      }]
    }]
  }]
}

我还没有想到不使用上千个for循环就如何做到这一点的想法。是否有人对如何在python或javascript上创建此网络提出建议？

Answer 1

为了创建所需的确切嵌套对象，我们将混合使用纯JavaScript和名为的D3方法d3.stratify。但是，请记住，有700万行（请参见下面的脚本）需要计算。

值得一提的是，对于此建议的解决方案，您必须将Kingdoms分开在不同的数据数组中（例如，使用Array.prototype.filter）。出现此限制是因为我们需要一个根节点，并且在Linnaean分类中，王国之间没有任何关系（除非您创建“ Domain”作为最高级别，它将作为所有真核生物的根，但是随后您将拥有相同的根节点）古细菌和细菌的问题）。

因此，假设您只有一个王国，就拥有此CSV（我添加了更多行）：

RecordID,kingdom,phylum,class,order,family,genus,species
1,Animalia,Chordata,Mammalia,Primates,Hominidae,Homo,Homo sapiens
2,Animalia,Chordata,Mammalia,Carnivora,Canidae,Canis,Canis latrans
3,Animalia,Chordata,Mammalia,Cetacea,Delphinidae,Tursiops,Tursiops truncatus
1,Animalia,Chordata,Mammalia,Primates,Hominidae,Pan,Pan paniscus

基于该CSV，我们将在此处创建一个名为的数组tableOfRelationships，顾名思义，该数组具有等级之间的关系：

const data = d3.csvParse(csv);

const taxonomicRanks = data.columns.filter(d => d !== "RecordID");

const tableOfRelationships = [];

data.forEach(row => {
  taxonomicRanks.forEach((d, i) => {
    if (!tableOfRelationships.find(e => e.name === row[d])) tableOfRelationships.push({
      name: row[d],
      parent: row[taxonomicRanks[i - 1]] || null
    })
  })
});

对于上面的数据，这是tableOfRelationships：

+---------+----------------------+---------------+
| (Index) |         name         |    parent     |
+---------+----------------------+---------------+
|       0 | "Animalia"           | null          |
|       1 | "Chordata"           | "Animalia"    |
|       2 | "Mammalia"           | "Chordata"    |
|       3 | "Primates"           | "Mammalia"    |
|       4 | "Hominidae"          | "Primates"    |
|       5 | "Homo"               | "Hominidae"   |
|       6 | "Homo sapiens"       | "Homo"        |
|       7 | "Carnivora"          | "Mammalia"    |
|       8 | "Canidae"            | "Carnivora"   |
|       9 | "Canis"              | "Canidae"     |
|      10 | "Canis latrans"      | "Canis"       |
|      11 | "Cetacea"            | "Mammalia"    |
|      12 | "Delphinidae"        | "Cetacea"     |
|      13 | "Tursiops"           | "Delphinidae" |
|      14 | "Tursiops truncatus" | "Tursiops"    |
|      15 | "Pan"                | "Hominidae"   |
|      16 | "Pan paniscus"       | "Pan"         |
+---------+----------------------+---------------+

看看：null作为父项Animalia的原因：这就是为什么我告诉您您需要按王国将数据集分开的原因null，整个表中只能有一个值。

最后，基于该表，我们使用d3.stratify()以下命令创建层次结构：

const stratify = d3.stratify()
    .id(function(d) { return d.name; })
    .parentId(function(d) { return d.parent; });

const hierarchicalData = stratify(tableOfRelationships);

这是演示。打开浏览器的控制台（该任务的片段不是很好），并检查children该对象的多个级别（）：

RecordID,kingdom,phylum,class,order,family,genus,species
1,Animalia,Chordata,Mammalia,Primates,Hominidae,Homo,Homo sapiens
2,Animalia,Chordata,Mammalia,Carnivora,Canidae,Canis,Canis latrans
3,Animalia,Chordata,Mammalia,Cetacea,Delphinidae,Tursiops,Tursiops truncatus
1,Animalia,Chordata,Mammalia,Primates,Hominidae,Pan,Pan paniscus

const data = d3.csvParse(csv);

const taxonomicRanks = data.columns.filter(d => d !== "RecordID");

const tableOfRelationships = [];

data.forEach(row => {
  taxonomicRanks.forEach((d, i) => {
    if (!tableOfRelationships.find(e => e.name === row[d])) tableOfRelationships.push({
      name: row[d],
      parent: row[taxonomicRanks[i - 1]] || null
    })
  })
});

PS：我不知道您将创建什么样的数据，但您实际上应该避免使用分类学排名。整个Linnaean分类法已过时，我们不再使用等级：由于系统发育系统是在60年代中期开发的，因此我们仅使用分类单位，而没有任何分类等级（此处为进化生物学老师）。另外，我对这700万行非常好奇，因为我们已经描述了100万种以上！

Answer 2

使用python和python-benedict库完全可以轻松完成所需的工作（它在Github上是开源的：

安装 pip install python-benedict

from benedict import benedict as bdict

# data source can be a filepath or an url
data_source = """
RecordID,kingdom,phylum,class,order,family,genus,species
1,Animalia,Chordata,Mammalia,Primates,Hominidae,Homo,Homo sapiens
2,Animalia,Chordata,Mammalia,Carnivora,Canidae,Canis,Canis
3,Plantae,nan,Magnoliopsida,Brassicales,Brassicaceae,Arabidopsis,Arabidopsis thaliana
4,Plantae,nan,Magnoliopsida,Fabales,Fabaceae,Phaseoulus,Phaseolus vulgaris
"""
data_input = bdict.from_csv(data_source)
data_output = bdict()

ancestors_hierarchy = ['kingdom', 'phylum', 'class', 'order', 'family', 'genus', 'species']
for value in data_input['values']:
    data_output['.'.join([value[ancestor] for ancestor in ancestors_hierarchy])] = bdict()

print(data_output.dump())
# if this output is ok for your needs, you don't need the following code

keypaths = sorted(data_output.keypaths(), key=lambda item: len(item.split('.')), reverse=True)

data_output['children'] = []
def transform_data(d, key, value):
    if isinstance(value, dict):
        value.update({ 'name':key, 'children':[] })
data_output.traverse(transform_data)

for keypath in keypaths:
    target_keypath = '.'.join(keypath.split('.')[:-1] + ['children'])
    data_output[target_keypath].append(data_output.pop(keypath))

print(data_output.dump())

第一个打印输出将是：

{
    "Animalia": {
        "Chordata": {
            "Mammalia": {
                "Carnivora": {
                    "Canidae": {
                        "Canis": {
                            "Canis": {}
                        }
                    }
                },
                "Primates": {
                    "Hominidae": {
                        "Homo": {
                            "Homo sapiens": {}
                        }
                    }
                }
            }
        }
    },
    "Plantae": {
        "nan": {
            "Magnoliopsida": {
                "Brassicales": {
                    "Brassicaceae": {
                        "Arabidopsis": {
                            "Arabidopsis thaliana": {}
                        }
                    }
                },
                "Fabales": {
                    "Fabaceae": {
                        "Phaseoulus": {
                            "Phaseolus vulgaris": {}
                        }
                    }
                }
            }
        }
    }
}

第二个打印输出将是：

{
    "children": [
        {
            "name": "Animalia",
            "children": [
                {
                    "name": "Chordata",
                    "children": [
                        {
                            "name": "Mammalia",
                            "children": [
                                {
                                    "name": "Carnivora",
                                    "children": [
                                        {
                                            "name": "Canidae",
                                            "children": [
                                                {
                                                    "name": "Canis",
                                                    "children": [
                                                        {
                                                            "name": "Canis",
                                                            "children": []
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                },
                                {
                                    "name": "Primates",
                                    "children": [
                                        {
                                            "name": "Hominidae",
                                            "children": [
                                                {
                                                    "name": "Homo",
                                                    "children": [
                                                        {
                                                            "name": "Homo sapiens",
                                                            "children": []
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ]
                        }
                    ]
                }
            ]
        },
        {
            "name": "Plantae",
            "children": [
                {
                    "name": "nan",
                    "children": [
                        {
                            "name": "Magnoliopsida",
                            "children": [
                                {
                                    "name": "Brassicales",
                                    "children": [
                                        {
                                            "name": "Brassicaceae",
                                            "children": [
                                                {
                                                    "name": "Arabidopsis",
                                                    "children": [
                                                        {
                                                            "name": "Arabidopsis thaliana",
                                                            "children": []
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                },
                                {
                                    "name": "Fabales",
                                    "children": [
                                        {
                                            "name": "Fabaceae",
                                            "children": [
                                                {
                                                    "name": "Phaseoulus",
                                                    "children": [
                                                        {
                                                            "name": "Phaseolus vulgaris",
                                                            "children": []
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ]
                        }
                    ]
                }
            ]
        }
    ]
}

Answer 3

var log = console.log;
var data = `
1,Animalia,Chordata,Mammalia,Primates,Hominidae,Homo,Homo sapiens
2,Animalia,Chordata,Mammalia,Carnivora,Canidae,Canis,Canis
3,Plantae,nan,Magnoliopsida,Brassicales,Brassicaceae,Arabidopsis,Arabidopsis thaliana
4,Plantae,nan,Magnoliopsida,Fabales,Fabaceae,Phaseoulus,Phaseolus vulgaris`;
//make array of rows with array of values
data = data.split("\n").map(v=>v.split(","));
//init tree
var tree = {};
data.forEach(row=>{
    //set current = root of tree for every row
    var cur = tree; 
    var id = false;
    row.forEach((value,i)=>{
        if (i == 0) {
            //set id and skip value
            id = value;
            return;
        }
        //If branch not exists create. 
        //If last value - write id
        if (!cur[value]) cur[value] = (i == row.length - 1) ? id : {};
        //Move link down on hierarhy
        cur = cur[value];
    });
}); 
log("Tree:");
log(JSON.stringify(tree, null, "  "));

//Now you have hierarhy in tree and can do anything with it.
var toStruct = function(obj) {
    let ret = [];
    for (let key in obj) {
        let child = obj[key];
        let rec = {};
        rec.name = key;
        if (typeof child == "object") rec.children = toStruct(child);
        ret.push(rec);
    }
    return ret;
}
var struct = toStruct(tree);
console.log("Struct:");
console.log(struct);

Answer 4

这看起来很简单，所以也许我不理解你的问题。

您想要的数据结构是一组嵌套的字典、键/值对。您的顶级王国词典为您的每个王国都有一个键，其值是门词典。门字典（对于一个王国）有每个门名称的键，每个键都有一个作为类字典的值，依此类推。

为了简化编码，您的属字典将为每个物种提供一个键，但物种的值将是空字典。

这应该是你想要的；不需要奇怪的库。

import csv

def read_data(filename):
    tree = {}
    with open(filename) as f:
        f.readline()  # skip the column headers line of the file
        for animal_cols in csv.reader(f):
            spot = tree
            for name in animal_cols[1:]:  # each name, skipping the record number
                if name in spot:  # The parent is already in the tree
                    spot = spot[name]  
                else:
                    spot[name] = {}  # creates a new entry in the tree
                    spot = spot[name]
    return tree

为了测试它，我使用了您的数据和pprint标准库中的数据。

from pprint import pprint
pprint(read_data('data.txt'))

得到

{'Animalia': {'Chordata': {'Mammalia': {'Carnivora': {'Canidae': {'Canis': {'Canis': {}}}},
                                        'Primates': {'Hominidae': {'Homo': {'Homo sapiens': {}}}}}}},
 'Plantae': {'nan': {'Magnoliopsida': {'Brassicales': {'Brassicaceae': {'Arabidopsis': {'Arabidopsis thaliana': {}}}},
                                       'Fabales': {'Fabaceae': {'Phaseoulus': {'Phaseolus vulgaris': {}}}}}}}}

再次阅读您的问题，您可能需要一个大表对（“来自更一般组的链接”、“链接到更具体的组”）。也就是说，“Animalia”链接到“Animalia:Chordata”，“Animalia:Chordata”链接到“Animalia:Chordata:Mammalia”等。不幸的是，数据中的“nan”意味着每个链接都需要全名。如果（父母，孩子）对是你想要的，这样走树：

def walk_children(tree, parent=''):
    for child in tree.keys():
        full_name = parent + ':' + child
        yield (parent, full_name)
        yield from walk_children(tree[child], full_name)

tree = read_data('data.txt')
for (parent, child) in walk_children(tree):
    print(f'parent="{parent}" child="{child}"')

给予：

parent="" child=":Animalia"
parent=":Animalia" child=":Animalia:Chordata"
parent=":Animalia:Chordata" child=":Animalia:Chordata:Mammalia"
parent=":Animalia:Chordata:Mammalia" child=":Animalia:Chordata:Mammalia:Primates"
parent=":Animalia:Chordata:Mammalia:Primates" child=":Animalia:Chordata:Mammalia:Primates:Hominidae"
parent=":Animalia:Chordata:Mammalia:Primates:Hominidae" child=":Animalia:Chordata:Mammalia:Primates:Hominidae:Homo"
parent=":Animalia:Chordata:Mammalia:Primates:Hominidae:Homo" child=":Animalia:Chordata:Mammalia:Primates:Hominidae:Homo:Homo sapiens"
parent=":Animalia:Chordata:Mammalia" child=":Animalia:Chordata:Mammalia:Carnivora"
parent=":Animalia:Chordata:Mammalia:Carnivora" child=":Animalia:Chordata:Mammalia:Carnivora:Canidae"
parent=":Animalia:Chordata:Mammalia:Carnivora:Canidae" child=":Animalia:Chordata:Mammalia:Carnivora:Canidae:Canis"
parent=":Animalia:Chordata:Mammalia:Carnivora:Canidae:Canis" child=":Animalia:Chordata:Mammalia:Carnivora:Canidae:Canis:Canis"
parent="" child=":Plantae"
parent=":Plantae" child=":Plantae:nan"
parent=":Plantae:nan" child=":Plantae:nan:Magnoliopsida"
parent=":Plantae:nan:Magnoliopsida" child=":Plantae:nan:Magnoliopsida:Brassicales"
parent=":Plantae:nan:Magnoliopsida:Brassicales" child=":Plantae:nan:Magnoliopsida:Brassicales:Brassicaceae"
parent=":Plantae:nan:Magnoliopsida:Brassicales:Brassicaceae" child=":Plantae:nan:Magnoliopsida:Brassicales:Brassicaceae:Arabidopsis"
parent=":Plantae:nan:Magnoliopsida:Brassicales:Brassicaceae:Arabidopsis" child=":Plantae:nan:Magnoliopsida:Brassicales:Brassicaceae:Arabidopsis:Arabidopsis thaliana"
parent=":Plantae:nan:Magnoliopsida" child=":Plantae:nan:Magnoliopsida:Fabales"
parent=":Plantae:nan:Magnoliopsida:Fabales" child=":Plantae:nan:Magnoliopsida:Fabales:Fabaceae"
parent=":Plantae:nan:Magnoliopsida:Fabales:Fabaceae" child=":Plantae:nan:Magnoliopsida:Fabales:Fabaceae:Phaseoulus"
parent=":Plantae:nan:Magnoliopsida:Fabales:Fabaceae:Phaseoulus" child=":Plantae:nan:Magnoliopsida:Fabales:Fabaceae:Phaseoulus:Phaseolus vulgaris"