将flatfiles加载到规范化的MySQL数据库中

Question

将数据从flatfiles加载到MySQL数据库,然后通过外键创建表之间的关系的最快方法是什么？

例如......我有一个平面文件格式:

[INDIVIDUAL]   [POP]  [MARKER]  [GENOTYPE]

"INDIVIDUAL1", "CEU", "rs55555","AA"  
"INDIVIDUAL1", "CEU", "rs535454","GA"  
"INDIVIDUAL1", "CEU", "rs555566","AT"  
"INDIVIDUAL1", "CEU", "rs12345","TT"  
...  
"INDIVIDUAL2", "JPT", "rs55555","AT"  

我需要加载到四个表中:

IND (id,fk_pop,name)  
POP (id,population)  
MARKER (id,rsid)  
GENOTYPE (id,fk_ind,fk_rsid,call)  

具体来说,如何以缩放的方式填充外键？这些数字在1000多个人的范围内,每个人有100万+基因型.

Answer 1

我会采取多步骤的方法来做到这一点.

1. 将数据加载到临时表中,该表与您拥有的文件格式匹配
2. 编写查询以执行其他插入,启动常规表,然后执行连接以获取FK值.

Answer 2

有一个更简单的方法。

首先，确保对那些应该有一个（名称、人口、rsid）的列有唯一约束。

然后使用类似以下内容：

 LOAD DATA INFILE 'data.txt' IGNORE INTO TABLE POP FIELDS TERMINATED BY ','
    ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES 
    (@name, population, @rsid, @call);
 LOAD DATA INFILE 'data.txt' IGNORE INTO TABLE MARKER FIELDS TERMINATED BY ',' 
    ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES 
    (@name, @population, rsid, @call);
 LOAD DATA INFILE 'data.txt' IGNORE INTO TABLE IND FIELDS TERMINATED BY ',' 
    ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES 
    (name, @population, @rsid, @call) 
    SET fk_pop = (SELECT id FROM POP WHERE population = @population);
 LOAD DATA INFILE 'data.txt' IGNORE INTO TABLE GENOTYPE FIELDS TERMINATED BY ',' 
    ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES 
    (@name, @population, @rsid, call)
    SET fk_ind = (SELECT id FROM IND where name = @name),
    fk_rsid = (SELECT id FROM MARKER where rsid = @rsid);

请注意，@ 用于指示变量，而不是列名。在前 2 个 LOAD DATA 中，这些仅用于忽略数据。在第二个2中，它们用于查找外键。

请注意，可能不会很快：）。