使用python将数据从xml文件填充到sqlite数据库

Question

我有一个问题涉及解决问题的一些指导.我有一个xml文件,我必须使用脚本语言填充到数据库系统(无论是,它可能是sqlite,mysql):Python.

有没有人知道如何进行？

• 我需要进一步阅读哪些技术？
• 我必须安装哪些环境？
• 关于同一主题的任何教程？

我已经尝试使用其他语言的基于树和sax的方法解析xml,但是从Python开始,我不知道从哪里开始.我已经知道如何设计我需要的数据库.

另一个问题是,Python是否可以单独执行数据库ddl查询？

Answer 1

我建议您研究ElementTree将XML文件解析到内存中(解析它,然后将它全部发送到SQL DB,可能更容易,但是如果文件很大,元素树也允许增量操作) - 它是标准Python库作为模块xml.etree.

我推荐sqlite3(也在标准Python库中)作为选择的关系数据库(如果你有选择),再次因为它方便而且容易 - 底层的SQLite嵌入式关系数据库也在其自己的站点上有很好的文档.如果您需要有关Python如何与关系数据库("DB-API")进行交互的一般教程,那么这里有一个很好的教程.

一旦你完全理解了etree和sqlite3 - 并且你不一定需要安装任何东西(这是他们的魅力的一部分;-) - 你基本上都已经设置好了.(当然是SQLite本身的安装(命令行和/或GUI工具),所以你可以查看你的数据库文件并调整它们可能很好,像oXygen或XMLmind这样的图形XML编辑器可以查看和/或调整你的XML ,但根本不需要任何类型的工具,与使用Python而不是其他语言进行XML解析和SQLite编写完全没有关系;-).

是的,如果您选择使用其他关系数据库,您可以完美地做到CREATE TABLE和其他DDL查询sqlite3(以及任何其他符合DB-API的Python模块;-).

Answer 2

你可以用手工做的这一切sqlite3和xmlstarlet.

• 将xml转换为csv
• 将csv导入db.

(也许这不是你想要的.但它很方便)

# content of xml file
$ cat artist.xml
<results><artist><id>Ae2300d8b0232c06c</id><name>?? ???(Sarah Brightman)</name><hotSongs><!--freemusic/song/result/Sa3f6b810d7f98646--><song><id>Sa3f6b810d7f98646</id><name>Gloomy Sunday</name><artist>?? ???(Sarah Brightman)</artist><artistId>Ae2300d8b0232c06c</artistId><album>????</album><duration>227.0</duration><canBeDownloaded>true</canBeDownloaded><hasFullLyrics>true</hasFullLyrics><canBeStreamed>true</canBeStreamed><albumId>Bfef909d41922984a</albumId><hasSimilarSongs>true</hasSimilarSongs><hasRecommendation>false</hasRecommendation></song><!--freemusic/song/result/Sbb94fa21258a8b51--><song><id>Sbb94fa21258a8b51</id><name>???(2008???????)(You And Me)</name><artist>??</artist><artist>?? ???(Sarah Brightman)</artist><artistId>A6682d46bd4bdfc84</artistId><artistId>Ae2300d8b0232c06c</artistId><album>??? 2008????????</album><duration>256.0</duration><canBeDownloaded>true</canBeDownloaded><hasFullLyrics>true</hasFullLyrics><canBeStreamed>true</canBeStreamed><albumId>Babf3ab3b7ef0e3ed</albumId><hasSimilarSongs>true</hasSimilarSongs><hasRecommendation>false</hasRecommendation></song><!--freemusic/song/result/S6a47788bef57ff87--><song><id>S6a47788bef57ff87</id><name>Scarborough Fair</name><artist>?? ???(Sarah Brightman)</artist><artistId>Ae2300d8b0232c06c</artistId><album>????</album><duration>251.0</duration><canBeDownloaded>true</canBeDownloaded><hasFullLyrics>true</hasFullLyrics><canBeStreamed>true</canBeStreamed><albumId>Bfef909d41922984a</albumId><hasSimilarSongs>true</hasSimilarSongs><hasRecommendation>false</hasRecommendation></song><!--freemusic/song/result/Sde1aa68da126ddfe--><song><id>Sde1aa68da126ddfe</id><name>Time To Say Goodbye</name><artist>?? ???(Sarah Brightman)</artist><artistId>Ae2300d8b0232c06c</artistId><album>Symphony: Live In Vienna</album><duration>275.0</duration><canBeDownloaded>true</canBeDownloaded><hasFullLyrics>true</hasFullLyrics><canBeStreamed>true</canBeStreamed><albumId>B0187937a07b940f7</albumId><hasSimilarSongs>true</hasSimilarSongs><hasRecommendation>false</hasRecommendation></song><!--freemusic/song/result/S7483cc8c44e06bbb--><song><id>S7483cc8c44e06bbb</id><name>Amazing Grace</name><artist>?? ???(Sarah Brightman)</artist><artistId>Ae2300d8b0232c06c</artistId><album>???? - ???</album><duration>186.0</duration><canBeDownloaded>true</canBeDownloaded><hasFullLyrics>true</hasFullLyrics><canBeStreamed>true</canBeStreamed><albumId>B38e85a3056c0381c</albumId><hasSimilarSongs>true</hasSimilarSongs><hasRecommendation>false</hasRecommendation></song><!--freemusic/song/result/S2d9683c7230e18de--><song><id>S2d9683c7230e18de</id><name>This Love</name><artist>?? ???(Sarah Brightman)</artist><artistId>Ae2300d8b0232c06c</artistId><album>????</album><duration>371.0</duration><canBeDownloaded>true</canBeDownloaded><hasFullLyrics>true</hasFullLyrics><canBeStreamed>true</canBeStreamed><albumId>Bfef909d41922984a</albumId><hasSimilarSongs>true</hasSimilarSongs><hasRecommendation>false</hasRecommendation></song><!--freemusic/song/result/Scfb4989195b44e57--><song><id>Scfb4989195b44e57</id><name>Here With Me</name><artist>?? ???(Sarah Brightman)</artist><artistId>Ae2300d8b0232c06c</artistId><album>????</album><duration>324.0</duration><canBeDownloaded>true</canBeDownloaded><hasFullLyrics>true</hasFullLyrics><canBeStreamed>true</canBeStreamed><albumId>Bfef909d41922984a</albumId><hasSimilarSongs>true</hasSimilarSongs><hasRecommendation>false</hasRecommendation></song><!--freemusic/song/result/S7bafc588178b932c--><song><id>S7bafc588178b932c</id><name>La Lune</name><artist>?? ???(Sarah Brightman)</artist><artistId>Ae2300d8b0232c06c</artistId><album>????</album><duration>173.0</duration><canBeDownloaded>true</canBeDownloaded><hasFullLyrics>false</hasFullLyrics><canBeStreamed>true</canBeStreamed><albumId>Bfef909d41922984a</albumId><hasSimilarSongs>true</hasSimilarSongs><hasRecommendation>false</hasRecommendation></song></hotSongs></artist></results>

# convert xml to csv file using xmlstarlet
$ xml sel -t -m '//song' -v 'concat(id, ",", name, ",", duration)' -n artist.xml | sed '$d' >artist.csv

# content of csv file
$ cat artist.csv
Sa3f6b810d7f98646,Gloomy Sunday,227.0
Sbb94fa21258a8b51,???(2008???????)(You And Me),256.0
S6a47788bef57ff87,Scarborough Fair,251.0
Sde1aa68da126ddfe,Time To Say Goodbye,275.0
S7483cc8c44e06bbb,Amazing Grace,186.0
S2d9683c7230e18de,This Love,371.0
Scfb4989195b44e57,Here With Me,324.0
S7bafc588178b932c,La Lune,173.0

# create sqlite database
$ sqlite3 song.db
sqlite> CREATE TABLE song(id, name, duration);

# enter csv mode to import csv file
sqlite> .mode csv
sqlite> .import artist.csv song

# check everything is ok
sqlite> .mode column
sqlite> .header on
sqlite> select * from song;
id                 name           duration
-----------------  -------------  ----------
Sa3f6b810d7f98646  Gloomy Sunday  227.0
Sbb94fa21258a8b51  ???(200  256.0
S6a47788bef57ff87  Scarborough F  251.0
Sde1aa68da126ddfe  Time To Say G  275.0
S7483cc8c44e06bbb  Amazing Grace  186.0
S2d9683c7230e18de  This Love      371.0
Scfb4989195b44e57  Here With Me   324.0
S7bafc588178b932c  La Lune        173.0