Python - 读取没有熊猫的镶木地板文件

Question

目前我正在使用下面的代码Python 3.5, Windows读取parquet文件。

import pandas as pd

parquetfilename = 'File1.parquet'
parquetFile = pd.read_parquet(parquetfilename, columns=['column1', 'column2'])  

但是，我想在不使用熊猫的情况下这样做。如何最好地做到这一点？我Python 2.7 and 3.6在Windows.

Answer 1

您可以用于duckdb此用途。它是一个类似于 SQLite 的嵌入式 RDBMS，但考虑到了 OLAP。有一个很好的 Python API 和一个 SQL 函数来导入 Parquet 文件：

import duckdb

conn = duckdb.connect(":memory:") # or a file name to persist the DB

# Keep in mind this doesn't support partitioned datasets,
# so you can only read one partition at a time
conn.execute("CREATE TABLE mydata AS SELECT * FROM parquet_scan('/path/to/mydata.parquet')")

# Export a query as CSV
conn.execute("COPY (SELECT * FROM mydata WHERE col = 'val') TO 'col_val.csv' WITH (HEADER 1, DELIMITER ',')")