使用python请求下载CSV

Question

这是我的代码:

import csv
import requests
with requests.Session() as s:
    s.post(url, data=payload)
    download = s.get('url that directly download a csv report')

这使我可以访问csv文件.我尝试了不同的方法来处理下载:

这将在一个字符串中提供csv文件:

print download.content

这将打印第一行并返回错误:_csv.Error:在未加引号的字段中看到的换行符

cr = csv.reader(download, dialect=csv.excel_tab)
for row in cr:
    print row

这将在每一行中打印一个字母,它不会打印整个内容:

cr = csv.reader(download.content, dialect=csv.excel_tab)
for row in cr:
    print row

我的问题是在这种情况下读取csv文件的最有效方法是什么.以及如何下载实际的csv文件.

谢谢

Answer 1

这应该有所帮助:

import csv
import requests

CSV_URL = 'http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv'


with requests.Session() as s:
    download = s.get(CSV_URL)

    decoded_content = download.content.decode('utf-8')

    cr = csv.reader(decoded_content.splitlines(), delimiter=',')
    my_list = list(cr)
    for row in my_list:
        print(row)

输出样本:

['street', 'city', 'zip', 'state', 'beds', 'baths', 'sq__ft', 'type', 'sale_date', 'price', 'latitude', 'longitude']
['3526 HIGH ST', 'SACRAMENTO', '95838', 'CA', '2', '1', '836', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '59222', '38.631913', '-121.434879']
['51 OMAHA CT', 'SACRAMENTO', '95823', 'CA', '3', '1', '1167', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '68212', '38.478902', '-121.431028']
['2796 BRANCH ST', 'SACRAMENTO', '95815', 'CA', '2', '1', '796', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '68880', '38.618305', '-121.443839']
['2805 JANETTE WAY', 'SACRAMENTO', '95815', 'CA', '2', '1', '852', 'Residential', 'Wed May 21 00:00:00 EDT 2008', '69307', '38.616835', '-121.439146']
[...]

相关问题的答案:https://stackoverflow.com/a/33079644/295246

编辑:如果您需要下载大文件(即stream=True),其他答案很有用.

Answer 2

为了简化这些答案,并在下载大文件时提高性能,下面的工作可能会更有效.

import requests
from contextlib import closing
import csv

url = "http://download-and-process-csv-efficiently/python.csv"

with closing(requests.get(url, stream=True)) as r:
    reader = csv.reader(r.iter_lines(), delimiter=',', quotechar='"')
    for row in reader:
        print row   

通过stream=True在GET请求中设置,当我们传递r.iter_lines() 给csv.reader()时,我们将生成器传递给csv.reader().通过这样做,我们启用csv.reader()来懒惰地遍历响应中的每一行for row in reader.

这避免了在我们开始处理之前将整个文件加载到内存中,从而大大减少了大文件的内存开销.

Answer 3

您还可以使用DictReader迭代字典{'columnname': 'value', ...}

import csv
import requests

response = requests.get('http://example.test/foo.csv')
reader = csv.DictReader(response.iter_lines())
for record in reader:
    print(record)

Answer 4

我喜欢《The Aelfinn》的回答，并持保留态度。我只能通过稍微缩短一点，删除多余的部分，使用真实的数据源，使其与2.x和3.x兼容并保持其他地方看到的高级别的内存效率来改善它们：

import csv
import requests

CSV_URL = 'http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv'

with requests.get(CSV_URL, stream=True) as r:
    lines = (line.decode('utf-8') for line in r.iter_lines())
    for row in csv.reader(lines):
        print(row)

糟糕的是，3.x在CSV方式上的灵活性较差，因为迭代器必须发出Unicode字符串（而requests确实如此bytes），因为仅2.x的版本for row in csv.reader(r.iter_lines()):-更像Python（更简短，更易于阅读）。无论如何，请注意上面的2.x / 3.x解决方案将无法处理OP所描述的情况，即在读取的数据中未引用NEWLINE的情况。

对于OP有关下载（相对于处理）实际CSV文件的问题，下面是另一个脚本，该脚本可以兼容2.x和3.x，具有最小的可读性和存储效率：

import os
import requests

CSV_URL = 'http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv'

with open(os.path.split(CSV_URL)[1], 'wb') as f, \
        requests.get(CSV_URL, stream=True) as r:
    for line in r.iter_lines():
        f.write(line)

Answer 5

我使用这段代码（我使用Python 3）：

import csv
import io
import requests

url = "http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv"
r = requests.get(url)
r.encoding = 'utf-8'  # useful if encoding is not sent (or not sent properly) by the server
csvio = io.StringIO(r.text, newline="")
data = []
for row in csv.DictReader(csvio):
    data.append(row)

Answer 6

转换为 Pandas DataFrame：

from io import StringIO
text=StringIO(download.content.decode('utf-8'))
df=pd.read_csv(text)