在python中读取Avro文件时出错

Question

我在Python中成功安装了Apache Avro。然后，我尝试按照以下说明将Avro文件读入Python。

https://avro.apache.org/docs/1.8.1/gettingstartedpython.html

我的目录中有一堆Avros，该目录已经在Python中设置为正确的路径。这是我的代码：

import avro.schema
from avro.datafile import DataFileReader, DataFileWriter
from avro.io import DatumReader, DatumWriter

reader = DataFileReader(open("part-00000-of-01733.avro", "r"), DatumReader())
for user in reader:
   print (user)
reader.close()

但是，它返回此错误：

https://avro.apache.org/docs/1.8.1/gettingstartedpython.html

我确实知道，在该指令的示例中，首先创建了一个架构。但是什么是avsc文件？在我的情况下，应如何创建它和相应的架构？理想情况下，我想将Avro文件读入Python，并将其保存为Python磁盘或数据框/列表类型中的csv格式，以进行进一步分析。我在Windows 7上使用Python 3。

编辑 我尝试了Stephane的代码，它返回了一个新的错误

import avro.schema
from avro.datafile import DataFileReader, DataFileWriter
from avro.io import DatumReader, DatumWriter

reader = DataFileReader(open("part-00000-of-01733.avro", "r"), DatumReader())
for user in reader:
   print (user)
reader.close()

EDITED2：Stephane的代码在大多数情况下都可以使用，但有时会引发这样的AssertionError

Traceback (most recent call last):
File "I:\DJ data\read avro.py", line 42, in <module>
for user in reader:
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\datafile.py", line 522, in __next__
datum = self.datum_reader.read(self.datum_decoder)
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 480, in read
return self.read_data(self.writer_schema, self.reader_schema, decoder)
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 525, in read_data
return self.read_record(writer_schema, reader_schema, decoder)
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 725, in read_record
field_val = self.read_data(field.type, readers_field.type, decoder)
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 523, in read_data
return self.read_union(writer_schema, reader_schema, decoder)
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 689, in read_union
return self.read_data(selected_writer_schema, reader_schema, decoder)
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 493, in read_data
return self.read_data(writer_schema, s, decoder)
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 503, in read_data
return decoder.read_utf8()
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 248, in read_utf8
input_bytes = self.read_bytes()
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 241, in read_bytes
return self.read(nbytes)
File "I:\Program Files\lib\site-packages\avro_python3-1.8.1-py3.5.egg\avro\io.py", line 171, in read
assert (len(input_bytes) == n), input_bytes
AssertionError: b'BlackRock Group\n\n17 December 2015\n\nFORM 8.3\n\nPUBLIC OPENING POSITION DISCLOSURE/DEALING DISCLOSURE BY\n\nA PERSON WITH INTERESTS IN RELEVANT SECURITIES REPRESENTING 1% OR MORE\n\nRule 8.3 of the Takeover Code (the "Code") \n\n\n   1.         KEY INFORMATION \n \n (a) Full name of discloser:                                                                        BlackRock, Inc. \n-------------------------------------------------------------------------------------------------  ----------------- \n (b) Owner or controller of interests and short positions disclosed, if diffe

Answer 1

您正在使用Windows和Python 3。

• 在Python 3中，默认情况下会open以文本模式打开文件。这意味着当发生进一步的读取操作时，Python将尝试将文件的内容从某些字符集解码为unicode。

• 您没有指定默认字符集，因此Python尝试对内容进行解码，就好像该内容是使用编码的charmap（Windows上默认）。

• 显然，您的avro文件未在charmap中进行编码，并且解码失败并出现异常

• 据我所记得，avro标头无论如何都是二进制内容...不是文本（对此不确定）。因此，也许首先您不应该尝试使用open解码文件：

reader = DataFileReader(open("part-00000-of-01733.avro", 'rb'), DatumReader())

（注意'rb'，二进制模式）

编辑：对于下一个问题（AttributeError），您遇到了一个已知错误，该错误未在1.8.1中修复。在下一个版本发布之前，您可以执行以下操作：

import avro.schema
from avro.datafile import DataFileReader, DataFileWriter, VALID_CODECS, SCHEMA_KEY
from avro.io import DatumReader, DatumWriter
from avro import io as avro_io


class MyDataFileReader(DataFileReader):
    def __init__(self, reader, datum_reader):
        """Initializes a new data file reader.

        Args:
          reader: Open file to read from.
          datum_reader: Avro datum reader.
        """
        self._reader = reader
        self._raw_decoder = avro_io.BinaryDecoder(reader)
        self._datum_decoder = None  # Maybe reset at every block.
        self._datum_reader = datum_reader

        # read the header: magic, meta, sync
        self._read_header()

        # ensure codec is valid
        avro_codec_raw = self.GetMeta('avro.codec')
        if avro_codec_raw is None:
            self.codec = "null"
        else:
            self.codec = avro_codec_raw.decode('utf-8')
        if self.codec not in VALID_CODECS:
            raise DataFileException('Unknown codec: %s.' % self.codec)

        self._file_length = self._GetInputFileLength()

        # get ready to read
        self._block_count = 0
        self.datum_reader.writer_schema = (
            schema.Parse(self.GetMeta(SCHEMA_KEY).decode('utf-8')))


reader = MyDataFileReader(open("part-00000-of-01733.avro", "r"), DatumReader())
for user in reader:
    print (user)
reader.close()

如此愚蠢的bug可能会发布，这很奇怪，这并不表示代码已经成熟！