AWS Athena: HIVE_BAD_DATA ERROR: Field type DOUBLE in parquet is incompatible with type defined in table schema

Question

I use AWS Athena to query some data stored in S3, namely partitioned parquet files with pyarrow compression.

I have three columns with string values, one column called "key" with int values and one column called "result" which have both double and int values.

With those columns, I created Schema like:

create external table (
    key int,
    result double,
    location string,
    vehicle_name string.
    filename string
)

When I queried the table, I would get

HIVE_BAD_DATA: Field results type INT64 in parquet is incompatible with type DOUBLE defined in table schema

So, I modified a schema with result datatype as INT.

Then I queried the table and got,

HIVE_BAD_DATA: Field results type DOUBLE in parquet is incompatible with type INT defined in table schema

I've looked around to try to understand why this might happen but found no solution.

Any suggestion is much appreciated.

Answer 1

在我看来，您有一些文件的列类型为 double，而某些文件的类型为 int。当您将表的列键入为 double 时，Athena 最终将读取相应列为 int 的文件并抛出此错误，如果您将表列键入为 int，则反之亦然。

据我所知，Athena 不进行类型强制，但即使这样做，类型也不兼容：Athena 中的 DOUBLE 列不能表示 Parquet INT64 列的所有可能值，而 Athena 中的 INT 列不能表示浮点数（对于 Parquet INT64，Athena 中需要一个 BIGINT 列）。

解决方案是确保您的文件都具有相同的架构。您可能需要在生成文件的代码中明确说明要生成的模式（例如，使其始终使用 DOUBLE）。