redshift Unload操作导致冗余数据

Question

我们使用 UNLOAD 命令对基于 s3 的外部表运行一些转换，并将数据以 PARQUET 格式发布到不同的 s3 存储桶中。

我在卸载操作中使用 ALLOWOVERWRITE 选项来替换文件（如果它们已存在）。这在大多数情况下都可以正常工作，但有时会为相同的数据插入重复的文件，这会导致外部表显示重复的数字。

例如，如果分区中的 parquet 是包含完整数据的 0000_part_00.parquet。在下一次运行中，卸载预计会覆盖此文件，但会插入新文件 0000_part_01.parquet，这会使总输出加倍。

如果我只是清理整个分区并再次重新运行，这将不会重复。这种不一致使我们的系统不可靠。

unload (<simple select statement>)
to 's3://<s3 bucket>/<prefix>/'
iam_role '<iam-role>' allowoverwrite
PARQUET
PARTITION BY (partition_col1, partition_col2);

谢谢。

Answer 1

为了防止冗余数据，您必须在语句中使用 Redshift 的CLEANPATH选项UNLOAD。请注意文档中的差异（也许 AWS 可以进一步澄清这一点）：

ALLOWOVERWRITE
By default, UNLOAD fails if it finds files that it would possibly overwrite. If ALLOWOVERWRITE is specified, UNLOAD overwrites existing files, including the manifest file.

CLEANPATH
The CLEANPATH option removes existing files located in the Amazon S3 path specified in the TO clause before unloading files to the specified location.
If you include the PARTITION BY clause, existing files are removed only from the partition folders to receive new files generated by the UNLOAD operation.
You must have the s3:DeleteObject permission on the Amazon S3 bucket. For information, see Policies and Permissions in Amazon S3 in the Amazon Simple Storage Service Console User Guide. Files that you remove by using the `CLEANPATH` option are permanently deleted and can't be recovered.
You can't specify the `CLEANPATH` option if you specify the `ALLOWOVERWRITE` option.

因此，正如@Vzzarr所说，ALLOWOVERWRITE仅覆盖与传入文件名共享相同名称的文件。对于不需要过去数据的状态保持不变的重复卸载操作，则必须使用CLEANPATH.

请注意，不能在同一个 UNLOAD 语句中同时使用ALLOWOVERWRITE和。CLEANPATH

这是一个例子：

ALLOWOVERWRITE
By default, UNLOAD fails if it finds files that it would possibly overwrite. If ALLOWOVERWRITE is specified, UNLOAD overwrites existing files, including the manifest file.

CLEANPATH
The CLEANPATH option removes existing files located in the Amazon S3 path specified in the TO clause before unloading files to the specified location.
If you include the PARTITION BY clause, existing files are removed only from the partition folders to receive new files generated by the UNLOAD operation.
You must have the s3:DeleteObject permission on the Amazon S3 bucket. For information, see Policies and Permissions in Amazon S3 in the Amazon Simple Storage Service Console User Guide. Files that you remove by using the `CLEANPATH` option are permanently deleted and can't be recovered.
You can't specify the `CLEANPATH` option if you specify the `ALLOWOVERWRITE` option.