ETL 连接器未在 aws 中加载

Question

我在胶水作业中使用 bigquery 连接器进行 aws胶水。几天前它工作正常，但现在突然出现以下错误：

LAUNCH ERROR | Glue ETL Marketplace - failed to download connector.Please refer logs for details.

以下是我在 cloudwatch 上遇到的完整错误

2021-11-08T11:33:02.045+05:00   Traceback (most recent call last): File "/usr/lib64/python3.7/runpy.py", line 193, in _run_module_as_main

2021-11-08T11:33:02.070+05:00   "__main__", mod_spec) File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/tmp/aws_glue_custom_connector_python/docker/unpack_docker_image.py", line 361, in <module>

2021-11-08T11:33:02.070+05:00   main() File "/tmp/aws_glue_custom_connector_python/docker/unpack_docker_image.py", line 351, in main

2021-11-08T11:33:02.070+05:00   res += download_jars_per_connection(conn, region, endpoint, proxy) File "/tmp/aws_glue_custom_connector_python/docker/unpack_docker_image.py", line 304, in download_jars_per_connection

2021-11-08T11:33:02.070+05:00   download_and_unpack_docker_layer(ecr_url, layer["digest"], dir_prefix, http_header) File "/tmp/aws_glue_custom_connector_python/docker/unpack_docker_image.py", line 168, in download_and_unpack_docker_layer

2021-11-08T11:33:02.070+05:00   layer = send_get_request(layer_url, header) File "/tmp/aws_glue_custom_connector_python/docker/unpack_docker_image.py", line 80, in send_get_request

2021-11-08T11:33:02.070+05:00   

2021-11-08T11:33:02.070+05:00   response.raise_for_status() File "/home/spark/.local/lib/python3.7/site-packages/requests/models.py", line 765, in raise_for_status

2021-11-08T11:33:02.071+05:00   raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request

2021-11-08T11:33:02.119+05:00   Glue ETL Marketplace - failed to download connector, activation script exited with code 

Answer 1

当 Glue 作业尝试使用连接器时，它必须以容器的形式下载连接器。连接器容器可在亚马逊公共 ECR 存储库中找到。要从 AWS 公共存储库中提取容器，我们必须将“AmazonEC2ContainerRegistryFullAccess”策略添加到您的 IAM 角色。我们也可以将访问权限限制为只读。

Answer 2

这很可能是权限问题。我遇到了这个问题，并暂时给予了宽松的权限，这似乎解决了问题。