小编Dee*_*mal的帖子

当我在 Airflow 1.10 版中使用KubernetesPodOperator运行docker 镜像时

一旦 pod 成功完成任务，airflow 会尝试通过 k8s 流客户端连接到 pod 来获取 xcom 值。

以下是我遇到的错误：

[2018-12-18 05:29:02,209] {{models.py:1760}} ERROR - (0)
Reason: Handshake status 403 Forbidden
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py", line 249, in websocket_call
    client = WSClient(configuration, get_websocket_url(url), headers)
  File "/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py", line 72, in __init__
    self.sock.connect(url, header=header)
  File "/usr/local/lib/python3.6/site-packages/websocket/_core.py", line 223, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File "/usr/local/lib/python3.6/site-packages/websocket/_handshake.py", line 79, in handshake
    status, resp = _get_resp_headers(sock)
  File "/usr/local/lib/python3.6/site-packages/websocket/_handshake.py", line 152, …

在运行使用
docker镜像运行 jar 的 DAG 时，给出了xcom_push=True，它会在单个 pod 中创建另一个容器以及docker镜像。

达格：

jar_task = KubernetesPodOperator(
    namespace='test',
    image="path to image",
    image_pull_secrets="secret",
    image_pull_policy="Always",
    node_selectors={"d-type":"na-node-group"},
    cmds=["sh","-c",..~running jar here~..],
    secrets=[secret_file],
    env_vars=environment_vars,
    labels={"k8s-app": "airflow"},
    name="airflow-pod",
    config_file=k8s_config_file,
    resources=pod.Resources(request_cpu=0.2,limit_cpu=0.5,request_memory='512Mi',limit_memory='1536Mi'),
    in_cluster=False,
    task_id="run_jar",
    is_delete_operator_pod=True,
    get_logs=True,
    xcom_push=True,
    dag=dag)

下面是JAR执行成功时的错误..

    [2018-11-27 11:37:21,605] {{logging_mixin.py:95}} INFO - [2018-11-27 11:37:21,605] {{pod_launcher.py:166}} INFO - Running command... cat /airflow/xcom/return.json
    [2018-11-27 11:37:21,605] {{logging_mixin.py:95}} INFO - 
    [2018-11-27 11:37:21,647] {{logging_mixin.py:95}} INFO - [2018-11-27 11:37:21,646] {{pod_launcher.py:173}} INFO - cat: can't open '/airflow/xcom/return.json': No such file or directory
    [2018-11-27 …

使用 Airflow worker 和 webserver/scheduler 作为在 EC2 上的 Kubernetes Engine 上运行的 Docker 镜像

我们有一个KubernetesPodOperator资源密集型任务，每 15 分钟运行一次。

在airflow-worker中收到这些错误作为电子邮件

Try 2 out of 3
Exception:
('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
Log: Link
Host: airflow-worker-deployment-123456789
Log file: /usr/local/airflow/logs/DAG_NAME/TASK_NAME/2019-03-14T10:50:00+00:00.log
Mark success: Link

知道它可以是什么吗？