kubernetes pod 和气流工作日志的 DNS

Question

\xe2\x9e\x9c k get pods -n edna    \nNAME                              READY   STATUS    RESTARTS   AGE\nairflow-79d5f59644-dd4k7          1/1     Running   0          16h\nairflow-worker-67bcf7844b-rq7r8   1/1     Running   0          22h\nbackend-65bcb6546-wvvqj           1/1     Running   0          2d16h\n\n

\n

因此，在airflow-79d5f59644-dd4k7 pod中运行的气流正在尝试从气流工作人员（celery/python，运行一个基于烧瓶的简单Web服务器处理日志）中提取日志，但它不能，因为域名airflow-worker-67bcf7844b- rq7r8未在airflow-79d5f59644-dd4k7内解析

\n

*** Log file does not exist: /usr/local/airflow/logs/hello_world/hello_task/2020-07-14T22:05:12.123747+00:00/1.log\n*** Fetching from: http://airflow-worker-67bcf7844b-rq7r8:8793/log/hello_world/hello_task/2020-07-14T22:05:12.123747+00:00/1.log\n*** Failed to fetch log file from worker. HTTPConnectionPool(host=\'airflow-worker-67bcf7844b-rq7r8\', port=8793): Max retries exceeded with url: /log/hello_world/hello_task/2020-07-14T22:05:12.123747+00:00/1.log (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7fd37d6a9790>: Failed to establish a new connection: [Errno -2] Name or service not known\'))\n\n

\n

我怎样才能做到这一点？

\n

我知道 Airflow 可以远程记录到 s3，但是有没有办法通过 POD 随机 hasotname 路由请求？

\n

我创建了一个 NodeType 服务，但 Airflow 不知道该服务的 DNS 名称，并尝试通过 Airflow Worker 的主机名访问日志（由 Celery 报告）。

\n

\xe2\x9e\x9c k get pods -n edna\nNAME                              READY   STATUS    RESTARTS   AGE\nairflow-79d5f59644-dd4k7          1/1     Running   0          16h\nairflow-worker-67bcf7844b-rq7r8   1/1     Running   0          22h\nbackend-65bcb6546-wvvqj           1/1     Running   0          2d17h\n\nkubectl get pods -n edna -l app=edna-airflow-worker \\                                                                        \n    -o go-template=\'{{range .items}}{{.status.podIP}}{{"\\n"}}{{end}}\'\n\'Tipz:\' kgp -n edna -l app=edna-airflow-worker \\ -o go-template=\'{{range .items}}{{.status.podIP}}{{" "}}{{end}}\'\n10.0.101.120\n

\n

进入airflow-79d5f59644-dd4k7 pod

\n

k exec -ti -n edna airflow-79d5f59644-dd4k7 bash  \n\n  [DEV] airflow-79d5f59644-dd4k7 app # curl -L http://airflow-worker-67bcf7844b-rq7r8:8793/log/hello_world/hello_task/2020-07-14T21:59:01.400678+00:00/1.log\n\ncurl: (6) Could not resolve host: airflow-worker-67bcf7844b-rq7r8; Unknown error\n  [DEV] airflow-79d5f59644-dd4k7 app # curl -L http://10.0.101.120:8793/log/hello_world/hello_task/2020-07-14T21:59:01.400678+00:00/1.log\n[2020-07-14 21:59:07,257] {{taskinstance.py:669}} INFO - Dependencies all met for <TaskInstance: hello_world.hello_task 2020-07-14T21:59:01.400678+00:00 [queued]>\n[2020-07-14 21:59:07,341] {{taskinstance.py:669}} INFO - Dependencies all met for <TaskInstance: hello_world.hello_task 2020-07-14T21:59:01.400678+00:00 [queued]>\n[2020-07-14 21:59:07,342] {{taskinstance.py:879}} INFO - \n--------------------------------------------------------------------------------\n[2020-07-14 21:59:07,342] {{taskinstance.py:880}} INFO - Starting attempt 1 of 1\n[2020-07-14 21:59:07,342] {{taskinstance.py:881}} INFO - \n--------------------------------------------------------------------------------\n[2020-07-14 21:59:07,348] {{taskinstance.py:900}} INFO - Executing <Task(PythonOperator): hello_task> on 2020-07-14T21:59:01.400678+00:00\n[2020-07-14 21:59:07,351] {{standard_task_runner.py:53}} INFO - Started process 5795 to run task\n[2020-07-14 21:59:07,912] {{logging_mixin.py:112}} INFO - Running %s on host %s <TaskInstance: hello_world.hello_task 2020-07-14T21:59:01.400678+00:00 [running]> airflow-worker-67bcf7844b-rq7r8\n[2020-07-14 21:59:07,989] {{logging_mixin.py:112}} INFO - Hello world! This is really cool!\n[2020-07-14 21:59:07,989] {{python_operator.py:114}} INFO - Done. Returned value was: Hello world! This is really cool!\n[2020-07-14 21:59:08,161] {{taskinstance.py:1065}} INFO - Marking task as SUCCESS.dag_id=hello_world, task_id=hello_task, execution_date=20200714T215901, start_date=20200714T215907, end_date=20200714T215908\n[2020-07-14 21:59:17,070] {{logging_mixin.py:112}} INFO - [2020-07-14 21:59:17,070] {{local_task_job.py:103}} INFO - Task exited with return code 0\n  [DEV] airflow-79d5f59644-dd4k7 app # \n\n\n

\n

Answer 1

解决方案

为worker pod的deployment.yaml提供以下ENV AIRFLOW__CORE__HOSTNAME_CALLABLE ：

env:
  - name: AIRFLOW__CORE__HOSTNAME_CALLABLE
    value: 'airflow.utils.net:get_host_ip_address'

或者只是更改airflow.cfg

然后气流尝试通过 POD 的 IP 进行访问，并且如果您的 POD 公开端口8793，则一切正常