我正在使用docker-compose来设置可扩展的气流群集.我的解决方案基于这个Dockerfile https://hub.docker.com/r/puckel/docker-airflow/
我的问题是将日志设置为从s3写入/读取.当一个dag完成后,我得到这样的错误
*** Log file isn't local.
*** Fetching here: http://ea43d4d49f35:8793/log/xxxxxxx/2017-06-26T11:00:00
*** Failed to fetch log file from worker.
*** Reading remote logs...
Could not read logs from s3://buckets/xxxxxxx/airflow/logs/xxxxxxx/2017-06-
26T11:00:00
我在这个airflow.cfg文件中设置了一个新的部分
[MyS3Conn]
aws_access_key_id = xxxxxxx
aws_secret_access_key = xxxxxxx
aws_default_region = xxxxxxx
然后在远程日志部分中指定s3路径 airflow.cfg
remote_base_log_folder = s3://buckets/xxxx/airflow/logs
remote_log_conn_id = MyS3Conn
我是否正确设置了这个并且有错误？这里有成功的秘诀吗？
- 更新
我尝试以URI和JSON格式导出,似乎都不起作用.然后我导出了aws_access_key_id和aws_secret_access_key,然后气流开始捡起它.现在我在工作日志中得到了他的错误
6/30/2017 6:05:59 PMINFO:root:Using connection to: s3
6/30/2017 6:06:00 PMERROR:root:Could not read logs from s3://buckets/xxxxxx/airflow/logs/xxxxx/2017-06-30T23:45:00
6/30/2017 6:06:00 PMERROR:root:Could not write logs to s3://buckets/xxxxxx/airflow/logs/xxxxx/2017-06-30T23:45:00
6/30/2017 6:06:00 …我在EC2上的docker容器中运行气流.当我尝试访问气流UI中的任务日志时,它将我重定向到localhost,然后我必须用EC2 ip替换localhost以查看日志.这样可行,但如果我不必每次都更改地址会更好.
我的问题是,我可以改变这种行为,如果是,在哪里？
我在这个网站上搜索了一段时间,但我能找到的只是人们对日志本身有问题,但没有这个重定向问题.我以为我会找到一些东西,airflow.cfg但我无法弄清楚这是不是正确的地方.
这是我的airflow.cfg,我可以根据需要发布任何其他文件:
[core]
# The home folder for airflow, default is ~/airflow
airflow_home = /usr/local/airflow
# The folder where your airflow pipelines live, most likely a
# subfolder in a code repository
# This path must be absolute
dags_folder = /usr/local/airflow/dags
# The folder where airflow should store its log files
# This path must be absolute
base_log_folder = /usr/local/airflow/logs
# Airflow can store logs remotely in AWS S3 or Google Cloud Storage. …