如何使用 Google Cloud Composer 设置 dbt？

Question

我正在尝试在 Google Cloud Composer 上安装 dbt，但遇到了依赖性问题。我已按照本文中的说明进行操作： https: //blog.doit-intl.com/setup-dbt-with-cloud-composer-ab702454e27b但是在步骤 2：在 Composer 中安装软件包（airflow-dbt 和 dbt），它已经失败了。

我在云构建日志中找到以下内容：

ERROR: snowflake-connector-python 2.3.6 has requirement boto3<1.16,>=1.4.4, but you'll have boto3 1.17.85 which is incompatible.
ERROR: snowflake-connector-python 2.3.6 has requirement requests<2.24.0, but you'll have requests 2.24.0 which is incompatible.
ERROR: networkx 2.5.1 has requirement decorator<5,>=4.3, but you'll have decorator 5.0.9 which is incompatible.
ERROR: hologram 0.0.13 has requirement jsonschema<3.2,>=3.0, but you'll have jsonschema 3.2.0 which is incompatible.
ERROR: dbt-core 0.19.1 has requirement idna<2.10, but you'll have idna 2.10 which is incompatible.
ERROR: dbt-core 0.19.1 has requirement requests<2.24.0,>=2.18.0, but you'll have requests 2.24.0 which is incompatible.
ERROR: dbt-snowflake 0.19.1 has requirement cryptography<4,>=3.2, but you'll have cryptography 3.0 which is incompatible.
ERROR: dbt-bigquery 0.19.1 has requirement google-api-core<1.24,>=1.16.0, but you'll have google-api-core 1.28.0 which is incompatible.
ERROR: dbt-redshift 0.19.1 has requirement boto3<1.16,>=1.4.4, but you'll have boto3 1.17.85 which is incompatible.

我当前的环境配置包含：composer-1.13.0-airflow-1.10.12

有没有人遇到过同样的问题并且您能够解决它吗？我还尝试安装日志中列出的要求的特定版本，但这并不能解决问题。

Answer 1

当您尝试在 Composer 环境中安装 dbt 时，会有点痛苦。但是，有一些解决方法。

1. 使用外部服务运行 dbt 作业，例如 Cloud Run。
2. 使用 Composer 的KubernetesPodOperator。我的同事在这里发表了一篇关于 dbt 讨论的好文章，介绍了设置过程。
3. 通过将 Composer 的环境变量 IGNORE_PYPI_DEPENDENCY_CONFLICTS 设置为 True 来忽略 Composer 的依赖冲突。但是，我不建议这样做，因为它可能会导致潜在的问题。
4. 在 Composer 中创建 Python 虚拟环境并安装 dbt 包。