gitlab 作业失败 - 图像拉取失败

Question

我正在尝试使用 Trivy 进行 docker 扫描并将其集成到 GitLab 中，管道已通过。\n但是作业失败了，不确定作业失败的原因。\ndocker 映像有效。\n启用共享运行程序后更新了新错误

\n

gitlab.yml

\n

Trivy_container_scanning:\n  stage: test\n  image: docker:stable-git\n  variables:\n    # Override the GIT_STRATEGY variable in your `.gitlab-ci.yml` file and set it to `fetch` if you want to provide a `clair-whitelist.yml`\n    # file. See https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html#overriding-the-container-scanning-template\n    # for details\n    GIT_STRATEGY: none\n    IMAGE: "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"\n  allow_failure: true\n  before_script:\n    - export TRIVY_VERSION=${TRIVY_VERSION:-v0.20.0}\n    - apk add --no-cache curl docker-cli\n    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY\n    - curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin ${TRIVY_VERSION}\n    - curl -sSL -o /tmp/trivy-gitlab.tpl https://github.com/aquasecurity/trivy/raw/${TRIVY_VERSION}/contrib/gitlab.tpl\n  script:\n    - trivy --exit-code 0 --cache-dir .trivycache/ --no-progress --format template --template "@/tmp/trivy-gitlab.tpl" -o gl-container-scanning-report.json $IMAGE\n    #- ./trivy \xe2\x80\x94 exit-code 0 \xe2\x80\x94 severity HIGH \xe2\x80\x94 no-progress \xe2\x80\x94 auto-refresh trivy-ci-test\n    #- ./trivy \xe2\x80\x94 exit-code 1 \xe2\x80\x94 severity CRITICAL \xe2\x80\x94 no-progress \xe2\x80\x94 auto-refresh trivy-ci-test\n\n  cache:\n    paths:\n      - .trivycache/\n  artifacts:\n    reports:\n      container_scanning: gl-container-scanning-report.json\n  dependencies: []\n  only:\n    refs:\n      - branches\n

\n

Dockerfile

\n

FROM composer:1.7.2\nRUN git clone https://github.com/aquasecurity/trivy-ci-test.git && cd trivy-ci-test && rm Cargo.lock && rm Pipfile.lock\nCMD apk add \xe2\x80\x94 no-cache mysql-client\nENTRYPOINT [\xe2\x80\x9cmysql\xe2\x80\x9d]\n

\n

作业错误：

\n

Running with gitlab-runner 13.2.4 (264446b2)\n  on gitlab-runner-gitlab-runner-76f48bbd84-8sc2l GCJviaG2\nPreparing the "kubernetes" executor\n30:00\nUsing Kubernetes namespace: gitlab-managed-apps\nUsing Kubernetes executor with image docker:stable-git ...\nPreparing environment\n30:18\nWaiting for pod gitlab-managed-apps/runner-gcjviag2-project-1020-concurrent-0pgp84 to be running, status is Pending\nWaiting for pod gitlab-managed-apps/runner-gcjviag2-project-1020-concurrent-0pgp84 to be running, status is Pending\nWaiting for pod gitlab-managed-apps/runner-gcjviag2-project-1020-concurrent-0pgp84 to be running, status is Pending\nWaiting for pod gitlab-managed-apps/runner-gcjviag2-project-1020-concurrent-0pgp84 to be running, status is Pending\nWaiting for pod gitlab-managed-apps/runner-gcjviag2-project-1020-concurrent-0pgp84 to be running, status is Pending\nWaiting for pod gitlab-managed-apps/runner-gcjviag2-project-1020-concurrent-0pgp84 to be running, status is Pending\nERROR: Job failed (system failure): prepare environment: image pull failed: Back-off pulling image "docker:stable-git". Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information\n

\n

另一个错误：

\n

Running with gitlab-runner 13.2.4 (264446b2)\n  on gitlab-runner-gitlab-runner-76f48bbd84-8sc2l GCJviaG2\nPreparing the "kubernetes" executor\n30:00\nUsing Kubernetes namespace: gitlab-managed-apps\nUsing Kubernetes executor with image $CI_REGISTRY/devops/docker-alpine-sdk:19.03.15 ...\nPreparing environment\n30:03\nWaiting for pod gitlab-managed-apps/runner-gcjviag2-project-1020-concurrent-0t7plc to be running, status is Pending\nERROR: Job failed (system failure): prepare environment: image pull failed: Failed to apply default image tag "/devops/docker-alpine-sdk:19.03.15": couldn't parse image reference "/devops/docker-alpine-sdk:19.03.15": invalid reference format. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information\n

\n

Answer 1

接下来是gitlab-org/gitlab-runner 问题 27664

\n

\n
• 要么是GitLab 基础设施问题
\n
• 或（刘布鲁斯评论）
\n

\n

\n

经过一番尝试和错误后，我和我们的团队发现问题是由于运行程序未能使用服务帐户密钥来提取图像造成的。
\n为了解决这个问题，我们使用自定义配置来指定image_pull_secrets格式.dockercfg，以便成功提取图像。

\n

runner-custom-config-map 的内容：

\n

\n

kind: ConfigMap\napiVersion: v1\nmetadata:\n  name: runner-custom-config-map\n  namespace: runner-namespace\ndata:\n  config.toml: |-\n    [[runners]]\n      [runners.kubernetes]\n        image_pull_secrets = ["secret_to_docker_cfg_file_with_sa_token"]\n

\n

\n

用于流道操作符规范：

\n

\n

spec:\n  concurrent: 1\n  config: runner-custom-config-map\n  gitlabUrl: \'https://example.gitlab.com\'\n  imagePullPolicy: Always\n  serviceaccount: kubernetes-service-account\n  token: gitlab-runner-registration-secret\n

\n

和secret_to_docker_cfg_file_with_sa_token：

\n

kind: Secret\napiVersion: v1\n  name: secret_to_docker_cfg_file_with_sa_token\n  namespace: plt-gitlab-runners\ndata:\n  .dockercfg: >-\n    __docker_cfg_file_with_pull_token__\ntype: kubernetes.io/dockercfg\n

\n\n

2022 年 6 月：该问题已由GitLab 15.0 的MR 3399
关闭： \n“在创建 pod 之前检查 serviceaccount 和 imagepullsecret 可用性”

\n

\n

防止在所需资源不可用时创建 pod。

\n

\n\n

2023 年 5 月：问题 27664添加了 ( Gabriel D\xc3\xadaz )：

\n

\n

经过很长时间终于发现了这背后的原因。
\n跑步者只是在跑步者注册期间缓存imagepullsecrets。无论如何，即使您重新启动运行程序部署或终止运行程序 Pod。
\n您可以做的唯一选择是完全取消注册运行器，提供新令牌，然后再次注册：

\n

\n

如果这样做，跑步者会立即读取新的imagepullsecrets更改。
\n此行为是否有记录？这让我发疯。我希望运行程序在每次管道运行时读取秘密值imagepullsecrets，但它只在注册期间读取它。

\n

\n

因此，问题 31066：“适用于 Kubernetes 的 GitLab 运行程序仅imagepullsecrets在运行程序注册期间读取，而不是在 pod/部署重新启动时读取，也不在新管道运行时读取”。

\n