使用 docker 执行器在 GitlabRunner 中构建 Micronaut docker 镜像

Question

TLDR：如何在 Openjdk 容器内使用 Gradle 构建 docker 镜像？

问题

我们正在使用 GitLab 管道在多模块 Gradle 项目中构建基于Micronaut的微服务套件。

目前，我们的管道使用 docker 命令行在单独的管道阶段构建 docker 镜像，但以这种方式添加新服务开始变得笨拙且麻烦。

因此，我不是使用单独的管道步骤来构建可执行文件，然后为每个可执行文件构建 docker 映像，而是考虑使用 gradle 与主要构建步骤一起构建 docker 映像。

Micronaut 的 gradle 插件包含并扩展了gradle-docker-plugin，并允许使用任务构建 docker 镜像dockerBuild。

整个 Gradle 构建步骤是openjdk:14通过自定义私有gradle-runner实例从 docker 镜像内执行的，因此该容器上没有任何 docker 相关位。

到目前为止我尝试过的

天真的方法

我的第一次尝试是简单地将dockerBuild目标添加到 Gradle 命令行中。该测试按预期失败，堆栈跟踪如下（为了清楚起见，进行了缩写）：

Execution failed for task ':my-service:dockerBuild'.
> com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.HttpHostConnectException: Connect to http://127.0.0.1:2375 [/127.0.0.1] failed: Connection refused
* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':my-service:dockerBuild'.
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:188)
    at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282)
    /.../
    at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
> Task :my-service:dockerBuild FAILED
Building image using context '/builds/my-group/my-project/my-service/build/docker'.
Using Dockerfile '/builds/my-group/my-project/my-service/build/docker/Dockerfile'
Using images 'registry.gitlab.com/my-group/my-project/my-service:1.0.1-SNAPSHOT+4b9f8460.179'.
    at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
    /.../
    at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
Caused by: java.lang.RuntimeException: com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.HttpHostConnectException: Connect to http://127.0.0.1:2375 [/127.0.0.1] failed: Connection refused
    at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:153)
    at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:8)
    at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
    at com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
Caused by: com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.HttpHostConnectException: Connect to http://127.0.0.1:2375 [/127.0.0.1] failed: Connection refused
    at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.socket.PlainConnectionSocketFactory$1.run(PlainConnectionSocketFactory.java:87)
    at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:84)
    /.../
    at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:67)
    at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:149)
    ... 3 more

经过一些初步的思考后，我发现这基本上是由于容器中运行的构建根本无法访问 docker 引擎远程 api 造成的。

Docker 套接字方法

后来用了几个平庸的Google-Fu/var/run/docker.sock应用程序，我找到了映射到容器文件系统的建议。

这促使我修改我的 gitlab-runner 配置：

Execution failed for task ':my-service:dockerBuild'.
> com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.HttpHostConnectException: Connect to http://127.0.0.1:2375 [/127.0.0.1] failed: Connection refused
* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':my-service:dockerBuild'.
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:188)
    at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282)
    /.../
    at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
> Task :my-service:dockerBuild FAILED
Building image using context '/builds/my-group/my-project/my-service/build/docker'.
Using Dockerfile '/builds/my-group/my-project/my-service/build/docker/Dockerfile'
Using images 'registry.gitlab.com/my-group/my-project/my-service:1.0.1-SNAPSHOT+4b9f8460.179'.
    at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
    /.../
    at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
Caused by: java.lang.RuntimeException: com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.HttpHostConnectException: Connect to http://127.0.0.1:2375 [/127.0.0.1] failed: Connection refused
    at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:153)
    at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:8)
    at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
    at com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
Caused by: com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.HttpHostConnectException: Connect to http://127.0.0.1:2375 [/127.0.0.1] failed: Connection refused
    at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.socket.PlainConnectionSocketFactory$1.run(PlainConnectionSocketFactory.java:87)
    at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:84)
    /.../
    at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:67)
    at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:149)
    ... 3 more

使用该配置运行管道产生了以下失败堆栈跟踪：

Execution failed for task ':my-service:dockerBuild'.
> java.io.IOException: native write() failed : Connection reset by peer
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':my-service:dockerBuild'.
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:188)
    at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:186)
    [...]
    at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
Caused by: java.lang.RuntimeException: java.io.IOException: native write() failed : Connection reset by peer
    at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:153)
    at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:8)
    at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
    at com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
Caused by: java.io.IOException: native write() failed : Connection reset by peer
    at com.github.dockerjava.httpclient5.UnixDomainSocket$UnixSocketOutputStream.write(UnixDomainSocket.java:319)
    at com.bmuschko.gradle.docker.shaded.org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:117)
    [...]
    at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:67)
    at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:149)
    ... 3 more
    Suppressed: java.io.IOException: native write() failed : Broken pipe
        at com.github.dockerjava.httpclient5.UnixDomainSocket$UnixSocketOutputStream.write(UnixDomainSocket.java:319)
        at com.bmuschko.gradle.docker.shaded.org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:117)
        at com.bmuschko.gradle.docker.shaded.org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flush(SessionOutputBufferImpl.java:126)
        [...]
        at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:8)
        at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
        at com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
        at java.base/java.lang.Thread.run(Thread.java:832)

这对我来说意味着，它尝试通过套接字进行连接，但连接被 docker API 服务器关闭。

据我所知，这个问题可能与 docker api 安全性有关，但所有教程都显示了 dind（Docker in Docker）的用法，这对我的用例并没有真正的帮助。

我还尝试使用特权访问运行容器，并尝试使用附加到构建作业的 dind 服务一次，但这些似乎都不起作用。

问题

因此，伟大而强大的互联网的问题是，是否有一种方法可以在 openjdk 容器内使用 Gradle 并利用 Docker 远程 API 来构建 docker 镜像？

如果是的话，我错过了什么？如果不是，有哪些替代方案（是的，我知道我可能可以创建一个单独的 gitlab shell 运行程序，但我想首先探索 Docker 选项）

Answer 1

我遇到了同样的问题，到目前为止我无法解决它（在 jdk 容器旁边有一个 docker 守护进程，企业限制:/）但是，有一个解决方法，就是使用 gradle- jib-plugin，这将让你为每个模块构建 docker 镜像，因此不需要任何 docker 守护进程（神奇！）