Google App Engine - 间歇性 502/连接被对等方重置

Question

在 Google App Engine (GAE) Flex 实例上运行 NodeJS 服务器时，我的应用程序中出现间歇性 502 错误。这些请求从未到达我的节点服务，但它们似乎与与对等方重置连接相关的 nginx 日志一致：

[error] 34#34: *25817 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.2.3.4, server: , request: "POST /endpoint HTTP/1.1", upstream: "http://172.17.0.1:8080/endpoint"
[error] 34#34: *27919 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.2.3.4, server: , request: "POST /endpoint HTTP/1.1", upstream: "http://172.17.0.1:8080/endpoint"
[error] 34#34: *28746 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.2.3.4, server: , request: "POST /endpoint HTTP/1.1", upstream: "http://172.17.0.1:8080/endpoint"
[error] 34#34: *28747 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.2.3.4, server: , request: "POST /endpoint HTTP/1.1", upstream: "http://172.17.0.1:8080/endpoint"
[error] 34#34: *24022 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.2.3.4, server: , request: "POST /endpoint HTTP/1.1", upstream: "http://172.17.0.1:8080/endpoint"
[error] 34#34: *29214 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.2.3.4, server: , request: "POST /endpoint HTTP/1.1", upstream: "http://172.17.0.1:8080/endpoint"

什么可能导致这种行为？CPU/内存负载并未接近资源限制，尽管当服务器处于某种负载下时，这种情况似乎确实发生得更频繁。

Answer 1

部署到 Google App Engine 时，负载均衡器被放置在实例前面。此负载平衡器的 HTTP 保持活动设置为600 秒。

\n

然后，负载均衡器连接到实例上的 nginx 服务，该服务使用 650 秒的保持活动时间，甚至在配置中提供了有用的注释，表示需要更高的时间以防止出现竞争情况。

\n

# GCLB uses a 10 minutes keep-alive timeout. Setting it to a bit more here\n# to avoid a race condition between the two timeouts.\nkeepalive_timeout 650;\n

\n

最后，nginx 反向代理到您的节点应用程序，该应用程序使用默认的保持活动时间... 5 秒

\n

这会导致超时之间的竞争条件（废话），您需要将节点服务器的超时设置为高于 650 秒。如果你使用的是expressjs，看起来像这样：

\n

const app = express();\nconst server = app.listen(process.env.PORT);\n\n//nginx uses a 650 second keep-alive timeout on GAE. Setting it to a bit more here to avoid a race condition between the two timeouts.\nserver.keepAliveTimeout = 700000; \n\n//ensure the headersTimeout is set higher than the keepAliveTimeout due to this nodejs regression bug: https://github.com/nodejs/node/issues/27363\nserver.headersTimeout = 701000; \n

\n

您可以查看启用保持活动的 Nginx 上游中的分析\xe2\x80\x98连接重置\xe2\x80\x99 错误，以获取上游服务器需要更大超时的技术（TCP 级别）解释。

\n