Go中如何自定义http.Client或http.Transport超时重试？

Question

我想实现http.Transport标准的自定义http.Client，如果客户端超时，它将自动重试。

PS 出于某种原因，定制http.Transport是必须具备的。我已经检查过hashcorp/go-retryablehttp，但是它不允许我使用我自己的http.Transport.

这是我的尝试，自定义组件：

type CustomTransport struct {
    http.RoundTripper
    // ... private fields
}

func NewCustomTransport(upstream *http.Transport) *CustomTransport {
    upstream.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
    // ... other customizations for transport
    return &CustomTransport{upstream}
}

func (ct *CustomTransport) RoundTrip(req *http.Request) (resp *http.Response, err error) {
    req.Header.Set("Secret", "Blah blah blah")
    // ... other customizations for each request

    for i := 1; i <= 5; i++ {
        resp, err = ct.RoundTripper.RoundTrip(req)
        if errors.Is(err, context.DeadlineExceeded) {
            log.Warnf("#%d got timeout will retry - %v", i, err)
            //time.Sleep(time.Duration(100*i) * time.Millisecond)
            continue
        } else {
            break
        }
    }

    log.Debugf("got final result: %v", err)
    return resp, err
}

调用者代码：

func main() {
    transport := NewCustomTransport(http.DefaultTransport.(*http.Transport))
    client := &http.Client{
        Timeout:   8 * time.Second,
        Transport: transport,
    }

    apiUrl := "https://httpbin.org/delay/10"

    log.Debugf("begin to get %q", apiUrl)
    start := time.Now()
    resp, err := client.Get(apiUrl)
    if err != nil {
        log.Warnf("client got error: %v", err)
    } else {
        defer resp.Body.Close()
    }
    log.Debugf("end to get %q, time cost: %v", apiUrl, time.Since(start))

    if resp != nil {
        data, err := httputil.DumpResponse(resp, true)
        if err != nil {
            log.Warnf("fail to dump resp: %v", err)
        }
        fmt.Println(string(data))
    }
}

我的实现没有按预期工作，一旦客户端超时，重试实际上不会发生。请参阅下面的日志：

2020-07-15T00:53:22.586 DEBUG   begin to get "https://httpbin.org/delay/10"
2020-07-15T00:53:30.590 WARN    #1 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 WARN    #2 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 WARN    #3 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 WARN    #4 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 WARN    #5 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 DEBUG   got final result: context deadline exceeded
2020-07-15T00:53:30.590 WARN    client got error: Get "https://httpbin.org/delay/10": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2020-07-15T00:53:30.590 DEBUG   end to get "https://httpbin.org/delay/10", time cost: 8.004182786s

您能告诉我如何解决这个问题，或者有任何方法/想法来实现这样的吗http.Client？

Answer 1

\n

请注意，http.Client 的 Timeout 字段或多或少已经过时。现在的最佳实践是使用 http.Request.Context() 进行超时。\xe2\x80\x93 脆弱

\n

\n

感谢@Flimzy 的启发！我尝试使用上下文进行超时控制而不是 http.Client 方式。这是代码：

\n

func (ct *CustomTransport) RoundTrip(req *http.Request) (resp *http.Response, err error) {\n    req.Header.Set("Secret", "Blah blah blah")\n    // ... other customizations for each request\n\n    for i := 1; i <= 5; i++ {\n        ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)\n        defer cancel()\n        //reqT := req.WithContext(ctx)\n        resp, err = ct.RoundTripper.RoundTrip(req.WithContext(ctx))\n        if errors.Is(err, context.DeadlineExceeded) {\n            log.Warnf("#%d got timeout will retry - %v", i, err)\n            //time.Sleep(time.Duration(100*i) * time.Millisecond)\n            continue\n        } else {\n            break\n        }\n    }\n

\n

根据日志，它有效（注意日志中的时间戳，它实际上重试了）：

\n

2020-07-16T00:06:12.788+0800    DEBUG   begin to get "https://httpbin.org/delay/10"\n2020-07-16T00:06:20.794+0800    WARN    #1 got timeout will retry - context deadline exceeded\n2020-07-16T00:06:28.794+0800    WARN    #2 got timeout will retry - context deadline exceeded\n2020-07-16T00:06:36.799+0800    WARN    #3 got timeout will retry - context deadline exceeded\n2020-07-16T00:06:44.803+0800    WARN    #4 got timeout will retry - context deadline exceeded\n2020-07-16T00:06:52.809+0800    WARN    #5 got timeout will retry - context deadline exceeded\n2020-07-16T00:06:52.809+0800    DEBUG   got final result: context deadline exceeded\n2020-07-16T00:06:52.809+0800    WARN    client got error: Get "https://httpbin.org/delay/10": context deadline exceeded\n2020-07-16T00:06:52.809+0800    DEBUG   end to get "https://httpbin.org/delay/10", time cost: 40.019334668s\n\n

\n