gRPC: Random CANCELLED exception on RPC calls

Question

I'm occasionally getting cancellation errors when calling gRPC methods.

Here's my client-side code (Using grpc-java 1.22.0 library):

public class MyClient {
    private static final Logger logger = LoggerFactory.getLogger(MyClient.class);
    private ManagedChannel channel; 
    private FooGrpc.FooStub fooStub;

    private final StreamObserver<Empty> responseObserver = new StreamObserver<>() {
        @Override
        public void onNext(Empty value) {
        }

        @Override
        public void onError(Throwable t) {
            logger.error("Error: ", t);
        }

        @Override
        public void onCompleted() {
        }
    };

    public MyClient() {
        this.channel = NettyChannelBuilder
            .forAddress(host, port)
            .sslContext(GrpcSslContexts.forClient().trustManager(certStream).build())
            .build();
        var pool = Executors.newCachedThreadPool(
                new ThreadFactoryBuilder().setNameFormat("foo-pool-%d").build());
        this.fooStub = FooGrpc.newStub(channel)
                .withExecutor(pool);
    }

    public void callFoo() {
        fooStub.withDeadlineAfter(500L, TimeUnit.MILLISECONDS)
                .myMethod(whatever, responseObserver);
    }
}

When I invoke callFoo() method, it usually works. Client sends a message and server receives it without problem.

But this call occasionally gives me an error:

io.grpc.StatusRuntimeException: CANCELLED: io.grpc.Context was cancelled without error
        at io.grpc.Status.asRuntimeException(Status.java:533) ~[handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:442) [handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) [handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) [handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) [handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700) [handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) [handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) [handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) [handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399) [handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:507) [handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66) [handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:627) [handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:515) [handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:686) [handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:675) [handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [handler-0.0.1-SNAPSHOT.jar:?]
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [handler-0.0.1-SNAPSHOT.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]

The weird thing is that even though the call gives an error at client side, the server does receive the request, mostly. But sometimes the server misses it.

甚至也不DEADLINE_EXCEEDED例外，它只是抛出CANCELLED: io.grpc.Context was cancelled without error。没有提供其他描述，所以我无法弄清楚为什么会这样。

总结一下：

1. 来自客户端的gRPC调用随机产生CANCELLED错误。
2. 当错误发生时，服务器有时会收到呼叫，但有时不会。

Answer 1

grpc-java支持自动期限和取消传播。当入站RPC导致出站RPC时，这些出站RPC会继承入站RPC的期限。另外，如果取消了入站RPC，则出站RPC也将被取消。

这是通过实现的io.grpc.Context。如果您执行的出站RPC的寿命要长于入站RPC，则应使用Context.fork()。

public void myRpcMethod(Request req, StreamObserver<Response> observer) {
  // ctx has all the values as the current context, but
  // won't be cancelled
  Context ctx = Context.current().fork();
  // Set ctx as the current context within the Runnable
  ctx.run(() -> {
    // Can start asynchronous work here that will not
    // be cancelled when myRpcMethod returns
  });
  observer.onNext(generateReply());
  observer.onCompleted();
}