Error with StatusCode Internal thrown when a server is unavailable for a client running on .NET Framework 4.8 instead of Unavailable

[SayakMukhopadhyay]

What version of gRPC and what language are you using?

Grpc.Net.Client 2.67.0 and C#

What operating system (Linux, Windows,...) and version?

Windows 11

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

.NET Framework 4.8

What did you do?

I have a GRPC server written in Java Spring Boot. I want to ensure proper retries if the server is unavailable. From what I understand, the client will retry if a retryable status code is returned. I would have thought that if the server is unavailable, I should get a status code of StatusCode.Unavailable but instead I get StatusCode.Internal. Setting StatusCode.Internal as a retryable status code works so far as the retries are concerned but that error is thrown for a variety of reasons that I don't want to retry on. Following is how I am setting up the channel

WinHttpHandler winHttpHandler = new WinHttpHandler();
winHttpHandler.ServerCertificateValidationCallback = TlsValidationCallback;

GrpcChannel channel = GrpcChannel.ForAddress("https://localhost:9090", new GrpcChannelOptions
{
    HttpHandler = winHttpHandler,
    ServiceConfig = new ServiceConfig
    {
        MethodConfigs =
        {
            new MethodConfig
            {
                Names = { MethodName.Default }, RetryPolicy =
                    new RetryPolicy
                    {
                        InitialBackoff = TimeSpan.FromSeconds(2),
                        MaxBackoff = TimeSpan.FromSeconds(30),
                        BackoffMultiplier = 1.5,
                        RetryableStatusCodes = { StatusCode.Internal }
                    }
            }
        }
    },
    MaxRetryAttempts = 5
});

Client = new HelloService.HelloServiceClient(channel);

Client.SendMessahe(new HelloRequest { Name = "hello"});

What did you expect to see?

I expected to get a StatusCode.Unavailable when the server is not running.

What did you see instead?

I am getting a StatusCode.Internal instead.

The full error message is

Unhandled Exception: Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. WinHttpException: Error 12029 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'A connection with the server could not be established'.", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request.") ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: Error 12029 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'A connection with the server could not be established'.
   at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
   at System.Net.Http.WinHttpHandler.<StartRequestAsync>d__122.MoveNext()
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Grpc.Net.Client.Internal.GrpcCall`2.<RunCall>d__82.MoveNext() in /_/src/Grpc.Net.Client/Internal/GrpcCall.cs:line 508
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Grpc.Net.Client.Internal.Retry.RetryCallBase`2.<GetResponseCoreAsync>d__80.MoveNext() in /_/src/Grpc.Net.Client/Internal/Retry/RetryCallBase.cs:line 119
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Grpc.Net.Client.Internal.HttpClientCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request) in /_/src/Grpc.Net.Client/Internal/HttpClientCallInvoker.cs:line 153
   at Grpc.Core.Interceptors.InterceptingCallInvoker.<BlockingUnaryCall>b__3_0[TRequest,TResponse](TRequest req, ClientInterceptorContext`2 ctx) in /_/src/Grpc.Core.Api/Interceptors/InterceptingCallInvoker.cs:line 53
   at Grpc.Core.ClientBase.ClientBaseConfiguration.ClientBaseConfigurationInterceptor.BlockingUnaryCall[TRequest,TResponse](TRequest request, ClientInterceptorContext`2 context, BlockingUnaryCallContinuation`2 continuation) in /_/src/Grpc.Core.Api/ClientBase.cs:line 205
   at Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request) in /_/src/Grpc.Core.Api/Interceptors/InterceptingCallInvoker.cs:line 50
   at MyRunner.RunnerService.RunnerServiceClient.ReplyRunner(HelloRequest request, CallOptions options) in my-folder\my-runner\MyRunner\obj\Debug\Protos\RunnerGrpc.cs:line 110
   at MyRunner.RunnerService.RunnerServiceClient.ReplyRunner(HelloRequest request, Metadata headers, Nullable`1 deadline, CancellationToken cancellationToken) in my-folder\my-runner\MyRunner\obj\Debug\Protos\RunnerGrpc.cs:line 105
   at MyRunner.RunnerCallbacks.ReplyRunner(RunnerReplyModel RunnerReplyModel) in my-folder\my-runner\MyRunner\RunnerCallbacks.cs:line 17
   at Beezlabs.RPAHive.Lib.RPARunnerTemplate.RunRunner(RunnerExecutionModel RunnerExecutionModel)
   at MyRunner.Program.<Main>d__0.MoveNext() in my-folder\my-runner\MyRunner\Program.cs:line 42
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at MyRunner.Program.<Main>(String[] args)

Anything else we should know about your project / environment?

TlsValidationCallback is a function that I am using to validate a custom certificate which is why managing my own WinHttpHandler is neded.

[JamesNK]

I don't think there is a way to avoid this. Perhaps looking at the internal exception type name (WinHttpHandler) and then parsing the exception message, but that is too easly broken. I think it's just a limitation when using WinHttpHandler.

[SayakMukhopadhyay]

Yeah, that is what I was afraid of. What's more, I was trying the following

try
{
     Client.SendMessage(new HelloRequest { Name = "hello"});
}
catch (Exception ex)
{
      if (ex.InnerException?.InnerException.Message.Contains(
            "A connection with the server could not be established") == true)
      {
         throw new RpcException(new Status(StatusCode.Unavailable, "oof"), ex.Message);
      }
}

to try and force throw an RPCException but this too causes an Internal status code.