[BUG] Azure Service Bus "Unable to cast object of type 'Microsoft.Azure.Amqp.Framing.Accepted'"

[JoeGaggler]

Library name and version

Azure.Messaging.ServiceBus 7.17.5

Describe the bug

I am experiencing the same issue as reported in #14836 - however that issue is locked, and so I am refiling this bug.

Stacktrace:

System.InvalidCastException: Unable to cast object of type 'Microsoft.Azure.Amqp.Framing.Accepted' to type 'Microsoft.Azure.Amqp.Transaction.Declared'.
   at Microsoft.Azure.Amqp.Transaction.Controller.DeclareAsync(CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpTransactionEnlistment.OnCreateAsync(TimeSpan timeout, CancellationToken cancellationToken)
   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout, CancellationToken cancellationToken)
   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpTransactionManager.EnlistAsync(Transaction transaction, AmqpConnectionScope connectionScope, TimeSpan timeout)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.SendBatchInternalAsync(AmqpMessage batchMessage, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.SendBatchInternalAsync(AmqpMessage batchMessage, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<SendAsync>b__24_0>d.MoveNext()
--- End of stack trace from previous location ---

Expected behavior

App code appears to be correct and is not throwing exceptions, so the expectation is that the ServiceBus library not throw either.

Actual behavior

Exception is thrown and caught internally, but operations might not be succeeding.

Reproduction Steps

using (var transactionScope = new TransactionScope(TransactionScopeOption.RequiresNew, TransactionScopeAsyncFlowOption.Enabled))
{
	await receiver.SetSessionStateAsync(binaryData, cancellationToken);
	await serviceBusSender.SendMessageAsync(outMessage, cancellationToken); // THROWS HERE! Note: in and out have same session id
	await receiver.CompleteMessageAsync(inMessage, CancellationToken.None);
	transactionScope.Complete();
}

Environment

Azure Container App
Base Image: mcr.microsoft.com/dotnet/aspnet:8.0

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[JoeGaggler]

We have enabled AzureEventSourceListener, which logs these at the same time as the exceptions:

Forwarding Azure Event: 92, TransactionDischargeException, Azure-Messaging-ServiceBus, transactionId(String)=222d15f3-cb67-441a-a38f-516870a395f6:388, amqpTransactionId(String)=, exception(String)=Azure.Messaging.ServiceBus.ServiceBusException: TransactionID is null or empty Reference:69c73c38-5818-447c-8fdb-ca9f94602a04, TrackingId:781780e6-d9ad-47a2-95ea-dab3b67105aa_G28, SystemTracker:gtm, Timestamp:2024-07-17T16:22:13 (GeneralError). For troubleshooting information, see https://aka.ms/azsdk/net/servicebus/exceptions/troubleshoot.

Forwarding Azure Event: 3, SendMessageException, Azure-Messaging-ServiceBus, identifier(String)={{{REDACTED}}}, exception(String)=System.InvalidCastException: Unable to cast object of type 'Microsoft.Azure.Amqp.Framing.Accepted' to type 'Microsoft.Azure.Amqp.Transaction.Declared'.
   at Microsoft.Azure.Amqp.Transaction.Controller.DeclareAsync(CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpTransactionEnlistment.OnCreateAsync(TimeSpan timeout, CancellationToken cancellationToken)
   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout, CancellationToken cancellationToken)
   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpTransactionManager.EnlistAsync(Transaction transaction, AmqpConnectionScope connectionScope, TimeSpan timeout)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.SendBatchInternalAsync(AmqpMessage batchMessage, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.SendBatchInternalAsync(AmqpMessage batchMessage, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<SendAsync>b__24_0>d.MoveNext()

Forwarding Azure Event: 89, TransactionInitializeException, Azure-Messaging-ServiceBus, transactionId(String)=222d15f3-cb67-441a-a38f-516870a395f6:388, exception(String)=System.InvalidCastException: Unable to cast object of type 'Microsoft.Azure.Amqp.Framing.Accepted' to type 'Microsoft.Azure.Amqp.Transaction.Declared'.
   at Microsoft.Azure.Amqp.Transaction.Controller.DeclareAsync(CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpTransactionEnlistment.OnCreateAsync(TimeSpan timeout, CancellationToken cancellationToken)

[jsquire]

@JoeGaggler: Thanks for reaching out and we regret that you're experiencing difficulties. Looking over the previous issue and implementation, this appears to be something in the AMQP transport library that Service Bus uses as a dependency. I've transferred this over to the repository where the Microsoft.Azure.Amqp library is maintained as #262 so that the folks best able to assist have visibility.