ObserverManager: fix Collection was modified

[ScarletKuro]

If you add a subscription while Notify is in process of enumerating the observers it can lead to exception: System.InvalidOperationException Collection was modified; enumeration operation may not execute.

This problem was also mentioned here #8284 (comment) in 3rd paragraph

This can be checked with a simple unit test (without grains):

[Fact]
public async Task CollectionModified()
{
	// Arrange
	var observerManager = new ObserverManager<int, int>(TimeSpan.FromHours(1), NullLogger.Instance);

	for (var i = 0; i < 1000; i++)
	{
		observerManager.Subscribe(i, i);
	}

	bool Predicate(int observer)
	{
		// Using predicate for an easier simulation of inserting item while enumerating
		if (observer == 500)
		{
			observerManager.Subscribe(1001, 1001);
		}

		return true;
	}

	Task NotificationAsync(int observer) => Task.CompletedTask;

	 // Act
 	var ex = await Record.ExceptionAsync(()=> observerManager.Notify(NotificationAsync, Predicate));

	 // Assert
 	Assert.Null(ex);
}

I did not add this to the current tests, since we don't really have any dedicated ObserverManager tests and it's unclear in what namespace to add it.

I'm not fan that we are creating a new list, but this is the simplest way to fix it, I guess.

Microsoft Reviewers: Open in CodeFlow

[ReubenBond]

This is an expensive approach. I wonder if we can come up with a cheaper one, possibly at the expense of complexity

[ScarletKuro]

Using ConcurrentDictionary can also fix this, but it's probably even more costly than creating a copy list in Notify.

[KSemenenko]

maybe https://learn.microsoft.com/en-us/dotnet/api/system.collections.frozen.frozenset?view=net-8.0 is good option here?

[KSemenenko]

Using ConcurrentDictionary can also fix this, but it's probably even more costly than creating a copy list in Notify.

we need some BenchmarkDotNet tests to check performance.

Apparently, we can use lock here, but the question is about the 'notify' method, which is asynchronous and can take a long time to execute.
Maybe this method should be run somewhere in a parallel thread?
Or to make a simple lock to make a list of Tasks and then run them in batches?

[slawomirpiotrowski]

I think it's not that easy to come up with a benchmark as relationship beetween amount of reads and writes would change the result.

Speaking about complexity: it might be efficient to cache ToList result and invalidate that cache whenever any writes to the dictionary are applied.

BUT It would be nice to commit any fix for the error if finding out the fastest solution is not that easy. Any fix is better than current (throwing exceptions) code.

[jamescarter-le]

This is still occurring in Orleans v8.1

[ScarletKuro]

Benchmark, copying the observers vs ConcurrentDictionary:

| Method                | NumberOfObservers | Mean         | Error       | StdDev      | Rank | Gen0    | Gen1    | Gen2    | Allocated |
|---------------------- |------------------ |-------------:|------------:|------------:|-----:|--------:|--------:|--------:|----------:|
| NotifyAsyncList       | 100               |     698.2 ns |     7.65 ns |     7.16 ns |    1 |  0.0315 |       - |       - |    1624 B |
| NotifyAsyncConcurrent | 100               |   1,122.6 ns |     9.34 ns |     8.28 ns |    2 |       - |       - |       - |      64 B |
| NotifyAsyncList       | 1000              |   6,849.2 ns |   120.44 ns |   152.32 ns |    3 |  0.3128 |       - |       - |   16024 B |
| NotifyAsyncConcurrent | 1000              |  10,430.6 ns |    34.06 ns |    28.44 ns |    4 |       - |       - |       - |      64 B |
| NotifyAsyncConcurrent | 10000             | 103,772.6 ns |   812.36 ns |   759.89 ns |    5 |       - |       - |       - |      64 B |
| NotifyAsyncList       | 10000             | 132,063.2 ns | 1,317.01 ns | 1,231.94 ns |    6 | 49.8047 | 49.8047 | 49.8047 |  160041 B |