Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a work in progress aimed to add code coverage to the F# backend. It will resolve #3688.
AltCover was chosen as the coverage tool.
Seemingly, all .NET code coverage tools assume that
dotnet test
will run your tests.Expecto is not build to expose tests to
dotnet test
, either plainly or via[<Tests>]
attributes common in .NET, so we needed to tackle this first. The NuGet packagesYoloDev.Expecto.TestSdk
andMicrosoft.NET.Test.Sdk
were added for this effect. After adding those, we also needed 1+[<Tests>]
attributes for the test-runner to discover - I opted to add this to the rootTests
module containing all tests, rather than drop the attribute in many files.[<Tests>]
seems to only be respected when attributing aTest
binding, soTests.fs
had to adjust a bit, in order to expose such in a newtests
binding.With these changes, we can now
dotnet test
within thefsharp-backend
directory, and it runs many of the tests well.With
dotnet test
now discovering and running those tests successfully, the next step was to involve a code coverage tool - AltCover. Altcover is installed as a global tool via thedotnet tool
CLI, installed in the dockerfile.With altcover installed, we can run
dotnet test /p:AltCover=true
.It seems to work at least at a basic level - here are some results:
Some outstanding tasks:
Currently, altcover is likely ensuring that we're testing things like TestUtils, and potentially some external dependencies
Tests
are included in the newtests
bindingSeveral test modules are commented out - some of them fail during
dotnet test
discovery, and others fail duringdotnet test
execution. The ones that fail due in discovery, I forget the details of now; will have to follow up. The ones that fail during execution are likely failing due to the sequenced aspects of those tests not being respected, though there may be additional reasons.currently, the coverage ends up at
fsharp-backend/tests/Tests/coverage
. I imagine we'd want to create a top-levelcoverage
folder, with a sub-folder for pretty .html outputaltcover
NuGet package in addition to the global tool (i.e. can we remove it?)I added this when borrowing code from MiniScaffold, and need to follow up
I took an initial attempt with ReportGenerator, but sticking with AltCover's Visualizer may be our best move here. I haven't yet spent much time here. It may make sense to actually install the visualizer tool on the dev's host machine (outside of container) and then inspect the resultant analysis file, and use the traditional ReportGenerator for CI or other uses.