Skip to content

Fuzzing YamlDotNet (C#) project with sydr‐fuzz (AFL and Sharpfuzz backend) (eng)

Jump to bottom
Daniel Kuts edited this page Apr 19, 2024 · 4 revisions

Introduction

This paper will demonstrate an approach to fuzzing C# applications using the Sydr-Fuzz interface based on the Sharpfuzz tool in conjunction with the AFLplusplus fuzzer. Sydr-Fuzz provides an interface for running hybrid fuzzing, using the dynamic symbolic execution capabilities of the Sydr in conjunction with libFuzzer and AFLplusplus fuzzers. In addition to fuzzing, Sydr-Fuzz offers a set of features for minimizing the corpus, collecting coverage, finding bugs in programs by checking security predicates, and analyzing crash severity with Casr. In addition to programs in compiled languages, Sydr-Fuzz supports fuzzing applications in Python, Java, and JavaScript. The next step was to add the ability to fuzz C# code through our Sydr-Fuzz tool. For fuzzing C# applications, the project binary files are first instrumented via Sharpfuzz and then fuzzing is performed via AFLplusplus.

Fuzz-target preparation

For an example of creating a fuzz-target, consider the YamlDotNet project. Instructions for installing and using the tool Sharpfuzz can be found in its GitHub repository, instructions for installing the fuzzer AFLplusplus can also be found in its GitHub repository. In our repository OSS-Sydr-Fuzz we already have the Docker container with a customized environment, which we will use for fuzzing.

The docker container is built and then started with the following commands:

$ docker build -t oss-sydr-fuzz-yamldotnet .
$ docker run --privileged --network host -v /etc/localtime:/etc/localtime:ro --rm -it -v $PWD:/fuzz oss-sydr-fuzz-yamldotnet /bin/bash

Let's have a look at fuzz-target Program.cs:

using SharpFuzz;
using YamlDotNet.Serialization;
using YamlDotNet.Serialization.NamingConventions;
using YamlDotNet.Core;
using YamlDotNet.Core.Events;
using System.IO;
using System.Text;
using YamlDotNet.RepresentationModel;

public class Program
{
    public static void Main(string[] args)
    {
        Fuzzer.OutOfProcess.Run(stream =>
        {
            try {
                string yml = File.ReadAllText(args[0]);
                var input = new StringReader(yml);

                var yaml = new YamlStream();
                var deserializer = new DeserializerBuilder()
                    .WithNamingConvention(CamelCaseNamingConvention.Instance)
                    .Build();
                var serializer = new SerializerBuilder()
                    .JsonCompatible()
                    .Build();

                var doc = deserializer.Deserialize(input);
                var json = serializer.Serialize(doc);
                var parser = new Parser(input);
                parser.Consume<StreamStart>();
                yaml.Load(input);
            }
            catch (YamlException) { }
            catch (System.InvalidOperationException) { }
            catch (System.ArgumentNullException) { }
            catch (System.ArgumentException) { }
        });
    }
}

First, you need to link the Sharpfuzz module to use its interface. Secondly, the necessary project libraries should be added if needed. Next, in the Main function you need to call the Fuzzer.OutOfProcess.Run() method, as a parameter to which we need to pass the function that runs the target of fuzzing. In this case, we pass a lambda-function that calls the fuzz-target functions.

Fuzzing C# code through Sharpfuzz considers searching for unhandled exceptions, so out target has a try-catch block, since we want to catch and ignore exceptions related to incorrect parsing of .yaml files and not to wrong behavior of the program.

It is preferably to build the fuzz-target in a new directory: create a new .NET console application there, copy the Program.cs there and add the Sharpfuzz library:

    $ mkdir build_fuzz && cd build_fuzz
    $ dotnet new console
    $ dotnet add package SharpFuzz
    $ cp -r /path/to/fuzz_target.cs .

Next, to build the fuzz-target, we need to link the fuzzing project module in the .csproj configuration file. To do this, either build the project itself (in the project directory using dotnet build or dotnet publish), find the compiled target_name.dll assembly (usually is located inside the bin/ directory) and specify the path to it in the build_fuzz.csproj file, or you can specify the path to the .csproj file of the project, then the project will be rebuilt automatically when building the fuzz-target. Examples of .csproj configuration:

<ItemGroup>
    <Reference Include="target_name">
      <HintPath>/path/to/bin/target_name.dll</HintPath>
    </Reference>
    <PackageReference Include="SharpFuzz" Version="2.1.1" />
</ItemGroup>

or

<ItemGroup>
    <ProjectReference Include="/path/to/csproj/target_name.csproj" />
    <PackageReference Include="SharpFuzz" Version="2.1.1" />
</ItemGroup>

After that, you need to build the fuzz-target and instrument it for fuzzing using the Sharpfuzz tool. It is strongly recommended to use the -p:CheckForOverfllowUnderflow=true compiler option when building a target. This enables the check of integer overflow in runtime, leading to catching bugs that won't be detected by the fuzzer.

    $ dotnet publish build_fuzz.csproj -c release -o bin -p:CheckForOverfllowUnderflow=true
    $ sharpfuzz bin/target_name.dll

Build is ready! Now we can move on to fuzzing.

Fuzzing preparation

Let's have a look at the YamlDotNet project configuration file parse_yaml.toml:

[sharpfuzz]
args = "-i /corpus -t 10000 -x /yaml.dict"
target = "/build_fuzz/bin/fuzz.dll @@"
casr_bin = "/build_fuzz/bin/release/net8.0/fuzz.dll"
jobs = 2

[cov]
build_dir = "/build_cov"

Here the [sharpfuzz] table is responsible for the configuration of the AFL++ fuzzer launch options. args - arguments, here it is obligatory to specify the path to the corpus (-i /path/to/corpus), target - path to .dll file of the fuzz-target, @@ here denotes that input is specified through the file, without this symbol input will be done through standard input, casr_bin - path to .dll file of the fuzz-target that is not instrumented by Sharpfuzz (this field is mandatory if the sydr-fuzz casr command will be used to analyze crash severity). Finally, jobs is the number of threads in which AFL++ will be run.

Next, let's move on to the [cov] table. It is responsible for the coverage configuration. Here build_dir is the directory where the fuzz-target is located and built.

To build coverage, you need to create a separate directory and copy the fuzz-target (the same one used for fuzzing) there. The .csproj file is configured in the same way as for fuzzing:

    $ mkdir build_cov && cd build_cov
    $ dotnet new console
    $ dotnet add package SharpFuzz
    $ cp -r /path/to/fuzz_target.cs .

Let's look at an example of build-file to see how you can build the fuzz-target.

# Make directories for fuzzing and coverage.
mkdir -p /build_fuzz /build_cov
cp /Program.cs /fuzz.csproj /build_fuzz
cp /Program.cs /fuzz.csproj /build_cov

# Build target for fuzzing.
cd /build_fuzz
dotnet publish fuzz.csproj -c release -o bin
sharpfuzz bin/YamlDotNet.dll

We have already made the fuzz-target and .csproj configuration file written according to the above rules. We create two directories (for fuzzing and coverage collection) and copy the target and configuration file into them. Then, as shown above, we build the project in the directory for fuzzing and instrument the .dll module of the project. It is not necessary to build the fuzz-target in the coverage directory, it will be built automatically when you start the coverage build via Sydr-Fuzz using dotnet build. If you need to build a project with specific settings, you can safely do it, in this case you will need to build it so that all .dll files will be located in the bin directory.

Fuzzing

Now, let's move on to fuzzing! Let's start Sydr-Fuzz with the following command:

    $ sydr-fuzz -c parse_yaml.toml run
[2024-03-13 17:12:20] [INFO] [AFL++] [*] Fuzzing test case #499 (2207 total, 0 crashes saved, state: in progress, mode=explore, perf_score=229, weight=0, favorite=1, was_fuzzed=1, exec_us=223, hits=32, map=1833, ascii=0, run_time=0:00:10:58)...
[2024-03-13 17:12:20] [INFO] [AFL++] [*] Fuzzing test case #502 (2207 total, 0 crashes saved, state: in progress, mode=explore, perf_score=172, weight=0, favorite=1, was_fuzzed=1, exec_us=187, hits=66, map=1537, ascii=0, run_time=0:00:10:58)...
[2024-03-13 17:12:20] [INFO] [AFL++] [*] Fuzzing test case #506 (2207 total, 0 crashes saved, state: in progress, mode=explore, perf_score=114, weight=0, favorite=1, was_fuzzed=1, exec_us=200, hits=174, map=1566, ascii=0, run_time=0:00:10:58)...
[2024-03-13 17:12:20] [INFO] [AFL++] [*] Fuzzing test case #507 (2207 total, 0 crashes saved, state: in progress, mode=explore, perf_score=114, weight=0, favorite=1, was_fuzzed=1, exec_us=262, hits=231, map=1140, ascii=0, run_time=0:00:10:58)...
[2024-03-13 17:12:20] [INFO] Found crash /fuzz/parse_yaml-out/crashes/crash-48745d7888086b915e559192e1c2dc785db5a1c1
[2024-03-13 17:12:21] [INFO] [AFL++] [*] Fuzzing test case #510 (2207 total, 0 crashes saved, state: in progress, mode=explore, perf_score=114, weight=0, favorite=1, was_fuzzed=1, exec_us=230, hits=155, map=1874, ascii=0, run_time=0:00:10:59)...
[2024-03-13 17:12:21] [INFO] [AFL++] [*] Fuzzing test case #514 (2208 total, 0 crashes saved, state: in progress, mode=explore, perf_score=128, weight=0, favorite=1, was_fuzzed=1, exec_us=182, hits=108, map=921, ascii=0, run_time=0:00:11:00)...
[2024-03-13 17:12:21] [INFO] [AFL++] [*] Fuzzing test case #515 (2210 total, 0 crashes saved, state: in progress, mode=explore, perf_score=128, weight=0, favorite=1, was_fuzzed=1, exec_us=176, hits=97, map=904, ascii=0, run_time=0:00:11:00)...
[2024-03-13 17:12:21] [INFO] [AFL++] [*] Fuzzing test case #516 (2210 total, 0 crashes saved, state: in progress, mode=explore, perf_score=172, weight=0, favorite=1, was_fuzzed=1, exec_us=186, hits=43, map=1558, ascii=0, run_time=0:00:11:00)...
[2024-03-13 17:12:21] [INFO] [AFL++] [*] Fuzzing test case #518 (2210 total, 0 crashes saved, state: in progress, mode=explore, perf_score=114, weight=0, favorite=1, was_fuzzed=1, exec_us=200, hits=160, map=1548, ascii=0, run_time=0:00:11:00)...
[2024-03-13 17:12:21] [INFO] [AFL++] [*] Fuzzing test case #523 (2210 total, 0 crashes saved, state: in progress, mode=explore, perf_score=300, weight=0, favorite=0, was_fuzzed=1, exec_us=194, hits=11, map=1599, ascii=0, run_time=0:00:11:00)...

[2024-03-13 17:12:24] [INFO] [AFL++] 
[2024-03-13 17:12:24] [INFO] [AFL++] +++ Testing aborted by user +++
[2024-03-13 17:12:24] [INFO] [AFL++] [!] 
[2024-03-13 17:12:24] [INFO] [AFL++] Performing final sync, this make take some time ...
[2024-03-13 17:12:24] [INFO] [AFL++] [!] Done!
[2024-03-13 17:12:24] [INFO] [AFL++] [+] We're done here. Have a nice day!
[2024-03-13 17:12:24] [INFO] [AFL++] 
[2024-03-13 17:12:24] [INFO] [RESULTS] Fuzzing corpuses are saved in workers queue directories. Run sydr-fuzz cmin subcommand to gather full corpus at "/fuzz/parse_yaml-out/corpus-old" and minimized corpus at "/fuzz/parse_yaml-out/corpus".
[2024-03-13 17:12:24] [INFO] [RESULTS] [afl_main] Statistics: 1664 new corpus items found, 6.86% coverage achieved, 0 crashes saved, 0 timeouts saved, total runtime 0 days, 0 hrs, 11 min, 3 sec
[2024-03-13 17:12:24] [INFO] [RESULTS] [afl_s01] Statistics: 1563 new corpus items found, 6.99% coverage achieved, 1 crashes saved, 2 timeouts saved, total runtime 0 days, 0 hrs, 11 min, 3 sec
[2024-03-13 17:12:24] [INFO] [RESULTS] timeout/crash: 2/1

After fuzzing is complete, we have over 4000 new files in the corpus as output. It will be a good idea to start minimization of the corpus:

sydr-fuzz -c parse_yaml.toml cmin
[2024-03-13 17:17:14] [INFO] [CMIN] corpus minimization tool for AFL++ (awk version)
[2024-03-13 17:17:14] [INFO] [CMIN] [*] Obtaining traces for 4312 input files in '/fuzz/parse_yaml-out/corpus-old'.
[2024-03-13 17:17:14] [INFO] [CMIN] [*] Creating 8 parallel tasks with about 539 items each.
[2024-03-13 17:17:14] [INFO] [CMIN] [*] Waiting for parallel tasks to complete ...
[2024-03-13 17:17:26] [INFO] [CMIN] [*] Done!
[2024-03-13 17:17:26] [INFO] [CMIN]     Processing file 1/4312
[2024-03-13 17:17:26] [INFO] [CMIN]     Processing file 44/4312
[2024-03-13 17:17:26] [INFO] [CMIN]     Processing file 87/4312
[2024-03-13 17:17:27] [INFO] [CMIN]     Processing file 130/4312
...
[2024-03-13 17:17:46] [INFO] [CMIN]     Processing file 4215/4312
[2024-03-13 17:17:46] [INFO] [CMIN]     Processing file 4258/4312
[2024-03-13 17:17:47] [INFO] [CMIN]     Processing file 4301/4312
[2024-03-13 17:17:49] [INFO] [CMIN] [+] Found 22119 unique tuples across 4312 files.
[2024-03-13 17:17:49] [INFO] [CMIN] [+] Narrowed down to 839 files, saved in '/fuzz/parse_yaml-out/corpus'.

Now our corpus has shrunk by a factor of about 5! Now we can move to the code coverage.

Coverage

We use 2 tools to collect C# code coverage: minicover and AltCover. minicover is used for html, clover, coveralls, xml, opencover, cobertura, text formats; AltCover is used for html or lcov formats. By default AltCover is used (because it allows to collect coverage in parallel mode), but if it is necessary to use minicover, then in the configuration file it is necessary to set the table [cov] in the following form:

[cov]
target = "/build_cov/Program.cs"
source = "/YamlDotNet"
build_dir = "/build_cov"
use_minicover = true

Here target is the path to the source code of the target, ```sourceis the path to the directory with the project's source code,build_dir`` is the directory where the fuzz-target is located and built, ``use_minicover`` is a flag indicating that the minicover tool is used (it's default value is false).

Now let's collect the coverage!

    $ sydr-fuzz -c parse_yaml.toml sharpcov html
[2024-03-13 17:23:13] [INFO] Running sharpcov html "/fuzz/parse_yaml.toml"
[2024-03-13 17:23:13] [INFO] Collecting coverage data for each file in corpus: /fuzz/parse_yaml-out/corpus
[2024-03-13 17:23:13] [INFO] Saving coverage data to /fuzz/parse_yaml-out/coverage/
[2024-03-13 17:23:13] [INFO] Launching dotnet build: cd "/build_cov" && "/usr/bin/dotnet" "build"
[2024-03-13 17:23:20] [INFO] Launching minicover instrumentation: "/root/.dotnet/tools/minicover" "instrument" "--workdir" "/" "--sources" "/build_cov/Program.cs" "--sources" "/YamlDotNet/" "--assemblies" "/build_cov/**/*.dll" "--coverage-file" "/fuzz/parse_yaml-out/coverage/coverage.json" "--hits-directory" "/fuzz/parse_yaml-out/coverage/coverage-hits"
[2024-03-13 17:23:21] [INFO] Launching coverage.
[2024-03-13 17:23:21] [INFO] Collecting coverage: 1/839
[2024-03-13 17:23:22] [INFO] Collecting coverage: 2/839
[2024-03-13 17:23:23] [INFO] Collecting coverage: 3/839
[2024-03-13 17:23:24] [INFO] Collecting coverage: 4/839
...
[2024-03-13 17:33:53] [INFO] Collecting coverage: 836/839
[2024-03-13 17:33:54] [INFO] Collecting coverage: 837/839
[2024-03-13 17:33:54] [INFO] Collecting coverage: 838/839
[2024-03-13 17:33:55] [INFO] Collecting coverage: 839/839
[2024-03-13 17:33:56] [INFO] Launching minicover report to html format: "/root/.dotnet/tools/minicover" "htmlreport" "--workdir" "/" "--coverage-file" "/fuzz/parse_yaml-out/coverage/coverage.json" "--no-fail" "--output" "/fuzz/parse_yaml-out/coverage"
[2024-03-13 17:33:57] [INFO] Coverage collecting to html format is done!

The coverage (in the case of minicover in html format) would look like this:

minicover_html

In case of collecting coverage with the AltCover tool, you can additionally use the reportgenerator tool. It can be used to generate coverage in many formats using .xml file generated after AltCover operation:

    $ reportgenerator -reports:/fuzz/parse_yaml-out/coverage/coverage.xml -reporttypes:Html -targetdir:/fuzz/parse_yaml-out/coverage/html

In the -reporttypes: option, specify the desired format. As a result, the directory /fuzz/parse_yaml-out/coverage/html will contain the coverage report in html format.

Crash Triage

Finally, we can use Casr to analyze crash severity. But before that, you should add a casr_bin field to the configuration file to [sharpfuzz] table, where you should specify the path to the binary file that not instrumented by Sharpfuzz:

[sharpfuzz]
...
target = "/build_fuzz/bin/fuzz.dll @@"
casr_bin = "/build_fuzz/bin/release/net8.0/fuzz.dll"
...

Now let's run a crash analysis using Casr:

    $ sydr-fuzz -c parse_yaml.toml casr

You can learn more about Casr in the [Casr] repository(https://github.com/ispras/casr) or from guide.

The output of the command will be as follows:

[2024-04-10 14:06:14] [INFO] [CASR-AFL] Analyzing 265 files...
[2024-04-10 14:06:14] [INFO] [CASR-AFL] Timeout for target execution is 30 seconds
[2024-04-10 14:06:14] [INFO] [CASR-AFL] Generating CASR reports...
[2024-04-10 14:06:14] [INFO] [CASR-AFL] Using 8 threads
[2024-04-10 14:06:15] [INFO] [CASR-AFL] Progress: 16/265
[2024-04-10 14:06:16] [INFO] [CASR-AFL] Progress: 36/265
...
[2024-04-10 14:06:26] [INFO] [CASR-AFL] Progress: 235/265
[2024-04-10 14:06:27] [INFO] [CASR-AFL] Progress: 256/265
[2024-04-10 14:06:28] [INFO] [CASR-AFL] Deduplicating CASR reports...
[2024-04-10 14:06:29] [INFO] [CASR-AFL] Number of reports before deduplication: 265. Number of reports after deduplication: 8
[2024-04-10 14:06:29] [INFO] [CASR-AFL] Clustering CASR reports...
[2024-04-10 14:06:29] [INFO] [CASR-AFL] Number of clusters: 5
[2024-04-10 14:06:29] [INFO] [CASR-AFL] ==> <cl1>
[2024-04-10 14:06:29] [INFO] [CASR-AFL] Crash: /fuzz/parse_yaml-out/casr/cl1/id:000011,sig:02,src:000333,time:3317,execs:15906,op:havoc,rep:2
[2024-04-10 14:06:29] [INFO] [CASR-AFL]   casrep: NOT_EXPLOITABLE: System.InvalidOperationException: /YamlDotNet/YamlDotNet/Core/Parser.cs:312
[2024-04-10 14:06:29] [INFO] [CASR-AFL]   Similar crashes: 1
[2024-04-10 14:06:29] [INFO] [CASR-AFL] Cluster summary -> System.InvalidOperationException: 1
[2024-04-10 14:06:29] [INFO] [CASR-AFL] ==> <cl2>
[2024-04-10 14:06:29] [INFO] [CASR-AFL] Crash: /fuzz/parse_yaml-out/casr/cl2/id:000054,sig:02,src:000993,time:35940,execs:149891,op:havoc,rep:1
[2024-04-10 14:06:29] [INFO] [CASR-AFL]   casrep: NOT_EXPLOITABLE: System.ArgumentOutOfRangeException: /YamlDotNet/YamlDotNet/Core/Scanner.cs:2001
[2024-04-10 14:06:29] [INFO] [CASR-AFL]   Similar crashes: 1
[2024-04-10 14:06:29] [INFO] [CASR-AFL] Cluster summary -> System.ArgumentOutOfRangeException: 1
[2024-04-10 14:06:29] [INFO] [CASR-AFL] ==> <cl3>
[2024-04-10 14:06:29] [INFO] [CASR-AFL] Crash: /fuzz/parse_yaml-out/casr/cl3/id:000065,sig:02,src:000835,time:69261,execs:299637,op:havoc,rep:2
[2024-04-10 14:06:29] [INFO] [CASR-AFL]   casrep: NOT_EXPLOITABLE: System.ArgumentException: /YamlDotNet/YamlDotNet/Core/TagName.cs:46
[2024-04-10 14:06:29] [INFO] [CASR-AFL]   Similar crashes: 1
[2024-04-10 14:06:29] [INFO] [CASR-AFL] Cluster summary -> System.ArgumentException: 1
[2024-04-10 14:06:29] [INFO] [CASR-AFL] ==> <cl4>
[2024-04-10 14:06:29] [INFO] [CASR-AFL] Crash: /fuzz/parse_yaml-out/casr/cl4/id:000000,sig:02,src:000023,time:198,execs:1077,op:havoc,rep:6
[2024-04-10 14:06:29] [INFO] [CASR-AFL]   casrep: NOT_EXPLOITABLE: System.ArgumentException: /YamlDotNet/YamlDotNet/Core/TagName.cs:51
[2024-04-10 14:06:29] [INFO] [CASR-AFL]   Similar crashes: 2
[2024-04-10 14:06:29] [INFO] [CASR-AFL] Cluster summary -> System.ArgumentException: 2
[2024-04-10 14:06:29] [INFO] [CASR-AFL] ==> <cl5>
[2024-04-10 14:06:29] [INFO] [CASR-AFL] Crash: /fuzz/parse_yaml-out/casr/cl5/id:000000,sig:02,src:000000,time:53,execs:474,op:havoc,rep:2
[2024-04-10 14:06:29] [INFO] [CASR-AFL]   casrep: NOT_EXPLOITABLE: System.ArgumentNullException: /YamlDotNet/YamlDotNet/Core/Tokens/TagDirective.cs:81
[2024-04-10 14:06:29] [INFO] [CASR-AFL]   Similar crashes: 3
[2024-04-10 14:06:29] [INFO] [CASR-AFL] Cluster summary -> System.ArgumentNullException: 3
[2024-04-10 14:06:29] [INFO] [CASR-AFL] SUMMARY -> System.ArgumentException: 3 System.ArgumentNullException: 3 System.ArgumentOutOfRangeException: 1 System.InvalidOperationException: 1
[2024-04-10 14:06:29] [INFO] Crashes and Casr reports are saved in /fuzz/parse_yaml-out/casr

As a result of the work, Casr reduced the number of crashes from 265 to 8! And then split them into 5 clusters. Next, let's consider the report of Casr work on one of the crosses (.casrep file inside the output directory):

Screenshot from 2024-04-10 14-13-05

The report includes crashline, stacktrace, environment information, and more, which will be useful for analyzing bugs!

Conclusion

This paper showed an approach to fuzzing C# code using the Sydr-Fuzz tool. Steps such as fuzzing, case minimization, coverage collection and crash triage were supported, and most importantly, all of these steps can be run quickly and easily through Sydr-Fuzz!

Clone this wiki locally