Fuzzing golang image (Go) project with sydr fuzz (go fuzz backend)
In this article we will try to apply hybrid fuzzing tool sydr-fuzz for fuzzing Go projects. Sydr-fuzz combines the power of Sydr - dynamic symbolic execution tool and AFLplusplus. It also supports another fuzzing engine libFuzzer, which we will use. We are lucky, go-fuzz has libFuzzer support, so we could try to use it as libFuzzer engine in sydr-fuzz. In this guide we will focus on preparing targets for go-fuzz(libFuzzer), Sydr, and code coverage. We will do hybrid fuzzing using Sydr & go-fuzz and then we will collect code coverage. Also, we will use our crash triage tool casr, and apply Sydr to check security predicates for finding interesting bugs using symbolic execution techniques. And of course, I'll focus on some interesting cases that I met during my experience in fuzzing Go projects.
Image decoders such as golang/image is a nice target to apply fuzzing. Before I start, there is already prepared for building docker container with all fuzz environment: targets for go-fuzz, Sydr, and code coverage.
go-fuzz has detailed instruction how to build fuzz targets and start fuzzing. Also there is a nice go-fuzz-corpus repository where you can find fuzz targets and initial corpus. In this repository there are fuzz targets for golang/image project (png, webp, tiff, jpeg, etc.). According to go-fuzz project manual we could just build fuzz target from go-fuzz-corpus repository, but let's just borrow fuzz targets and pretend that we write them from scratch.
Okay, let's write fuzz target for webp decoder. First, I clone golang/image repository and create fuzz.go file there with the following contents:
package image
import (
"bytes"
"golang.org/x/image/webp"
)
func FuzzWebp(data []byte) int {
cfg, err := webp.DecodeConfig(bytes.NewReader(data))
if err != nil {
return 0
}
if cfg.Width*cfg.Height > 4000000 {
return 0
}
if _, err := webp.Decode(bytes.NewReader(data)); err != nil {
return 0
}
return 1
}I slightly modified the fuzz target using image-rs webp target:
if cfg.Width*cfg.Height > 4000000 { // originally 1e6To build fuzz target we can use just use go-fuzz-build with -libfuzzer option:
go-fuzz-build -libfuzzer -func=FuzzWebp -o webp.a
clang -fsanitize=fuzzer webp.a -o fuzz_webpGreat, now we have fuzz target for Go project that looks and works like libFuzzer fuzz target. Let's build a target for DSE tool (Sydr).
For this purpose we will create a binary that reads input file and calls FuzzWebp function. Let's create a file cmd/sydr_webp/main.go with the following contents:
package main
import (
"os"
"golang.org/x/image"
)
func main() {
data, _ := os.ReadFile(os.Args[1])
image.FuzzWebp(data)
}To build target for Sydr (dse tool) we need to do the following:
cd cmd/sydr_webp && go buildNice, we've got sydr_webp binary for Sydr. It remains for us to build the binary for collecting coverage. It's took a lot of time for me to investigate how to do it. We don't have nice options such as: "-C instrument-coverage" for Rust, or "-fprofile-instr-generate -fcoverage-mapping" for C/C++. In OSS-Fuzz is done a lot work for that, but I can't use their work outside OSS-Fuzz environment. I found some nice hack suitable for me and hybrid fuzzing approach that I use. Let's talk about it later in Coverage section. Before we start fuzzing let's build docker container first.
We are going to start hybrid fuzzing using sydr-fuzz with Sydr & go-fuzz. Here is the configuration file for sydr-fuzz:
[sydr]
target = "/image/cmd/sydr_webp/sydr_webp @@"
[libfuzzer]
path = "/image/fuzz_webp"
args = "-dict=/webp.dict -rss_limit_mb=8192 /go-fuzz-corpus/webp/corpus"Let's start fuzzing:
# sydr-fuzz -c webp.toml run
According to logs after 6 hours we found a first crash, nice!
[INFO] #5398923 REDUCE ft: 6376 corp: 1094/12459Kb lim: 175231 exec/s: 226 rss: 1516Mb L: 69/175231 MS: 1 EraseBytes-
[INFO] [LIBFUZZER] run time : 0 days, 6 hrs, 36 min, 54 sec
[INFO] [LIBFUZZER] last new find : 0 days, 0 hrs, 0 min, 30 sec
[INFO] #5402010 REDUCE ft: 6376 corp: 1094/12459Kb lim: 175231 exec/s: 226 rss: 1516Mb L: 2450/175231 MS: 5 ChangeByte-ManualDict-InsertRepeatedBytes-EraseBytes-PersAutoDict- DE: "CCIP"-"AIMN"-
[INFO] #5402116 REDUCE ft: 6376 corp: 1094/12459Kb lim: 175231 exec/s: 226 rss: 1516Mb L: 49/175231 MS: 1 EraseBytes-
[INFO] #5402357 REDUCE ft: 6376 corp: 1094/12459Kb lim: 175231 exec/s: 226 rss: 1516Mb L: 56/175231 MS: 1 EraseBytes-
[INFO] [SYDR] execs: 273, sat{opt|sopt|fuzzmem}: 17649{8575|4553|2350}, unsat: 29348, timeout: 94, oom: 1
[INFO] Launching Sydr: "/builds/dse/gitlab-jobs/oss-sydr-fuzz/projects/image-go/sydr/sydr" "--no-console-log" "-o" "/builds/dse/gitlab-jobs/oss-sydr-fuzz/projects/image-go/webp-out/corpus" "--optimistic" "--wait-jobs" "-s" "60" "--fuzzmem" "--fuzzmem-models" "-c" "/builds/dse/gitlab-jobs/oss-sydr-fuzz/projects/image-go/webp-out/sydr/cache" "-m" "8192" "-f" "/builds/dse/gitlab-jobs/oss-sydr-fuzz/projects/image-go/webp-out/sydr/seeds/b75bc8d30b99ba6eb03817c79a0d29010dabd3c6" "--flat" "b75bc8d30b99ba6eb03817c79a0d29010dabd3c6" "--log-file" "/builds/dse/gitlab-jobs/oss-sydr-fuzz/projects/image-go/webp-out/sydr/logs/log_b75bc8d30b99ba6eb03817c79a0d29010dabd3c6.txt" "--stats-file" "/builds/dse/gitlab-jobs/oss-sydr-fuzz/projects/image-go/webp-out/sydr/stats/stats_b75bc8d30b99ba6eb03817c79a0d29010dabd3c6.json" "--" "/image/cmd/sydr_webp/sydr_webp" "/builds/dse/gitlab-jobs/oss-sydr-fuzz/projects/image-go/webp-out/sydr/seeds/b75bc8d30b99ba6eb03817c79a0d29010dabd3c6"
[INFO] #5406157 REDUCE ft: 6376 corp: 1094/12459Kb lim: 175231 exec/s: 226 rss: 1516Mb L: 64/175231 MS: 1 EraseBytes-
[INFO] SUMMARY: libFuzzer: deadly signal /builds/dse/gitlab-jobs/oss-sydr-fuzz/projects/image-go/webp-out/crashes/crash-9c8db1037a649c084d5a92e5dd6bb33332113e8a
Also, you could see, that go-fuzz (libFuzzer) log a slightly differ from original libFuzzer: it doesn't have cov:, only ft: field. Let's wait till fuzzing is finished.
[INFO] [SYDR] execs: 423, sat{opt|sopt|fuzzmem}: 20585{9634|4975|3451}, unsat: 32720, timeout: 176, oom: 3
[INFO] [RESULTS] Fuzzing corpus is saved in /builds/dse/gitlab-jobs/oss-sydr-fuzz/projects/image-go/webp-out/corpus
[INFO] [RESULTS] oom/leak/timeout/crash: 37/0/0/81
[INFO] [RESULTS] Fuzzing results are saved in /builds/dse/gitlab-jobs/oss-sydr-fuzz/projects/image-go/webp-out/crashes
After 11 hours fuzzing has ended and we have found 81 crashes!
Before we go further, let's minimize the input corpus:
# sydr-fuzz -c webp.toml cmin
[INFO] Original fuzzing corpus saved as /builds/dse/gitlab-jobs/oss-sydr-fuzz/projects/image-go/webp-out/corpus-old
[INFO] Minimizing corpus /builds/dse/gitlab-jobs/oss-sydr-fuzz/projects/image-go/webp-out/corpus
[INFO] libFuzzer environment: ASAN_OPTIONS=allocator_may_return_null=1
[INFO] Launching libFuzzer: "/image/fuzz_webp" "-merge=1" "-rss_limit_mb=8192" "-artifact_prefix=/builds/dse/gitlab-jobs/oss-sydr-fuzz/projects/image-go/webp-out/crashes/" "-close_fd_mask=3" "-verbosity=2" "-dict=/webp.dict" "/builds/dse/gitlab-jobs/oss-sydr-fuzz/projects/image-go/webp-out/corpus" "/builds/dse/gitlab-jobs/oss-sydr-fuzz/projects/image-go/webp-out/corpus-old"
[INFO] MERGE-OUTER: 4760 files, 0 in the initial corpus, 0 processed earlier
[INFO] MERGE-OUTER: attempt 1
[INFO] MERGE-OUTER: successful in 1 attempt(s)
[INFO] MERGE-OUTER: the control file has 763040 bytes
[INFO] MERGE-OUTER: consumed 0Mb (32Mb rss) to parse the control file
[INFO] MERGE-OUTER: 831 new files with 6460 new features added; 0 new coverage edges
Minimization narrowed down 4760 files to 831, good. Let's try to check security predicates!
The idea behind security predicates is shortly described in xlnt guide. Security predicates are intended to find integer overflows, oobs, and division's by zero. To check if result is true positive, sydr-fuzz verifies generated inputs on sanitizers (ASAN, UBSAN). Integer overflows often doesn't lead program to crash. So, building target with UBSAN is strongly recommended when you using security predicates. But, unfortunately for Go I haven't find any ability to build target with UBSAN. Also, there is no overflow-checks = true option that we have for Rust projects. Anyway, let's check security predicates, maybe we will get new crashes.
# sydr-fuzz -c web.toml security -j 64
[INFO] [RESULTS] Security predicates results are saved in /fuzz/webp-out/security
[INFO] [RESULTS] Verified errors are saved in /fuzz/webp-out/security-verified
[INFO] [RESULTS] Unique errors are saved in /fuzz/webp-out/security-unique
[INFO] [RESULTS] Security total/verified/unique: 3964/0/0
[INFO] [RESULTS] Unverified intoverflow/bounds/zerodiv/null/negsize: 3938/26/0/0/0
[INFO] [RESULTS] Verified intoverflow/bounds/zerodiv/null/negsize: 0/0/0/0/0
[INFO] [RESULTS] Unique intoverflow/bounds/zerodiv/null/negsize : 0/0/0/0/0
[INFO] [RESULTS] oom/leak/timeout/crash: 37/0/0/81
[INFO] [RESULTS] Crashes are saved in /fuzz/webp-out/crashes
To my regret, no new crashes were found, moving on.
As I told before, collecting coverage is a real pain when you fuzz Go projects using go-fuzz libFuzzer mode. There is a lot of code in OSS-Fuzz to support code coverage collection for go-fuzz libFuzzer, but we need OSS-Fuzz infrastructure (docker images, scripts, etc). Coverage collection works for go tests and for fuzzing with gofuzz. For some period of time I was seeking the solution for my case: I have an input corpus after fuzzing go-fuzz (libFuzzer), I want to collect coverage on this corpus. I noticed, that go-fuzz tool have an interesting option -dumpcover. It updates coverage profile every new input found during fuzzing campaign. What if we use our output corpus from fuzzing with go-fuzz (libFuzzer) as input corpus for go-fuzz and ask to dumpcover? Let's try!
First I'll prepare the initial corpus for go-fuzz in golang/image project root directory:
# cp -r /fuzz/webp-out/corpus /image/corpus
Then I'll build go-fuzz fuzz target:
# go-fuzz-build -func=FuzzWebp -o fuzz_webp.zip
At last, we start fuzzing for coverage:
# go-fuzz -bin=fuzz_webp.zip -dumpcover
2023/03/03 17:08:55 workers: 12, corpus: 831 (3s ago), crashers: 0, restarts: 1/0, execs: 0 (0/sec), cover: 0, uptime: 3s
2023/03/03 17:08:58 workers: 12, corpus: 831 (6s ago), crashers: 0, restarts: 1/21, execs: 42 (7/sec), cover: 1224, uptime: 6s
2023/03/03 17:09:01 workers: 12, corpus: 831 (9s ago), crashers: 0, restarts: 1/23, execs: 276 (31/sec), cover: 1262, uptime: 9s
2023/03/03 17:09:04 workers: 12, corpus: 831 (12s ago), crashers: 0, restarts: 1/67, execs: 815 (68/sec), cover: 1295, uptime: 12s
2023/03/03 17:09:07 workers: 12, corpus: 834 (1s ago), crashers: 0, restarts: 1/153, execs: 1840 (123/sec), cover: 1309, uptime: 15s
2023/03/03 17:09:10 workers: 12, corpus: 834 (4s ago), crashers: 1, restarts: 1/2544, execs: 71235 (3954/sec), cover: 1309, uptime: 18s
2023/03/03 17:09:13 workers: 12, corpus: 834 (7s ago), crashers: 1, restarts: 1/3345, execs: 167258 (7958/sec), cover: 1309, uptime: 21s
2023/03/03 17:09:16 workers: 12, corpus: 834 (10s ago), crashers: 1, restarts: 1/3877, execs: 294689 (12269/sec), cover: 1309, uptime: 24s
2023/03/03 17:09:19 workers: 12, corpus: 834 (13s ago), crashers: 1, restarts: 1/4534, execs: 399020 (14769/sec), cover: 1309, uptime: 27s
2023/03/03 17:09:22 workers: 12, corpus: 834 (16s ago), crashers: 1, restarts: 1/4965, execs: 466755 (15548/sec), cover: 1309, uptime: 30s
2023/03/03 17:09:25 workers: 12, corpus: 834 (19s ago), crashers: 1, restarts: 1/5300, execs: 519472 (15733/sec), cover: 1309, uptime: 33s
2023/03/03 17:09:28 workers: 12, corpus: 834 (22s ago), crashers: 2, restarts: 1/5200, execs: 582438 (16171/sec), cover: 1309, uptime: 36s
2023/03/03 17:09:31 workers: 12, corpus: 834 (25s ago), crashers: 2, restarts: 1/4996, execs: 724489 (18568/sec), cover: 1309, uptime: 39s
2023/03/03 17:09:34 workers: 12, corpus: 834 (28s ago), crashers: 2, restarts: 1/5112, execs: 828161 (19710/sec), cover: 1309, uptime: 42s
2023/03/03 17:09:37 workers: 12, corpus: 834 (31s ago), crashers: 2, restarts: 1/5394, execs: 933312 (20732/sec), cover: 1309, uptime: 45s
2023/03/03 17:09:40 workers: 12, corpus: 834 (34s ago), crashers: 2, restarts: 1/5587, execs: 1033722 (21528/sec), cover: 1309, uptime: 48s
^C2023/03/03 17:09:42 shutting down...
So we could see that go-fuzz imported our corpus (831 files) and created a coverprofile! Nice, let's generate html report.
# go tool cover -html=coverprofile
cover: inconsistent NumStmt: changed from 0 to 1
There is a problem in go-fuzz, when it dumps coverage. Let's try a proposed fix with sed:
# sed -i '/0.0,1.1/d' coverprofile
Running go tool cover again:
# go tool cover -html=coverprofile
HTML output written to /tmp/cover2240572277/coverage.html
Oh my God, we've got the coverage, let's take a look!
For crash triage I use casr via sydr-fuzz casr subcommand:
sydr-fuzz -c webp.toml casr
You can learn more about casr from it's repository or from my other fuzzing tutorial.
Let's look at casr output:
[INFO] Analyzing 81 files...
[INFO] Timeout for target execution is 30 seconds
[INFO] Using 6 threads
[INFO] casr-san: creating ASAN reports...
[INFO] Progress: 19/81
[INFO] Progress: 41/81
[INFO] Progress: 66/81
[INFO] Casr-cluster: deduplication of casr reports...
[INFO] Reports before deduplication: 81; after: 1
[INFO] Copying inputs...
[INFO] casr-gdb: adding crash reports...
[INFO] Using 1 threads
[WARN] casr-gdb: no crash on input /fuzz/webp-out/crashes/crash-003be3ca633b2073c7a6b1c2cae1e72995a0cab3
[INFO] Done!
[INFO] ==> <casr>
[INFO] Crash: /fuzz/webp-out/casr/crash-003be3ca633b2073c7a6b1c2cae1e72995a0cab3
[INFO] casr-san: NOT_EXPLOITABLE: GoPanic: /image/webp/decode.go:157
[INFO] casr-gdb: No crash
[INFO] Similar crashes: 1
[INFO] Cluster summary -> GoPanic: 1
[INFO] SUMMARY -> GoPanic: 1
[INFO] Crashes and Casr reports are saved in /fuzz/webp-out/casr
After deduplication we have only one crash. Let's look at it's casr report.
We can see that panic is occurred on memory allocation of a byte array. Value w*h is not sanitized and controlled by user. I proposed a fix, hope it will be merged.
In this article I tried to shed some light on interesting aspects of fuzzing Go projects. I've showed how to use hybrid fuzzing with sydr-fuzz, minimize corpus, collect code coverage, check security predicates, and triage crashes. Though, Sydr and sydr-fuzz are commercial products, and if you don't have access to them, you could try to do some parts without them.
- Preparing fuzz targets is in the same way for go-fuzz with libFuzzer, and also for other symbolic executors, such as Fuzzolic. Also you could build and use provided docker container for fuzzing.
- I haven't met any integration between libFuzzer based Fuzzer and symbolic executor in open source, so you could do just fuzzing with go-fuzz.
- Checking security predicates is a Sydr feature (paper). Though there are some similar academic papers in the wild.
- To collect coverage after go-fuzz with libFuzzer you could you approach that I described in Coverage section.
- casr is open-sourced! You can use this algorithm for crash triage.
Andrey Fedotov