BPF Conformance

This project measures the conformance of a BPF runtime to the ISA. To measure conformance the BPF runtime under test is built into a plugin process that does:

1. Accept both BPF byte code an initial memory.
2. Execute the byte code.
3. Return the value in %r0 at the end of the execution.

eBPF runtime implementations that are currently measured using this project

1. Linux Kernel via libbpf
2. uBPF
3. eBPF for Windows / bpf2c
4. rbpf
5. Prevail Verifier
6. bpftime / llvmbpf, The result is here

Note: Linux Kernel is treated as the authorative eBPF implementation.

Prerequisites

Ubuntu

sudo apt-get install -y libboost-dev libboost-filesystem-dev libboost-program-options-dev libelf-dev lcov libbpf-dev

MacOS

brew install cmake ninja ccache boost

Building

Run cmake -S . -B build to configure the project, then run cmake --build build to build the project.

Usage

Options:
  --help                         Print help messages
  --test_file_path arg           Path to test file
  --test_file_directory arg      Path to test file directory
  --plugin_path arg              Path to plugin
  --plugin_options arg           Options to pass to plugin
  --list_instructions arg        List instructions used and not used in tests
  --list_used_instructions arg   List instructions used in tests
  --list_unused_instructions arg List instructions not used in tests
  --debug (true|false)           Print debug information
  --xdp_prolog (true|false)      XDP prolog
  --elf (true|false)             ELF format
  --cpu_version arg              CPU version
  --include_regex arg            Include regex
  --exclude_regex arg            Exclude regex

--xdp_prolog true should only be used with the libbpf_plugin.

Using a published package

Select the desired version from bpf_conformance

Assume the package is named: "ghcr.io/alan-jowett/bpf_conformance:main"

docker run --privileged -it --rm ghcr.io/alan-jowett/bpf_conformance:main src/bpf_conformance_runner --test_file_directory tests --plugin_path libbpf_plugin/libbpf_plugin --cpu_version v3

Running the test

Linux (test require Linux kernel BPF support):

cmake --build build --target test --

Note: The libbpf_plugin requires root or BPF permissions.

Using bpf_conformance as a static lib

The BPF Conformance tests can also be used as a static library as part of another tests.

1. Include include/bpf_conformance.h
2. Link against libbpf_conformance.a and boost_filesystem (depending on platform).
3. Invoke bpf_conformance, passing it a list of test files.

Interpreting results

On completion of the test the bpf_conformance tools prints the list of tests that passes/failed and a summary count.

sudo build/src/bpf_conformance --test_file_directory tests --plugin_path build/libbpf_plugin/libbpf_plugin
Test results:
PASS: "tests/add.data"
PASS: "tests/add64.data"
PASS: "tests/alu-arith.data"
PASS: "tests/alu-bit.data"
PASS: "tests/alu64-arith.data"
PASS: "tests/alu64-bit.data"
PASS: "tests/arsh-reg.data"
PASS: "tests/arsh.data"
PASS: "tests/arsh32-high-shift.data"
PASS: "tests/arsh64.data"
PASS: "tests/be16-high.data"
PASS: "tests/be16.data"
PASS: "tests/be32-high.data"
PASS: "tests/be32.data"
PASS: "tests/be64.data"
PASS: "tests/call_unwind_fail.data"
PASS: "tests/div-by-zero-reg.data"
PASS: "tests/div32-high-divisor.data"
PASS: "tests/div32-imm.data"
PASS: "tests/div32-reg.data"
PASS: "tests/div64-by-zero-reg.data"
PASS: "tests/div64-imm.data"
PASS: "tests/div64-reg.data"
PASS: "tests/exit-not-last.data"
PASS: "tests/exit.data"
PASS: "tests/jeq-imm.data"
PASS: "tests/jeq-reg.data"
PASS: "tests/jge-imm.data"
PASS: "tests/jgt-imm.data"
PASS: "tests/jgt-reg.data"
PASS: "tests/jit-bounce.data"
PASS: "tests/jle-imm.data"
PASS: "tests/jle-reg.data"
PASS: "tests/jlt-imm.data"
PASS: "tests/jlt-reg.data"
PASS: "tests/jne-reg.data"
PASS: "tests/jset-imm.data"
PASS: "tests/jset-reg.data"
PASS: "tests/jsge-imm.data"
PASS: "tests/jsge-reg.data"
PASS: "tests/jsgt-imm.data"
PASS: "tests/jsgt-reg.data"
PASS: "tests/jsle-imm.data"
PASS: "tests/jsle-reg.data"
PASS: "tests/jslt-imm.data"
PASS: "tests/jslt-reg.data"
PASS: "tests/lddw.data"
PASS: "tests/lddw2.data"
PASS: "tests/ldxb-all.data"
PASS: "tests/ldxb.data"
PASS: "tests/ldxdw.data"
PASS: "tests/ldxh-all.data"
PASS: "tests/ldxh-all2.data"
PASS: "tests/ldxh-same-reg.data"
PASS: "tests/ldxh.data"
PASS: "tests/ldxw-all.data"
PASS: "tests/ldxw.data"
PASS: "tests/le16.data"
PASS: "tests/le32.data"
PASS: "tests/le64.data"
PASS: "tests/lsh-reg.data"
PASS: "tests/mem-len.data"
PASS: "tests/mod-by-zero-reg.data"
PASS: "tests/mod.data"
PASS: "tests/mod32.data"
PASS: "tests/mod64-by-zero-reg.data"
PASS: "tests/mod64.data"
PASS: "tests/mov.data"
PASS: "tests/mul32-imm.data"
PASS: "tests/mul32-reg-overflow.data"
PASS: "tests/mul32-reg.data"
PASS: "tests/mul64-imm.data"
PASS: "tests/mul64-reg.data"
PASS: "tests/neg.data"
PASS: "tests/neg64.data"
PASS: "tests/prime.data"
PASS: "tests/rsh-reg.data"
PASS: "tests/rsh32.data"
PASS: "tests/stack.data"
PASS: "tests/stb.data"
PASS: "tests/stdw.data"
PASS: "tests/sth.data"
PASS: "tests/stw.data"
PASS: "tests/stxb-all.data"
PASS: "tests/stxb-all2.data"
PASS: "tests/stxb-chain.data"
PASS: "tests/stxb.data"
PASS: "tests/stxdw.data"
PASS: "tests/stxh.data"
PASS: "tests/stxw.data"
PASS: "tests/subnet.data"
Passed 91 out of 91 tests.

bpf_conformance_runner plugins

bpf_conformance_runner invokes plugins using command-line execution, and uses stdin, stdout, and stderr to exchange additional information as follows:

<plugin name> [<base16 memory bytes>] [<plugin options>] [--elf]

where:

• <plugin name>: the plugin executable name
• <base16 memory bytes>: if present, the contents of the mem section of a test data file
• <plugin options>: any additional options to pass to the plugin
• --elf: if present, indicates that the data passed to stdin will be in ELF format

The program is then passed to the plugin via stdin, either as raw bytecode or (if --elf is specified) in ELF format. The plugin must then either:

• compute a successful result and output the final contents of r0 in hex (either with or without a leading "0x") to stdout and exit with a status of 0, OR
• output an error message to stderr and exit with a non-zero status.

Additional plugins can be created based on the above specification.