Instruction tracing

.github/workflows/build.yml

@@ -8,7 +8,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - run: sudo apt-get update
-      - run: sudo apt-get install -y libusb-1.0-0-dev libzmq3-dev meson libsdl2-dev libdwarf-dev libdw-dev libelf-dev libcapstone-dev python3-pip ninja-build protobuf-compiler
+      - run: sudo apt-get install -y libusb-1.0-0-dev libzmq3-dev meson libsdl2-dev libdwarf-dev libdw-dev libelf-dev libcapstone-dev python3-pip ninja-build protobuf-compiler cmake
       - run: sudo pip3 install meson==1.2.0
       - uses: actions/checkout@v3
       - run: meson setup ./build
@@ -27,7 +27,7 @@ jobs:
     name: Build and Test Orbetto on macOS
     runs-on: macos-13
     steps:
-      - run: brew install zmq sdl2 libelf protobuf meson ninja capstone dwarfutils
+      - run: brew install zmq sdl2 libelf protobuf meson ninja capstone dwarfutils cmake
       - uses: actions/checkout@v3
       - run: meson setup ./build
         working-directory: ext/orbetto

.gitignore

@@ -8,6 +8,10 @@ dist/
 /ext/orbetto/subprojects/orbuculum
 /ext/orbetto/subprojects/libdwarf*
 /ext/orbetto/subprojects/packagecache
+/ext/orbetto/subprojects/CRoaring
 /ext/orbetto/*.perf
 /ext/orbetto/*.gz
 /ext/orbetto/*.debug
+/ext/orbetto/*.zip
+/ext/orbetto/*.roar
+/ext/orbetto/trace.swo

ext/orbetto/meson.build

@@ -15,6 +15,17 @@ perfbuf_sources = cpp_gen.process(proto_sources, preserve_path_from: meson.curre
 # Compile and link the protobuf source files
 perfbuf_lib = static_library('perfbuf', sources: perfbuf_sources, dependencies: [dependency('protobuf')])
 
+# Specify the include directory for Roaring Bitmap
+cmake = import('cmake')
+rm = cmake.subproject('croaring')
+# Fetch the dependency object
+message('CMake targets:\n - ' + '\n - '.join(rm.target_list()))
+rm_dep1 = rm.dependency('roaring-headers-cpp')
+rm_dep2 = rm.dependency('roaring-headers')
+rm_dep = rm.dependency('roaring')
+rm_inc = rm.get_variable('roaring_headers_cpp_inc') + rm.get_variable('roaring_headers_inc')
+
+
 # Clone the orbuculum git repository
 orbuculum = subproject('orbuculum')
 liborb = orbuculum.get_variable('liborb')
@@ -24,8 +35,8 @@ git_version_info_h = orbuculum.get_variable('git_version_info_h')
 # Compile and link everything together
 executable('orbetto',
     sources: ['src/orbetto.cpp', 'src/device.cpp', git_version_info_h],
-    include_directories: ['src'] + perfbuf_lib.private_dir_include() + orbinc,
-    dependencies: [dependency('protobuf')] + orbuculum.get_variable('dependencies'),
+    include_directories: ['src'] + perfbuf_lib.private_dir_include() + orbinc + rm_inc,
+    dependencies: [dependency('protobuf')] + orbuculum.get_variable('dependencies') + [rm_dep] + [rm_dep1] + [rm_dep2],
     link_with: [liborb, perfbuf_lib],
     install: true,
 )

ext/orbetto/metrics/METRICS.md

@@ -0,0 +1,54 @@
+# Trace Metrics with Plotly in Dash
+
+As explained in the TRACE.md readme, perfetto is a very useful tool for displaying trace data,
+but is limited in data size and display variability.
+Therefore, as a proof of concept, some metrics are calculated in Python.
+We use the perfbuf file generated by orbetto, the SQL trace processor from perfetto
+and a program counter bitmap generated by CRoaring in orbetto.
+
+## Visualising metrics:
+
+To start the local dash server you need to pass a perfbuf file when analysing itm data or a bitmap for statement data.
+Depending on which argument you pass, different metrics can be displayed. Use -h to see all options.
+If you are using the -diff flag, you will need to provide a second perfbuf file to display the metrics based on the difference between the
+between the two files. This is very useful for comparing different versions or settings between two runs.
+
+```sh
+# cd orbetto/metrics
+# two metrics
+python3 metrics.py -f /path/to/orbetto.perf -wq -dwq
+# difference between two traces on one metric
+python3 metrics.py -f /path/to/orbetto_1.perf -f2 /path/to/orbetto_1.perf -hp -diff
+# Code coverage for pc bitmap (beta not completely accurate)
+python3 metrics.py -bm /path/to/bitmap.roar -elf /path/to/elf_file.elf -cc
+```
+
+Once executed, copy the https address into your browser and use the plotly interface to display the desired plot.
+
+Note: Some metrics take a while to calculate (mainly code coverage).
+
+## Advanced metrics and goal
+
+There are already some simple statistical techniques included that can detect outliers, but the bigger picture of this work
+is to do regression testing and fuzzing with statistics derived from the instruction trace. For example:
+- Regularity of sensor readings [Done].
+- Comm link throughputs
+- Scheduling latency vs timeouts [Done].
+- Thread progress vs semaphores
+- Callstack changes (code coverage) [Done].
+
+All these metrics need to be combined into a large latent space that can be diffed (a kind of distance measure) between different versions of PX4
+to detect bugs in the code before it flies.
+
+## GCOV
+
+There has been a quick investigation to use a more advanced and also tested tool. The most common is gcov, which is a
+test coverage programme from GCC.
+Two files are required for gcov to display code coverage:
+- .gcno, which is an advanced source file you get when compiling with the GCC -ftest-coverage option
+- .gcda, which is created when the GCC -fprofile-arcs option is run.
+
+It should be simple to get the .gcno file as the flag is already a compile option in PX4.
+Theoretically it should be possible to reverse engineer the .gcda file just from the PC values,
+(documentation can be found here: https://github.com/gcc-mirror/gcc/blob/master/gcc/gcov-io.h)
+However, it is unclear how long this will take and how robust it will be.