chore: merge the yocto scripts into yocto manifest

README.md

@@ -35,15 +35,32 @@ setup a Yocto Project build environment for you!
 
 ## Manifest Files
 
-* **default.xml** - External releasable components. Used for release builds.
+This repository contains several manifest files:
+* **tdx-base.xml** - Minimal confidential computing image configuration (uses [tdx-base_yocto_build_config.env](env_files/tdx-base_yocto_build_config.env))
+* **tdx-bob.xml** - Searcher node image configuration (uses [bob_yocto_build_config.env](env_files/bob_yocto_build_config.env))
+* **tdx-rbuilder.xml** - Builder/validator node image configuration (uses [rbuilder_yocto_build_config.env](env_files/rbuilder_yocto_build_config.env))
+
+## Build Profiles
+
+This repository supports multiple build profiles, each tailored for specific use cases and defined by its corresponding manifest file:
+
+* **tdx-base**: Minimal confidential computing image. 
+  See [tdx-base/README.md](config_files/tdx-base/README.md) for details.
+
+* **tdx-bob**: Specialized image for running searcher nodes with podman support and searcher-specific configurations. 
+  See [tdx-bob/README.md](config_files/tdx-bob/README.md) for details.
+
+* **tdx-rbuilder**: Comprehensive environment for running Ethereum validators and builders, including reth and lighthouse clients. 
+  See [tdx-rbuilder/README.md](config_files/tdx-rbuilder/README.md) for details.
+
+Each profile's configuration and setup scripts are maintained in the `config_files/` directory. See the profile-specific READMEs for detailed information about features, requirements, and build configurations.
 
 ## Reproducing image measurements
 
 **Make sure you are checked out on the correct commit (in this repo).**
 
-Building images is as simple as installing docker and running `make measurements-<name>`. For list of images run `make help`.  
+Building images and generating their reproducible build measurements is as simple as installing docker and running `make measurements-<name>`. For list of images run `make help`.  
 Make sure you have plenty of disk space available (200 GBs), and that your session will not time out as a fresh build can take over an hour on slow hardware.  
-To get
 
 Built images as well as measurements will be available in `./reproducible-build/artifacts-<image name>`.  
 
@@ -98,6 +115,7 @@ $ cd yocto/tdx
 * Clone the Yocto meta layer source using yocto manifest as show below.
 ```
 $ repo init -u https://github.com/flashbots/yocto-manifests.git -b main -m <manifest_file>.xml
+# Replace <manifest_file> with tdx-base.xml, tdx-bob.xml, or tdx-rbuilder.xml depending on your needs
 ```
 A successful initialization will end with a message stating that Repo is
 initialized in your working directory. Your directory should now contain a
@@ -165,9 +183,9 @@ $ git clone https://github.com/flashbots/yocto-manifests.git
 ``` 
 2. adjust the yocto build configuration in [env_files](https://github.com/flashbots/yocto-manifests/tree/main/env_files) for the target image you want to build 
 3. make the desired image using, these are the current options: 
-   - image-base: builds a basic tdx-image and outputs the image artifacts in */reproducile-build/artifacts-base*
-   - image-bob: builds the image for the BOB project with the searcher embedded ss-key and podman support and outputs the image artifacts in */reproducile-build/artifacts-bob*
-   - image-rbuilder: builds the image for with rbuilder/reth/lighthouse and outputs the image artifacts in */reproducile-build/artifacts-rbuilder*
+   - image-base: builds a basic tdx-image and outputs the image artifacts in */reproducible-build/artifacts-base* (see [tdx-base profile](config_files/tdx-base/README.md))
+   - image-bob: builds the image for the BOB project with the searcher embedded ss-key and podman support and outputs the image artifacts in */reproducible-build/artifacts-bob* (see [tdx-bob profile](config_files/tdx-bob/README.md))
+   - image-rbuilder: builds the image for with rbuilder/reth/lighthouse and outputs the image artifacts in */reproducible-build/artifacts-rbuilder* (see [tdx-rbuilder profile](config_files/tdx-rbuilder/README.md))
 ```
 $ make image-<target-name>
 ```

config_files/Makefile

@@ -0,0 +1,14 @@
+.PHONY: build gen-measurements
+
+build:
+	cd srcs/poky/ && bitbake cvm-image-azure
+
+gen-measurements:
+	mkdir -p measurements
+	CURRENT_DIR=$(shell pwd) && \
+	IMAGE_DIR=$$CURRENT_DIR/srcs/poky/build/tmp/deploy/images/tdx && \
+	cd srcs/poky/meta-confidential-compute/scripts/measured-boot && \
+	for script in precalculate_pcr*; do \
+		output_file="$$CURRENT_DIR/measurements/$$(basename $$script .sh | sed 's/precalculate_//')_output.json"; \
+		./$$script $$IMAGE_DIR/cvm-image-azure-tdx.rootfs.wic.vhd $$output_file >/dev/null; \
+	done

config_files/tdx-base/README.md

@@ -0,0 +1,36 @@
+# TDX Base Profile
+
+This is the minimal TEE-enabled profile that serves as a foundation for other specialized images. It provides basic AMD SEV-SNP/TDX support with essential security features.
+
+## Features
+- Basic AMD SEV-SNP/TDX support
+- TPM2 integration
+- Minimal system footprint
+
+## Included Layers
+- meta-confidential-compute: Core confidential computing support
+- meta-openembedded: Basic system utilities and libraries
+- poky: Base Yocto distribution
+
+## Build Configuration
+- Image Type: `cvm-image-azure`
+- Package Format: IPK
+- Supported Machine: tdx
+- Distribution: cvm
+
+## Environment Variables
+The following environment variables can be set to customize the build:
+- `DEBUG_TWEAKS_ENABLED`: Enable debug features (default: 1)
+- `DISK_ENCRYPTION_KEY_STORAGE`: Configure disk encryption key storage location (optional)
+
+## Usage
+```bash
+make image-base
+```
+
+For measurement generation:
+```bash
+make measurements-base
+```
+
+Build artifacts will be available in `reproducible-build/artifacts-base/`.

config_files/tdx-base/patches/patch-local.conf

@@ -0,0 +1,70 @@
+--- conf/local.conf	2024-06-03 15:49:59.294281280 +0000
++++ /home/ubuntu/poky/build/conf/local.conf	2024-06-03 10:16:24.809382127 +0000
+@@ -36,7 +36,7 @@
+ #MACHINE ?= "genericx86-64"
+ #
+ # This sets the default machine to be qemux86-64 if no other machine is selected:
+-MACHINE ??= "qemux86-64"
++#MACHINE ??= "qemux86-64"
+
+ # These are some of the more commonly used values. Looking at the files in the
+ # meta/conf/machine directory, or the conf/machine directory of any additional layers
+@@ -91,7 +91,7 @@
+ # Ultimately when creating custom policy, people will likely end up subclassing 
+ # these defaults.
+ #
+-DISTRO ?= "poky"
++#DISTRO ?= "poky"
+ # As an example of a subclass there is a "bleeding" edge policy configuration
+ # where many versions are set to the absolute latest code from the upstream 
+ # source control systems. This is just mentioned here as an example, its not
+@@ -110,7 +110,7 @@
+ #  - 'package_rpm' for rpm style packages
+ # E.g.: PACKAGE_CLASSES ?= "package_rpm package_deb package_ipk"
+ # OE-Core defaults to ipkg, whilst Poky defaults to rpm:
+-# PACKAGE_CLASSES ?= "package_rpm"
++PACKAGE_CLASSES ?= "package_ipk"
+
+ #
+ # SDK target architecture
+@@ -238,8 +238,8 @@
+ # (CDN) kindly provided by JSDelivr, uncomment one of the SSTATE_MIRRORS lines, not both.
+ # Using the CDN rather than the yoctoproject.org address is suggested/preferred.
+ #
+-#BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws'
+-#SSTATE_MIRRORS ?= "file://.* http://cdn.jsdelivr.net/yocto/sstate/all/PATH;downloadfilename=PATH"
++BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws'
++SSTATE_MIRRORS ?= "file://.* http://cdn.jsdelivr.net/yocto/sstate/all/PATH;downloadfilename=PATH"
+ #
+ ###SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH"
+
+@@ -249,7 +249,7 @@
+ #
+ # By default native qemu will build with a builtin VNC server where graphical output can be
+ # seen. The line below enables the SDL UI frontend too.
+-PACKAGECONFIG:append:pn-qemu-system-native = " sdl"
++# PACKAGECONFIG:append:pn-qemu-system-native = " sdl"
+ # By default libsdl2-native will be built, if you want to use your host's libSDL instead of 
+ # the minimal libsdl built by libsdl2-native then uncomment the ASSUME_PROVIDED line below.
+ #ASSUME_PROVIDED += "libsdl2-native"
+@@ -269,8 +269,8 @@
+ #
+ # A shared hash equivalent server can be set with "<HOSTNAME>:<PORT>" format
+ #
+-#BB_HASHSERVE = "auto"
+-#BB_SIGNATURE_HANDLER = "OEEquivHash"
++BB_HASHSERVE = "auto"
++BB_SIGNATURE_HANDLER = "OEEquivHash"
+
+ #
+ # Memory Resident Bitbake
+@@ -286,3 +286,9 @@
+ # track the version of this file when it was generated. This can safely be ignored if
+ # this doesn't mean anything to you.
+ CONF_VERSION = "2"
++MACHINE_FEATURES_NATIVE:append = " efi"
++MACHINE_FEATURES:append = " efi"
++EXTRA_IMAGEDEPENDS += "ovmf"
++
++MACHINE ?= "tdx"
++DISTRO ?= "cvm"

config_files/tdx-base/setup

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+# Check if the first argument is -v or --verbose
+verbose=0
+if [[ $1 == "-v" ]] || [[ $1 == "--verbose" ]]; then
+    verbose=1
+    # Shift the positional parameters to the left, effectively removing the first argument
+    shift
+fi
+
+# Define a function for verbose output
+function verbose_output {
+    if [[ $verbose -eq 1 ]]; then
+        echo "$1"
+    fi
+}
+
+# Set CURRENT_PATH to the current working directory
+CURRENT_PATH=$(pwd)
+verbose_output "Set CURRENT_PATH to $CURRENT_PATH"
+
+# Switch to the oe-init-build-env directory
+pushd $CURRENT_PATH/srcs/poky/
+verbose_output "Switched to $(pwd)"
+
+# Source the oe-init-build-env script
+source oe-init-build-env
+verbose_output "Sourced the oe-init-build-env script"
+
+# Add the  meta-evm, meta-confidential-compute, meta-secure-core/meta-tpm2, meta-openembedded/meta-python, meta-openembedded/meta-oe layers meta-rust-bin meta-clang to bblayers.conf
+bitbake-layers add-layer ../meta-confidential-compute
+
+verbose_output "Added the meta-evm and meta-confidential-compute layers to bblayers.conf"
+
+# Return to the original directory
+popd
+verbose_output "Returned to $(pwd)"
+
+# Apply patches
+for patch in $CURRENT_PATH/srcs/yocto-manifests/config_files/tdx-base/patches/patch-*; do
+    # Extract the filename from the patch file and remove the "patch-" prefix
+    filename=$(basename $patch | sed 's/^patch-//')
+    verbose_output "Processing patch file $patch"
+
+    # Apply the patch to the corresponding file in srcs/poky/build/conf/
+    patch -N $CURRENT_PATH/srcs/poky/build/conf/$filename -i $patch
+    verbose_output "Applied patch to $CURRENT_PATH/srcs/poky/build/conf/$filename"
+done

config_files/tdx-bob/README.md

@@ -0,0 +1,46 @@
+# TDX BOB (Builder/Searcher) Profile
+
+This profile creates a specialized image for running searcher nodes in confidential compute environments. It includes podman support and searcher-specific configurations.
+
+## Features
+- AMD SEV-SNP/TDX support
+- Secure container runtime isolation via podman
+- SSH key-based authentication
+- TPM2 measurements and attestation
+
+## Included Layers
+- meta-confidential-compute: Core confidential computing support
+- meta-openembedded: System utilities and libraries
+- meta-secure-core: Security features
+- meta-virtualization: Container support
+- meta-custom-podman: Customized podman configuration
+- meta-searcher: Searcher-specific features
+- poky: Base Yocto distribution
+
+## Build Configuration
+- Image Type: `cvm-image-azure`
+- Package Format: IPK
+- Supported Machine: tdx
+- Distribution: cvm
+
+## Required Configuration
+Before building, you must set the following in env_files/bob_yocto_build_config.env:
+- `SEARCHER_SSH_KEY`: SSH public key for searcher access (required)
+
+## Usage
+```bash
+# First set SEARCHER_SSH_KEY in env_files/bob_yocto_build_config.env
+make image-bob
+```
+
+For measurement generation:
+```bash
+make measurements-bob
+```
+
+Build artifacts will be available in `reproducible-build/artifacts-bob/`.
+
+## Notes
+- Ensure SSH key is properly configured before building
+- The image is optimized for searcher workloads
+- Includes container management capabilities

config_files/tdx-bob/patches/patch-local.conf

@@ -0,0 +1,70 @@
+--- conf/local.conf	2024-06-03 15:49:59.294281280 +0000
++++ /home/ubuntu/poky/build/conf/local.conf	2024-06-03 10:16:24.809382127 +0000
+@@ -36,7 +36,7 @@
+ #MACHINE ?= "genericx86-64"
+ #
+ # This sets the default machine to be qemux86-64 if no other machine is selected:
+-MACHINE ??= "qemux86-64"
++#MACHINE ??= "qemux86-64"
+
+ # These are some of the more commonly used values. Looking at the files in the
+ # meta/conf/machine directory, or the conf/machine directory of any additional layers
+@@ -91,7 +91,7 @@
+ # Ultimately when creating custom policy, people will likely end up subclassing 
+ # these defaults.
+ #
+-DISTRO ?= "poky"
++#DISTRO ?= "poky"
+ # As an example of a subclass there is a "bleeding" edge policy configuration
+ # where many versions are set to the absolute latest code from the upstream 
+ # source control systems. This is just mentioned here as an example, its not
+@@ -110,7 +110,7 @@
+ #  - 'package_rpm' for rpm style packages
+ # E.g.: PACKAGE_CLASSES ?= "package_rpm package_deb package_ipk"
+ # OE-Core defaults to ipkg, whilst Poky defaults to rpm:
+-# PACKAGE_CLASSES ?= "package_rpm"
++PACKAGE_CLASSES ?= "package_ipk"
+
+ #
+ # SDK target architecture
+@@ -238,8 +238,8 @@
+ # (CDN) kindly provided by JSDelivr, uncomment one of the SSTATE_MIRRORS lines, not both.
+ # Using the CDN rather than the yoctoproject.org address is suggested/preferred.
+ #
+-#BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws'
+-#SSTATE_MIRRORS ?= "file://.* http://cdn.jsdelivr.net/yocto/sstate/all/PATH;downloadfilename=PATH"
++BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws'
++SSTATE_MIRRORS ?= "file://.* http://cdn.jsdelivr.net/yocto/sstate/all/PATH;downloadfilename=PATH"
+ #
+ ###SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH"
+
+@@ -249,7 +249,7 @@
+ #
+ # By default native qemu will build with a builtin VNC server where graphical output can be
+ # seen. The line below enables the SDL UI frontend too.
+-PACKAGECONFIG:append:pn-qemu-system-native = " sdl"
++# PACKAGECONFIG:append:pn-qemu-system-native = " sdl"
+ # By default libsdl2-native will be built, if you want to use your host's libSDL instead of 
+ # the minimal libsdl built by libsdl2-native then uncomment the ASSUME_PROVIDED line below.
+ #ASSUME_PROVIDED += "libsdl2-native"
+@@ -269,8 +269,8 @@
+ #
+ # A shared hash equivalent server can be set with "<HOSTNAME>:<PORT>" format
+ #
+-#BB_HASHSERVE = "auto"
+-#BB_SIGNATURE_HANDLER = "OEEquivHash"
++BB_HASHSERVE = "auto"
++BB_SIGNATURE_HANDLER = "OEEquivHash"
+
+ #
+ # Memory Resident Bitbake
+@@ -286,3 +286,9 @@
+ # track the version of this file when it was generated. This can safely be ignored if
+ # this doesn't mean anything to you.
+ CONF_VERSION = "2"
++MACHINE_FEATURES_NATIVE:append = " efi"
++MACHINE_FEATURES:append = " efi"
++EXTRA_IMAGEDEPENDS += "ovmf"
++
++MACHINE ?= "tdx"
++DISTRO ?= "cvm"