Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How do I configure two TDX host machine for identical measurements #270

Open
fwoodruff-ab opened this issue Nov 13, 2024 · 9 comments
Open

How do I configure two TDX host machine for identical measurements #270

fwoodruff-ab opened this issue Nov 13, 2024 · 9 comments

Comments

@fwoodruff-ab
Copy link

Describe the support request
Running this repo multiple times produces slightly different images but for any given run, a particular image is created, which can be widely shared and audited.

I am finding that running the same TDX .qcow2 image (same file checksum) on different host machines configured identically by the same cloud provider I get the same measurement. However I now have a bare metal machine and all nonzero measurements except vendor ID are different (MRSEAM, MRTD, RTMR[0:2] etc.

How do I configure two bare metal TDX machines to produce the same attestation? How might I get an identical attestation to the cloud provider? How else can I audit the firmware TDX guest cloud providers are using?

I have attached a system report to comply with the requested formatting here although I don't believe it is relevant.

System report

Git ref

d792cec63042a704591f79d730401db3c5e8b27a

Operating system details

Distributor ID:	Ubuntu
Description:	Ubuntu 24.04.1 LTS
Release:	24.04
Codename:	noble

Kernel version

6.8.0-1013-intel #20-Ubuntu SMP PREEMPT_DYNAMIC Thu Oct  3 17:38:00 UTC 2024 x86_64 x86_64 GNU/Linux

TDX kernel logs

[    5.055379] virt/tdx: BIOS enabled: private KeyID range [64, 128)
[    5.055383] virt/tdx: Disable ACPI S3. Turn off TDX in the BIOS to use ACPI S3.
[   15.823342] virt/tdx: TDX module: attributes 0x0, vendor_id 0x8086, major_version 1, minor_version 5, build_date 20240129, build_num 698
[   15.823352] virt/tdx: CMR: [0x100000, 0x77800000)
[   15.823357] virt/tdx: CMR: [0x100000000, 0x407a000000)
[   15.823360] virt/tdx: CMR: [0x4080000000, 0x807c000000)
[   15.823363] virt/tdx: CMR: [0x8080000000, 0xc07c000000)
[   15.823365] virt/tdx: CMR: [0xc080000000, 0x1007c000000)
[   19.947613] virt/tdx: 4202516 KB allocated for PAMT
[   19.947623] virt/tdx: module initialized
...
[    5.055379] virt/tdx: BIOS enabled: private KeyID range [64, 128)
[    5.055383] virt/tdx: Disable ACPI S3. Turn off TDX in the BIOS to use ACPI S3.
[   15.823342] virt/tdx: TDX module: attributes 0x0, vendor_id 0x8086, major_version 1, minor_version 5, build_date 20240129, build_num 698
[   15.823352] virt/tdx: CMR: [0x100000, 0x77800000)
[   15.823357] virt/tdx: CMR: [0x100000000, 0x407a000000)
[   15.823360] virt/tdx: CMR: [0x4080000000, 0x807c000000)
[   15.823363] virt/tdx: CMR: [0x8080000000, 0xc07c000000)
[   15.823365] virt/tdx: CMR: [0xc080000000, 0x1007c000000)
[   19.947613] virt/tdx: 4202516 KB allocated for PAMT
[   19.947623] virt/tdx: module initialized

TDX CPU instruction support

CPU supports TDX according to /proc/cpuinfo

Model specific registers (MSRs)

MK_TME_ENABLED bit: 1 (expected value: 1)
SEAM_RR bit: 1 (expected value: 1)
NUM_TDX_PRIV_KEYS: 40
SGX_AND_MCHECK_STATUS: 0 (expected value: 0)
Production platform: Production (expected value: Production)

CPU details

 INTEL(R) XEON(R) PLATINUM 8570

QEMU package details

Status: Installed
Package: qemu-system-x86
Version: 1:8.2.2+ds-0ubuntu2+tdx1.0
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-release/ubuntu noble/main amd64 Packages

Libvirt package details

Status: Installed
Package: libvirt-clients
Version: 10.0.0-2ubuntu8.3+tdx1.2
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-release/ubuntu noble/main amd64 Packages

OVMF package details

Status: Installed
Package: ovmf
Version: 2024.02-3+tdx1.0
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-release/ubuntu noble/main amd64 Packages

sgx-dcap-pccs package details

Status: Installed
Package: sgx-dcap-pccs
Version: 1.21-0ubuntu1
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-attestation-release/ubuntu noble/main amd64 Packages

tdx-qgs package details

Status: Installed
Package: tdx-qgs
Version: 1.21-0ubuntu2.2
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-attestation-release/ubuntu noble/main amd64 Packages

sgx-ra-service package details

Status: Installed
Package: sgx-ra-service
Version: 1.21-0ubuntu2.2
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-attestation-release/ubuntu noble/main amd64 Packages
Description: Intel(R) Software Guard Extensions Multi-Package Registration Agent Service

sgx-pck-id-retrieval-tool package details

Status: Installed
Package: sgx-pck-id-retrieval-tool
Version: 1.21-0ubuntu2.2
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-attestation-release/ubuntu noble/main amd64 Packages

QGSD service status

● qgsd.service - Intel(R) TD Quoting Generation Service
     Loaded: loaded (/usr/lib/systemd/system/qgsd.service; enabled; preset: enabled)
     Active: active (running) since Thu 2024-11-07 08:41:18 PST; 41min ago
    Process: 3650 ExecStartPre=/bin/chown -R qgsd:qgsd /var/opt/qgsd/ (code=exited, status=0/SUCCESS)
    Process: 3682 ExecStartPre=/bin/chmod 0750 /var/opt/qgsd/ (code=exited, status=0/SUCCESS)
    Process: 3693 ExecStartPre=/usr/share/qgs/linksgx.sh (code=exited, status=0/SUCCESS)
    Process: 3720 ExecStart=/usr/bin/qgs (code=exited, status=0/SUCCESS)
   Main PID: 3731 (qgs)
      Tasks: 5 (limit: 629145)
     Memory: 13.8M (peak: 14.3M)
        CPU: 428ms
     CGroup: /system.slice/qgsd.service
             └─3731 /usr/bin/qgs

Nov 07 09:21:45 sdp qgsd[3731]: unpack message successfully in thread [79d944107740]
Nov 07 09:21:45 sdp qgsd[3731]: tee_att_get_quote_size return 0x1100f
Nov 07 09:21:45 sdp qgsd[3731]: call tee_att_init_quote
Nov 07 09:21:46 sdp qgsd[3731]: [QPL] No certificate data for this platform.
Nov 07 09:21:46 sdp qgsd[3731]: [get_platform_quote_cert_data ../td_ql_logic.cpp:302] Error returned from the p_sgx_get_quote_config API. 0xe011
Nov 07 09:21:46 sdp qgsd[3731]: tee_att_init_quote return 0x11001
Nov 07 09:21:46 sdp qgsd[3731]: tee_att_get_quote_size return 0x1100f
Nov 07 09:21:46 sdp qgsd[3731]: resp_size is 0
Nov 07 09:21:46 sdp qgsd[3731]: About to shutdown and close socket
Nov 07 09:21:46 sdp qgsd[3731]: erased a connection, now [0]

PCCS service status

● pccs.service - Provisioning Certificate Caching Service (PCCS)
     Loaded: loaded (/usr/lib/systemd/system/pccs.service; enabled; preset: enabled)
     Active: active (running) since Thu 2024-11-07 08:49:11 PST; 33min ago
       Docs: https://github.com/intel/SGXDataCenterAttestationPrimitives/blob/master/QuoteGeneration/pccs/README.md
   Main PID: 5236 (node)
      Tasks: 15 (limit: 629145)
     Memory: 47.9M (peak: 59.5M)
        CPU: 3.546s
     CGroup: /system.slice/pccs.service
             └─5236 /usr/bin/node /opt/intel/sgx-dcap-pccs/pccs_server.js

Nov 07 09:21:45 sdp node[5236]: 2024-11-07 09:21:45.874 [info]: Client Request-ID : 5896dc0a037745e7a08d6e11b23b56d8
Nov 07 09:21:46 sdp node[5236]: 2024-11-07 09:21:46.442 [info]: Request-ID is : undefined
Nov 07 09:21:46 sdp node[5236]: 2024-11-07 09:21:46.442 [error]: Intel PCS server returns error(401).{ "statusCode": 401, "message": "Access denied due to invalid subscription key. Make sure you use valid one or no subscription key at all." }
Nov 07 09:21:46 sdp node[5236]: 2024-11-07 09:21:46.442 [error]: Error: No cache data for this platform.
Nov 07 09:21:46 sdp node[5236]:     at Module.getPckCertFromPCS (file:///opt/intel/sgx-dcap-pccs/services/logic/commonCacheLogic.js:159:11)
Nov 07 09:21:46 sdp node[5236]:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Nov 07 09:21:46 sdp node[5236]:     at async LazyCachingMode.getPckCertFromPCS (file:///opt/intel/sgx-dcap-pccs/services/caching_modes/cachingMode.js:126:12)
Nov 07 09:21:46 sdp node[5236]:     at async Module.getPckCert (file:///opt/intel/sgx-dcap-pccs/services/pckcertService.js:118:16)
Nov 07 09:21:46 sdp node[5236]:     at async getPckCert (file:///opt/intel/sgx-dcap-pccs/controllers/pckcertController.js:77:25)
Nov 07 09:21:46 sdp node[5236]: 2024-11-07 09:21:46.445 [info]: 127.0.0.1 - - [07/Nov/2024:17:21:46 +0000] "GET /sgx/certification/v4/pckcert?qeid=363071E31D0A099CF232DC8CB9DF1948&encrypted_ppid=7E6867A54B2411CFB190A7B1BE231AAE6BE8317CC0BFE4D76E51AE1C4E55CFF9F7C4FD78AFAC84DE13E37282ACBFF88DFBFD0AC8159FD1B76754E10FA8DFCD493CDF124748883F7818302AD85C3BC488F71655E240044CCBADFEFD53E93C981C03BA6DFFD6E28D08FF3C959897A6025860A97908B937318D5CC57CD9A4E0CC1181042E5F144024F8040D38EAD01CF64173B014BB189CA01D5E03FEF7B3D4E98ABFC6DB06901243129719D21A6368D9143E7AB993808361CEFED8BCE9E16752AD4747D38725CA4DE5CC46D3D4EA49CCC6DC568033CB7A93264DAFE5A9DC8D34863A9371FED049B3AD695BAAF7C86DC46E589B33F90A8262BA8C2AEAD34733113DD838810F49FB4D594838899613E8A70754D50647F29F7EE95DE528120D52ACE2DB60F3B1BDA8CEC1F271315F6BF3D35B8896AB624DFD3A1A85DCF487737C58F91B9E66DA9848B89B4CEEA8473D57DD940B6EE5B729FFC586C7FFECB5CCD95FF2052D9EA15A3216CC7B65E901D695C7DF14450E5F55C443C13D5A455E9A478B30&cpusvn=0202191B03FF00060000000000000000&pcesvn=0F00&pceid=0000 HTTP/1.1" 404 32 "-" "-"

MPA registration logs (last 30 lines)

[07-11-2024 08:29:19] INFO: SGX Registration Agent version: 1.21.100.3
[07-11-2024 08:29:19] INFO: Starts Registration Agent Flow.
[07-11-2024 08:29:19] INFO: SGX MP Server configuration flag indicates that Registration Server won't save encrypted platform keys.
[07-11-2024 08:29:19] INFO: Platform registration request (PLATFORM_MANIFEST) won't be send to Registration Server.
[07-11-2024 08:29:19] INFO: Please use management tool or PCKCertIDRetrievalTool to read PLATFORM_MANIFEST.
[07-11-2024 08:29:19] INFO: Finished Registration Agent Flow.
[07-11-2024 08:41:18] INFO: SGX Registration Agent version: 1.21.100.3
[07-11-2024 08:41:18] INFO: Starts Registration Agent Flow.
[07-11-2024 08:41:24] INFO: Registration Flow - PLATFORM_ESTABLISHMENT or TCB_RECOVERY passed successfully.
[07-11-2024 08:41:24] INFO: Finished Registration Agent Flow.
@syncronize-issues-to-jira Syncronize issues to Jira
Copy link

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/PEK-1458.

This message was autogenerated

@hector-cao
Copy link
Collaborator 

However I now have a bare metal machine and all nonzero measurements except vendor ID are different (MRSEAM, MRTD, RTMR[0:2] etc.

can you elaborate a little bit please ? for this third machine, you are saying that the values are different to the first 2 machines ?

@fwoodruff-ab
Copy link
Author 

However I now have a bare metal machine and all nonzero measurements except vendor ID are different (MRSEAM, MRTD, RTMR[0:2] etc.

can you elaborate a little bit please ? for this third machine, you are saying that the values are different to the first 2 machines ?

Hi Hector,

Yes, the third machine is a bare metal machine. The first two machines are on Google Cloud, where we only have TDX Guest access.

@hector-cao
Copy link
Collaborator

hector-cao commented Nov 14, 2024 •
edited
Loading

and you are using the same guest image for Google Cloud and bare metal ? I dont know how Google Cloud run the guest because i can impact the event log contents and therefore the RTMR measurement values

@fwoodruff-ab
Copy link
Author

Yes it's the same image.

because it can impact the event log contents

Can I turn event logs off maybe?

@hector-cao
Copy link
Collaborator

hector-cao commented Nov 15, 2024 •
edited
Loading

You cannot, the event logs are used to compute the measurement values RTMR
If you have 2 bare metal TDX host, i will be interested to see the measurement values for the same guest image on those 2 machines

@kostko
Copy link

kostko commented Nov 20, 2024 •
edited
Loading

There is a bunch of differences between hypervisors on bare metal machines and also between cloud providers, some thoughts:

  • Cloud providers usually do not provide a way for one to configure the virtual firmware, which in case of TDX has the role of TDVF and performs early boot. The TDVF is responsible for measuring all later boot stages, including the kernel, cmdline, etc. If the firmware is different, the later measurements can be completely different.
  • Even when using the same firmware, the way the hypervisor generates and loads certain artifacts (e.g. ACPI tables) can impact measurements.
  • Hardware configuration (number of CPUs, memory) also impacts measurements as these attributes are reflected in TD HOB and/or ACPI tables.
  • The firmware load process itself is also measured into MRTD. But there are multiple ways of loading the firmware, depending on how pages are initialized and measured. For example, there are subtle differences between some kernel/qemu patchsets which result in different order of MEM.PAGE.ADD/MR.EXTEND executions and thus different MRTD measurements for the same TDVF image.

@hector-cao
Copy link
Collaborator

@kostko Thanks for this feedback, i m not sure that TDVF is responsible for measuring all later boot stages, it might depends on what is the boot flow, in case the boot flow involves shim, grub, these 2 components will record their own event logs whose digests will contribute to the measurement values (RTMR)

@kostko
Copy link

kostko commented Nov 21, 2024

Yes of course, all the later boot stages may (and do) further extend the RTMRs. But those components (boot loader, kernel, apps, ...) you can control quite easily to ensure determinism.

It is more tricky to ensure determinism of early boot due to the issues I mentioned above.

@mythi mythi mentioned this issue Dec 11, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kostko @hector-cao @fwoodruff-ab