CXL emulation: cxl devices are shown in cxl list but they are not accessible in SYSRAM mode or DAX mode

[hextag]

'cxl list' lists the following devices. I also verified using lspci -t that these devices are present in PCIe hierarchy. However, these devices cannot be accessed in DAX mode or SysRAM mode.

[root@fedora devices]# cxl list
[
  {
    "memdev":"mem0",
    "ram_size":268435456,
    "serial":0,
    "host":"0000:c3:00.0"
  },
  {
    "memdev":"mem2",
    "ram_size":268435456,
    "serial":0,
    "host":"0000:c2:00.0"
  },
  {
    "memdev":"mem3",
    "pmem_size":268435456,
    "serial":0,
    "host":"0000:36:00.0"
  },
  {
    "memdev":"mem1",
    "pmem_size":268435456,
    "serial":0,
    "host":"0000:37:00.0"
  }
]

There are two issues which I'm facing,

1. daxctl list doesn't list any devices. There are also no dax nodes in /dev/ when I did ls

   • This is possibly because of bios or firmware configuration. I'm trying to debug this based on this issue -
     cxl list, no matching device found #246 (comment) which asks to tweek bios/firmware settings to check if EFI_SP_MEMORY is configured properly in firmware.

2. I expected these devices to be in SysRam mode if they are not in dax mode because of above issue. But, these devices are not in SysRAM mode based on the output of lsmem.
   Following are the logs.

[root@fedora mem0]# lsmem
RANGE                                 SIZE  STATE REMOVABLE BLOCK
0x0000000000000000-0x000000007fffffff   2G online       yes  0-15
0x0000000180000000-0x000000027fffffff   4G online       yes 48-79

Memory block size:       128M
Total online memory:       6G
Total offline memory:      0B

I can't use the following command of daxctl reconfigure to reconfigure the devices since I don't see daxx.y nodes in /dev/ as of now,

daxctl reconfigure-device --mode=system-ram --online-movable --force /dev/daxX.Y

Is the issue of not seeing these nodes in SysRam mode also related to Issue 1 of not seeing dax nodes in /dev/ ?

System Configuration

Kernel: Linux fedora 6.8.0-rc6 #2 SMP PREEMPT_DYNAMIC Fri Oct 11 17:04:26 CDT 2024 x86_64 GNU/Linux

[stellarhopper]

@hextag look at the cxl create-region command - you'll want to create a CXL region by specifying which memdevs you want in it, and the interleave settings etc, and then you can do device-dax / system-ram setup on it.

[hextag]

@stellarhopper Thanks, I'm now able to create a /dev/daxX.Y node and mmap() on it.
Following are the commands I used.

cxl create-region -m -d decoder0.0 -w 2 -g 8192 mem1 mem3

sudo ndctl create-namespace --mode=devdax --force --region=region0

[hextag]

Hi @stellarhopper,

I exited qemu and launched qemu CXL VM again. However, I get error this time as shown below when trying to create namespace after creating region0.

[root@fedora ~]# sudo ndctl create-namespace --mode=devdax --force --region=region0
failed to create namespace: No space left on device

I used same command to create region as above from previous qemu launch which is succesful

cxl create-region -m -d decoder0.0 -w 2 -g 8192 mem2 mem3 -t pmem
[  133.823499] cxl region0: Bypassing cpu_cache_invalidate_memregion() for testing!
{
  "region":"region0",
  "resource":"0xb90000000",
  "size":"512.00 MiB (536.87 MB)",
  "type":"pmem",
  "interleave_ways":2,
  "interleave_granularity":8192,
  "decode_state":"commit",
  "mappings":[
    {
      "position":1,
      "memdev":"mem3",
      "decoder":"decoder7.0"
    },
    {
      "position":0,
      "memdev":"mem2",
      "decoder":"decoder6.0"
    }
  ]
}

This is command I use to launch qemu,

run_qemu.sh --cxl --git-qemu --rebuild none

Can you please help me how to fix the 'failed to create namespace: No space left on device' error when trying to create namespace. I'm using exact same commands as last qemu launch.

I'm retrying to launch qemu with --rebuild wipe to see if it helps to resolve this error

Update:

I don't see this error with --rebuild wipe passed to run_qemu.sh

[stellarhopper]

@hextag Without actually trying it, I'm guessing the ndctl namespace is auto-reassembled when you create the cxl region again.
Does ndctl list (maybe with -i) show the namespace after your cxl create-region step in the rebuild=none case?