OpenMPI 5.0.x maps/binds to cores even if asked to use hwthreads #12967
Labels
Comments
LourensVeen
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Background information
I'm trying to start an MPI program on a particular subset of resources within an allocation (or locally, the results are the same). I'm mapping with a rankfile and binding the MPI processes to a specific set of CPUs that way. My software is set up to specify logical CPUs, these being hwthreads on my 8C16T laptop.
My test suite includes OpenMPI 3.1.6 and 4.1.6, which both work fine, but I can't get OpenMPI 5.0.x to bind to hwthreads, it insists on interpreting the numbers in the rankfile as core ids. As a result, everything runs fine but in the wrong place.
What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)
I tested 5.0.1, 5.0.3 and 5.0.5, which don't work, and 4.1.6 and 3.1.6, which do. Versions 5.0.x use hwloc 2.11.1, while the older ones use hwloc 1.11.13.
Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)
I used Spack v0.22 and v0.23, with SLURM support and external PMIX, hwloc and libevent.
Please describe the system on which you are running
Kubuntu 22.04
I'm running inside of a Docker container, and can reproduce the problem both in a SLURM allocation and outside of one, but only on 5.0.x.
Lenovo P14s, AMD 7 PRO 5850U CPU with 8 cores and 16 hwthreads
(output of ompi_info --all omitted because the message is too long)
Output of lstopo -v
Using hwloc 2.11.1Output of lstopo --of xml
Using hwloc 2.11.1Output of mpirun test command
No special hardware, connections are either local or over TCP between different Docker containers representing fake cluster nodes.
Details of the problem
When I use
mpirunto start a program and ask for OpenMPI to map processes using a rankfile and hwthreads, it assigns whole cores to each process instead of individual threads. That is, the slots in the rankfile are always interpreted as core ids, not as hwthread (logical CPU) ids:This is as expected, because OpenMPI interprets the numbers as core ids by default, and cores 0 and 1 map to hwthreads {0, 1} and {2, 3} respectively. So let's use
--use-hwthread-cpusto fix that:On pre-5.0.x, the above prints
{0, 1}and{2, 3}, but 5.0.x seems to persist in using cores rather than hwthreads. It does complain about me using old syntax however, so let's try the newer one:shell$ mpirun -n 2 --map-by rankfile:file=rankfile:hwtcpus python -c 'import os; print(os.sched_getaffinity(0))' {0, 1, 2, 3} {4, 5, 6, 7}That fixes the warning, but not the problem. Let's see what it's actually doing:
Okay, I don't think that was supposed to happen. Let's try a different way:
Ah, it's trying to bind to cores, maybe that's it?
Nope. Maybe try the old syntax again?
That
Cpu Type: COREmay be the problem, but how to convince OpenMPI that I have hwthreads to bind to? And why does it work on earlier versions?I can't find anything in the 5.0.x docs that suggests that this is intended, so I think it's a bug, either in the code (the ORTE to PRRTE switch maybe?) or in the docs. Or perhaps in my brain, if I missed something. At any rate, any help in fixing it would be much appreciated!
The text was updated successfully, but these errors were encountered: