running multi-node with Exclusive Process

[angainor]

When running all_reduce_perf across 2 nodes / 4 GH200 GPUs per node I run into trouble: nccl-tests tell me GPUs are busy, although they are not. I have the following SLURM job:

#SBATCH --ntasks-per-node=4 --gpus-per-node=4 --nodes=2
srun ./build/all_reduce_perf -b 8 -e 128M -f 2

The output looks good in the start, but then the program fails:

# nccl-tests version 2.17.8 nccl-headers=22606 nccl-library=22606
# Collective test starting: all_reduce_perf
# nThread 1 nGpus 1 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid 3617715 on    gpu-1-1 device  0 [0009:01:00] NVIDIA GH200 120GB
#  Rank  1 Group  0 Pid 3617716 on    gpu-1-1 device  1 [0019:01:00] NVIDIA GH200 120GB
#  Rank  2 Group  0 Pid 3617717 on    gpu-1-1 device  2 [0029:01:00] NVIDIA GH200 120GB
#  Rank  3 Group  0 Pid 3617718 on    gpu-1-1 device  3 [0039:01:00] NVIDIA GH200 120GB
#  Rank  4 Group  0 Pid 3230358 on    gpu-1-7 device  0 [0009:01:00] NVIDIA GH200 120GB
#  Rank  5 Group  0 Pid 3230359 on    gpu-1-7 device  1 [0019:01:00] NVIDIA GH200 120GB
#  Rank  6 Group  0 Pid 3230360 on    gpu-1-7 device  2 [0029:01:00] NVIDIA GH200 120GB
#  Rank  7 Group  0 Pid 3230361 on    gpu-1-7 device  3 [0039:01:00] NVIDIA GH200 120GB
gpu-1-1: Test CUDA failure common.cu:1304 'CUDA-capable device(s) is/are busy or unavailable'
 .. gpu-1-1 pid 3617715: Test failure common.cu:1189
gpu-1-7: Test CUDA failure common.cu:1304 'CUDA-capable device(s) is/are busy or unavailable'
 .. gpu-1-7 pid 3230359: Test failure common.cu:1189
gpu-1-7: Test CUDA failure common.cu:1304 'CUDA-capable device(s) is/are busy or unavailable'
 .. gpu-1-7 pid 3230361: Test failure common.cu:1189
gpu-1-7: Test CUDA failure common.cu:1304 'CUDA-capable device(s) is/are busy or unavailable'
 .. gpu-1-7 pid 3230358: Test failure common.cu:1189
gpu-1-7: Test CUDA failure common.cu:1304 'CUDA-capable device(s) is/are busy or unavailable'
 .. gpu-1-7 pid 3230360: Test failure common.cu:1189
gpu-1-1: Test CUDA failure common.cu:1304 'CUDA-capable device(s) is/are busy or unavailable'
 .. gpu-1-1 pid 3617718: Test failure common.cu:1189

Now, my GPUs are configured with Exclusive process, so multiple processes cannot use one GPU. I guess this is what is happening in nccl-tests, because when I change to Default:

nvidia-smi -i 0,1,2,3 -c 0

the test runs through. Inspection with nvidia-smi shows that indeed, many processes use GPU0:

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   3203007      C   ...ests-2.17.8/./build/all_reduce_perf        788MiB |
|    0   N/A  N/A   3203008      C   ...ests-2.17.8/./build/all_reduce_perf        556MiB |
|    0   N/A  N/A   3203009      C   ...ests-2.17.8/./build/all_reduce_perf        556MiB |
|    0   N/A  N/A   3203010      C   ...ests-2.17.8/./build/all_reduce_perf        556MiB |
|    1   N/A  N/A   3203008      C   ...ests-2.17.8/./build/all_reduce_perf        790MiB |
|    2   N/A  N/A   3203009      C   ...ests-2.17.8/./build/all_reduce_perf        786MiB |
|    3   N/A  N/A   3203010      C   ...ests-2.17.8/./build/all_reduce_perf        784MiB |
+-----------------------------------------------------------------------------------------+

Or rather, the 3 processes that should use GPUs 1,2,3 also in addition have a process running on GPU 0.

Is it possible to run nccl-tests with Exclusive process? Or do I have to consider changing that setting on the system? Also, do you know if this is this only nccl-tests limitation, or a general nccl issue?

[sjeaugey]

This looks like a bug, which we may have fixed. Can you try with a newer version?

[angainor]

@sjeaugey so git main? I was running with 2.17.8 - that's the newest I can find. I also tried with an earlier 2.14.1.

[angainor]

@sjeaugey I see the same behavior with main.

[sjeaugey]

So you ran with NCCL 2.29 and the latest NCCL perf tests?

[angainor]

I ran with NCCL/2.26.6 and latest perf tests. I can try newer NCCL and report back, if you think this will help.

[sjeaugey]

Ok thanks. I remember such a bug in the past which was fixed, that's why I wanted to try something more recent. 2.26 should be recent enough.

We should try to repro and confirm whether we see the same or not.

EDIT: I tried on a random machine with NCCL 2.26.6 and the latest NCCL tests and I could not see the same problem. So either it's related to that specific platform, or something else is going on.

[angainor]

@sjeaugey FYI, I have tested now with NCCL/2.29.2 and nccl-tests:main and the behavior is the same.

[sjeaugey]

I still can't reproduce, even on a similar system. Did you set any environment variable which could alter the behavior of the NCCL library or the NCCL test?

[angainor]

Interesting. I guess I should describe the system better. This is a Cray EX system. I am compiling ncc-tests using OpenMPI with libfabric support. There are a bunch of envars set:

NCCL_NET_GDR_LEVEL=PHB
OMPI_MCA_mtl=ofi
OMPI_MCA_pml=cm
OMPI_MCA_ras_base_launch_orted_on_hn=1
PRTE_MCA_ras_base_launch_orted_on_hn=1
FI_CXI_RX_MATCH_MODE=hybrid
FI_PROVIDER=cxi

I start the tests as follows:

#SBATCH --ntasks-per-node=4 --gpus-per-node=4 --nodes=2
srun ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 1

I will also test with the Cray MPI instead and let you know.

[angainor]

Note that this is a Slingshot11 system, where you have to run the jobs through SLURM in order for the interconnect to work.

[angainor]

@sjeaugey Yes, that's it - compiled with Cray MPI and started with srun - all works fine with Exclusive process. There is only 1 process per GPU at any time. But compiled with OpenMPI and started with srun I see this problem.

[angainor]

@sjeaugey Not sure this is relevant, but OpenMPI expects cudaChooseDevice(), cudaSetDevice(), etc. to be called before MPI_Init. Maybe in nccl-tests it's done later?

[sjeaugey]

I see. I've been using Open MPI in my experiments. But I'm also not using the same network plugin. Could it be the network plugin could be allocating GPU resources before the device is set?

[angainor]

@sjeaugey Thanks a lot - that was helpful. I think OpenMPI + libfabric does something during the initialization that is incompatible with either NCCL, or nccl-tests. A solution (workaround?) to this is to turn off HMEM support in the OFI mtl:

export OMPI_MCA_mtl_ofi_disable_hmem=true

or to not use the OFI mtl at all, e.g.

export OMPI_MCA_pml=ob1
export OMPI_MCA_btl=tcp,sm,self

I guess both solutions have their issues for hybrid NCCL-MPI applications. Do you have any thoughts on this?