Create VGPU changes with VFIO Framework

[JunAr7112]

No description provided.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cdesiniotis]

Thanks @JunAr7112, this is a good start. As we make iterations on this and get more familiar with the internals here, it may be valuable to create a new internal/vgpu package that hides away the vfio vs mdev framework complexity. We need to think through what the right interface would be, but I imagine we will need methods for 1) getting all vGPU devices, 2) getting all parent devices (of which you can create a vGPU device on top of), 3) creating a vGPU device. The pkg/vgpu/config.go file, which is concerned with getting / setting a particular vGPU config, can invoke these methods without having to know what vfio / mdev is.

[cdesiniotis]

Question -- does the name reported by NVML, e.g. vgpuTypeId.GetName(), align exactly with the name we were using before? (the names stored in vgpuDev.MDEVType)

[JunAr7112]

I added a helper function to ensure they exactly align

[cdesiniotis]

So the name reported by NVML contains the product prefix? For example, is NVML returning NVIDIA A100-4C as the type name for the A100-4C device?

[JunAr7112]

Yes I checked this manually earlier and added the parseVGPUTypeName(rawName string) to verify that we would only be getting A100-4C. The name reported by NVML included a prefix.

[cdesiniotis]

I don't think the nvidia directory should be included in this path. The "path to the VF" is simply /sys/bus/pci/devices/<BDF>/virtfn<N>. Other parts of the code are not intuitive to me because vfAddr includes the nvidia directory.

[JunAr7112]

local-agadiyar@ipp1-2284:/sys/bus/pci/devices/0000:41:00.0/virtfn0/nvidia$ cat current_vgpu_type
687
local-agadiyar@ipp1-2284:/sys/bus/pci/devices/0000:41:00.0/virtfn0/nvidia$ cat creatable_vgpu_types
ID : vGPU Name

The current_vgpu_type and creatable_vgpu_types files are located in the nvidia folder. This way we don't need to append nvidia onto another address variable

[cdesiniotis]

Question: Shouldn't NvidiaPCIDevice represent the VF (IIUC device is currently a PF)? If so, then I don't see the need to have VirtualFunctionaPath as a separate field. If we instead used the VF here, I think that would simplify the code in a few places and make this easier to read. Note, the nvpci.NvidiaPCIDevice type allows you to go from the VF to the backing PF via device.SriovInfo.VirtualFunction.PhysicalFunction.

[JunAr7112]

I think the issue here is that nvpci doesn't have a built in way to get virtual functions from the physical function. That is why I am storing the physical device ( via nvdevices, err := m.nvlib.Nvpci.GetGPUs() ) as well as the path to the virtual function.

[cdesiniotis]

As indicated in a prior comment, if parentDevice was just of type nvpci.NvidiaPCIDevice (and represented the VF), this code would be replaced by:

vgpuTypeNumberBytes, err := os.ReadFile(filepath.Join(parentDevice.NvidiaPCIDevice.Path, "current_vgpu_type"))

[JunAr7112]

See above comment.

[cdesiniotis]

Why is this field needed? Shouldn't NvidiaPCIDevice.Path represent the path to the virtual function?

[JunAr7112]

See above comment. nvpci.NvidiaPCIDevice is storing the physical function

[cdesiniotis]

Question -- does nvlib.Interface need to exist anymore?

[JunAr7112]

No we are no longer using the nvlib.Interface. I don't think we need it for any of the other projects either

[cdesiniotis]

So the name reported by NVML contains the product prefix? For example, is NVML returning NVIDIA A100-4C as the type name for the A100-4C device?

[cdesiniotis]

To improve readability, what if we constructed a map, named supportedVgpuTypes, prior as such:

supportedVgpuTypes := map[string]bool{}
for _, vgpu := range supportedVGPUs {
  name, ret := vgpu.GetName()
  if ret != nvml.SUCCESS {
    continue
  }
  name = parseVGPUTypeName(name)
  supportedVgpuNames[name] = true
}

Then this for loop would simplify to

Suggested change

	found := false
	for _, vgpuTypeId := range supportedVGPUs {
	rawName, ret := vgpuTypeId.GetName()
	if ret != nvml.SUCCESS {
	continue
	}
	name := parseVGPUTypeName(rawName)
	if name == key {
	found = true
	sanitizedConfig[key] = val
	break
	}
	if name == strippedKey {
	found = true
	sanitizedConfig[strippedKey] = val
	break
	}
	}
	if _, ok := supportedVgpuTypes[key]; ok {
	sanitizedConfig[key] = val
	} else if _, ok := supportedVgpuTypes[strippedKey]; ok {
	sanitizedConfig[strippedKey] = val
	} else {
	return fmt.Errorf("vGPU type %s is not supported on GPU (index=%d, address=%s)", key, gpu, device.Address)
	}

[JunAr7112]

Ok I broke this into two for loops.