Why Does KIND_MODEL Validation Forbid `gpus` Field when Backend API Supports `device_id` for All Instance Kinds

[dhanushSB96]

I'm developing a custom accelerator backend and noticed that model_config_utils.cc (lines 1698–1704) forbids using the gpus field with KIND_MODEL, even though the backend API TRITONBACKEND_ModelInstanceDeviceId() returns a device_id for all instance kinds (KIND_GPU, KIND_MODEL, KIND_CPU). It simply returns the device_id assigned during instance creation, regardless of kind.

The device_id is stored during instance creation and passed to backends through this API, and backends can use this device_id to select their target device, regardless of whether it’s a CUDA GPU or a custom accelerator.

Is there a specific reason for this validation? For custom accelerator backends, the gpus field would be useful for device selection, similar to CUDA GPU backends.

I know we can use device_id or a parameter-based workaround, but the gpus field would be more flexible. I mainly want to understand the reason for this validation and whether it would be possible to relax this validation check.

Use Case:
For custom accelerator backends, the gpus field would be a natural way to specify device IDs, similar to how CUDA GPU backends use it. The field name gpus might be a misnomer for non-CUDA devices, but the functionality (device selection) is exactly what custom accelerator backends need.

Environment:
Triton Server Version: 25.09
Backend: Custom
Platform: Linux

[JiwaniZakir]

The validation in model_config_utils.cc (lines 1698–1704) that rejects gpus for KIND_MODEL was almost added when KIND_MODEL was introduced to represent model-parallelism or vendor-specific placement — the assumption being that device assignment for those backends would be handled internally by the backend itself rather than via the Triton config. The gpus field was intentionally scoped to KIND_GPU because historically it mapped directly to CUDA device IDs, and allowing it for KIND_MODEL would have been semantically ambiguous at the time. The disconnect you're seeing is that TRITONBACKEND_ModelInstanceDeviceId() is deliberately backend-agnostic and just surfaces whatever device_id was assigned at instance creation — it was designed without a corresponding relaxation of the config validation layer. Relaxing the check in NormalizeInstanceGroup() to permit gpus for KIND_MODEL when a non-GPU backend is in use would be the minimal change needed, though it would also require a decision on whether gpus should be renamed or aliased (e.g., devices) to avoid misleading semantics for non-CUDA accelerators.

[dhanushSB96]

Thanks for the detailed explanation — that makes sense and aligns with what I suspected.

I understand that when KIND_MODEL was introduced, the intent was to keep device assignment fully backend‑managed, and that gpus was intentionally scoped to KIND_GPU due to its historical mapping to CUDA device IDs. The clarification around the semantic ambiguity at the time is very helpful.

I agree that the disconnect stems from the backend API (TRITONBACKEND_ModelInstanceDeviceId()) being deliberately backend‑agnostic, without a corresponding relaxation in the config validation layer. From a non-CUDA accelerator backend perspective, relaxing the check in NormalizeInstanceGroup() to allow gpus for KIND_MODEL (when a non‑GPU backend is in use) seems like a minimal and pragmatic change, especially since Triton already assigns and propagates device_id uniformly.

I also agree that gpus is not ideal semantically for non‑CUDA devices, and that an alias or a more generic field like devices could be a cleaner longer‑term solution. Happy to follow the recommended direction here, and I’m also open to contributing a PR if there’s agreement on the desired behavior and scope.