Skip to content

[26.04_linux-nvidia-bos] ats always on v4#401

Draft
nirmoy wants to merge 6 commits intoNVIDIA:26.04_linux-nvidia-bosfrom
nirmoy:my-ats-always-on-v4
Draft

[26.04_linux-nvidia-bos] ats always on v4#401
nirmoy wants to merge 6 commits intoNVIDIA:26.04_linux-nvidia-bosfrom
nirmoy:my-ats-always-on-v4

Conversation

@nirmoy
Copy link
Copy Markdown
Collaborator

@nirmoy nirmoy commented Apr 30, 2026 •
edited
Loading

BugLink: https://bugs.launchpad.net/ubuntu/+source/linux-nvidia-6.17/+bug/2150727

Summary:

  • Revert the older ATS always-on backport stack currently in the 26.04 BOS branch.
  • Backport v4 of the ATS always-on lore series:
    • PCI support for CXL.cache-capable devices.
    • PCI quirks for pre-CXL NVIDIA GPU and CX10 devices.
    • arm-smmu-v3 support for devices that require ATS to stay enabled.
  • Keep NVIDIA SAUCE metadata, per-patch lore backport URLs, and Nirmoy backporter Signed-off-by trailers.

Backport notes:

  • Patch 1 is adapted to the 26.04 base where PCI_DVSEC_CXL_CACHE_CAPABLE already exists.
  • Patch 2 is applied after reverting the older ATS PCI quirk support.
  • Patch 3 is applied after reverting the older ATS arm-smmu-v3 support.

Validation:

  • Local patchscan: pass.
  • Local validate-pr: pass.
  • git diff --check: pass.
  • git show --check HEAD~6..HEAD: pass.
  • Targeted x86 object build: drivers/pci/ats.o, drivers/pci/quirks.o, drivers/pci/pci.o.

@nirmoy nirmoy marked this pull request as draft April 30, 2026 09:26
@nirmoy nirmoy changed the title My ats always on v4 ats always on v4 Apr 30, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 30, 2026 •
edited
Loading

PR Validation Report

Patchscan ✅ No Missing Fixes

All cherry-picked commits checked — no missing upstream fixes found.

PR Lint ✅ All checks passed

Details 
Checking 6 commits...

Cherry-pick digest:
┌──────────────┬───────────────────────────────────────────────┬────────────┬─────────┬───────────────────────────┐
│ Local        │ Referenced upstream / Patch subject           │ Patch-ID   │ Subject │ SoB chain                 │
├──────────────┼───────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 80c64428969c │ iommu/arm-smmu-v3: allow ats to be always on  │ noted      │ found   │ ok, backporter: nirmoyd   │
├──────────────┼───────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ d4d63f93e055 │ pci: allow ats to be always on for pre-cxl de │ noted      │ found   │ ok, backporter: nirmoyd   │
├──────────────┼───────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ f95b57224e51 │ pci: allow ats to be always on for cxl.cache  │ noted      │ found   │ ok, backporter: nirmoyd   │
├──────────────┼───────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ ab2033808093 │ [Revert] pci: allow ats to be always on for c │ N/A        │ N/A     │ nirmoyd                   │
├──────────────┼───────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 7323790a599c │ [Revert] pci: allow ats to be always on for n │ N/A        │ N/A     │ nirmoyd                   │
├──────────────┼───────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ a4c077a94504 │ [Revert] iommu/arm-smmu-v3: allow ats to be a │ N/A        │ N/A     │ nirmoyd                   │
└──────────────┴───────────────────────────────────────────────┴────────────┴─────────┴───────────────────────────┘

Lint: all checks passed.

@nirmoy nirmoy force-pushed the my-ats-always-on-v4 branch from ed22fc3 to 072939e Compare April 30, 2026 09:36
@nirmoy nirmoy changed the title ats always on v4 [26.04_linux-nvidia-bos] ats always on v4 Apr 30, 2026
nirmoy added 3 commits April 30, 2026 02:44
@nirmoy nirmoy force-pushed the my-ats-always-on-v4 branch 2 times, most recently from 15c3560 to 520182f Compare April 30, 2026 09:53
nicolinc added 3 commits April 30, 2026 03:03
@nicolinc @nirmoy
NVIDIA: VR: SAUCE: PCI: Allow ATS to be always on for CXL.cache capab…
f95b572 
…le devices

BugLink: https://bugs.launchpad.net/ubuntu/+source/linux-nvidia-6.17/+bug/2150727

Controlled by the IOMMU driver, ATS is usually enabled "on demand" when a
given PASID on a device is attached to an I/O page table. This is working
even when a device has no translation on its RID (i.e., the RID is IOMMU
bypassed).

However, certain PCIe devices require non-PASID ATS on their RID even when
the RID is IOMMU bypassed. Call this "always on".

For example, CXL spec r4.0 notes in sec 3.2.5.13 Memory Type on CXL.cache:
 "To source requests on CXL.cache, devices need to get the Host Physical
  Address (HPA) from the Host by means of an ATS request on CXL.io."

In other words, the CXL.cache capability requires ATS; otherwise, it can't
access host physical memory.

Introduce a new pci_ats_always_on() helper for the IOMMU driver to scan a
PCI device and shift ATS policies between "on demand" and "always on".

Add the support for CXL.cache devices first. Pre-CXL devices will be added
in quirks.c file.

Note that pci_ats_always_on() validates against pci_ats_supported(), so we
ensure that untrusted devices (e.g. external ports) will not be always on.
This maintains the existing ATS security policy regarding potential side-
channel attacks via ATS.

Cc: linux-cxl@vger.kernel.org
Suggested-by: Vikram Sethi <vsethi@nvidia.com>
Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Tested-by: Nirmoy Das <nirmoyd@nvidia.com>
Acked-by: Nirmoy Das <nirmoyd@nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
[Nirmoy: Adapt to already existing PCI_DVSEC_CXL_CACHE_CAPABLE.]
(backported from https://lore.kernel.org/r/f6734b9dad0050138676f11ecd14e9db1cf6b697.1777269009.git.nicolinc@nvidia.com)
Signed-off-by: Nirmoy Das <nirmoyd@nvidia.com>
@nicolinc @nirmoy
NVIDIA: VR: SAUCE: PCI: Allow ATS to be always on for pre-CXL devices
d4d63f9 
BugLink: https://bugs.launchpad.net/ubuntu/+source/linux-nvidia-6.17/+bug/2150727

Some NVIDIA GPU/NIC devices, though they don't implement CXL config space,
have many CXL-like properties. Call this kind "pre-CXL".

Similar to CXL.cache capability, these pre-CXL devices also require the ATS
function even when their RIDs are IOMMU bypassed, i.e. keep ATS "always on"
v.s. "on demand" when a non-zero PASID line gets enabled in SVA use cases.

Introduce pci_dev_specific_ats_always_on() quirk function to scan a list of
IDs for these devices. Then, include it in pci_ats_always_on().

Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Nirmoy Das <nirmoyd@nvidia.com>
Tested-by: Nirmoy Das <nirmoyd@nvidia.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
[Nirmoy: Apply after reverting older ATS always-on PCI quirk support.]
(backported from https://lore.kernel.org/r/1a8cf5e88051ab5c10417edb94df598ecbc810cf.1777269009.git.nicolinc@nvidia.com)
Signed-off-by: Nirmoy Das <nirmoyd@nvidia.com>
@nicolinc @nirmoy
NVIDIA: VR: SAUCE: iommu/arm-smmu-v3: Allow ATS to be always on
80c6442 
BugLink: https://bugs.launchpad.net/ubuntu/+source/linux-nvidia-6.17/+bug/2150727

When a device's default substream attaches to an identity domain, the SMMU
driver currently sets the device's STE between two modes:

  Mode 1: Cfg=Translate, S1DSS=Bypass, EATS=1
  Mode 2: Cfg=bypass (EATS is ignored by HW)

When there is an active PASID (non-default substream), mode 1 is used. And
when there is no PASID support or no active PASID, mode 2 is used.

The driver will also downgrade an STE from mode 1 to mode 2, when the last
active substream becomes inactive.

However, there are PCIe devices that demand ATS to be always on. For these
devices, their STEs have to use the mode 1 as HW ignores EATS with mode 2.

Change the driver accordingly:
  - always use the mode 1
  - never downgrade to mode 2
  - allocate and retain a CD table (see note below)

Note that these devices might not support PASID, i.e. doing non-PASID ATS.
In such a case, the ssid_bits is set to 0. However, s1cdmax must be set to
a !0 value in order to keep the S1DSS field effective. Thus, when a master
requires ats_always_on, set its s1cdmax to at least 1, meaning that the CD
table will have a dummy entry (SSID=1) that will never be used.

Now for these devices, arm_smmu_cdtab_allocated() will always return true,
v.s. false prior to this change. When its default substream is attached to
an IDENTITY domain, its first CD is NULL in the table, which is a totally
valid case. Thus, add "!master->ats_always_on" to the condition.

Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Tested-by: Nirmoy Das <nirmoyd@nvidia.com>
Acked-by: Nirmoy Das <nirmoyd@nvidia.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
[Nirmoy: Apply after reverting older ATS always-on arm-smmu-v3 support.]
(backported from https://lore.kernel.org/r/7403163ebf59380f88c7503b3adf0dae07428df8.1777269009.git.nicolinc@nvidia.com)
Signed-off-by: Nirmoy Das <nirmoyd@nvidia.com>
@nirmoy nirmoy force-pushed the my-ats-always-on-v4 branch from 520182f to 80c6442 Compare April 30, 2026 10:04
@nvmochs
Copy link
Copy Markdown
Collaborator

nvmochs commented Apr 30, 2026

@nirmoy I see this received a RB tag from DaveJ. Do we need to take this in now or can we wait a week to see if it lands in -next?

@nirmoy
Copy link
Copy Markdown
Collaborator Author

nirmoy commented Apr 30, 2026

@nirmoy I see this received a RB tag from DaveJ. Do we need to take this in now or can we wait a week to see if it lands in -next?

There is no rush, we can wait it to get merge in the -next

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@nirmoy @nvmochs @nicolinc