chore(base-image): Migrate Konflux builds to UBI9/RHEL9 by davdhacs · Pull Request #2562 · stackrox/scanner

davdhacs · 2026-01-15T04:28:55Z

Description

Migrates scanner builds from UBI8/RHEL8 to UBI9/RHEL9 base images.

Key Changes

Konflux Base Images:

Builder: brew.registry.redhat.io/rh-osbs/openshift-golang-builder:rhel_9_1.24
Scanner runtime: registry.access.redhat.com/ubi9-minimal:latest
Scanner DB: registry.redhat.io/rhel9/postgresql-15:latest
Image names: rhacs-scanner-rhel9, rhacs-scanner-slim-rhel9, rhacs-scanner-db-rhel9, rhacs-scanner-db-slim-rhel9

Non-Konflux Base Images:

Scanner: ubi9-minimal
Scanner DB: ubi9 / ubi9-minimal
Vulnerabilities: ubi9-minimal

RPM/Repo Updates:

rpms.lock.yaml: xz updated from RHEL 8 (5.2.4) to RHEL 9 (5.2.5)
rpms.rhel.repo: repos updated from rhel8 to rhel9
PostgreSQL download script: pg_rhel_major=9

Tekton Labels:

CPE labels: el8 → el9

UBI9 Compatibility Fixes:

update-ca-trust extract -o /etc/pki/ca-trust/extracted for unprivileged containers (RHBZ#2241240)
cp --recursive --no-dereference --no-clobber in restore-all-dir-contents
microdnf install -y xz (explicit -y flag)

Checklist

Investigated and inspected CI test results

Testing Performed

TBD

openshift-ci · 2026-01-15T04:29:19Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

davdhacs · 2026-01-27T14:57:14Z

@tommartensen fyi this is the parallel scanner update for UBI9. In this, we also have to add the update-ca-trust workaround for UBI9 changed perms on second execution.

The arm build fails because tar hits a bug in a syscall in qemu; I still have some debug logging around that, and I expect the arm arch build to fail because of it.

davdhacs · 2026-02-18T05:39:24Z

/test all

Migrate all scanner base images from UBI8/RHEL8 to UBI9/RHEL9: Konflux base images: - Builder: openshift-golang-builder:rhel_9_golang_1.25 - Scanner runtime: ubi9-minimal - Scanner DB: rhel9/postgresql-15 Non-Konflux base images: - Scanner: ubi9-minimal - Scanner DB: ubi9 / ubi9-minimal - Vulnerabilities: ubi9-minimal Other updates: - Tekton CPE labels: el8 -> el9 - rpms.lock.yaml: xz updated to RHEL 9 version (5.2.5-8.el9_0) - rpms.rhel.repo: RHEL 8 repos -> RHEL 9 repos - PostgreSQL download script: pg_rhel_major=9 - Image name labels: rhel8 -> rhel9 - microdnf install -y xz (explicit -y for UBI9 compatibility) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

In UBI9, update-ca-trust fails when running as an unprivileged user (nobody:nobody) because it attempts to write to system-wide paths. Use the -o flag to specify a user-writable output directory. Also switch restore-all-dir-contents to use --no-clobber to avoid overwriting CA trust files that were already updated at runtime. See: https://bugzilla.redhat.com/show_bug.cgi?id=2241240 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

davdhacs · 2026-02-18T21:32:25Z

/retest

davdhacs · 2026-02-19T00:07:01Z

@tommartensen could you review this? (the ubi9 upgrade for scanner(v2); includes the update-ca-trust permissions workaround)

tommartensen

LGTM to me from a Konflux perspective. Have you deployed and smoke tested the resulting images (like we did for collector)?

davdhacs · 2026-02-19T14:44:43Z

LGTM to me from a Konflux perspective. Have you deployed and smoke tested the resulting images (like we did for collector)?

Yes, I tested with these changes (minus the master-merge [empty] commits). I'll re-do the smoke test with this and the latest master collector builds.

BradLugo · 2026-02-23T17:06:39Z

Note to self and @stackrox/scanner: we need to follow up on #2092 and update the image for s390x to avoid using a different version of psql.

mclasmeier · 2026-03-03T13:49:09Z

image/scanner/scripts/import-additional-cas

-update-ca-trust extract
+# The -o flag is required for running as an unprivileged user in containers.
+# See: https://bugzilla.redhat.com/show_bug.cgi?id=2241240
+update-ca-trust extract -o /etc/pki/ca-trust/extracted


I'd like to understand this better.
Is this path /etc/pki/ca-trust/extracted referenced somewhere? Is it defaulted to somewhere?

I think it's self-referential: https://src.fedoraproject.org/rpms/ca-certificates/blob/rawhide/f/update-ca-trust#_129-138

/etc/pki/ca-trust/extracted is the default. We're re-setting it to get the non-root user operation of setting a -o path.
Here you can see it hardcoded in the version of update-ca-trust in ubi9-minimal: https://pkgs.devel.redhat.com/cgit/rpms/ca-certificates/tree/update-ca-trust?h=rhel-9.4.0#n10
And this explains why we need to use the -o arg: https://pkgs.devel.redhat.com/cgit/rpms/ca-certificates/tree/update-ca-trust?h=rhel-9.4.0#n88
long-version:
In UBI8, the update-ca-trust trust extract execution did not change the permissions on the files it wrote. In the version in UBI9, the directory-hash output was added and write permissions are removed when it is written.
The -o(USER_DEST) arg was added to allow running as non-root because it does a chmod +/- around adding links into the directory-hash directory: https://src.fedoraproject.org/rpms/ca-certificates/pull-request/8#request_diff
Looking forward in https://src.fedoraproject.org/rpms/ca-certificates/c/7dc60cb it was reworked but changes it so the symlinks are only added when the USER_DIR is not specified (so we will still need the -o arg if the ca-certificates package updates are backported).

example usage: here this path is in the search paths for golang ssl: https://go.dev/src/crypto/x509/root_linux.go

more information in the man page, https://www.linux.org/docs/man8/update-ca-trust.html

Far future / TODO: The cert setup could be re-worked for these images so that it does not rely on update-ca-trust to add custom certs. Maybe that part isn't needed anymore, but I think that reaches outside of the scope of upgrading to UBI9. There was discussion about init container instead of the double-run that sensor and scanner do, and there are other options especially when we eventually switch to the golang built-in ssl.

Thank you @davdhacs

…migration

…migration # Conflicts: # image/db/rhel/konflux.Dockerfile

davdhacs · 2026-03-09T16:32:46Z

/test ?

davdhacs · 2026-03-10T04:25:22Z

The test failures are not-related to UBI9/changes in this PR.

davdhacs · 2026-03-10T04:25:34Z

#2888
test fix pr

…migration # Conflicts: # image/db/rhel/konflux.Dockerfile

davdhacs · 2026-03-10T20:51:26Z

/retest e2e-tests

davdhacs · 2026-03-10T23:33:48Z

/test e2e-tests

davdhacs · 2026-03-11T01:06:46Z

/test e2e-tests

davdhacs · 2026-03-11T03:47:19Z

/test e2e-tests

openshift-ci · 2026-03-11T04:41:58Z

@davdhacs: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-tests	`f65ee67`	link	false	`/test e2e-tests`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

davdhacs added the konflux-build Run Konflux in PR. Push commit to trigger it. label Jan 15, 2026

openshift-ci bot added the do-not-merge/work-in-progress label Jan 15, 2026

davdhacs force-pushed the konflux-ubi9-rhel9-migration branch from d9cdc31 to d2f149d Compare February 5, 2026 16:58

davdhacs force-pushed the konflux-ubi9-rhel9-migration branch from 34c5d67 to 5a2e599 Compare February 18, 2026 05:37

davdhacs marked this pull request as ready for review February 18, 2026 05:39

davdhacs requested review from a team and rhacs-bot as code owners February 18, 2026 05:39

openshift-ci bot removed the do-not-merge/work-in-progress label Feb 18, 2026

rhacs-bot requested a review from a team February 18, 2026 05:48

davdhacs and others added 2 commits February 18, 2026 14:01

davdhacs force-pushed the konflux-ubi9-rhel9-migration branch from 5a2e599 to b3addf3 Compare February 18, 2026 21:07

empty commit

46d33f6

davdhacs requested a review from tommartensen February 19, 2026 00:06

tommartensen reviewed Feb 19, 2026

View reviewed changes

BradLugo approved these changes Feb 23, 2026

View reviewed changes

mclasmeier reviewed Mar 3, 2026

View reviewed changes

davdhacs added 2 commits March 5, 2026 13:22

Merge remote-tracking branch 'origin/master' into konflux-ubi9-rhel9-…

1ec1ed6

…migration

Merge remote-tracking branch 'origin/master' into konflux-ubi9-rhel9-…

64b335a

…migration # Conflicts: # image/db/rhel/konflux.Dockerfile

Merge remote-tracking branch 'origin/master' into konflux-ubi9-rhel9-…

f65ee67

…migration # Conflicts: # image/db/rhel/konflux.Dockerfile

Conversation

davdhacs commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Key Changes

Checklist

Testing Performed

Uh oh!

openshift-ci bot commented Jan 15, 2026

Uh oh!

davdhacs commented Jan 27, 2026

Uh oh!

davdhacs commented Feb 18, 2026

Uh oh!

davdhacs commented Feb 18, 2026

Uh oh!

davdhacs commented Feb 19, 2026

Uh oh!

tommartensen left a comment

Choose a reason for hiding this comment

Uh oh!

davdhacs commented Feb 19, 2026

Uh oh!

BradLugo commented Feb 23, 2026

Uh oh!

mclasmeier Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

BradLugo Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

davdhacs Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

davdhacs Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

davdhacs Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

davdhacs Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

mclasmeier Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

davdhacs commented Mar 9, 2026

Uh oh!

davdhacs commented Mar 10, 2026

Uh oh!

davdhacs commented Mar 10, 2026

Uh oh!

davdhacs commented Mar 10, 2026

Uh oh!

davdhacs commented Mar 10, 2026

Uh oh!

davdhacs commented Mar 11, 2026

Uh oh!

davdhacs commented Mar 11, 2026

Uh oh!

openshift-ci bot commented Mar 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

davdhacs commented Jan 15, 2026 •

edited

Loading