[Feature] Fleet: Auto-upgrade is not retroactive and does not reconcile clusters to latest patch version

[sphtd]

Is your feature request related to a problem? Please describe.

Auto-upgrade in Azure Kubernetes Fleet Manager is currently event-driven and not state-reconciliating.
If an auto-upgrade profile is created after a Kubernetes patch version has already been released, no updaterun is triggered. As a result, clusters can remain on outdated patch versions indefinitely until a new version is released.

Example:

Cluster version: 1.33.5
Available versions: 1.33.6, 1.33.7
Auto-upgrade profile created after 1.33.7 rollout
Result: no upgrade happens

Additionally:

If a cluster is stopped during maintenance window, the upgrade is missed and not retried
There is no mechanism to “catch up” to the latest allowed patch version
This leads to non-deterministic behavior and breaks the expectation of automatic patching.

Describe the solution you'd like

Auto-upgrade should behave as a desired state reconciliation system, not only event-driven.

Expected behavior:

When auto-upgrade profile is created or updated:
system evaluates current cluster version
if cluster is behind → trigger updaterun immediately
System should periodically reconcile:
if cluster version < latest allowed by policy → trigger upgrade
If upgrade fails (e.g. cluster stopped):
retry within maintenance window until successful

Optional configuration:

reconcileOnCreate: true
catchUpToLatestPatch: true
retryUntilSuccess: true
Describe alternatives you've considered

Current workarounds:
Manually triggering upgrades via CLI/API
Changing upgrade policy to force re-evaluation
Building custom automation (pipelines/scripts) to detect and upgrade outdated clusters

These approaches:
add operational overhead
defeat the purpose of “auto-upgrade”
introduce additional failure points

Additional context

Current behavior depends on:
timing of Kubernetes version rollout
timing of auto-upgrade profile creation
cluster availability during maintenance window

This makes the system non-deterministic:

two identical clusters may end up on different versions depending on timing
The system already supports manual updaterun triggering, which indicates that the capability exists, but is not used automatically for reconciliation.

Expected behavior for users:
clusters should converge to the latest allowed version defined by policy, regardless of when the profile was created or whether an event was missed.

[microsoft-github-policy-service]

@sjwaight would you be able to assist?

[serbrech]

Thank you for the detailed and thoughtful feedback. This is exactly the kind of input we are looking for to help evolve Fleet auto‑upgrade in the right direction!

Current auto‑upgrade behavior

The behavior you describe is accurate and intentional. Fleet auto‑upgrade is designed with a safety‑first principle.
Creating or updating an auto‑upgrade profile is deliberately passive, We avoid automatically triggering upgrades on profile creation because upgrades can't be rolled back easily, if at all.

Instead, automation takes effect from the next Kubernetes version rollout, and the explicit generate operation is available when users want to take action immediately. This is only necessary upon creation of the profile. This will be surfaced in the portal to improve the user experience too.

Similarly, upgrades failures due to stopped clusters or other blocking conditions are surfaced through updateruns rather than retried automatically. AKS does not support upgrade operation on stopped clusters, and automatically starting clusters can have significant customer impact (cost, quota usage/quota noisy neighbor, and any workload side-effect).
Skipping them by default would not be a considered safe, as we would leave cluster behind silently.

Two identical clusters may end up on different versions depending on timing

This is true for the period of time between auto-upgrade profile creation, and new kubernetes or node image rollout, which can be fixed by manually generating an updaterun, once.
Any later divergence would be related to customer constraints (strategy/maintenance window/region).

How we're addressing the gaps

In some cases, these defaults are not what a particular user expects, and we are iterating in this area.
The goal of Multi-Cluster upgrade in Fleet is to provide a system to converge the fleet to the defined configuration, but always within the realities of multi‑cluster upgrades. Ordering, scale, failure handling, and customer‑specific constraints all influence when and how upgrades can safely proceed.

We are actively iterating in this area:

• maxConcurrency settings allow better control over the cadence of upgrades, without imposing a complex grouping structure
• failureThreshold will allow to relax the failure conditions of the upgrade system
• We are considering further Retry options/apis, although in the vast majority of failures, the user needs to take an action before retrying.

User feedback like this is directly shaping those decisions and helps us determine where relaxing today’s safety constraints makes sense.