Multiple sequential MCD replica reduction repeatedly picks the same machine for deletion

[aaronfern]

How to categorize this issue?

/area auto-scaling
/kind bug
/priority 3

What happened:
We recently identified a bug where when there are multiple sequential MCD replica scale downs by an external actor (for example cluster-autoscaler), we noticed that the MCS controller repeatedly picks the same machine for deletion.
This causes problems, specially with how the external actor processed the scale down. For example, when using cluster-autoscaler with the mcm provider, cluster-autoscaler will cordon nodes that it has marked for termination. This leads to cluster states with multiple cordoned nodes that that are not deleted.

What you expected to happen:
When selecting a node for deletion due to reducing replicas, MCM should not select nodes already in a terminating state.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• Others:

[takoverflow]

Closed with #1086 and #1087

[aaronfern]

Since #1095 extends #1087, should this issue not track that too?

[takoverflow]

Since it was a follow-up cleanup PR, I thought linking it to the original PR itself should be sufficient. But you're right maybe the issue should stay open till that PR goes in as well.

/reopen

[gardener-prow]

@takoverflow: Reopened this issue.

Details

In response to this:

Since it was a follow-up cleanup PR, I thought linking it to the original PR itself should be sufficient. But you're right maybe the issue should stay open till that PR goes in as well.

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.