add 'balance --max-moves-per-osd' option

[janveverka]

Adds the option --max-moves-per-osd=N to the balance command:

  --max-moves-per-osd MAX_MOVES_PER_OSD
                        only consider moves between OSDs with fewer movements
                        than the given maximum. This includes movements that are
                        both ongoing and planned. It applies to the movement
                        source OSDs as well as the destination ones.

Using this option ensures that the found movements would not cause the number of simultaneously ongoing moves per OSD to be greater than the given MAX_MOVES_PER_OSD.

This helps with making the movements more spread out across the cluster and avoiding overload of some OSDs to a point that it negatively affects clients. This may be useful on some heavily loaded mission critical clusters, especially when running on spinning drives.

[TheJJ]

hey! great idea, this surely is useful.

i think the implementation can be optimized a bit :)
your code basically does explicit tracking, but this for example disregards movements that are un-done in the process. this also is something that changes during optimization, so it should be part of the simulated cluster state PGMappings.

so it would be better to add the query of planned OSD moves as a feature/new function to PGMappings.
the number of planned moves is simply

@lru_cache(max=...)
def get_osd_remaps_num(self, osdid: int):
    return len(self._osd_pgs_up[osdid] ^ self._osd_pgs_acting[osdid])

that can be returned from a cached function (which is cache-reset when a new move is recorded).

[janveverka]

@TheJJ , thanks for the feedback! I'll look into optimizing it.

[TheJJ]

did you have a chance to look at this again? :)

[janveverka]

Not yet, sorry. It's still on my radar though. :-)