[Bug]: 500 server error when re-applying replica groups service

[r4victor]

Steps to reproduce

1. Have an old replica group service submitted before Set explicit GPU defaults in ResourcesSpec and improve default GPU vendor selection #3573
2. Re-apply the same service with the same name – the server will fail with 500 and the following log:

  Top-level `resources` is not allowed when `replicas` is a list. Specify `resources` in each replica group instead. (type=value_error)
__root__
  Missing configuration (type=value_error)

Actual behaviour

The problem is the old run can no longer pass the replica groups check because default resources changed in #3573

dstack/src/dstack/_internal/core/models/configurations.py

Lines 1011 to 1018 in 8657913

	resources = values.get("resources")
	default_resources = ResourcesSpec()
	if resources and resources.dict() != default_resources.dict():
	raise ValueError(
	"Top-level `resources` is not allowed when `replicas` is a list. "
	"Specify `resources` in each replica group instead."
	)

Expected behaviour

Although we can handle the error, the check itself is very fragile since it depends on default ResourcesSpec() staying the same.

dstack version

master

Server logs

File "/Users/r4victor/Projects/dstack/dstack/src/dstack/_internal/server/routers/runs.py", line 132, in get_plan
    run_plan = await runs.get_plan(
               ^^^^^^^^^^^^^^^^^^^^
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/_internal/server/services/runs/__init__.py", line 347, in get_plan
    current_resource = await get_run_by_name(
                       ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/_internal/server/services/runs/__init__.py", line 297, in get_run_by_name
    return run_model_to_run(run_model, return_in_api=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/_internal/server/services/runs/__init__.py", line 757, in run_model_to_run
    run_spec = get_run_spec(run_model)
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/_internal/server/services/runs/__init__.py", line 117, in get_run_spec
    return RunSpec.__response__.parse_raw(run_model.run_spec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic/main.py", line 572, in pydantic.main.BaseModel.parse_raw
  File "pydantic/main.py", line 549, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 364, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 2 validation errors for RunSpecResponse
configuration -> ServiceConfigurationResponse -> __root__
  Top-level `resources` is not allowed when `replicas` is a list. Specify `resources` in each replica group instead. (type=value_error)
__root__
  Missing configuration (type=value_error)

Additional information

No response

[r4victor]

It seems services with replica groups not working at all after #3573 – not only re-apply. When applying a service I get CLI error:

Traceback (most recent call last):
  File "/Users/r4victor/Projects/dstack/dstack/.venv/bin/dstack", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/_internal/cli/main.py", line 95, in main
    args.func(args)
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/_internal/cli/commands/apply.py", line 98, in _command
    configurator.apply_configuration(
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/_internal/cli/services/configurators/run.py", line 105, in apply_configuration
    run_plan = self.api.runs.get_run_plan(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/api/_public/runs.py", line 520, in get_run_plan
    run_plan = self._api_client.runs.get_plan(self._project, run_spec)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/r4victor/Projects/dstack/dstack/src/dstack/api/server/_runs.py", line 80, in get_plan
    return parse_obj_as(RunPlan.__response__, resp.json())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic/tools.py", line 38, in pydantic.tools.parse_obj_as
  File "pydantic/main.py", line 364, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 2 validation errors for ParsingModel[RunPlanResponse]
__root__ -> effective_run_spec -> configuration -> ServiceConfigurationResponse -> __root__
  Top-level `resources` is not allowed when `replicas` is a list. Specify `resources` in each replica group instead. (type=value_error)
__root__ -> effective_run_spec -> __root__
  Missing configuration (type=value_error)

Apparently, this is because server modifies the resources by setting gpu vendor but the replica group check relies on default resources being unmodified.