feat(studio): multi-platform playground — Android + iOS + HarmonyOS + Computer#2371
Open
feat(studio): multi-platform playground — Android + iOS + HarmonyOS + Computer#2371
Conversation
Prepare the contract layer for supporting iOS, HarmonyOS and Computer alongside the existing Android runtime. No runtime behaviour changes — the legacy channel names are now aliases that resolve to the same generic channel string. - `PlaygroundBootstrap` type replaces `AndroidPlaygroundBootstrap` (kept as deprecated alias). - `PlaygroundRuntimeService` interface replaces `AndroidPlaygroundRuntimeService` (kept as deprecated alias). - `IPC_CHANNELS.getPlaygroundBootstrap` / `restartPlayground` are the canonical names; `getAndroidPlaygroundBootstrap` / `restartAndroidPlayground` now point to the same values. - `StudioRuntimeApi` gains the generic methods with `@deprecated` tags on the old ones. - electron-contract.test.ts updated to verify alias identity.
…ony + Computer) Replace the single-platform Android runtime service with a unified multi-platform runtime that registers all four device platforms: Android — ADB + scrcpy preview iOS — WebDriverAgent (manual host:port, mjpeg preview) HarmonyOS — HDC device discovery, screenshot preview Computer — local display enumeration, screenshot preview Architecture: - `multi-platform-runtime.ts` builds a `RegisteredPlaygroundPlatform[]` array from the four platform packages, each resolved lazily via `require.resolve` — missing packages are marked unavailable instead of crashing. - Calls `prepareMultiPlatformPlayground(platforms)` from the generic `@midscene/playground` infra to get a unified `sessionManager` that routes to the selected platform. - Launches a SINGLE HTTP server via `launchPreparedPlaygroundPlatform`; the renderer talks to this one server. - The platform selector UI (cards) is built into the playground's session setup form. Renderer changes: - `StudioPlaygroundProvider` calls the generic `getPlaygroundBootstrap` / `restartPlayground` IPC instead of the Android-specific ones. - `usePlaygroundController` no longer passes `defaultDeviceType` — the multi-platform session manager owns platform selection. - MainContent + Playground panel text de-Androidified: "Playground starting..." / "Runtime Error" / "Retry runtime" instead of hardcoded "Android" everywhere. - Context type gains `restartPlayground` alongside the deprecated `restartAndroidPlayground` alias. Build: - `package.json` adds `@midscene/ios`, `@midscene/harmony`, `@midscene/computer`, `@midscene/computer-playground` as deps. - `rsbuild.config.ts` marks all four platform packages as externals for the main process bundle. - `preload/index.ts` exposes the new IPC methods + legacy aliases.
Deploying midscene with
|
| Latest commit: |
1d0b8fd
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://58e67c24.midscene.pages.dev |
| Branch Preview URL: | https://feat-studio-multi-platform.midscene.pages.dev |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 648374008a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+100
to
+101
| prepare: async () => | ||
| harmonyPlaygroundPlatform.prepare({ staticDir: harmonyStaticDir }), |
There was a problem hiding this comment.
Registering Harmony with harmonyPlaygroundPlatform.prepare(...) wires Studio to the CLI-oriented @midscene/harmony flow, where prepare() calls selectDevice() and can invoke process.exit(1) when no HDC targets are found (packages/harmony/src/platform.ts). In the Electron main process this means selecting the Harmony card can block on a terminal prompt (multiple devices) or terminate the app (no devices) instead of returning a recoverable setup error, so Harmony support is effectively broken for normal GUI usage.
Useful? React with 👍 / 👎.
Previously only the Android section in the sidebar and overview had a
working onConnect / onClick handler. Other platform sections rendered
devices but clicking them was a no-op.
Generalize device click handling for the multi-platform session manager:
- Sidebar: replace the Android-specific `androidDevices` array with a
`buildDeviceItemsForPlatform(platformKey, devices)` factory that
produces onClick handlers for every platform section. Each handler
sets `platformId` + `{platformId}.deviceId` in the form and calls
`createSession` — matching the field prefixing convention from
`prepareMultiPlatformPlayground`.
- DeviceList overview: the `onConnect` handler now uses the `platform`
argument (was `_platform`) to set the same prefixed form fields.
- `selected` state is computed across all platforms (flattened device
id set) instead of Android-only.
The multi-platform server routes the session to the correct backend
based on `platformId`. Clicking an Android device works exactly as
before; clicking an iOS/Harmony/Computer device sets the right form
values and triggers a session create through the same unified path.
The multi-platform session manager requires a `platformId` in the form before it returns targets from `getSetupSchema`. Without it, the initial `refreshSessionSetup` poll gets a "Choose a platform" response with no targets, and the sidebar stays empty. Seed `platformId: 'android'` into the form via `useLayoutEffect` before the first poll fires. The controller's interval picks up the value on its next tick and returns Android targets immediately — matching the pre-multi-platform behaviour where devices appeared on boot. When the user clicks a different platform section in the sidebar, the onClick handler overrides `platformId` and triggers a fresh `refreshSessionSetup` with the new platform, swapping the device list.
The multi-platform session manager only returns targets for the currently selected platform. That left non-Android sidebar sections empty even when Harmony or Computer devices were connected. Add an independent device-discovery service that scans ALL platforms concurrently (ADB, HDC, display enumeration) and returns a flat DiscoveredDevice[] via a new IPC channel `studio:discover-devices`. Wiring: - Main: `device-discovery.ts` runs `scanAndroidDevices()`, `scanHarmonyDevices()`, `scanComputerDisplays()` via `Promise.allSettled` — each scan is isolated, failures on one platform don't block others. iOS is omitted (requires manual WDA). - Preload: exposes `discoverDevices()` on `StudioRuntimeApi`. - Renderer: `StudioPlaygroundProvider` polls `discoverDevices()` every 5 seconds. Results are bucketed by `platformId` into `DiscoveredDevicesByPlatform` and exposed on the context. - Sidebar: merges `discoveredDevices` into `deviceBuckets` so devices from ALL platforms appear simultaneously. Discovered devices that are already in the session-setup bucket (from the selected platform's `listTargets`) are deduplicated by id. Result: plug in an Android device AND an HDC device → both appear in their respective sidebar sections at the same time.
Two build errors: 1. `@midscene/android` was not in studio's package.json deps (only `@midscene/android-playground` was). TS couldn't resolve the import for `getConnectedDevicesWithDetails`. 2. Harmony's `getConnectedDevices()` returns `HarmonyDeviceInfo[]` (objects with `deviceId` field), not `string[]`. Updated the mapper to destructure `.deviceId`. Also add `@midscene/android` to the rsbuild externals list so the main bundle doesn't try to inline it.
When a non-Android device was connected, `buildGenericConnectedDeviceItem` used `runtimeInfo.title` (e.g. "Midscene HarmonyOS Playground") as the device label, since `metadata.sessionDisplayName` was unset and the function checked `.title` before `.metadata.deviceId`. Swap the priority: show the concrete `metadata.deviceId` (the serial number from `hdc list targets`) when available, fall back to `.title` only when neither `sessionDisplayName` nor `deviceId` is set. Also add Screen Recording permission note: dev-mode Electron binary at node_modules/electron/dist/Electron.app needs explicit macOS Screen Recording permission for the Computer platform to capture screenshots.
The computer platform stores its display id in `metadata.displayId` (not `metadata.deviceId`). `buildGenericConnectedDeviceItem` only checked `metadata.deviceId`, so after connecting to a computer display the connected entry got `id: "computer-connected"` while the discovery entry had `id: "0"` — no dedup match, both appeared in the sidebar. Check `metadata.displayId` as a fallback so the connected entry uses `id: "0"` and the discovery entry with the same id is suppressed.
The device label in the top bar was hardcoded to read from `resolveAndroidDeviceLabel`, which returned "No Android device selected" when connected to a non-Android platform (Computer, Harmony, iOS). Resolve the label from `runtimeInfo.metadata` first — check `sessionDisplayName`, then `deviceId`, then `displayId` — so any connected platform shows its actual device name. Fall back to the Android resolver for backward compat, and finally to a generic "No device selected" instead of the Android-specific string.
…ller Lets hosts seed the session-setup form on mount (e.g. to pre-select a default platform) so the first refreshSessionSetup poll lands on a real platform instead of the generic "Choose a platform" setup, without forcing the host into a post-mount setFieldsValue workaround.
- Delete legacy Android-only aliases: android-runtime.ts, AndroidPlaygroundBootstrap / AndroidPlaygroundRuntimeService types, legacy IPC_CHANNELS entries, and the restartAndroidPlayground StudioRuntimeApi / context field. - Drop unreachable unavailableReason branches in the runtime (workspace deps are always present) by driving registration from a table and letting resolveStaticDir fail fast. - Log per-platform scan failures in device-discovery via the shared getDebug logger instead of swallowing them; normalize the dynamic imports and fix DisplayInfo field usage (name/primary, not label/width). - Share StudioPlatformId + STUDIO_PLATFORM_IDS across main and renderer so bucketing can never silently drop a new platform. - Move bucketDiscoveredDevices and a new resolveConnectedDeviceLabel into selectors so the label logic is defined once and MainContent no longer re-reads runtime metadata by hand. - Seed the default platformId via usePlaygroundController's initialFormValues option, removing the post-mount setFieldsValue hack. - Gate device-discovery polling on the ready phase, flag sidebar placeholder rows with isPlaceholder so they can't get "selected", and surface an iOS setup hint when no iOS devices are discovered. - Tests: add bucket-discovered-devices.test.ts and resolveConnectedDeviceLabel cases; wire @main/@preload/@renderer/@shared aliases into vitest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Evolve Studio from Android-only to a multi-platform playground that supports Android, iOS, HarmonyOS, and Computer with real device connections. Web is excluded per design.
Replaces the single-platform `createAndroidPlaygroundRuntimeService()` with a unified `createMultiPlatformRuntimeService()` that registers all four platform descriptors and launches ONE server via `prepareMultiPlatformPlayground()` from `@midscene/playground`. The renderer talks to this single server; the built-in platform selector card UI routes to the correct backend.
Platform support matrix
Changes
Main process
Renderer
Build
Known limitations
Test plan