-
Notifications
You must be signed in to change notification settings - Fork 70
Description
Summary
Ensure that stream mappers (hashing, field-renaming, row-filtering, encryption, field-filtering) are properly handled during cross-workspace connection migration workflows.
Requested by: Aaron ("AJ") Steers (@aaronsteers)
Related Devin session: https://app.devin.ai/sessions/eb636b8d14fc4895b3b894ea399f998e
Related issues:
- feat: Enhance MCP tools for cross-workspace connector migration #978 (parent migration issue)
- Cross-linked Ops MCP issue: TBD (will be created and linked)
Current State
What works today
Mappers are preserved during catalog dump/import via the existing tools:
-
Catalog dump (
dump_raw_catalog()→web_backend/connections/get): The API response includesmappersper stream in thesyncCatalog, with each mapper havingtypeandmapperConfiguration. -
Catalog import (
import_raw_catalog()→web_backend/connections/update): The platform'sCatalogConverter.toConfiguredInternal()correctly converts mappers from API model to internal model (lines 305-312 ofCatalogConverter.kt), preservingtypeandmapperConfiguration.
So the basic mapper round-trip (dump from source connection → import to target connection) does work for mappers without secrets.
What does NOT work
-
Mapper secrets are masked on dump: The platform's
MapperSecretHelper.maskMapperSecrets()masks secret fields (e.g., encryption keys) before returning the catalog via the API. This means:- When dumping a catalog from the source connection, mapper secrets (e.g., AES encryption keys) will be replaced with
"**********"or similar masked values - When importing the dumped catalog to the target connection, the masked values won't work
- The user would need to manually re-enter mapper secrets for the target connection
- When dumping a catalog from the source connection, mapper secrets (e.g., AES encryption keys) will be replaced with
-
No MCP tools for mapper management: Neither Coral MCP nor Ops MCP has any tools for:
- Listing mappers configured on a connection
- Adding/removing/updating individual mappers
- Validating mappers against the target connection's streams (the platform has a
/v1/web_backend/mappers/validateendpoint, but it's not exposed via MCP)
-
No mapper compatibility validation: When migrating, there's no check that the target workspace/platform version supports all the mapper types used by the source connection.
Supported Mapper Types
From the platform's StreamMapperType enum:
| Mapper Type | Has Secrets? | Migration Concern |
|---|---|---|
hashing |
No | Should migrate cleanly |
field-renaming |
No | Should migrate cleanly |
row-filtering |
No | Should migrate cleanly |
field-filtering |
No | Should migrate cleanly |
encryption |
Yes (encryption keys) | Secrets will be masked, need re-entry |
Proposed Improvements (P2 - not blocking initial migration)
1. Document mapper secret limitation
At minimum, document that mapper secrets (currently only encryption mappers) will need to be manually re-entered after migration.
2. Add mapper listing to describe_cloud_connection
Enhance the describe_cloud_connection MCP tool to include a summary of configured mappers per stream (type + non-secret config), so users can see what needs attention.
3. Add mapper validation tool
Expose the platform's /v1/web_backend/mappers/validate endpoint as an MCP tool to validate that mappers from a dumped catalog are compatible with the target connection.
Priority
P2 — Not blocking for initial migration. Basic mapper preservation works via catalog dump/import. Only encryption mapper secrets need manual re-entry, which is an edge case for most migrations.