Skip to content

feat: Support mapper/mapping preservation in cross-workspace connection migration #980

@devin-ai-integration

Description

@devin-ai-integration

Summary

Ensure that stream mappers (hashing, field-renaming, row-filtering, encryption, field-filtering) are properly handled during cross-workspace connection migration workflows.

Requested by: Aaron ("AJ") Steers (@aaronsteers)
Related Devin session: https://app.devin.ai/sessions/eb636b8d14fc4895b3b894ea399f998e
Related issues:


Current State

What works today

Mappers are preserved during catalog dump/import via the existing tools:

  1. Catalog dump (dump_raw_catalog()web_backend/connections/get): The API response includes mappers per stream in the syncCatalog, with each mapper having type and mapperConfiguration.

  2. Catalog import (import_raw_catalog()web_backend/connections/update): The platform's CatalogConverter.toConfiguredInternal() correctly converts mappers from API model to internal model (lines 305-312 of CatalogConverter.kt), preserving type and mapperConfiguration.

So the basic mapper round-trip (dump from source connection → import to target connection) does work for mappers without secrets.

What does NOT work

  1. Mapper secrets are masked on dump: The platform's MapperSecretHelper.maskMapperSecrets() masks secret fields (e.g., encryption keys) before returning the catalog via the API. This means:

    • When dumping a catalog from the source connection, mapper secrets (e.g., AES encryption keys) will be replaced with "**********" or similar masked values
    • When importing the dumped catalog to the target connection, the masked values won't work
    • The user would need to manually re-enter mapper secrets for the target connection
  2. No MCP tools for mapper management: Neither Coral MCP nor Ops MCP has any tools for:

    • Listing mappers configured on a connection
    • Adding/removing/updating individual mappers
    • Validating mappers against the target connection's streams (the platform has a /v1/web_backend/mappers/validate endpoint, but it's not exposed via MCP)
  3. No mapper compatibility validation: When migrating, there's no check that the target workspace/platform version supports all the mapper types used by the source connection.


Supported Mapper Types

From the platform's StreamMapperType enum:

Mapper Type Has Secrets? Migration Concern
hashing No Should migrate cleanly
field-renaming No Should migrate cleanly
row-filtering No Should migrate cleanly
field-filtering No Should migrate cleanly
encryption Yes (encryption keys) Secrets will be masked, need re-entry

Proposed Improvements (P2 - not blocking initial migration)

1. Document mapper secret limitation

At minimum, document that mapper secrets (currently only encryption mappers) will need to be manually re-entered after migration.

2. Add mapper listing to describe_cloud_connection

Enhance the describe_cloud_connection MCP tool to include a summary of configured mappers per stream (type + non-secret config), so users can see what needs attention.

3. Add mapper validation tool

Expose the platform's /v1/web_backend/mappers/validate endpoint as an MCP tool to validate that mappers from a dumped catalog are compatible with the target connection.


Priority

P2 — Not blocking for initial migration. Basic mapper preservation works via catalog dump/import. Only encryption mapper secrets need manual re-entry, which is an edge case for most migrations.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions