Skip to content

Improve deployment orchestrator performance#1984

Merged
shangyian merged 1 commit intoDataJunction:mainfrom
shangyian:optimize-deploy
Apr 10, 2026
Merged

Improve deployment orchestrator performance#1984
shangyian merged 1 commit intoDataJunction:mainfrom
shangyian:optimize-deploy

Conversation

@shangyian
Copy link
Copy Markdown
Collaborator

@shangyian shangyian commented Apr 9, 2026

Summary

Reduce database interaction in deployment orchestrator

  • Reduce DB loads for tags/owners to only the fields that are needed and skip unused relationships.
  • Copy fast-path: skips bulk_validate_node_data entirely and uses pre-validated columns (this eliminates ANTLR4 type re-parsing for branch copies)
  • Bulk flush per level when deploying nodes per level and per-dim link flush eliminated
  • Reduce export load options: new Node.export_load_options() / NodeRevision.export_load_options() skip
    materializations, availability, and created_by
  • Skip redundant session.refresh for owners

Column type caching

  • Column type is now cached with lru_cache(maxsize=1024) so that repeated ANTLR4 parses for the same type
    string (e.g., bigint, float) are eliminated
  • We pre-populate the cache with common primitive types at server startup, so DB loads for routine types cost zero ANTLR4 calls

Metrics instrumentation

Instrument some metrics for deployment

Test Plan

  • PR has an associated issue: #
  • make check passes
  • make test shows 100% unit test coverage

Deployment Plan

@netlify
Copy link
Copy Markdown

netlify bot commented Apr 9, 2026

Deploy Preview for thriving-cassata-78ae72 canceled.

Name Link
🔨 Latest commit 7402716
🔍 Latest deploy log https://app.netlify.com/projects/thriving-cassata-78ae72/deploys/69d83cbc0fec9a0008eae29f

@shangyian shangyian changed the title Optimize deploy Improve deployment orchestrator performance Apr 9, 2026
@shangyian shangyian marked this pull request as ready for review April 10, 2026 00:22
@shangyian shangyian merged commit d50f437 into DataJunction:main Apr 10, 2026
17 checks passed
@shangyian shangyian deleted the optimize-deploy branch April 10, 2026 00:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant