refactor(common): modularize Js UDF handling with plan an exec phases#1900
Draft
refactor(common): modularize Js UDF handling with plan an exec phases#1900
Conversation
LNSD
changed the title
LNSD
changed the title
LNSD
force-pushed
the
lnsd/refactor-common-attach-js-udfs
branch
from
60a7b4b to
5325a07
Compare
LNSD
force-pushed
the
lnsd/refactor-common-attach-js-udfs
branch
from
5325a07 to
9181fc8
Compare
LNSD
force-pushed
the
lnsd/refactor-common-attach-js-udfs
branch
2 times, most recently
from
e70a4ff to
2c2b6d1
Compare
Decouple `datasets-derived` from `ScalarUDF` and `js-runtime` by moving Arrow type validation to the deserialization boundary, so a successfully deserialized `Function` is always valid. - Move `Function`/`FunctionSource` from `datasets-common` into `datasets-derived` with custom `Deserialize` that validates Arrow types against JS UDF-supported primitives - Remove `js-runtime` dependency from `datasets-derived`; `Dataset::function_by_name` returns `&Function` instead of constructing `ScalarUDF` - Simplify `SelfSchemaProvider::from_manifest_udfs` by removing redundant `schema_name` parameter Signed-off-by: Lorenzo Delgado <lorenzo@edgeandnode.com>
Decouple IsolatePool from catalog resolution by splitting JS UDF handling into planning and execution phases, moving runtime resource attachment to service initialization. - Introduce `udfs` module with `PlanJsUdf` (planning) and `ExecJsUdf` (execution) types - Remove `IsolatePool` from `AmpCatalogProvider`, pass it at service/worker level - Move JS UDF attach logic from `DetachedLogicalPlan` into `ExecContext` - Enhance `Function` type validation with separate input/output type checks - Denormalize `FunctionSource` fields into `PlanJsUdf` for direct field access Signed-off-by: Lorenzo Delgado <lorenzo@edgeandnode.com>
LNSD
force-pushed
the
lnsd/refactor-common-attach-js-udfs
branch
from
2c2b6d1 to
e6fb678
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Separate UDF lifecycle into plan-time placeholders and exec-time bindings so the
V8 isolate pool is only needed at execution, not during schema resolution.
udfs::plan::PlanJsUdfandudfs::exec::ExecJsUdfto split planning from executionExecContext::attach()to rewrite plan UDFs into executable ones in a single traversalIsolatePoolfromAmpCatalogProviderintoExecContextdatasets-derivedfunctions intofunctions.rswith builder patternjs-runtimedependency fromadmin-apianddatasets-derived