-
Notifications
You must be signed in to change notification settings - Fork 70
Open
Description
Summary
Add Cython compilation to the PyAirbyte build pipeline so that PyPI-published wheels contain compiled .so/.pyd extensions instead of readable .py source files, providing basic code obfuscation for distribution from the private repo.
Context
This pattern has been prototyped in the airbyte-ops-mcp and connector-builder-mcp repos:
- Ops MCP PR: https://github.com/airbytehq/airbyte-ops-mcp/pull/343
- Connector Builder MCP PR: https://github.com/airbytehq/connector-builder-mcp/pull/208
- Related ops-mcp issue: https://github.com/airbytehq/airbyte-ops-mcp/issues/346
The approach uses:
hatch-cythonizebuild hook (maintained fork of hatch-cython) to compile.py→.c→.soduring wheel builds- Build-time injection of Cython config (not stored in
pyproject.toml) to avoid breakinguv sync/ editable installs - Post-build source stripping to remove
.pyfiles from wheels where compiled.socounterparts exist - Exclusion of modules that break under Cython compilation (e.g., MCP/Pydantic presentation layers)
Key Learnings from Ops MCP Implementation
- Cython config must NOT be in
pyproject.toml— it runs during editable installs (uv sync), breaking CI workflows. Solution: inject at build time via a script. - Pydantic/FastMCP modules must be excluded — Cython-compiled functions (
cyfunction) don't expose type annotations the same way, breaking Pydantic schema generation. __init__.pyfiles must be excluded — required for Python package structure.- Platform-specific wheels — must build per OS (Linux, macOS) and Python version (3.10, 3.11, 3.12).
Action Items
- Evaluate whether this pattern should be applied to PyAirbyte
- If yes, adapt the
enable_cython_build.pyandstrip_source_from_wheel.pyscripts for PyAirbyte's package structure - Identify which modules (if any) need to be excluded from compilation
- Integrate with existing publish/release workflows
Devin session: https://app.devin.ai/sessions/7c8622f9452141d1bbad93f74079278f
Requested by: Aaron ("AJ") Steers (@aaronsteers)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels