fix(source-hubspot): use v3 properties API for CRM search streams and fix test cache leaks#74097
Draft
suisuixia42 wants to merge 6 commits intomasterfrom
Draft
fix(source-hubspot): use v3 properties API for CRM search streams and fix test cache leaks#74097suisuixia42 wants to merge 6 commits intomasterfrom
suisuixia42 wants to merge 6 commits intomasterfrom
Conversation
The v2 properties API (/properties/v2/{entity}/properties) returns all
properties including those with high data sensitivity. When these
sensitive properties are passed in the CRM search request body, HubSpot
returns 403 MISSING_SCOPES requiring granular scopes like
`tickets.highly_sensitive.v2` or `crm.objects.deals.sensitive.read.v2`.
These scopes are not available on legacy OAuth apps and require HubSpot
review for new-style apps.
Switch CRM search streams to discover properties via the v3 API
(/crm/v3/properties/{entity}) which excludes sensitive properties by
default, preventing the 403 errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Run ruff format to satisfy pre-commit format check. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…che, and fix mock_v3_properties URL Co-Authored-By: suisui.xia <suisui.xia@airbyte.io>
Contributor
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
|
Contributor
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
Contributor
|
Deploy preview for airbyte-docs ready! ✅ Preview Built with commit 5e91cd6. |
Contributor
|
…est_streams.py Co-Authored-By: suisui.xia <suisui.xia@airbyte.io>
devin-ai-integration
bot
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR targets the following PR:
What
CRM search streams (
companies,contacts,deals,tickets,leads,deal_splits,engagements_*) usePropertiesFromEndpointto discover HubSpot properties for search requests. The v2 API (/properties/v2/{entity}/properties) returns all properties including those withdataSensitivity: "high", which causes 403 MISSING_SCOPES errors on Enterprise accounts with sensitive custom properties.This PR (based on #74032 by hlepouse-vasco) switches to the v3 API which excludes sensitive properties by default, and adds test infrastructure to prevent HTTP cache leaks between tests.
How
Core fix (from #74032):
manifest.yaml: SwitchPropertiesFromEndpointpath from/properties/v2/{entity}/propertiesto/crm/v3/properties/{entity}, updatefield_pathfrom[]to["results"](v3 wraps response array)Test infrastructure (from #74032):
mock_dynamic_schema_requests,mock_dynamic_schema_requests_with_skip,mock_properties,mock_all_properties) now register both v2 and v3 mocksCrmSearchPropertiesRequestBuilderfor v3 property URLtest_source.pyandtest_streams.pyupdated to register v3 mocksAdditional changes on top of #74032:
conftest.py: Addedimport requests_cacheandclear_http_cacheautouse fixture that clears the CDK's shared in-memory SQLite HTTP cache before each test (preventsrequests_cachefrom bypassingrequests_mock)conftest.py: Changedmock_v3_propertieshelper URL from relative (/crm/v3/properties/{entity}) to absolute (https://api.hubapi.com/crm/v3/properties/{entity})test_streams.py: Changeddeal_splittest mocks from relative to absolute URLs forendpoint_path(/crm/v3/objects/deal_split/search→https://api.hubapi.com/...) andproperties_path(/properties/v2/deal_split/properties→https://api.hubapi.com/...). Without full URLs, theclear_http_cachefixture exposes aNoMockAddresserror that was previously hidden by cached responses from earlier tests.Review guide
conftest.pyclear_http_cache fixture —autouse=True(runs for every test). Verify this doesn't introduce unexpected side effects or slowdowns for tests that don't use HTTP caching.conftest.pymock_v3_properties — Uses full URL (https://api.hubapi.com/...) withregister_uri. Tests that call bothmock_v3_propertiesandmock_dynamic_schema_requestswill register the v3 mock twice. Verify double-registration doesn't cause issues withrequests_mock.test_streams.pydeal_split tests — Now use full URLs while some other test mocks still use relative URLs. This is intentional (theclear_http_cachefixture surfaces the mismatch for deal_split tests specifically), but verify no other tests have the same latent relative-URL issue that could surface under different test ordering.integrations/__init__.pymock_properties v3 response construction — The v3 response is built by starting withv3_records = copy.deepcopy(templates)and then appending built records. This meansv3_recordscontains both the original template records AND the newly built records. Verify this produces the correct JSON structure for the v3 API response format{"results": [...]}. The v2 path usesRootHttpResponseBuilderwhich may handle templates differently.manifest.yamlPropertiesFromEndpoint — Core fix: v2 → v3 API path change. Verify this correctly excludes sensitive properties and that the v3 response format{"results": [...]}matches actual API behavior.integrations/__init__.pymock_all_properties — Now checksexisting_v2andexisting_v3separately to avoid double-registration. Verify the logic correctly identifies already-mocked entities for both v2 and v3 URLs.User Impact
Users with HubSpot Enterprise accounts who have marked custom properties as sensitive will no longer experience 403 errors on CRM search streams. Backward-compatible — accounts without sensitive properties are unaffected.
Can this PR be safely reverted and rolled back?
No state/config format changes. Safe to revert.
Requested by: suisuixia42
Link to Devin run