Make GCS optional for transfer elevation cache and cleanup#608
Open
jeremyzilar wants to merge 1 commit intostagingfrom
Open
Make GCS optional for transfer elevation cache and cleanup#608jeremyzilar wants to merge 1 commit intostagingfrom
jeremyzilar wants to merge 1 commit intostagingfrom
Conversation
When GCS_BUCKET_NAME is unset, get_cached_elevations() returns an empty dict instead of failing with 'Cannot determine path without bucket name'. dump_cached_elevations() and cleanup_locations() skip GCS operations in that case, allowing transfers to run locally without GCS credentials. Made-with: Cursor
Member
|
I'd like to minimize any modification or continued use of the transfer functionality. For this purpose I implemented PR #597. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes Google Cloud Storage (GCS) optional for the data transfer pipeline. When
GCS_BUCKET_NAMEis not set, the transfer runs without GCS instead of failing withValueError: Cannot determine path without bucket name.Problem
Running
python -m transfers.transferlocally (without GCS credentials) failed during well transfer initialization.WellTransferercallsget_cached_elevations()to load a precomputed elevation cache from GCS. IfGCS_BUCKET_NAMEis empty or unset, the GCS client creates a bucket object with no name, andblob.exists()raises:This blocked local development and testing of the transfer without GCS.
Solution
Guard all GCS usage in
well_transfer_util.pywith a check forGCS_BUCKET_NAME:get_cached_elevations()– Returns an empty dict when GCS is not configured. The transfer proceeds without a cache; elevation lookups may be slower or use defaults.dump_cached_elevations()– No-op when GCS is not configured. The in-memory cache is not persisted.cleanup_locations()– Skips GCS read/write when GCS is not configured. Location updates (state, county, quad_name) still run using in-memory lookups; only the GCS-backed cache is skipped.Behavior
GCS_BUCKET_NAMEsetGCS_BUCKET_NAMEunset or emptyTesting
python -m transfers.transferwithGCS_BUCKET_NAMEunset in.env; the transfer should complete successfully.GCS_BUCKET_NAMEset and valid credentials, the transfer should still use GCS for elevation and location cleanup caches.