Skip to content

Make GCS optional for transfer elevation cache and cleanup#608

Open
jeremyzilar wants to merge 1 commit intostagingfrom
FIX-GCS_BUCKET_NAME
Open

Make GCS optional for transfer elevation cache and cleanup#608
jeremyzilar wants to merge 1 commit intostagingfrom
FIX-GCS_BUCKET_NAME

Conversation

@jeremyzilar
Copy link

Summary

Makes Google Cloud Storage (GCS) optional for the data transfer pipeline. When GCS_BUCKET_NAME is not set, the transfer runs without GCS instead of failing with ValueError: Cannot determine path without bucket name.

Problem

Running python -m transfers.transfer locally (without GCS credentials) failed during well transfer initialization. WellTransferer calls get_cached_elevations() to load a precomputed elevation cache from GCS. If GCS_BUCKET_NAME is empty or unset, the GCS client creates a bucket object with no name, and blob.exists() raises:

ValueError: Cannot determine path without bucket name.

This blocked local development and testing of the transfer without GCS.

Solution

Guard all GCS usage in well_transfer_util.py with a check for GCS_BUCKET_NAME:

  • get_cached_elevations() – Returns an empty dict when GCS is not configured. The transfer proceeds without a cache; elevation lookups may be slower or use defaults.
  • dump_cached_elevations() – No-op when GCS is not configured. The in-memory cache is not persisted.
  • cleanup_locations() – Skips GCS read/write when GCS is not configured. Location updates (state, county, quad_name) still run using in-memory lookups; only the GCS-backed cache is skipped.

Behavior

Scenario Before After
GCS_BUCKET_NAME set Uses GCS for elevation cache and location cleanup cache Same behavior
GCS_BUCKET_NAME unset or empty Transfer crashes on well transfer init Transfer runs; GCS operations are skipped

Testing

  • Run python -m transfers.transfer with GCS_BUCKET_NAME unset in .env; the transfer should complete successfully.
  • With GCS_BUCKET_NAME set and valid credentials, the transfer should still use GCS for elevation and location cleanup caches.

When GCS_BUCKET_NAME is unset, get_cached_elevations() returns an empty
dict instead of failing with 'Cannot determine path without bucket name'.
dump_cached_elevations() and cleanup_locations() skip GCS operations in
that case, allowing transfers to run locally without GCS credentials.

Made-with: Cursor
@jeremyzilar jeremyzilar requested a review from jirhiker March 18, 2026 13:27
@jeremyzilar jeremyzilar self-assigned this Mar 18, 2026
@jirhiker
Copy link
Member

I'd like to minimize any modification or continued use of the transfer functionality. For this purpose I implemented PR #597.

Copy link
Member

@jirhiker jirhiker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see inline comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants