Skip to content

fix(fsspec): _info() should honor self.dircache#663

Merged
kylebarron merged 1 commit intodevelopmentseed:mainfrom
fvaleye:fix/info-honors-dircache
Apr 10, 2026
Merged

fix(fsspec): _info() should honor self.dircache#663
kylebarron merged 1 commit intodevelopmentseed:mainfrom
fvaleye:fix/info-honors-dircache

Conversation

@fvaleye
Copy link
Copy Markdown
Contributor

@fvaleye fvaleye commented Apr 10, 2026

Summary

This PR eliminates unnecessary network calls in FsspecStore._info() by ensuring it consults the directory cache before falling back to a network request, bringing it into alignment with standard fsspec implementations.

Currently, FsspecStore._info() issues an unconditional head_async() and ignores self.dircache, even though AbstractFileSystem._ls_from_cache is documented as the standard way for callers to surface pre-known file metadata.

For comparison, both s3fs and gcsfs consult _ls_from_cache() first and only fall back to a network call on miss. obstore.fsspec.FsspecStore is the outlier here: pre-populating fs.dircache (or having it populated by a prior _ls) is ignored.

Notes

When the parent directory is cached, and the requested child is not in the listing, _info() now raises FileNotFoundError immediately instead of falling through to a _ls(). This matches s3fs._info and the documented contract of _ls_from_cache: a cached parent listing is authoritative, so a missing child definitively does not exist.

@ds-release-bot ds-release-bot Bot added the fix label Apr 10, 2026
Copy link
Copy Markdown
Member

@kylebarron kylebarron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

This is why fsspec is so awful to work with. Each implementation needs to know specifics of other internals of fsspec to be correct. Even if fsspec decides to implement caching, it should do it at a level that doesn't require implementors to know about it!

(FWIW I'm happy to merge PRs fixing fsspec handling, but in general I recommend using the non-fsspec API)

@kylebarron kylebarron merged commit 996978a into developmentseed:main Apr 10, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants