fix(fsspec): _info() should honor self.dircache#663
Merged
kylebarron merged 1 commit intodevelopmentseed:mainfrom Apr 10, 2026
Merged
fix(fsspec): _info() should honor self.dircache#663kylebarron merged 1 commit intodevelopmentseed:mainfrom
kylebarron merged 1 commit intodevelopmentseed:mainfrom
Conversation
kylebarron
approved these changes
Apr 10, 2026
Member
kylebarron
left a comment
There was a problem hiding this comment.
Thank you!
This is why fsspec is so awful to work with. Each implementation needs to know specifics of other internals of fsspec to be correct. Even if fsspec decides to implement caching, it should do it at a level that doesn't require implementors to know about it!
(FWIW I'm happy to merge PRs fixing fsspec handling, but in general I recommend using the non-fsspec API)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR eliminates unnecessary network calls in
FsspecStore._info()by ensuring it consults the directory cache before falling back to a network request, bringing it into alignment with standard fsspec implementations.Currently,
FsspecStore._info()issues an unconditionalhead_async()and ignoresself.dircache, even thoughAbstractFileSystem._ls_from_cacheis documented as the standard way for callers to surface pre-known file metadata.For comparison, both s3fs and gcsfs consult
_ls_from_cache()first and only fall back to a network call on miss.obstore.fsspec.FsspecStoreis the outlier here: pre-populatingfs.dircache(or having it populated by a prior_ls) is ignored.Notes
When the parent directory is cached, and the requested child is not in the listing,
_info()now raisesFileNotFoundErrorimmediately instead of falling through to a_ls(). This matchess3fs._infoand the documented contract of_ls_from_cache: a cached parent listing is authoritative, so a missing child definitively does not exist.