feat!: open_reader accepts optional size to skip HEAD request

[fvaleye]

Summary

Add an optional size parameter to obstore.open_reader / open_reader_async that lets callers skip the HEAD request used to fetch the file size when the size is already known from external metadata.

create_reader currently HEADs the object solely to populate an ObjectMeta, but BufReader::with_capacity consumes only meta.location and meta.size from it. There is a network round-trip per file to fetch a single u64.

The same pattern already exists one layer up in the arrow-rs workspace: parquet::arrow::async_reader::ParquetObjectReader::with_file_size(u64) accepts an optional size hint as Option<u64>. Python-side similar examples exist in fsspec.AbstractBufferedFile(size=None) and pyarrow.dataset.FileFormat.make_fragment(file_size=None).

When size is provided, create_reader constructs the ObjectMeta directly with the hint and skips the HEAD entirely. When size is None, behavior is the same as before.

Notes

The caller is responsible for the size value being accurate.
Failure modes:

• A hint larger than the actual file surfaces as an OSError at read time, when a range request past EOF is issued (Invalid range).
• A hint smaller than the actual file causes silent truncation: the reader treats the hint as authoritative EOF.

[kylebarron]

Thanks for the PR!

I think best to no longer store meta: ObjectMeta internally.

Curious if it's worth making the breaking changes non-breaking and deprecated for a release cycle, or if it's worth just removing .meta now. What do you think?

[kylebarron]

Thank you!

[kylebarron]

Forgot to mark this as breaking because it removes the meta param