Add the option to use smart_open to open S3 files

[pkerpedjiev]

This provides more flexibility into how the files are loaded and could potentially allow loading http-based files or any other type that smart_open supports.

[pkerpedjiev]

First pass at a PR. Would you be interested in something like this. If so, I'm happy to add tests. If not, I'm also happy to just create a new package.

[polyrand]

I'll take a look at this as soon as possible. But some thoughts from a quick overview:

• I'd be more inclined to put this behind an optional package dependency, so it can be installed with s3sqlite[smart_open]
• It should add the dependency to requirements.txt

I should probably update this repo to use pyproject.toml + uv. But this was mostly an experiment and I haven't looked at this in a while. But we can still get a PR merged and then I'll do the tooling update before pushing a new version.

Also, just to understand better. What would be the use case for using smart_open? i.e: What use-case motivates this change?

I think there's significant overlap between what smart_open and fsspec provide (in terms of capabilities). So just want to review the intended use case, because it may be more optimal to make it fully compatible with any fsspec implementation (instead of just s3fs) instead of adding smart_open.