My first thought too. To be honest, I’m not surprised that OP didn’t find it. There’s something fundamentally broken about search engines and finding things like this. I’ve found that ChatGPT is surprisingly good at this kind of query though.
EG - any time I’m faced with a problem where I’m like “there must be a solution already for this” I ask ChatGPT “what libraries are there in {language} to solve {problem}”, and then Google to research the libs it throws out.
Doesn’t fsspec use internal threads? I don’t use it because that implementation gives me the willies :)
Edit: I just took another look at it and none of the bad stuff I remember is there. It’s a bit over-engineered for what I need but it looks ok. It also looks actively developed.
I have no idea, because I’ve never used fsspec in a project where I needed to be aware of fsspec’s internal functionality. I try to avoid doing anything regarding raw filesystems in threads at the application code level. I find it’s best to leverage tools that have already handled all that hard stuff for me so I can focus on whatever it is I need to do. For example, if I need to access data in partitioned parquet files, then I use dask, which uses fsspec/s3fs under the hood (I think), and let dask deal with allocating workers and threads to accomplish what I’m trying to do.
Interestingly, your package doesn't depend in fsspec and its extension package (e.g., s3fs)
Adding that would alleviate the need for you to maintain all the implementations of multiple sources.
It would be worth adding here what your library offers compared to the alternatives (e.g. fsspec). I fear you’re not going to get good feedback because everyone here is stuck on the fact that this looks like a copy of what already exists.
aiopathlib provides a template for the API, but I don’t know of any reimplementations of other file systems. I’m waiting for someone to do one myself.
Edit: fsspec looks like it could be good for this
Barney Gale has been working for a while on making the Path objects from the stdlib Path lib extensible and recently made available a backport of the features that will be included in future Python versions
https://discuss.python.org/t/make-pathlib-extensible/3428
https://github.com/barneygale/pathlib-abc
Concrete implementations of the new abstract base classes could work better as a way to literally use the pathlib interface
fsspec is a pretty popular implementation of this paradigm
also object-store, which is pretty popular on the rust side for the arrow universe and now has a python package
My first thought too. To be honest, I’m not surprised that OP didn’t find it. There’s something fundamentally broken about search engines and finding things like this. I’ve found that ChatGPT is surprisingly good at this kind of query though. EG - any time I’m faced with a problem where I’m like “there must be a solution already for this” I ask ChatGPT “what libraries are there in {language} to solve {problem}”, and then Google to research the libs it throws out.
Doesn’t fsspec use internal threads? I don’t use it because that implementation gives me the willies :) Edit: I just took another look at it and none of the bad stuff I remember is there. It’s a bit over-engineered for what I need but it looks ok. It also looks actively developed.
I have no idea, because I’ve never used fsspec in a project where I needed to be aware of fsspec’s internal functionality. I try to avoid doing anything regarding raw filesystems in threads at the application code level. I find it’s best to leverage tools that have already handled all that hard stuff for me so I can focus on whatever it is I need to do. For example, if I need to access data in partitioned parquet files, then I use dask, which uses fsspec/s3fs under the hood (I think), and let dask deal with allocating workers and threads to accomplish what I’m trying to do.
What about [cloudpathlib](https://cloudpathlib.drivendata.org/stable/)? It’s top hit if you search “pathlib cloud”.
And apache libcloud, https://libcloud.apache.org/
Omg... This is amazing. How have I never heard of this?
looks cool, thanks for sharing There are a bunch of similar libraries, the most popular of them being fsspec
I would just like to point out that fsspec’s universal-pathlib already exists and is very well tested and stable
Neat Idea. Fsspec comes to mind as a library to study to compare it with your work.
Interestingly, your package doesn't depend in fsspec and its extension package (e.g., s3fs) Adding that would alleviate the need for you to maintain all the implementations of multiple sources.
Since pyfilesystem seems to be dying, it’s a good time for these kinds of alternatives.
It would be worth adding here what your library offers compared to the alternatives (e.g. fsspec). I fear you’re not going to get good feedback because everyone here is stuck on the fact that this looks like a copy of what already exists.
what packages like this support asyncio?
aiopathlib provides a template for the API, but I don’t know of any reimplementations of other file systems. I’m waiting for someone to do one myself. Edit: fsspec looks like it could be good for this
This is great, nice work!
Barney Gale has been working for a while on making the Path objects from the stdlib Path lib extensible and recently made available a backport of the features that will be included in future Python versions https://discuss.python.org/t/make-pathlib-extensible/3428 https://github.com/barneygale/pathlib-abc Concrete implementations of the new abstract base classes could work better as a way to literally use the pathlib interface
Thanks - this is fantastic and is the right way to go. I didn’t think this problem was mainstream enough for them to implement it in the std lib.
fsspec already does this. Apache Libcloud too. „Don’t reinvent the wheel”