I've never seen the term "dataframe" used as it is on this webste, and the commenters here seem to all use it. Judging by the examples it seems to just refer to a "row" from e.g. a CSV or SQL query. So is that all it is, or am I missing something?
Polars: Fast DataFrame library for Rust and Python
91–100 of 131 posts
Re: Polars: Fast DataFrame library for Rust and Python
#92I've never seen the term "dataframe" used as it is on this webste, and the commenters here seem to all use it. Judging by the examples it seems to just refer to a "row" from e.g. a CSV or SQL query. So is that all it is, or am I missing something?
A "dataframe" is a "table"
Re: Polars: Fast DataFrame library for Rust and Python
#93I've never seen the term "dataframe" used as it is on this webste, and the commenters here seem to all use it. Judging by the examples it seems to just refer to a "row" from e.g. a CSV or SQL query. So is that all it is, or am I missing something?
It's a column oriented data structure.
Re: Polars: Fast DataFrame library for Rust and Python
#94Does anybody here know dataframe systems that are able to handle file sizes bigger than the available RAM? Is polars able to handle this? I am only aware of disk.frame (diskframe.com), but don't know how well it performs.
spark dataframe api..
Re: Polars: Fast DataFrame library for Rust and Python
#95There are so many dataframe libraries, many of which have APIs closely following pandas, but not drop-in replacements. I wish we could agree on a standard describing the core parts of what a dataframe must do, such that code depending only on those operations can easily move between dataframes.
That's SQL isn't it?
Well, relational algebra/calculus, but close enough.
Re: Polars: Fast DataFrame library for Rust and Python
#96Does anybody here know dataframe systems that are able to handle file sizes bigger than the available RAM? Is polars able to handle this? I am only aware of disk.frame (diskframe.com), but don't know how well it performs.
Apache Spark.
Re: Polars: Fast DataFrame library for Rust and Python
#97From the python docs:
> No Index
> They are not needed. Not having them makes things easier. Convince me otherwise
The library in general seem interesting. I'm not 100% sold on the syntax (as usual project is called select...), but it is not pandas which is already a huge plus.
Re: Polars: Fast DataFrame library for Rust and Python
#98Polars could bring the best of both worlds together if it can codegen python api calls to their Rust equivalent. A user conducts ad-hoc analysis and rapid development with Python. When the work is ready to ship, the user invokes a codegen to transform into Rust-equivalent api calls, resulting in a new rust module.
Re: Polars: Fast DataFrame library for Rust and Python
#99Does anybody here know dataframe systems that are able to handle file sizes bigger than the available RAM? Is polars able to handle this? I am only aware of disk.frame (diskframe.com), but don't know how well it performs.
Vaex
Re: Polars: Fast DataFrame library for Rust and Python
#100Does anybody here know dataframe systems that are able to handle file sizes bigger than the available RAM? Is polars able to handle this? I am only aware of disk.frame (diskframe.com), but don't know how well it performs.
I believe Vaex can do this, in addition to GPU processing and reading direct from s3. https://github.com/vaexio/vaex
To you and all the other sibling comments: Thanks a lot! Exactly what I have been looking for!
With regard to Vaex, I would really be interested in an independent benchmark comparing it to dask, spark, data.table etc. However, I have seen in the comments that others also can't find that.