31–40 of 131 posts
Earlier quoted context omitted.
He is basically describing benefits of the rest language so it's perfectly credible
How so? Does Rust have zero runtime overhead? I would find that hard to believe.
In my world, anything that isn't "identical to R's dplyr API but faster" just isn't quite worth switching for. There's absolutely no contest: dplyr has the most productive API and that matters to me more than anything else. But I'm glad to see Polars moves away from the kludgey sprawl of the Pandas API towards the perfection of dplyr... while also being blazingly fast! Now just mix in a bit of DSL so people aren't ob…
Also worth plugging the advanced speed of R’s data.table package which continues to trump dplyr to this day. The syntax is also more compact and straightforward once you understand how to query data with it.
Everything is fine "once you understand how to use it", even assembly code, but it's not equally expressive or intuitive. So I don't value data.table speed that much, it's my thinking and typing speed that's usually the limiting factor. I would always recommend dplyr over anything else for someone learning how to use tables.
I also can't help but point out that data.table has the worst first FAQ answer I've ever seen in software documentation: https://cran.r-project.org/web/packages/data.table/vignettes.... Just astonishingly bad. I could write an essay about the unique and diverse ways in which this thing is both incredibly poorly organized and deeply user-hostile.
But if you truly have a need for speed on large datasets, it may be for you.
In my world, anything that isn't "identical to R's dplyr API but faster" just isn't quite worth switching for. There's absolutely no contest: dplyr has the most productive API and that matters to me more than anything else. But I'm glad to see Polars moves away from the kludgey sprawl of the Pandas API towards the perfection of dplyr... while also being blazingly fast! Now just mix in a bit of DSL so people aren't ob…
Also worth plugging the advanced speed of R’s data.table package which continues to trump dplyr to this day. The syntax is also more compact and straightforward once you understand how to query data with it.
In my world, anything that isn't "identical to R's dplyr API but faster" just isn't quite worth switching for. There's absolutely no contest: dplyr has the most productive API and that matters to me more than anything else. But I'm glad to see Polars moves away from the kludgey sprawl of the Pandas API towards the perfection of dplyr... while also being blazingly fast! Now just mix in a bit of DSL so people aren't ob…
dplyr API is not ideal in my experience. Overly verbose and confusing group/melt/cast operators. I much much prefer data.table. In your edit you mention concision, data.table is practically the platonic ideal of that!
The tidyverse has the most advanced and intuitive versions of all the things you mention IMO. It has evolved a lot in the past couple years and your impressions of it could be out of date.
There is also the dtplyr backend for data.table speed with dplyr syntax, but I don't even bother because dplyr is almost always fast enough for me.
Earlier quoted context omitted.
Why? The benchmarks speak volumes. https://h2oai.github.io/db-benchmark/
The benchmarks speak volumes of dishonesty. They sorted the results by speed of 1st run. For a language like Julia, which is JIT-compiled, that's not a fair comparison, considering that you compile once and run millions of times. Note also that Julia would be number 1 in almost all of those benchmarks if you were to rank by speed of second run (as expected...). It's funny because once you notice it those benchmarks a…
Not really. They are designed to showcase a common use case across multiple technologies.
The beauty of this benchmark is that there is a hardware limit included so that it forces you to create novel solutions to perform well.
>Note also that Julia would be number 1 in almost all of those benchmarks if you were to rank by speed of second run (as expected...). It's funny because once you notice it those benchmarks are basically an ad for Julia.
Not sure where you're getting that but even on second run Julia doesn't really compete with DT/Polars
Earlier quoted context omitted.
Also worth plugging the advanced speed of R’s data.table package which continues to trump dplyr to this day. The syntax is also more compact and straightforward once you understand how to query data with it.
I don't like it as much as dplyr and I stand behind that. It's too "clever", especially with respect to joins. Everything is fine "once you understand how to use it", even assembly code, but it's not equally expressive or intuitive. So I don't value data.table speed that much, it's my thinking and typing speed that's usually the limiting factor. I would always recommend dplyr over anything else for someone learning h…
https://rdatatable.gitlab.io/data.table/articles/datatable-i...
Which is why it isn't really linked anywhere else.
Earlier quoted context omitted.
Why? The benchmarks speak volumes. https://h2oai.github.io/db-benchmark/
The benchmarks speak volumes of dishonesty. They sorted the results by speed of 1st run. For a language like Julia, which is JIT-compiled, that's not a fair comparison, considering that you compile once and run millions of times. Note also that Julia would be number 1 in almost all of those benchmarks if you were to rank by speed of second run (as expected...). It's funny because once you notice it those benchmarks a…
Not true. If we'd rank them by second run Julia would be:
- On simple query: 1st, 1st, 4th, 1st, 5th (down 1).
- On advanced query: 3rd, 6th, 6th, 4th (up 1), - (out of memory).
> The databases (and spark) will have to read from disk. They have no chance of competing with anything that's reading from ram, no matter how slow it is.
Not true. Upon quick peek on the bench code, ClickHouse and Spark use in-memory table. I assume other engines too.
Earlier quoted context omitted.
I don't like it as much as dplyr and I stand behind that. It's too "clever", especially with respect to joins. Everything is fine "once you understand how to use it", even assembly code, but it's not equally expressive or intuitive. So I don't value data.table speed that much, it's my thinking and typing speed that's usually the limiting factor. I would always recommend dplyr over anything else for someone learning h…
The FAQ isn't for new data.table users. https://rdatatable.gitlab.io/data.table/articles/datatable-i... Which is why it isn't really linked anywhere else.
Earlier quoted context omitted.
The benchmarks speak volumes of dishonesty. They sorted the results by speed of 1st run. For a language like Julia, which is JIT-compiled, that's not a fair comparison, considering that you compile once and run millions of times. Note also that Julia would be number 1 in almost all of those benchmarks if you were to rank by speed of second run (as expected...). It's funny because once you notice it those benchmarks a…
>The benchmarks speak volumes of dishonesty. Not really. They are designed to showcase a common use case across multiple technologies. The beauty of this benchmark is that there is a hardware limit included so that it forces you to create novel solutions to perform well. >Note also that Julia would be number 1 in almost all of those benchmarks if you were to rank by speed of second run (as expected...). It's funny be…
It's like -- Julia is the Rory Gilmore of programming languages.