Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pandas vs. Julia – cheat sheet and comparison

datascientyst.com

Post reply on HN
Print view View original

61–70 of 138 posts

Re: Pandas vs. Julia – cheat sheet and comparison

#61

Post by rgavuliak »

joeman1000 wrote: post #57
rgavuliak wrote: post #37

The cheatsheet goes wrong already for the first example of declaring a df: - you could do a range in python (range(11, 14)) - columns are called col_1 & col_2 vs a & b (both sets are horrible names) - pandas defines index of 0, 1, 3, while Julia would most likely have 0, 1, 2?

What do you mean? Julia is 1-indexed.

Ok, in that case Python would be 0, 1, 3 and Julia 1, 2, 3. My point is that the example explicitly skips an index in the definition of a data frame for Python, but it doesn't for Julia.

Re: Pandas vs. Julia – cheat sheet and comparison

#62

Post by nevereasonfroma »

martinsmit wrote: post #54
cookieperson wrote: post #33

Earlier quoted context omitted.

Pretty sure dataframes jl isn't the fastest dataframes library out there. Think it's Polars, which has bindings in Rust and python. If I remember correctly the runner up is data.table. Similarly SQL/SQLite can often beat all of these So switching to Julia for speed in this context may not even make sense anyways...

I agree with your conclusion but want to add that switching from Julia may not make sense either. According to these benchmarks: https://h2oai.github.io/db-benchmark/ , DF.jl is the fastest library for some things, data.table for others, polars for others. Which is fastest depends on the query and whether it takes advantage of the features/properties of each. For what it's worth, data.table is my favourite to use and…

duckdb's fork, updated 2023.04 (h2oai is 2021.06): https://duckdblabs.github.io/db-benchmark/

repo: https://github.com/duckdblabs/db-benchmark

Re: Pandas vs. Julia – cheat sheet and comparison

#63

Post by ChrisRackauckas »

martinsmit wrote: post #54
cookieperson wrote: post #33

Earlier quoted context omitted.

Pretty sure dataframes jl isn't the fastest dataframes library out there. Think it's Polars, which has bindings in Rust and python. If I remember correctly the runner up is data.table. Similarly SQL/SQLite can often beat all of these So switching to Julia for speed in this context may not even make sense anyways...

I agree with your conclusion but want to add that switching from Julia may not make sense either. According to these benchmarks: https://h2oai.github.io/db-benchmark/ , DF.jl is the fastest library for some things, data.table for others, polars for others. Which is fastest depends on the query and whether it takes advantage of the features/properties of each. For what it's worth, data.table is my favourite to use and…

Indeed DataFrames.jl isn't and won't be the fastest way to do many things. It makes a lot of trade offs in performance for flexibility. The columns of the dataframe can be any indexable array, so while most examples use 64-bit floating point numbers, strings, and categorical arrays, the nice thing about DataFrames.jl is that using arbitrary precision floats, pointers to binaries, etc. are all fine inside of a DataFrame without any modification. This is compared to things like the Pandas allowed datatypes (https://pbpython.com/pandas_dtypes.html). I'm quite impressed by the DataFrames.jl developers given how they've kept it dynamic yet seem to have achieved pretty good performance. Most of it is smart use of function barriers to avoid the dynamism in the core algorithms. But from that knowledge it's very clear that systems should be able to exist that outperform it even with the same algorithms, in some cases just by tens of nanoseconds but in theory that bump is always there.

In the Julia world the one which optimizes to be fully non-dynamic is TypedTables (https://github.com/JuliaData/TypedTables.jl) where all column types are known at compile time, removing the dynamic dispatch overhead. But in Julia the minor performance gain of using TypedTables vs the major flexibility loss is the reason why you pretty much never hear about it. Probably not even worth mentioning but it's a fun tidbit.

> For what it's worth, data.table is my favourite to use and I believe it has the nicest ergonomics of the three I spoke about.

I would be interested to hear what about the ergonomics of data.table you find useful. if there are some ideas that would be helpful for DataFrames.jl to learn from data.table directly I'd be happy to share it with the devs. Generally when I hear about R people talk about tidyverse. Tidier (https://github.com/TidierOrg/Tidier.jl) is making some big strides in bringing a tidy syntax to Julia and I hear that it has had some rapid adoption and happy users, so there are some ongoing efforts to use the learnings of R API's but I'm not sure if someone is looking directly at the data.table parts.

Re: Pandas vs. Julia – cheat sheet and comparison

#64

Post by stoniejohnson »

joeman1000 wrote: post #56

The thing that keeps me coming back to Julia is the ability to pipe (or whatever you want to call it). It makes DataFrame operations a lot cleaner since I don't need to modify in place or create new DFs at intermediate steps in a process. Here's a video showing this sort of workflow in R: https://youtu.be/W3e8qMBypSE

Is that not also available in Pandas?

https://pandas.pydata.org/docs/reference/api/pandas.DataFram...

Re: Pandas vs. Julia – cheat sheet and comparison

#65

Post by ChrisRackauckas »

joeman1000 wrote: post #56

The thing that keeps me coming back to Julia is the ability to pipe (or whatever you want to call it). It makes DataFrame operations a lot cleaner since I don't need to modify in place or create new DFs at intermediate steps in a process. Here's a video showing this sort of workflow in R: https://youtu.be/W3e8qMBypSE

Julia has a pipe syntax (|>). But I think the bigger part here is more generally APIs built around it, which people are doing some things to port tidy syntax (https://github.com/TidierOrg/Tidier.jl).

Re: Pandas vs. Julia – cheat sheet and comparison

#66

Post by ChrisRackauckas »

misja111 wrote: post #38
jstx1 wrote: post #10

This seems very poor - the comparison is between pandas and DataFrames.jl, not Julia; syntax comparison is very surface-level; cheatsheats are low resolution; the learning curve section says nothing about the learning curve; and the conclusion is "do whatever you like".

Well Pandas is a framework, not a language, so it only makes sense to compare it to DataFrames.jl and not to Julia as a lanugage. But I agree this should have been reflected in the title of the article.

I both agree and disagree. It does look weird as library vs language. However at the same time, in Pandas everything tends to be in the Pandas library, whereas when using DataFrames.jl you tend to mix it with a lot of features that are external. Most of the calls just use overloaded functions from Julia's Base library (mean, first, last, findall). The Pandas model is to look at the Pandas docs and find the Pandas dataframe function that does your job. The DataFrames.jl model is to do whatever you would have done normally in Julia, like use the sort function, but now just use it on a DataFrame. The idea of DataFrames.jl is that you know the language and so it extends/adds as few functions as possible (joins, groupby, split-apply-combine I think are it?). This plus many other calls use functions from the more general Julia data science ecosystem (CSV.jl, JSON.jl, ...). So the title ends up being a bit apples and oranges, but the usage is also quite apples and oranges and the cheat sheet does accurately reflect that.

Re: Pandas vs. Julia – cheat sheet and comparison

#67

Post by newswasboring »

marginalia_nu wrote: post #6
retrochameleon wrote: post #3

Unreadable on mobile

You do a lot of software development on mobile?

Look up the termux community. There are actually people in third world countries who are learning programming using their phones. I've seen awesome builds which use Android phones as their primary CPU unit and cobble on scavenged monitors and keyboards and mice. It's honestly fucking awesome

Re: Pandas vs. Julia – cheat sheet and comparison

#68

Post by chaxor »

joeman1000 wrote: post #56

The thing that keeps me coming back to Julia is the ability to pipe (or whatever you want to call it). It makes DataFrame operations a lot cleaner since I don't need to modify in place or create new DFs at intermediate steps in a process. Here's a video showing this sort of workflow in R: https://youtu.be/W3e8qMBypSE

In pandas you can chain commands by wrapping the whole command in (). Personally IMO looks far 'cleaner' than all of the ugly %>% everywhere.

Re: Pandas vs. Julia – cheat sheet and comparison

#69

Post by chaxor »

cookieperson wrote: post #34
freilanzer wrote: post #26

Earlier quoted context omitted.

I have done both complex and trivial stuff in both languages and Julia isn't more inconvenient for trivial things.

Just make sure you find the appropriate documentation because the package changes it's syntax an awful lot over the past four years or so and there are lots of tutorials, videos, and blogs that don't apply anymore. Similarly make sure you research the ecosystem because everything in Julia is very fragmented, IE pandas.loadcsv will require two or more packages in it's Julia equivalent.

This is definitely clouded by personal preferences far too much. Acting as if python is void of issues in change over time? We all know the incredible pain of trying to get some ML package running written 3 months ago (let alone 3 years ago) and how much time is spent remaking some conda env inside a docker inside qemu inside... just to get the stupid thing to load. So don't act like python doesn't have it's problems in change over time.

Re: Pandas vs. Julia – cheat sheet and comparison

#70

Post by chaxor »

ChrisRackauckas wrote: post #63
martinsmit wrote: post #54

Earlier quoted context omitted.

I agree with your conclusion but want to add that switching from Julia may not make sense either. According to these benchmarks: https://h2oai.github.io/db-benchmark/ , DF.jl is the fastest library for some things, data.table for others, polars for others. Which is fastest depends on the query and whether it takes advantage of the features/properties of each. For what it's worth, data.table is my favourite to use and…

Indeed DataFrames.jl isn't and won't be the fastest way to do many things. It makes a lot of trade offs in performance for flexibility. The columns of the dataframe can be any indexable array, so while most examples use 64-bit floating point numbers, strings, and categorical arrays, the nice thing about DataFrames.jl is that using arbitrary precision floats, pointers to binaries, etc. are all fine inside of a DataFra…

I really hope people don't come from R to Julia. People who use R are not good programmers, and will degrade the core of the language and it's principles. It would be a shame to see the equivalent of tacking on 6 different object oriented systems to a base language and fragmenting the community completely.
Post reply on HN