Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pandas vs. Julia – cheat sheet and comparison

datascientyst.com

Post reply on HN
Print view View original

121–130 of 138 posts

Re: Pandas vs. Julia – cheat sheet and comparison

#121

Post by cookieperson »

chaxor wrote: post #69
cookieperson wrote: post #34

Earlier quoted context omitted.

Just make sure you find the appropriate documentation because the package changes it's syntax an awful lot over the past four years or so and there are lots of tutorials, videos, and blogs that don't apply anymore. Similarly make sure you research the ecosystem because everything in Julia is very fragmented, IE pandas.loadcsv will require two or more packages in it's Julia equivalent.

This is definitely clouded by personal preferences far too much. Acting as if python is void of issues in change over time? We all know the incredible pain of trying to get some ML package running written 3 months ago (let alone 3 years ago) and how much time is spent remaking some conda env inside a docker inside qemu inside... just to get the stupid thing to load. So don't act like python doesn't have it's problems…

No clue why python was even mentioned here. I hardly if ever use python. Strawman arguments are logical fallacies.

Re: Pandas vs. Julia – cheat sheet and comparison

#122

Post by cookieperson »

affinepplan wrote: post #58
cookieperson wrote: post #34

Earlier quoted context omitted.

Just make sure you find the appropriate documentation because the package changes it's syntax an awful lot over the past four years or so and there are lots of tutorials, videos, and blogs that don't apply anymore. Similarly make sure you research the ecosystem because everything in Julia is very fragmented, IE pandas.loadcsv will require two or more packages in it's Julia equivalent.

I notice you coming into every single thread about Julia to criticize the language and the community. Do you have a vendetta or something?

Not a vendetta, just sick of people talking about Julia like it's this flawless thing to bring in users to make $ off of the community, only for the new comers to enter find problems and get gas lit over their existence. It's a gross cycle and someone has to say something about it.

Re: Pandas vs. Julia – cheat sheet and comparison

#123

Post by cookieperson »

patrick451 wrote: post #92
affinepplan wrote: post #58

Earlier quoted context omitted.

I notice you coming into every single thread about Julia to criticize the language and the community. Do you have a vendetta or something?

The Rust Evangalism Strike force normalized the practice. This is what we do now.

??? What's evangelical about pointing out that anyone interested in using dataframes jl should be aware that there is a lot of out dated content on how to use it and to be mindful of that??? It's honest and helpful.

Re: Pandas vs. Julia – cheat sheet and comparison

#124

Post by cookieperson »

hpcjoe wrote: post #95
affinepplan wrote: post #58

Earlier quoted context omitted.

I notice you coming into every single thread about Julia to criticize the language and the community. Do you have a vendetta or something?

Back in my more perl-ish days, I recall pythonistas doing this. Sad to see little has changed.

Is it that I have done something wrong here, or is it possible the Julia community can't handle its own reality. If you reread what I wrote here... You'd actually see it's advice. It's advice from someone who has helped people learn Julia for years. Common stumbling points.

Re: Pandas vs. Julia – cheat sheet and comparison

#125

Post by cookieperson »

chaxor wrote: post #70
ChrisRackauckas wrote: post #63

Earlier quoted context omitted.

Indeed DataFrames.jl isn't and won't be the fastest way to do many things. It makes a lot of trade offs in performance for flexibility. The columns of the dataframe can be any indexable array, so while most examples use 64-bit floating point numbers, strings, and categorical arrays, the nice thing about DataFrames.jl is that using arbitrary precision floats, pointers to binaries, etc. are all fine inside of a DataFra…

I really hope people don't come from R to Julia. People who use R are not good programmers, and will degrade the core of the language and it's principles. It would be a shame to see the equivalent of tacking on 6 different object oriented systems to a base language and fragmenting the community completely.

It might surprise you to learn that Julia is actively relying on code written in/for R to perform computations. You might be surprised to find out that people who can write R can also write C++ C and other languages of their choosing. You also might be surprised to learn that some of the most vetted statistical code exists in the R ecosystem. If I were someone recruiting for a niche language that had a weak ecosystem, personally I'd take all the help I could get. Can learn Julia with a background in any other programming language in a few weeks... The same can't be said about martingales... But you get to choose your strategy here...

Re: Pandas vs. Julia – cheat sheet and comparison

#126

Post by cookieperson »

ayhanfuat wrote: post #50
cookieperson wrote: post #22

Earlier quoted context omitted.

SQLite is often much faster then dataframes jl and pandas.

I highly doubt that SQLite is faster than pandas, let alone dataframes.jl, for analytical workloads.

Might surprise you how many people are using pandas or dataframes for OLTP on a daily basis because they don't know better.

Re: Pandas vs. Julia – cheat sheet and comparison

#127

Post by shele »

cookieperson wrote: post #125
chaxor wrote: post #70

Earlier quoted context omitted.

I really hope people don't come from R to Julia. People who use R are not good programmers, and will degrade the core of the language and it's principles. It would be a shame to see the equivalent of tacking on 6 different object oriented systems to a base language and fragmenting the community completely.

It might surprise you to learn that Julia is actively relying on code written in/for R to perform computations. You might be surprised to find out that people who can write R can also write C++ C and other languages of their choosing. You also might be surprised to learn that some of the most vetted statistical code exists in the R ecosystem. If I were someone recruiting for a niche language that had a weak ecosystem…

And thus we who transitioned to Julia from R and know a bit about martingales and less about programming have long been trying to degrade the core of the language and its principles by making `mean` a Base function.

Re: Pandas vs. Julia – cheat sheet and comparison

#128

Post by martinsmit »

ChrisRackauckas wrote: post #63
martinsmit wrote: post #54

Earlier quoted context omitted.

I agree with your conclusion but want to add that switching from Julia may not make sense either. According to these benchmarks: https://h2oai.github.io/db-benchmark/ , DF.jl is the fastest library for some things, data.table for others, polars for others. Which is fastest depends on the query and whether it takes advantage of the features/properties of each. For what it's worth, data.table is my favourite to use and…

Indeed DataFrames.jl isn't and won't be the fastest way to do many things. It makes a lot of trade offs in performance for flexibility. The columns of the dataframe can be any indexable array, so while most examples use 64-bit floating point numbers, strings, and categorical arrays, the nice thing about DataFrames.jl is that using arbitrary precision floats, pointers to binaries, etc. are all fine inside of a DataFra…

> Tidier

I have not tried it. I like that the project makes broadcasting invisible, I dislike that it tries to completely replicate R's semantics and Tidyverse's syntax. Two examples: firstly, the tuples vs scalars thing doesn't seem very Julia to me. Secondly, I love that DF.jl has :column_name and variable_name as separate syntax. Tidier.jl drops this convention (from what I see in the readme).

> I'm not sure if someone is looking directly at the data.table parts

I believe there was some effort to make an i-j-by syntax in Julia but it fell through or stopped getting worked on. By this syntax I mean something like:

  # An example of using i, j, and by
  @dt flights [
    carrier == "AA",
    (mean(:arr_delay), mean(:dep_delay)),
    by = (:origin, :dest, :month)]

  # An example of expressions in by
  @dt flights [_, nrows, by = (:dep_delay > 0, :arr_delay > 0)]
The idea of ijby (as I understand it) is that it has a consistent structure: row selection/filtering comes before column selection/filtering, and is optionally followed by "by" and then other keyword arguments which augment the data that the core "ij" operations act upon.

data.table also has some nifty syntax like

  data[, x := x + 1] # update in place
  data[, x := x/nrows(.SD), by = y] # .SD =  references data subset currently being worked on
which make it more concise than dplyr.

The conciseness and structure that comes from data.table and its tendency to be much less code than comparable tidyverse transformations through some well-informed choices and reservations of syntax make it nicer for me to use.

Re: Pandas vs. Julia – cheat sheet and comparison

#129

Post by affinepplan »

ChrisRackauckas wrote: post #63
martinsmit wrote: post #54

Earlier quoted context omitted.

I agree with your conclusion but want to add that switching from Julia may not make sense either. According to these benchmarks: https://h2oai.github.io/db-benchmark/ , DF.jl is the fastest library for some things, data.table for others, polars for others. Which is fastest depends on the query and whether it takes advantage of the features/properties of each. For what it's worth, data.table is my favourite to use and…

Indeed DataFrames.jl isn't and won't be the fastest way to do many things. It makes a lot of trade offs in performance for flexibility. The columns of the dataframe can be any indexable array, so while most examples use 64-bit floating point numbers, strings, and categorical arrays, the nice thing about DataFrames.jl is that using arbitrary precision floats, pointers to binaries, etc. are all fine inside of a DataFra…

> I would be interested to hear what about the ergonomics of data.table you find useful. if there are some ideas that would be helpful for DataFrames.jl to learn from data.table directly I'd be happy to share it with the devs.

Personally, my main usability gripe is that it's difficult to do row-wise transformations that try to combine multiple columns by name. I know one can do ``` transform(df, AsTable() => foo ∘ Tables.NamedTupleIterator) ```

But this is 1) kind of wordy and 2) can come with enormous compile times (making it unusable) for wide tables

Re: Pandas vs. Julia – cheat sheet and comparison

#130

Post by geokon »

time_to_smile wrote: post #90
joeman1000 wrote: post #56

The thing that keeps me coming back to Julia is the ability to pipe (or whatever you want to call it). It makes DataFrame operations a lot cleaner since I don't need to modify in place or create new DFs at intermediate steps in a process. Here's a video showing this sort of workflow in R: https://youtu.be/W3e8qMBypSE

I'm still not entirely convinced that pipes aren't an anti-pattern. Absolutely an improvement over nested function calls: a(b(c(d))) vs d |> c |> b |> a but I'm not convinced pipes are better than more verbose code that explains each step: step1 = c(d) step2 = b(step1) result = a(step2) I've written a lot of tidy R and do understand the specific use cases where it really doesn't make sense to use the more verbose for…

I suggest trying to lean into it more

I suggest that trying to strictly only bind output to a symbol if it will be used in multiple places.

So when I read code and I see some "intermediary" value bound - it tells me immediately "this thing will be used in several spots". Thereby bindings actually start to convey extra information

Anyway, it's just something that's worked for me. In all other scenarios I will use threading/pipeline (maybe Clojure specific). If steps are confusing/complex then you make a local named lambda or add in the extreme case.. comments

Post reply on HN