Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pandas vs. Julia – cheat sheet and comparison

datascientyst.com

Post reply on HN
Print view View original

111–120 of 138 posts

Re: Pandas vs. Julia – cheat sheet and comparison

#111

Post by mrtranscendence »

DNF2 wrote: post #107
patrick451 wrote: post #94

Earlier quoted context omitted.

> Sorry, I mean my np.somenamespace.another.namespace.sparce head :) I really don't understand what you are trying to complain about. Namespaces are nice. Dumping everything into the global namespace sucks.

This is a problem for non-dispatch or singular dispatch languages, it's significantly different in the context of multiple dispatch. Namespaces are, well, not bad, but sometimes they are a solution to a problem that does not necessarily exist.

Is it really that different? It's fine in languages with multiple dispatch to have its builtin functions in the default namespace, but that doesn't mean that we'd want functions for everything in the default namespace. Namespacing is still useful.

For one thing, I've found it very useful when folks use syntax like `import numpy as np`, because then when I see `np.foo` I can trace back where `foo` comes from and look up the relevant documentation.

The complaint about `open("foo.txt").readlines()` vs `readlines("foo.txt")` is a red herring IMO, because nothing stops anyone from implementing a generic `readlines()` function in Python that can take a string file name or a file handler object. It's just that nobody really cares enough to because it's a complete non-issue.

Re: Pandas vs. Julia – cheat sheet and comparison

#112

Post by DNF2 »

mrtranscendence wrote: post #111
DNF2 wrote: post #107

Earlier quoted context omitted.

This is a problem for non-dispatch or singular dispatch languages, it's significantly different in the context of multiple dispatch. Namespaces are, well, not bad, but sometimes they are a solution to a problem that does not necessarily exist.

Is it really that different? It's fine in languages with multiple dispatch to have its builtin functions in the default namespace, but that doesn't mean that we'd want functions for everything in the default namespace. Namespacing is still useful. For one thing, I've found it very useful when folks use syntax like `import numpy as np`, because then when I see `np.foo` I can trace back where `foo` comes from and look…

But Julia _does_ have namespaces, and you can import everything with the `import` statement, or you can retrieve only the functions you need. This is also what is generally done during package development, while dumping everything is for interactive use.

Re: Pandas vs. Julia – cheat sheet and comparison

#113

Post by rgavuliak »

markkitti wrote: post #83
rgavuliak wrote: post #61

Earlier quoted context omitted.

Ok, in that case Python would be 0, 1, 3 and Julia 1, 2, 3. My point is that the example explicitly skips an index in the definition of a data frame for Python, but it doesn't for Julia.

Surely Python would not skip the 2.

The code snippet explicitly defines the index to be 0, 1, 3. It may just be a typo, but if even simple examples are written sloppily, how can I trust the sheet:

df = pd.DataFrame( {'col_1': [11, 12, 13], 'col_2': [21, 22, 23]}, index=[0, 1, 3])

Re: Pandas vs. Julia – cheat sheet and comparison

#114

Post by DNF2 »

gnramires wrote: post #108
DNF2 wrote: post #105

Earlier quoted context omitted.

The argument isn't silently treated as a double, it is explicitly and loudly treated as a double, because it is a literal double. And this is not an advantage to the designer exclusively, it is very much an advantage for the end user that the treatment is explicit, consistent and predictable, instead of 'magically' reinterpreting the meaning of literals based on guessing the intent of the user. Basically, you seem to…

I think the argument here is that parsing of the string '1e-300' maybe should be context dependent. In this case, 1e-300 is being parsed as a double, and then forwarded to the function. Maybe it could be parsed as a bigfloat whenever it is an argument of a function expecting a bigfloat.

Yes, I got that argument, and that is exactly what I was arguing against. You cannot and should not parse the literal double `1e-300` differently dependent on which function it is later passed to. This is what `big"1e-300"` or `BigFloat("1e-300")` is for, where the BigFloat constructor parses the string.

Re: Pandas vs. Julia – cheat sheet and comparison

#115

Post by DNF2 »

jakobnissen wrote: post #110
gnramires wrote: post #108

Earlier quoted context omitted.

I think the argument here is that parsing of the string '1e-300' maybe should be context dependent. In this case, 1e-300 is being parsed as a double, and then forwarded to the function. Maybe it could be parsed as a bigfloat whenever it is an argument of a function expecting a bigfloat.

Julia does this for exponentiation, i.e. a literal exponent is parsef differently that exponentiation by a variable. I think it was a mistake. Invariably, a new user discovers the discrepancy and is thoroughly confused. Let's not repeat the same mistake with big nums.

I don't think this has anything to do with whether 1e-300 and 10.0^-300 is parsed differently (and perhaps that is a mistake). The poster seems to want to parse 1e-300 directly as a BigFloat in the call `BigFloat(1e-300)`, because of the function it is passed to.

Re: Pandas vs. Julia – cheat sheet and comparison

#116

Post by marginalia_nu »

EuAndreh wrote: post #81
marginalia_nu wrote: post #51

Earlier quoted context omitted.

Mobile compatible websites are strictly worse though. It's why almost all desktop websites are just a hideous jumble of boxes these days. It's virtually impossible to make a website that is well designed on both desktop and mobile. As long as the affordances of mouse+keyboard and touchscreen are as different as they are, one of the user groups needs will suffer a detrimental compromise.

An empty HTML page works on both. Add some margin and spacing and it keeps working. After that, what you can do is ruin it, targetting a webpage for a specific screen size.

Right, but if you aren't able to make assumptions about screen size, you can't use for example a table, which is very useful if you want to convey data.

A phone screen just isn't big enough to display tables with more than two or three columns. There's no reason desktop users should need to be crippled in this fashion.

Tables are a very powerful tool for conveying lots of structured data in a way that's useful and intuitive. Avoiding them means losing out on this.

This is the exact problem the website we're discussing is having. You just can't show the relatively small amount of information side-by-side on mobile the way they are trying to. The only solution to it is to make it less intuitive and show them on top of each other in a complete jumble.

Re: Pandas vs. Julia – cheat sheet and comparison

#117

Post by Nilshg »

ChrisRackauckas wrote: post #63
martinsmit wrote: post #54

Earlier quoted context omitted.

I agree with your conclusion but want to add that switching from Julia may not make sense either. According to these benchmarks: https://h2oai.github.io/db-benchmark/ , DF.jl is the fastest library for some things, data.table for others, polars for others. Which is fastest depends on the query and whether it takes advantage of the features/properties of each. For what it's worth, data.table is my favourite to use and…

Indeed DataFrames.jl isn't and won't be the fastest way to do many things. It makes a lot of trade offs in performance for flexibility. The columns of the dataframe can be any indexable array, so while most examples use 64-bit floating point numbers, strings, and categorical arrays, the nice thing about DataFrames.jl is that using arbitrary precision floats, pointers to binaries, etc. are all fine inside of a DataFra…

> Indeed DataFrames.jl isn't and won't be the fastest way to do many things

Agreed, and the DF.jl developers are aware and very open about this fact - the core design trades off flexibility and user friendliness over speed (while of course trying to be as performant as possible within those constraints).

One thing that hasn't been mentioned so far is InMemoryDatasets.jl, which as far as I know is the closest to polars in Julia-land in that it chooses a different point on the flexibility-performance curve more towards the performance end. It's not very widely used as far as I can tell but could be interesting for users who need more performance than DF.jl can deliver - some benchmarks from early versions suggested performance is on par with polars: https://discourse.julialang.org/t/ann-a-new-lightning-fast-p...

Re: Pandas vs. Julia – cheat sheet and comparison

#118

Post by tombert »

sundarurfriend wrote: post #99
tombert wrote: post #19

Earlier quoted context omitted.

I have not done anything even remotely significant in Julia, but the little I played with didn't seem to indicate to me that it would be bad for trivial stuff...what trivial stuff is hard in Julia but easy in Python?

I think they just mean that with trivial projects, it's not worth trying them in a new language since the performance benefits are probably going to be minimal, and wouldn't really show off Julia's strengths.

That's fair; I have done my fair share of scripting in Node.js just because I'm familiar with it and it's fast enough to do most anything.

Re: Pandas vs. Julia – cheat sheet and comparison

#119

Post by cookieperson »

hpcjoe wrote: post #93
cookieperson wrote: post #34

Earlier quoted context omitted.

Just make sure you find the appropriate documentation because the package changes it's syntax an awful lot over the past four years or so and there are lots of tutorials, videos, and blogs that don't apply anymore. Similarly make sure you research the ecosystem because everything in Julia is very fragmented, IE pandas.loadcsv will require two or more packages in it's Julia equivalent.

> Julia is very fragmented No, it isn't. I'm using Pandas, DF.jl, and even polars at work. DF.jl is by far the best/easiest/quickest to use, as its syntax is consistent. Polars is a bit more annoying as its syntax is further along the learning curve that I have gotten yet. Pandas ... what to say about a library that will happily return a pd.Series in one moment, and a pd.DataFrame in another, for the same function ca…

Would it surprise you to know I don't use python almost ever?

Yes the ecosystem is very fragmented. It's done so by design. One of the major contributors to the language wrote a paper about it.

Re: Pandas vs. Julia – cheat sheet and comparison

#120

Post by cookieperson »

Nilshg wrote: post #60
cookieperson wrote: post #34

Earlier quoted context omitted.

Just make sure you find the appropriate documentation because the package changes it's syntax an awful lot over the past four years or so and there are lots of tutorials, videos, and blogs that don't apply anymore. Similarly make sure you research the ecosystem because everything in Julia is very fragmented, IE pandas.loadcsv will require two or more packages in it's Julia equivalent.

To be clear on this: DataFrames, like most of the Julia ecosystem, follows SemVer. DataFrames 1.0 was released over two years ago (March 2021), and the API has been stable ever since. Furthermore, Bogumil Kaminski, one of the main developers behind DataFrames, makes sure that the DataFrames tutorials he has created here ( https://github.com/bkamins/Julia-DataFrames-Tutorial ) are updated on every new release.

Beside the point. Old information on the internet abounds with the old syntax. It's a common hiccup for beginners in Julia. Ie they'd google something, try the syntax it errored out, they google it another way found it was updated try that nope that's outdated now too, etc. So it's worth mentioning.
Post reply on HN