Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pandas 3.0

pandas.pydata.org

Post reply on HN
Print view View original

121–125 of 125 posts

Re: Pandas 3.0

#121

Post by noitpmeder »

pelasaco wrote: post #88
edschofield wrote: post #18

The design of Pandas is inferior in every way to Polars: API, memory use, speed, expressiveness. Pandas has been strictly worse since late 2023 and will never close the gap. Polars is multithreaded by default, written in a low-level language, has a powerful query engine, supports lazy, out-of memory execution, and isn’t constrained by any compatibility concerns with a warty, eager-only API and pre-Arrow data types th…

are many of the mentioned issues not just some vibe-code sessions away from done?

Give it a shot and report back when you get them merged

Re: Pandas 3.0

#123

Post by v3ss0n »

gkbrk wrote: post #22
v3ss0n wrote: post #20

Earlier quoted context omitted.

Sounds too much like an advertisement. Also we need to watch out when diving into Polars . Polars is VC backed Opensource project with cloud offering , which may become an opencore project - we know how those goes.

> we know how those go They get forked and stay open source? At least this is what happens to all the popular ones. You can't really un-open-source a project if users want to keep it open-source.

not many can maintain a complex project in full time.

Re: Pandas 3.0

#124

Post by richardbachman »

sampo wrote: post #36
edschofield wrote: post #18

The design of Pandas is inferior in every way to Polars: API, memory use, speed, expressiveness. Pandas has been strictly worse since late 2023 and will never close the gap. Polars is multithreaded by default, written in a low-level language, has a powerful query engine, supports lazy, out-of memory execution, and isn’t constrained by any compatibility concerns with a warty, eager-only API and pre-Arrow data types th…

Historically 18 years ago, Pandas started as a project by someone working in finance to use Python instead of Excel, yet be nicer than using just raw Python dicts and Numpy arrays. For better or worse, like Excel and like the simpler programming languages of old, Pandas lets you overwrite data in place. Prepare some data df_pandas = pd.DataFrame({'a': [1, 2, 3, 4, 5], 'b': [10, 20, 30, 40, 50]}) df_polars = pl.from_p…

`row_index()` was also recently added.

  df.with_columns(pl.col.b + pl.row_index().is_between(1, 3))
  # shape: (5, 2)
  # ┌─────┬─────┐
  # │ a   ┆ b   │
  # │ --- ┆ --- │
  # │ i64 ┆ i64 │
  # ╞═════╪═════╡
  # │ 1   ┆ 10  │
  # │ 2   ┆ 21  │
  # │ 3   ┆ 31  │
  # │ 4   ┆ 41  │
  # │ 5   ┆ 50  │
  # └─────┴─────┘
> Polars has an optimization to overwite a single value

I believe it is just "syntax sugar" for calling `Series.scatter()`[1]

> it doesn't allow slicing

I believe you are correct:

  df_polars[1:3, "b"] += 1
  # TypeError: cannot use "slice(1, 3, None)" for indexing
You can do:

  df_polars[list(range(1, 4)), "b"] += 1
Perhaps nobody has requested slice syntax? It seems like it would be easy to add.

[1]: https://github.com/pola-rs/polars/blob/9079e20ae59f8c75dcce8...

Re: Pandas 3.0

#125

Post by thijsn »

goatlover wrote: post #115
thijsn wrote: post #112

Earlier quoted context omitted.

Polars is indeed more verbose when coming from pandas, but in my experience it is an advantage for when you're reading that same code after not having touched it for months. pandas is write-optimized, so you can quickly and powerfully transform your data. Once you're used to it, it allows you to quickly get your work done. But figuring out what is happening in that code after returning to it a while later is a lot ha…

I don't agree that more verbose code is necessarily more readable when the shorter code looks like familiar math. All you have to do is learn how operators broadcast across array-like structures, how slicing and filtering works. Perhaps with more complicated examples the shorter code becomes harder to read after months away? Mathematicians are able to handle a lot of compact equations. No doubt some of this comes dow…

Oh I don't mean to say verbose makes it more readable by default, I agree with you on that. I mostly meant that because the API is declarative (more geared at describing the result you want instead of the operations) it is easier to understand what's going on. A side effect of that is that it might be more verbose, which is the case of Polars vs pandas. In the end it's a personal thing which one you like the most. I do believe that if your deliverable is insights you get out of your analysis I can imagine that a less verbose API is practical to get things done quickly. But if you create pipelines that your colleagues have to quickly understand (or you in a couple of months) a read-optimized one makes more sense, even though it might take slightly more effort to write.
Post reply on HN