Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pandas 3.0

pandas.pydata.org

Post reply on HN
Print view View original

101–110 of 125 posts

Re: Pandas 3.0

#101

Post by vaylian »

edschofield wrote: post #18

The design of Pandas is inferior in every way to Polars: API, memory use, speed, expressiveness. Pandas has been strictly worse since late 2023 and will never close the gap. Polars is multithreaded by default, written in a low-level language, has a powerful query engine, supports lazy, out-of memory execution, and isn’t constrained by any compatibility concerns with a warty, eager-only API and pre-Arrow data types th…

> The design of Pandas is inferior in every way to Polars

I used Pandas a lot with Jupyter notebooks. I don't have any experience with Polars. Is it also possible to work with Polars dataframes in Jupyter notebooks?

Re: Pandas 3.0

#102

Post by disgruntledphd2 »

vaylian wrote: post #101
edschofield wrote: post #18

The design of Pandas is inferior in every way to Polars: API, memory use, speed, expressiveness. Pandas has been strictly worse since late 2023 and will never close the gap. Polars is multithreaded by default, written in a low-level language, has a powerful query engine, supports lazy, out-of memory execution, and isn’t constrained by any compatibility concerns with a warty, eager-only API and pre-Arrow data types th…

> The design of Pandas is inferior in every way to Polars I used Pandas a lot with Jupyter notebooks. I don't have any experience with Polars. Is it also possible to work with Polars dataframes in Jupyter notebooks?

Yes. Most things just work with Polars. The one issue for me is the need for geopandas.

Re: Pandas 3.0

#103

Post by disgruntledphd2 »

quentindanjou wrote: post #78
v3ss0n wrote: post #20

Earlier quoted context omitted.

Sounds too much like an advertisement. Also we need to watch out when diving into Polars . Polars is VC backed Opensource project with cloud offering , which may become an opencore project - we know how those goes.

I was also thinking that this comment looks like an AD. Pandas does not have any paid option and isn't made directly for profit.

To be fair, as someone who's fought pandas for many years I agree with basically everything they said. The API design for Polars is much, much more intuitive. It's a base R to dplyr level change.

Re: Pandas 3.0

#104

Post by ciupicri »

jtrueb wrote: post #97
EForEndeavour wrote: post #55

Earlier quoted context omitted.

Do you mean the new default datetime resolution of microseconds instead of the previous nanosecond resolution? Obviously this will require adjustments to any code that requires ns resolution, but I'd bet that's a tiny minority of all pandas code ever written. Do you have a particular use case in mind for the problems this will cause?

I would describe it as the huge majority, reflecting on my pandas use over the years. Pretty much all of the data worth exploring in pandas over excel, some data gui, or polars involves timestamps.

Yeah, but is nanosecond-level resolution necessary? In many scenarios, a resolution of one second is adequate.

Re: Pandas 3.0

#105

Post by datsci_est_2015 »

jvican wrote: post #99
datsci_est_2015 wrote: post #94

Earlier quoted context omitted.

Might be cool once PySpark integrates with Polars, but for now like many others I’m stuck with dropping into pandas for non-vectorized operations

Is there any plan for this?

Not that I’m aware of. The Spark ecosystem seems a little too “stable” to be putting effort into that kind of development.

Edit: hah, based on the sibling comment, I stand corrected

Re: Pandas 3.0

#106

Post by claytonjy »

lvl155 wrote: post #19
postalcoder wrote: post #3

I've migrated off of pandas to polars for my workflows to reap the benefit of, in my experience a 10-20x speedup on average. I can't imagine anything bringing me back short of a performance miracle. LLMs have made syntax almost a non-barrier.

Went from pandas to polars to duckdb. As mentioned elsewhere SQL is the most readable for me and LLM does most of the coding on my end (quant). So I need it at the most readable and rudimentary/step-wise level. OT, but I can’t imagine data science being a job category for too long. It’s got to be one of the first to go in AI age especially since the market is so saturated with mediocre talents.

Even before LLMs, Data Science was being replaced by more specialization, IME.

Data Engineers took over the plumbing once they moved on from Scala and Spark. ML Engineers took over the modeling (and LLMs are now killing this job too, as it’s rare to need model training outside of big labs). Data analysts have to know SQL and python these days, and most DS are now just this, but with a nicer title and higher pay.

Once upon a time I thought DS would be much more about deeper statistics and causal inference, but those have proven to be rare, niche needs outside soft science academia.

Re: Pandas 3.0

#107

Post by condwanaland »

aidos wrote: post #98
data-ottawa wrote: post #65

Earlier quoted context omitted.

No other modern language will compete with R on ergonomics because of how it allows functions to read the context they’re called in, and S expressions are incredibly flexibly. The R manual is great. To say pandas just copied it but worse is overly dismissive. The core of pandas has always been indexing/reindexing, split-apply-combine, and slicing views. It’s a different approach than R’s data tables or frames.

> allows functions to read the context they’re called in Can you show an example? Seems interesting considering that code knowing about external context is not generally a good pattern when it comes to maintainability (security, readability). I’ve lived through some horrific 10M line coldfusion codebases that embraced this paradigm to death - they were a whole other extreme where you could _write_ variables in the sc…

Say I have a dataframe called 'penguins'

I can write code like: penguin_sizes Here, weight and height are columns inside the dataframe. But I can refer to them as if they were objects in the environment (I., e without quotes) because the select function looks for them inside the penguins dataframe (it's first argument)

This is a very simple example but it's used extensively in some R paradigms

Re: Pandas 3.0

#108

Post by devin-petersohn »

jvican wrote: post #99
datsci_est_2015 wrote: post #94

Earlier quoted context omitted.

Might be cool once PySpark integrates with Polars, but for now like many others I’m stuck with dropping into pandas for non-vectorized operations

Is there any plan for this?

Funny enough, I actually just (2 weeks ago) added support for streaming from Pyspark to Polars/DuckDB/etc through Arrow PyCapsule. By streaming, I mean actually streaming, not collecting all data at once. It won't be released probably until May/June but it's there: https://github.com/apache/spark/commit/ecf179c3485ba8bac72af...

Re: Pandas 3.0

#109

Post by mynameisash »

datsci_est_2015 wrote: post #93
mynameisash wrote: post #42

Earlier quoted context omitted.

Not GP, but as a data engineer who has worked with data scientists for 20 years, I think the assessment is unfortunately true. I used to work on teams where DS would put a ton of time into building quality models, gating production with defensible metrics. Now, my DS counterparts are writing prompts and calling it a day. I'm not at all convinced that the results are better, but I guess if you don't spend time (=money…

In what field do you work? > writing prompts and calling it a day What does this mean? They’re not creating pull requests and maintaining learning / analytics systems? This kind of vagueposting gets on my nerves.

> They’re not creating pull requests and maintaining learning / analytics systems?

Sure, they check prompts into git. And there are a few notebooks that have been written and deployed, but most of that is collecting data and handing it off to ChatGPT. No, they're not maintaining learning/analytics systems. My team builds our data processing pipelines, and we support everything in production.

> This kind of vagueposting gets on my nerves.

What is vague about my comment?

Whereas in the past, the DS teams I worked with would do feature engineering and rigorous evaluation of models with retraining based on different criteria, now I'm seeing that teams are being lazy and saying, "We'll let the LLM do things. It can handle unstructured data, and we can give it new data without additional work on our part." Hence, they're simply writing a prompt and not doing much more.

Re: Pandas 3.0

#110

Post by thereisnospork »

goatlover wrote: post #89
sampo wrote: post #36

Earlier quoted context omitted.

Historically 18 years ago, Pandas started as a project by someone working in finance to use Python instead of Excel, yet be nicer than using just raw Python dicts and Numpy arrays. For better or worse, like Excel and like the simpler programming languages of old, Pandas lets you overwrite data in place. Prepare some data df_pandas = pd.DataFrame({'a': [1, 2, 3, 4, 5], 'b': [10, 20, 30, 40, 50]}) df_polars = pl.from_p…

The Polars code puts me off as being too verbose and requiring too many steps. I love the broadcasting ability that Pandas gets from Numpy. It's what sceintific computing should look like in my opinon. Maybe R, Julia or some array-based language does it a bit better than Numpy/Pandas, but it's certainly not like the Polars example.

Likewise, I was considering trying Polaris until I saw that example. The pandas example is a good approximation of how I think and want to transform/process data even if it is ugly under the hood. I do occasionally find numpy and pandas annoying wrt when the return a view vs a copy but the cure seems worse than the disease.
Post reply on HN