Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pandas 3.0

pandas.pydata.org

Post reply on HN
Print view View original

91–100 of 125 posts

Re: Pandas 3.0

#91

Post by swyx »

OutOfHere wrote: post #87
swyx wrote: post #82

Earlier quoted context omitted.

its not even about the ingest, every major semver change now is a problem because now LLMs will need to contextually distinguish whether or not they are expected to output Pandas 2 or 3, unless ofc you explicitly prompt it.

I wouldn't worry about it because over a longer period, this automatically leans toward the more recent versions. There are multiple forces that exist to make this happen. The main exception is for legacy code requiring maintenance when they are unwilling to upgrade Pandas.

yes but a lot of legacy code wil still need to be maintained and written. you dont see how this can be confusing/annoying?

Re: Pandas 3.0

#92

Post by datsci_est_2015 »

lvl155 wrote: post #19
postalcoder wrote: post #3

I've migrated off of pandas to polars for my workflows to reap the benefit of, in my experience a 10-20x speedup on average. I can't imagine anything bringing me back short of a performance miracle. LLMs have made syntax almost a non-barrier.

Went from pandas to polars to duckdb. As mentioned elsewhere SQL is the most readable for me and LLM does most of the coding on my end (quant). So I need it at the most readable and rudimentary/step-wise level. OT, but I can’t imagine data science being a job category for too long. It’s got to be one of the first to go in AI age especially since the market is so saturated with mediocre talents.

> It’s got to be one of the first to go in AI age especially since the market is so saturated with mediocre talents.

Depends what your definition of “to go” means. Responsibilities swallowed by peers? Sure, and new job titles might pop up like Research & Development Engineer or something.

The discipline of creating automated systems to extract insights from data to create business value? I can’t really see that going anywhere. I mean, why tf would we be building so many data centers if there’s no value in the data they’re storing.

Re: Pandas 3.0

#93

Post by datsci_est_2015 »

mynameisash wrote: post #42
iugtmkbdfil834 wrote: post #29

Earlier quoted context omitted.

This is interesting. I wanted to dig into it a little since I am not sure I am following the logic of that statement. Do you mean that AI would take over the field, because by default most people there are already not producing anything that a simple 'talk to data' LLM won't deliver?

Not GP, but as a data engineer who has worked with data scientists for 20 years, I think the assessment is unfortunately true. I used to work on teams where DS would put a ton of time into building quality models, gating production with defensible metrics. Now, my DS counterparts are writing prompts and calling it a day. I'm not at all convinced that the results are better, but I guess if you don't spend time (=money…

In what field do you work?

> writing prompts and calling it a day

What does this mean? They’re not creating pull requests and maintaining learning / analytics systems?

This kind of vagueposting gets on my nerves.

Re: Pandas 3.0

#94

Post by datsci_est_2015 »

edschofield wrote: post #18

The design of Pandas is inferior in every way to Polars: API, memory use, speed, expressiveness. Pandas has been strictly worse since late 2023 and will never close the gap. Polars is multithreaded by default, written in a low-level language, has a powerful query engine, supports lazy, out-of memory execution, and isn’t constrained by any compatibility concerns with a warty, eager-only API and pre-Arrow data types th…

Might be cool once PySpark integrates with Polars, but for now like many others I’m stuck with dropping into pandas for non-vectorized operations

Re: Pandas 3.0

#95

Post by nothrowaways »

data-ottawa wrote: post #59
edschofield wrote: post #18

The design of Pandas is inferior in every way to Polars: API, memory use, speed, expressiveness. Pandas has been strictly worse since late 2023 and will never close the gap. Polars is multithreaded by default, written in a low-level language, has a powerful query engine, supports lazy, out-of memory execution, and isn’t constrained by any compatibility concerns with a warty, eager-only API and pre-Arrow data types th…

Pandas deserves a ton of respect in my opinion. I built my career on knowing it well and using it daily for a decade, so I’m biased. Pandas created the modern Python data stack when there was not really any alternatives (except R and closed source). The original split-apply-combine paradigm was well thought out, simple, and effective, and the built in tools to read pretty much anything (including all of your awful cs…

Very well articulated.

Re: Pandas 3.0

#96

Post by mjhay »

thegabriele wrote: post #28
postalcoder wrote: post #3

I've migrated off of pandas to polars for my workflows to reap the benefit of, in my experience a 10-20x speedup on average. I can't imagine anything bringing me back short of a performance miracle. LLMs have made syntax almost a non-barrier.

" 10-20x speedup on average. " Is this everyone's experience?

It’s a typical experience. Polars is fast, and Pandas is very slow and memory-hungry. It would be one thing if Pandas had a good API, but it doesn’t.

Re: Pandas 3.0

#97

Post by jtrueb »

EForEndeavour wrote: post #55
jtrueb wrote: post #26

That timestamp resolution discrepancy is going to cause so many problems

Do you mean the new default datetime resolution of microseconds instead of the previous nanosecond resolution? Obviously this will require adjustments to any code that requires ns resolution, but I'd bet that's a tiny minority of all pandas code ever written. Do you have a particular use case in mind for the problems this will cause?

I would describe it as the huge majority, reflecting on my pandas use over the years. Pretty much all of the data worth exploring in pandas over excel, some data gui, or polars involves timestamps.

Re: Pandas 3.0

#98

Post by aidos »

data-ottawa wrote: post #65
vegabook wrote: post #54

Earlier quoted context omitted.

"revolutionary"? It just copied and pasted the decades-old R (previous "S") dataframe into Python, including all the paradigms (with worse ergonomics since it's not baked into the language).

No other modern language will compete with R on ergonomics because of how it allows functions to read the context they’re called in, and S expressions are incredibly flexibly. The R manual is great. To say pandas just copied it but worse is overly dismissive. The core of pandas has always been indexing/reindexing, split-apply-combine, and slicing views. It’s a different approach than R’s data tables or frames.

> allows functions to read the context they’re called in

Can you show an example? Seems interesting considering that code knowing about external context is not generally a good pattern when it comes to maintainability (security, readability).

I’ve lived through some horrific 10M line coldfusion codebases that embraced this paradigm to death - they were a whole other extreme where you could _write_ variables in the scope of where you were called from!

Re: Pandas 3.0

#99

Post by jvican »

datsci_est_2015 wrote: post #94
edschofield wrote: post #18

The design of Pandas is inferior in every way to Polars: API, memory use, speed, expressiveness. Pandas has been strictly worse since late 2023 and will never close the gap. Polars is multithreaded by default, written in a low-level language, has a powerful query engine, supports lazy, out-of memory execution, and isn’t constrained by any compatibility concerns with a warty, eager-only API and pre-Arrow data types th…

Might be cool once PySpark integrates with Polars, but for now like many others I’m stuck with dropping into pandas for non-vectorized operations

Is there any plan for this?

Re: Pandas 3.0

#100

Post by __mharrison__ »

edschofield wrote: post #18

The design of Pandas is inferior in every way to Polars: API, memory use, speed, expressiveness. Pandas has been strictly worse since late 2023 and will never close the gap. Polars is multithreaded by default, written in a low-level language, has a powerful query engine, supports lazy, out-of memory execution, and isn’t constrained by any compatibility concerns with a warty, eager-only API and pre-Arrow data types th…

"every way" is strong words.

Pandas is better for plotting and third party integration.

Post reply on HN