Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pandas 3.0

pandas.pydata.org

Post reply on HN
Print view View original

111–120 of 125 posts

Re: Pandas 3.0

#111

Post by [deleted] »

goatlover wrote: post #89
sampo wrote: post #36

Earlier quoted context omitted.

Historically 18 years ago, Pandas started as a project by someone working in finance to use Python instead of Excel, yet be nicer than using just raw Python dicts and Numpy arrays. For better or worse, like Excel and like the simpler programming languages of old, Pandas lets you overwrite data in place. Prepare some data df_pandas = pd.DataFrame({'a': [1, 2, 3, 4, 5], 'b': [10, 20, 30, 40, 50]}) df_polars = pl.from_p…

The Polars code puts me off as being too verbose and requiring too many steps. I love the broadcasting ability that Pandas gets from Numpy. It's what sceintific computing should look like in my opinon. Maybe R, Julia or some array-based language does it a bit better than Numpy/Pandas, but it's certainly not like the Polars example.

[deleted]

Re: Pandas 3.0

#112

Post by thijsn »

goatlover wrote: post #89
sampo wrote: post #36

Earlier quoted context omitted.

Historically 18 years ago, Pandas started as a project by someone working in finance to use Python instead of Excel, yet be nicer than using just raw Python dicts and Numpy arrays. For better or worse, like Excel and like the simpler programming languages of old, Pandas lets you overwrite data in place. Prepare some data df_pandas = pd.DataFrame({'a': [1, 2, 3, 4, 5], 'b': [10, 20, 30, 40, 50]}) df_polars = pl.from_p…

The Polars code puts me off as being too verbose and requiring too many steps. I love the broadcasting ability that Pandas gets from Numpy. It's what sceintific computing should look like in my opinon. Maybe R, Julia or some array-based language does it a bit better than Numpy/Pandas, but it's certainly not like the Polars example.

Polars is indeed more verbose when coming from pandas, but in my experience it is an advantage for when you're reading that same code after not having touched it for months.

pandas is write-optimized, so you can quickly and powerfully transform your data. Once you're used to it, it allows you to quickly get your work done. But figuring out what is happening in that code after returning to it a while later is a lot harder compared to Polars, which is more read-optimized. This read-optimized API coincidentally allows the engine to perform more optimizations because all implicit knowledge about data must be typed out instead of kept in your head.

Re: Pandas 3.0

#113

Post by jtrueb »

ciupicri wrote: post #104
jtrueb wrote: post #97

Earlier quoted context omitted.

I would describe it as the huge majority, reflecting on my pandas use over the years. Pretty much all of the data worth exploring in pandas over excel, some data gui, or polars involves timestamps.

Yeah, but is nanosecond-level resolution necessary? In many scenarios, a resolution of one second is adequate.

I don't need nanosecond accuracy. I just know there are a lot of scripts expecting it.

Re: Pandas 3.0

#114

Post by datsci_est_2015 »

mynameisash wrote: post #109
datsci_est_2015 wrote: post #93

Earlier quoted context omitted.

In what field do you work? > writing prompts and calling it a day What does this mean? They’re not creating pull requests and maintaining learning / analytics systems? This kind of vagueposting gets on my nerves.

> They’re not creating pull requests and maintaining learning / analytics systems? Sure, they check prompts into git. And there are a few notebooks that have been written and deployed, but most of that is collecting data and handing it off to ChatGPT. No, they're not maintaining learning/analytics systems. My team builds our data processing pipelines, and we support everything in production. > This kind of vagueposti…

I have never heard of this. What kind of insights are being generated? What kind of data? Am I unaware that we’re at the point that I can give a CSV of e.g. industrial measurement data to an LLM and it provides reliable and repeatable output? Are people making decisions based on the LLM output? Do the people making those decisions based on that output know that it might be completely hallucinated and the only response they’ll get from the “Data Scientists” is a shoulder shrug?

So many questions. That’s why I called it vague. I don’t know how any data scientist could read this and not have a million follow up questions. Is this offline learning? Online learning? What are the guardrails? Are there guardrails? Mostly, wtf?

Re: Pandas 3.0

#115

Post by goatlover »

thijsn wrote: post #112
goatlover wrote: post #89

Earlier quoted context omitted.

The Polars code puts me off as being too verbose and requiring too many steps. I love the broadcasting ability that Pandas gets from Numpy. It's what sceintific computing should look like in my opinon. Maybe R, Julia or some array-based language does it a bit better than Numpy/Pandas, but it's certainly not like the Polars example.

Polars is indeed more verbose when coming from pandas, but in my experience it is an advantage for when you're reading that same code after not having touched it for months. pandas is write-optimized, so you can quickly and powerfully transform your data. Once you're used to it, it allows you to quickly get your work done. But figuring out what is happening in that code after returning to it a while later is a lot ha…

I don't agree that more verbose code is necessarily more readable when the shorter code looks like familiar math. All you have to do is learn how operators broadcast across array-like structures, how slicing and filtering works. Perhaps with more complicated examples the shorter code becomes harder to read after months away? Mathematicians are able to handle a lot of compact equations.

No doubt some of this comes down to preference as to what's considered readable. I never really bought that argument that regular expressions create more problems than they're worth. Perhaps I side on the expressivity end of the readability debate.

Re: Pandas 3.0

#116

Post by datsci_est_2015 »

claytonjy wrote: post #106
lvl155 wrote: post #19

Earlier quoted context omitted.

Went from pandas to polars to duckdb. As mentioned elsewhere SQL is the most readable for me and LLM does most of the coding on my end (quant). So I need it at the most readable and rudimentary/step-wise level. OT, but I can’t imagine data science being a job category for too long. It’s got to be one of the first to go in AI age especially since the market is so saturated with mediocre talents.

Even before LLMs, Data Science was being replaced by more specialization, IME. Data Engineers took over the plumbing once they moved on from Scala and Spark. ML Engineers took over the modeling (and LLMs are now killing this job too, as it’s rare to need model training outside of big labs). Data analysts have to know SQL and python these days, and most DS are now just this, but with a nicer title and higher pay. Once…

Reading a comment like this makes me realize how broad the title “Data Scientist” is, especially this tidbit:

> as it’s rare to need model training outside of big labs

Do you think there are pre-trained models for e.g. process optimization for the primary metallurgy process for steel manufacturing? Industrial engineers don’t know anything about machine learning (by trade), and there are companies that bring specialized Data Science know-how to that industry to improve processes using modern data-driven methods, especially model building.

It’s almost like 99% of comments on this topic think that DS begins at image classification and ends at LLMs, with maybe a little bit of landing page A/B testing or something. Wild.

> Once upon a time I thought DS would be much more about deeper statistics and causal inference, but those have proven to be rare, niche needs outside soft science academia.

This is my entire career lol.

Re: Pandas 3.0

#117

Post by data-ottawa »

vegabook wrote: post #90
data-ottawa wrote: post #69

Earlier quoted context omitted.

Map is one operation pandas does nicely that most other “wrap a fast language” dataframe tools do poorly. When it feels like you’re writing some external udf thats executed in another environment, it does not feel as nice as throwing in a lambda, even if the lambda is not ideal.

you have map_elements in polars which does exactly this. https://docs.pola.rs/api/python/dev/reference/expressions/ap... You can also iter_rows into a lambda if you really want to. https://docs.pola.rs/api/python/stable/reference/dataframe/a... Personally I find it extremely rare that I need to do this given Polars expressions are so comprehensive, including when.then.otherwise when all else fails.

That one has a bit more friction than pandas because the return schema requirement -- pandas let's you get away with this bad practice.

It also does batches when you declare scalar outputs, but you can't control the batch size, which usually isn't an issue, but I've run into situations where it is.

Re: Pandas 3.0

#118

Post by data-ottawa »

condwanaland wrote: post #107
aidos wrote: post #98

Earlier quoted context omitted.

> allows functions to read the context they’re called in Can you show an example? Seems interesting considering that code knowing about external context is not generally a good pattern when it comes to maintainability (security, readability). I’ve lived through some horrific 10M line coldfusion codebases that embraced this paradigm to death - they were a whole other extreme where you could _write_ variables in the sc…

Say I have a dataframe called 'penguins' I can write code like: penguin_sizes Here, weight and height are columns inside the dataframe. But I can refer to them as if they were objects in the environment (I., e without quotes) because the select function looks for them inside the penguins dataframe (it's first argument) This is a very simple example but it's used extensively in some R paradigms

Yes, this exactly.

And its why you can do plot(x, sin) and get properly labelled graphs. It also powers the formula API that made caret and glm modules so easy to use.

Re: Pandas 3.0

#119

Post by [deleted] »

data-ottawa wrote: post #117
vegabook wrote: post #90

Earlier quoted context omitted.

you have map_elements in polars which does exactly this. https://docs.pola.rs/api/python/dev/reference/expressions/ap... You can also iter_rows into a lambda if you really want to. https://docs.pola.rs/api/python/stable/reference/dataframe/a... Personally I find it extremely rare that I need to do this given Polars expressions are so comprehensive, including when.then.otherwise when all else fails.

That one has a bit more friction than pandas because the return schema requirement -- pandas let's you get away with this bad practice. It also does batches when you declare scalar outputs, but you can't control the batch size, which usually isn't an issue, but I've run into situations where it is.

[deleted]

Re: Pandas 3.0

#120

Post by bikelang »

edschofield wrote: post #18

The design of Pandas is inferior in every way to Polars: API, memory use, speed, expressiveness. Pandas has been strictly worse since late 2023 and will never close the gap. Polars is multithreaded by default, written in a low-level language, has a powerful query engine, supports lazy, out-of memory execution, and isn’t constrained by any compatibility concerns with a warty, eager-only API and pre-Arrow data types th…

All of this is true and I agree with you - but this comment comes off a bit disrespectful.
Post reply on HN