Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What category theory teaches us about dataframes

mchav.github.io

Post reply on HN
Print view View original

51–60 of 68 posts

Re: What category theory teaches us about dataframes

#51

Post by heavenlyblue »

bjourne wrote: post #32
few wrote: post #3

I felt like one or two decades ago, all the rage was about rewriting programs into just two primitives: map and reduce. For example filter can be expressed as: is_even = lambda x: x % 2 == 0 mapped = map(lambda x: [x] if is_even(x) else [], data) filtered = reduce(lambda x, y: x + y, mapped, []) But then the world moved on from it because it was too rigid

Reductions are painful because they specify a sequence of ordered operations. Runtime is O(N), where N is the sequence length, regardless of amount of hardware. So you want to work at a higher level where you can exploit commutativity and independence of some (or even most) operations.

Reduce is massively parallel for commutative operations

Re: What category theory teaches us about dataframes

#52

Post by gwerbin »

sweezyjeezy wrote: post #23
rich_sasha wrote: post #2

The article starts well, on trying to condense pandas' gaziliion of inconsistent and continuously-deprecated functions with tens of keyword arguments into a small, condensed set of composable operations - but it lost me then. The more interesting nugget for me is about this project they mention: https://modin.readthedocs.io/en/latest/index.html called Modin, which apparently went to the effort of analysing common pan…

The pandas API is awful, but it's kind of interesting why. It was started as a financial time series manipulation library ('panels') in a hedge fund and a lot of the quirks come from that. For example the unique obsession with the 'index' - functions seemingly randomly returning dataframes with column data as the index, or having to write index=False every single time you write to disk, or it appending the index to t…

> The pandas API is awful

I hate to be the "you're holding it wrong" guy but 90% of "Pandas bad!" posts I find are either outright misinformed or mischaracterizing one person's particular opinion as some kind of common truth. This one is both!

> That comes from the assumption that there is almost always a meaningful index (timestamps)

The index can be literally any unique row label or ID. It's idiosyncratic among "data frames" (SQL has no equivalent concept, and the R community has disowned theirs), but it's really not such a crazy thing to have row labels built into your data table. Excel supports this in several different ways (frozen columns, VLOOKUP) and users expect it in just about any table-oriented GUI tool.

> having to write index=False every single time you write to disk

If you're actually using the index as it's meant to be used, you'd see why this isn't the default setting.

> functions seemingly randomly returning dataframes with column data as the index

I assume you're talking about the behavior of .groupby() and .rolling()? It's never been random. Under-documented and hard to reason about group_keys= and related options, yes. But not random.

> appending the index to the Series numpy data leading to incredibly confusing bugs

I've been using Pandas professionally almost daily since 2015 and I have no idea what this means.

Re: What category theory teaches us about dataframes

#53

Post by gwerbin »

pavodive wrote: post #8

When I started reading about pandas complexity and the smaller set of operations needed, couldn't help but think of R's data.table simplicity. Granted, it's got more than 15 functions, but its simplicity seems to me very similar to what the author presented in the end.

data.table "simplicity" is actually a huge set of features, they just have a clever and compact way to express those features in code. At the same time, there is effectively no standard-eval programmatic interface for it, which makes it a headache for building programs rather than scripting with. data.table is amazing, but it is anything but simple IMO.

Re: What category theory teaches us about dataframes

#54

Post by gwerbin »

getnormality wrote: post #6

Hmm. Folks trying to discover the elegant core of data frame manipulation by studying... pandas usage patterns. When R's dplyr solved this over a decade ago, mostly by respecting SQL and following its lead. The pandas API feels like someone desperately needed a wheel and had never heard of a wheel, so they made a heptagon, and now millions of people are riding on heptagon wheels. Because it's locked in now, everyone…

> There's no reason to invent a completely new API for them

Yes there is: SQL is one of many possible ways to interact with tabular data, why should it be the only one? R data frames literally pioneered an alternative API. Dplyr is fantastic for many reasons, one of those being that people like the verb-based approach

Furthermore I argue that dplyr is not particularly similar to SQL in the way you actually use it and how it's actually interpreted/executed.

As for the rest I feel like you're just stating your preferences as fact.

Re: What category theory teaches us about dataframes

#55

Post by gwerbin »

mr_toad wrote: post #47
getnormality wrote: post #6

Hmm. Folks trying to discover the elegant core of data frame manipulation by studying... pandas usage patterns. When R's dplyr solved this over a decade ago, mostly by respecting SQL and following its lead. The pandas API feels like someone desperately needed a wheel and had never heard of a wheel, so they made a heptagon, and now millions of people are riding on heptagon wheels. Because it's locked in now, everyone…

> just by porting SQL to your language You make it sound like writing an SQL parser and query engine is a trivial task. Have you ever looked at the implementation of a query engine to see what’s actually involved? You can’t just ‘build on SQL’, you have to build a substantial library of functions to build SQL on top of.

Also it's not like dplyr is anything close to a "port" of SQL. You could in theory collect dplyr verbs and compile them to SQL, sure. That's what ORMs typically do, and what the Spark API does (and its descendants such as Polars).

"Porting" SQL to your language usually means inventing a new API for relational and/or tabular data access that feels ergonomic in the host language, and then either compiling it to SQL or executing it in some kind of array processing backend, or DataFusion if you're fancy like that.

Re: What category theory teaches us about dataframes

#56

Post by _diyar »

gwerbin wrote: post #52
sweezyjeezy wrote: post #23

Earlier quoted context omitted.

The pandas API is awful, but it's kind of interesting why. It was started as a financial time series manipulation library ('panels') in a hedge fund and a lot of the quirks come from that. For example the unique obsession with the 'index' - functions seemingly randomly returning dataframes with column data as the index, or having to write index=False every single time you write to disk, or it appending the index to t…

> The pandas API is awful I hate to be the "you're holding it wrong" guy but 90% of "Pandas bad!" posts I find are either outright misinformed or mischaracterizing one person's particular opinion as some kind of common truth. This one is both! > That comes from the assumption that there is almost always a meaningful index (timestamps) The index can be literally any unique row label or ID. It's idiosyncratic among "da…

I think the commenter you are replying to might well understand these nuances. The point is not that Pandas is inscrutable, but instead that it‘s annoying to use in many common use-cases.

Re: What category theory teaches us about dataframes

#57

Post by caseyross »

Interesting idea. I feel like it could be productive to categorize operations by their result shape as well:

- Row select: From N rows, produce 0-N rows.

- Column select: From N columns, produce 0-N columns.

- Table add: From MxN and OxP tables, produce max M+OxN+P table.

- Table subtract: From MxN and OxP tables, produce min 0x0 table.

This line of thinking reveals some normally hard-to-see similarities, such as `groupby` and `dedupe` sharing the same underlying mechanism. (i.e., both are "collapsing" row selects.)

Re: What category theory teaches us about dataframes

#58

Post by rdevilla »

rich_sasha wrote: post #2

The article starts well, on trying to condense pandas' gaziliion of inconsistent and continuously-deprecated functions with tens of keyword arguments into a small, condensed set of composable operations - but it lost me then. The more interesting nugget for me is about this project they mention: https://modin.readthedocs.io/en/latest/index.html called Modin, which apparently went to the effort of analysing common pan…

> Pandas is clearly quite ergonomic for various exploratory interactive analyses, but the API is, imo, awful.

Having previously inherited (and now dispossessed) an un-disentangleable pile of Python, pandas, and SQL hacks reminiscent of a spreadsheet rammed with inscrutable Excel formulae, I have no idea how data scientists collaborate on anything with this technology. It's like when bioinformatics was full of write-only Perl code that was maybe executed successfully once for the purposes of a study or paper, and was kept around for future archaeologists to hopefully one day resuscitate when the need may arise again.

If programmers are expected to just throw garbage like this at the next asshole with the misfortune to have to maintain code that was never designed to be maintained, it's not a surprise that the industry is once again moving towards write-only code, this time produced at scale by LLMs.

It's like we're back to Visual Studio Ultimate slopping out 10k lines of XAML in response to your dragging and dropping in the WYSIWYG. There is a reason nobody does this any more.

Re: What category theory teaches us about dataframes

#59

Post by esafak »

hermitcrab wrote: post #31
whattheheckheck wrote: post #28

Earlier quoted context omitted.

Have you heard of the book Mathematics for Big data https://github.com/Accla/d4m He says himself the ideas are more important than the software package

D4M seems to be a library, not a book. Or am I missing something?

https://mitpress.mit.edu/9780262038393/mathematics-of-big-da...

Re: What category theory teaches us about dataframes

#60

Post by getnormality »

gwerbin wrote: post #55
mr_toad wrote: post #47

Earlier quoted context omitted.

> just by porting SQL to your language You make it sound like writing an SQL parser and query engine is a trivial task. Have you ever looked at the implementation of a query engine to see what’s actually involved? You can’t just ‘build on SQL’, you have to build a substantial library of functions to build SQL on top of.

Also it's not like dplyr is anything close to a "port" of SQL. You could in theory collect dplyr verbs and compile them to SQL, sure. That's what ORMs typically do, and what the Spark API does (and its descendants such as Polars). "Porting" SQL to your language usually means inventing a new API for relational and/or tabular data access that feels ergonomic in the host language, and then either compiling it to SQL or…

dplyr straightforwardly transpiles to SQL through the dbplyr package, so it's semantically pretty close to a port, even though the syntax is a bit different (better).
Post reply on HN