Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What category theory teaches us about dataframes

mchav.github.io

Post reply on HN
Print view View original

31–40 of 68 posts

Re: What category theory teaches us about dataframes

#31

Post by hermitcrab »

whattheheckheck wrote: post #28
hermitcrab wrote: post #20

I guess this article is an interesting exercise from a pure maths point of view. But, as someone developing a drag and drop data wrangling tool the important thing is creating a set of composable operations/primitive that are meaningful and useful to your end user. We have ended up 73 distinct transforms in Easy Data Transform. Sure they overlap to an extent, but feel they are at the right semantic level for our user…

Have you heard of the book Mathematics for Big data https://github.com/Accla/d4m He says himself the ideas are more important than the software package

D4M seems to be a library, not a book. Or am I missing something?

Re: What category theory teaches us about dataframes

#32

Post by bjourne »

few wrote: post #3

I felt like one or two decades ago, all the rage was about rewriting programs into just two primitives: map and reduce. For example filter can be expressed as: is_even = lambda x: x % 2 == 0 mapped = map(lambda x: [x] if is_even(x) else [], data) filtered = reduce(lambda x, y: x + y, mapped, []) But then the world moved on from it because it was too rigid

Reductions are painful because they specify a sequence of ordered operations. Runtime is O(N), where N is the sequence length, regardless of amount of hardware. So you want to work at a higher level where you can exploit commutativity and independence of some (or even most) operations.

Re: What category theory teaches us about dataframes

#33

Post by hermitcrab »

mrlongroots wrote: post #22
hermitcrab wrote: post #20

I guess this article is an interesting exercise from a pure maths point of view. But, as someone developing a drag and drop data wrangling tool the important thing is creating a set of composable operations/primitive that are meaningful and useful to your end user. We have ended up 73 distinct transforms in Easy Data Transform. Sure they overlap to an extent, but feel they are at the right semantic level for our user…

Algebras are also nice for implementations. If you can decompose a domain into a few algebraic primitives you can write nice SIMD/CUDA kernels for those primitives. To your point, I wonder if the 73 distinct transforms were just different defaults/usability wrappers over these. And you may also get into situations where kernels can be fused together or other batching constraints enable optimizations that nice algebra…

They are effectively highly level wrappers over the most primitive operations. High enough level that they can be used from a GUI, rather than code.

It is a balance. Too few transforms and they become to low level for my users. Too many and you struggle to find the transform you want.

Re: What category theory teaches us about dataframes

#34

Post by rich_sasha »

bbkane wrote: post #11
rich_sasha wrote: post #2

The article starts well, on trying to condense pandas' gaziliion of inconsistent and continuously-deprecated functions with tens of keyword arguments into a small, condensed set of composable operations - but it lost me then. The more interesting nugget for me is about this project they mention: https://modin.readthedocs.io/en/latest/index.html called Modin, which apparently went to the effort of analysing common pan…

Check out polars- I find it much more intuitive than pandas as it looks closer to SQL (and I learned SQL first). Maybe you'll feel the same way!

I've looked at Polars. My sense is that Pandas is an interactive data analysis library poorly suited to production uses, and Polars is the other way around. Seemed quite verbose for example. Sometimes doing `series["2026"]` is exactly the right thing to type.

Re: What category theory teaches us about dataframes

#35

Post by mamcx »

getnormality wrote: post #12
DangitBobby wrote: post #10

Earlier quoted context omitted.

I guess I have pandas brain because I definitely want to drop duplicates, 100% of the time I'm worried about duplicates and 99% of the time the only thing I want to do with duplicates is drop them. When you've got 19 columns it's _really fucking annoying_ if the tool you're using doesn't have an obvious way to say `select distinct on ( ) from my_shit`. Close second at say, 98% of the time, I want to a get a count of…

Duplicates in source data are almost always a sign of bad data modeling, or of analysts and engineers disregarding a good data model. But I agree that this ubiquitous antipattern that nobody should be doing can still be usefully made concise. There should be a select distinct * operation. And FWIW I personally hate writing raw SQL. But the problem with the API is not the data operations available, it's the syntax and…

> Duplicates in source data are almost always a sign of bad data modeling

Nope. Duplicates in source data(INPUT) is natural, correct and MUST be supported or almost all data become impossible.

What is the actual problem is the OUTPUT. Duplicates on the OUTPUT need to be controlled and explicit. In general, we need in the OUTPUT a unique rowby a N-key, but probably not need it to be unique for the rest, so, in the relational model, you need unique for a combination of columns (rarely, by ALL of them).

Re: What category theory teaches us about dataframes

#36

Post by tikhonj »

hermitcrab wrote: post #20

I guess this article is an interesting exercise from a pure maths point of view. But, as someone developing a drag and drop data wrangling tool the important thing is creating a set of composable operations/primitive that are meaningful and useful to your end user. We have ended up 73 distinct transforms in Easy Data Transform. Sure they overlap to an extent, but feel they are at the right semantic level for our user…

You can have both: you start with a small, mathematically inspired algebraic core, then you express the higher-level more user-friendly operations in terms of the algebraic core.

As long as your core primitives are well designed (easier said than done!), this accomplishes two things: it makes your implementation simpler, and it helps guide and constrain your user-facing design. This latter aspect is a bit unintuitive (why would you want more constraints to work around?), but I've seen it lead to much better interface designs in multiple projects. By forcing yourself to express user-level affordances in terms of a small conceptual core, you end up with a user design that is more internally consistent and composable.

Re: What category theory teaches us about dataframes

#37

Post by doug_durham »

getnormality wrote: post #12
DangitBobby wrote: post #10

Earlier quoted context omitted.

I guess I have pandas brain because I definitely want to drop duplicates, 100% of the time I'm worried about duplicates and 99% of the time the only thing I want to do with duplicates is drop them. When you've got 19 columns it's _really fucking annoying_ if the tool you're using doesn't have an obvious way to say `select distinct on ( ) from my_shit`. Close second at say, 98% of the time, I want to a get a count of…

Duplicates in source data are almost always a sign of bad data modeling, or of analysts and engineers disregarding a good data model. But I agree that this ubiquitous antipattern that nobody should be doing can still be usefully made concise. There should be a select distinct * operation. And FWIW I personally hate writing raw SQL. But the problem with the API is not the data operations available, it's the syntax and…

Duplicates are a sign of reality. Only where you have the resources to have dedicated people clean and organize data do you have well modeled data. Pandas is a power tool for making sense of real data.

Re: What category theory teaches us about dataframes

#38

Post by doug_durham »

getnormality wrote: post #6

Hmm. Folks trying to discover the elegant core of data frame manipulation by studying... pandas usage patterns. When R's dplyr solved this over a decade ago, mostly by respecting SQL and following its lead. The pandas API feels like someone desperately needed a wheel and had never heard of a wheel, so they made a heptagon, and now millions of people are riding on heptagon wheels. Because it's locked in now, everyone…

SQL only works on well defined data sets that obey relational calculus rules. Pandas is a power tool for dealing with data as you find it. Without Pandas you are stuck with tools like Excel.

Re: What category theory teaches us about dataframes

#39

Post by jimbokun »

tikhonj wrote: post #36
hermitcrab wrote: post #20

I guess this article is an interesting exercise from a pure maths point of view. But, as someone developing a drag and drop data wrangling tool the important thing is creating a set of composable operations/primitive that are meaningful and useful to your end user. We have ended up 73 distinct transforms in Easy Data Transform. Sure they overlap to an extent, but feel they are at the right semantic level for our user…

You can have both: you start with a small, mathematically inspired algebraic core, then you express the higher-level more user-friendly operations in terms of the algebraic core. As long as your core primitives are well designed (easier said than done!), this accomplishes two things: it makes your implementation simpler, and it helps guide and constrain your user-facing design. This latter aspect is a bit unintuitive…

For one thing it gives users of your library fewer concepts to learn.

Re: What category theory teaches us about dataframes

#40

Post by entropicdrifter »

rich_sasha wrote: post #34
bbkane wrote: post #11

Earlier quoted context omitted.

Check out polars- I find it much more intuitive than pandas as it looks closer to SQL (and I learned SQL first). Maybe you'll feel the same way!

I've looked at Polars. My sense is that Pandas is an interactive data analysis library poorly suited to production uses, and Polars is the other way around. Seemed quite verbose for example. Sometimes doing `series["2026"]` is exactly the right thing to type.

You can do that in Polars, too
Post reply on HN