Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What category theory teaches us about dataframes

mchav.github.io

Post reply on HN
Print view View original

41–50 of 68 posts

Re: What category theory teaches us about dataframes

#41

Post by jimbokun »

hermitcrab wrote: post #33
mrlongroots wrote: post #22

Earlier quoted context omitted.

Algebras are also nice for implementations. If you can decompose a domain into a few algebraic primitives you can write nice SIMD/CUDA kernels for those primitives. To your point, I wonder if the 73 distinct transforms were just different defaults/usability wrappers over these. And you may also get into situations where kernels can be fused together or other batching constraints enable optimizations that nice algebra…

They are effectively highly level wrappers over the most primitive operations. High enough level that they can be used from a GUI, rather than code. It is a balance. Too few transforms and they become to low level for my users. Too many and you struggle to find the transform you want.

You don’t have to limit the transforms you offer users to just the core ones. But for your own sanity you can implement the none core ones in terms of the core ones.

Re: What category theory teaches us about dataframes

#44

Post by hermitcrab »

jimbokun wrote: post #39
tikhonj wrote: post #36

Earlier quoted context omitted.

You can have both: you start with a small, mathematically inspired algebraic core, then you express the higher-level more user-friendly operations in terms of the algebraic core. As long as your core primitives are well designed (easier said than done!), this accomplishes two things: it makes your implementation simpler, and it helps guide and constrain your user-facing design. This latter aspect is a bit unintuitive…

For one thing it gives users of your library fewer concepts to learn.

Yes, but fewer concepts may not be simpler in practice. E.g. assembler is simpler than C++, but I wouldn't want to write a big program in assembler.

Re: What category theory teaches us about dataframes

#46

Post by ux266478 »

bjourne wrote: post #32
few wrote: post #3

I felt like one or two decades ago, all the rage was about rewriting programs into just two primitives: map and reduce. For example filter can be expressed as: is_even = lambda x: x % 2 == 0 mapped = map(lambda x: [x] if is_even(x) else [], data) filtered = reduce(lambda x, y: x + y, mapped, []) But then the world moved on from it because it was too rigid

Reductions are painful because they specify a sequence of ordered operations. Runtime is O(N), where N is the sequence length, regardless of amount of hardware. So you want to work at a higher level where you can exploit commutativity and independence of some (or even most) operations.

You're right it's primarily a runtime + compiler + language issue. I really don't understand why people tried to force functional programming in environments without decent algebraic reasoning mechanisms.

Modern graph reducers have inherent confluence and aren't reliant on explicit commutation. They can do everything parallel and out of order (until they have to talk to some extrinsic thing like getting input or spitting out output), including arbitrary side-effectual mutation. We really live in the future.

Re: What category theory teaches us about dataframes

#47

Post by mr_toad »

getnormality wrote: post #6

Hmm. Folks trying to discover the elegant core of data frame manipulation by studying... pandas usage patterns. When R's dplyr solved this over a decade ago, mostly by respecting SQL and following its lead. The pandas API feels like someone desperately needed a wheel and had never heard of a wheel, so they made a heptagon, and now millions of people are riding on heptagon wheels. Because it's locked in now, everyone…

> just by porting SQL to your language

You make it sound like writing an SQL parser and query engine is a trivial task. Have you ever looked at the implementation of a query engine to see what’s actually involved? You can’t just ‘build on SQL’, you have to build a substantial library of functions to build SQL on top of.

Re: What category theory teaches us about dataframes

#48

Post by michaelbarton »

gregw2 wrote: post #13
DangitBobby wrote: post #10

Earlier quoted context omitted.

I guess I have pandas brain because I definitely want to drop duplicates, 100% of the time I'm worried about duplicates and 99% of the time the only thing I want to do with duplicates is drop them. When you've got 19 columns it's _really fucking annoying_ if the tool you're using doesn't have an obvious way to say `select distinct on ( ) from my_shit`. Close second at say, 98% of the time, I want to a get a count of…

You articulate your case well, thank you! I always warn people (particularly junior people) though that blindly dropping duplicates is a dangerous habit because it helps you and others in your organization ignore the causes of bad data quickly without getting them fixed at the source. Over time, that breeds a lot of complexity and inefficiency. And it can easily mask flaws in one's own logic or understanding of the d…

Exactly. It’s not that getting rid of duplicates is bad, is that they may be a symptom of something worse. E.g. incorrect aggregation logic

Re: What category theory teaches us about dataframes

#49

Post by toxik »

bjourne wrote: post #32
few wrote: post #3

I felt like one or two decades ago, all the rage was about rewriting programs into just two primitives: map and reduce. For example filter can be expressed as: is_even = lambda x: x % 2 == 0 mapped = map(lambda x: [x] if is_even(x) else [], data) filtered = reduce(lambda x, y: x + y, mapped, []) But then the world moved on from it because it was too rigid

Reductions are painful because they specify a sequence of ordered operations. Runtime is O(N), where N is the sequence length, regardless of amount of hardware. So you want to work at a higher level where you can exploit commutativity and independence of some (or even most) operations.

You can reduce in parallel. That was the whole point of MapReduce. For example, the sum abcdefgh can be found by first ab, cd, ef, gh; then those results (ab)(cd), (ef)(gh); then the final result by (abcd)(efgh). That's just three steps to compute seven sums.

Re: What category theory teaches us about dataframes

#50

Post by toxik »

Pandas and so on exist for the same reason Django's ORM and SqlAlchemy do: people do not want to string interpolate to talk to their database. SQL is great for DBA's, and absolutely sucks for programmers. Microsoft was really onto something with LINQ, in my opinion.
Post reply on HN