Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Category Theory ∩ Machine Learning

github.com

Post reply on HN
Print view View original

11–20 of 73 posts

Re: Category Theory ∩ Machine Learning

#11

Post by red_trumpet »

adamnemecek wrote: post #3

I have recently written a paper on understanding machine learning via the lens of Hopf algebra https://arxiv.org/abs/2302.01834 . Hopf algebras (which are really just tensors with recurrence relations built in) subsume convnets, transformers and diffusion model and also provide a theoretically better autodiff that operates within single layers as opposed to across entire graphs. Furthermore, there is a correspondence…

Your paper didn't pass my smell test at all, tbh. For example the formula you write about "product" and "coproduct" in section 3 is literally identical (as "=" is symmetric). In section 4.2 you write "the product is the standard tensor product" with a formula that doesn't at all involve the map m: A \otimes A \to A. The formula you write is the induced product on A \otimes A, assuming that you already have a product on A. The formula for "coproduct" is just an example[1] of a coproduct, not every coproduct has to look that way.

[1] https://en.wikipedia.org/wiki/Coalgebra#Examples

Re: Category Theory ∩ Machine Learning

#12

Post by adamnemecek »

hgsgm wrote: post #9
adamnemecek wrote: post #4

Earlier quoted context omitted.

The field needs better foundations. CT is pretty good.

Why? How? The OP GitHub site doesn't promote any material that introduces the concepts at all. The "survey" paper at the top is nigh-impenetrable. I'm sure the category theorists are having fun modelling machine learning, but it doesn't show how machine learning benefits from the category theory.

Residual connection serves as a feedback/trace a la trace in traced monoidal categories. That is one insight I have gleaned from CT.

Re: Category Theory ∩ Machine Learning

#13

Post by hgsgm »

gexaha wrote: post #6
adamnemecek wrote: post #3

I have recently written a paper on understanding machine learning via the lens of Hopf algebra https://arxiv.org/abs/2302.01834 . Hopf algebras (which are really just tensors with recurrence relations built in) subsume convnets, transformers and diffusion model and also provide a theoretically better autodiff that operates within single layers as opposed to across entire graphs. Furthermore, there is a correspondence…

could you advertise your research a bit less often, please? i see your post like literally almost every other day here

Or at least explain it in more accessible way. Every time Adam posts about the paper, it gets confused comments and no engagement on the content, because it's pretty deep graduate level pure math, which is occasionally seen but rare on HN.

Re: Category Theory ∩ Machine Learning

#14

Post by Yoric »

hgsgm wrote: post #9
adamnemecek wrote: post #4

Earlier quoted context omitted.

The field needs better foundations. CT is pretty good.

Why? How? The OP GitHub site doesn't promote any material that introduces the concepts at all. The "survey" paper at the top is nigh-impenetrable. I'm sure the category theorists are having fun modelling machine learning, but it doesn't show how machine learning benefits from the category theory.

Category Theory (just as all mathematical models for programming or subsets thereof) are building blocks for reasoning on what we build. Past applications of such mathematical models include:

- programming languages with semantics that are better adapted to specific problems (e.g. Rust's ownership);

- better compilers (see e.g. Haskell's supercompiler, which puts to shame `constexpr`-style features);

- better static analyzers (e.g. better type systems, abstract interpretation, model checkers).

In the case of Machine Learning, it might some day help us create Machine Learning that we can understand and trust better. Or it might fail. Or it might help us invent something different entirely, in 30 years.

Re: Category Theory ∩ Machine Learning

#15

Post by resource0x »

adamnemecek wrote: post #12
hgsgm wrote: post #9

Earlier quoted context omitted.

Why? How? The OP GitHub site doesn't promote any material that introduces the concepts at all. The "survey" paper at the top is nigh-impenetrable. I'm sure the category theorists are having fun modelling machine learning, but it doesn't show how machine learning benefits from the category theory.

Residual connection serves as a feedback/trace a la trace in traced monoidal categories. That is one insight I have gleaned from CT.

You beat me to this. Indeed, of all categories the monoidal ones are the most potent. Look how nicely they fit in crypto ledgers: https://www.cl.cam.ac.uk/events/syco/3/slides/Nester.pdf

\s

Re: Category Theory ∩ Machine Learning

#16

Post by donnowhy »

category theory is 'native 2-dimensional' math. i.e. category theory explains everything in terms of graphs, where a graph is made from two different sorts of 'entities', nodes and vertices i.e. categories and morphisms

this being math, I wonder to which extent can category theory be re-expressed in terms of sets.

perhaps a better question is if category theory can be re-expressed (or founded on) functions?

lastly, I wonder if category theory can be expressed in terms of functions (i think maybe it can, without sets?) why shouldn't it be expressible in terms of sets (for some reason I don't think just sets are sufficient, may have to define functions (which possible in terms of sets) before 'expressing' categories starting with set theory)?

Re: Category Theory ∩ Machine Learning

#17

Post by ianandrich »

adamnemecek wrote: post #3

I have recently written a paper on understanding machine learning via the lens of Hopf algebra https://arxiv.org/abs/2302.01834 . Hopf algebras (which are really just tensors with recurrence relations built in) subsume convnets, transformers and diffusion model and also provide a theoretically better autodiff that operates within single layers as opposed to across entire graphs. Furthermore, there is a correspondence…

I just read your Coinductive guide to inductive transformer heads paper.

My mind is blown.

Is the Hopf Algebra based ML framework you are working on on your github? I took a glance, but you have 1500 repositories and it wasn't on the first few of them.

Re: Category Theory ∩ Machine Learning

#18

Post by jurynulifcation »

red_trumpet wrote: post #11
adamnemecek wrote: post #3

I have recently written a paper on understanding machine learning via the lens of Hopf algebra https://arxiv.org/abs/2302.01834 . Hopf algebras (which are really just tensors with recurrence relations built in) subsume convnets, transformers and diffusion model and also provide a theoretically better autodiff that operates within single layers as opposed to across entire graphs. Furthermore, there is a correspondence…

Your paper didn't pass my smell test at all, tbh. For example the formula you write about "product" and "coproduct" in section 3 is literally identical (as "=" is symmetric). In section 4.2 you write "the product is the standard tensor product" with a formula that doesn't at all involve the map m: A \otimes A \to A. The formula you write is the induced product on A \otimes A, assuming that you already have a product…

adamnemecek has posted too many comments and is in cooldown phase, but he's asked me to post this comment: "It's the programmers equal sign. I think that the surrounding text provides a decent explanation what the deal is.

You are right, there's a missing sentence fragment, "standard tensor product that satisfies the property...".

Read the Diaconis paper. "

--- This isn't a sock puppet and I hope this isn't against site rules. I just wanted to try and help facilitate good discussion. I think trumpet brought up some interesting criticisms and felt Adam had a legitimate interest in responding ASAP.

Re: Category Theory ∩ Machine Learning

#19

Post by voxl »

donnowhy wrote: post #16

category theory is 'native 2-dimensional' math. i.e. category theory explains everything in terms of graphs, where a graph is made from two different sorts of 'entities', nodes and vertices i.e. categories and morphisms this being math, I wonder to which extent can category theory be re-expressed in terms of sets. perhaps a better question is if category theory can be re-expressed (or founded on) functions? lastly, I…

Set theory is fine, see the (Stack Project)[https://stacks.math.columbia.edu/browse] which develops a ton of modern Category Theory on ZFC (Zermelo-Fraenkel Set Theory with the Axiom of Choice) alone.

Alternative foundations of mathematics (Set Theory, Category Theory, Type Theory, and all their variations) can all mutually interpret the other by just postulating sufficiently large universes. You don't pick or advocate one based off its ability to encode mathematics, but instead based on its ability to express your intention and ideas.

Really its no different from programming language preference in my book.

Re: Category Theory ∩ Machine Learning

#20

Post by Koshkin »

donnowhy wrote: post #16

category theory is 'native 2-dimensional' math. i.e. category theory explains everything in terms of graphs, where a graph is made from two different sorts of 'entities', nodes and vertices i.e. categories and morphisms this being math, I wonder to which extent can category theory be re-expressed in terms of sets. perhaps a better question is if category theory can be re-expressed (or founded on) functions? lastly, I…

> i.e. categories and morphisms

Objects, not categories.

Post reply on HN