Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hierarchical Modeling (H-Nets)

cartesia.ai

Post reply on HN
Print view View original

11–20 of 23 posts

Re: Hierarchical Modeling (H-Nets)

#11

Post by miven »

gdiamos wrote: post #7

How does it handle images?

As far as I understand the "chunking" of input bytes is learned completely end to end, so it's basically up to the model to figure out how to most efficiently delineate and aggregate the information from the inputs according to the patterns provided to it during training.

Since it's end to end this allows them to apply this process not only to raw byte encodings but basically representations of any level, such as stacking two stages of aggregation one after another.

So in principle they could either let the model do its thing on raw bytes of an image or alternatively maybe cut it up into tiny patches ViT-style and feed that to their H-Net.

I wonder how hard would it be to adapt chunking to work in 2D and what would that even look like.

Some other notes on how multimodal inputs could be handled using this architecture are mentioned in Albert Gu's (one of the author's) blog, although only briefly, there's still much to figure out it would seem: https://goombalab.github.io/blog/2025/hnet-future/#alternati...

Re: Hierarchical Modeling (H-Nets)

#12

Post by marviel »

miven wrote: post #11
gdiamos wrote: post #7

How does it handle images?

As far as I understand the "chunking" of input bytes is learned completely end to end, so it's basically up to the model to figure out how to most efficiently delineate and aggregate the information from the inputs according to the patterns provided to it during training. Since it's end to end this allows them to apply this process not only to raw byte encodings but basically representations of any level, such as sta…

Thanks for sharing this blog post is a great speculative deep-dive.

Re: Hierarchical Modeling (H-Nets)

#14

Post by lukebechtel »

cubefox wrote: post #13

As Mamba didn't make it, will H-Nets replace Transformers?

It's meant to replace the BPE tokenizer piece, so it isn't a full Language Model by itself.

In fact in Gu's blog post (linked in a post below) it's mentioned that they created a Mamba model that used this in place of the tokenizer.

Re: Hierarchical Modeling (H-Nets)

#15

Post by vannevar »

>The best AI architectures in use today treat all inputs equally.

Doesn't this architecture also treat all inputs equally? It seems like an encoder that preprocesses the input by inferring hierarchy. But don't all models essentially do that while training?

Re: Hierarchical Modeling (H-Nets)

#16

Post by modeless »

vannevar wrote: post #15

>The best AI architectures in use today treat all inputs equally. Doesn't this architecture also treat all inputs equally? It seems like an encoder that preprocesses the input by inferring hierarchy. But don't all models essentially do that while training?

If I understand correctly, each level of the hierarchy divides its input into chunks of variable size, but outputs a fixed amount for each chunk. The chunking is learned. The model can choose to compress data by making its input chunks bigger, depending on their content.

Re: Hierarchical Modeling (H-Nets)

#17

Post by blurbleblurble »

modeless wrote: post #3

I don't know if this is the one but something like this is clearly the future IMO. We need more levels of hierarchy to efficiently generalize to longer sequences with high level structure. Back when Byte Latent Transformers came out I thought extending the idea to more levels of hierarchy was the way to go, and this seems to be basically that? Another article about H-Nets: https://main-horse.github.io/posts/hnet-inf/

Yes... This seems like a generalization of "large concept models" in a certain way

Re: Hierarchical Modeling (H-Nets)

#18

Post by blurbleblurble »

Hand wavy idea: I wonder if we couldn't take this to another level and have some kind of general graph representation along with hierarchical reductions of it.

I sort of disagree with the assertion that "language is fundamentally hierarchical" in that it supposes there is a single abstraction hierarchy that's universally preferable or correct. That's just not true. It doesn't hurt anybody and it's definitely simpler to choose just one useful one (a hierarchy) but why learn only one? Why not learn multiple and also learn how to modulate between them?

Re: Hierarchical Modeling (H-Nets)

#20

Post by yorwba »

cubefox wrote: post #13

As Mamba didn't make it, will H-Nets replace Transformers?

Their architecture uses a mix of Transformer and Mamba layers. The question isn't whether it will replace Transformers, but whether it'll become part of the toolkit or whether it'll get abandoned like many other promising approaches.
Post reply on HN