Hierarchical Modeling (H-Nets)
cartesia.ai
Hierarchical Modeling (H-Nets)
1–10 of 23 posts
Re: Hierarchical Modeling (H-Nets)
#2> H-Net demonstrates three important results on language modeling:
> 1. H-Nets scale better with data than state-of-the-art Transformers with BPE tokenization, while learning directly from raw bytes. This improved scaling is even more pronounced on domains without natural tokenization boundaries, like Chinese, code, and DNA.
> 2. H-Nets can be stacked together to learn from deeper hierarchies, which further improves performance.
> 3. H-Nets are significantly more robust to small perturbations in input data like casing, showing an avenue for creating models that are more robust and aligned with human reasoning.
Re: Hierarchical Modeling (H-Nets)
#3I don't know if this is the one but something like this is clearly the future IMO. We need more levels of hierarchy to efficiently generalize to longer sequences with high level structure. Back when Byte Latent Transformers came out I thought extending the idea to more levels of hierarchy was the way to go, and this seems to be basically that?
Another article about H-Nets: https://main-horse.github.io/posts/hnet-inf/
Re: Hierarchical Modeling (H-Nets)
#4> H-Net demonstrates three important results on language modeling: > 1. H-Nets scale better with data than state-of-the-art Transformers with BPE tokenization, while learning directly from raw bytes. This improved scaling is even more pronounced on domains without natural tokenization boundaries, like Chinese, code, and DNA. > 2. H-Nets can be stacked together to learn from deeper hierarchies, which further improves…
Re: Hierarchical Modeling (H-Nets)
#5I've only skimmed the paper, but it looks interesting and credible, so I've added it to my reading list.
Thank you for sharing on HN!
---
EDIT: The hierarchical composition and routing aspects of this work vaguely remind me of https://github.com/glassroom/heinsen_routing/ but it has been a while since I played with that. UPDATE: After spending a bit more time on the OP, it's different, but the ideas are related, like routing based on similarity.
Re: Hierarchical Modeling (H-Nets)
#6I've only skimmed the paper, but it looks interesting and credible, so I've added it to my reading list. Thank you for sharing on HN! --- EDIT: The hierarchical composition and routing aspects of this work vaguely remind me of https://github.com/glassroom/heinsen_routing/ but it has been a while since I played with that. UPDATE: After spending a bit more time on the OP, it's different, but the ideas are related, like…
No problem! I'm still parsing it myself, but it seems promising in theory, and the result curves are impressive.
Re: Hierarchical Modeling (H-Nets)
#7How does it handle images?
Re: Hierarchical Modeling (H-Nets)
#8How does it handle images?
it mentions native multimodality somewhere in either the Arxiv or post -- seems like it might handle it well?
Re: Hierarchical Modeling (H-Nets)
#9Seems likely to be relevant for memory formation/consolidation/management.
Big, if so.
Re: Hierarchical Modeling (H-Nets)
#10[deleted]