Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

World model on million-length video and language with RingAttention

largeworldmodel.github.io

Post reply on HN
Print view View original

1–10 of 62 posts

Re: World model on million-length video and language with RingAttention

#4

Post by gchamonlive »

cranberryturkey wrote: post #2

wow. talk about that show with the Lost guy as an eccentric billionaire? this is what he built as a surveillance system.

Person of interest, and it is actually a very well thought and down to earth show. Quite interesting to see many of the elements in the show coming to life with recent AI advancements.

Re: World model on million-length video and language with RingAttention

#7

Post by bbor »

Because it might not be clear:

  … d)Fully open-sourced a family of 7B parameter models capable of processing long text documents (LWM-Text, LWM-Text-Chat) and videos (LWM, LWM-Chat) of over 1M tokens.
https://huggingface.co/LargeWorldModel

In terms of content, I am blown away yet again by the SoTA speeding on by as I try to catch up. Can someone with a more cynical eye point me to competitors or problems with this approach? Because as it stands… that jump to a context length of a million tokens is pretty impressive to an outsider.

Re: World model on million-length video and language with RingAttention

#8

Post by kromem »

It's pretty wild watching technology develop where I genuinely don't have a confident idea of just how far it will progress by December in February of the same year.

Open models have just been on fire lately, and the next generation of SotA models to pull synthetic data from in training the next generation of open models each taking nuanced and clever approaches to infrastructure improvements has me pretty much considering all bets to be off.

At this point, the bottleneck is increasingly the human ability to adapt to improving tools than limitations in the tools themselves.

Re: World model on million-length video and language with RingAttention

#9

Post by benob »

bbor wrote: post #7

Because it might not be clear: … d)Fully open-sourced a family of 7B parameter models capable of processing long text documents (LWM-Text, LWM-Text-Chat) and videos (LWM, LWM-Chat) of over 1M tokens. https://huggingface.co/LargeWorldModel In terms of content, I am blown away yet again by the SoTA speeding on by as I try to catch up. Can someone with a more cynical eye point me to competitors or problems with this app…

Look at how RingAttention is implemented: it's blockwise attention distributed among many GPUs, in other words bruteforce parallelization. For inference they use TPUs v4-128, not running this at home any time soon.

Re: World model on million-length video and language with RingAttention

#10

Post by colechristensen »

kromem wrote: post #8

It's pretty wild watching technology develop where I genuinely don't have a confident idea of just how far it will progress by December in February of the same year. Open models have just been on fire lately, and the next generation of SotA models to pull synthetic data from in training the next generation of open models each taking nuanced and clever approaches to infrastructure improvements has me pretty much consi…

Physics probably felt similar a hundred years ago.
Post reply on HN