Berkeley
Wernstrom! (shakes fist)
World model on million-length video and language with RingAttention
11–20 of 62 posts
Re: World model on million-length video and language with RingAttention
#12It feels like Matei is everywhere, impressive!
Re: World model on million-length video and language with RingAttention
#13Because it might not be clear: … d)Fully open-sourced a family of 7B parameter models capable of processing long text documents (LWM-Text, LWM-Text-Chat) and videos (LWM, LWM-Chat) of over 1M tokens. https://huggingface.co/LargeWorldModel In terms of content, I am blown away yet again by the SoTA speeding on by as I try to catch up. Can someone with a more cynical eye point me to competitors or problems with this app…
Look at how RingAttention is implemented: it's blockwise attention distributed among many GPUs, in other words bruteforce parallelization. For inference they use TPUs v4-128, not running this at home any time soon.
What is this month's best choice to run at home?
Re: World model on million-length video and language with RingAttention
#14wow. talk about that show with the Lost guy as an eccentric billionaire? this is what he built as a surveillance system.
Person of interest, and it is actually a very well thought and down to earth show. Quite interesting to see many of the elements in the show coming to life with recent AI advancements.
exactly. back when i watched tv that was my favorite show.
Re: World model on million-length video and language with RingAttention
#15Figure 2 and 3 are incredible, and I hope they're true in real life scenarios.
Re: World model on million-length video and language with RingAttention
#16Earlier quoted context omitted.
Look at how RingAttention is implemented: it's blockwise attention distributed among many GPUs, in other words bruteforce parallelization. For inference they use TPUs v4-128, not running this at home any time soon.
What is this month's best choice to run at home?
ollama
Re: World model on million-length video and language with RingAttention
#17Some pretty fascinating collaborators:
- Matei Zaharia, a CTO of Databricks - Pieter Abbeel Director of the Berkeley Robot Learning Lab, Co-Director of the Berkeley Artificial Intelligence Research (BAIR) lab - Two talented PhD students: Hao Liu, Wilson Yan
Re: World model on million-length video and language with RingAttention
#18Because it might not be clear: … d)Fully open-sourced a family of 7B parameter models capable of processing long text documents (LWM-Text, LWM-Text-Chat) and videos (LWM, LWM-Chat) of over 1M tokens. https://huggingface.co/LargeWorldModel In terms of content, I am blown away yet again by the SoTA speeding on by as I try to catch up. Can someone with a more cynical eye point me to competitors or problems with this app…
Look at how RingAttention is implemented: it's blockwise attention distributed among many GPUs, in other words bruteforce parallelization. For inference they use TPUs v4-128, not running this at home any time soon.
Is there any information on a suggested inference setup? I guess they had something different in mind than TPU v4-128 when they put it on HuggingFace?
Re: World model on million-length video and language with RingAttention
#19Earlier quoted context omitted.
Look at how RingAttention is implemented: it's blockwise attention distributed among many GPUs, in other words bruteforce parallelization. For inference they use TPUs v4-128, not running this at home any time soon.
Is there any information on a suggested inference setup? I guess they had something different in mind than TPU v4-128 when they put it on HuggingFace?
looking at https://github.com/LargeWorldModel/LWM - they seem to indeed suggest to use a TPU vm
Re: World model on million-length video and language with RingAttention
#20I wonder why are the example videos this specific clip compilation format.
It feels to me that to navigate that, you essentially have to index 500 10-seconds videos, and that looks a lot easier than retrieving information that is in an actual 1 hour long video, because the later one will have a lot more of easy to mix-up moments. So maybe it hides an inability to answer questions about actual long videos (in the paper, the other example videos cap at 3 minutes length for what I can see).
On the other hand, maybe it's just for results presentation purposes, because it is much more readily "verifiable" for everyone than saying "trust us, in this very long video, there's the correct answer unarguably".
So if someone happens to more about that, I'd be very interested