Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

World model on million-length video and language with RingAttention

largeworldmodel.github.io

Post reply on HN
Print view View original

31–40 of 62 posts

Re: World model on million-length video and language with RingAttention

#31

Post by brucethemoose2 »

dsrtslnd23 wrote: post #18
benob wrote: post #9

Earlier quoted context omitted.

Look at how RingAttention is implemented: it's blockwise attention distributed among many GPUs, in other words bruteforce parallelization. For inference they use TPUs v4-128, not running this at home any time soon.

Is there any information on a suggested inference setup? I guess they had something different in mind than TPU v4-128 when they put it on HuggingFace?

Its llama 7B, so anything that runs that.

You can quantize the cache and fit quite a bit on GPUs. At least 75k on my mere 24GB 3090, maybe 200K with a fancy quantization repo.

Re: World model on million-length video and language with RingAttention

#32

Post by brucethemoose2 »

bbor wrote: post #7

Because it might not be clear: … d)Fully open-sourced a family of 7B parameter models capable of processing long text documents (LWM-Text, LWM-Text-Chat) and videos (LWM, LWM-Chat) of over 1M tokens. https://huggingface.co/LargeWorldModel In terms of content, I am blown away yet again by the SoTA speeding on by as I try to catch up. Can someone with a more cynical eye point me to competitors or problems with this app…

Its more of a tech demo since its llama 7B (a model that is, TBH, obsoleted by mistral 7B), and its dataset is not that great.

We've had Yi 6B 200K context for some time, which is also quite good.

The problem, of course, is hardware requirements and vram. This one is particularly hairy since its not a GQA model.

Re: World model on million-length video and language with RingAttention

#33

Post by brucethemoose2 »

We've been testing it in the local llm Discords, turns out its just a llama 7B finetune that can run on any old GPU (which is cool).

https://huggingface.co/brucethemoose/LargeWorldModel_LWM-Tex...

https://huggingface.co/dranger003/LWM-Text-Chat-128K-iMat.GG...

And its long context recall is quite good! We've already kind of discovered this with Yi, but there are some things one can do with a mega context that you just can't get with RAG.

Re: World model on million-length video and language with RingAttention

#34

Post by simcop2387 »

brucethemoose2 wrote: post #33

We've been testing it in the local llm Discords, turns out its just a llama 7B finetune that can run on any old GPU (which is cool). https://huggingface.co/brucethemoose/LargeWorldModel_LWM-Tex... https://huggingface.co/dranger003/LWM-Text-Chat-128K-iMat.GG... And its long context recall is quite good! We've already kind of discovered this with Yi, but there are some things one can do with a mega context that you jus…

> And its long context recall is quite good! We've already kind of discovered this with Yi, but there are some things one can do with a mega context that you just can't get with RAG.

I've got to imagine that a mega-context like this can help RAG work in ways that just isn't possible otherwise. i.e. bring in many more search results or surrounding context around the results so that the processing can do much more.

Re: World model on million-length video and language with RingAttention

#35

Post by catchnear4321 »

pk-protect-ai wrote: post #30
catchnear4321 wrote: post #24

Earlier quoted context omitted.

the sentence itself should be alarming. the claim is that a dataset was created. of words and of videos, and that it was created from public datasets of books and videos, those datasets containing books, and videos. it takes too many words to say almost nothing. nothing to see here. if that isn’t the intent, then the authors need to do better.

The information is in the model card though: Books3 dataset 700B text-image pairs from Laion-2B-en, filtered to only keep images with at least 256 resolution 400M text-image pairs from COYO-700M, filtered to only keep images with at least 256 resolution 10M text-video pairs from WebVid10M 3M text-video pairs from a subset of InternVid10M 73K text-video chat pairs from Valley-Instruct-73K 100K text-video chat pairs fr…

wouldn’t that have been a cleaner explanation than the sentence provided? books and videos, see model card. the redundant language is a smell whether the emitter wishes to acknowledge or not. the point still stands, the hot mess of a sentence didn’t need to be that way.

> …so petty and pedantic…

if nothing else, think of the language models that need to digest this. sure you can send in gobbledygook and get out plausibly sense, but why?

llms will push pedantry to the forefront. or suffer from it. who knows. have fun.

Re: World model on million-length video and language with RingAttention

#36

Post by PeterHolzwarth »

colechristensen wrote: post #10
kromem wrote: post #8

It's pretty wild watching technology develop where I genuinely don't have a confident idea of just how far it will progress by December in February of the same year. Open models have just been on fire lately, and the next generation of SotA models to pull synthetic data from in training the next generation of open models each taking nuanced and clever approaches to infrastructure improvements has me pretty much consi…

Physics probably felt similar a hundred years ago.

It gives one an appreciation for 1940s, 1950s film, where scientists were somewhat idolized as being on the forefront of a massive new future - the world these pop culture expressions came from was alive with just non-stop growth in understanding of math, medicine, physics, and on down the technology list. Had to have been heady days to be alive and even slightly interested in technology and the tools it uses.

Re: World model on million-length video and language with RingAttention

#37

Post by azinman2 »

knightoffaith wrote: post #27

Amazing that you can just shove a ton of multimodal data into a big transformer and get a really good multimodal model. I wonder where things will top out. For many years a lot of people (including me) were saying "you can't just take existing architectures, scale them up, feed them a lot of data, and expect something mpressive", but here we are.

Why not? This was the conclusion of “The unreasonable effectiveness of data”: https://static.googleusercontent.com/media/research.google.c...

Re: World model on million-length video and language with RingAttention

#38

Post by okwhateverdude »

DougMerritt wrote: post #13
benob wrote: post #9

Earlier quoted context omitted.

Look at how RingAttention is implemented: it's blockwise attention distributed among many GPUs, in other words bruteforce parallelization. For inference they use TPUs v4-128, not running this at home any time soon.

What is this month's best choice to run at home?

If you pull the llama.cpp repo and use their convert/quantize tools on the pytorch version of the models uploaded to huggingface, they will load just fine into ollama:

https://old.reddit.com/r/LocalLLaMA/comments/18av9aw/quick_s...

https://github.com/ggerganov/llama.cpp/discussions/2948

You can run ollama (and a web UI) pretty trivially via docker:

docker run -d --gpus=all -v /some/dir/for/ollama/data:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:latest

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway --name ollama-webui ghcr.io/ollama-webui/ollama-webui:main

That particular webui will let you upload models (with configuration). Other wise, you can use the api directly (you'll need to POST a `blob` first):

https://github.com/ollama/ollama/blob/main/docs/api.md#creat...

Post reply on HN