> Scalable Perceptual Generative Model
If you wanna call it Bagel, just call it Bagel. No need to make up a justification.
Bagel: Open-source unified multimodal model
11–20 of 35 posts
Re: Bagel: Open-source unified multimodal model
#12Earlier quoted context omitted.
If the model uses FP16 precision and has 7 billion active parameters, it would require approximately 14 GB of VRAM. I didn't read the paper.
How can you calculate required VRAM from precision and parameter number?
Rule of thumb is parameter_count * precision. Precision can be anything [32,16,8,4] bits. 32bits is sometimes used in training (although less now I guess), and rarely in inference. For a while now "full" precision is 16bit (fp16, bf16), fp8 is 8bit, int4 is 4bit, and so on. Everything that's not "full" precision is also known as quantised. fp8 is a quantised version of the "full" model.
So quick napkin math can give you the VRAM usage for loading the model. 7b can be ~14GB full, 7GB in fp8 and ~3.5GB in 4bit (AWQ, int4, q4_k_m, etc). But that's just to load the model in VRAM. You also need some available VRAM to run inference, and there are a lot of things to consider there too. You need to be able to run a forward pass on the required context, you can keep a kv cache to speed up inference, you can do multiple sessions in parallel, and so on.
Context length is important to take into account because images take a lot of tokens. So what you could do with a 7b LLM at full precision on a 16GB VRAM GPU might not be possible with a VLM, because the context of your query might not fit into the remaining 2GB.
Re: Bagel: Open-source unified multimodal model
#13Earlier quoted context omitted.
If the model uses FP16 precision and has 7 billion active parameters, it would require approximately 14 GB of VRAM. I didn't read the paper.
How can you calculate required VRAM from precision and parameter number?
Realistically you probably just want to look at the file size on huggingface and add ~2 gigs for OS/Firefox tabs and and a bit for context (depends but lets say 1-2)
The direct parm conversion math tends to be much less reliable than one would expect once quants are involved.
e.g.
7B @ Q8 = 7.1gb [0]
30B @ Q8 = 34.6gb [1]
btw you can also roughly estimate expected output speed too if you know the device memory throughput. Noting that this doesn't work for MoEs
Also recently discovered that in CPU mode llama.cpp does memory mapping. For some models it loads less than a quarter into memory.
Re: Bagel: Open-source unified multimodal model
#14These days, papers come with an advertisement video
As someone who used to be in the academia, I think is isn't bad in itself , I just worry that by comparison it raises the burden of effort that one has to make in order to get their work noticed.
Compared to the effort required to play in that field at all, making a video is almost negligible.
Re: Bagel: Open-source unified multimodal model
#15I'm interested in potential alternatives to ChatGPT's advanced voice mode. When I see the word "multimodal" I'm hopeful the model understands text + voice but instead it almost always seems to refer to text + images. Is there a keyword that I can use to look for models that work with voice similar to ChatGPT's advanced voice mode?
Re: Bagel: Open-source unified multimodal model
#16Oh no it's The Everything Bagel.
Re: Bagel: Open-source unified multimodal model
#17bagel
Re: Bagel: Open-source unified multimodal model
#18Is it from ByteDance Team, right? The team behind TikTok, CapCut, BuzzVideo and more.
Any thoughts on that?
Re: Bagel: Open-source unified multimodal model
#19A quick test in the "demo" link doesn't show it to be "as smart" as it appeared in the demos on the page. I really hope it does all it's promising to do, but I'm skeptic so far.
Re: Bagel: Open-source unified multimodal model
#20I'm interested in potential alternatives to ChatGPT's advanced voice mode. When I see the word "multimodal" I'm hopeful the model understands text + voice but instead it almost always seems to refer to text + images. Is there a keyword that I can use to look for models that work with voice similar to ChatGPT's advanced voice mode?
Google Gemini Live is pretty good.
If you want to try only voice, Try unmute.sh by Kyutai which will be eventually open-sourced