Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fuyu-8B: A multimodal architecture for AI agents

adept.ai

Post reply on HN
Print view View original

51–60 of 60 posts

Re: Fuyu-8B: A multimodal architecture for AI agents

#51

Post by JimDabell »

mark_l_watson wrote: post #39

This looks so cool, and from reading the Hugging Face model card it should be easy enough to run. I do almost all of my work with text, NLP, IR, etc., and I have wanted to try multi-modal models. I just bookmarked the model card page. I am also getting even more excited by the explosion of work on open models. I still haven’t adjusted to how good mistral-7B is, and it runs on my Mac without breaking a sweat.

I gave it a shot on an M1 Max with 64GB RAM yesterday and it consumed all available RAM and hit a wall. I can run other, larger models without any problems so I assume it’s not an intrinsic limitation, but I didn’t spend any time debugging it.

Mistral-7B is incredible for its size!

Re: Fuyu-8B: A multimodal architecture for AI agents

#52

Post by 3abiton »

fpgaminer wrote: post #19

The architecture is quite compelling. I would not have expected it to work as well as it does. Glancing at the benchmarks it's basically on par with other VLMs in its class, despite having no separate image encoder. Is there an associated paper? Or more specifically, details on the training dataset? It must have been a mix of text and VLM tasks, otherwise one or the other capability would have rotted during training.…

This is the first multimodal model i hear about that is open source. Are there already other alternatives?

Re: Fuyu-8B: A multimodal architecture for AI agents

#53

Post by tellarin »

snats wrote: post #27
thefcpk wrote: post #25

One thing that puzzles me is the lack of multilingual models... it is a bit sad to see everything through the English language.

Yes, but currently there is a project called Aya[1] from Cohere4AI that I think it is trying to create multilingual models. [1] aya.for.ai

And the project is looking for contributors across many languages!

Full disclaimer: I'm a contributor and a big believer in the project.

Re: Fuyu-8B: A multimodal architecture for AI agents

#54

Post by outofpaper »

MattPalmer1086 wrote: post #48
startupsfail wrote: post #43

Earlier quoted context omitted.

If you take some copyrighted data, a set of books, for example. And count words in these books and then plot a distribution of top 100 word frequencies. The copyright for that new image would belong to you.

Copyright in the specific image sure, but not the graph itself. Someone else could do the same thing and make their own graph image.

Exactly. Data is not covered by American copyright and artifacts generated by LLM and diffusion tools are not covered by copyright protection unless there was human involvement and humans are transparent about how they participated in the creation of the artifacts.

Re: Fuyu-8B: A multimodal architecture for AI agents

#55

Post by coder543 »

3abiton wrote: post #52
fpgaminer wrote: post #19

The architecture is quite compelling. I would not have expected it to work as well as it does. Glancing at the benchmarks it's basically on par with other VLMs in its class, despite having no separate image encoder. Is there an associated paper? Or more specifically, details on the training dataset? It must have been a mix of text and VLM tasks, otherwise one or the other capability would have rotted during training.…

This is the first multimodal model i hear about that is open source. Are there already other alternatives?

The Fuyu pre-trained model is not open source. At best, it is source-available. It's also not the only multimodal model you can run locally.

A few other examples include LLaVA[0], IDEFICS[1][2], and CogVLM[3]. Mini-GPT[4] might be another one to look at. I'm pretty sure all of these have better licenses than Fuyu. Fuyu's architecture does sound really interesting, but the license on the pre-trained model is a complete non-starter for almost anything.

[0]: https://github.com/haotian-liu/LLaVA

[1]: https://huggingface.co/blog/idefics

[2]: https://huggingface.co/HuggingFaceM4/idefics-80b-instruct

[3]: https://github.com/THUDM/CogVLM

[4]: https://github.com/Vision-CAIR/MiniGPT-4

Re: Fuyu-8B: A multimodal architecture for AI agents

#56

Post by bluecrab »

coder543 wrote: post #55
3abiton wrote: post #52

Earlier quoted context omitted.

This is the first multimodal model i hear about that is open source. Are there already other alternatives?

The Fuyu pre-trained model is not open source. At best, it is source-available. It's also not the only multimodal model you can run locally. A few other examples include LLaVA[0], IDEFICS[1][2], and CogVLM[3]. Mini-GPT[4] might be another one to look at. I'm pretty sure all of these have better licenses than Fuyu. Fuyu's architecture does sound really interesting, but the license on the pre-trained model is a complet…

[dead]

Re: Fuyu-8B: A multimodal architecture for AI agents

#57

Post by startupsfail »

outofpaper wrote: post #54
MattPalmer1086 wrote: post #48

Earlier quoted context omitted.

Copyright in the specific image sure, but not the graph itself. Someone else could do the same thing and make their own graph image.

Exactly. Data is not covered by American copyright and artifacts generated by LLM and diffusion tools are not covered by copyright protection unless there was human involvement and humans are transparent about how they participated in the creation of the artifacts.

For now there is a lot of human involvement. You pretty much need a team of engineers or an equivalent to get anything besides minor fine tuning done. And there is usually human labor involved at labeling, feedback and evaluation stages.

Re: Fuyu-8B: A multimodal architecture for AI agents

#59

Post by outofpaper »

startupsfail wrote: post #57
outofpaper wrote: post #54

Earlier quoted context omitted.

Exactly. Data is not covered by American copyright and artifacts generated by LLM and diffusion tools are not covered by copyright protection unless there was human involvement and humans are transparent about how they participated in the creation of the artifacts.

For now there is a lot of human involvement. You pretty much need a team of engineers or an equivalent to get anything besides minor fine tuning done. And there is usually human labor involved at labeling, feedback and evaluation stages.

The issue circles back to their needing to be transparent about how they did the work.

When it comes to intellectual property there are two methods of protecting it: either you can keep it a trade secret and only use it in house (the secret sauce approach) or you keep things out in the open and seek copyright or patent or trademark protection. You can't have it both ways and even more so with AI co-created artifacts. If they are transparent about all the steps involved and what the humans did then they can seek protection for the human created parts. This also allows others to then replicate these steps and to create similar artifacts.

It sounds like they and many other "AI" teams want patent protection without having to register for it. These teams are trying to write their own licenses to rights they do not have.

Post reply on HN