This looks great! Is there any software that supports these? Llama.cpp, Ollama, LM studio, etc are really convenient, but I don't think they have image support yet?
Fuyu-8B: A multimodal architecture for AI agents
31–40 of 60 posts
Re: Fuyu-8B: A multimodal architecture for AI agents
#32Hey I work at Adept and helped make this! Happy to answer questions. The thing I think is especially neat/notable is how simple you can make the model architecture while still getting good performance. I expect we'll continue to see bits of these models get deleted in the next few years Note that you can get the model weights on HuggingFace here: https://huggingface.co/adept/fuyu-8b
What’s the situation with the license? Your blog post says you are open sourcing it, but it’s currently only available under a non-commercial license instead. Is an open source release forthcoming?
Yeah... in the blog post, they do explicitly mention "cc-by-nc", which I find disappointing.
Anything that Adept is "excited to see what the community builds on top of it" would only serve Adept and no one else! What incentive does the community have to build on top of Fuyu, when the community can't benefit from its own work? If Adept wants to benefit from word-of-mouth discussion of their models and from community contributions that make those models work better, as has happened dramatically with Llama 2, then they need to give the community the opportunity to benefit too.
Also weird: if you look at the tags on Hugging Face, you'll see it is listed as "cc". This comes from the README[0] metadata. "cc" is not really a license.
[0]: https://huggingface.co/adept/fuyu-8b/blob/main/README.md?cod...
Re: Fuyu-8B: A multimodal architecture for AI agents
#33Hey I work at Adept and helped make this! Happy to answer questions. The thing I think is especially neat/notable is how simple you can make the model architecture while still getting good performance. I expect we'll continue to see bits of these models get deleted in the next few years Note that you can get the model weights on HuggingFace here: https://huggingface.co/adept/fuyu-8b
Thank you for the release!
What can you tell us about this:
> Our internal models (based on Fuyu) have extra capabilities related to our product. In particular,
> 1. They can reliably perform OCR on high-resolution images
> 2. They can do fine-grained localization of text and UI elements within those images
> 3. They can answer questions about images of UIs
Is this just a matter of additional fine tuning, or are there architectural differences?
Re: Fuyu-8B: A multimodal architecture for AI agents
#34Thank you to the amazing team at Adept.ai for making this available!
For anyone interested in contributing to a fully open source alternative, join us at https://github.com/OpenAdaptAI/OpenAdapt
Lots of interesting work to be done, including integrating with Fuyu-8B!
Re: Fuyu-8B: A multimodal architecture for AI agents
#35Why don‘t these benchmarks judge the likelihood of the example answer? Just taking the MAP predictions seems like a waste of information
Re: Fuyu-8B: A multimodal architecture for AI agents
#36Earlier quoted context omitted.
> a list of numbers can not be copyrighted Any digital object can be represented as a list of numbers (this is precisely the origin of the term digit al). Since there is clearly precedent for copyrighted digital objects (media, software, etc), reducing something to "a list of numbers" is not a useful distinction in regard to copyright law.
IANAL but as far as I remember, you can't copyright a list of objective facts, for example a phone book containing a list of phone numbers. Model weights are clearly not in that category. Happy to be corrected if I misremember.
Model weight are akin to markov chains and compressed data. They are direct representations of the data they where created from in the same way that markov chains are created from hidden markov chains and Zipped files are created from files.
Zipping a file does not grant the copyright protection of the zipped output beyond the copyright of the original file.
Moreover the American federal registrar has officially stated that AI generated artifacts are not eligible for copyright https://www.federalregister.gov/documents/2023/03/16/2023-05....
Re: Fuyu-8B: A multimodal architecture for AI agents
#37Thank you to the amazing team at Adept.ai for making this available! For anyone interested in contributing to a fully open source alternative, join us at https://github.com/OpenAdaptAI/OpenAdapt Lots of interesting work to be done, including integrating with Fuyu-8B!
"fully open source", but there is no license?
https://github.com/OpenAdaptAI/OpenAdapt/blob/30581e47fa9aec...
https://github.com/OpenAdaptAI/OpenAdapt/issues/246
And Fuyu is under a non-commercial license, so there's not much to be done with it unless someone trains a new Fuyu-architecture model from scratch.
Re: Fuyu-8B: A multimodal architecture for AI agents
#38Fascinating! I love seeing more multimodal ML. Thanks for sharing!
Re: Fuyu-8B: A multimodal architecture for AI agents
#39This looks so cool, and from reading the Hugging Face model card it should be easy enough to run. I do almost all of my work with text, NLP, IR, etc., and I have wanted to try multi-modal models. I just bookmarked the model card page.
I am also getting even more excited by the explosion of work on open models. I still haven’t adjusted to how good mistral-7B is, and it runs on my Mac without breaking a sweat.
Re: Fuyu-8B: A multimodal architecture for AI agents
#40Thank you to the amazing team at Adept.ai for making this available! For anyone interested in contributing to a fully open source alternative, join us at https://github.com/OpenAdaptAI/OpenAdapt Lots of interesting work to be done, including integrating with Fuyu-8B!
"fully open source", but there is no license? https://github.com/OpenAdaptAI/OpenAdapt/blob/30581e47fa9aec... https://github.com/OpenAdaptAI/OpenAdapt/issues/246 And Fuyu is under a non-commercial license, so there's not much to be done with it unless someone trains a new Fuyu-architecture model from scratch.
Thank you for pointing this out! You are correct that we have not yet decided on a license.
I will admit my ignorance on this topic, and I didn't want us to rush into selecting a license that is inappropriate.
Which one should we choose?