Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fuyu-8B: A multimodal architecture for AI agents

adept.ai

Post reply on HN
Print view View original

11–20 of 60 posts

Re: Fuyu-8B: A multimodal architecture for AI agents

#11

Post by Q6T46nT668w6i3m »

tasdfqwer0897 wrote: post #2

Hey I work at Adept and helped make this! Happy to answer questions. The thing I think is especially neat/notable is how simple you can make the model architecture while still getting good performance. I expect we'll continue to see bits of these models get deleted in the next few years Note that you can get the model weights on HuggingFace here: https://huggingface.co/adept/fuyu-8b

Neat idea! Are the batches encoded as tokens into the input sequence? This is something I really like about the multi-modal PALM papers since it enables the multi-modal tokens to be referenced.

Re: Fuyu-8B: A multimodal architecture for AI agents

#12

Post by visarga »

tasdfqwer0897 wrote: post #2

Hey I work at Adept and helped make this! Happy to answer questions. The thing I think is especially neat/notable is how simple you can make the model architecture while still getting good performance. I expect we'll continue to see bits of these models get deleted in the next few years Note that you can get the model weights on HuggingFace here: https://huggingface.co/adept/fuyu-8b

Do you offer paid API access to larger models?

Re: Fuyu-8B: A multimodal architecture for AI agents

#13

Post by rafaelero »

thatcherc wrote: post #4

Really cool that the image patches are converted to tokens with just a linear projection instead of a big embedding model! I wonder if that trick will prove viable for other multimodel media like audio.

Not using embeddings/lookup table means they can't generate image/audio, which to me it's a severe limitation. Why bother going to the process of generating a multimodal transformer if it's able to generate nothing but text?

Re: Fuyu-8B: A multimodal architecture for AI agents

#14

Post by axiom92 »

GaggiX wrote: post #8
famouswaffles wrote: post #6

Oh wow. This seems to be the best released vlm model. The chart/UI understanding displayed in particular is superb.

>This is by far the best open source vlm model LLaVA 1.5 is very good, at least at describing images. http://llava.hliu.cc/

Right, but no separate image encoder + half the size could be very helpful for many applications.

Re: Fuyu-8B: A multimodal architecture for AI agents

#15

Post by leodriesch »

rafaelero wrote: post #13
thatcherc wrote: post #4

Really cool that the image patches are converted to tokens with just a linear projection instead of a big embedding model! I wonder if that trick will prove viable for other multimodel media like audio.

Not using embeddings/lookup table means they can't generate image/audio, which to me it's a severe limitation. Why bother going to the process of generating a multimodal transformer if it's able to generate nothing but text?

For an AI agent that should navigate a computer (which is Adepts use case IIRC) it should work, as it only has to output commands.

Re: Fuyu-8B: A multimodal architecture for AI agents

#16

Post by mandelken »

JimDabell wrote: post #9
tasdfqwer0897 wrote: post #2

Hey I work at Adept and helped make this! Happy to answer questions. The thing I think is especially neat/notable is how simple you can make the model architecture while still getting good performance. I expect we'll continue to see bits of these models get deleted in the next few years Note that you can get the model weights on HuggingFace here: https://huggingface.co/adept/fuyu-8b

What’s the situation with the license? Your blog post says you are open sourcing it, but it’s currently only available under a non-commercial license instead. Is an open source release forthcoming?

You can download the weights on Hugginface.

I believe the copyright on AI model weights in the US is not fully established, but so far it has been held that a list of numbers can not be copyrighted, so likely the same applies to model weights. Note that you don't have to enter into an agreement with Adept to use the model.

Alternatively, use and download the weights in Japan that has explicitly no copyright on AI models.

Re: Fuyu-8B: A multimodal architecture for AI agents

#17

Post by Philpax »

rafaelero wrote: post #13
thatcherc wrote: post #4

Really cool that the image patches are converted to tokens with just a linear projection instead of a big embedding model! I wonder if that trick will prove viable for other multimodel media like audio.

Not using embeddings/lookup table means they can't generate image/audio, which to me it's a severe limitation. Why bother going to the process of generating a multimodal transformer if it's able to generate nothing but text?

Many applications only need input, not output.

Re: Fuyu-8B: A multimodal architecture for AI agents

#19

Post by fpgaminer »

The architecture is quite compelling. I would not have expected it to work as well as it does. Glancing at the benchmarks it's basically on par with other VLMs in its class, despite having no separate image encoder.

Is there an associated paper? Or more specifically, details on the training dataset? It must have been a mix of text and VLM tasks, otherwise one or the other capability would have rotted during training. But I wonder if they trained off strictly VLM corpora, or also used plain image-text datasets like CLIP. It would be interesting if only the former.

Also makes me wonder if it could be trained on something like CommonCrawl where all the images are retained and interspersed correctly throughout the text. This model could theoretically train just fine off that, and it would unlock a whole new dataset effectively.

And has there been an inspection of what the model is outputting for predicted image "tokens"? Is it correctly predicting projected image patches to any degree of accuracy? And could therefore also generate images inline with text if another de-projection layer was trained?

Re: Fuyu-8B: A multimodal architecture for AI agents

#20

Post by GaggiX »

axiom92 wrote: post #14
GaggiX wrote: post #8

Earlier quoted context omitted.

>This is by far the best open source vlm model LLaVA 1.5 is very good, at least at describing images. http://llava.hliu.cc/

Right, but no separate image encoder + half the size could be very helpful for many applications.

The 7B LLaVa model is smaller, even considering the image encoder (CLIP-L).
Post reply on HN