Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PaLI-3 Vision Language Models

arxiv.org

Post reply on HN
Print view View original

21–29 of 29 posts

Re: PaLI-3 Vision Language Models

#21

Post by staticman2 »

light_hue_1 wrote: post #18

No comparison against GPT-4V? How embarrassing! Where are they going to submit this? A conference where no one knows about GPT-4V? Ridiculous. It's getting really awkward seeing these papers from Google. "We're here too! We're totally not woefully behind everyone else in the field!". No model, no reasonable comparisons, just generic bragging. I'm astounded as an ML researcher how Google can be doing so incredibly bad…

I just tried translating a camera image of a japanese manga page with Chatgpt vs Bard, and Bard greatly outperformed ChatGPT in recognizing the japanese kanji, for what it's worth.

Re: PaLI-3 Vision Language Models

#22

Post by Geee »

light_hue_1 wrote: post #18

No comparison against GPT-4V? How embarrassing! Where are they going to submit this? A conference where no one knows about GPT-4V? Ridiculous. It's getting really awkward seeing these papers from Google. "We're here too! We're totally not woefully behind everyone else in the field!". No model, no reasonable comparisons, just generic bragging. I'm astounded as an ML researcher how Google can be doing so incredibly bad…

Fundamentally the reason is that they can't make money with it, and AI eats into their search revenue. Search as a business is dead, and AI can't bring in the money.

Re: PaLI-3 Vision Language Models

#24

Post by nerdponx »

buildbot wrote: post #3

Something that stood out to me skimming the paper - that was somewhat buried - they finetune the model on each benchmark. "Finally, for each individual task (benchmark), we fine-tune the PaLI-3 model with frozen ViT image encoder on the task’s training data as described in the cor- responding section. For most tasks, we fine-tune the 812×812 resolution checkpoint, but for two document understanding tasks, we go up to…

Are they just fine-tuning part of the model on the "unsupervised" portion of the training data? I think that's not entirely unfair because it might be realistic. If you have a big corpus of data and a pre-existing model, you might want to fine tune the latter using the former. However it's certainly a generous benchmark and doesn't reflect real-world "online" usage.

Re: PaLI-3 Vision Language Models

#25

Post by biomattr »

light_hue_1 wrote: post #18

No comparison against GPT-4V? How embarrassing! Where are they going to submit this? A conference where no one knows about GPT-4V? Ridiculous. It's getting really awkward seeing these papers from Google. "We're here too! We're totally not woefully behind everyone else in the field!". No model, no reasonable comparisons, just generic bragging. I'm astounded as an ML researcher how Google can be doing so incredibly bad…

Maybe its a conscious choice not to be the most frontier model? Cause its hard to believe they're not capable of being the hare in this race.

Re: PaLI-3 Vision Language Models

#26

Post by kolja005 »

sgd99 wrote: post #23

can anyone explain how these visual tokens which are concatenated with the tokenizer outputs for the encoder are created?

I was a little confused about this too. The authors say in the paper:

"The outputs of the ViT image encoder before pooling form the visual tokens, which are linearly projected and prepended to the embedded input text tokens."

I took a look at the HuggingFace implementation of ViT [1]. After the ViT encoder blocks there's a layer norm and then a pooling layer (line 595), where the pooling layer involves taking the first token output from the layer norm and running it through a dense layer. So, it looks like in PaLI-3 the tokens are the hidden states output by the layer norm after the ViT encoder blocks.

[1] https://github.com/huggingface/transformers/blob/main/src/tr...

Re: PaLI-3 Vision Language Models

#27

Post by buildbot »

atorodius wrote: post #17
buildbot wrote: post #3

Something that stood out to me skimming the paper - that was somewhat buried - they finetune the model on each benchmark. "Finally, for each individual task (benchmark), we fine-tune the PaLI-3 model with frozen ViT image encoder on the task’s training data as described in the cor- responding section. For most tasks, we fine-tune the 812×812 resolution checkpoint, but for two document understanding tasks, we go up to…

That's normal for ML

To finetune on each benchmark? I'd say it's not in our modern era of in-context learning, though of course fine-tuning has it's place as well for making smaller models better in one domain than a generalist larger model.

Re: PaLI-3 Vision Language Models

#28

Post by standardly »

light_hue_1 wrote: post #18

No comparison against GPT-4V? How embarrassing! Where are they going to submit this? A conference where no one knows about GPT-4V? Ridiculous. It's getting really awkward seeing these papers from Google. "We're here too! We're totally not woefully behind everyone else in the field!". No model, no reasonable comparisons, just generic bragging. I'm astounded as an ML researcher how Google can be doing so incredibly bad…

I imagine their AI model isn't off the ground becuase it hasn't integrated with ads yet. They ruined search and youtube for more ad impressions, and likely have the same strategy with AI.

Re: PaLI-3 Vision Language Models

#29

Post by sgd99 »

kolja005 wrote: post #26
sgd99 wrote: post #23

can anyone explain how these visual tokens which are concatenated with the tokenizer outputs for the encoder are created?

I was a little confused about this too. The authors say in the paper: "The outputs of the ViT image encoder before pooling form the visual tokens, which are linearly projected and prepended to the embedded input text tokens." I took a look at the HuggingFace implementation of ViT [1]. After the ViT encoder blocks there's a layer norm and then a pooling layer (line 595), where the pooling layer involves taking the fir…

thank you!
Post reply on HN