PaLI-3 Vision Language Models
arxiv.org
PaLI-3 Vision Language Models
1–10 of 29 posts
Re: PaLI-3 Vision Language Models
#2maybe someone more informed can help me understand why they didn't compared to Llava (https://llava-vl.github.io/)?
Re: PaLI-3 Vision Language Models
#3Something that stood out to me skimming the paper - that was somewhat buried - they finetune the model on each benchmark.
"Finally, for each individual task (benchmark), we fine-tune the PaLI-3 model with frozen ViT image encoder on the task’s training data as described in the cor- responding section. For most tasks, we fine-tune the 812×812 resolution checkpoint, but for two document understanding tasks, we go up to 1064×1064 resolution"
S̶o̶ ̶t̶h̶i̶s̶ ̶i̶s̶ ̶c̶o̶m̶p̶a̶r̶i̶n̶g̶ ̶a̶ ̶s̶m̶a̶l̶l̶e̶r̶ ̶m̶o̶d̶e̶l̶ ̶f̶i̶n̶e̶t̶u̶n̶e̶d̶ ̶p̶e̶r̶ ̶b̶e̶n̶c̶h̶m̶a̶r̶k̶ ̶t̶o̶ ̶l̶a̶r̶g̶e̶r̶ ̶m̶o̶d̶e̶l̶s̶ ̶t̶h̶a̶t̶ ̶I̶ ̶p̶r̶e̶s̶u̶m̶e̶ ̶a̶r̶e̶ ̶n̶o̶t̶,̶ ̶t̶h̶o̶u̶g̶h̶ ̶I̶ ̶h̶a̶v̶e̶ ̶n̶o̶t̶ ̶r̶e̶a̶d̶ ̶t̶h̶e̶ ̶P̶a̶l̶i̶-̶X̶ ̶p̶a̶p̶e̶r̶.̶
Edit - No, I was wrong, Palm-X is also fine-tuned before each task/set of tasks.
Impressive improvement!!!
Re: PaLI-3 Vision Language Models
#4maybe someone more informed can help me understand why they didn't compared to Llava ( https://llava-vl.github.io/ )?
Maybe they just didn’t know about llava while conducting their research. It can take days to train a model sometimes.
Re: PaLI-3 Vision Language Models
#5[deleted]
Re: PaLI-3 Vision Language Models
#6Re: PaLI-3 Vision Language Models
#7maybe someone more informed can help me understand why they didn't compared to Llava ( https://llava-vl.github.io/ )?
The purpose of this research is to compare large vision-language models where the vision component is pre-trained using different techniques, namely on image classification versus unsupervised contrastive pre-training (see OpenAI's CLIP). PaLI-3 also isn't an instruction-tuned model, so comparing it to Llava would be a little apples-to-oranges.
Re: PaLI-3 Vision Language Models
#8I probably could have guessed that contrastive pre-training works better for downstream vision-language tasks than image classification pre-training, but it's nice to see this hypothesis thoroughly tested here.
Also, hacks to get LLMs to generate structured output seem to be mostly getting the job done. I'm less optimistic about this approach for traditional vision tasks where language is the interface, however. Are we going to get models to output a pixel-wise segmentation mask as text? I want to doubt it but seeing how LLMs are about to output long sequences of structured text leaves my mind open.
Re: PaLI-3 Vision Language Models
#9[dead]
Re: PaLI-3 Vision Language Models
#10I probably could have guessed that contrastive pre-training works better for downstream vision-language tasks than image classification pre-training, but it's nice to see this hypothesis thoroughly tested here. Also, hacks to get LLMs to generate structured output seem to be mostly getting the job done. I'm less optimistic about this approach for traditional vision tasks where language is the interface, however. Are…
I don’t see why not- “segment anything” from meta seems to handle labeled pixel-wise segmentation masks fairly well. You can also get rough masks today by looking at where the text part of the model attends to in the image part.