Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MaMMUT: A simple vision-encoder text-decoder architecture for multimodal tasks

ai.googleblog.com

Post reply on HN
Print view View original

31–36 of 36 posts

Re: MaMMUT: A simple vision-encoder text-decoder architecture for multimodal tasks

#31

Post by AdoHaha »

reaperman wrote: post #14
fpgaminer wrote: post #7

Earlier quoted context omitted.

A research paper by itself isn't worth nothing, sure, but without the ability to reproduce the paper or even check their results it's ... not worth much.

The architecture is described enough to re-implement it and train it on known datasets/benchmarks such as VQA2. A single man with a medical degree named Phil Wang ('lucidrains')[0] has the ability to reproduce most of these papers by himself . He has 246 GitHub repos[1], most of which appear to be reproductions of models which are only described in papers that had no associated code or models released, such as [2]. O…

He indeed re-implemented the MaMMUT: https://github.com/lucidrains/MaMMUT-pytorch

Re: MaMMUT: A simple vision-encoder text-decoder architecture for multimodal tasks

#33

Post by sitkack »

m00x wrote: post #30
sitkack wrote: post #27

Earlier quoted context omitted.

Given the exposure hn has had to blip-2 it might be good to provide more context. https://arxiv.org/abs/2301.12597 https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... Were you expecting a response?

I'm not sure what you mean. HN has a wide array of people that have multiple sources of information. I was throwing a question out there hoping an expert in the matter would know. They're both multi-modal models for zero-shot answering. I'm not sure what the purpose of this comment is. If no experts see this, I won't get an answer. You don't need to reply with something completely useless that adds nothing to the top…

I am recommending that you add more context to your question since it has been posted a couple times but has had zero engagement.

HN is not an AIML discussion forum. Putting 10% more work in the question could yield 400% better responses.

Linking to what BLIP2 is a good start

https://huggingface.co/docs/transformers/main/en/model_doc/b...

https://github.com/salesforce/LAVIS

Re: MaMMUT: A simple vision-encoder text-decoder architecture for multimodal tasks

#34

Post by m00x »

sitkack wrote: post #33
m00x wrote: post #30

Earlier quoted context omitted.

I'm not sure what you mean. HN has a wide array of people that have multiple sources of information. I was throwing a question out there hoping an expert in the matter would know. They're both multi-modal models for zero-shot answering. I'm not sure what the purpose of this comment is. If no experts see this, I won't get an answer. You don't need to reply with something completely useless that adds nothing to the top…

I am recommending that you add more context to your question since it has been posted a couple times but has had zero engagement. HN is not an AIML discussion forum. Putting 10% more work in the question could yield 400% better responses. Linking to what BLIP2 is a good start https://huggingface.co/docs/transformers/main/en/model_doc/b... https://github.com/salesforce/LAVIS

I'm not expecting someone to go out and do the work for me. If someone happens to know, I'd love a response, otherwise I don't expect one.

This reply serves exactly no purpose.

Re: MaMMUT: A simple vision-encoder text-decoder architecture for multimodal tasks

#35

Post by m00x »

brianjking wrote: post #32
m00x wrote: post #19

Does anyone know the performance of this vs BLIP-2?

I don't think Google has released the weights or anything outside of what is linked here, I'd love to know though. Have you looked at YOLOv8 vs BLIP2?

YOLOv8 is an object detection model, BLIP2 and MaMMUT are imagetext generative models.

Re: MaMMUT: A simple vision-encoder text-decoder architecture for multimodal tasks

#36

Post by brianjking »

m00x wrote: post #35
brianjking wrote: post #32

Earlier quoted context omitted.

I don't think Google has released the weights or anything outside of what is linked here, I'd love to know though. Have you looked at YOLOv8 vs BLIP2?

YOLOv8 is an object detection model, BLIP2 and MaMMUT are image text generative models.

YOLOv8 can do image segmentation with labeling (it might need SAM for labels) - but yeah, I understand.

For my use they're kind of able to be used interchangeably in this case, depending on which can detect objects in the image and generate a caption or image/text match (BLIP2) or detect via object detection or segmentation in SAM.

Post reply on HN