Meta-Transformer: A unified framework for multimodal learning
kxgong.github.io
Meta-Transformer: A unified framework for multimodal learning
1–10 of 39 posts
Re: Meta-Transformer: A unified framework for multimodal learning
#2This seems like a step in the dangerous direction.
Re: Meta-Transformer: A unified framework for multimodal learning
#3Yo dawg, we heard you like transformers so we put transformers on your transformers so you can train while you train.
The spider web graph shows metatransformers performing worse to their counterparts in almost all fields. Is there a reason I should not believe that an expert model will always outperform a general purpose one, even if it's a metatransformer?
Re: Meta-Transformer: A unified framework for multimodal learning
#4This seems like a step in the dangerous direction.
Why?
Re: Meta-Transformer: A unified framework for multimodal learning
#5Yo dawg, we heard you like transformers so we put transformers on your transformers so you can train while you train. The spider web graph shows metatransformers performing worse to their counterparts in almost all fields. Is there a reason I should not believe that an expert model will always outperform a general purpose one, even if it's a metatransformer?
I mean, there is a somewhat unique value proposition of a multimodal framework like this meta transfirmer. Its goal isn't necessarily to beat expert models in their own game, but to provide a unified framework for processing diverse modalities of data.
I think it aims to leverage the cross-modal relationships and unified learning, which might not be possible with expert models designed for only a single modality.
Even if it performs slightly worse on some tasks, the ability to handle multiple modalities within a single framework is an pretty sweet advantage in scenarios where data from various sources need to be processed simultaneously, and patterns across modalities need to be captured somehow.
A general-purpose model could also be a more cost-effective solution in some cases, ensemble experts are difficult to scale and parallelize.
Re: Meta-Transformer: A unified framework for multimodal learning
#6Yeah, that's where I thought it would go shortly after I tried GPT-4 from openAI. We're clearly at the transformer limits imho (comparing the effectiveness between 3.5 and 4, and the number of parameter in each model is why I think we reached a soft cap).
So since it'll be hard to go deeper, going broader by interlacing different model types might be a way to pierce through.
Re: Meta-Transformer: A unified framework for multimodal learning
#7We need to start ingesting raw scientific data through these models and see what it comes up with. What could these models identify by parsing through raw JWST or Hubble data? Or training against every published scientific paper? Is anyone doing this sort of thing already?
Re: Meta-Transformer: A unified framework for multimodal learning
#8We need to start ingesting raw scientific data through these models and see what it comes up with. What could these models identify by parsing through raw JWST or Hubble data? Or training against every published scientific paper? Is anyone doing this sort of thing already?
Meta's Galactica was an attempt to train an LLM predominantly on scientific papers, articles and so on. It failed pretty spectacularly but Galactica 2, if that's ever a things, might rectify that.
Re: Meta-Transformer: A unified framework for multimodal learning
#9According to the website, the model can then fine-tuned for certain tasks such as image classification.
1. How does the multi-model help here in improving the accuracy of image classification when training is combined from text, images, and audio?
2. How about the speed? I would imagine a model with text, audio and image data would be larger compared to text-only models?
Re: Meta-Transformer: A unified framework for multimodal learning
#10Yo dawg, we heard you like transformers so we put transformers on your transformers so you can train while you train. The spider web graph shows metatransformers performing worse to their counterparts in almost all fields. Is there a reason I should not believe that an expert model will always outperform a general purpose one, even if it's a metatransformer?
performance is bounded and so outperformance will approach episilon?