(Potentially dumb) side q: What is Meta's play with their recent AI releases?
Google and Microsoft have tons to gain through search and their cloud; Meta's direction I struggle to understand.
Meta AI Unleashes Megabyte, a Scalable Model Architecture
181–190 of 213 posts
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#182(Potentially dumb) side q: What is Meta's play with their recent AI releases? Google and Microsoft have tons to gain through search and their cloud; Meta's direction I struggle to understand.
Advertisement categorization perhaps? Or automated moderation?
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#183(Potentially dumb) side q: What is Meta's play with their recent AI releases? Google and Microsoft have tons to gain through search and their cloud; Meta's direction I struggle to understand.
If your question is 'why is Meta open sourcing their models ?', stratechery had an article on the topic : https://stratechery.com/2023/free-meta-open-sources-another-...
> Zuckerberg was specifically talking about cloud infrastructure software, but the same point applies to AI capabilities as well: Meta isn’t selling its capabilities; rather, it sells a canvas for users to put whatever content they desire, and to consume the content created by other users. It follows, then, that Meta ought to be fairly agnostic about how and where that content is created; by extension, if Meta were to open source its content creation models, the most obvious place where the content of those models would be published is on Meta platforms. To put it another way, Meta’s entire business is predicated on content being a commodity; making creation into a commodity as well simply provides more grist for the mill.
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#184Earlier quoted context omitted.
Meta has no horse in the race (i.e. they don't have a search engine). So, they don't mind throwing random things out. Withholding it won't really make much of a difference for them, as they don't have a way to productionize the tech.
Meta still has the Portal line of products - which were competitors to the Alexa, Siri and similar Google audio assistant product lines. I just searched and it looks like Meta are currently partnering with Amazon to license Alexa on these devices, but I could imagine they might want to replace it with their own LLM eventually. I am a bit surprised that no one is talking about how these new LLM models will disrupt Ale…
Meta discontinued the Portal line
https://www.theverge.com/2022/11/11/23454019/meta-portal-sma...
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#185Astrophysicists use a similar algorithm to build giant particle evolution models of galaxies and cosmology. It would be computationally prohibitive to compute the gravitational or electromagnetic force for every combinatorial pair of particles. So the mass/force are averaged in a hierarchical set of subcube centroids.
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#186My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…
I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough. Think of a human with below average intelligence. Then think of a human genius. Now consider how incredibly similar their brains are, despite the massive performance gap. It's not like one has 10x the number of neurons/synapses/connections etc. of the other. They're both healthy human brains, and you need pow…
The whole idea of the singularity is that AI starts improving itself. Until that happens, normal human progress in the LLM field is just normal human progress, and will probably follow a similar path of human progress where there's a breakthrough followed by lots of low hanging fruit and hype, then a plateauing and refinement/productization.
LLMs can help humans develop new LLMs faster, but mostly in implementation (CoPilot and ChatGPT), and that's not really the important part. I have yet to see an LLM come up with original ideas.
Given that training data seems to be a big bottleneck, and LLMs are really good at generating text, I think that maybe we can start to talk about the possibility of "singularity" once LLMs are able to generate their own training data that increases their abilities. After all, humans are able to do this. That is the history of human knowledge.
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#187My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall). This is part of why I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of t…
I don’t think I follow this argument. AI has been dropping in costs and complexity the more engineering time is spent on it. It seems like the bottleneck is humans creating new AI techniques right? If an AI is capable of developing new AI techniques unsupervised, isn’t that by definition the singularity? Heck doesn’t even need to be unsupervised. If it can even do most of the heavy lifting for a human I feel like tha…
> If an AI is capable of developing new AI techniques unsupervised, isn’t that by definition the singularity?
AI is not capable of developing new AI techniques unsupervised.
Honestly the lowest hanging fruit should be the ability for LLMs to generate their own textual training data that leads to an improvement in model abilities, since that is what they are supposed to be good at. Until then, it's still garbage in garbage out.
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#188(Potentially dumb) side q: What is Meta's play with their recent AI releases? Google and Microsoft have tons to gain through search and their cloud; Meta's direction I struggle to understand.
If your question is 'why is Meta open sourcing their models ?', stratechery had an article on the topic : https://stratechery.com/2023/free-meta-open-sources-another-... > Zuckerberg was specifically talking about cloud infrastructure software, but the same point applies to AI capabilities as well: Meta isn’t selling its capabilities; rather, it sells a canvas for users to put whatever content they desire, and to con…
Boom - thanks!
For me I still feel they're underplaying it and in some ways diluting their own niche by making content too saturated but it's probably just that I hate social media lol
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#189Earlier quoted context omitted.
The lack of efficiency in the brain is a trade off for adaptability. There's no doubt that we can transmit signals faster than neurons can fire but the cost is drastically reduced adaptation. There are micro, meso, and macroscopic networks in the brain that have different degrees of "adaptability". This isn't even considering the variety in neurons or the additional signalling cascades by non neuronal tissue. How doe…
There are a few counters I have to this and one would be that AI could still end up 'smarter' than us, but have no innate desire to survive. The paperclip maximizer scenarios are an example of this. AI could very well create a highly destructive scenario not only for humans, but also itself because it is "intelligent" but not "aligned" with the idea of survival and evolution.
I erased a lengthy response I was writing because I think our sentiments may actually align. In short, I'm about as certain that these computers aren't going to achieve intelligence in my lifetime as I'm certain that the world is round. But none of that means these models aren't dangerous. Misinformation and accelerating backdoor access to network infrastructures are a couple that we are already see happening.
Re: Meta AI Unleashes Megabyte, a Scalable Model Architecture
#190Ugh. I just spent the past few days reading this exact paper and preparing a detailed presentation on it for my job as an AI researcher, and headlines like this make me roll my eyes very hard. MEGABYTE is indeed a cool new architecture, but it is still very much just a proof of concept at the moment. The paper shows that the model can compete with (but not decimate) vanilla Transformers on the scale of ~1B parameters…
When I started reading this and immediately came across the word "groundbreaking" I paused for a sec and thought to myself: Let me just pretend this word isn't there, it's probably an exaggeration. It takes a special kind of adaptation to filter out all the 10%-40% bluff usually found in these kinds of news articles.
Its always better to just get at the underlying, usually far dryer scientific paper. Even if its not in your field try and read the intro and the conclusion, the authors will describe why this is generally relevant with much less flowery language than the press release. Even a word like “novel” is bold to use too many times.