Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small AI Models Gain Traction In places with unreliable networks

spectrum.ieee.org

Post reply on HN
Print view View original

81–90 of 92 posts

Re: Small AI Models Gain Traction In places with unreliable networks

#81

Post by dTal »

dofm wrote: post #62
simianwords wrote: post #34

Earlier quoted context omitted.

No this will never work. Domain specific models will never be a thing because intelligence carries over and compounds. Why didn’t OpenAI release a math specific model? Why not a literature specific one? Why do they instead have generic models of different sizes? And how did all labs converge on this? Why does Fable just not train on non cybersec and non biology data but instead have clearly costly and annoying classi…

> No this will never work. This bet is too early. > Why didn’t OpenAI release a math specific model? Why not a literature specific one? Why do they instead have generic models of different sizes? And how did all labs converge on this? Because they have a very early product and they could train it, brute force, with access to an extraordinarily large pool of money. So did all the other labs. Because it was thus easier…

Rats are a weird choice for that comparison. They're some of the smartest animals on the planet, some say smarter than dogs.

Re: Small AI Models Gain Traction In places with unreliable networks

#82

Post by roadside_picnic »

stingraycharles wrote: post #32
N_Lens wrote: post #18

I strongly believe this premise in the article is correct - we will see a lot of tiny, hyper specialized models for individual tasks, and perhaps that will converge with an orchestration layer for a generalized intelligence that controls these specialized tiny models, that will be quite capable. I don't foresee AGI arising out training bigger LLMs (Though investors won't realise that for a while yet). It's actually h…

> It's actually how organic brains work - specialized tasks are offloaded to local cortical columns. How are small isolated language models more similar to that than MoE in LLMs?

MoEs don't route like most people imagine. They aren't learning topic based experts despite the name

The original Mixtral paper [0] (in the "Routing analysis" section) found:

"surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic"

A quick skim of more recent analysis on MoE shows that this hasn't changed. MoE models do appear to work, but don't appear to do what the name implies, if anything they're routing based on the structure of the text and not the semantic content (and we're still not entirely sure what they're doing).

0. https://arxiv.org/pdf/2401.04088

Re: Small AI Models Gain Traction In places with unreliable networks

#83

Post by andy99 »

thereitgoes456 wrote: post #36
simianwords wrote: post #34

Earlier quoted context omitted.

No this will never work. Domain specific models will never be a thing because intelligence carries over and compounds. Why didn’t OpenAI release a math specific model? Why not a literature specific one? Why do they instead have generic models of different sizes? And how did all labs converge on this? Why does Fable just not train on non cybersec and non biology data but instead have clearly costly and annoying classi…

DeepMind did release a math specific model. And OpenAI has released a coding specific model. The answer to your question is “because the market isn’t big enough”, not because it doesn’t work. Why would knowing about 2019 internet memes help you in any way at coding?

> Why would knowing about 2019 internet memes help you in any way at coding?

99.99% of the knowledge an LLM has is useless for a given scenario, the hard part is knowing what the .01% that’s needed is. Knowing as much as it can means the model can handle edge cases, turns of phrase, etc.

Put another way, it avoids overfitting. That’s basically the insight that’s given way to the current AI boom.

Re: Small AI Models Gain Traction In places with unreliable networks

#84

Post by dofm »

dTal wrote: post #81
dofm wrote: post #62

Earlier quoted context omitted.

> No this will never work. This bet is too early. > Why didn’t OpenAI release a math specific model? Why not a literature specific one? Why do they instead have generic models of different sizes? And how did all labs converge on this? Because they have a very early product and they could train it, brute force, with access to an extraordinarily large pool of money. So did all the other labs. Because it was thus easier…

Rats are a weird choice for that comparison. They're some of the smartest animals on the planet, some say smarter than dogs.

Which is exactly what makes them a good comparison, IMO. They are arguably smarter than dogs with about two fifths as many neurons. A shorter lifespan, faster breeding and many more threats has made them better.

The point I am getting to elliptically is that larger models aren't necessarily the solution. They are one solution pathway.

It is fully possible (I think actually likely) that the ultimately successful path for LLMs will emerge from the pressure of keeping them small, not making them large. Very small, domain specific models could well outperform large models in their domains, and they might even show that domain specialisation is not necessarily much of a limitation, just a useful impetus to stay small. Like how rats can drive those little cars.

(I think the frontier models are potentially already too big. Can't prove it or even close to it, but it feels like this is going to be a story.)

Re: Small AI Models Gain Traction In places with unreliable networks

#85

Post by Danox »

ACCount37 wrote: post #73
andy99 wrote: post #33

Earlier quoted context omitted.

General purpose models are always more robust and generally better than smaller narrower models. My bet is that compute will catch up and any “small” model will still be generally capable, just smaller than sota, rather than intentionally narrow. The exception would be for very well defined tasks where the data distribution never varies, but these are rare and don’t really need “AI” anyway when they do exist.

You're getting downvoted, but you're completely right. There are very few cases in which narrowing a model down is buying you anything worthwhile. It seems like for LLMs, "general intelligence" is expensive, but "one more domain" is fairly cheap.

The demand for smaller models and single purpose AI agents will only get bigger in the future. It’s no different than going for mainframe computers back in the day to personal computers. It’s inevitable…..

Re: Small AI Models Gain Traction In places with unreliable networks

#86

Post by prmph »

> The RxScanner is a handheld spectrometer that scans a pill with infrared light, then sends the item’s molecular profile to an AI model equipped with a pharmaceutical database. In seconds, the AI identifies the medication from its molecular profile—or reports that it’s phony.

Is every tech, including database search "AI" now?

Re: Small AI Models Gain Traction In places with unreliable networks

#87

Post by andy99 »

Danox wrote: post #85
ACCount37 wrote: post #73

Earlier quoted context omitted.

You're getting downvoted, but you're completely right. There are very few cases in which narrowing a model down is buying you anything worthwhile. It seems like for LLMs, "general intelligence" is expensive, but "one more domain" is fairly cheap.

The demand for smaller models and single purpose AI agents will only get bigger in the future. It’s no different than going for mainframe computers back in the day to personal computers. It’s inevitable…..

Has nothing to do with demand, it’s a question of the most efficient way to train a good model. Narrower models are in general worse than general purpose ones, so you’re better using the largest general purpose model that fits in your compute budget vs trying to somehow remove capability or knowledge and assume the model stays as capable for the task you want.

Re: Small AI Models Gain Traction In places with unreliable networks

#89

Post by plastic-enjoyer »

ACCount37 wrote: post #72
plastic-enjoyer wrote: post #38

Earlier quoted context omitted.

> General purpose models are always more robust and generally better than smaller narrower models. What do you mean with more robust?

Less weird unexpected failures, more innate ability to handle edge cases gracefully. Quite important when you're running high on automation and low on oversight.

This may be speculative, but couldn't robustness emerge by having a number of specialized models, that are interconnected and feed into each other? Are there any arguments from ML that would speak against this?

Re: Small AI Models Gain Traction In places with unreliable networks

#90

Post by ACCount37 »

plastic-enjoyer wrote: post #89
ACCount37 wrote: post #72

Earlier quoted context omitted.

Less weird unexpected failures, more innate ability to handle edge cases gracefully. Quite important when you're running high on automation and low on oversight.

This may be speculative, but couldn't robustness emerge by having a number of specialized models, that are interconnected and feed into each other? Are there any arguments from ML that would speak against this?

That does work. Even if you drop the "specialized" part. Ensembles of the same architecture at the same scale trained on the same data do outperform a singular model of the same line - especially on corner cases. Successes of an ensemble correlate stronger than failures do.

The usual argument against is that if you have "a number of specialized models" that perform well in ensemble, you can take that ensemble, and distill it into a single larger model (dense or integrated sparse, like MoE), and get the same improvement in performance with an efficiency win.

This works because having those "specialized models" duplicates a lot of the highly conserved "low level" wiring that's required for a model to function at all. As such, you end up running a small scale version of the same "backbone" computational processes many times. "Merging" those models into a larger, denser model allows for a singular strong "backbone" to be used for everything.

Post reply on HN