Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small AI Models Gain Traction In places with unreliable networks

spectrum.ieee.org

Post reply on HN
Print view View original

61–70 of 92 posts

Re: Small AI Models Gain Traction In places with unreliable networks

#61

Post by bluerooibos »

tim-fan wrote: post #4

Is anyone making LLM-in-a-box for emergency supply kits yet? I feel that would be handy in all sorts of situations when networks are down.

> Is anyone making LLM-in-a-box for emergency supply kits yet?

Maybe someone should be making this, but for rebuilding society in the event of a disaster - a solar-powered black box with most of humanity's knowledge within. Even something running one of the Qwen models would be useful.

"So, we had a nuclear war and need to start from scratch. How do I turn this rock into a computer chip?"

Re: Small AI Models Gain Traction In places with unreliable networks

#62

Post by dofm »

simianwords wrote: post #34
N_Lens wrote: post #18

I strongly believe this premise in the article is correct - we will see a lot of tiny, hyper specialized models for individual tasks, and perhaps that will converge with an orchestration layer for a generalized intelligence that controls these specialized tiny models, that will be quite capable. I don't foresee AGI arising out training bigger LLMs (Though investors won't realise that for a while yet). It's actually h…

No this will never work. Domain specific models will never be a thing because intelligence carries over and compounds. Why didn’t OpenAI release a math specific model? Why not a literature specific one? Why do they instead have generic models of different sizes? And how did all labs converge on this? Why does Fable just not train on non cybersec and non biology data but instead have clearly costly and annoying classi…

> No this will never work.

This bet is too early.

> Why didn’t OpenAI release a math specific model? Why not a literature specific one? Why do they instead have generic models of different sizes? And how did all labs converge on this?

Because they have a very early product and they could train it, brute force, with access to an extraordinarily large pool of money. So did all the other labs. Because it was thus easier to scrape everything rather than spend enormous effort (with tools that did not really exist) to partition the training set. Any number of other "because"s.

It's just what they are doing now and it showed the earliest results.

LLMs are still less intelligent than rats, which have tiny brains.

Re: Small AI Models Gain Traction In places with unreliable networks

#63

Post by dofm »

simianwords wrote: post #42
fasterik wrote: post #40

Earlier quoted context omitted.

Your examples (math, literature) involve natural language. It stands to reason that a general language model will be more competitive in those domains. If you want examples of successful domain-specific models, look at AlphaZero and AlphaFold. LLMs aren't anywhere close to achieving that level of competence at abstract strategy games or protein folding. "This will never work" is a pretty confident assertion for a fie…

> I don't foresee AGI arising out training bigger LLMs (Though investors won't realise that for a while yet). This is what the parent said. AGI won't rise out of AlphaZero and AlphaFold in the same way AGI won't rise out of Houdini chess engine. This is the industry consensus.

AGI is a macguffin (or a shaggy dog) for a story told to investors. It has never been plausible on the timescales suggested and it almost certainly will not emerge from LLMs.

Re: Small AI Models Gain Traction In places with unreliable networks

#64

Post by dmezzetti »

100% agree on this.

I've been working on small local models for years with txtai (https://github.com/neuml/txtai). I've published close to 100 models that can run local for RAG, Agents, Vector Search and more (https://huggingface.co/NeuML/collections).

Re: Small AI Models Gain Traction In places with unreliable networks

#65

Post by dofm »

SwellJoe wrote: post #7
tim-fan wrote: post #4

Is anyone making LLM-in-a-box for emergency supply kits yet? I feel that would be handy in all sorts of situations when networks are down.

This is couched in prepper nonsense, but it's got LLM, WikiPedia, maps, etc. A bunch of genuinely useful stuff to keep on a USB stick or whatever: https://www.projectnomad.us/ But, the current model you really want for an emergency kit is Gemma 4 12B QAT 4-bit. At ~7GB on disk, it's small enough to run on a tablet or any modern computer, slowly if you don't have a GPU or modern Apple silicon, but exceedingly smart fo…

The 12B QAT model is really overlooked because the tech industry has been so desperate for the LLM bet to play out to "product market fit" (which means please IPO now) that it has become convinced that coding models are the only things that matter.

Re: Small AI Models Gain Traction In places with unreliable networks

#66

Post by dofm »

wahnfrieden wrote: post #6
cdnsteve wrote: post #5

Earlier quoted context omitted.

Can you expand what you mean?

They want to ask the iOS Foundation model (frontier on device intelligence for something small) for instance about emergency procedures and life-saving info. I wouldn’t trust that model with much at all though. More likely to find what you need from miniature survival guides.

> They want to ask the iOS Foundation model (frontier on device intelligence for something small)

This is a bit of a straw man, TBH.

For one thing, "LLM-in-a-box" doesn't necesssarily imply a device as small as a phone.

For another, you'd need to convince people that the iOS Foundation model is the "frontier" of LLMs that run on phones when it is really not. AFAIK it is noticeably outperformed by the Gemma 4 E2B model and certainly the E4B.

https://blog.google/innovation-and-ai/technology/developers-...

Here is a common-or-garden youtube video that includes a demonstration of how much better the E2B is:

https://www.youtube.com/watch?v=sTxyBUbdZcA

Whether this idea (LLMs for emergency/survival scenarios) has value, I don't know, so I am not offering an opinion, but you should approach it with a good faith argument.

I am an LLM cynic but I suppose if I was to be without connectivity but with power for a while, a device with the Gemma 4 E2B or E4B model on it might be helpful or interesting to have. If such a device had the 12B QAT model on it, that really would cross the line to utility. Not sure it has value in the OP's scenario, still.

Re: Small AI Models Gain Traction In places with unreliable networks

#67

Post by bluGill »

bluerooibos wrote: post #61
tim-fan wrote: post #4

Is anyone making LLM-in-a-box for emergency supply kits yet? I feel that would be handy in all sorts of situations when networks are down.

> Is anyone making LLM-in-a-box for emergency supply kits yet? Maybe someone should be making this, but for rebuilding society in the event of a disaster - a solar-powered black box with most of humanity's knowledge within. Even something running one of the Qwen models would be useful. "So, we had a nuclear war and need to start from scratch. How do I turn this rock into a computer chip?"

If you are rebuilding society most of this knowledge is useless for centuries. You don't have enough labor to build and maintain factories. You will spend centuries in the hunter gather phase struggling to survive, while slowly building agriculture. You will be lucky if you can teach your grandkids to read - since that will be a useless skill.

Print important knowledge on paper and store it in a desert. in 2000 years society and population will advance enough to get a jump start based on our knowledge.

Re: Small AI Models Gain Traction In places with unreliable networks

#68

Post by InsomniacL »

thereitgoes456 wrote: post #36
simianwords wrote: post #34

Earlier quoted context omitted.

No this will never work. Domain specific models will never be a thing because intelligence carries over and compounds. Why didn’t OpenAI release a math specific model? Why not a literature specific one? Why do they instead have generic models of different sizes? And how did all labs converge on this? Why does Fable just not train on non cybersec and non biology data but instead have clearly costly and annoying classi…

DeepMind did release a math specific model. And OpenAI has released a coding specific model. The answer to your question is “because the market isn’t big enough”, not because it doesn’t work. Why would knowing about 2019 internet memes help you in any way at coding?

> Why would knowing about 2019 internet memes help you in any way at coding?

https://github.com/Brainrotlang/brainrot

"Brainrot is a meme-inspired programming language that translates common programming keywords into internet slang and meme references."

Re: Small AI Models Gain Traction In places with unreliable networks

#70

Post by jonbodner »

bluGill wrote: post #67
bluerooibos wrote: post #61

Earlier quoted context omitted.

> Is anyone making LLM-in-a-box for emergency supply kits yet? Maybe someone should be making this, but for rebuilding society in the event of a disaster - a solar-powered black box with most of humanity's knowledge within. Even something running one of the Qwen models would be useful. "So, we had a nuclear war and need to start from scratch. How do I turn this rock into a computer chip?"

If you are rebuilding society most of this knowledge is useless for centuries. You don't have enough labor to build and maintain factories. You will spend centuries in the hunter gather phase struggling to survive, while slowly building agriculture. You will be lucky if you can teach your grandkids to read - since that will be a useless skill. Print important knowledge on paper and store it in a desert. in 2000 years…

There is no rebuilding society because of energy. All of the oil and coal that could be extracted by a civilization that only has wood for fuel has been used. If we go back to the Stone Age, we aren’t returning to the present. If we’re lucky, we will get to the Middle Ages.
Post reply on HN