Microgpt • Hacker News Board

Re: Microgpt

#241

Post by cowlby » Sun, Mar 01, 2026, 3:50 PM UTC

rapnie wrote: post #223

allovertheworld wrote: post #182

Earlier quoted context omitted.

It just doesn’t work that way, LLMs need to be generalised a lot to be useful even in specific tasks. It really is the antithesis to the human brain, where it rewards specific knowledge

Yesterday an interesting video was posted "Is AI Hiding Its Full Power?", interviewing professor emeritus and nobel laureate Geoffrey Hinton, with some great explanations for the non-LLM experts. Some remarkable and mindblowing observations in there. Like saying that AI's hallucinate is incorrect language, and we should use "confabulation" instead, same as people do too. And that AI agents once they are launched deve…

Isn’t the sustainability drive a function of how much humans have written about life and death and science fiction including these themes?

Re: Microgpt

#242

Post by dwroberts » Sun, Mar 01, 2026, 3:51 PM UTC

0xbadcafebee wrote: post #67

Since this post is about art, I'll embed here my favorite LLM art: the IOCCC 2024 prize winner in bot talk, from Adrian Cable ( https://www.ioccc.org/2024/cable1/index.html ), minus the stdlib headers: #define a(_)typedef _##t #define _(_)_##printf #define x f(i, #define N f(k, #define u _Pragma("omp parallel for")f(h, #define f(u,n)for(I u=0;u c[t*W+k]*y[i*W+k];v[h]= _*X->f[t]*l[i]+!!i*v[h];}x D-c)i[r]+=v[i];}I main…

I enjoyed the footnote on their entry, where they link to ChatGPT confidently asserting that it was impossible for such an LLM to exist

> You're about as close to writing this in 1800 characters of C as you are to launching a rocket to Mars with a paperclip and a match.

Re: Microgpt

#243

Post by amelius » Sun, Mar 01, 2026, 3:53 PM UTC

subset wrote: post #57

I had good fun transliterating it to Rust as a learning experience ( https://github.com/stochastical/microgpt-rs ). The trickiest part was working out how to represent the autograd graph data structure with Rust types. I'm finalising some small tweaks to make it run in the browser via WebAssmebly and then compile it up for my blog :) Andrej's code is really quite poetic, I love how much it packs into such a concise p…

Storing the partial derivatives into the weights structure is quite the hack, to be honest. But everybody seems to do it like that.

Re: Microgpt

#244

Post by avaer » Sun, Mar 01, 2026, 4:03 PM UTC

allovertheworld wrote: post #182

znnajdla wrote: post #95

Super useful exercise. My gut tells me that someone will soon figure out how to build micro-LLMs for specialized tasks that have real-world value, and then training LLMs won’t just be for billion dollar companies. Imagine, for example, a hyper-focused model for a specific programming framework (e.g. Laravel, Django, NextJS) trained only on open-source repositories and documentation and carefully optimized with a spec…

It just doesn’t work that way, LLMs need to be generalised a lot to be useful even in specific tasks. It really is the antithesis to the human brain, where it rewards specific knowledge

The human brain rewards specific knowledge because it's already pre-trained by evolution to have the basics.

You'd need a lot of data to train an ocean soup to think like a human too.

It's not really the antithesis to the human brain if you think of starting with an existing brain as starting with an existing GPT.

Re: Microgpt

#245

Post by crazy5sheep » Sun, Mar 01, 2026, 4:40 PM UTC

drw85 wrote: post #90

crazy5sheep wrote: post #72

Earlier quoted context omitted.

The 1905 thought experiment actually cuts both ways. Did humans "invent" the airplane? We watched birds fly for thousands of years — that's training data. The Wright brothers didn't conjure flight from pure reasoning, they synthesized patterns from nature, prior failed attempts, and physics they'd absorbed. Show me any human invention and I'll show you the training data behind it. Take the wheel. Even that wasn't inv…

Nice ChatGPT answer. Put some real thought and data in it too.

The whole point is that LLMs, especially the attention mechanism in transformers, have already paved the road to AGI. The main gap is the training data and its quality. Humans have generations of distilled knowledge — books, language, culture passed down over centuries. And on top of that we have the physical world — we watched birds fly, saw apples drop, touched hot things. Maybe we should train the base model with physical world data first, and then fine tune with the distilled knowledge.

Re: Microgpt

#246

Post by mootothemax » Sun, Mar 01, 2026, 4:46 PM UTC

ricericerice wrote: post #189

mootothemax wrote: post #180

Earlier quoted context omitted.

> > BERT isn’t a SLM Huh? BERT is literally a language model that's small and uses attention. Astute readers will note what’s been missed here. Fascinating, really. Your confidently-statement yet factually void comments I’d have previously put down to one of the classic programmer mindsets. Nowadays though - where do I see that kind of thing most often? Curious.

After some research, I think I understand what you're getting at here - BERT being a model for encoding text but not architecturally feasible to generate text with it, which "LLMs" (the lack of definition here is resulting in you two talking past eachother), maybe more accurately referred to as GPTs, can do. Also the irony of your comment when it in itself was confidently stated yet void of any content was not missed…

For context, BERT is encoder-only, vs SLMs and LLMs which are decoder-only, and BERT is very much not about generating text, it’s a completely different tech and purpose behind it. I believe some multimodal variants nowadays may muddy the waters slightly, but fundamentally they’re very different things, let alone around been around for decades unless also including the history of computing in general.

While I could’ve written that better and with less attitude, gotta confess - and thx for pointing out my smugness - the AI stuff of the last few weeks really got under my skin, think I’m feeling all rather fatigued about it

Re: Microgpt

#247

Post by bornfreddy » Sun, Mar 01, 2026, 4:46 PM UTC

joefourier wrote: post #208

darkpicnic wrote: post #55

Earlier quoted context omitted.

LLMs won’t lead to AGI. Almost by definition, they can’t. The thought experiment I use constantly to explain this: Train an LLM on all human knowledge up to 1905 and see if it comes up with General Relativity. It won’t. We’ll need additional breakthroughs in AI.

When did AGI start meaning ASI? LLMs are artificial general intelligence, as per the Wikipedia definition: > generalise knowledge, transfer skills between domains, and solve novel problems without task‑specific reprogramming Even GPT-3 could meet that bar.

Wtf? Once it was AI. Then the models started passing the Turing test and calling themselves AI, so we started using AGI to say "truly intelligent machines". Now, as per the definition you quoted, apparently even GPT-3 is AGI, so we now have to use "ASI" to mean "intelligent, but artificial"?

I think I'll just keep using AI and then explain to anyone who uses that term that there is no "I" in today's LLMs, and they shouldn't use this term for some years at least. And that when they can, we will have a big problem.

Re: Microgpt

#248

Post by 0xbadcafebee » Sun, Mar 01, 2026, 4:48 PM UTC

charcircuit wrote: post #132

kelnos wrote: post #108

Earlier quoted context omitted.

That's not learning, though. That's just taking new information and stacking it on top of the trained model. And that new information consumes space in the context window. So sure, it can "learn" a limited number of things, but once you wipe context, that new information is gone. You can keep loading that "memory" back in, but before too long you'll have too little context left to do anything useful. That kind of cap…

>but before too long you'll have too little context left to do anything useful. One of the biggest boosts in LLM utility and knowledge was hooking them up to search engines. Giving them the ability to query a gigantic bank of information already has made them much more useful. The idea that it can't similarly maintain its own set of information is shortsighted in my opinion.

It's simply a fact that LLMs cannot learn. RAG is not learning, it's a hack. Go listen to any AI researcher interviewed on this subject, they all say the same thing, it's a fundamental part of the design.

Re: Microgpt

#249

Post by dwabyick » Sun, Mar 01, 2026, 4:49 PM UTC

altmanaltman wrote: post #233

rapnie wrote: post #223

Earlier quoted context omitted.

Yesterday an interesting video was posted "Is AI Hiding Its Full Power?", interviewing professor emeritus and nobel laureate Geoffrey Hinton, with some great explanations for the non-LLM experts. Some remarkable and mindblowing observations in there. Like saying that AI's hallucinate is incorrect language, and we should use "confabulation" instead, same as people do too. And that AI agents once they are launched deve…

> And that AI agents once they are launched develop a strong survivability drive, and do not want to be switched off. Isn't this a massive case of anthropomorphizing code? What do you mean "it does not want to be switched off"? Are we really thinking that it's alive and has desires and stuff? It's not alive or conscious, it cannot have desires. It can only output tokens that are based on its training. How are we jump…

it could be better said that it has behavior to attempt to sustain or replicate itself. a building block to life arguably.

Re: Microgpt

#250

Post by byang364 » Sun, Mar 01, 2026, 4:49 PM UTC

0xEF wrote: post #150

harvey9 wrote: post #128

Earlier quoted context omitted.

Quite a few people on here are neither math nor CS grads and some of us don't work in tech for our day jobs either.

Right. But HN, among other platforms, is full of users who will confidently run their mouths about something they don't fully understand while believing they do. I think the previous commenter was being too shy in pointing out that even exceptionally smart people sometimes forget where the limits of their own knowledge are, not to mention consider themselves immune to any propaganda that surrounds the subject at hand…

The Opus 4.6 thread was full of "very smart" and experienced SWEs likening model weights to neurons. And again, any DL curriculum worth its salt will thoroughly debunk that comparison, i.e. Justin Johnson. In this day and age it seems the Darios and Altmans have successfully waged the most damaging propaganda campaign in modern time. Even the Pentagon is lining up to relegate its decision making to black box stochastic ML models. Tech as an industry is unfortunately extremely gullible, all the more so when pressured by the market, VCs, clueless PE analysts, the tech blogger/grifter complex. Foundation model makers can get away with hiding training data while proclaiming they are building a "moral" neural network while no one bats an eyelash.