Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How LLMs work

0xkato.xyz

Post reply on HN
Print view View original

111–120 of 293 posts

Re: How LLMs work

#111

Post by trollbridge »

ai_slop_hater wrote: post #36
trollbridge wrote: post #26

Earlier quoted context omitted.

We don't really know why language works with humans, either. If you raise a baby from birth, you kind of observe how it is learning language, but the process is also rather mysterious. My eldest son's first word was to actually imitate a cow mooing, and then after that to imitate a motor noise of a tractor or truck. And then after that a meow. (His first complete sentence was "King Graham fell"...) My next child took…

Human brain capabilities are truly amazing, imagine if people didn’t treat their children as if they are stupid and didn’t constantly lie to them, because kids are stupid right, they wouldn’t understand. What heights could be reached.

They aren't stupid, but they aren't quite ready to handle the full responsibilities of the world and worry about things they don't need to worry about.

My son is very worried about black holes lately when he learned anything that goes into one can't get out. He's pretty concerned astronauts could get stuck in one some day. So I explained to him that Hawking radiation does actually mean you can eventually get out; it just takes some time.

I didn't think it pertinent to mention spaghettification, the fact anywhere near a black hole will be really hot, or that cosmic censorship means whatever Hawking-radiates from a black hole wouldn't be an astronaut anymore.

It was also fun to hear Hawking speak. He wanted to know if Hawking was a robot. I said no, but he has a robot talk for him. Not quite true, but close enough.

Re: How LLMs work

#112

Post by trollbridge »

jumploops wrote: post #35
trollbridge wrote: post #26

Earlier quoted context omitted.

We don't really know why language works with humans, either. If you raise a baby from birth, you kind of observe how it is learning language, but the process is also rather mysterious. My eldest son's first word was to actually imitate a cow mooing, and then after that to imitate a motor noise of a tractor or truck. And then after that a meow. (His first complete sentence was "King Graham fell"...) My next child took…

Completely agree! It’s interesting to me how similar attempting to understand LLMs is to neuroscience. “When we turn this bit off, this other thing happens… if we change these weights the Eiffel Tower is now in Rome” We’re basically just probing around and trying to reverse engineer an emergent system. To your point, this system may be quite different from model to model (human to human) although some similarities li…

Computer vision ends up displaying emergent behaviour. It just "figures out" things.

Re: How LLMs work

#113

Post by layla5alive »

Borealid wrote: post #90
AltruisticGapHN wrote: post #88

I don't like how most LLM explainer articles and videos say that essentially a LLM " predicts the next word". I'm a developer but not very good at maths and I still don't understand any of it. A LLM clearly has some "visual" capacity. You ask Gemini to build something with Canvas and it's able to reason about the shape of things. Like recently I waanted a checkbox that has like a gradient flowing around the edge. It…

Your casual understanding is imprecise. At all times the LLM is, indeed, predicting the next token. Anything it does emerges from that. It did not "figure anything out". It predicted that text describing the use of a radial gradient was likely to follow text describing your problem.

Lol, the bird did not 'fly' - it just flapped its wings and generated lift!

Re: How LLMs work

#114

Post by sirsinsalot »

2muchcoffeeman wrote: post #110
crossroadsguy wrote: post #86

Earlier quoted context omitted.

What hopes/paths does a mere CS bachelor (not deep into stats/maths), and mid level dev (native mobile only; 10-15 years exp.), have about not only understanding it (maybe not fully) but getting possibly into this as a career? Not expecting churning out models and AI systems from the first weeks/months but entry/employment into this field? (If I can be honest, and I am not being disparaging about anything lest it mig…

I think you need to ask what you actually want to do with the AI. If you want to be a researcher and come out with the next breakthrough, get ready to go back to school and learn some math. If you just need to learn how to use it well and build things with it, then you probably just need to have a high level understanding. Same as programming. I’d bet most programmers have no idea about the physics that makes compute…

You missed the third and most important reason to learn: fun.

Which sums up HN these days.

Re: How LLMs work

#115

Post by slickytail »

antirez wrote: post #75
malwrar wrote: post #10

Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I…

There is a different way to look at this: that is, actually the Transformer is a minimal complication of what the based model is: in theory the neural network could be just a huge FFN, which is anyway the part of the Transformer that does the heavy lifting. But this would be impossibile to train both numerically and computationally, so the Transformer encodes enough priors for it to work: the causal attention, and th…

[dead]

Re: How LLMs work

#116

Post by sirsinsalot »

redox99 wrote: post #100
10GBps wrote: post #15

Earlier quoted context omitted.

Yep. It's nearly identical to the neural nets we were using in the 90s. Back then even a supercomputer wasn't big enough or fast enough to do what we do today. I have to wonder though. Is this all a human brain is? A similar thing to an LLM just scaled exponentially larger. I mean a brain is not just neurons with simple connections to each other. The neurons, axons, dendrites, , etc in a brain are all holding and pro…

In the 90s you didn't have norm layers, residuals, attention, and some more. So you're missing a lot of the building blocks that make LLMs. It's not a matter of just having the compute.

I think the attention mechanism is so simple but so revolutionary that people forget it.

Like the best leaps in thinking, once it is made, is is immediately obvious and intuitive.

Re: How LLMs work

#117

Post by locallost »

AltruisticGapHN wrote: post #88

I don't like how most LLM explainer articles and videos say that essentially a LLM " predicts the next word". I'm a developer but not very good at maths and I still don't understand any of it. A LLM clearly has some "visual" capacity. You ask Gemini to build something with Canvas and it's able to reason about the shape of things. Like recently I waanted a checkbox that has like a gradient flowing around the edge. It…

I don't want to pretend I can explain LLMs, but the same "math" can be applied for visual and non visual things. The dot product of two vectors gives you the angle between them. This is true in 2 or 3 dimensions. But it's also true in 4, 5, 6...n dimensions even though we cannot visualize a 4d space. That it's an angle is relevant for you in the space you can comprehend, but for math or a machine it works in any number of dimensions. So it does need to understand anything visually if the math checks out.

Re: How LLMs work

#119

Post by qsera »

layla5alive wrote: post #113
Borealid wrote: post #90

Earlier quoted context omitted.

Your casual understanding is imprecise. At all times the LLM is, indeed, predicting the next token. Anything it does emerges from that. It did not "figure anything out". It predicted that text describing the use of a radial gradient was likely to follow text describing your problem.

Lol, the bird did not 'fly' - it just flapped its wings and generated lift!

More like being suspended by a thread...

Re: How LLMs work

#120

Post by qsera »

AltruisticGapHN wrote: post #88

I don't like how most LLM explainer articles and videos say that essentially a LLM " predicts the next word". I'm a developer but not very good at maths and I still don't understand any of it. A LLM clearly has some "visual" capacity. You ask Gemini to build something with Canvas and it's able to reason about the shape of things. Like recently I waanted a checkbox that has like a gradient flowing around the edge. It…

>is the LLM is able to represent things in space

It is imitating the text written by humans who can represent things in space.

Post reply on HN