81–90 of 100 posts
Earlier quoted context omitted.
Not when the only way to get anything out of it is using these matrices to line up word tokens. Or play with jpeg noise.
Unclear what you are saying here
Earlier quoted context omitted.
Stochastic parrot is way too simplistic a dismissal. It's hand wavy. But I don't get what's so impressive about the nuanced language of a language model that has been given datacenter amounts of compute and virtually all of written word ever put digitalized. Yeah it's the first actually functioning natural language interface. At what cost though. It's completely out of proportion with the benefits and only bubble lev…
Are you referring to the (one-time) training cost or the cost per inferred token? The latter is pretty acceptable these days, especially with smaller models.
Also claiming training costs are 'one off' when there's so much resources being poured into training new and bigger models is disingenuous.
Earlier quoted context omitted.
Yes.
I just tried with 4o and it didn't notice anything, it just rewrote the statements with different words.
Earlier quoted context omitted.
https://news.ycombinator.com/item?id=40721115
"Explain" is a variation of "what do you think about", so you get similar results. https://news.ycombinator.com/item?id=40723601
Earlier quoted context omitted.
Are you referring to the (one-time) training cost or the cost per inferred token? The latter is pretty acceptable these days, especially with smaller models.
Small models can't do impressive language corner cases. And the benefits of the models that can scale linearly while their cost grows geometrically. Also claiming training costs are 'one off' when there's so much resources being poured into training new and bigger models is disingenuous.
Earlier quoted context omitted.
> The theory goes - that if the model "understands" this scenario, then internally, it has has created something that models the real world. But this is merely a definition of what it means to "understand" something. For example just tabulating many input/output combinations would not follow this definition.
From the preface to the 20th anniversary edition of Godel, Escher, Bach: "Meaning cannot be kept out of formal systems when sufficiently complex isomorphisms arise. Meaning comes in despite one's best efforts to keep symbols meaningless! ...When a system of "meaningless" symbols has patterns in it that accurately track, or mirror, various phenomena in the world, then that tracking or mirroring imbues the symbols with…
Earlier quoted context omitted.
Unclear what you are saying here
Claiming LLMs are manipulating 'high dimension concepts' not words when all they can do with these 'high dimension concepts' is outputting words is meaningless.
And, an LLM can output whatever you want. Change the last linear layer from (embedding dimension -> vocab size) to (embedding dimension -> sentiment categories) for example, and hey presto it can produce sentiment analysis (with surprisingly few samples, because... it's learned all the concepts in the transformer blocks in pre-training).
You gotta actually have a clue about how they work. This stuff is no longer magic, there are many resources to learn them in detail.