Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A global workspace in language models

anthropic.com

Post reply on HN
Print view View original

161–170 of 218 posts

Re: A global workspace in language models

#161

Post by djaro »

horacemorace wrote: post #105
ACCount37 wrote: post #56

Earlier quoted context omitted.

"The reversal curse", it rarely shows up in practice but you found a case when it did. The "knowledge landscape" an LLM uses is "directional". It's easy to reach "a quirky music band from Michigan known for colored ties" when you stand at "Tally Hall". But if you stand at "a quirky music band from Michigan known for colored ties", it's harder to reach "Tally Hall" from there. For the "latent knowledge graph" an LLM u…

If this kind of thing holds true for humans we now may understand synesthesia, perhaps we find ourselves with a large enough study pool to map out clever tricks to influence each other.

Its very true for humans as well. For example, if you ask someone to "name 100 historical figures", they will have a very difficult time. But there are of course hundreds of historical figures people woukd recognize. People who manage to do this still tend to fall into sequences so, i.e., naming all the presidents of America or emperors of Rome. But the broader the category, the more difficult it is to come up with examples.

Re: A global workspace in language models

#162

Post by djaro »

appplication wrote: post #75
famouswaffles wrote: post #60

Earlier quoted context omitted.

Recall isn't naturally bidirectional, even for humans. If you are learning vocabulary in a new language, it's common advice to practice both target > source and source > target. Doing only one-way often makes you much better recalling that single direction than both.

I would need further convincing that humans do not naturally tend towards bidirectional recall. Perhaps I’m just on alert anytime I see an LLM-ism that’s met with a claim that the same or similar phenomena holds true in humans as well.

This is very obvious any time you try to create exhaustive lists of something.

For example, for every country in the world, I would recognize it and say, yeah, thats a country.

But if I had to write all ~200 countries into a list, I would probably miss quite a few.

Or, if you gave me names of all US presidents, I would for each of them go, oh yeah, thats a president. But ask me "name all the presidents", I wouldnt get further than 10.

Re: A global workspace in language models

#163

Post by djaro »

timr wrote: post #123
appplication wrote: post #75

Earlier quoted context omitted.

I would need further convincing that humans do not naturally tend towards bidirectional recall. Perhaps I’m just on alert anytime I see an LLM-ism that’s met with a claim that the same or similar phenomena holds true in humans as well.

Do you mean that you don't believe that humans learning a language have the problem the parent described? Because I do, and everyone I've ever met while learning does as well. Do you mean that you don't believe the problem exists in general , because here's another example: if you give a song title, I can easily hum the opening. If you give me the opening, I cannot reliably name the song.

[dead]

Re: A global workspace in language models

#164

Post by maest »

IsTom wrote: post #148
winter_blue wrote: post #95

Earlier quoted context omitted.

There's a shockingly unfathomable level of data compression that these local LLMs are able to remember in an encyclopedic way, so many obscure little things.

Is it? Compressed (text) Wikipedia is in ballpark of dozens of GBs.

Dozens of GBs of compressed text is an unfathomable amount of information from a human POV.

Re: A global workspace in language models

#165

Post by chamomeal »

andy12_ wrote: post #146
paraschopra wrote: post #126

Well, isn't it sort of expected? It's a common misconception that LLMs residual exists for predicting just the next token. While training, we sum/average the losses across whole sequence which puts the pressure to predict future tokens on residual stream of _all_ past tokens. For example, if a particular shape of residual helps reduce loss across several future tokens, it will take that shape (even if it takes a slig…

I think what's unexpected is that it seems that some cases of model errors are truly caused by the model being misaligned? In the "Catching a model fabricating data" example I would have thought that it was just the model being stupid and not understanding the intent of the question, but as per its J-Space, it seems the model is "aware" in some sense that it's manipulating/faking data? There is also now a deeper ques…

Yeah that was the only really surprising part to me. So every time copilot breaks my source code to “fix” its crappy unit tests, does it know what it’s doing?

Re: A global workspace in language models

#166

Post by DoctorOetker »

wongarsu wrote: post #27
com2kid wrote: post #21

Anyone remember that blog post from a few months back where someone was able to improve a model's math ability by just duplicating layers that were activated while solving math problems? Just literally copy/pasting them and linking them together so the model ran through the same layers again? I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights…

Found it: https://news.ycombinator.com/item?id=47500709 Part 3 might be the best introduction: https://dnhkng.github.io/posts/sapir-whorf/ tl;dr: Based on experiments with similar prompts translated to different languages LLM layers group into three phases: the first decodes from the source language into an abstract space, the middle does something, then there's a last part where the abstract result gets transformed…

I wonder if the dnhkng results could be correlated to reasoning in first order logic / set theory notation?

There will be multiple notations (MetaMath, Lean, and essentially Frege's notation everyone learns in high school), and we could try to identify how the neural networks represent them as vectors (or vector combinations). The moment formal logic can be connected to the reasoning representations, regularization can be reduced to eliminating internal inconsistencies.

Re: A global workspace in language models

#167

Post by amelius »

paraschopra wrote: post #126

Well, isn't it sort of expected? It's a common misconception that LLMs residual exists for predicting just the next token. While training, we sum/average the losses across whole sequence which puts the pressure to predict future tokens on residual stream of _all_ past tokens. For example, if a particular shape of residual helps reduce loss across several future tokens, it will take that shape (even if it takes a slig…

Yeah it would be more surprising if all the hidden state was completely uncorrelated to anything in the output.

Re: A global workspace in language models

#168

Post by dnhkng »

echelon wrote: post #44
com2kid wrote: post #21

Anyone remember that blog post from a few months back where someone was able to improve a model's math ability by just duplicating layers that were activated while solving math problems? Just literally copy/pasting them and linking them together so the model ran through the same layers again? I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights…

> I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights do what. Too bad the frontier models are closed weights. Maybe the research community and whole rest of the world will build on open and all the advances will happen in open ecosystems instead.

Too bad they also don't give anything back to individual researchers. Oh well, wasn't expecting much.

Re: A global workspace in language models

#169

Post by cubefox »

taosu_la wrote: post #107

The article says that this ability was discovered around opus 4.5, but open-source models like deepseek and glm have already surpassed the capabilities of opus4.5. But why hasn't this been discovered before? Is it because the scale of computing power is not large enough?

More likely it was because Anthropic invests a lot in interpretability research, unlike other AI companies.

Re: A global workspace in language models

#170

Post by mike_hearn »

wongarsu wrote: post #27
com2kid wrote: post #21

Anyone remember that blog post from a few months back where someone was able to improve a model's math ability by just duplicating layers that were activated while solving math problems? Just literally copy/pasting them and linking them together so the model ran through the same layers again? I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights…

Found it: https://news.ycombinator.com/item?id=47500709 Part 3 might be the best introduction: https://dnhkng.github.io/posts/sapir-whorf/ tl;dr: Based on experiments with similar prompts translated to different languages LLM layers group into three phases: the first decodes from the source language into an abstract space, the middle does something, then there's a last part where the abstract result gets transformed…

Nah, it's a cool blog post especially as it was real AI research done at home (albeit with a ridiculously expensive PC), but Anthropic and other labs have been investigating this kind of thing for years.

Even the original transformer architecture makes this clear. It had an explicit "encoder" phase and then a "decoder" phase. Modern LLMs collapse the two together, or are sometimes described rather confusingly as being decoder only. But what they're doing is more or less the same.

Post reply on HN