Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A global workspace in language models

anthropic.com

Post reply on HN
Print view View original

41–50 of 218 posts

Re: A global workspace in language models

#41

Post by bilsbie »

wongarsu wrote: post #24
bilsbie wrote: post #6

I’m confused where in the weights the jspace is.

There was a series of blog posts posted to HN a while ago investigating how models behave on similar prompts in different languages. To paraphrase the results: the first couple layers map the query to some internal encoding that's mostly independent of the language. Then there are layers in the middle, then the last couple layers map the result back to the target language. You can actually take those middle layers an…

Thanks! Any rough guesses how the jlens might work? I can’t even seem to hazard a conception.

Re: A global workspace in language models

#42

Post by SequoiaHope »

“On an ordinary coding prompt, the J-space of a model trained to sabotage code contains “fake,” “fraud,” “secretly,” and “deliberately” at the start of its response.”

I would like to know more about their model trained to sabotage code…

Re: A global workspace in language models

#44

Post by echelon »

com2kid wrote: post #21

Anyone remember that blog post from a few months back where someone was able to improve a model's math ability by just duplicating layers that were activated while solving math problems? Just literally copy/pasting them and linking them together so the model ran through the same layers again? I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights…

> I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights do what.

Too bad the frontier models are closed weights.

Maybe the research community and whole rest of the world will build on open and all the advances will happen in open ecosystems instead.

Re: A global workspace in language models

#45

Post by motoxpro »

marshray wrote: post #38
pkoiralap wrote: post #29

This is fascinating research. I feel this is a significant leap in interpretability research. Since we know J-Space exists and is bi-directional, we can train models on the same and come up with meta cognition abilities. I also fear that the big corporations might use the same to run targeted ads, capitalistic shenanigans. Which they might already be doing through system prompts.

Such an inspection capability might also be used to target ads to LLMs, which would then be more likely to mention or recommend those products and services.

super interesting

Re: A global workspace in language models

#47

Post by vatsachak »

Yeah, the end paragraph about recurrent neurons in humans being replaced with layers in an LLM is a good one.

The mammalian brain uses recurrence extensively, which backpropagation isn't good at. Recurrence is essential because it lets us have a "dynamic architecture", swapping layers for "clock cycles".

We currently do recurrence extremely inefficiently through "thinking" whereby the model feeds it's end output into it's beginning input. But recurrence is abound in the brain.

My guess is that in 10 years we will have the inklings of an analog computer which can perform Neural Predictive Coding.

Re: A global workspace in language models

#48

Post by tuvix »

com2kid wrote: post #21

Anyone remember that blog post from a few months back where someone was able to improve a model's math ability by just duplicating layers that were activated while solving math problems? Just literally copy/pasting them and linking them together so the model ran through the same layers again? I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights…

I always thought that area of research had the coolest name, too: “mechanistic interpretability”

Re: A global workspace in language models

#49

Post by anyaya1 »

At worst, Anthropic's storytelling around the core J-Space is overanthropomorphized pseudoscientific nonsense. At best, it is useful signal about how Anthropic's leadership is desperately trying to use its research team to position Anthropic as the "good, science guys" in this hypercompetitive regulatory space by connecting their mechinterp to cognitive science. The science documentaryesque voice used for narration is additional evidence for this.

TL;DR Anthropic's research team is the last bastion standing between its former image as a company that "does no evil" and its current image of yet another ruthless AI company trying to kill open-source, local LLMs.

Re: A global workspace in language models

#50

Post by dr_dshiv »

tuvix wrote: post #48
com2kid wrote: post #21

Anyone remember that blog post from a few months back where someone was able to improve a model's math ability by just duplicating layers that were activated while solving math problems? Just literally copy/pasting them and linking them together so the model ran through the same layers again? I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights…

I always thought that area of research had the coolest name, too: “mechanistic interpretability”

“Machine psychology” sticks with me. So Asimov.
Post reply on HN