31–40 of 218 posts
This, taken in combination with the SAE paper, the golden-gate claude paper, the feelings / introspection paper, and note in the fable system card (that they are silently nerfing responses about activation shaping), is basically confirmation to me that they have a new technique they they are using during training (along the vibe space of these mechinterp papers), and its probably some kind of representation learning…
As long as language models are liars, such as documented here recently: https://distrowatch.com/weekly.php?issue=20260706#freebsd We should really stop giving these liar models any further credibility.
Don't get me wrong - I personally "trust" an LLM as a source of facts about as far as I could throw a rack of GPUs. But this article you linked takes a whole lot of words to cast LLMs as the villian for amplifying a bit of bad information originally published by a usually reliable and widely-cited source:
"In short, either Phoronix mocked up the screenshots to demonstrate what the feature could look like, or perhaps they were testing a preview snapshot for FreeBSD 15.1 which was never shipped. Either way, it looks like other blogs and reviewers picked up on this and shared the information, presenting it as a feature which would be (or was included) in FreeBSD's latest version."
It would be really cool if they could expose this information to customers somehow. Imagine: - having a log of the most prominent J-space tokens during your customer support chatbot's interactions with a user, so you can have more introspection into why a particular outcome happened - being able to detect certain thoughts associated with undesirable behavior (hallucinations, overstepping authority, lying, etc.) and t…
https://github.com/anthropics/jacobian-lens
Looks like it should be easy to use on open weights models.
Anyone remember that blog post from a few months back where someone was able to improve a model's math ability by just duplicating layers that were activated while solving math problems? Just literally copy/pasting them and linking them together so the model ran through the same layers again? I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights…
Yeah! I still think about that sometimes. Mind-blowing that worked at all, let alone improved performance.
As long as language models are liars, such as documented here recently: https://distrowatch.com/weekly.php?issue=20260706#freebsd We should really stop giving these liar models any further credibility.
They might as well change their name to Anthropomorphic at this point.
This is fascinating research. I feel this is a significant leap in interpretability research. Since we know J-Space exists and is bi-directional, we can train models on the same and come up with meta cognition abilities. I also fear that the big corporations might use the same to run targeted ads, capitalistic shenanigans. Which they might already be doing through system prompts.
I’m confused where in the weights the jspace is.
There was a series of blog posts posted to HN a while ago investigating how models behave on similar prompts in different languages. To paraphrase the results: the first couple layers map the query to some internal encoding that's mostly independent of the language. Then there are layers in the middle, then the last couple layers map the result back to the target language. You can actually take those middle layers an…
> Recent research on smaller models has shown hints of shared grammatical mechanisms across languages. We investigate this by asking Claude for the "opposite of small" across different languages, and find that the same core features for the concepts of smallness and oppositeness activate, and trigger a concept of largeness, which gets translated out into the language of the question.