Viewing profile — leogao

> The purpose here is not to responsibly warn us of a real threat. If that were the aim there would be a lot more shutting down of data centres and a lot less selling of nuclear-we…

39% just doesn't pass basic muster. in the past year, my rent hasn't near-doubled. it doesn't cost anywhere near twice as much as last year to buy food or clothing or transportatio…

the converse is that good hiring managers who can spot actual undervalued talent have a huge advantage.

in the grand scheme of things, this is a very small amount of plastic waste, and as far as resources go, one of the less scarce ones. at some point, the cost of the hand wringing t…

where are you getting this 39% figure? inflation in 2025 was only 2.7%.

this kind of cynicism hurts young people who actually do good work on their own though.

I'm not an expert at hardware, so take this with a grain of salt, but there are two main reasons: - Discrete optimisation is always going to be harder than continuous optimization.…

I'm also very excited about SAE/Transcoder based approaches! I think the big tradeoff is that our approach (circuit sparsity) is aiming for a full complete understanding at any cos…

There's been some work (e.g RASP - https://arxiv.org/abs/2106.06981 ) on taking logical computations and compiling them into transformer weights.

Mixture of experts sparsity is very different from weight sparsity. In a mixture of experts, all weights are nonzero, but only a small fraction get used on each input. On the other…

For what it's worth, we think it's unfortunately quite unlikely that frontier models will ever be trained with extreme unstructured sparsity, even with custom sparsity optimized ha…

you can always have an LVM layer for atomic snapshots

btrfs has eaten my data within the last decade. (not even because of the broken erasure coding, which I was careful to avoid!) not sure I'm willing to give it another chance. I'd m…

> Sentences like the opening line of the Declaration of Independence simply do not occur in conversation. maybe not in your conversations

The GPT-4 pretraining set included chess games in PGN notation from 1800+ ELO players. I can't comment on any other models.

Nvidia SASS has a similar instruction too (LOP3.LUT)

I looked into the question of what counts as reasonable care and wrote up my conclusions here: https://www.lesswrong.com/posts/kBg5eoXvLxQYyxD6R/my-takes-o...

In domains like ML, people care way more about the half precision FLOPs than single precision.

Is there a better test than the one that is normally prescribed, to distinguish the false positive cases you describe?

The paper cites Anthropic's work extensively.

Note that we focus on random positive activations, which are less susceptible to interpretability illusions than top activations (but also look less impressive as a result). We als…

The paper introduces substantial improvements over the methodology in the Anthropic SAE paper, and the research was done concurrently.

We were planning to release the paper around this time independent of the other events you mention. I think it is still predominantly accurate to say that we have no idea how LLMs …

This project has been in the works for about a year. The initial commit to the public repo was not really closely related to this project, it was part of the release of the Transfo…