Viewing profile — mxwsn

This is the same reasoning behind why Yann Lecun thought test-time scaling would not work for LLMs: compounding error. Instead, the more tokens LLMs use, the better their performan…

No, there are more training tokens than parameters in LLMs. They are in the classical first descent setting.

> Here’s a thought experiment: suppose that a mathematician solved a major problem by having a long exchange with an LLM in which the mathematician played a useful guiding role but…

Great summary. The fact that the auto encoding task is not grounded in thoughts, and their initial training on guessed internal thoughts, raise serious concerns on faithfulness. Fe…

Diffusion and flow matching models generate samples by iterative denoising. Iterative denoising means passing input to the neural network, running a forward pass, and taking the ou…

How do you know that width scaling has been the driving force of improvement?

The Jacobian is first derivatives, but for a function mapping N to M dimensions. It's the first derivative of every output wrt every input, so it will be an N x M matrix. The gradi…

Wow! The title suggests introductory material, but in my opinion this has strong potential to win test of time awards for research.

That's really interesting. What if they RAG search related videos from the prompt, and condition on that to generate? That might explain fidelity like this

Why is not the diffusion training objective? The technique is known as self-conditioning right? Is it an issue with conditional Tweedie's?

AI with ability but without responsibility is not enough for dramatic socioeconomic change, I think. For now, the critical unique power of human workers is that you can hold them r…

Has anyone come across any really cool artifacts? I'd be curious to see

Stablecoins transferred $27 trillion in 2024 - more than Visa and Mastercard combined. This is right in the article. Stablecoins operate using decentralized ledgers on e.g. Ethereu…

Gemini has beat it already, but using a different and notably more helpful harness. The creator has said they think harness design is the most important factor right now, and that …

Huh, I imagined this was because of relaxing regulation.

Good read, thanks for sharing

> But what is the original purpose of AI research? I will speak for myself here, but I know many other AI researchers will say the same: the ultimate goal is to understand how huma…

This ought to be called the qwerty effect, for how the qwerty keyboard layout can't be usurped at this point. It was at the right place at the right time, even though arguably its …

What's surprising about this is how sparsely defined the rewards are. Even if the model learns the formatting reward, if it never chances upon a solution, there isn't any feedback/…

I used sublime from 2013 to 2021. It was great. Since, I've switched to VS Code and haven't looked back.

Context is a challenge for LLMs, but the challenge feels of a different quality to me, than the challenge of incorporating local context into automated decision-making AI like algo…

My interest was piqued, but the extrapolation in [1] is uh... not the most convincing. If there were more data points then sure, maybe

OK - there's always a nonzero chance of hallucination. There's also a non-zero chance that macroscale objects can do quantum tunnelling, but no one is arguing that we "need to live…