Viewing profile — valine

> The two modified satellites will test hardware planned for Starlink V3 and will attempt to scan Starship’s heat shield and transmit imagery down to operators to test methods of a…

Apple silicon has been an unmitigated success so it makes sense they’d go with Ternus. On a related note Apple needs to add Ternus to their spell check dictionary

Yup exactly, in principle it helps with both inference speed by reducing memory bandwidth usage and also reduces the memory footprint of your kvcache.

So let’s start with a really simple decoder transformer with a single layer and single attention head, and train it to predict the next token in a sequence of text. To predict the …

The allegations of fraud made by the people invading my home and terrorizing my friends? I’ll take those with a grain of salt. And for the record, I’m not afraid of ICE, never said…

I lived my whole life in the twin cities and have a lot of friends, US citizens, who are too scared to go out to eat right now because of the ICE raids. If that wasn’t the point it…

> About the second point, I've been under the impression that because LLMs are trained on average code, they infer that the bugs and architectural flaws are desirable This is reall…

Humans don’t learn to write messy complex code. Messy, complex code is the default, writing clean code takes skill. You’re assuming the LLM produces extra complexity because it’s m…

Image generation and image input are two totally different things. This is about feeding text into LLMs as images, it has nothing to do with image generation.

Here's my version, took about 5 minutes to create inside the ChatGPT web interface. https://valine.github.io/vibe-coded-ant-game/ I don't know if this game was vibe coded, but it c…

Both Claude and GPT5 can single shot this type of game. The score counter looks exactly like the type of thing Claude spits out.

Same. There are really only two features I care about in a phone: a high refresh rates and weight. At 165 grams the iPhone air is by far the lightest 120hz phone apple has ever mad…

Embedding models are not always trained with the rest of the model. That’s the whole idea behind VLLMs. First layer embeddings are so interchangeable you can literally feed in the …

>> For each LLM, we extract static, token-level embeddings from its input embedding layer (the ‘E‘matrix). This choice aligns our analysis with the context-free nature of stimuli t…

So instead of next token prediction its next event prediction. At some point this just loops around and we're back to teaching models to predict the next token in the sequence.

You’re thinking about this like the final layer of the model is all that exists. It’s highly likely reasoning is happening at a lower layer, in a different latent space that can’t …

My personal theory is that it’s an emergent property of many attention heads working together. If each attention head is a bird, reasoning would be the movement of the flock.

That’s true yeah. The model can do that because calculating latents is independent of next token prediction. You do a forward pass for each token in your sequence without the final…

Attention computes a weighted average of all previous latents. So yes, it’s a new token as input to the forward pass, but after it feeds through an attention head it contains a lit…

The dimensionality I suppose depends on the vocab size and your hidden dimension size, but that’s not really relevant. It’s a single linear projection to go from latents to logits.…

The lower dimensional logits are discarded, the original high dimensional latents are not. But yeah, the LLM doesn’t even know the sampler exists. I used the last layer as an examp…

I think it’s helpful to remember that language models are not producing tokens, they are producing a distribution of possible next tokens. Just because your sampler picks a sequenc…

The rendering is done with OpenGL, and for remote viewing I just render to an offscreen framebuffer and stream it back to the client with WebRTC. The code for that isn’t public yet…

Thanks, I will do a deep writeup on that at some point.