Viewing profile — atgctg

https://platform.openai.com/docs/guides/latest-model System card: https://cdn.openai.com/pdf/3a4153c8-c748-4b71-8e31-aecbde944...

The paper's Table 7 shows DyT reducing overall LLaMA 7B inference time by 7.8% and training time by 8.2%. That is not insignificant.

You can get a free trial right now to Stratechery Plus through Asianometry: https://stratechery.passport.online/member/plan/4ycW4SE71Cy6... Source: https://substack.com/home/post/p…

You have to store the KV cache, not the tokens. For Gemma 27B (probably slightly larger than Flash), this would be: Size of KV cache = 2 * (num_layers) * (num_kv_heads * dim_head) …

Works using math CSS injection [1]: ```math \ce{$\unicode[goombafont; color:red; pointer-events: none; z-index: -10; position: fixed; top: 0; left: 0; height: 100vh; object-fit: co…

Tiktoken added support for GPT-4o: https://github.com/openai/tiktoken/commit/9d01e5670ff50eb74c... It has an increased vocab size of 200k.

Seems like they are working on adding that capability: > We're exploring whether we can responsibly provide the ability to generate NSFW content in age-appropriate contexts through…

A lot of transformer explanations fail to mention what makes self attention so powerful. Unlike traditional neural networks with fixed weights, self-attention layers adaptively wei…

It would be interesting to have attention visualized as well, similar to how it's done in BertViz: https://github.com/jessevig/bertviz

As an example, INT8 support in WebGPU would enable running quantized models, allowing larger LLMs to run locally in the browser. See Limitations section here: https://fleetwood.dev…

Running at ~40 fps on M2 Max: https://www.reddit.com/r/macgaming/comments/1435ukq/cyberpun...

That video is about analog computers

It's hard to distinguish non-Google projects with Google Sans in their templates from actual Google Research papers, as the font is meant to be exclusively used by Google[1]. [1] h…

Delightful intro, turns out it's written by the master storyteller, Ted Chiang.