Viewing profile — thw20

Good work! This is very interesting. Here's a related work that construct low-rank approximation for attention: https://arxiv.org/abs/2505.12942 . Maybe the idea of Query calibrati…

The up to date paper documenting and analysing the observation is now available on arxiv!

This project reveals an interesting phenomena, where LLM converts semantic non-informative tokens to attention sinks through middle layer MLP. The converted sinks are termed second…

This is so amazing. What a masterpiece for intro to reinforcement learning in llm.