Viewing profile — lappa

ChatGPT, request minimal necessary diff to make a specific change, review, ctrl+c, ctrl+v

Good post. I think you mixed torch.eye with torch.full though

This isn't suggesting no one understands how these models are architected, nor is anyone saying that SDPA / matrix multiplication isn't understood by those who create these systems…

I use the SingleFile extension to archive every page I visit. It's easy to set up, but be warned, it takes up a lot of disk space. $ du -h ~/archive/webpages 1.1T /home/andrew/arch…

OpenAI clearly states that they train on your data https://help.openai.com/en/articles/5722486-how-your-data-is...

It's easy to argue that Llama-3.3 8B performs better than GPT-3.5. Compare their benchmarks, and try the two side-by-side. Phi-4 is yet another step towards a small, open, GPT-4 le…

Great project, looking forward to seeing more as this develops. Also FYI, your mail server seems to be down.

Provided a constant temperature of 1.0, you can train the model on prompts with probablistic requests, with loss determined by KL divergence. Expectation: 80% left, 20% right Model…

Look at it from an algorithmic perspective. In computer science many algorithms take a non-constant number of steps to execute. However, in transformers models, there are a limited…

Very interested in the expansion of RL for transformers, but I can't quite tell what this project is. Could you please add links to the documentation to the readme where it states …

A few days ago I saw a post using NeuralFlow to help explain the repetition problem. https://old.reddit.com/r/LocalLLaMA/comments/1ap8mxh/what_ca... > I’ve done some investigation …

Not a material science expert, however per their paper, they use DFT to verify the stability, then use the verification status to improve the model. >candidate structures filtered …

Paper: https://www.nature.com/articles/s41586-023-06735-9

Excellent, appears Amazon has introduced two important things here: - Rope theta of 100,000, likely from the Llama 2 Long paper which found that a large theta helped regulate atten…

More data, more parameters, more compute all result in a better model per "Scaling Laws for Neural Language Models" https://browse.arxiv.org/pdf/2001.08361v1.pdf Largeness is a val…

From the paper https://github.com/EleutherAI/math-lm https://huggingface.co/EleutherAI/llemma_34b https://huggingface.co/EleutherAI/llemma_7b

Great work, lots of useful information here. The only thing I wish you did different was explored alpha > 2 * r. In this blog post, the author found that alpha of 4 * r (where r=64…

"For example, HyperAttention makes the inference time of ChatGLM2 50% faster on 32k context length while perplexity increases from 5.6 to 6.3." "when half of all attention layers a…

This seems half-baked and there are numerous faulty assumptions in this article. For example, Bitcoin miners cannot computer gradients. Their ASICs can only calculate double-sha256…

Interesting how this method quantizes different layers / modules in a manner that minimizes perplexity as it adjusts parameters. I'd be interested to see how 2.5 bit quantization c…