Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I made a transformer to predict a simple sequence manually

vgel.me

Post reply on HN
Print view View original

101–104 of 104 posts

Re: I made a transformer to predict a simple sequence manually

#101

Post by juliangoldsmith »

fransje26 wrote: post #98
thomasahle wrote: post #78

I thought I understood transformers well, even though I had never implemented them. Then one day I implemented them, and they didn't work/train nearly as well as the standard pytorch transformer. I eventually realized that I had ignored the dropout, because I thought my data could never overfit. (I trained the transformer to add numbers, and I never showed it the same pair twice.) Turns out dropout has a much bigger…

Would you have any literature to recommend, to help get some insight in how to approach the task?

Andrej Karpathy has a series on YouTube where he builds everything up from scratch. He implements a transformer when he builds a GPT-style model: https://www.youtube.com/watch?v=kCc8FmEb1nY

Re: I made a transformer to predict a simple sequence manually

#102

Post by thomasahle »

fransje26 wrote: post #98
thomasahle wrote: post #78

I thought I understood transformers well, even though I had never implemented them. Then one day I implemented them, and they didn't work/train nearly as well as the standard pytorch transformer. I eventually realized that I had ignored the dropout, because I thought my data could never overfit. (I trained the transformer to add numbers, and I never showed it the same pair twice.) Turns out dropout has a much bigger…

Would you have any literature to recommend, to help get some insight in how to approach the task?

There are a lot of "simple transformer" implementations on github. But I'll recommend Microsofts new Phi 1.5: https://huggingface.co/microsoft/phi-1_5/blob/main/modeling_... It's well written, and very modern, including rotary embeddings and a kv-cache for inference.

Re: I made a transformer to predict a simple sequence manually

#103

Post by fransje26 »

thomasahle wrote: post #102
fransje26 wrote: post #98

Earlier quoted context omitted.

Would you have any literature to recommend, to help get some insight in how to approach the task?

There are a lot of "simple transformer" implementations on github. But I'll recommend Microsofts new Phi 1.5: https://huggingface.co/microsoft/phi-1_5/blob/main/modeling_... It's well written, and very modern, including rotary embeddings and a kv-cache for inference.

Thank you!

Re: I made a transformer to predict a simple sequence manually

#104

Post by fransje26 »

juliangoldsmith wrote: post #101
fransje26 wrote: post #98

Earlier quoted context omitted.

Would you have any literature to recommend, to help get some insight in how to approach the task?

Andrej Karpathy has a series on YouTube where he builds everything up from scratch. He implements a transformer when he builds a GPT-style model: https://www.youtube.com/watch?v=kCc8FmEb1nY

Thank you for the tip!
Post reply on HN