Embarrassingly simple self-distillation improves code generation
1–10 of 227 posts
Re: Embarrassingly simple self-distillation improves code generation
#2> simple self-distillation (SSD):
Sorry apple, SSD is already taken, you can't use that acronym.
Re: Embarrassingly simple self-distillation improves code generation
#3[deleted]
Re: Embarrassingly simple self-distillation improves code generation
#4> simple self-distillation (SSD): Sorry apple, SSD is already taken, you can't use that acronym.
ATT=All TLAs are Taken
Re: Embarrassingly simple self-distillation improves code generation
#5[flagged]
Re: Embarrassingly simple self-distillation improves code generation
#6Haven't read the paper yet, but it is interesting how seemingly simple many breakthroughs in ML are. Even transformers are like that. Maybe it's hindsight bias.
I suppose we just don't have a deeper underlying theory to lean on and help us 'design' anything.
Re: Embarrassingly simple self-distillation improves code generation
#7[flagged]
That's... almost every AI paper.
Re: Embarrassingly simple self-distillation improves code generation
#8> simple self-distillation (SSD): Sorry apple, SSD is already taken, you can't use that acronym.
You're right, I offer these alternatives:
Consistency Preservation Update (CPU)
Guided Probability Update (GPU)
History-aware Distillation Driving (HDD)
Probability Smoothing Update (PSU)
Re: Embarrassingly simple self-distillation improves code generation
#9Incredible, will translate to better coding models in the near future.
We really need to develop better tools to understand what's happening inside these NNs. Working with high-D spaces is not something we're good at, and we're basically throwing stuff at it and seeing if it sticks.
Re: Embarrassingly simple self-distillation improves code generation
#10It's cringe worthy to see that the original paper itself is editorialised.
Title should be: Simple Self-Distillation Improves Code Generation