what is the big deal with obsidian ? I see a lot of people use it but I'm more than happy with giving an LLM a local sqlite table , embedding api and asking the agent to maintain its own memory
Embarrassingly simple self-distillation improves code generation
181–190 of 227 posts
Re: Embarrassingly simple self-distillation improves code generation
#182One sentence summary: We fine-tuned a general-purpose model to produce valid benchmark code results and it got better at producing benchmark code results; we didn't bother to evaluate it on anything the model used to be good at.
Not really? If you read it, there is no validation, no correctness signal, no verification, none of that. They're just passing in benchmark inputs, collecting the outputs (regardless of their quality), training on those outputs, and then sweeping the decode settings (temp, topk) of the resulting model. Their conclusion is that this results in a better model than the original - even when taking into consideration the…
Not only that, they additionally ran an experiment with the training temperature turned way up (2.0) and truncation turned off such that the majority of SFT examples were incoherent (63% IIRC). Yet the model finetuned on these broken examples still improved over baseline.
Re: Embarrassingly simple self-distillation improves code generation
#183It seems that self-distillation is the way to go for LLM.
Self-distillation has been shown recently as very efficient and effective back in January this year by MIT and ETH team in their Self-Distillation Fine-Tuning (SDFT) LLM system [1],[2].
This paper is also their closest competitor named On-Policy Self-Distillation in the comparison table.
I hope they keep the original work real name that is Self-Distillation Fine-Tuning or SDFT. Imagine later paper citing this very paper as cross-entropy self-distillation instead of their very own given name Simple Self-Distillation or SSD. Although I'd have admitted it's a lousy name that breaks the namespace with common SSD nomenclature for solid-dtate drive, as others have rightly pointed.
I think they should given the proper credit to this earlier seminal earlier on SDFT but apparently they just put it as one as of the systems in their benchmark but not explaining much of the connection and lineage which is a big thing in research publication.
[1] Self-Distillation Enables Continual Learning:
https://arxiv.org/abs/2601.19897
[2] Self-Distillation Enables Continual Learning:
Re: Embarrassingly simple self-distillation improves code generation
#184[flagged]
Re: Embarrassingly simple self-distillation improves code generation
#185most codebases dont have traces to train on. if you use rlm-workflow you will build up rich traceability in the form of requirements, plans, implementation artifacts, along with worktree diffs. with these, you can then use self-distillation on models or use autoagent to improve your harness. https://github.com/doubleuuser/rlm-workflow
Re: Embarrassingly simple self-distillation improves code generation
#186Earlier quoted context omitted.
Doing a tool call for autocomplete is not going to make coding agents faster. I do think there is some merit in a tool that dumps all namespaces and reachable symbols so the agent can do its own autocomplete without a round-trip.
Doesn’t need to be a tool call. As a human coder you don’t summon intellisense. It’s just popped up into your visual field as extra input - contextual cues. You could force intellisense state into the context vector the LLM receives.
Not really, because the LLM loop doesn't have the ability to get updates from the agent live. It would have to somehow be integrated all the way down the stack.
Re: Embarrassingly simple self-distillation improves code generation
#187[dead]
Re: Embarrassingly simple self-distillation improves code generation
#188It's interesting that LLMs improve skills, especially on harder problems, just by practicing them. That's effectively what's going on.
Re: Embarrassingly simple self-distillation improves code generation
#189Earlier quoted context omitted.
Doesn’t need to be a tool call. As a human coder you don’t summon intellisense. It’s just popped up into your visual field as extra input - contextual cues. You could force intellisense state into the context vector the LLM receives.
Not really, because the LLM loop doesn't have the ability to get updates from the agent live. It would have to somehow be integrated all the way down the stack.
LLMs can have whatever abilities we build for them. The fact we currently start their context out with a static prompt which we keep feeding in on every iteration of the token prediction loop is a choice. We don’t have to keep doing that if there are other options available.
Re: Embarrassingly simple self-distillation improves code generation
#190Earlier quoted context omitted.
Why would this be surprising? That’s exactly how much of the code they were trained on is presented in PRs, Forums, etc.
Is that true? That depends on how their web scraping works, like whether it runs client-side highlighting, strips out HTML tags, etc.
The highlighting isn't what matters, its the pretext. E.g. An LLM seeing "```python" before a code block is going to better recall python codeblocks by people that prefixed them that way.