241–250 of 255 posts
this is awesome. i'm the founder/maintainer at teale.com (open source distributed inference app) and one of the biggest challenges has been an actually usable/reliable local model that can run on 8gb macbook air's, 6gb android smartphones, etc... will get some of our test machines serving needle
This sounds like it could be really powerful, but I don't understand how to use it. I tried 'git add commit and push' which is something I might ask an llm to do but is a waste of tokens. It output: [{"name":"create_note","arguments":{"text":"commit and push","title":"git add"}}] Which is kind of a little way there but not useable on its own
Hmm.. this might make it feasible to build something like a command line program where you can optionally just specify the arguments in natural language. Although I know people will object to including an extra 14 MB and the computation for "parsing" and it could be pretty bad if everyone started doing that. But it's really interesting to me that that may be possible now. You can include a fine-tuned model that under…
Earlier quoted context omitted.
There are two answers already and neither is entirely adequate. In normal LLM training, you take a set of documents and have it learn to predict the future, then have some private RLHF/RLVR etc. data that it learns to produce good chat outputs from. In distillation, you take a set of prompts you are interested in, and record the big LLM's outputs, then train your small model to produce the same output as the big LLM.…
> In distillation, you take a set of prompts you are interested in, and record the big LLM's outputs, then train your small model to produce the same output as the big LLM. Why use the bigger LLM outputs for this and not human outputs? If we assume that human responses to prompts are better than sota models (in some cases they are) then why use the big model at all?
You can set up model distillation as a weekend batch job.
This is neat, and matches an observation I saw with early Claude Code usage: Sonnet would often call tools quickly to gather more context, whereas Opus would spend more time reasoning and trying to solve a problem with the context it had. This led to lots of duplicated functions and slower development, though the new models (GPT-5.5 and Opus 4.6) seem to suffer from this less. My takeaway was that “dumber” (i.e. smal…
The key is to not run LLMs in loops. This trend of agentic frameworks is silly, and mostly exists to make LLM companies more revenue. An LLM is mostly useless but is much more useful and reliable with one shot tooling. I have a suite or tools ive built for myself on top of the openrouter api for very specific tasks. Press button amd LLM does (one) useful thing, not press button and let LLM run tool calls in a loop fo…