131–140 of 160 posts
I'm playing around with this in LMStudio (in huggingface -> use this model dropdown -> LMStudio) It's really impressive so far, so quick to respond on a mac mini M2. And it appears to be accurate at least for the obvious questions. I couldn't get it to work as an autocomplete of Zed unfortunately. It looks like it's hardwired to work with some providers and LMStudio is not included in the prediction engines list. Has…
Wow, I can even chat about C code with that model with LM Studio on my Macbook at 200 tokens per seconds
Also that's crazy, M4 Mac?
What type of hardware do I need to run a small model like this? I don't do Apple.
1.5B models can run on CPU inference at around 12 tokens per second if I remember correctly.
I'm very green to this so forgive if this question sounds silly: Would instead of the RL step a constrained decoding say via something like xgrammar fix syntax generation issue ?
> Would instead of the RL step a constrained decoding say via something like xgrammar fix syntax generation issue ? It can, but you have to consider two things here: a) constrained decoding ensures adherence to syntax, not semantics. Say you're editing a field in an enum in rust. You can write syntactically correct rust code that doesn't address the new field further in the code (say in a switch). You'd get correctly…
The other one is that constrained decoding only works on CFGs (simpler grammars like JSON schemas) since only these ones can produce automatas which can be used for constrained decoding. Programming languages like Python and C++ aren't CFGs so it doesn't work.
Also constrained decoding generally worsens model quality since the model would be generating off-policy. So RL helps push corrected syntax back on-policy.
I've been trying my hands at implementing an emacs package for inline completions with this. I have it mostly working and performance is good enough but I haven't been blown away by the quality of its suggestions unfortunately. Which I guess is expected from a 1.5B model. I'd love to see them making a larger model in the 10-20b range maybe? I know most people wouldn't be able to run that on their machines, but some c…
This seems great for code, but can this be used for non-code use cases?
Do you plan to release Sweep 3B/7B on HF?
Based on qwen2.5-coder? seems like a "why not/resume embellish/show VC" type release I guess
[flagged]
> On the infra side, training a 1.5B model in ~4 hours on 8×H100 is impressive. It's hard to compare without more details about the training process and the dataset, but, is it? Genuine question, because I had the opposite impression. Like, for example, recently I did a full finetuning run on a 3B model chewing through a 146k entry dataset (with 116k entries having reasoning traces, so they're not short) in 7 hours o…
is there any llm lsp it can integrate well with?