Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stable Code 3B: Coding on the Edge

stability.ai

Post reply on HN
Print view View original

141–148 of 148 posts

Re: Stable Code 3B: Coding on the Edge

#141

Post by emadm »

refulgentis wrote: post #135
DreamGen wrote: post #115

Earlier quoted context omitted.

> Stable LM zephyr is the best 3b chat model By what measure? Phi 2 seems better as far as I can tell from benchmarks and usage and has much more permissive license.

Setting aside I've tried both, we'll bore each other to death if we just assert one is better: From first principles, Phi 2 is extremely unlikely to be better, it's a base model and doesn't know how to chat. (see README on HF repo and also "Responses by phi-2 are off, it's depressed and insults me for no reason whatsover?", https://huggingface.co/microsoft/phi-2/discussions/61 ) re: Benchmarks, see https://huggingfac…

Yeah, Phi-2 is weird on chat, StableLM beats it on some metrics, Phi-2 does on others but also doesn't really have system integration yet.

The base model of StableLM 3b zephyr is actually under an even more permissive license (we didn't change in retrospect) and is the best base to train on for MacBooks with 8gb RAM, edge devices etc.

With LLM Farm quantised you can run it faster than you can read on a iPhone or whatever.

https://huggingface.co/stabilityai/stablelm-3b-4e1t

It's also one of the only models with fully dataset, training and other transparency: https://stability.wandb.io/stability-llm/stable-lm/reports/S...

Re: Stable Code 3B: Coding on the Edge

#142

Post by emadm »

rsynnott wrote: post #127
emadm wrote: post #77

Earlier quoted context omitted.

Business model is bundling so you have a one stop shop for good quality models of every modality and cultural variants of them. These go on bedrock, on chip, on prem etc and our consulting partners take them to the end user. On the innovation side stable diffusion turbo does like 100 cats with hats per second and the video model outperforms runway, pika etc on blind tests. Stable audio was one of the time innovation…

> On the innovation side stable diffusion turbo does like 100 cats with hats per second 2028: Energy use on hat-cat generation exceeds energy use on bitcoin.

There are not enough cats on the internet. We are working to fix this.

Re: Stable Code 3B: Coding on the Edge

#145

Post by yreg »

londons_explore wrote: post #19
lfkdev wrote: post #6

How is this compared to the current GitHub Copilot?

If you just want to get stuff done, use the best tools like a Milwaukee Drill - and right now, thats copilot/gpt-4. If you don't want to be tied to a company and like opensource, feel free to connect a toy motor to an AA battery to drill your holes... Or to use Llama/Stable Code 3B.

A self-hosted solution is a common requirement for security reasons.

Re: Stable Code 3B: Coding on the Edge

#146

Post by knicholes »

knicholes wrote: post #137
mlboss wrote: post #89

Earlier quoted context omitted.

Did you build a machine with 4x 3090 ? I looking for a way to build such a machine for ML training.

I did! I started by going to vast.ai. I was able to look at the specs of the top-scoring machines. I started with the motherboard (as I knew it could support my 3090s, because some PCIe busses can't handle all that data). Then of course I copied everything else that I could. I ended up using PCIe extenders and zip-tieing (plastic, I should use metal zip ties instead) the cards to a rack I got from Lowes. I'm not too…

Here are the parts I can find CPU: https://www.newegg.com/amd-epyc-7252-socket-sp3/p/N82E168191... Motherboard: https://www.newegg.com/asrock-rack-romed8-2t/p/N82E168131400... SSD: https://www.newegg.com/samsung-2tb-980-pro/p/N82E16820147796... Computer Power Supply (the GPUs have their own power supplies, run apart from the computer, which I'm told is bad, but... seems to work for years now for me): https://www.newegg.com/corsair-hx-series-hx1200-cp-9020140-n... PCIe 4.0 x16 Risers: https://www.newegg.com/p/N82E16812987068?Item=N82E1681298706... Tower: https://www.newegg.com/black-fractal-design-meshify-2-compac... RAM: https://www.newegg.com/nemix-ram-16gb-288-pin-ddr4-sdram/p/1...

Total, without GPUs and their power supplies: $2900

Re: Stable Code 3B: Coding on the Edge

#147

Post by a_wild_dandan »

unshavedyak wrote: post #93
a_wild_dandan wrote: post #44

Earlier quoted context omitted.

This is phenomenal. And runs fast! The 33b version might be my MacBook's new coding daily driver.

How are you using it? I need to find some sane way to use this stuff from Helix/terminal..

There are many workflows, with hardware-dependent requirements. Three which work for my MacBook:

1. Clone & make llama.cpp. It's a CLI program that runs models, e.g. `./main -m -p `.

2. Another CLI option is `ollama`, which I believe can download/cache models for you.

3. A GUI like LM Studio provides a wonderful interface for configuring, and interacting with, your models. LM Studio also provides a model catalog for you to pick from.

Assuming that your hardware is sufficient, options 1 & 2 should satisfy your terminal needs. Option 3 is an excellent playground for trying new models/configurations/etc.

Models are heavy. To fit one in your silicon and run it quickly, you'll want to use a quantized model. It's a model's "distilled" version -- say 80% smaller for a 0.1% accuracy loss. TheBloke on HuggingFace is one specialist in distilling. After finding a model you like, you can download some flavor of quantization he made, e.g: `huggingface-cli download TheBloke/neural-chat-7B-v3-3-GGUF neural-chat-7b-v3-3.Q4_K_M.gguf --local-dir .`; then use your favorite model runner (e.g. llama.cpp) to run it.

Hope that gets you started. Cheers!

Re: Stable Code 3B: Coding on the Edge

#148

Post by mlboss »

knicholes wrote: post #146
knicholes wrote: post #137

Earlier quoted context omitted.

I did! I started by going to vast.ai. I was able to look at the specs of the top-scoring machines. I started with the motherboard (as I knew it could support my 3090s, because some PCIe busses can't handle all that data). Then of course I copied everything else that I could. I ended up using PCIe extenders and zip-tieing (plastic, I should use metal zip ties instead) the cards to a rack I got from Lowes. I'm not too…

Here are the parts I can find CPU: https://www.newegg.com/amd-epyc-7252-socket-sp3/p/N82E168191... Motherboard: https://www.newegg.com/asrock-rack-romed8-2t/p/N82E168131400... SSD: https://www.newegg.com/samsung-2tb-980-pro/p/N82E16820147796... Computer Power Supply (the GPUs have their own power supplies, run apart from the computer, which I'm told is bad, but... seems to work for years now for me): https://www.newe…

Thanks man! Super helpful.
Post reply on HN