Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

StableCode

stability.ai

Post reply on HN
Print view View original

21–30 of 109 posts

Re: StableCode

#21

Post by runako »

nwoli wrote: post #16
runako wrote: post #6

Is this a "product" that one could install and use or a model that one should expect an OEM to integrate into a product before programmers can use it? I'm asking because I don't see any links that would help me figure out how to try it out.

Ctrl-F for “Code for using StableCode Instruct to generate a response to a given instruction.” and you’ll see a super straightforward piece of code to copy to test it out for generating code

Thanks! The verbiage at the beginning of the announcement seems to go out of its way to not call StableCode a "model," which was confusing. By contrast, the recent release of SDXL 1.0 is described as a "model" in its announcement.

Re: StableCode

#22

Post by ben_w »

nwoli wrote: post #17
sebzim4500 wrote: post #2

Hard to believe it can work that well when it only has 3B parameters, but I'd love to be proven wrong.

Reminder that GPT-2 was considered “too dangerous” to be released at just 1.5B weights

My memory may be imperfect, but I thought it was more "we aren't sure and we want to promote a culture of safety" rather than "this is definitely unsafe… oh wait never mind"?

Re: StableCode

#23

Post by johndough »

eyegor wrote: post #13
thewataccount wrote: post #8

Earlier quoted context omitted.

I was impressed enough by replit's 2.7B model that I'm convinced it's doable. I have a 4090 and consider that the "max expected card for a consumer to own". Also exllama doesn't support non-llama models and the creator doesn't seem interested in adding support for wizardcoder/etc. Because of this, using the alternatives are prohibitively slow to use a quantized 16B model on a 4090 (if the exllama author reads this _p…

I'm fairly confident a coding specific model should be a lot smaller - 3b should be plenty if not 1b or less. As it stands, there are quite a few 7-13b model sizes that can predict natural language quite well. Code seems at its surface a much simpler language, strict grammars, etc so I wouldn't think it needs to be anywhere near as large as the nlp models. Right now people are retraining nlp models to work with code,…

> Code seems at its surface a much simpler language

When using GitHub Copilot, I often write a brief comment first and most of the time, it is able to complete my code faster than if I had written it myself. For my workflow, a good code model must therefore also be able to understand natural text well.

Although I am not sure to which degree the ability to understand natural text and the ability to generate natural text are related. Perhaps a bit of text generation capabilities can be traded off against faster execution and fewer parameters.

Re: StableCode

#24

Post by arugulum »

ben_w wrote: post #22
nwoli wrote: post #17

Earlier quoted context omitted.

Reminder that GPT-2 was considered “too dangerous” to be released at just 1.5B weights

My memory may be imperfect, but I thought it was more "we aren't sure and we want to promote a culture of safety" rather than "this is definitely unsafe… oh wait never mind"?

It's actually even less remarkable than that. It was an experiment in having a limited release, to shift the field toward a different release convention.

> Nearly a year ago we wrote in the OpenAI Charter: “we expect that safety and security concerns will reduce our traditional publishing in the future, while increasing the importance of sharing safety, policy, and standards research,” and we see this current work as potentially representing the early beginnings of such concerns, which we expect may grow over time.

> This decision, as well as our discussion of it, is an experiment: while we are not sure that it is the right decision today, we believe that the AI community will eventually need to tackle the issue of publication norms in a thoughtful way in certain research areas.

> We will further publicly discuss this strategy in six months.

https://openai.com/research/better-language-models

Re: StableCode

#25

Post by arcanemachiner »

yohannparis wrote: post #9
runako wrote: post #6

Is this a "product" that one could install and use or a model that one should expect an OEM to integrate into a product before programmers can use it? I'm asking because I don't see any links that would help me figure out how to try it out.

To be honest, you’d better buy GitHub co-pilot and enjoy the productivity boost at a cheap price. Trying to download/install/setup/use StableCode is worth it only if you want to learn all those steps as well. If what you care is the final result, just buy an existing service.

I may put all my open source stuff on GitHub, but hell will freeze over before I willingly let Microsoft get a whiff of my private data, no matter how irrelevant it may be.

GitHub Copilot sounds pretty neat though, I will admit that.

Re: StableCode

#27

Post by thewataccount »

nwoli wrote: post #20
thewataccount wrote: post #10

Earlier quoted context omitted.

> Stability AI, Apple, Meta, etc are clearly at the finish line I'm very optimistic and expect them to catch up. I've used the open models a lot, to be clear they are starting to compare to GPT3.5Turbo right now, they can't compete with GPT4 at all. GPT4 is almost a year old from when it finished training I think? I expect open source models to stay ~1.5 years behind. That said they will eventually be "good enough".…

> Keep in mind too though that using and scaling GPUs is not free. You have to run the models somewhere. Long or medium term these will probably be dirt cheap to just run in the background though. It might be within 3-5 years since parallel compute is still growing and isn’t as bounded by moores law stagnation

I get decent performance with my 4090, enough that LLMs with exllama at 30B quantitized are very usable. But we're severely VRAM limited, especially on lower end hardware which rarely sees > 10GB of VRAM.

I don't know how much slower it could be and still be useful though. The big thing is we need more VRAM, 30B is context length limited with only 24GB of vram, I've only barely made it above 3.2k tokens before running out.

I hope you're right, that it becomes common for systems to have either dedicated TPU type stuff similar to smartphones, and that they absolutely load the crap with VRAM (which I don't think is even that expensive?)

Models will also get smaller but I'm skeptical we'll get GPT4 performance with any useful context length under 24GB VRAM any time soon.

Re: StableCode

#28

Post by thewataccount »

gsuuon wrote: post #19
eyegor wrote: post #13

Earlier quoted context omitted.

I'm fairly confident a coding specific model should be a lot smaller - 3b should be plenty if not 1b or less. As it stands, there are quite a few 7-13b model sizes that can predict natural language quite well. Code seems at its surface a much simpler language, strict grammars, etc so I wouldn't think it needs to be anywhere near as large as the nlp models. Right now people are retraining nlp models to work with code,…

I'd really like to see smaller models trained on only one specific language, with it's own language specific tokenizer. I imagine the reduction in vocab size would translate to handling more context easier?

I think simply having the vocab more code friendly (e.g. codex) would make the biggest difference, whitespace is the biggest one (afaik every space is a token), but consider how many languages continue `for(int i=0;`, `) {\n`, `} else {`, 'import ', etc.

My understanding is that a model properly trained on multiple languages will beat an expert based system. I feel like programming languages overlap, and interop with each other enough that I wouldn't want to specialize it in just one language.

Re: StableCode

#29

Post by GeneralMayhem »

johndough wrote: post #23
eyegor wrote: post #13

Earlier quoted context omitted.

I'm fairly confident a coding specific model should be a lot smaller - 3b should be plenty if not 1b or less. As it stands, there are quite a few 7-13b model sizes that can predict natural language quite well. Code seems at its surface a much simpler language, strict grammars, etc so I wouldn't think it needs to be anywhere near as large as the nlp models. Right now people are retraining nlp models to work with code,…

> Code seems at its surface a much simpler language When using GitHub Copilot, I often write a brief comment first and most of the time, it is able to complete my code faster than if I had written it myself. For my workflow, a good code model must therefore also be able to understand natural text well. Although I am not sure to which degree the ability to understand natural text and the ability to generate natural te…

Understanding should be much easier, for the same reason humans (e.g. children, foreign-language learners) can always understand more than they can say: human language is fairly low-entropy, so if there's a word you don't understand, you can pick up most of the meaning from context. On the other hand, producing natural-sounding language requires knowing every single word you're going to use.
Post reply on HN