Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

StableCode

stability.ai

Post reply on HN
Print view View original

41–50 of 109 posts

Re: StableCode

#41

Post by coder543 »

lolinder wrote: post #35
miohtama wrote: post #31

Earlier quoted context omitted.

The model, source, etc. are available under permissive terms https://huggingface.co/stabilityai/stablecode-instruct-alpha... You can “run it locally”. Very handy if you do not trust automatically sending all your code to someone in the United States.

> to reproduce, distribute, and create derivative works of the Software Products solely for your non-commercial research purposes I wouldn't call these terms permissive. It's in line with the recent trend in released AI models, but fairly restrictive in what you're actually allowed to do with it.

The Completion model appears to place the model weights under the Apache 2 license, which is a permissive license: https://huggingface.co/stabilityai/stablecode-completion-alp...

The Instruct model has that non-commercial restriction, but I'm not sure why. They say it was trained with Alpaca-formatted questions and responses, but I'm not sure if that includes the original Alpaca dataset.

Re: StableCode

#42

Post by maaaaattttt »

sebzim4500 wrote: post #37
jstummbillig wrote: post #30

Earlier quoted context omitted.

When they don't voluntarily answer the question, you know the answer.

It's not easy to compare them, to be fair. I guess you could come up with a thousand example prompts and pay some students to pick which output is better, but I can also see why you wouldn't bother. It probably depends on language, type of prompt, etc.

One could team up with Hackerrank/leetcode, let the model code in the interface (maybe there's an API for that already, no idea), execute their code verbatim and see how many test cases they get right the first time around. Then, like for humans, give them a clue about one of the tests no passing (or code not working, too slow, etc.). Give points based on the difficulty of the question and the number of clues needed.

I guess the obvious caveat is that these model are probably overfitted on these types of questions. But a specific benchmark could be made containing question kept secret for models. Time to build "Botrank" I guess.

Re: StableCode

#43

Post by hmottestad »

yohannparis wrote: post #9
runako wrote: post #6

Is this a "product" that one could install and use or a model that one should expect an OEM to integrate into a product before programmers can use it? I'm asking because I don't see any links that would help me figure out how to try it out.

To be honest, you’d better buy GitHub co-pilot and enjoy the productivity boost at a cheap price. Trying to download/install/setup/use StableCode is worth it only if you want to learn all those steps as well. If what you care is the final result, just buy an existing service.

I have bought into co-pilot, but I can’t say it’s that much of a productivity boost. More often than not it recommends something completely wrong. I guess it might be more useful if I did more spring boot or maybe hibernate.

I’ve found chat gpt to be more helpful in general. I can paste some code in and have a discussion about what I want it to fix for me.

Re: StableCode

#44

Post by karmasimida »

3rd3 wrote: post #26

How does it compare to GitHub Copilot?

On HumanEval, Copilot is 40+ on pass@1 comparing to 26 for stable code 3b.

HumanEval is abused but this model is only good for its size, it is no match for Copilot … yet

Re: StableCode

#45

Post by UncleOxidant »

karmasimida wrote: post #44
3rd3 wrote: post #26

How does it compare to GitHub Copilot?

On HumanEval, Copilot is 40+ on pass@1 comparing to 26 for stable code 3b. HumanEval is abused but this model is only good for its size, it is no match for Copilot … yet

> On HumanEval, Copilot is 40+ on pass@1 comparing to 26 for stable code 3b.

Can you put those numbers into context for those who haven't done HumanEval? Are those percentages so that 40+ means 40+% and 26 is 26%? If so does that imply both would be failing scores?

Re: StableCode

#46

Post by RomanPushkin »

Is it good at algos?

From interviews:

Implement queue that supports three methods:

* push

* pop

* peek(i)

peek returns element by its index. All three methods should have O(1) complexity [write code in Ruby].

ChatGPT wasn't able to solve that last time I tried https://twitter.com/romanpushkin/status/1617037136364199938

Re: StableCode

#47

Post by barrotes »

capableweb wrote: post #38
cutler wrote: post #34

Yet another site whose data privacy policy amounts to nothing more than an Accept button. Refuse to use such sites.

It's a model you download and run yourself, on your own hardware. No privacy policy needed.

He's referring to the actual website, which doesn't give you the option of reject profilation cookies (mandatory in Europe). I commented about another website posted here few days ago. It gets me mad too

Re: StableCode

#48

Post by UncleOxidant »

miohtama wrote: post #31
3rd3 wrote: post #26

How does it compare to GitHub Copilot?

The model, source, etc. are available under permissive terms https://huggingface.co/stabilityai/stablecode-instruct-alpha... You can “run it locally”. Very handy if you do not trust automatically sending all your code to someone in the United States.

Hmmm... so on that hugging face page there's a text box where you enter input then you click the 'compute' button.

So I asked it to "Write a python function that computes the square of the input number."

And it responds with:

     def square(x):
Which seems quite underwhelming.

Re: StableCode

#49

Post by smcleod »

cutler wrote: post #34

Yet another site whose data privacy policy amounts to nothing more than an Accept button. Refuse to use such sites.

Use uBlock Origin and then you won't have to see them ;)

Re: StableCode

#50

Post by voxl »

RomanPushkin wrote: post #46

Is it good at algos? From interviews: Implement queue that supports three methods: * push * pop * peek(i) peek returns element by its index. All three methods should have O(1) complexity [write code in Ruby]. ChatGPT wasn't able to solve that last time I tried https://twitter.com/romanpushkin/status/1617037136364199938

in what world is a hashtable lookup worst case O(1)? Your own solution doesn't match your requirements.

If you want amortized complexity then a simple vector suffices.

Post reply on HN