SantaCoder: A new 1.1B code model for generation and infilling

moyix wrote: post #2

Despite being only 1.1B params, SantaCoder outperforms Facebook's InCoder (6.7B params) and Salesforce's CodeGen-Multi-2.7B. Paper: https://hf.co/datasets/bigcode/admin/resolve/main/BigCode_Sa... Dataset search: https://huggingface.co/spaces/bigcode/santacoder-search Model weights: https://huggingface.co/bigcode/santacoder

Any idea how this, and those other two models, would compare to GitHub Copilot?

Re: SantaCoder: A new 1.1B code model for generation and infilling

#5

Post by seinecle » Thu, Dec 22, 2022, 8:21 PM UTC

snowder wrote: post #3

I am having trouble getting the demo to run. It just errors out

Same here

Re: SantaCoder: A new 1.1B code model for generation and infilling

#6

Post by rytill » Thu, Dec 22, 2022, 8:46 PM UTC

Are there model weights?

Re: SantaCoder: A new 1.1B code model for generation and infilling

#7

Post by moyix » Thu, Dec 22, 2022, 8:46 PM UTC

isoprophlex wrote: post #4

moyix wrote: post #2

Despite being only 1.1B params, SantaCoder outperforms Facebook's InCoder (6.7B params) and Salesforce's CodeGen-Multi-2.7B. Paper: https://hf.co/datasets/bigcode/admin/resolve/main/BigCode_Sa... Dataset search: https://huggingface.co/spaces/bigcode/santacoder-search Model weights: https://huggingface.co/bigcode/santacoder

Any idea how this, and those other two models, would compare to GitHub Copilot?

Based on the reverse engineering done by Parth Thakkar [1], the model used by Copilot is probably about 10x as large (12B parameters), so I would expect Copilot to still win pretty handily (especially since the Codex models are generally a lot better trained than Salesforce CodeGen or InCoder). It's also a little bit hard to compare directly because as Parth documents, there are a lot of extra smarts that go into Copilot on the client side.

The SantaCoder paper does have some benchmarks on MultiPL-E though, so you could compare them to the Codex results on that benchmark reported here (but keep in mind that code-davinci-002 is probably even larger than the model used by Copilot): https://arxiv.org/abs/2208.08227

[1] https://thakkarparth007.github.io/copilot-explorer/posts/cop...

Re: SantaCoder: A new 1.1B code model for generation and infilling

#8

Post by moyix » Thu, Dec 22, 2022, 8:47 PM UTC

rytill wrote: post #6

Are there model weights?

Yep! https://huggingface.co/bigcode/santacoder

Re: SantaCoder: A new 1.1B code model for generation and infilling

#9

Post by moyix » Thu, Dec 22, 2022, 8:48 PM UTC

seinecle wrote: post #5

snowder wrote: post #3

I am having trouble getting the demo to run. It just errors out

Same here

Might be overloaded – if you have a GPU you can try running it locally by getting the model weights here: https://huggingface.co/bigcode/santacoder

Re: SantaCoder: A new 1.1B code model for generation and infilling

#10

Post by bogwog » Thu, Dec 22, 2022, 8:59 PM UTC

moyix wrote: post #9

seinecle wrote: post #5

Earlier quoted context omitted.

Same here

Might be overloaded – if you have a GPU you can try running it locally by getting the model weights here: https://huggingface.co/bigcode/santacoder

Any idea how much GPU memory you'd need to run this locally?

EDIT: just tried it and it didn't seem to go past ~6gb