Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SantaCoder: A new 1.1B code model for generation and infilling

huggingface.co

Post reply on HN
Print view View original

51–60 of 76 posts

Re: SantaCoder: A new 1.1B code model for generation and infilling

#51

Post by Jap2-0 »

Jap2-0 wrote: post #50
zomglings wrote: post #22

Looks like Santa's going to be too busy to deliver presents this Christmas: def all_prime_elements(sequence): """Returns every prime element of the sequence.""" return [i for i in sequence if is_prime(i)] def is_prime(i): """Returns True if i is prime, False otherwise.""" if i == 2: return True if i == 3: return True if i == 5: return True if i == 7: return True if i == 11: return True if i == 13: return True if i ==…

Mine never defines is_prime(): def all_prime_elements(sequence): """Returns every prime element of the sequence.""" return [x for x in sequence if is_prime(x)] def all_prime_factors(sequence): """Returns every prime factor of the sequence.""" return [x for x in sequence if is_prime(x) and x!= 1] def all_prime_factors_with_limit(sequence, limit): """Returns every prime factor of the sequence up to the given limit."""…

If I prompt it, it actually comes up with a decent function:

  def is_prime(element):
    """Returns whether a number is prime."""
    if element
Of course, if you ask it to keep generating past that there's the usual slow descent into madness.

Re: SantaCoder: A new 1.1B code model for generation and infilling

#52

Post by stevenhuang »

wokwokwok wrote: post #45
bogdanoff_2 wrote: post #36

Are there any models that edit or generate file trees?

Like, an entire repository of files? No. How would a model get trained on that? You'd have to pass in the entire repository for each sample. It's prohibitively difficult to create that sort of model. If you want that, you'll have to build tooling on top of a text model (ie. an application that calls a model repeatedly), that takes a prompt and breaks it up into per-file prompts, then incrementally generates the files…

> so generally speaking 'generate an entirely application' one-step models currently don't exist.

This already exists. People have created full apps in ChatGPT by doing this. Many examples online.

Re: SantaCoder: A new 1.1B code model for generation and infilling

#53

Post by ilaksh »

morgante wrote: post #38
ilaksh wrote: post #29

Earlier quoted context omitted.

Have you been able to get your rate limit increased for code-davinci-002? It defaults to a very small amount.

I recommend trying davinci-003 as well.

Yeah, been messing with that a lot.

Re: SantaCoder: A new 1.1B code model for generation and infilling

#54

Post by throwaway0x7E6 »

zomglings wrote: post #22

Looks like Santa's going to be too busy to deliver presents this Christmas: def all_prime_elements(sequence): """Returns every prime element of the sequence.""" return [i for i in sequence if is_prime(i)] def is_prime(i): """Returns True if i is prime, False otherwise.""" if i == 2: return True if i == 3: return True if i == 5: return True if i == 7: return True if i == 11: return True if i == 13: return True if i ==…

I see Yandere Simulator got into the training dataset

Re: SantaCoder: A new 1.1B code model for generation and infilling

#57

Post by moyix »

armchairhacker wrote: post #43

If you haven't noticed, bigcode has also released "The Stack", a 3TB (!) dataset of code ( https://huggingface.co/datasets/bigcode/the-stack ). Also, they have a special policy where "The Stack" only contains permissively-licensed code, and anyone can see if their data is included and opt-out. It's true they haven't actually trained a model on the stack, and this is...not copilot. But I like what they're doing and I…

> It's true they haven't actually trained a model on the stack

What do you mean? SantaCoder is trained on The Stack:

> Dataset

> The base training dataset for the experiments in this paper contains 268 GB of Python, Java and JavaScript files from The Stack v1.1 (Kocetkov et al., 2022) after removing data from opt-out requests, near-deduplication, PII-redaction (see Section 4), and filtering based on line-length and percentage of alphanumeric characters. This dataset was also decontaminated by removing files that contained test-samples from the following benchmarks: HumanEval (Chen et al., 2021), APPS (Hendrycks et al., 2021), MBPP (Austin et al., 2021) and MultiPL-E (Cassano et al., 2022).

It's definitely not on par with Copilot yet, but SantaCoder is a trial run for a larger & better model that they're planning to train in 2023. Stay tuned! :)

Re: SantaCoder: A new 1.1B code model for generation and infilling

#58

Post by moyix »

Vetch wrote: post #44
moyix wrote: post #2

Despite being only 1.1B params, SantaCoder outperforms Facebook's InCoder (6.7B params) and Salesforce's CodeGen-Multi-2.7B. Paper: https://hf.co/datasets/bigcode/admin/resolve/main/BigCode_Sa... Dataset search: https://huggingface.co/spaces/bigcode/santacoder-search Model weights: https://huggingface.co/bigcode/santacoder

SantaCoder's impressive but that's probably misleading. It's reported that incoder doesn't generate as diverse a set of solutions but does do better at the ones it generates. This means it performs well at a lower number of tries when compared to other similar models, which is what matters in practice. The numbers reported here required many trials. With a fuller context and just a handful of tries, it's unlikely tha…

The amount of context is dictated by the benchmark, but I agree it would be good to see what the pass@1 and pass@10 numbers are – if the raw data is available somewhere that can easily be computed.

Re: SantaCoder: A new 1.1B code model for generation and infilling

#59

Post by moyix »

obastani wrote: post #49
moyix wrote: post #7

Earlier quoted context omitted.

Based on the reverse engineering done by Parth Thakkar [1], the model used by Copilot is probably about 10x as large (12B parameters), so I would expect Copilot to still win pretty handily (especially since the Codex models are generally a lot better trained than Salesforce CodeGen or InCoder). It's also a little bit hard to compare directly because as Parth documents, there are a lot of extra smarts that go into Cop…

Just out of curiosity, in what sense is Codex is better trained than CodeGen?

OpenAI hasn't said exactly how they trained code-davinci-002 so this is speculative, but I'm reasonably sure it was trained on more data and languages than CodeGen and for longer. It was also trained using fill-in-the middle [1].

[1] https://arxiv.org/abs/2207.14255

Re: SantaCoder: A new 1.1B code model for generation and infilling

#60

Post by arcticfox »

teaearlgraycold wrote: post #56
ipnon wrote: post #55

I think we’re a good 10 years away from “read my codebase and add the feature my PM asked for.”

We’ll become diff revisers

ChatGPT is great at analyzing diffs so… I’m gonna be reviewing the reviews
Post reply on HN