SantaCoder: A new 1.1B code model for generation and infilling • Hacker News Board

Re: SantaCoder: A new 1.1B code model for generation and infilling

#41

Post by yazzku » Fri, Dec 23, 2022, 1:22 AM UTC

vsskanth wrote: post #39

Didn't work for me - def closest_point_on_curve(target,curve): """Returns the 3D point on the curve closest to target 3D point""" Response: - return closest_point_on_curve_3d(target,curve)

For the following example it just goes on generating a never-end sequence of calls:

def point_line_projection(line,point): """Returns the perpendicular projection of the point on the line.""" return line.point_projection(point)

def line_intersection(line1,line2): """Returns the intersection point of two lines.""" return line1.intersection(line2)

def line_intersection_point(line1,line2): """Returns the intersection point of two lines.""" return line1.intersection_point(line2)

def line_intersection_point_line(line1,line2): """Returns the intersection point of two lines.""" return line1.intersection_point_line(line2)

def line_intersection_point_line_parallel(line1,line2): """Returns the intersection point of two lines.""" return line1.intersection_point_line_parallel(line2)

def line_intersection_point_line_parallel_point(line1,line2,point): """Returns the intersection point of two lines.""" return line1.intersection_point_line_parallel

...

Re: SantaCoder: A new 1.1B code model for generation and infilling

#42

Post by varunkmohan » Fri, Dec 23, 2022, 1:26 AM UTC

There are a bunch of really good ideas used to train this model - multi query attention, infilling, near deduplication and dataset cleaning.

I do wish that the demo was a little more interactive (not needing to click buttons to create a generation) since it makes it hard to see the full power of the model.

One of the things we tried at Codeium for our playground on browser was to make it super clear how well the model performs by making the experience interactive - https://www.codeium.com/playground

Re: SantaCoder: A new 1.1B code model for generation and infilling

#43

Post by armchairhacker » Fri, Dec 23, 2022, 1:43 AM UTC

If you haven't noticed, bigcode has also released "The Stack", a 3TB (!) dataset of code (https://huggingface.co/datasets/bigcode/the-stack). Also, they have a special policy where "The Stack" only contains permissively-licensed code, and anyone can see if their data is included and opt-out.

It's true they haven't actually trained a model on the stack, and this is...not copilot. But I like what they're doing and I think it should be appreciated. Honestly, I may even say they're doing with code what stability.ai is doing with images.

Re: SantaCoder: A new 1.1B code model for generation and infilling

#44

Post by Vetch » Fri, Dec 23, 2022, 1:46 AM UTC

moyix wrote: post #2

Despite being only 1.1B params, SantaCoder outperforms Facebook's InCoder (6.7B params) and Salesforce's CodeGen-Multi-2.7B. Paper: https://hf.co/datasets/bigcode/admin/resolve/main/BigCode_Sa... Dataset search: https://huggingface.co/spaces/bigcode/santacoder-search Model weights: https://huggingface.co/bigcode/santacoder

SantaCoder's impressive but that's probably misleading. It's reported that incoder doesn't generate as diverse a set of solutions but does do better at the ones it generates. This means it performs well at a lower number of tries when compared to other similar models, which is what matters in practice. The numbers reported here required many trials.

With a fuller context and just a handful of tries, it's unlikely that 6.7B version of incoder will be outperformed by SantaCoder.

Re: SantaCoder: A new 1.1B code model for generation and infilling

#45

Post by wokwokwok » Fri, Dec 23, 2022, 1:54 AM UTC

bogdanoff_2 wrote: post #36

Are there any models that edit or generate file trees?

Like, an entire repository of files?

No.

How would a model get trained on that? You'd have to pass in the entire repository for each sample. It's prohibitively difficult to create that sort of model.

If you want that, you'll have to build tooling on top of a text model (ie. an application that calls a model repeatedly), that takes a prompt and breaks it up into per-file prompts, then incrementally generates the files passing the context of previous files, and the 'context' would be too large, so you'd get large scale consistency errors.

Broadly speaking the number of tokens = the size of the text it can generate.

With small models, the number is trivial (code fragment), so generally speaking 'generate an entirely application' one-step models currently don't exist.

That said, stable diffusion has proved that you can iterate in latent space and use a VAE to upscale to larger sizes to reduce the over all model size while still having output that is ~order of magnitude larger than the latent space.

...so it's not totally out of the question that's coming.

However, right now? no.

Re: SantaCoder: A new 1.1B code model for generation and infilling

#46

Post by HideousKojima » Fri, Dec 23, 2022, 2:19 AM UTC

zomglings wrote: post #22

Looks like Santa's going to be too busy to deliver presents this Christmas: def all_prime_elements(sequence): """Returns every prime element of the sequence.""" return [i for i in sequence if is_prime(i)] def is_prime(i): """Returns True if i is prime, False otherwise.""" if i == 2: return True if i == 3: return True if i == 5: return True if i == 7: return True if i == 11: return True if i == 13: return True if i ==…

Makes me wonder if it used the "CS Grad" memes as part of its training set:

https://i.imgur.com/RGLFim0_d.webp?maxwidth=2560&fidelity=hi...

Re: SantaCoder: A new 1.1B code model for generation and infilling

#47

Post by jillesvangurp » Fri, Dec 23, 2022, 2:20 AM UTC

ilaksh wrote: post #11

Is anyone else here building AI programming services based on models like this? I see a lot of comments saying the models can't do much programming. But I just suspect there must be a silent contingent that is also working on services like that. And maybe less likely to promote the abilities of these models because it encourages competition.

I've been pretty impressed with chat gpt generating working implementations of various algorithms in different languages. Crucially, it actually knows about algorithms. I was trying to get it to generate some algorithm for calculating concave hulls the other day and ended up learning a thing or two about various algorithms for that in this space. Almost but not quite worked for my use case. It seems limited in the amount of code it can generate in one go. But otherwise, I was pretty impressed.

So, we're not that far off from basically pair programming with an AI that will do most of the boring/tedious work we currently do manually. Something like chat gpt integrated into an IDE could be useful right now.

Re: SantaCoder: A new 1.1B code model for generation and infilling

#48

Post by DethNinja » Fri, Dec 23, 2022, 2:24 AM UTC

What is the memory requirements of this model?

Re: SantaCoder: A new 1.1B code model for generation and infilling

#49

Post by obastani » Fri, Dec 23, 2022, 3:00 AM UTC

moyix wrote: post #7

isoprophlex wrote: post #4

Earlier quoted context omitted.

Any idea how this, and those other two models, would compare to GitHub Copilot?

Based on the reverse engineering done by Parth Thakkar [1], the model used by Copilot is probably about 10x as large (12B parameters), so I would expect Copilot to still win pretty handily (especially since the Codex models are generally a lot better trained than Salesforce CodeGen or InCoder). It's also a little bit hard to compare directly because as Parth documents, there are a lot of extra smarts that go into Cop…

Just out of curiosity, in what sense is Codex is better trained than CodeGen?

Re: SantaCoder: A new 1.1B code model for generation and infilling

#50

Post by Jap2-0 » Fri, Dec 23, 2022, 3:16 AM UTC

zomglings wrote: post #22

Looks like Santa's going to be too busy to deliver presents this Christmas: def all_prime_elements(sequence): """Returns every prime element of the sequence.""" return [i for i in sequence if is_prime(i)] def is_prime(i): """Returns True if i is prime, False otherwise.""" if i == 2: return True if i == 3: return True if i == 5: return True if i == 7: return True if i == 11: return True if i == 13: return True if i ==…

Mine never defines is_prime():

  def all_prime_elements(sequence):
    """Returns every prime element of the sequence."""
    return [x for x in sequence if is_prime(x)]


  def all_prime_factors(sequence):
    """Returns every prime factor of the sequence."""
    return [x for x in sequence if is_prime(x) and x!= 1]


  def all_prime_factors_with_limit(sequence, limit):
    """Returns every prime factor of the sequence up to the given limit."""
    return [x for x in sequence if is_prime(x) and x!= 1]


  def all_prime_factors_with_limit_and_step(sequence, limit, step):
    """Returns every prime factor of the sequence up to the given limit."""
    return [x for x in sequence if is_prime(x) and x!= 1]


  def all_prime_factors_with_limit_and_step_and_start(sequence, limit, step, start):
    """Returns every prime factor of the sequence up to the given limit."""
    return [x for x in sequence if is_prime(x) and x!= 1]


  def all_prime_factors_with_limit_and_step_and_start_and_end(sequence, limit, step, start, end):
    """Returns every prime factor of the sequence up to the given limit."""
    return [x for x in sequence if is_prime(x) and x!= 1]


  def all_prime_factors_with_limit_and_step_and_start_and_end_and_step(sequence, limit, step, start, end, step):
    """Returns every prime factor of the sequence up to the given limit."""
    return [x for x in sequence if is_prime(x) and x!= 1]

(and so on)