Live data from Hacker News

Code Llama, a state-of-the-art large language model for coding

ai.meta.com

191–200 of 525 posts

Re: Code Llama, a state-of-the-art large language model for coding

#191
post #132
post #130

Earlier quoted context omitted.

That’s just one perspective… Another perspective is that LLMs enable programmers to skip a lot of the routine and boring aspects of coding - looking up stuff, essentially - so they can focus on the fun parts that engage creativity.

But it won't stop there. Why would it stop at some arbitrarily defined boundary? The savings associated with no longer having to pay programmers the amounts of money that they believe they are worth (high enough to result in collusion between employers) are just too tempting.

Some form of AI will eventually take over almost all existing jobs. Whether those jobs evolve or not somehow and new jobs replace them, we will see.

But it's definitely not just programmers. And it will take time.

Society needs to adjust. Stopping progress would not be a solution and is not possible.

However, hopefully we can pause before we create digital animals with hyperspeed reasoning and typical animal instincts like self-preservation. Researchers like LeCun are already moving on from things like LLMs and working on approaches that really imitate animal cognition (like humans) and will eventually blow all existing techniques out of the water.

The path that we are on seems to make humans obsolete within three generations or so.

So the long term concern is not jobs, but for humans to lose control of the planet in less than a century.

On the way there we might be able to manage a new golden age -- a crescendo for human civilization.

Re: Code Llama, a state-of-the-art large language model for coding

#192
post #177

Earlier quoted context omitted.

The paper states it was instruction fine tuned with synthetic data (LLM generated instructions) ala another paper (“Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor”). The github repo associated with that paper is linked below. It links to the paper on arxiv, but also has some data in the repo. https://github.com/orhonovich/unnatural-instructions

Maybe they used GPT-4 to train it. OpenAI terms of use don't allow that to be released commercially.

This is the most likely explanation for both why they wouldn't release it and wouldn't explain why.

Re: Code Llama, a state-of-the-art large language model for coding

#193
post #134

Copilot has been working great for me thus far, but it's limited by its interface. It seems like it only knows how to make predictions for the next bit of text. Is anyone working on a code AI that can suggest refactorings? "You should pull these lines into a function, it's repetitive" "You should change this structure so it is easier to use" Etc

There's an instruct model in there, you can definitely use it for this, that's one of the objectives. An instruct model means that you can ask it to do what you want, including asking it to give you refactoring ideas from the code you will give it.

How do I access it from my IDE? Jetbrains/VSCode?

Re: Code Llama, a state-of-the-art large language model for coding

#194

Copilot has been working great for me thus far, but it's limited by its interface. It seems like it only knows how to make predictions for the next bit of text. Is anyone working on a code AI that can suggest refactorings? "You should pull these lines into a function, it's repetitive" "You should change this structure so it is easier to use" Etc

I let mine generate whatever it likes, then add a comment below such as "# Refactor the above to foo.." Works fairly well at times.

Can it suggest deletions? Just seems like I don't know how to use it.

Re: Code Llama, a state-of-the-art large language model for coding

#196
post #122

Never before in the history of mankind was a group so absolutely besotted with the idea of putting themselves out of a job.

When mechanized textile machinery was invented, the weavers that had jobs after their introduction were those that learned how to use them.

Re: Code Llama, a state-of-the-art large language model for coding

#197
post #8

Interesting that there's a 34B model. That was missing from the original Llama 2 release. I wonder if it's still usable for general non-code chat tasks or if the code fine tuning destroyed that. It should be the best model that would still fit on 24GB gaming GPUs with quantization, because 70B doesn't fit.

Looks like they left out another model though. In the paper they mention a "Unnatural Code Llama" which wipes the floor with every other model/finetune on every benchmark except for slightly losing to Code Llama Python on MBPP pass@100 and slightly losing to GPT-4 on HumanEval pass@1 which is insane. Meta says later on that they aren't releasing it and give no explanation. I wonder why given how incredible it seems t…

It's "unnatural" because it was finetuned on generated data using another model, almost certainly gpt-4 (whose TOS forbid this).

Re: Code Llama, a state-of-the-art large language model for coding

#198
post #177

Earlier quoted context omitted.

The paper states it was instruction fine tuned with synthetic data (LLM generated instructions) ala another paper (“Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor”). The github repo associated with that paper is linked below. It links to the paper on arxiv, but also has some data in the repo. https://github.com/orhonovich/unnatural-instructions

Maybe they used GPT-4 to train it. OpenAI terms of use don't allow that to be released commercially.

I've seen this argued a lot but is it fact? OpenAI was able to train on data from other platforms and surely, those platforms weren't letting their data go if they could help it. Unless some new laws have been passed, I don't think OpenAI can legally prevent others from using their data to train models. OpenAI can't have their cake and eat it too. After all, any content generated by AI can't be copyrighted.

Re: Code Llama, a state-of-the-art large language model for coding

#199

Earlier quoted context omitted.

Looks like they aren't releasing a pretty interesting model too. In the paper they mention a "Unnatural Code Llama" which wipes the floor with every other model/finetune on every benchmark except for slightly losing to Code Llama Python on MBPP pass@100 and slightly losing to GPT-4 on HumanEval pass@1 which is insane. Meta says later on that they aren't releasing it and give no explanation. I wonder why given how inc…

Note that current GPT-4 pass@1 for HumanEval is closer to 90% than to 67% reported in GPT-4 technical report, as reported, e.g., in [1] [1] https://arxiv.org/abs/2305.01210

Good point, I guess Meta should be using that number in their chart
Post reply on HN