Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code Llama, a state-of-the-art large language model for coding

ai.meta.com

Post reply on HN
Print view View original

481–490 of 525 posts

Re: Code Llama, a state-of-the-art large language model for coding

#481

Post by [deleted] »

zaphirplane wrote: post #408
belenos46 wrote: post #361

Earlier quoted context omitted.

I've done that (maybe it was fizzbuzz, now that I'm thinking about it) and boy howdy does that get the people you're interviewing with agitated. Saying "I'm interviewing for a architect level container orchestration position. If I'm reinventing the wheel writing algorithms, something is terribly wrong" shuts them up, but doesn't make them any happier.

Is the job role just a for example ? what does and container orchestration architect do ? Something like this cluster should use envoy and Prometheus. The new clusters rate isn’t usually high enough for the stack to change. Real question I love these non conventional (swe, sre, pm, manager ) roles in tech

[deleted]

Re: Code Llama, a state-of-the-art large language model for coding

#482

Post by kateklink »

Someone1234 wrote: post #315

Business opportunity: I'd pay money for NICE desktop software that can run all these different models (non-subscription, "2-year updates included, then discount pricing" modal perhaps). My wishlist: - Easy plug & play model installation, and trivial to change which model once installed. - Runs a local web server, so I can interact with it via any browser - Ability to feed a model a document or multiple documents and…

we're going in this direction for code models with Refact https://github.com/smallcloudai/refact/ - right now you self-host code models, fine-tune them on local files, get the model running locally inside your IDE

Re: Code Llama, a state-of-the-art large language model for coding

#483

Post by kateklink »

irrational wrote: post #350
Someone1234 wrote: post #315

Business opportunity: I'd pay money for NICE desktop software that can run all these different models (non-subscription, "2-year updates included, then discount pricing" modal perhaps). My wishlist: - Easy plug & play model installation, and trivial to change which model once installed. - Runs a local web server, so I can interact with it via any browser - Ability to feed a model a document or multiple documents and…

I work for a Fortune 100 company with 80,000+ employees. All of us are explicitly forbidden from using any sort of AI/LLM tool without written permission from the head of legal AND the CEO. In other words, nobody is going to get permission. The concerns are 2 fold - 1. We might inadvertently use someone else’s intellectual property. 2. Someone else might gain access to our intellectual property. What you are describi…

we try to eliminate this problem by using code models trained only on permissevely licensed code, then you can run them locally without sending code anywhere

Re: Code Llama, a state-of-the-art large language model for coding

#484

Post by kateklink »

firecall wrote: post #375
alsobrsp wrote: post #338

Earlier quoted context omitted.

I have been using refact.ai on my laptop, it has been quite good. https://github.com/smallcloudai/refact/blob/main/README.md

I wish these things worked with anything other than VSCode or JerBrains tools! VSCode is such a bloated hog of an editor! Every time I open VSCode it’s bugging with badges to update extensions… and it’s so slow!

having more plugin support is in our plans for sure. We're also open for contributions.

Re: Code Llama, a state-of-the-art large language model for coding

#485

Post by kkielhofner »

woadwarrior01 wrote: post #464
selfhoster11 wrote: post #450

Earlier quoted context omitted.

A 7B model at 8-bit quantization takes up 7 GB of RAM. Less if you use a 6-bit quantization, which is nearly as good. Otherwise it's just a question of having enough system RAM and CPU cores, plus maybe a small discrete GPU.

You’ll need a bit more than 7GB (~1 GB or so), even at 8 bit quantization, because of the KV-cache. LLM inference is notoriously inefficient without it, because it’s autoregressive.

Some projects such as lmdeploy[0] can quantize the KV cache[1] as well to save some VRAM.

Speaking of lmdeploy, it doesn't seem to be widely known but it also supports quantization with AWQ[2] which appears to be superior to the more widely used GPTQ.

The serving backend is Nvidia Triton Inference Server. Not only is Triton extremely fast and efficient, they have a custom TurboMind backend for Triton. With this lmdeploy delivers the best performance I've seen[3].

On my development workstation with an RTX 4090, llama2-chat-13b, AWQ int4, and KV cache int8:

8 concurrent sessions (batch 1): 580 tokens/s

1 concurrent session (batch 1): 105 tokens/s

This is out of the box, I haven't spent any time further optimizing it.

[0] - https://github.com/InternLM/lmdeploy

[1] - https://github.com/InternLM/lmdeploy/blob/main/docs/en/kv_in...

[2] - https://github.com/InternLM/lmdeploy/tree/main#quantization

[3] - https://github.com/InternLM/lmdeploy/tree/main#performance

Re: Code Llama, a state-of-the-art large language model for coding

#486

Post by ranguna »

mchiang wrote: post #240
jerrysievert wrote: post #233

Earlier quoted context omitted.

while it supports it, so far I've only managed to get infinite streams of near nonsense from the ollama models (codellama:7b-q4_0 and codellama:latest) my questions were asking how to construct an indexam for postgres in c, how to write an r-tree in javascript, and how to write a binary tree in javascript.

still modifying the code completion (foundation / python models) to see what's causing the behavior. Have had some good success with the instruct model: codellama:7b-instruct

Nice work with ollama!

Any plans to add the 13B quant models?

Re: Code Llama, a state-of-the-art large language model for coding

#487

Post by throwmeback »

lynx23 wrote: post #444
csmpltn wrote: post #326

Earlier quoted context omitted.

> "I'd fail an interview candidate that suggested adding 1 each time for subsequent prime testing" Congratulations! You must be that arrogant guy everybody hates interviewing with, the one with the superiority complex. How about instead of just failing people over literally nothing (wasting everybody's time and money) - just ask the candidate whether they could somehow reduce the search space by utilizing the propert…

Downvoting this is not enough. I'd pay $5 (reddit style) to have this properly killfiled if HN allowed for that. Besides, not "everybody hates them, only those inmature enough to still need intellectual babysitting.

Whoa there Satan

Re: Code Llama, a state-of-the-art large language model for coding

#488

Post by TeMPOraL »

chaboud wrote: post #440
awwaiid wrote: post #432

Earlier quoted context omitted.

Two when you include -2, which I certainly think we should in this circumstance.

Are we including it just to poke the bear? Prime numbers are typically defined as numbers with no positive divisors other than one and the number in question.

A definition which marks 1 as being prime :).

Re: Code Llama, a state-of-the-art large language model for coding

#489

Post by bakuninsbart »

noduerme wrote: post #330
blibble wrote: post #284

Earlier quoted context omitted.

it's not that though, there's several other typical optimisations in there just not the super obvious one that demonstrates extremely basic understanding of what a prime number is

I'm mad at myself now that it has eaten 15 minutes of my time trying to come up with the right optimization. What's the trick? 2, +1, and then +2 from there on seems obvious but once you get to 9 is it worth building a list of nonprimes to skip?

Exactly, multiples of 2 can't be a prime, so skipping this saves a function call (that itself only does one comparison), so it is a very simple and relatively obvious optimization. But actually every multiple of a number per definition cannot be a prime, you can't just change the step size, but there are some nifty tricks to approach the problem from a different angle.

Re: Code Llama, a state-of-the-art large language model for coding

#490

Post by TeMPOraL »

NhanH wrote: post #465
maleldil wrote: post #461

Earlier quoted context omitted.

I don't get it. This is an extremely basic fact that most people can figure out after thinking about primes for a minute. Maybe if you ask for "what's an easy optimisation here?" This would make the candidate think more closely about invariants that their code should hold, which in itself is a very valuable skill.

Because I know enough not to write prime testing code that resemble anything like that loop to have to care about reducing its search space. If you actually want to test my knowledge about prime number, you can ask and I will tell you about using some probabilistic choice instead, and that I know fast deterministic one might exist, but I am not up to date on the state of the art. If I have to write the loop above, I…

Same. Also, if I'm suspecting it's a "can they into loops" fizzbuzz test, I'd be wary of reducing the search space for the simple reason that it makes the code slightly more complex, introducing a chance to make e.g. off-by-one error, which would lose me points if the reviewer has the kind of "pedntic over irrelevant things" attitude this subthread is criticizing.
Post reply on HN