Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Codestral: Mistral's Code Model

mistral.ai

Post reply on HN
Print view View original

21–30 of 225 posts

Re: Codestral: Mistral's Code Model

#21

Post by sharbloop »

fnbr wrote: post #15
andruby wrote: post #10

This is an open weights 22B model. The download on Huggingface is 44GB. Is there a rule-of-thumb estimate for how much RAM this would need to be used locally? Is the RAM requirement the same for a GPU and "unified" RAM like Apple silicon?

The rule of thumb is roughly 44gb, as most models are trained in bf16, and require 16 bits per parameter, so 2 bytes. You need a bit more for activations, so maybe 50GB? you need enough RAM and HBM (GPU RAM) so it’s a constraint on both.

Which GPU card can I buy to run this model? Can it run on commercial RTX3090 or does it need a custom GPU?

Re: Codestral: Mistral's Code Model

#22

Post by tkellogg »

swyx wrote: post #13

i've been noticing that there's a divergence in philosophy between Llama style LLMs (Mistral are Meta alums so I'm counting them in tehre) and OpenAI/GPT style LLMs when it comes to code. GPT3.5+ prioritized code very heavily - there's no CodeGPT, its just GPT4, and every version is better than the last. Whereas the Llama/Mistral models are now shipping the general language model first, then adding CodeLlama/Codestra…

The OpenAI philosophy is that adding modes improves everything. Sure, it’s astronomically expensive, but I tend to think they’re on to something.

Re: Codestral: Mistral's Code Model

#23

Post by [deleted] »

andruby wrote: post #10

This is an open weights 22B model. The download on Huggingface is 44GB. Is there a rule-of-thumb estimate for how much RAM this would need to be used locally? Is the RAM requirement the same for a GPU and "unified" RAM like Apple silicon?

[deleted]

Re: Codestral: Mistral's Code Model

#25

Post by ddavis »

My favorite thing to ask the models designed for programming is: "Using Python write a pure ASGI middleware that intercepts the request body, response headers, and response body, stores that information in a dict, and then JSON encodes it to be sent to an external program using a function called transmit." None of them ever get it right :)

Re: Codestral: Mistral's Code Model

#26

Post by [deleted] »

swyx wrote: post #13

i've been noticing that there's a divergence in philosophy between Llama style LLMs (Mistral are Meta alums so I'm counting them in tehre) and OpenAI/GPT style LLMs when it comes to code. GPT3.5+ prioritized code very heavily - there's no CodeGPT, its just GPT4, and every version is better than the last. Whereas the Llama/Mistral models are now shipping the general language model first, then adding CodeLlama/Codestra…

[deleted]

Re: Codestral: Mistral's Code Model

#27

Post by Novosell »

fnbr wrote: post #15
andruby wrote: post #10

This is an open weights 22B model. The download on Huggingface is 44GB. Is there a rule-of-thumb estimate for how much RAM this would need to be used locally? Is the RAM requirement the same for a GPU and "unified" RAM like Apple silicon?

The rule of thumb is roughly 44gb, as most models are trained in bf16, and require 16 bits per parameter, so 2 bytes. You need a bit more for activations, so maybe 50GB? you need enough RAM and HBM (GPU RAM) so it’s a constraint on both.

Most GPUs still use GDDR I'm pretty sure, not HBM. Do you mean VRAM?

Re: Codestral: Mistral's Code Model

#28

Post by IMTDb »

Is there a way to use this within VSCode like copilot , meaning having the "shadow code" appear while you code instead of having to tho back-and-forth between the editor and a chat-like interface ?

For me, a significant component of the quality of these tools resides on the "client" side; being able to engineer a prompt that will yield to accurate code being generated by the model. The prompt needs to find and embed the right chunks from the user current workspace, or even from his entire org repos. The model is "just" one piece of the puzzle.

Re: Codestral: Mistral's Code Model

#29

Post by imachine1980_ »

memothon wrote: post #18
swyx wrote: post #13

i've been noticing that there's a divergence in philosophy between Llama style LLMs (Mistral are Meta alums so I'm counting them in tehre) and OpenAI/GPT style LLMs when it comes to code. GPT3.5+ prioritized code very heavily - there's no CodeGPT, its just GPT4, and every version is better than the last. Whereas the Llama/Mistral models are now shipping the general language model first, then adding CodeLlama/Codestra…

>Zuck has mentioned recently That's a really surprising thing to hear, where did you see that? The only quote I've seen is this one: >“One hypothesis was that coding isn’t that important because it’s not like a lot of people are going to ask coding questions in WhatsApp,” he says. “It turns out that coding is actually really important structurally for having the LLMs be able to understand the rigor and hierarchical s…

Make Sense, they want better interaction whit users for Whatsapp, Instagram and Facebook marketers, content creation and moderation,and their glasses(ai /ar) I don't see in that context why the should push more effort into llm coding, is sad anyways

Re: Codestral: Mistral's Code Model

#30

Post by bongodongobob »

ddavis wrote: post #25

My favorite thing to ask the models designed for programming is: "Using Python write a pure ASGI middleware that intercepts the request body, response headers, and response body, stores that information in a dict, and then JSON encodes it to be sent to an external program using a function called transmit." None of them ever get it right :)

Cool, you've identified that your prompt is inadequate for the task.

'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?'

Post reply on HN