Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Codestral: Mistral's Code Model

mistral.ai

Post reply on HN
Print view View original

31–40 of 225 posts

Re: Codestral: Mistral's Code Model

#31

Post by Havoc »

sharbloop wrote: post #21
fnbr wrote: post #15

Earlier quoted context omitted.

The rule of thumb is roughly 44gb, as most models are trained in bf16, and require 16 bits per parameter, so 2 bytes. You need a bit more for activations, so maybe 50GB? you need enough RAM and HBM (GPU RAM) so it’s a constraint on both.

Which GPU card can I buy to run this model? Can it run on commercial RTX3090 or does it need a custom GPU?

3090 or 4090 will be able to run quantized 22B models.

Though realistically for code completion smaller models will be better due to speed

Re: Codestral: Mistral's Code Model

#32

Post by sanex »

ddavis wrote: post #25

My favorite thing to ask the models designed for programming is: "Using Python write a pure ASGI middleware that intercepts the request body, response headers, and response body, stores that information in a dict, and then JSON encodes it to be sent to an external program using a function called transmit." None of them ever get it right :)

Can you get it right without an IDE?

Re: Codestral: Mistral's Code Model

#34

Post by TechDebtDevin »

bongodongobob wrote: post #30
ddavis wrote: post #25

My favorite thing to ask the models designed for programming is: "Using Python write a pure ASGI middleware that intercepts the request body, response headers, and response body, stores that information in a dict, and then JSON encodes it to be sent to an external program using a function called transmit." None of them ever get it right :)

Cool, you've identified that your prompt is inadequate for the task. 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?'

Damn, show us your brilliant prompt then. LLMs cannot do this, not even in python, of which there are libraries like Blacksheep that honestly make it a trivial task.

Re: Codestral: Mistral's Code Model

#35

Post by meiraleal »

IMTDb wrote: post #28

Is there a way to use this within VSCode like copilot , meaning having the "shadow code" appear while you code instead of having to tho back-and-forth between the editor and a chat-like interface ? For me, a significant component of the quality of these tools resides on the "client" side; being able to engineer a prompt that will yield to accurate code being generated by the model. The prompt needs to find and embed…

I created a simple CLI app that does this in my workspace, which is under source control so after the LLM execution all the changes are highlighted by diff and the LLM also creates a COMMIT_EDITMSG file describing what it changed. Now I don't use chatgpt anymore, only this cli tool.

I never saw something like this integrated directly on VSCode tho (and isn't my preferred workflow anyway, command line works better).

Re: Codestral: Mistral's Code Model

#36

Post by nicce »

ddavis wrote: post #25

My favorite thing to ask the models designed for programming is: "Using Python write a pure ASGI middleware that intercepts the request body, response headers, and response body, stores that information in a dict, and then JSON encodes it to be sent to an external program using a function called transmit." None of them ever get it right :)

I usually through some complex Rust code with lifetime requirements. And ask them to fix it. LLMs aren't capable on providing much help for that in general, other than some very basic cases.

The best way to get your work done is still to look into Rust forums.

Re: Codestral: Mistral's Code Model

#37

Post by pyepye »

IMTDb wrote: post #28

Is there a way to use this within VSCode like copilot , meaning having the "shadow code" appear while you code instead of having to tho back-and-forth between the editor and a chat-like interface ? For me, a significant component of the quality of these tools resides on the "client" side; being able to engineer a prompt that will yield to accurate code being generated by the model. The prompt needs to find and embed…

Not using Codestral (yet) but check out Continue.dev[1] with Ollama[2] running llama3:latest and starcoder2:3b. It gives you a locally running chat and edit via llama3 and autocomplete via starcoder2.

It's not perfect but it's getting better and better.

[1] https://www.continue.dev/ [2] https://ollama.com/

Re: Codestral: Mistral's Code Model

#38

Post by jasonjmcghee »

sashank_1509 wrote: post #12

Seems nice but some preliminary testing against GPT-4o shows it’s lacking a bit. It does a pretty good job for easy questions though

GPT-4o is really oddly hit or miss for code.

Sometimes it outperforms GPT-4 in quality by a fair amount, and other times it starts repeating itself. Duplicating function definitions, even misremembering what things are named.

It seems to have to do with length. If the output exceeds a few thousand tokens, it seems to experience some pretty bad failure modes.

Re: Codestral: Mistral's Code Model

#40

Post by [deleted] »

ddavis wrote: post #25

My favorite thing to ask the models designed for programming is: "Using Python write a pure ASGI middleware that intercepts the request body, response headers, and response body, stores that information in a dict, and then JSON encodes it to be sent to an external program using a function called transmit." None of them ever get it right :)

[deleted]
Post reply on HN