Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Codestral: Mistral's Code Model

mistral.ai

Post reply on HN
Print view View original

71–80 of 225 posts

Re: Codestral: Mistral's Code Model

#71

Post by croes »

colesantiago wrote: post #17

I'm so happy now LLMs are democratising access to programming, especially open models like what Meta with Llama and Mistral is doing with Codestral are doing. The abundance of programming is going to allow almost everyone to become a great programmer. This is so exciting to see and each day programming is becoming a solved problem so we can focus on other things.

>The abundance of programming is going to allow almost everyone to become a great programmer.

How do you become a great programmer if you don't really program?

Re: Codestral: Mistral's Code Model

#72

Post by behnamoh »

analyte123 wrote: post #48

The license for this [1] prohibits use of the model and its outputs for any commercial activity, or even any "live" (whatever that means) conditions, commercial or not. There seems to be an exclusion for using the code outputs as part of "development". But wait! It also prohibits "any internal usage by employees in the context of the company's business activities". However you interpret these clauses, this puts their…

From the website:

> licensed under the new Mistral AI Non-Production License, which means that you can use it for research and testing purposes. ...

Which basically means "we give you this model. Go find its weaknesses and report on r/locallama. Then we'll use that to improve our commercial model which we won't open-source."

I'm sick of abusing the word "open-source" in this field.

Re: Codestral: Mistral's Code Model

#73

Post by whiplash451 »

icedchai wrote: post #62
colesantiago wrote: post #17

I'm so happy now LLMs are democratising access to programming, especially open models like what Meta with Llama and Mistral is doing with Codestral are doing. The abundance of programming is going to allow almost everyone to become a great programmer. This is so exciting to see and each day programming is becoming a solved problem so we can focus on other things.

I'm skeptical. I've run into people who used LLMs to code, then can't debug it without someone else's help. It may get you 80% there though.

It does not get you 80% there if it achieves what you described. It rather gets you 100% into trouble.

Re: Codestral: Mistral's Code Model

#74

Post by behnamoh »

swyx wrote: post #13

i've been noticing that there's a divergence in philosophy between Llama style LLMs (Mistral are Meta alums so I'm counting them in tehre) and OpenAI/GPT style LLMs when it comes to code. GPT3.5+ prioritized code very heavily - there's no CodeGPT, its just GPT4, and every version is better than the last. Whereas the Llama/Mistral models are now shipping the general language model first, then adding CodeLlama/Codestra…

> Zuck

No, if anything he said Meta realized coding abilities make the model overall better, so they focused on those more than before.

Re: Codestral: Mistral's Code Model

#75

Post by JimDabell »

ddavis wrote: post #25

My favorite thing to ask the models designed for programming is: "Using Python write a pure ASGI middleware that intercepts the request body, response headers, and response body, stores that information in a dict, and then JSON encodes it to be sent to an external program using a function called transmit." None of them ever get it right :)

I normally ask about building a multi-tenant system using async SQLAlchemy 2 ORM where some tables are shared between tenants in a global PostgreSQL schema and some are in a per-tenant schema.

Nothing gets it right first time, but when ChatGPT 4 first came out, I could talk to it more and it would eventually get it right. Not long after that though, ChatGPT degraded. It would get it wrong on the first try, but with every subsequent follow up it would forget one of the constraints. Then when it was prompted to fix that one, it forgot a different one. And eventually it would cycle through all of the constraints, getting at least one wrong each time.

Since then benchmarks came out showing that ChatGPT “didn’t really degrade”, but all of the benchmarks seemed focused on single question/answer pairs and not actual multi-turn chat. For this kind of thing, ChatGPT 4 has never managed to recover to as good as it was when it was first released in my experience.

It’s been months since I’ve had to deal with that kind of code, so I might be forgetting something, but I just tried it with Codestral and it spat out something that looked reasonable very quickly on its first try.

Re: Codestral: Mistral's Code Model

#76

Post by TechDebtDevin »

Closi wrote: post #64
TechDebtDevin wrote: post #54

Earlier quoted context omitted.

I see your point but hand holding isn't really a good way to benchmark a models coding capabilities.

Depends if benchmarking is the aim, rather than decreasing the time it takes to build things.

Well sure, but that wasn't what we were discussing. The original comment says they use that as their benchmark. While their coding task is a bit complex compared to other benchmarking prompts, it's not that crazy. Here is an example of prompts used for benchmarking with Python for reference:

https://huggingface.co/datasets/mbpp?row=98

At the end of the day LLMs in their current iteration aren't intended to do even moderately difficult tasks on their own but it's fun to query them to see progress when new claims are made.

Re: Codestral: Mistral's Code Model

#77

Post by kergonath »

ddavis wrote: post #61
bongodongobob wrote: post #30

Earlier quoted context omitted.

Cool, you've identified that your prompt is inadequate for the task. 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?'

It's something I know how to do after figuring it out myself and discovering the potential sharp edges, so I've made it into a fun game to test the models. I'd argue that it's a great prompt (to keep using consistently over time) to see the evolution of this wildly accelerating field.

Do you notice any progress over time?

Re: Codestral: Mistral's Code Model

#78

Post by croes »

smokel wrote: post #56
colesantiago wrote: post #17

I'm so happy now LLMs are democratising access to programming, especially open models like what Meta with Llama and Mistral is doing with Codestral are doing. The abundance of programming is going to allow almost everyone to become a great programmer. This is so exciting to see and each day programming is becoming a solved problem so we can focus on other things.

In my experience these tools amplify the quality of a programmer. I have seen good programmers dramatically increase their productivity, but I've also seen others copy-pasting for loops inside other for loops where one loop would definitely suffice. We're not quite there yet.

I'm curious for the long-term effect.

I observe a certain laziness in myself when it comes to certain problems. It's easier to ask a LLM and debug provided code, but I ask myself if I'm losing some problem solving capabilities in the long run because of this.

Similar to the loss of speed in doing mental arithmetic because of calculators on the smartphone.

Re: Codestral: Mistral's Code Model

#79

Post by maskil »

colesantiago wrote: post #17

I'm so happy now LLMs are democratising access to programming, especially open models like what Meta with Llama and Mistral is doing with Codestral are doing. The abundance of programming is going to allow almost everyone to become a great programmer. This is so exciting to see and each day programming is becoming a solved problem so we can focus on other things.

I would argue the opposite is true.

My experience with coding with LLMs is that the only thing it's really good at is generating boilerplate that it has more-or-less seen before (essentially a library, even if is somewhat adapted), however it is incapable of the creative thinking that developers regularly need to engage in when architecting a solution for their use case.

Re: Codestral: Mistral's Code Model

#80

Post by TechDebtDevin »

bottom999mottob wrote: post #52
bongodongobob wrote: post #45

Earlier quoted context omitted.

My point is that you shouldn't expect to one shot everything. Have it start by writing a spec, then outline classes and methods, then write the code, and feed it debug stuff.

Exactly, expecting one shot 100% working code with one prompt is ridiculous at this point. It's why libraries like Aider are so useful, because you can iteratively diff generated code until it's useable.

Sure it's impossible at this point, but the point of a benchmark isn't to complete the task it's to test it's efficacy overall and to see progress. None of them are 100% at even the simplistic python benchmarks, doesn't mean we shouldn't measure that capability. But sure, I get it. That's not how they are intended to be used but that's also not the point the commenter was laying out.
Post reply on HN