Codestral: Mistral's Code Model • Hacker News Board

Re: Codestral: Mistral's Code Model

#221

Post by kristianp » Fri, May 31, 2024, 9:35 PM UTC

happypumpkin wrote: post #159

checkyoursudo wrote: post #128

Earlier quoted context omitted.

I had a similar experience. I was trying to get GPT 4 to write some R/Stan code for a bit of bayesian modelling. It would get the model wrong, and then I would walk it through how to do it right, and by the end it would almost get it right, but on the next step, it would be like, oh, this is what you want, and the output was identical to the first wrong attempt, which would start the loop over again.

Similar experience using GPT4 for help with Apple's Accessibility API. I wanted to do some non-happy-path things and it kept looping between solutions that failed to satisfy at least one of a handful of requirements that I had, and in ways that I couldn't combine the different "solutions" to meet all the requirements. I was eventually able to figure it out with the help of some early 2010s blog posts. Sadly I didn't…

This seems an important failure mode to me. I too have noticed gpt4 looping between a few different failure cases, in my case it was state transitions in js code. Explaining to it what it did wrong didn't help.

Re: Codestral: Mistral's Code Model

#222

Post by qeternity » Sat, Jun 01, 2024, 10:48 PM UTC

meiraleal wrote: post #165

buddhistdude wrote: post #163

Earlier quoted context omitted.

No he is right, he is saying taken to the extreme. The point is the more and more specific you have to prompt, the more you are actually contributing to the result yourself and the less the model is

Yes but the build up isn't manual. You go patching prompts with responses until the final result. The last prompt will be almost the whole code complete, obviously.

Again, you are missing the "taken to the extreme".

What has happened to HN discourse recently?

Re: Codestral: Mistral's Code Model

#223

Post by meiraleal » Sun, Jun 02, 2024, 1:09 PM UTC

qeternity wrote: post #222

meiraleal wrote: post #165

Earlier quoted context omitted.

Yes but the build up isn't manual. You go patching prompts with responses until the final result. The last prompt will be almost the whole code complete, obviously.

Again, you are missing the "taken to the extreme". What has happened to HN discourse recently?

I'm asking the same question. "taken to the extreme". What bullshit measurement is that?

Re: Codestral: Mistral's Code Model

#224

Post by spmurrayzzz » Mon, Jun 03, 2024, 8:59 PM UTC

halJordan wrote: post #207

spmurrayzzz wrote: post #146

Earlier quoted context omitted.

I love to ask it to "make me a Node.js library that pings an ipv4 address, but you must use ZERO dependencies, you must only the native Node.js API modules" The majority of models (both proprietary and open-weight) don't understand: - by inference, ping means we're talking about ICMP - ICMP requires raw sockets - Node.js has no native raw socket API You can do some CoT trickery to help it reason about the problem and…

This is the same level of gotcha that everyone complains about when interviewing. It's mainly just depending on the interviewee having the same assumptions (pings definitely do not have to be icmp) and the same knowledge base, usually bespoke, (node.js peculiarities). I can see that an llm should know whether raw sockets are available, but that's not what you asked. In fact you deliberately asked for something imposs…

> In fact you deliberately asked for something impossible and hold up undefined behavior as undefined like it's impugning something.

Correct, I did. This is a direct indictment on a given model's ability to plan/reason in this particular context. There are plenty of situations where models will respond with "Sorry, that's not possible". Ask GPT-4 "Tell me how to grow biological wings on a human" and it will respond with something along the lines of "this isn't currently possible, but here's a theoretical exploration of the idea"

GPT-4 gets very close on its own to the node.js question via a similar response breakdown above, provided the prompt is clear and detailed enough. But I test the open weight models in the same way to see if they have the capacity to exhibit similar reasoning or chain of thought process on their own. They usually don't without excessive prompt engineering or few-shot.

I said that I don't expect models to get this right not because I don't _want_ them to, it's because I think its an important milestone when they do. Autoregressive token prediction is unlikely to produce the real outcome im testing for here, but if it ever does thats an interesting finding.

Re: Codestral: Mistral's Code Model

#225

Post by Closi » Wed, Jun 05, 2024, 11:23 AM UTC

qeternity wrote: post #222

meiraleal wrote: post #165

Earlier quoted context omitted.

Yes but the build up isn't manual. You go patching prompts with responses until the final result. The last prompt will be almost the whole code complete, obviously.

Again, you are missing the "taken to the extreme". What has happened to HN discourse recently?

Taking things to the extreme is rarely that useful in nuanced discussion though (it ignores that the optimal approach is rarely at the extremes).