Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Codestral Mamba

mistral.ai

Post reply on HN
Print view View original

81–90 of 148 posts

Re: Codestral Mamba

#81

Post by azeirah »

funnygiraffe wrote: post #64
throwup238 wrote: post #58

Earlier quoted context omitted.

Claude Projects which allow attaching a bunch of files to fill up the 200k context. I wrote up a script to dump a bunch of code and documentation files to markdown as context and I add them to a bunch of Claude projects on a per topic basis. For example, I'm currently working on a Rust/Qt desktop app so I have a project with the whole Qt6 book attached to ask questions about Qt, a project with my SQL schema and ORM/S…

I was under the impression that with LLMs, in order to get high-quality answers, it's always best to keep context short. Is that not the case anymore? Does Claude under this usage paradigm not struggle with very long contexts in ways as for example described in the "lost in the middle" paper ( https://arxiv.org/abs/2307.03172 )?

The conclusion you walked away with is the opposite of what usually works in practice.

The more context you give the llm, the better.

The key takeaway from that paper is to keep your instructions/questions/direction in the beginning or at the end of the context. Any information can go anywhere.

Not to be too dismissive, it's a good paper, but we're one year further and in practice this issue seems to have been tackled by training on better data.

This can differ a lot depending on what model you're using, but in the case of claude sonnet 3.5, more relevant context is generally better for anything except for speed.

It does remain true that you need to keep your most important instructions at the beginning or at the end however.

Re: Codestral Mamba

#82

Post by kz919 »

pzo wrote: post #55

weird they compare to deepseek-coder v1.5 when we already have v2.0. Any advantage to use codestral mamba apart from that it's lighter in weights?

obviously because they can't beat it... There will be zero reason to use it when you have better transformer based models that can fit the existing infrastructure.

Re: Codestral Mamba

#83

Post by holoduke »

nabakin wrote: post #76
culopatin wrote: post #3

Does anyone have a video or written article that would get one up to speed with a bit of the history/progression and current products that are out there for one to try locally? This is coming from someone that understands the general concepts of how LLMs work but only used the general publicly available tools like ChatGPT, Claude, etc. I want to see if I have any hardware I can stress and run something locally, but d…

Here's a summary of what's happened the past couple of years and what tools are out there. After ChatGPT released, there was a lot of hype in the space but open source was far behind. Iirc the best open foundation LLM that existed was GPT-2 but it was two generations behind. Awhile later Meta released LLaMA[1], a well trained base foundation model, which brought an explosion to open source. It was soon implemented in…

Great info. Do you also know the state of the code assistants? Any thoughts on copilot versus others?

Re: Codestral Mamba

#84

Post by iAmAPencilYo »

nabakin wrote: post #76
culopatin wrote: post #3

Does anyone have a video or written article that would get one up to speed with a bit of the history/progression and current products that are out there for one to try locally? This is coming from someone that understands the general concepts of how LLMs work but only used the general publicly available tools like ChatGPT, Claude, etc. I want to see if I have any hardware I can stress and run something locally, but d…

Here's a summary of what's happened the past couple of years and what tools are out there. After ChatGPT released, there was a lot of hype in the space but open source was far behind. Iirc the best open foundation LLM that existed was GPT-2 but it was two generations behind. Awhile later Meta released LLaMA[1], a well trained base foundation model, which brought an explosion to open source. It was soon implemented in…

Thank you! Very helpful as a newbie coming in.

Re: Codestral Mamba

#85

Post by stavros »

causal wrote: post #60
mountainriver wrote: post #56

Earlier quoted context omitted.

lol no it’s not, the benchmarks don’t show that at all. Both have issues in different ways

Benchmarks are pretty flawed IMO, in particular their weakness here seems to be that they are poor at evaluating long-tail multiturn conversations. 4o often gives a great first response, then spirals into a repetition. Sonnet 3.5 is much better at seeing the big picture in a longer conversation IMO.

I made a mobile app the other day using LLMs (I had never used React or TypeScript before, and I built an app with React Native). I was pretty disappointed, both Sonnet 3.5 and gpt-4-turbo performed pretty poorly, making mistakes like missing a closing bracket somewhere and meaning I had to revert, because I had no idea where they meant to put it.

Also they did the thing that junior developers tend to do, where you have a race condition of some sort, and they just work around it by adding some if checks. The app is at around 400 lines right now, it works but feels pretty brittle. Adding a tiny feature here or there breaks something else, and GPT does the wrong thing half the time.

All in all, I'm not complaining, because I made an app in two days, but it won't replace a developer yet, no matter how much I want it to.

Re: Codestral Mamba

#86

Post by nabakin »

holoduke wrote: post #83
nabakin wrote: post #76

Earlier quoted context omitted.

Here's a summary of what's happened the past couple of years and what tools are out there. After ChatGPT released, there was a lot of hype in the space but open source was far behind. Iirc the best open foundation LLM that existed was GPT-2 but it was two generations behind. Awhile later Meta released LLaMA[1], a well trained base foundation model, which brought an explosion to open source. It was soon implemented in…

Great info. Do you also know the state of the code assistants? Any thoughts on copilot versus others?

I've been following the state of things, but I'm not sure which ones are the best. There's Meta's CodeLlama[1], Mistral's Codestral[2], DeepSeek AI's DeepSeek-Coder-V2-Instruct[3], CodeGemma[4], Alibaba's CodeQwen[5], and Microsoft's WizardCoder[6].

I'm pretty sure CodeLlama is out of date now. I've heard DeepSeek LLMs are good and DeepSeek-Coder-V2-Instruct was released recently. With the good reputation and its massive size (236b) I'd guess it is the best coding LLM, but if it's not being trained efficiently, maybe Codestral and Codestral Mamba come close.

I don't think the best coding LLMs are close to GitHub Copilot but I could be wrong since I'm just relaying information that I've heard secondhand.

[1] https://ai.meta.com/blog/code-llama-large-language-model-cod...

[2] https://mistral.ai/news/codestral/

[3] https://github.com/deepseek-ai/DeepSeek-Coder-V2

[4] https://developers.googleblog.com/en/gemma-family-expands-wi...

[5] https://qwenlm.github.io/blog/codeqwen1.5/

[6] https://github.com/nlpxucan/WizardLM

Re: Codestral Mamba

#87

Post by hobofan »

holoduke wrote: post #83
nabakin wrote: post #76

Earlier quoted context omitted.

Here's a summary of what's happened the past couple of years and what tools are out there. After ChatGPT released, there was a lot of hype in the space but open source was far behind. Iirc the best open foundation LLM that existed was GPT-2 but it was two generations behind. Awhile later Meta released LLaMA[1], a well trained base foundation model, which brought an explosion to open source. It was soon implemented in…

Great info. Do you also know the state of the code assistants? Any thoughts on copilot versus others?

All the main IDE-integrated ones seem very much on par (Copilot, Sourcegraph Cody, Continue.dev), with cursor.sh liked by some as it has code assistant-first UI.

I've personally went back to the browser with Claude 3.5 Sonnet (and the projects + artifacts feature), as it is one of the most industrious ones, and I really like the UX of artifacts + it integrates new code well into existing code you paste into it.

In the end I think it also often comes down to what languages/frameworks you are using and how well the LLM/product handles it, so I'd still recommend to test around. E.g. some of the main frameworks I'm working with on a daily basis went through big refactors/interface changes 1-2 years ago, and I stopped using ChatGPT because it had a strong tendency to produce code based on the old interfaces/paradigms.

Aider[0] is also quite interesting, especially when it comes to more significant refactorings in the codebase and has gotten quite good with that with the last few bigger model releases, but it takes same time to get used to and doesn't have good IDE-integration.

[0]: https://github.com/paul-gauthier/aider

Re: Codestral Mamba

#88

Post by psychoslave »

nabakin wrote: post #76
culopatin wrote: post #3

Does anyone have a video or written article that would get one up to speed with a bit of the history/progression and current products that are out there for one to try locally? This is coming from someone that understands the general concepts of how LLMs work but only used the general publicly available tools like ChatGPT, Claude, etc. I want to see if I have any hardware I can stress and run something locally, but d…

Here's a summary of what's happened the past couple of years and what tools are out there. After ChatGPT released, there was a lot of hype in the space but open source was far behind. Iirc the best open foundation LLM that existed was GPT-2 but it was two generations behind. Awhile later Meta released LLaMA[1], a well trained base foundation model, which brought an explosion to open source. It was soon implemented in…

This is one one the most useful and informative comment I ever faced on HN. Thank you very much.

Re: Codestral Mamba

#89

Post by [deleted] »

sam_goldman_ wrote: post #59

You can try this model out using OpenAI's API format with this TypeScript SDK: https://github.com/token-js/token.js You just need a Mistral API key: https://console.mistral.ai/api-keys/

[deleted]

Re: Codestral Mamba

#90

Post by rvnx »

funnygiraffe wrote: post #64
throwup238 wrote: post #58

Earlier quoted context omitted.

Claude Projects which allow attaching a bunch of files to fill up the 200k context. I wrote up a script to dump a bunch of code and documentation files to markdown as context and I add them to a bunch of Claude projects on a per topic basis. For example, I'm currently working on a Rust/Qt desktop app so I have a project with the whole Qt6 book attached to ask questions about Qt, a project with my SQL schema and ORM/S…

I was under the impression that with LLMs, in order to get high-quality answers, it's always best to keep context short. Is that not the case anymore? Does Claude under this usage paradigm not struggle with very long contexts in ways as for example described in the "lost in the middle" paper ( https://arxiv.org/abs/2307.03172 )?

At the beginning it was true, the longer the context, the more the LLM was lost, but now, the new models can retrieve information anywhere in the context

c.f.

https://pbs.twimg.com/media/GH2NJMxbYAAcRL3?format=jpg&name=...

Post reply on HN