Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Codestral Mamba

mistral.ai

Post reply on HN
Print view View original

131–140 of 148 posts

Re: Codestral Mamba

#131

Post by nabakin »

visarga wrote: post #109
nabakin wrote: post #76

Earlier quoted context omitted.

Here's a summary of what's happened the past couple of years and what tools are out there. After ChatGPT released, there was a lot of hype in the space but open source was far behind. Iirc the best open foundation LLM that existed was GPT-2 but it was two generations behind. Awhile later Meta released LLaMA[1], a well trained base foundation model, which brought an explosion to open source. It was soon implemented in…

Overall a good write up, but I have a few quips: > Awhile later Meta released LLaMA[1], I think Stable Diffusion was first to release a SOTA model (August 2022) that worked locally, not in language but image generation, but it set the tone for Meta. LLaMA only came in February 2023. > The company Mistral had proven itself in the past with very impressive LLaMA finetunes Mistal is not a finetune of LLaMA, it is a mode…

> I think Stable Diffusion was first to release a SOTA model (August 2022) that worked locally, not in language but image generation, but it set the tone for Meta. LLaMA only came in February 2023.

Sure, I was only covering LLMs though. If I wanted to cover image generation models and tools as well, the comment would be double its size.

> Mistal is not a finetune of LLaMA, it is a model trained from scratch. Also, Mistral was most of the time better than LLaMA during this period.

Oh, that's right. Iirc it was just the Llama 2 architecture that was used with sliding window attention.

> Quantization does not improve accuracy, except if you trade off precision for longer context maybe, but not on similar prompts. It is like JPEG compression, the original is always better for a specific image, but for the same byte size you get more resolution from JPEG than say... a PNG.

I'm well aware of how quantization works. I meant quantization methods were increasingly able to retain accuracy. Such as methods which quantize less important weights more heavily, improving accuracy for the same LLM size.

Re: Codestral Mamba

#132

Post by imtringued »

mbowcut2 wrote: post #54
causal wrote: post #38

Earlier quoted context omitted.

It should be corrected but the interesting aspect of this release is the architecture. To stay competitive while only needing linear inference time and supporting 256k context is pretty neat.

THIS. People don't realize the importance of Mamba competing on par with transformers.

Linear attention is terrible for chatbot style request-response applications, but if you're giving the model the prompt and then let it scan the codebase and fill in the middle, then linear attention should work pretty decently. The performance benefit should also have a much bigger impact, since you're reprocessing the same code over and over again.

Re: Codestral Mamba

#133

Post by razodactyl »

solarkraft wrote: post #112

I kinda just want something that can keep up with the original version of Copilot. It was so much better than the crap they’re pumping out now (keeps messing up syntax and only completing a few characters at a time).

Supposedly they were training on feedback provided by the plugin itself but that approach doesn't make sense to me because:

- I don't remember the shortcuts most of the time.

- When I run completions I double take and realise they're wrong.

- I am not a good source of data.

All this information is being fed back into the model as positive feedback. So perhaps reason for it to have gone downhill.

I recall it being amazing at coding back in the day, now I can't trust it.

Of course, it's anecdotal which is also problematic in itself but I have definitely noticed the issue where it will fail and stop autocompleting or provide completely irrelevant code.

Re: Codestral Mamba

#134

Post by M4v3R »

karolist wrote: post #119
heeton wrote: post #116

Earlier quoted context omitted.

Have you tried supermaven? It replaced copilot for me a couple of months ago.

I tried it, uses GPT-4o, the $10 sign up credit dissapeared in a few hours of intense coding, I'm not paying $500/mo for a fancy autocomple. Manual instruct style chat about code with Claude-Sonnet-3.5 is the best price/perf I've tried so far, through poe.com I use around 30k credits per day of coding of the 1M monthly allotment, I think it was $200/y. It's not available directly in my country. I've tried a bunch of…

Supermaven does not use GPT-4o for their autocompletions, they have their own model for that and for your paid subscription you get unlimited use of autocomplete.

There is a separate, optional feature called Supermaven Chat which is basically a wrapper around GPT-4o / Claude 3.5 where you can chat about fragments of your code, refactor them etc. This costs money, but you can also supply your own API key for OpenAI/Anthropic models. Or just use the web interface of any LLM if you just want to chat about your codebase.

Re: Codestral Mamba

#135

Post by Zambyte »

TechDebtDevin wrote: post #62
Agentus wrote: post #27

Earlier quoted context omitted.

I'm looking to run something on a 24gb GPU for the purpose of running wild with agentic use of LLMs. Is there anything worth trying that would fit on that amount of vRAM? Or are all the open-source PC-sized LLMs laughable still?

You can run the llama 70b based models faster than 10 tkn/s on 24gb vram. I've found that the quality of this class of LLMs is heavily swayed by your configuration and system prompting and results may vary. This Reddit post seems to have some input on the topic: https://www.reddit.com/r/LocalLLaMA/comments/1cj4det/llama_3... I haven't used any agent frameworks other than messing around with langchain a bit so I can't…

You would probably get the same tokens per second with llama 3 70b if you just unplugged the 24gb GPU. For something that actually fits in 24gb of VRAM, I recommend gemma 2 27b up to q6. I use q4 and it works quite well for my needs.

Re: Codestral Mamba

#136

Post by PoignardAzur »

lolinder wrote: post #98

I know it's just a throwaway line, but the bit about Cleopatra at the top feels in poor taste. It's completely inaccurate in that no one has ever attributed her death to a "mamba", and even the asp that some sources claim has been disputed. But even aside from that, it just feels weird that a human being's death has turned into a random reference you can make in a throwaway joke while advertising a product. They're c…

Also, saying that she had a "glorious destiny" is incredibly inaccurate historically. She was kinda crap at her job.

Re: Codestral Mamba

#138

Post by regularfry »

DalasNoin wrote: post #130
bhouston wrote: post #8

What are the steps required to get this running in VS Code? If they had linked to the instructions in their post (or better yet a link to a one click install of a VS Code Extension), it would help a lot with adoption. (BTW I consider it malpractice that they are at the top of hacker news with a model that is of great interest to a large portion of the users where and they do not have a monetizable call to action on t…

I feel like local models could be an amazing coding experience because you could disconnect from the internet. Usually I need to open chatgpt or google every so often to solve some issue or generate some function, but this also introduces so many distractions. imagine being able to turn off internet completely and only have a chat assistant that runs locally. I fear though that it is just going to be a bit to slow at…

I don't have a gut feel for how much difference the Mamba arch makes to inference speed, nor how much quantisation is likely to ruin things, but as a rough comparison Mistral-7B at 4 bits per param is very usable on CPU.

The issue with using any local models for code generation comes up with doing so in a professional context: you lose any infrastructure the provider might have for avoiding regurgitation of copyright code, so there's a legal risk there. That might not be a barrier in your context, but in my day-to-day it certainly is.

Re: Codestral Mamba

#139

Post by trissi »

thot_experiment wrote: post #66

Does anyone have a favorite FIM capable model? I've been using codellama-13b through ollama w/ a vim extension i wrote and it's okay but not amazing, I definitely get better code most of the time out of Gemma-27b but no FIM (and for some reason codellama-34b has broken inference for me)

I use deepseek-coder-7b-instruct-v1.5 & DeepSeek-Coder-V2-Lite-Instruct when I want speed & codestral-22B-v0.1 when I want smartness.

All of those are FIM capable, but especially deepseek-v2-lite is very picky with its prompt template so make sure you use it correctly...

Depending on your hardware codestral-22B might be fast enough for everything, but for me it's a bit to slow...

If you can run it deepseek v2 non-light is amazing, but it requires loads of VRAM

Post reply on HN