Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Codestral: Mistral's Code Model

mistral.ai

Post reply on HN
Print view View original

191–200 of 225 posts

Re: Codestral: Mistral's Code Model

#191

Post by Terretta »

daghamm wrote: post #190
Terretta wrote: post #178

Earlier quoted context omitted.

Yes, RAM requirement is BnL same for GPU and using the metal/GPU in Apple Silicon. Running LLM models on a MacBook Pro with Apple Silicon vs. a PC with an Nvidia 4090 GPU has trade-offs. My 128GB MacBook Pro handles models using up to 96GB of unified memory, running at a little under half the speed of a 4090. If you use a quantized version of full floating point model, you can run the largest open models available. W…

"It's not clear to me why Macbooks and Mac Studio Ultras with maxed out RAM aren't selling" Aren't these machines extremly expensive and generally not upgradable?

They cost less than the 4 - 8 graphics cards and monster desktop PC that would be needed, plus, hey, it's a laptop.

Re: Codestral: Mistral's Code Model

#192

Post by rldjbpin »

foobiekr wrote: post #65
analyte123 wrote: post #48

The license for this [1] prohibits use of the model and its outputs for any commercial activity, or even any "live" (whatever that means) conditions, commercial or not. There seems to be an exclusion for using the code outputs as part of "development". But wait! It also prohibits "any internal usage by employees in the context of the company's business activities". However you interpret these clauses, this puts their…

There's some irony in the fact that people will ignore this license in exactly the same way Mistral and all the other LLM guys ignore the copyright and licensing on the works they ingest.

should it be morally ok to not follow these kinds of license, maybe except when you are selling a service without making any changes? i wonder what people visiting this site thinks about this.

Re: Codestral: Mistral's Code Model

#193

Post by Terretta »

tyfon wrote: post #186
Terretta wrote: post #178

Earlier quoted context omitted.

Yes, RAM requirement is BnL same for GPU and using the metal/GPU in Apple Silicon. Running LLM models on a MacBook Pro with Apple Silicon vs. a PC with an Nvidia 4090 GPU has trade-offs. My 128GB MacBook Pro handles models using up to 96GB of unified memory, running at a little under half the speed of a 4090. If you use a quantized version of full floating point model, you can run the largest open models available. W…

I have a 5940x with 128 gb ram. It's a bit slower perhaps than the mac, but i get the best of both worlds. That is I get a lot of RAM to hold the model and I can offload as much of it as possible to the GPU. This works especially well with models like mixtral 8x22, but also models like llama3 and the old large bloom model. I also get the utility of running Linux instead of the closed up mac os. But running large mode…

I get the utility of a laptop that runs 20 hours on battery and slips in the side pocket of my carry-on or shoulder bag. (The Mac can also split between RAM and GPU.) Mixtral 8x22 and Llama 3 70b stream at roughly the same speed as last year's GPT-4.

> closed up MacOS

https://github.com/apple-oss-distributions/distribution-macO...

  curl https://alx.sh | sh
https://asahilinux.org/

I prefer the "utility" of BSDs, but that's just a preference.

Re: Codestral: Mistral's Code Model

#194

Post by aitchnyu »

belter wrote: post #115
foobiekr wrote: post #65

Earlier quoted context omitted.

There's some irony in the fact that people will ignore this license in exactly the same way Mistral and all the other LLM guys ignore the copyright and licensing on the works they ingest.

And nobody will sue anybody because suing means...discovery....

Umm, are you saying its mutually assured destruction for the book pirate (AI company) and api wrapper startup?

Re: Codestral: Mistral's Code Model

#195

Post by Zambyte »

demosthanos wrote: post #130
Zambyte wrote: post #101

Earlier quoted context omitted.

The fact that I can downloaded it and run it myself is a pretty meaningful amount of openness to me. I can easily ignore their bogus claims about what I'm allowed to do with it due to their distribution model. I can't necessarily do the same with a propriety service, as they can cut me off if the way I use the output makes them sad :(

> I can easily ignore their bogus claims about what I'm allowed to do with it due to their distribution model. If you're talking about exclusively personally use, sure. If you're talking about a business setting in a jurisdiction that Mistral can sue in, not so much. Being able to use it in a business setting is a pretty darn important part of what Open Source has always meant (it's why it exists as a term at all).

> If you're talking about a business setting in a jurisdiction that Mistral can sue in, not so much.

I'm reminded of the Japanese concept called Sosumi :)

> Being able to use it in a business setting is a pretty darn important part of what Open Source has always meant (it's why it exists as a term at all).

I'm quite familiar with the history of that term, but neither I nor Mistral used it. None of their models have been open source; they have been open weight. You can argue that they are actually "weight available" given the terms they write next to the download link, but since there has been no ruling on whether weights themselves are covered by copyright (and I think that would be terribly bogus if they are), I simply choose not to care what they write in their "terms of use".

Re: Codestral: Mistral's Code Model

#196

Post by stalked_why »

piskov wrote: post #142
isaacrolandson wrote: post #135

Will this run on an M3 48GB?

You’ll need 44GB just for the weights By default only 75% of unified memory is available to GPU if you have >36GB. So with 48 total only 36 is available for GPU with is lower than 44. tldr; without quantization you will not be able to run it.

https://medium.com/@tedisaacs/codestral-for-ai-coding-models...

This article says 88GB without quantization. Though it then goes on to make a ridiculous claim that if you had 128GB of RAM, then using up 88GB of 128GB would make everything else really slow because I guess the think the remaining 40GB of RAM somehow isn't enough for your OS and desktop apps.

So it’s probably not a very authoritative source.

Re: Codestral: Mistral's Code Model

#197

Post by lioeters »

Sytten wrote: post #150

Is there a vscode extension that could plug any model out there and have a similar experience to copilot. I always want to try them but I cant be bothered to do a whole setup each time.

Haven't used it personally, but someone up-thread mentioned:

> Llama Coder is a better and self-hosted Github Copilot replacement for VS Code

https://github.com/ex3ndr/llama-coder

And:

> Open-source VS Code and JetBrains extensions that enable you to easily create your own modular AI software development system

https://github.com/continuedev/continue

Re: Codestral: Mistral's Code Model

#198

Post by [deleted] »

Terretta wrote: post #191
daghamm wrote: post #190

Earlier quoted context omitted.

"It's not clear to me why Macbooks and Mac Studio Ultras with maxed out RAM aren't selling" Aren't these machines extremly expensive and generally not upgradable?

They cost less than the 4 - 8 graphics cards and monster desktop PC that would be needed, plus, hey, it's a laptop.

[deleted]

Re: Codestral: Mistral's Code Model

#199

Post by talldayo »

Terretta wrote: post #193
tyfon wrote: post #186

Earlier quoted context omitted.

I have a 5940x with 128 gb ram. It's a bit slower perhaps than the mac, but i get the best of both worlds. That is I get a lot of RAM to hold the model and I can offload as much of it as possible to the GPU. This works especially well with models like mixtral 8x22, but also models like llama3 and the old large bloom model. I also get the utility of running Linux instead of the closed up mac os. But running large mode…

I get the utility of a laptop that runs 20 hours on battery and slips in the side pocket of my carry-on or shoulder bag. (The Mac can also split between RAM and GPU.) Mixtral 8x22 and Llama 3 70b stream at roughly the same speed as last year's GPT-4. > closed up MacOS https://github.com/apple-oss-distributions/distribution-macO... curl https://alx.sh | sh https://asahilinux.org/ I prefer the "utility" of BSDs, but th…

Asahi is Fischer-Price tiers of support compared to what even Nvidia, the most loathed OEM on Linux, provides for free to their users. If that's the best option available, it should be no wonder that server customers are avoiding Apple like the plague. Apple has to beg their audience to reverse-engineer their own OpenCL drivers if they want them; Nvidia ships them alongside CUDA. These two companies are not the same.

Have you ever seen the inside of a datacenter? Why is it that surprising to you that nobody perks up when you start waxing on about battery life? Even terms of power-to-performance, Apple's latest chips get ethered by Nvidia's server offerings.

This "Apple for Inference" meme is so dead that I can only feel sad when I see people unironically promoting it. You actually think serious customers are going to load up Asahi (even funnier, MacOS) on their Mac Pro... so they can inference half as fast as a single Blackwell GPU? You think the industry is doing this shit? I don't even think the Steve Jobs apologists are dumb enough to fall for this one, you must be a particularly aspirational shareholder.

Re: Codestral: Mistral's Code Model

#200

Post by poopypoopington »

Liquix wrote: post #152
IshKebab wrote: post #148

Earlier quoted context omitted.

> So basically I, as an open source author, had my code eaten up by Mistral without my consent Not necessarily. You consented to people reading your code and learning from it when you posted it on Github. Whether or not there's an issue with AI doing the same remains to be settled. It certainly isn't clear cut that separate consent would be required.

MIT/BSD code is fair game, but isn't the whole point of GPL/AGPL "you can read and share and use this, but you can't take it and roll it into your closed commercial product for profit"? It seems like what Mistral and co are doing is a fundamental violation of the one thing GPL is striving to enforce.

Is there an updated version of these license(s) that explicitly excludes projects from being used for training of AIs?
Post reply on HN