Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DBRX: A new open LLM

databricks.com

Post reply on HN
Print view View original

21–30 of 360 posts

Re: DBRX: A new open LLM

#21

Post by PheonixPharts »

XCSme wrote: post #11

I am planning to buy a new GPU. If the GPU has 16GB of VRAM, and the model is 70GB, can it still run well? Also, does it run considerably better than on a GPU with 12GB of VRAM? I run Ollama locally, mixtral works well (7B, 3.4GB) on a 1080ti, but the 24.6GB version is a bit slow (still usable, but has a noticeable start-up time).

While GPUs are still the kings of speed, if you are worried about VRAM I do recommend a maxed out Mac Studio.

Llama.cpp + quantized models on Apple Silicon is an incredible experience, and having 192 GB of unified memory to work with means you can run models that just aren't feasible on a home GPU setup.

It really boils down to what type of local development you want to do. I'm mostly experimenting with things where the time to response isn't that big of a deal, and not fine-tuning the models locally (which I also believe GPUs are still superior for). But if your concern is "how big of a model can I run" vs "Can I have close to real time chat", the unified memory approach is superior.

Re: DBRX: A new open LLM

#22

Post by K0balt »

Y_Y wrote: post #13
vlovich123 wrote: post #10

Earlier quoted context omitted.

I thought float4 sacrificed a negligible cost in evaluation quality for a 8x reduction in RAM?

A free lunch? Wouldn't that be nice! Sometimes the quantization process improves the accuracy a little (probably by implicit regularization) but a model that's at or near capacity (as it should be) is necessarily hurt by throwing away most of the information. Language models often quantize well to small fixed-point types like int4, but it's not a magic wand.

I find that q6 and 5+ are subjectively as good as raw tensor files. 4 bit quality reduction is very detectable though. Of course there must be a loss of information, but perhaps there is a noise floor or something like that.

Re: DBRX: A new open LLM

#23

Post by hexomancer »

mochomocha wrote: post #16
natsucks wrote: post #14

it's twice the size of mixtral and barely beats it.

It's a MoE model, so it offers a different memory/compute latency trade-off than standard dense models. Quoting the blog post: > DBRX uses only 36 billion parameters at any given time. But the model itself is 132 billion parameters, letting you have your cake and eat it too in terms of speed (tokens/second) vs performance (quality).

Mixtral is also a MoE model, hence the name: mixtral.

Re: DBRX: A new open LLM

#24

Post by Icko »

viktour19 wrote: post #8

It's great how we went from "wait.. this model is too powerful to open source" to everyone trying to shove down their 1% improved model down the throats of developers

I'm 90% certain that OpenAI has some much beefier model they are not releasing - remember the Q* rumour?

Re: DBRX: A new open LLM

#25

Post by chpatrick »

jasonjmcghee wrote: post #15
XCSme wrote: post #11

I am planning to buy a new GPU. If the GPU has 16GB of VRAM, and the model is 70GB, can it still run well? Also, does it run considerably better than on a GPU with 12GB of VRAM? I run Ollama locally, mixtral works well (7B, 3.4GB) on a 1080ti, but the 24.6GB version is a bit slow (still usable, but has a noticeable start-up time).

> mixtral works well Do you mean mistral? mixtral is 8x7B and requires like 100GB of RAM Edit: (without quant as others have pointed out) can definitely be lower, but haven't heard of a 3.4GB version

The quantized one works fine on my 24GB 3090.

Re: DBRX: A new open LLM

#26

Post by ingenieroariel »

TLDR: A model that could be described as "3.8 level" that is good at math and openly available with a custom license.

It is as fast as 34B model, but uses as much memory as a 132B model. A mixture of 16 experts, activates 4 at a time, so has more chances to get the combo just right than Mixtral (8 with 2 active).

For my personal use case (a top of the line Mac Studio) it looks like the perfect size to replace GPT-4 turbo for programming tasks. What we should look out for is people using them for real world programming tasks (instead of benchmarks) and reporting back.

Re: DBRX: A new open LLM

#28

Post by [deleted] »

ingenieroariel wrote: post #26

TLDR: A model that could be described as "3.8 level" that is good at math and openly available with a custom license. It is as fast as 34B model, but uses as much memory as a 132B model. A mixture of 16 experts, activates 4 at a time, so has more chances to get the combo just right than Mixtral (8 with 2 active). For my personal use case (a top of the line Mac Studio) it looks like the perfect size to replace GPT-4 t…

[deleted]

Re: DBRX: A new open LLM

#29

Post by brainless »

viktour19 wrote: post #8

It's great how we went from "wait.. this model is too powerful to open source" to everyone trying to shove down their 1% improved model down the throats of developers

I feel quite the opposite. Improvements, even tiny ones are great. But what's more important is that more companies release under open license.

Training models isn't cheap. Individuals can't easily do this, unlike software development. So we need companies to do this for the foreseeable future.

Re: DBRX: A new open LLM

#30

Post by _ea1k »

kwerk wrote: post #17
jasonjmcghee wrote: post #15

Earlier quoted context omitted.

> mixtral works well Do you mean mistral? mixtral is 8x7B and requires like 100GB of RAM Edit: (without quant as others have pointed out) can definitely be lower, but haven't heard of a 3.4GB version

I have two 3090s and it runs fine with `ollama run mixtral`. Although OP definitely meant mistral with the 7B note

ollama run mixtral will default to the quantized version (4bit IIRC). I'd guess this is why it can fit with two 3090s.
Post reply on HN