Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DBRX: A new open LLM

databricks.com

Post reply on HN
Print view View original

61–70 of 360 posts

Re: DBRX: A new open LLM

#61

Post by jobigoud »

emmender2 wrote: post #46

this proves that all llm models converge to a certain point when trained on the same data. ie, there is really no differentiation between one model or the other. Claims about out-performance on tasks are just that, claims. the next iteration of llama or mixtral will converge. LLMs seem to evolve like linux/windows or ios/android with not much differentiation in the foundation models.

It's even possible they converge when trained on different data, if they are learning some underlying representation. There was recent research on face generation where they trained two models by splitting one training set in two without overlap, and got the two models to generate similar faces for similar conditioning, even though each model hadn't seen anything that the other model had.

Re: DBRX: A new open LLM

#63

Post by sangnoir »

hexomancer wrote: post #23
mochomocha wrote: post #16

Earlier quoted context omitted.

It's a MoE model, so it offers a different memory/compute latency trade-off than standard dense models. Quoting the blog post: > DBRX uses only 36 billion parameters at any given time. But the model itself is 132 billion parameters, letting you have your cake and eat it too in terms of speed (tokens/second) vs performance (quality).

Mixtral is also a MoE model, hence the name: mix tral.

Despite both being MoEs, thr architectures are different. DBRX has double the number of experts in the pool (16 vs 8 for Mixtral), and doubles the active experts (4 vs 2)

Re: DBRX: A new open LLM

#64

Post by wokwokwok »

XCSme wrote: post #39
llm_trw wrote: post #18

Earlier quoted context omitted.

>If the GPU has 16GB of VRAM, and the model is 70GB, can it still run well? Also, does it run considerably better than on a GPU with 12GB of VRAM? No, it can't run at all. >I run Ollama locally, mixtral works well (7B, 3.4GB) on a 1080ti, but the 24.6GB version is a bit slow (still usable, but has a noticeable start-up time). That is not mixtral, that is mistral 7b. The 1080ti is slower than running inference on curr…

> No, it can't run at all. https://s3.amazonaws.com/i.snag.gy/ae82Ym.jpg EDIT: This was ran on a 1080ti + 5900x. Initial generation takes around 10-30seconds (like it has to upload the model to GPU), but then it starts answering immediately, at around 3 words per second.

Did you check your GPU utilization?

Typically when it runs that way it runs on the CPU, not the GPU.

Are you sure you're actually offloading any work to the GPU?

At least with llama.cpp, there is no 'partially put a layer' into the GPU. Either you do, or you don't. You pick the number of layers. If the model is too big, the layers won't fit and it can't run at all.

The llama.cpp `main` executable will tell you in it's debug information when you use the -ngl flag; see https://github.com/ggerganov/llama.cpp/blob/master/examples/...

It's also possible you're running (eg. if you're using ollama) and quantized version of the model which reduces the memory requirements and quality of the model outputs.

Re: DBRX: A new open LLM

#65

Post by throwaway74432 »

emmender2 wrote: post #46

this proves that all llm models converge to a certain point when trained on the same data. ie, there is really no differentiation between one model or the other. Claims about out-performance on tasks are just that, claims. the next iteration of llama or mixtral will converge. LLMs seem to evolve like linux/windows or ios/android with not much differentiation in the foundation models.

LLMs are a commodity

https://www.investopedia.com/terms/c/commodity.asp

Re: DBRX: A new open LLM

#66

Post by occamrazor »

CharlesW wrote: post #36
hn_acker wrote: post #31

Even though the README.md calls the license the Databricks Open Source License, the LICENSE file includes paragraphs such as > You will not use DBRX or DBRX Derivatives or any Output to improve any other large language model (excluding DBRX or DBRX Derivatives). and > If, on the DBRX version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates,…

> This is a source-available model, not an open model. To me, "source available" implies that everything you need to reproduce the model is also available, and that doesn't appear to be the case. How is the resulting model more "free as in freedom" than a compiled binary?

I like:

- “open weights” for no training data and no restrictions on use,

- “weights available” for no training data and restrictions on use, like in this case.

Re: DBRX: A new open LLM

#67

Post by loudmax »

XCSme wrote: post #42
PheonixPharts wrote: post #21

Earlier quoted context omitted.

While GPUs are still the kings of speed, if you are worried about VRAM I do recommend a maxed out Mac Studio. Llama.cpp + quantized models on Apple Silicon is an incredible experience, and having 192 GB of unified memory to work with means you can run models that just aren't feasible on a home GPU setup. It really boils down to what type of local development you want to do. I'm mostly experimenting with things where…

I already have 128GB of RAM (DDR4), and was wondering if upgrading from a 1080ti (12GB) to a 4070ti super (16GB), would make a big difference. I assume the FP32 and FP16 operations are already a huge improvement, but also the 33% increased VRAM might lead to fewer swaps between VRAM and RAM.

I have an RTX 3080 with 10GB of VRAM. I'm able to run models larger than 10GB using llama.cpp and offloading to the GPU as much as can fit into VRAM. The remainder of the model runs on CPU + regular RAM.

The `nvtop` command displays a nice graph of how much GPU processing and VRAM is being consumed. When I run a model that fits entirely into VRAM, say Mistral 7B, nvtop shows the GPU processing running at full tilt. When I run a model bigger than 10GB, say Mixtral or Llama 70B with GPU offloading, my CPU will run full tilt and the VRAM is full, but the GPU processor itself will operate far below full capacity.

I think what is happening here is that the model layers that are offloaded to the GPU do their processing, then the GPU spends most of the time waiting for the much slower CPU to do its thing. So in my case, I think upgrading to a faster GPU would make little to no difference when running the bigger models, so long as the VRAM is capped at the same level. But upgrading to a GPU with more VRAM, even a slower GPU, should make the overall speed faster for bigger models because the GPU would spend less time waiting for the CPU. (Of course, models that fit entirely into VRAM will run faster on a faster GPU).

In my case, the amount of VRAM absolutely seems to be the performance bottleneck. If I do upgrade, it will be for a GPU with more VRAM, not necessarily a GPU with more processing power. That has been my experience running llama.cpp. YMMV.

Re: DBRX: A new open LLM

#68

Post by comp_raccoon »

shnkr wrote: post #12
simonw wrote: post #6

Earlier quoted context omitted.

The most detailed answer to that I've seen is the original LLaMA paper, which described exactly what that model was trained on (including lots of scraped copyrighted data) https://arxiv.org/abs/2302.13971 Llama 2 was much more opaque about the training data, presumably because they were already being sued at that point (by Sarah Silverman!) over the training data that went into the first Llama! A couple of things I'v…

my question was specific to databricks model. If it followed llama or openai, they could add a line or two about it .. make the blog complete.

they have a technical report coming! knowing the team, they will do a great job disclosing as much as possible.

Re: DBRX: A new open LLM

#69

Post by pandastronaut »

tylermw wrote: post #59
pandastronaut wrote: post #53

Earlier quoted context omitted.

Even starting at 30%, the MMLU graph is false. The four bars are wrong. Even their own 73,7% is not at the right height. The Mixtral 71.4% is below the 70% mark of the axis. This is really the kind of marketing trick that makes me avoid a provider / publisher. I can't build trust this way.

I believe they are using the percentages as part of the height of the bar chart! I thought I'd seen every way someone could do dataviz wrong (particularly with a bar chart), but this one is new to me.

Interesting! It is probably one of the worst trick I have seen in a while for a bar graph. Never seen this one before. Trust vanishes instantly facing that kind of dataviz.

Re: DBRX: A new open LLM

#70

Post by renewiltord »

mpeg wrote: post #35

The scale on that bar chart for "Programming (Human Eval)" is wild. Manager: "looks ok, but can you make our numbers pop? just make the LLaMa bar smaller"

Yeah, this is why I ask climate scientists to use a proper 0 K graph but they always zoom it in to exaggerate climate change. Display correctly with 0 included and you’ll see that climate change isn’t a big deal.

It’s a common marketing and fear mongering trick.

Post reply on HN