Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DBRX: A new open LLM

databricks.com

Post reply on HN
Print view View original

261–270 of 360 posts

Re: DBRX: A new open LLM

#261

Post by taneq »

MajimasEyepatch wrote: post #212
IshKebab wrote: post #95

Earlier quoted context omitted.

That sounds unsurprising? Like if you take any set of numbers, randomly split it in two, then calculate the average of each half... it's not surprising that they'll be almost the same. If you took two different training sets then it would be more surprising. Or am I misunderstanding what you mean?

It doesn't really matter whether you do this experiment with two training sets created independently or one training set split in half. As long as both are representative of the underlying population, you would get roughly the same results. In the case of human faces, as long as the faces are drawn from roughly similar population distributions (age, race, sex), you'll get similar results. There's only so much variati…

If not are sampled from the same population then they’re not really independent, even if they’re totally disjoint.

Re: DBRX: A new open LLM

#262

Post by espadrine »

swalsh wrote: post #77
djoldman wrote: post #4

Model card for base: https://huggingface.co/databricks/dbrx-base > The model requires ~264GB of RAM I'm wondering when everyone will transition from tracking parameter count vs evaluation metric to (total gpu RAM + total CPU RAM) vs evaluation metric. For example, a 7B parameter model using float32s will almost certainly outperform a 7B model using float4s. Additionally, all the examples of quantizing recently releas…

> The model requires ~264GB of RAM This feels as crazy as Grok. Was there a generation of models recently where we decided to just crank on the parameter count?

Not recently. GPT-3 from 2020 requires even more RAM; the open-source BLOOM from 2022 did too.

In my view, the main value of larger models is distillation (which we particularly witness, for instance, with how Claude Haiku matches release-day GPT-4 despite being less than a tenth of the cost). Hopefully the distilled models will be easier to run.

Re: DBRX: A new open LLM

#263

Post by bevekspldnw »

c1b wrote: post #257
bevekspldnw wrote: post #145

Earlier quoted context omitted.

I had gone the Mac Studio route initially, but I ended up with getting an A6000 for about the same price as a Mac and putting that in a Linux server onder my desk. Ollama makes it dead simple to serve it over my local network, so I can be on my M1 Air and using it no differently than if on my laptop. The difference is that the A6000 absolutely smokes the Mac.

> The difference is that the A6000 absolutely smokes the Mac. Memory Bandwidth : Mac Studio wins (about the same @ ~800) VRAM : Mac Studio wins (4x more) TFLOPs: A6000 wins (32 vs 38)

VRAM in excess of the model one is using isn’t useful per se. My use cases require high throughput, and on many tasks the A6000 executes inference at 2x speed.

Re: DBRX: A new open LLM

#264

Post by blueblimp »

ianbutler wrote: post #220

The approval on the base model is not feeling very open. Plenty of people still waiting on a chance to download it, where as the instruct model was an instant approval. The base model is more interesting to me for finetuning.

The license allows to reproduce/distribute/copy, so I'm a little surprised there's an approval process at all.

Re: DBRX: A new open LLM

#265

Post by zubairshaik »

hintymad wrote: post #238
Closi wrote: post #147

Earlier quoted context omitted.

Demonstrating you can do it yourself shows a level of investment and commitment to AI in your platform that integrating LLAMA does not. And from a corporate perspective, it means that you have in-house capability to work at the cutting-edge of AI to be prepared for whatever comes next.

> Demonstrating you can do it yourself shows a level of investment and commitment to AI in your platform that integrating LLAMA does not. I buy this argument. It looks that's not what AWS does, though, yet they don't have problem attracting LLM users. Maybe AWS already got enough reputation?

I may be misunderstanding, but doesn't Amazon have it's own models in the form of Amazon Titan[0]? I know they aren't competitive in terms of output quality but surely in terms of cost there can be some use cases for them.

[0] https://aws.amazon.com/bedrock/titan/

Re: DBRX: A new open LLM

#266

Post by rmbyrro »

hintymad wrote: post #238
Closi wrote: post #147

Earlier quoted context omitted.

Demonstrating you can do it yourself shows a level of investment and commitment to AI in your platform that integrating LLAMA does not. And from a corporate perspective, it means that you have in-house capability to work at the cutting-edge of AI to be prepared for whatever comes next.

> Demonstrating you can do it yourself shows a level of investment and commitment to AI in your platform that integrating LLAMA does not. I buy this argument. It looks that's not what AWS does, though, yet they don't have problem attracting LLM users. Maybe AWS already got enough reputation?

It's easier because 70% of the market already has an AWS account and a sizeable budget allocated to it. The technical team is literally one click away from any AWS service.

Re: DBRX: A new open LLM

#267

Post by llm_trw »

XCSme wrote: post #39
llm_trw wrote: post #18

Earlier quoted context omitted.

>If the GPU has 16GB of VRAM, and the model is 70GB, can it still run well? Also, does it run considerably better than on a GPU with 12GB of VRAM? No, it can't run at all. >I run Ollama locally, mixtral works well (7B, 3.4GB) on a 1080ti, but the 24.6GB version is a bit slow (still usable, but has a noticeable start-up time). That is not mixtral, that is mistral 7b. The 1080ti is slower than running inference on curr…

> No, it can't run at all. https://s3.amazonaws.com/i.snag.gy/ae82Ym.jpg EDIT: This was ran on a 1080ti + 5900x. Initial generation takes around 10-30seconds (like it has to upload the model to GPU), but then it starts answering immediately, at around 3 words per second.

Congratulations on using CPU inference.

Re: DBRX: A new open LLM

#268

Post by dumbfounder »

arcticfox wrote: post #242
bobbylarrybobby wrote: post #221

Earlier quoted context omitted.

I mean, faces are faces, right? If the training data set is large and representative I don't see why any two (representative) halves of the data would lead to significantly different models.

I think that's the point; language is language. If there's some fundamental limit of what type of intelligence the current breed of LLMs can extract from language, at some point it doesn't matter how good or expansive the content of the training set is. Maybe we are finally starting to hit an architectural limit at this point.

But information is not information. They may be able to talk in the same style, but not about the same things.

Re: DBRX: A new open LLM

#269

Post by belter »

brucethemoose2 wrote: post #237

I would note the actual leading models right now (IMO) are: - Miqu 70B (General Chat) - Deepseed 33B (Coding) - Yi 34B (for chat over 32K context) And of course, there are finetunes of all these. And there are some others in the 34B-70B range I have not tried (and some I have tried, like Qwen, which I was not impressed with). Point being that Llama 70B, Mixtral and Grok as seen in the charts are not what I would call…

For all the Model Cards and License notices, I find it interesting there is not much information on the contents of the dataset used for training. Specifically, if it contains data subject to Copyright restrictions. Or did I miss that?

Re: DBRX: A new open LLM

#270

Post by jfrankle »

zopper wrote: post #194

Interesting that they haven't release DBRX MoE-A and B. For many use-cases, smaller models are sufficient. Wonder why that is?

Honestly, just a matter of having the time to clean everything up and get it out. The ancillary code, model cards, etc. take a surprising amount of time.
Post reply on HN