Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DBRX: A new open LLM

databricks.com

Post reply on HN
Print view View original

231–240 of 360 posts

Re: DBRX: A new open LLM

#231

Post by coldtea »

dheera wrote: post #200
djoldman wrote: post #4

Model card for base: https://huggingface.co/databricks/dbrx-base > The model requires ~264GB of RAM I'm wondering when everyone will transition from tracking parameter count vs evaluation metric to (total gpu RAM + total CPU RAM) vs evaluation metric. For example, a 7B parameter model using float32s will almost certainly outperform a 7B model using float4s. Additionally, all the examples of quantizing recently releas…

I'm more wondering when we'll have algorithms that will "do their best" given the resources they detect. That would be what I call artificial intelligence. Giving up because "out of memory" is not intelligence.

>Giving up because "out of memory" is not intelligence.

When people can't remember the facts/theory/formulas needed to answer some test question, or can't memorize some complicated information because it's too much, they usually give up too.

So, giving up because of "out of memory" sure sounds like intelligence to me.

Re: DBRX: A new open LLM

#232

Post by declaredapple »

dheera wrote: post #199
Mandelmus wrote: post #146

Earlier quoted context omitted.

And it appears to be at ~80 GB of RAM via quantisation.

That's a tricky number. Does it run on an 80GB GPU, does it auto-shave some parameters to fit in 79.99GB like any articifially "intelligent" piece of code would do, or does it give up like an unintelligent piece of code?

What?

Are you asking if the framework automatically quantizes/prunes the model on the fly?

Or are you suggesting the LLM itself should realize it's too big to run, and prune/quantize itself? Your references to "intelligent" almost leads me to the conclusion that you think the LLM should prune itself. Not only is this a chicken and egg problem, but LLMs are statistical models, they aren't inherently self bootstraping.

Re: DBRX: A new open LLM

#233

Post by Y_Y »

vlovich123 wrote: post #113
Y_Y wrote: post #13

Earlier quoted context omitted.

A free lunch? Wouldn't that be nice! Sometimes the quantization process improves the accuracy a little (probably by implicit regularization) but a model that's at or near capacity (as it should be) is necessarily hurt by throwing away most of the information. Language models often quantize well to small fixed-point types like int4, but it's not a magic wand.

I didn’t suggest a free lunch, just that the 8x reduction in RAM (+ faster processing) does not result in an 8x growth in the error. Thus a quantized model will outperform a non-quantized one on a evaluation/RAM metric.

That's not a good metric.

Re: DBRX: A new open LLM

#234

Post by sparks1970 »

stri8ed wrote: post #127
swalsh wrote: post #74

Earlier quoted context omitted.

The models are commodities, and the API's are even similar enough that there is zero stickiness. I can swap one model for another, and usually not have to change anything about my prompts or rag pipelines. For startups, the lesson here is don't be in the business of building models. Be in the business of using models. The cost of using AI will probably continue to trend lower for the foreseeable future... but you can…

Or be in the business of building infrastructure for AI inference.

Or be in the business of selling .ai domain names.

Re: DBRX: A new open LLM

#235

Post by lfmunoz4 »

bboygravity wrote: post #131

Less than 1 week after Nancy Pelosi bought a 5M USD share in Databricks, this news is published. https://twitter.com/PelosiTracker_/status/177119703064106223... Crime pays in the US.

I see these types of jokes everywhere. I cannot understand that hints of corruption are so blatant (i.e. a politician consistently beating the market) yet people keep voting for the same politician. Don't see how that is possible, must be these joke are only on internet and mainstream media never mentions this.

Re: DBRX: A new open LLM

#236

Post by samatman »

hiddencost wrote: post #162
jjtheblunt wrote: post #138

I’d like to know how Nancy Pelosi, who sure as hell doesn’t know what Apache Spark is, bought $1 million worth (and maybe $5million) of Databricks stock days ago. https://www.dailymail.co.uk/sciencetech/article-13228859/amp...

You know she has advisors, right?

If someone "advises" you that a company is about to do something major, and this isn't public information, and you take action on the stock market accordingly, that's insider trading.

Re: DBRX: A new open LLM

#237

Post by brucethemoose2 »

I would note the actual leading models right now (IMO) are:

- Miqu 70B (General Chat)

- Deepseed 33B (Coding)

- Yi 34B (for chat over 32K context)

And of course, there are finetunes of all these.

And there are some others in the 34B-70B range I have not tried (and some I have tried, like Qwen, which I was not impressed with).

Point being that Llama 70B, Mixtral and Grok as seen in the charts are not what I would call SOTA (though mixtral is excellent for the batch size 1 speed)

Re: DBRX: A new open LLM

#238

Post by hintymad »

Closi wrote: post #147
hintymad wrote: post #121

Earlier quoted context omitted.

Thanks! Why do they not focus on hosting other open models then? I suspect other models will soon catch up with their advantages in faster inference and better benchmark results. That said, maybe the advantage is aligned interests: they want customers to use their platforms, so they can keep their models open. In contrast, Mistral removed their commitment to open source as they found a potential path to profitability…

Demonstrating you can do it yourself shows a level of investment and commitment to AI in your platform that integrating LLAMA does not. And from a corporate perspective, it means that you have in-house capability to work at the cutting-edge of AI to be prepared for whatever comes next.

> Demonstrating you can do it yourself shows a level of investment and commitment to AI in your platform that integrating LLAMA does not.

I buy this argument. It looks that's not what AWS does, though, yet they don't have problem attracting LLM users. Maybe AWS already got enough reputation?

Re: DBRX: A new open LLM

#239

Post by gigatexal »

benrutter wrote: post #203
gigatexal wrote: post #72

data engineer here, offtopic, but am i the only guy tired of databricks shilling their tools as the end-all, be-all solutions for all things data engineering?

Lord no! I'm a data engineer also, feel the same. The part that I find most maddening is it seems pretty devoid from sincerely attempting to provide value. Things databricks offers that makes peoples lives easier: - Out the box kubernetes with no set up - Preconfigured spark Those are genuinely really useful, but then there's all this extra stuff that makes people's lives worse or drives bad practice: - Everything is…

Glad I’m not the only one. Especially with this notebook stuff they’re pushing. It’s an anti pattern I think.

Re: DBRX: A new open LLM

#240

Post by jph00 »

brucethemoose2 wrote: post #237

I would note the actual leading models right now (IMO) are: - Miqu 70B (General Chat) - Deepseed 33B (Coding) - Yi 34B (for chat over 32K context) And of course, there are finetunes of all these. And there are some others in the 34B-70B range I have not tried (and some I have tried, like Qwen, which I was not impressed with). Point being that Llama 70B, Mixtral and Grok as seen in the charts are not what I would call…

Miqu is a leaked model -- no license is provided to use it. Yi 34B doesn't allow commercial use. Deepseed 33B isn't much good at stuff outside of coding.

So it's fair to say that DBRX is the leading general purpose model that can be used commercially.

Post reply on HN