Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DBRX: A new open LLM

databricks.com

Post reply on HN
Print view View original

281–290 of 360 posts

Re: DBRX: A new open LLM

#281

Post by nerpderp82 »

pandastronaut wrote: post #53
glutamate wrote: post #45

Earlier quoted context omitted.

I think the case for "axis must always go to 0" is overblown. Zero isn't always meaningful, for instance chance performance or performance of trivial algorithms is likely >0%. Sometimes if axis must go to zero you can't see small changes. For instance if you plot world population 2014-2024 on an axis going to zero, you won't be able to see if we are growing or shrinking.

Even starting at 30%, the MMLU graph is false. The four bars are wrong. Even their own 73,7% is not at the right height. The Mixtral 71.4% is below the 70% mark of the axis. This is really the kind of marketing trick that makes me avoid a provider / publisher. I can't build trust this way.

MMLU is not a good benchmark and needs to stop being used.

I can't find the section, but at the end of one of https://www.youtube.com/@aiexplained-official/videos he runs down a deep dive of the questions and answers in MMLU, and there are so many typos, omissions, and errors in the questions and the answers that it should no longer be used.

This is it, with the corret time offset into the video https://www.reddit.com/r/OpenAI/comments/18i02oe/mmlu_is_not...

The original longer complaint against MMLU https://www.youtube.com/watch?v=hVade_8H8mE

Re: DBRX: A new open LLM

#282

Post by wongarsu »

ok_dad wrote: post #279
jph00 wrote: post #240

Earlier quoted context omitted.

Miqu is a leaked model -- no license is provided to use it. Yi 34B doesn't allow commercial use. Deepseed 33B isn't much good at stuff outside of coding. So it's fair to say that DBRX is the leading general purpose model that can be used commercially.

Model weights are just constants in a mathematical equation, they aren’t copyrightable. It’s questionable whether licenses to use them only for certain purposes are even enforceable. No human wrote the weights so they aren’t a work of art/authorship by a human. Just don’t use their services, use the weights at home on your machines so you don’t bypass some TOS.

Photographs aren't human-made either, yet they are copyrightable. I agree that both the letter and the spirit of copyright law are in favor of models not being copyrightable, but it will take years until there's a court ruling either way. Until then proceed with caution and expect rights holders to pretend like copyright does apply. Not that they will come after your home setup either way.

Re: DBRX: A new open LLM

#283

Post by K0balt »

Taek wrote: post #243
K0balt wrote: post #22

Earlier quoted context omitted.

I find that q6 and 5+ are subjectively as good as raw tensor files. 4 bit quality reduction is very detectable though. Of course there must be a loss of information, but perhaps there is a noise floor or something like that.

At what parameter count? Its been established that quantization has less of an effect on larger models. By the time you are at 70B quantization to 4 bits basically is negligible

I work mostly with mixtral and mistral 7b these days, but I did work with some 70b models before mistral came out, and I was not impressed with the 4 bit Llama-2 70b.

Re: DBRX: A new open LLM

#284

Post by batshit_beaver »

fragmede wrote: post #218
n2d4 wrote: post #166

Earlier quoted context omitted.

There's at least an argument to be made that this is because all the models are heavily trained on GPT-4 outputs (or whatever the SOTA happens to be during training). All those models are, in a way, a product of inbreeding.

But is it the kind of inbreeding that gets you Downs, or the kwisatz haderach?

Yes

Re: DBRX: A new open LLM

#285

Post by raincole »

hipadev23 wrote: post #277
jph00 wrote: post #240

Earlier quoted context omitted.

Miqu is a leaked model -- no license is provided to use it. Yi 34B doesn't allow commercial use. Deepseed 33B isn't much good at stuff outside of coding. So it's fair to say that DBRX is the leading general purpose model that can be used commercially.

[flagged]

Absolutely.

People say LLM are foundamentally just statistics so training one on copyrightable materials is okay. Well perhaps, but pure statistics data are not copyrightable. Feel free to use leaked models.

Re: DBRX: A new open LLM

#286

Post by panarky »

briandw wrote: post #119

Worse than the chart crime of truncating the y axis is putting LLaMa2's Human Eval scores on there and not comparing it to Code Llama Instruct 70b. DBRX still beats Code Llama Instruct's 67.8 but not by that much.

> chart crime of truncating the y axis

If you chart the temperature of the ocean do you keep the y-axis anchored at zero Kelvin?

Re: DBRX: A new open LLM

#287

Post by petesergeant »

This makes me bearish on OpenAI as a company. When a cloud company can offer a strong model for free by selling the compute, what competitive advantage does a company who want you to pay for the model have left? Feels like they might get Netscape’d.

Re: DBRX: A new open LLM

#288

Post by [deleted] »

brucethemoose2 wrote: post #237

I would note the actual leading models right now (IMO) are: - Miqu 70B (General Chat) - Deepseed 33B (Coding) - Yi 34B (for chat over 32K context) And of course, there are finetunes of all these. And there are some others in the 34B-70B range I have not tried (and some I have tried, like Qwen, which I was not impressed with). Point being that Llama 70B, Mixtral and Grok as seen in the charts are not what I would call…

[deleted]

Re: DBRX: A new open LLM

#289

Post by [deleted] »

emmender2 wrote: post #46

this proves that all llm models converge to a certain point when trained on the same data. ie, there is really no differentiation between one model or the other. Claims about out-performance on tasks are just that, claims. the next iteration of llama or mixtral will converge. LLMs seem to evolve like linux/windows or ios/android with not much differentiation in the foundation models.

[deleted]

Re: DBRX: A new open LLM

#290

Post by Zuiii »

wongarsu wrote: post #282
ok_dad wrote: post #279

Earlier quoted context omitted.

Model weights are just constants in a mathematical equation, they aren’t copyrightable. It’s questionable whether licenses to use them only for certain purposes are even enforceable. No human wrote the weights so they aren’t a work of art/authorship by a human. Just don’t use their services, use the weights at home on your machines so you don’t bypass some TOS.

Photographs aren't human-made either, yet they are copyrightable. I agree that both the letter and the spirit of copyright law are in favor of models not being copyrightable, but it will take years until there's a court ruling either way. Until then proceed with caution and expect rights holders to pretend like copyright does apply. Not that they will come after your home setup either way.

Photos involve creativity. Photos that don't involve creativity aren't usually considered copyrightable by the courts (hence why all copyright cases I followed include arguments that establish why creativity was a material to creating the work).

Weights, on the other hand, are a product of a purely mechanical process. Sure, the process itself required creativity as did the composition of data, but the creation of the weights themselves do not.

Model weights are effectively public domain data according to the criteria outlined in statement issued by the US copyright office a year ago: https://www.federalregister.gov/documents/2023/03/16/2023-05...

Post reply on HN