Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DBRX: A new open LLM

databricks.com

Post reply on HN
Print view View original

51–60 of 360 posts

Re: DBRX: A new open LLM

#51

Post by nilstycho »

glutamate wrote: post #45
mpeg wrote: post #35

The scale on that bar chart for "Programming (Human Eval)" is wild. Manager: "looks ok, but can you make our numbers pop? just make the LLaMa bar smaller"

I think the case for "axis must always go to 0" is overblown. Zero isn't always meaningful, for instance chance performance or performance of trivial algorithms is likely >0%. Sometimes if axis must go to zero you can't see small changes. For instance if you plot world population 2014-2024 on an axis going to zero, you won't be able to see if we are growing or shrinking.

I agree with your general point, but world population is still visibly increasing on that interval.

https://ourworldindata.org/explorers/population-and-demograp...

Perhaps "global mean temperature in Kelvin" would be a comparable example.

Re: DBRX: A new open LLM

#52

Post by whimsicalism »

hn_acker wrote: post #31

Even though the README.md calls the license the Databricks Open Source License, the LICENSE file includes paragraphs such as > You will not use DBRX or DBRX Derivatives or any Output to improve any other large language model (excluding DBRX or DBRX Derivatives). and > If, on the DBRX version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates,…

identical to llama fwiw

Re: DBRX: A new open LLM

#53

Post by pandastronaut »

glutamate wrote: post #45
mpeg wrote: post #35

The scale on that bar chart for "Programming (Human Eval)" is wild. Manager: "looks ok, but can you make our numbers pop? just make the LLaMa bar smaller"

I think the case for "axis must always go to 0" is overblown. Zero isn't always meaningful, for instance chance performance or performance of trivial algorithms is likely >0%. Sometimes if axis must go to zero you can't see small changes. For instance if you plot world population 2014-2024 on an axis going to zero, you won't be able to see if we are growing or shrinking.

Even starting at 30%, the MMLU graph is false. The four bars are wrong. Even their own 73,7% is not at the right height. The Mixtral 71.4% is below the 70% mark of the axis. This is really the kind of marketing trick that makes me avoid a provider / publisher. I can't build trust this way.

Re: DBRX: A new open LLM

#55

Post by mnemoni_c »

emmender2 wrote: post #46

this proves that all llm models converge to a certain point when trained on the same data. ie, there is really no differentiation between one model or the other. Claims about out-performance on tasks are just that, claims. the next iteration of llama or mixtral will converge. LLMs seem to evolve like linux/windows or ios/android with not much differentiation in the foundation models.

Yea it feels like transformer LLMs are in or getting closer to diminishing returns. Will need some new breakthrough, likely entirely new approach, to get to AGI levels

Re: DBRX: A new open LLM

#56

Post by TZubiri »

glutamate wrote: post #45
mpeg wrote: post #35

The scale on that bar chart for "Programming (Human Eval)" is wild. Manager: "looks ok, but can you make our numbers pop? just make the LLaMa bar smaller"

I think the case for "axis must always go to 0" is overblown. Zero isn't always meaningful, for instance chance performance or performance of trivial algorithms is likely >0%. Sometimes if axis must go to zero you can't see small changes. For instance if you plot world population 2014-2024 on an axis going to zero, you won't be able to see if we are growing or shrinking.

Then you can plot it on a greater timescale, or plot the change rate

Re: DBRX: A new open LLM

#57

Post by [deleted] »

hn_acker wrote: post #31

Even though the README.md calls the license the Databricks Open Source License, the LICENSE file includes paragraphs such as > You will not use DBRX or DBRX Derivatives or any Output to improve any other large language model (excluding DBRX or DBRX Derivatives). and > If, on the DBRX version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates,…

[deleted]

Re: DBRX: A new open LLM

#58

Post by adolph »

hn_acker wrote: post #31

Even though the README.md calls the license the Databricks Open Source License, the LICENSE file includes paragraphs such as > You will not use DBRX or DBRX Derivatives or any Output to improve any other large language model (excluding DBRX or DBRX Derivatives). and > If, on the DBRX version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates,…

Maybe the license is “open” as in a can of beer, not OSS.

Re: DBRX: A new open LLM

#59

Post by tylermw »

pandastronaut wrote: post #53
glutamate wrote: post #45

Earlier quoted context omitted.

I think the case for "axis must always go to 0" is overblown. Zero isn't always meaningful, for instance chance performance or performance of trivial algorithms is likely >0%. Sometimes if axis must go to zero you can't see small changes. For instance if you plot world population 2014-2024 on an axis going to zero, you won't be able to see if we are growing or shrinking.

Even starting at 30%, the MMLU graph is false. The four bars are wrong. Even their own 73,7% is not at the right height. The Mixtral 71.4% is below the 70% mark of the axis. This is really the kind of marketing trick that makes me avoid a provider / publisher. I can't build trust this way.

I believe they are using the percentages as part of the height of the bar chart! I thought I'd seen every way someone could do dataviz wrong (particularly with a bar chart), but this one is new to me.

Re: DBRX: A new open LLM

#60

Post by occamrazor »

pandastronaut wrote: post #53
glutamate wrote: post #45

Earlier quoted context omitted.

I think the case for "axis must always go to 0" is overblown. Zero isn't always meaningful, for instance chance performance or performance of trivial algorithms is likely >0%. Sometimes if axis must go to zero you can't see small changes. For instance if you plot world population 2014-2024 on an axis going to zero, you won't be able to see if we are growing or shrinking.

Even starting at 30%, the MMLU graph is false. The four bars are wrong. Even their own 73,7% is not at the right height. The Mixtral 71.4% is below the 70% mark of the axis. This is really the kind of marketing trick that makes me avoid a provider / publisher. I can't build trust this way.

It‘s more likely to be incompetence than malice: even their 73.7% is closer to 72% than to 74%.
Post reply on HN