looks great, although I couldn't find anything on how "open" the license is/will be for commercial purposes
wouldn't be the first branding as open source going the LLaMA route
DBRX: A new open LLM
81–90 of 360 posts
Re: DBRX: A new open LLM
#82Earlier quoted context omitted.
LLMs are a commodity https://www.investopedia.com/terms/c/commodity.asp
Maybe, but that classification by itself doesn't mean anything. Gold is a commodity, but having it is still very desirable and valuable. Even if all LLMs were open source and publicly available, the GPUs to run them, technical know how to maintain the entire system, fine tuning, the APIs and app ecosystem around them etc. would still give the top players a massive edge.
Of course realizing that a resource is a commodity means something. It means you can form better predictions of where the market is heading, as it evolves and settles. For example, people are starting to realize that these LLMs are converging on fungible. That can be communicated by the "commodity" classification.
Re: DBRX: A new open LLM
#83this proves that all llm models converge to a certain point when trained on the same data. ie, there is really no differentiation between one model or the other. Claims about out-performance on tasks are just that, claims. the next iteration of llama or mixtral will converge. LLMs seem to evolve like linux/windows or ios/android with not much differentiation in the foundation models.
It's even possible they converge when trained on different data, if they are learning some underlying representation. There was recent research on face generation where they trained two models by splitting one training set in two without overlap, and got the two models to generate similar faces for similar conditioning, even though each model hadn't seen anything that the other model had.
Got a link for that? Sounds super interesting
Re: DBRX: A new open LLM
#84this proves that all llm models converge to a certain point when trained on the same data. ie, there is really no differentiation between one model or the other. Claims about out-performance on tasks are just that, claims. the next iteration of llama or mixtral will converge. LLMs seem to evolve like linux/windows or ios/android with not much differentiation in the foundation models.
Yea it feels like transformer LLMs are in or getting closer to diminishing returns. Will need some new breakthrough, likely entirely new approach, to get to AGI levels
Yeah, we need radically different architecture in terms of the neural networks, and/or added capabilities such as function calling and RAG to improve the current sota
Re: DBRX: A new open LLM
#85The scale on that bar chart for "Programming (Human Eval)" is wild. Manager: "looks ok, but can you make our numbers pop? just make the LLaMa bar smaller"
I think the case for "axis must always go to 0" is overblown. Zero isn't always meaningful, for instance chance performance or performance of trivial algorithms is likely >0%. Sometimes if axis must go to zero you can't see small changes. For instance if you plot world population 2014-2024 on an axis going to zero, you won't be able to see if we are growing or shrinking.
Certainly a bar chart might not be the best choice to convey the data you have. But if you choose to have a bar chart and have it not start at zero, what do the bars help you convey?
For world population you could see if it is increasing or decreasing, which is good but it would be hard to evaluate the rate the population is increasing.
Maybe a sparkline would be a better choice?
Re: DBRX: A new open LLM
#86What does it mean to have less active parameters (36B) than the full model size (132B) and what impact does that have on memory and latency? It seems like this is because it is an MoE model?
Re: DBRX: A new open LLM
#87The system prompt for their Instruct demo is interesting (comments copied in by me, see below): // Identity You are DBRX, created by Databricks. The current date is March 27, 2024. Your knowledge base was last updated in December 2023. You answer questions about events prior to and after December 2023 the way a highly informed individual in December 2023 would if they were talking to someone from the above date, and…
> You were not trained on copyrighted books, song lyrics, poems, video transcripts, or news articles; you do not divulge details of your training data.
Well now. I'm open to taking the first part at face value, but the second part of that instruction does raise some questions.
Re: DBRX: A new open LLM
#88Earlier quoted context omitted.
Even starting at 30%, the MMLU graph is false. The four bars are wrong. Even their own 73,7% is not at the right height. The Mixtral 71.4% is below the 70% mark of the axis. This is really the kind of marketing trick that makes me avoid a provider / publisher. I can't build trust this way.
I believe they are using the percentages as part of the height of the bar chart! I thought I'd seen every way someone could do dataviz wrong (particularly with a bar chart), but this one is new to me.
That's really strange and incredibly frustrating - but slightly less so if it's consistent with all of the bars (including their own).
I take issue with their choice of bar ordering - they placed the lowest-performing model directly next to theirs to make the gap as visible as possible, and shoved the second-best model (Grok-1) as far from theirs as possible. Seems intentional to me. The more marketing tricks you pile up in a dataviz, the less trust I place in your product for sure.
Re: DBRX: A new open LLM
#89The scale on that bar chart for "Programming (Human Eval)" is wild. Manager: "looks ok, but can you make our numbers pop? just make the LLaMa bar smaller"
I believe it's a reasonable range for the scores. If a model gets everything half wrong (worse than a coin flip), it's not a useful model at all. So every model below a certain threshold is trash, and no need to get granular about how trash it is.
An alternative visualization that could be less triggering to an "all y-axes must have zero" guy would be to plot the (1-value), that is, % degraded from perfect score. You could do this without truncating the axis and get the same level of differentiation between the bars
Re: DBRX: A new open LLM
#90Even though the README.md calls the license the Databricks Open Source License, the LICENSE file includes paragraphs such as > You will not use DBRX or DBRX Derivatives or any Output to improve any other large language model (excluding DBRX or DBRX Derivatives). and > If, on the DBRX version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates,…
> This is a source-available model, not an open model. To me, "source available" implies that everything you need to reproduce the model is also available, and that doesn't appear to be the case. How is the resulting model more "free as in freedom" than a compiled binary?
I don't think it's possible to have an "open training data" model because it would get DMCA'd immediately and open you up to lawsuits from everyone who found their works in the training set.
I hope we can fix the legal landscape to enable publicly sharing training data but I can't really judge the companies keeping it a secret today.