DBRX: A new open LLM
databricks.com
DBRX: A new open LLM
1–10 of 360 posts
Re: DBRX: A new open LLM
#2[deleted]
Re: DBRX: A new open LLM
#3GenAI novice here. what is training data made of how is it collected? I guess no one will share details on it, otherwise a good technical blog post with lots of insights!
>At Databricks, we believe that every enterprise should have the ability to control its data and its destiny in the emerging world of GenAI.
>The main process of building DBRX - including pretraining, post-training, evaluation, red-teaming, and refining - took place over the course of three months.
Re: DBRX: A new open LLM
#4Model card for base: https://huggingface.co/databricks/dbrx-base
> The model requires ~264GB of RAM
I'm wondering when everyone will transition from tracking parameter count vs evaluation metric to (total gpu RAM + total CPU RAM) vs evaluation metric.
For example, a 7B parameter model using float32s will almost certainly outperform a 7B model using float4s.
Additionally, all the examples of quantizing recently released superior models to fit on one GPU doesnt mean the quantized model is a "win." The quantized model is a different model, you need to rerun the metrics.
Re: DBRX: A new open LLM
#5What’s the process to deliver and test a quantized version of this model?
This model is 264GB, so can only be deployed in server settings.
Quantized mixtral at 24G is just small enough where it can be running on premium consumer hardware (ie 64GB RAM)
Re: DBRX: A new open LLM
#6GenAI novice here. what is training data made of how is it collected? I guess no one will share details on it, otherwise a good technical blog post with lots of insights! >At Databricks, we believe that every enterprise should have the ability to control its data and its destiny in the emerging world of GenAI. >The main process of building DBRX - including pretraining, post-training, evaluation, red-teaming, and refi…
The most detailed answer to that I've seen is the original LLaMA paper, which described exactly what that model was trained on (including lots of scraped copyrighted data) https://arxiv.org/abs/2302.13971
Llama 2 was much more opaque about the training data, presumably because they were already being sued at that point (by Sarah Silverman!) over the training data that went into the first Llama!
A couple of things I've written about this:
- https://simonwillison.net/2023/Aug/27/wordcamp-llms/#how-the...
Re: DBRX: A new open LLM
#7GenAI novice here. what is training data made of how is it collected? I guess no one will share details on it, otherwise a good technical blog post with lots of insights! >At Databricks, we believe that every enterprise should have the ability to control its data and its destiny in the emerging world of GenAI. >The main process of building DBRX - including pretraining, post-training, evaluation, red-teaming, and refi…
The training data is pretty much anything you can read on the internet plus books.
This is then cleaned up to remove nonsense, some technical files, and repeated files.
From this, they tend to weight some sources more - e.g. Wikipedia gets a pretty high weighting in the data mix. Overall these data mixes have multiple trillion token counts.
GPT-4 apparently trained on multiple epochs of the same data mix. So would assume this one did too as it’s a similar token count
Re: DBRX: A new open LLM
#8It's great how we went from "wait.. this model is too powerful to open source" to everyone trying to shove down their 1% improved model down the throats of developers
Re: DBRX: A new open LLM
#9GenAI novice here. what is training data made of how is it collected? I guess no one will share details on it, otherwise a good technical blog post with lots of insights! >At Databricks, we believe that every enterprise should have the ability to control its data and its destiny in the emerging world of GenAI. >The main process of building DBRX - including pretraining, post-training, evaluation, red-teaming, and refi…
The training data is pretty much anything you can read on the internet plus books. This is then cleaned up to remove nonsense, some technical files, and repeated files. From this, they tend to weight some sources more - e.g. Wikipedia gets a pretty high weighting in the data mix. Overall these data mixes have multiple trillion token counts. GPT-4 apparently trained on multiple epochs of the same data mix. So would as…
https://arxiv.org/abs/2305.10429 found that people are overweighting Wikipedia and downweighting Wikipedia improves things across the board INCLUDING PREDICTING NEXT TOKEN ON WIKIPEDIA, which is frankly amazing.
Re: DBRX: A new open LLM
#10Model card for base: https://huggingface.co/databricks/dbrx-base > The model requires ~264GB of RAM I'm wondering when everyone will transition from tracking parameter count vs evaluation metric to (total gpu RAM + total CPU RAM) vs evaluation metric. For example, a 7B parameter model using float32s will almost certainly outperform a 7B model using float4s. Additionally, all the examples of quantizing recently releas…
I thought float4 sacrificed a negligible cost in evaluation quality for a 8x reduction in RAM?