really noob question - so to run on a GPU you need a 264GB RAM GPU?
and if you ran on a 264GB CPU would it be super slow?
DBRX: A new open LLM
291–300 of 360 posts
Re: DBRX: A new open LLM
#292Earlier quoted context omitted.
Thanks! Why do they not focus on hosting other open models then? I suspect other models will soon catch up with their advantages in faster inference and better benchmark results. That said, maybe the advantage is aligned interests: they want customers to use their platforms, so they can keep their models open. In contrast, Mistral removed their commitment to open source as they found a potential path to profitability…
Commoditize your complements: https://gwern.net/complement If Databricks makes their money off model serving and doesn't care whose model you use, they are incentivized to help the open models be competitive with the closed models they can't serve.
At this point it's a cliché to share this article, as much as I love gwern lol.
Re: DBRX: A new open LLM
#293Even though the README.md calls the license the Databricks Open Source License, the LICENSE file includes paragraphs such as > You will not use DBRX or DBRX Derivatives or any Output to improve any other large language model (excluding DBRX or DBRX Derivatives). and > If, on the DBRX version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates,…
1. Open source is a well-defined model and I reasonably expect Databricks to be aware of this due to their use of open source models in their other projects.
2. The stated licensing terms are clearly and decisively not open source.
3. It is reasonable to conclude that this model is dual licensed, under this restrictive proprietary license, and an undisclosed open source license.
4. Just use this Model under the open source license with the assumption that they will release the open source license later.
I jest. In all seriousness, you should just disregard their licensing terms entirely as copyright does not apply to weight. https://news.ycombinator.com/item?id=39847147
Re: DBRX: A new open LLM
#294really noob question - so to run on a GPU you need a 264GB RAM GPU? and if you ran on a 264GB CPU would it be super slow?
The model's weights can be sharded across multiple GPU's. A "common" training server could contain (for instance) eight "A100" GPU's, each with 40 GB (or up to 80 GB) a piece for a total of 320 GB working VRAM. Since they're connected to each other in the same PC, they can communicate with each other quickly enough to calculate in coordination in this fashion. This setup is _very_ expensive of course. Probably in the hundreds of thousands of dollars.
If you're hoping to run the model yourself, you will need enough money and expertise to rent and deploy it to a server with as many GPU's. Alternatively, volunteers and other researchers will be able to quantize (compress) the model and make it easier to run on configurations without as much VRAM.
If you ran it on CPU it may indeed be super slow, but it's possible it's fast enough for the purposes of running the model rather than trying to train that model. I am seeing (limited) success with the maxed out Mac lineup ($4500) using the beefy M1/M2 line of CPU's.
Re: DBRX: A new open LLM
#295Worse than the chart crime of truncating the y axis is putting LLaMa2's Human Eval scores on there and not comparing it to Code Llama Instruct 70b. DBRX still beats Code Llama Instruct's 67.8 but not by that much.
> chart crime of truncating the y axis If you chart the temperature of the ocean do you keep the y-axis anchored at zero Kelvin?
If you chart the temperature of the ocean are you measuring it in Kelvin?
Re: DBRX: A new open LLM
#296Earlier quoted context omitted.
And it appears to be at ~80 GB of RAM via quantisation.
That's a tricky number. Does it run on an 80GB GPU, does it auto-shave some parameters to fit in 79.99GB like any articifially "intelligent" piece of code would do, or does it give up like an unintelligent piece of code?
Are you aware how Macs present memory? Their 'unified' memory approach means you could run an 80GB model on a 128GB machine.
There's no concept of 'dedicated GPU memory' as per conventional amd64 arch machines.
Re: DBRX: A new open LLM
#297Earlier quoted context omitted.
Commoditize your complements: https://gwern.net/complement If Databricks makes their money off model serving and doesn't care whose model you use, they are incentivized to help the open models be competitive with the closed models they can't serve.
At this point it's a cliché to share this article, as much as I love gwern lol.
There is always the lucky 10k.
Re: DBRX: A new open LLM
#298I would note the actual leading models right now (IMO) are: - Miqu 70B (General Chat) - Deepseed 33B (Coding) - Yi 34B (for chat over 32K context) And of course, there are finetunes of all these. And there are some others in the 34B-70B range I have not tried (and some I have tried, like Qwen, which I was not impressed with). Point being that Llama 70B, Mixtral and Grok as seen in the charts are not what I would call…
It's Deepseek, not Deepseed, just so people can actually find the model.
Re: DBRX: A new open LLM
#299really noob question - so to run on a GPU you need a 264GB RAM GPU? and if you ran on a 264GB CPU would it be super slow?
Adding to ShamelessC's answer - the other option is to wait for quantised versions of this model. A q4 will be around 70GB, and probably acceptable. A q5 or higher would be preferred, but we're still a good way under the 260GB.
You still need extra RAM to breath, but that's a lot more palatable.
This is why the Mac range - with unified memory - is appealing, as you can allocate most of your (say) 256GB of RAM to the GPU.
Conventional (desktop) CPU / RAM would be painfully slow.
Re: DBRX: A new open LLM
#300Earlier quoted context omitted.
Excellent comment. Shows good awareness of economic forces at play here. We are just going to use whatever LLM is best fast/cheap and the giants are in an arms race to deliver just that. But only two companies in this epic techno-cold war have an economic moat but the other moat is breaking down inside the moat of the other company. The moat inside the moat cannot run without the parent moat.
Intriguing comment that I don't quite follow. Can you please elaborate?
Probably OpenAI running on Azure. But it was still convoluted.