Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DBRX: A new open LLM

databricks.com

Post reply on HN
Print view View original

201–210 of 360 posts

Re: DBRX: A new open LLM

#201

Post by brandall10 »

PheonixPharts wrote: post #21
XCSme wrote: post #11

I am planning to buy a new GPU. If the GPU has 16GB of VRAM, and the model is 70GB, can it still run well? Also, does it run considerably better than on a GPU with 12GB of VRAM? I run Ollama locally, mixtral works well (7B, 3.4GB) on a 1080ti, but the 24.6GB version is a bit slow (still usable, but has a noticeable start-up time).

While GPUs are still the kings of speed, if you are worried about VRAM I do recommend a maxed out Mac Studio. Llama.cpp + quantized models on Apple Silicon is an incredible experience, and having 192 GB of unified memory to work with means you can run models that just aren't feasible on a home GPU setup. It really boils down to what type of local development you want to do. I'm mostly experimenting with things where…

Wait for the M3 Ultra and it will be 256GB and markedly faster.

Re: DBRX: A new open LLM

#202

Post by visarga »

dheera wrote: post #200
djoldman wrote: post #4

Model card for base: https://huggingface.co/databricks/dbrx-base > The model requires ~264GB of RAM I'm wondering when everyone will transition from tracking parameter count vs evaluation metric to (total gpu RAM + total CPU RAM) vs evaluation metric. For example, a 7B parameter model using float32s will almost certainly outperform a 7B model using float4s. Additionally, all the examples of quantizing recently releas…

I'm more wondering when we'll have algorithms that will "do their best" given the resources they detect. That would be what I call artificial intelligence. Giving up because "out of memory" is not intelligence.

No but some model serving tools like llama.cpp do their best. It's just a matter of choosing the right serving tools. And I am not sure LLMs could not optimize their memory layout. Why not? Just let them play with this and learn. You can do pretty amazing things with evolutionary methods where the LLMs are the mutation operator. You evolve a population of solutions. (https://arxiv.org/abs/2206.08896)

Re: DBRX: A new open LLM

#203

Post by benrutter »

gigatexal wrote: post #72

data engineer here, offtopic, but am i the only guy tired of databricks shilling their tools as the end-all, be-all solutions for all things data engineering?

Lord no! I'm a data engineer also, feel the same. The part that I find most maddening is it seems pretty devoid from sincerely attempting to provide value.

Things databricks offers that makes peoples lives easier:

- Out the box kubernetes with no set up

- Preconfigured spark

Those are genuinely really useful, but then there's all this extra stuff that makes people's lives worse or drives bad practice:

- Everything is a notebook

- Local development is discouraged

- Version pinning of libraries has very ugly/bad support

- Clusters take 5 minutes to load even if you just want to "print('hello world')"

Sigh! I worked at a company that was databricks heavy and an still suffering PTSD. Sorry for the rant.

Re: DBRX: A new open LLM

#205

Post by breezeTrowel »

swalsh wrote: post #77
djoldman wrote: post #4

Model card for base: https://huggingface.co/databricks/dbrx-base > The model requires ~264GB of RAM I'm wondering when everyone will transition from tracking parameter count vs evaluation metric to (total gpu RAM + total CPU RAM) vs evaluation metric. For example, a 7B parameter model using float32s will almost certainly outperform a 7B model using float4s. Additionally, all the examples of quantizing recently releas…

> The model requires ~264GB of RAM This feels as crazy as Grok. Was there a generation of models recently where we decided to just crank on the parameter count?

Cranking up the parameter count is literally how the current LLM craze got started. Hence the "large" in "large language model".

Re: DBRX: A new open LLM

#206

Post by alexott »

melondonkey wrote: post #105
gigatexal wrote: post #72

data engineer here, offtopic, but am i the only guy tired of databricks shilling their tools as the end-all, be-all solutions for all things data engineering?

Data scientist here that’s also tired of the tools. We put so much effort in trying to educate DSes in our company to get away from notebooks and use IDEs like VS or RStudio and databricks has been a step backwards cause we didn’t get the integrated version

There is VSCode extension, plus databricks-connect… plus DABs. There are a lot customers doing local only development

Re: DBRX: A new open LLM

#207

Post by YetAnotherNick »

emmender2 wrote: post #46

this proves that all llm models converge to a certain point when trained on the same data. ie, there is really no differentiation between one model or the other. Claims about out-performance on tasks are just that, claims. the next iteration of llama or mixtral will converge. LLMs seem to evolve like linux/windows or ios/android with not much differentiation in the foundation models.

Even in the most liberal interpretation of prove, it doesn't do that. GPT-4 was trained before OpenAI has any special data or deal with microsoft or the product market fit. Yet, no model has beaten it in a year. And google, microsoft, meta definitely have better data and more compute.

Re: DBRX: A new open LLM

#208

Post by alexott »

pandastronaut wrote: post #154
melondonkey wrote: post #105

Earlier quoted context omitted.

Data scientist here that’s also tired of the tools. We put so much effort in trying to educate DSes in our company to get away from notebooks and use IDEs like VS or RStudio and databricks has been a step backwards cause we didn’t get the integrated version

Thank you ! I am so tired of all those unmaintainable nor debugable notebooks. Years ago, Databricks had a specific page on their documentation where they stated that notebooks where not for production grade software. It has been removed. And now you have a chatgpt like in their notebooks ... What a step backwards. How can all those developers be so happy without having the bare minimum tools to diagnosis their code…

It’s less about notebooks, but more about SDLC practices. Notebooks may encourage writing throwaway code, but if you split code correctly, then you can do unit testing, write modular code, etc. And ability to use “arbitrary files” as Python packages exists for quite a while, so you can get best of both worlds - quick iteration, plus ability to package your code as a wheel and distribute

P.S. here is a simple example of unit testing: https://github.com/alexott/databricks-nutter-repos-demo - I wrote it more than three years ago.

Re: DBRX: A new open LLM

#209

Post by resource_waste »

ml_hardware wrote: post #122
djoldman wrote: post #4

Model card for base: https://huggingface.co/databricks/dbrx-base > The model requires ~264GB of RAM I'm wondering when everyone will transition from tracking parameter count vs evaluation metric to (total gpu RAM + total CPU RAM) vs evaluation metric. For example, a 7B parameter model using float32s will almost certainly outperform a 7B model using float4s. Additionally, all the examples of quantizing recently releas…

Looks like someone has got DBRX running on an M2 Ultra already: https://x.com/awnihannun/status/1773024954667184196?s=20

I find 500 tokens considered 'running' a stretch.

Cool to play with for a few tests, but I can't imagine using it for anything.

Re: DBRX: A new open LLM

#210

Post by alexott »

benrutter wrote: post #203
gigatexal wrote: post #72

data engineer here, offtopic, but am i the only guy tired of databricks shilling their tools as the end-all, be-all solutions for all things data engineering?

Lord no! I'm a data engineer also, feel the same. The part that I find most maddening is it seems pretty devoid from sincerely attempting to provide value. Things databricks offers that makes peoples lives easier: - Out the box kubernetes with no set up - Preconfigured spark Those are genuinely really useful, but then there's all this extra stuff that makes people's lives worse or drives bad practice: - Everything is…

A lot of things has changed quite long ago - not everything is notebook, local dev is fully supported, version pinning wasn’t a problem, cluster startup time heavily dependent on underlying cloud provider, and serverless notebooks/jobs are coming
Post reply on HN