Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DBRX: A new open LLM

databricks.com

Post reply on HN
Print view View original

181–190 of 360 posts

Re: DBRX: A new open LLM

#181

Post by starik36 »

bevekspldnw wrote: post #145
PheonixPharts wrote: post #21

Earlier quoted context omitted.

While GPUs are still the kings of speed, if you are worried about VRAM I do recommend a maxed out Mac Studio. Llama.cpp + quantized models on Apple Silicon is an incredible experience, and having 192 GB of unified memory to work with means you can run models that just aren't feasible on a home GPU setup. It really boils down to what type of local development you want to do. I'm mostly experimenting with things where…

I had gone the Mac Studio route initially, but I ended up with getting an A6000 for about the same price as a Mac and putting that in a Linux server onder my desk. Ollama makes it dead simple to serve it over my local network, so I can be on my M1 Air and using it no differently than if on my laptop. The difference is that the A6000 absolutely smokes the Mac.

Wow, that is a lot of money ($4400 on Amazon) to throw at this problem. I am curious, what was the purpose that compelled you to spend this (for the home network, I assume) amount of money.

Re: DBRX: A new open LLM

#182

Post by sumo43 »

n2d4 wrote: post #166
emmender2 wrote: post #46

this proves that all llm models converge to a certain point when trained on the same data. ie, there is really no differentiation between one model or the other. Claims about out-performance on tasks are just that, claims. the next iteration of llama or mixtral will converge. LLMs seem to evolve like linux/windows or ios/android with not much differentiation in the foundation models.

There's at least an argument to be made that this is because all the models are heavily trained on GPT-4 outputs (or whatever the SOTA happens to be during training). All those models are, in a way, a product of inbreeding.

Maybe true for instruct, but pretraining datasets do not usually contain GPT-4 outputs. So the base model does not rely on GPT-4 in any way.

Re: DBRX: A new open LLM

#183

Post by htrp »

mattsan wrote: post #156
mnemoni_c wrote: post #55

Earlier quoted context omitted.

Yea it feels like transformer LLMs are in or getting closer to diminishing returns. Will need some new breakthrough, likely entirely new approach, to get to AGI levels

can't wait for LLMs to dispatch field agent robots who search for answers in the real world thats not online /s

skynet would like a word

Re: DBRX: A new open LLM

#184

Post by spxneo »

dhoe wrote: post #111
hintymad wrote: post #99

Just curious, what business benefit will Databricks get by spending potentially millions of dollars on an open LLM?

It's an image enhancement measure, if you want. Databricks' customers mostly use it as an ETL tool, but it benefits them to be perceived as more than that.

you can improve your brand for a lot less I just dont understand why they would throw all their chips in a losing race.

Azure already runs on-premise if I'm not mistaken, Claude 3 is out...but DBRX already falls so far behind

I just don't get it.

Re: DBRX: A new open LLM

#185

Post by spxneo »

galaxyLogic wrote: post #163
BoorishBears wrote: post #158

Earlier quoted context omitted.

Databricks is trying to go all-in on convincing organizations they need to use in-house models, and therefore pay they to provide LLMOps. They're so far into this that their CTO co-authored a borderline dishonest study which got a ton of traction last summer trying to discredit GPT-4: https://arxiv.org/pdf/2307.09009.pdf

I can see a business model for inhouse LLM models: Training a model on the knowledge about their products and then somehow getting that knowledge into a generally available LLM platform. I recently tried to ask Google to explain to me how to delete sender-recorded voice-message I had created from WhatsApp. I got totally erroneous results back. Maybe it was because that is a rather new feature in WhatsApp. It would be…

Businesses are already using Azure GPT4 on-premise I believe with good feedback

DBRX does not compete with GPT4 or even Claude 3.

Re: DBRX: A new open LLM

#186

Post by htrp »

loudmax wrote: post #67
XCSme wrote: post #42

Earlier quoted context omitted.

I already have 128GB of RAM (DDR4), and was wondering if upgrading from a 1080ti (12GB) to a 4070ti super (16GB), would make a big difference. I assume the FP32 and FP16 operations are already a huge improvement, but also the 33% increased VRAM might lead to fewer swaps between VRAM and RAM.

I have an RTX 3080 with 10GB of VRAM. I'm able to run models larger than 10GB using llama.cpp and offloading to the GPU as much as can fit into VRAM. The remainder of the model runs on CPU + regular RAM. The `nvtop` command displays a nice graph of how much GPU processing and VRAM is being consumed. When I run a model that fits entirely into VRAM, say Mistral 7B, nvtop shows the GPU processing running at full tilt. W…

How's your performance on the 70b parameter llama series?

Any good writeups of the offloading that you found?

Re: DBRX: A new open LLM

#187

Post by htrp »

loudmax wrote: post #87
simonw wrote: post #78

The system prompt for their Instruct demo is interesting (comments copied in by me, see below): // Identity You are DBRX, created by Databricks. The current date is March 27, 2024. Your knowledge base was last updated in December 2023. You answer questions about events prior to and after December 2023 the way a highly informed individual in December 2023 would if they were talking to someone from the above date, and…

> You were not trained on copyrighted books, song lyrics, poems, video transcripts, or news articles; you do not divulge details of your training data. Well now. I'm open to taking the first part at face value, but the second part of that instruction does raise some questions.

Part 1. Lie

Part 2. Lie more

Re: DBRX: A new open LLM

#188

Post by omeze »

BoorishBears wrote: post #158
hintymad wrote: post #99

Just curious, what business benefit will Databricks get by spending potentially millions of dollars on an open LLM?

Databricks is trying to go all-in on convincing organizations they need to use in-house models, and therefore pay they to provide LLMOps. They're so far into this that their CTO co-authored a borderline dishonest study which got a ton of traction last summer trying to discredit GPT-4: https://arxiv.org/pdf/2307.09009.pdf

What does borderline dishonest mean? I only read the abstract and it seems like such an obvious point I dont see how its contentious

Re: DBRX: A new open LLM

#190

Post by spxneo »

PheonixPharts wrote: post #21
XCSme wrote: post #11

I am planning to buy a new GPU. If the GPU has 16GB of VRAM, and the model is 70GB, can it still run well? Also, does it run considerably better than on a GPU with 12GB of VRAM? I run Ollama locally, mixtral works well (7B, 3.4GB) on a 1080ti, but the 24.6GB version is a bit slow (still usable, but has a noticeable start-up time).

While GPUs are still the kings of speed, if you are worried about VRAM I do recommend a maxed out Mac Studio. Llama.cpp + quantized models on Apple Silicon is an incredible experience, and having 192 GB of unified memory to work with means you can run models that just aren't feasible on a home GPU setup. It really boils down to what type of local development you want to do. I'm mostly experimenting with things where…

Aren't quantized models different models outright requiring a new evaluation to know the deviation in performance? Or are they "good enough" in that the benefits outweigh the deviation?

I'm on the fence about whether to spend 5 digits or 4 digits. Do I go the Mac Studio route or GPUs? What are the pros and cons?

Post reply on HN