Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DBRX: A new open LLM

databricks.com

Post reply on HN
Print view View original

221–230 of 360 posts

Re: DBRX: A new open LLM

#221

Post by bobbylarrybobby »

jobigoud wrote: post #61
emmender2 wrote: post #46

this proves that all llm models converge to a certain point when trained on the same data. ie, there is really no differentiation between one model or the other. Claims about out-performance on tasks are just that, claims. the next iteration of llama or mixtral will converge. LLMs seem to evolve like linux/windows or ios/android with not much differentiation in the foundation models.

It's even possible they converge when trained on different data, if they are learning some underlying representation. There was recent research on face generation where they trained two models by splitting one training set in two without overlap, and got the two models to generate similar faces for similar conditioning, even though each model hadn't seen anything that the other model had.

I mean, faces are faces, right? If the training data set is large and representative I don't see why any two (representative) halves of the data would lead to significantly different models.

Re: DBRX: A new open LLM

#222

Post by tartrate »

pandastronaut wrote: post #53
glutamate wrote: post #45

Earlier quoted context omitted.

I think the case for "axis must always go to 0" is overblown. Zero isn't always meaningful, for instance chance performance or performance of trivial algorithms is likely >0%. Sometimes if axis must go to zero you can't see small changes. For instance if you plot world population 2014-2024 on an axis going to zero, you won't be able to see if we are growing or shrinking.

Even starting at 30%, the MMLU graph is false. The four bars are wrong. Even their own 73,7% is not at the right height. The Mixtral 71.4% is below the 70% mark of the axis. This is really the kind of marketing trick that makes me avoid a provider / publisher. I can't build trust this way.

Seems fixed now

Re: DBRX: A new open LLM

#223

Post by ec109685 »

For coding evals, it seems like unless you are super careful, they can be polluted by the training data.

Are there standard ways to avoid that type of score inflation?

Re: DBRX: A new open LLM

#224

Post by m3kw9 »

wavemode wrote: post #180
m3kw9 wrote: post #125

These tiny “state of the art” performance increases are really indicative the current architecture for LLM(Transformers + Mixture of Experts) is maxed out even if you train it more/differently. The writings are on all over the walls.

It would not surprise me if this is what has delayed OpenAI in releasing a new model. After more than a year since GPT-4, they may have by now produced some mega-trained mega-model, but running it is so expensive, and its eval improvement over GPT-4 so marginal, that releasing it to the public simply makes no commercial sense just yet. They may be working on how to optimize it to reduce cost, or re-engineer it to imp…

These “state of the art” llm barely eking out a win isn’t a threat to OpenAI and they can take their sweet time sharpening sword that will come down hard on these LLMs

Re: DBRX: A new open LLM

#225

Post by InitEnabler »

XCSme wrote: post #41
jasonjmcghee wrote: post #15

Earlier quoted context omitted.

> mixtral works well Do you mean mistral? mixtral is 8x7B and requires like 100GB of RAM Edit: (without quant as others have pointed out) can definitely be lower, but haven't heard of a 3.4GB version

I have 128GB, but something is weird with Ollama. Even though for the Ollama Docker I only allow 90GB, it ends up using 128GB/128GB, so the system because very slow (mouse freezes).

What docker flags are you running?

Re: DBRX: A new open LLM

#226

Post by falcor84 »

dheera wrote: post #200
djoldman wrote: post #4

Model card for base: https://huggingface.co/databricks/dbrx-base > The model requires ~264GB of RAM I'm wondering when everyone will transition from tracking parameter count vs evaluation metric to (total gpu RAM + total CPU RAM) vs evaluation metric. For example, a 7B parameter model using float32s will almost certainly outperform a 7B model using float4s. Additionally, all the examples of quantizing recently releas…

I'm more wondering when we'll have algorithms that will "do their best" given the resources they detect. That would be what I call artificial intelligence. Giving up because "out of memory" is not intelligence.

I suppose you could simulate dementia by loading as much of the weights as space permits and then just stopping. Then during inference, replace the missing weights with calls to random(). I'd actually be interested in seeing the results.

Re: DBRX: A new open LLM

#227

Post by loudmax »

htrp wrote: post #186
loudmax wrote: post #67

Earlier quoted context omitted.

I have an RTX 3080 with 10GB of VRAM. I'm able to run models larger than 10GB using llama.cpp and offloading to the GPU as much as can fit into VRAM. The remainder of the model runs on CPU + regular RAM. The `nvtop` command displays a nice graph of how much GPU processing and VRAM is being consumed. When I run a model that fits entirely into VRAM, say Mistral 7B, nvtop shows the GPU processing running at full tilt. W…

How's your performance on the 70b parameter llama series? Any good writeups of the offloading that you found?

Performance of 70b models is like 1 token every few seconds. And that's fitting the whole model into system RAM, not swap. It's interesting because some of the larger models are quite good, but too annoyingly slow to be practical for most use cases.

The Mixtral models run surprisingly well. They can run better than 1 token per second, depending on quantization. Still slow, but approaching a more practical level of usefulness.

Though if you're planning on accomplishing real work with LLMs, the practical solution for most people is probably to rent a GPU in the cloud.

Re: DBRX: A new open LLM

#228

Post by rayval »

spxneo wrote: post #177
swalsh wrote: post #74

Earlier quoted context omitted.

The models are commodities, and the API's are even similar enough that there is zero stickiness. I can swap one model for another, and usually not have to change anything about my prompts or rag pipelines. For startups, the lesson here is don't be in the business of building models. Be in the business of using models. The cost of using AI will probably continue to trend lower for the foreseeable future... but you can…

Excellent comment. Shows good awareness of economic forces at play here. We are just going to use whatever LLM is best fast/cheap and the giants are in an arms race to deliver just that. But only two companies in this epic techno-cold war have an economic moat but the other moat is breaking down inside the moat of the other company. The moat inside the moat cannot run without the parent moat.

Intriguing comment that I don't quite follow. Can you please elaborate?

Re: DBRX: A new open LLM

#229

Post by falcor84 »

emmender2 wrote: post #46

this proves that all llm models converge to a certain point when trained on the same data. ie, there is really no differentiation between one model or the other. Claims about out-performance on tasks are just that, claims. the next iteration of llama or mixtral will converge. LLMs seem to evolve like linux/windows or ios/android with not much differentiation in the foundation models.

> this proves that all llm models converge to a certain point when trained on the same data

They are also all trained to do well on the same evals, right? So doesn't it just boil down to neural nets being universal function approximators?

Re: DBRX: A new open LLM

#230

Post by bevekspldnw »

starik36 wrote: post #181
bevekspldnw wrote: post #145

Earlier quoted context omitted.

I had gone the Mac Studio route initially, but I ended up with getting an A6000 for about the same price as a Mac and putting that in a Linux server onder my desk. Ollama makes it dead simple to serve it over my local network, so I can be on my M1 Air and using it no differently than if on my laptop. The difference is that the A6000 absolutely smokes the Mac.

Wow, that is a lot of money ($4400 on Amazon) to throw at this problem. I am curious, what was the purpose that compelled you to spend this (for the home network, I assume) amount of money.

Large scale document classification tasks in very ambiguous contexts. A lot of my work goes into using big models to generate training data for smaller models.

I have multiple millions of documents so GPT is cost prohibitive, and too slow. My tools of choice tend to be a first pass with Mistral to check task performance and if lacking using Mixtral.

Often I find with a good prompt Mistral will work as well as Mixtral and is about 10x faster.

I’m on my “home” network, but it’s a “home office” for my startup.

Post reply on HN