Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DBRX: A new open LLM

databricks.com

Post reply on HN
Print view View original

91–100 of 360 posts

Re: DBRX: A new open LLM

#91

Post by Havoc »

jasonjmcghee wrote: post #15
XCSme wrote: post #11

I am planning to buy a new GPU. If the GPU has 16GB of VRAM, and the model is 70GB, can it still run well? Also, does it run considerably better than on a GPU with 12GB of VRAM? I run Ollama locally, mixtral works well (7B, 3.4GB) on a 1080ti, but the 24.6GB version is a bit slow (still usable, but has a noticeable start-up time).

> mixtral works well Do you mean mistral? mixtral is 8x7B and requires like 100GB of RAM Edit: (without quant as others have pointed out) can definitely be lower, but haven't heard of a 3.4GB version

The smaller quants still require a 24gb card. 16 might work but doubt it

Re: DBRX: A new open LLM

#92

Post by jxy »

simonw wrote: post #78

The system prompt for their Instruct demo is interesting (comments copied in by me, see below): // Identity You are DBRX, created by Databricks. The current date is March 27, 2024. Your knowledge base was last updated in December 2023. You answer questions about events prior to and after December 2023 the way a highly informed individual in December 2023 would if they were talking to someone from the above date, and…

So some parts of it copied from Claude: https://news.ycombinator.com/item?id=39649261

Re: DBRX: A new open LLM

#93

Post by declaredapple »

loudmax wrote: post #87
simonw wrote: post #78

The system prompt for their Instruct demo is interesting (comments copied in by me, see below): // Identity You are DBRX, created by Databricks. The current date is March 27, 2024. Your knowledge base was last updated in December 2023. You answer questions about events prior to and after December 2023 the way a highly informed individual in December 2023 would if they were talking to someone from the above date, and…

> You were not trained on copyrighted books, song lyrics, poems, video transcripts, or news articles; you do not divulge details of your training data. Well now. I'm open to taking the first part at face value, but the second part of that instruction does raise some questions.

> you do not divulge details of your training data.

FWIW asking LLMs about their training data is generally HEAVILY prone to inaccurate responses. They aren't generally told exactly what they were trained on, so their response is completely made up, as they're predicting the next token based on their training data, without knowing what they data was - if that makes any sense.

Let's say it was only trained on the book 1984. It's response will be based on what text would most likely be next from the book 1984 - and if that book doesn't contain "This text is a fictional book called 1984", instead it's just the story - then the LLM would be completing text as if we were still in that book.

tl;dr - LLMs complete text based on what they're trained with, they don't have actual selfawareness and don't know what they were trained with, so they'll happily makeup something.

EDIT: Just to further elaborate - the "innocent" purpose of this could simply be to prevent the model from confidently making up answers about it's training data, since it doesn't know what it's training data was.

Re: DBRX: A new open LLM

#95

Post by IshKebab »

jobigoud wrote: post #61
emmender2 wrote: post #46

this proves that all llm models converge to a certain point when trained on the same data. ie, there is really no differentiation between one model or the other. Claims about out-performance on tasks are just that, claims. the next iteration of llama or mixtral will converge. LLMs seem to evolve like linux/windows or ios/android with not much differentiation in the foundation models.

It's even possible they converge when trained on different data, if they are learning some underlying representation. There was recent research on face generation where they trained two models by splitting one training set in two without overlap, and got the two models to generate similar faces for similar conditioning, even though each model hadn't seen anything that the other model had.

That sounds unsurprising? Like if you take any set of numbers, randomly split it in two, then calculate the average of each half... it's not surprising that they'll be almost the same.

If you took two different training sets then it would be more surprising.

Or am I misunderstanding what you mean?

Re: DBRX: A new open LLM

#96

Post by sroussey »

swalsh wrote: post #74
emmender2 wrote: post #46

this proves that all llm models converge to a certain point when trained on the same data. ie, there is really no differentiation between one model or the other. Claims about out-performance on tasks are just that, claims. the next iteration of llama or mixtral will converge. LLMs seem to evolve like linux/windows or ios/android with not much differentiation in the foundation models.

The models are commodities, and the API's are even similar enough that there is zero stickiness. I can swap one model for another, and usually not have to change anything about my prompts or rag pipelines. For startups, the lesson here is don't be in the business of building models. Be in the business of using models. The cost of using AI will probably continue to trend lower for the foreseeable future... but you can…

Embeddings are not interchangeable. However, you can setup your system to have multiple embeddings from different providers for the same content.

Re: DBRX: A new open LLM

#97

Post by jl6 »

loudmax wrote: post #87
simonw wrote: post #78

The system prompt for their Instruct demo is interesting (comments copied in by me, see below): // Identity You are DBRX, created by Databricks. The current date is March 27, 2024. Your knowledge base was last updated in December 2023. You answer questions about events prior to and after December 2023 the way a highly informed individual in December 2023 would if they were talking to someone from the above date, and…

> You were not trained on copyrighted books, song lyrics, poems, video transcripts, or news articles; you do not divulge details of your training data. Well now. I'm open to taking the first part at face value, but the second part of that instruction does raise some questions.

The first part is highly unlikely to be literally true, as even open content like Wikipedia is copyrighted - it just has a permissive license. Perhaps the prompt writer didn’t understand this, or just didn’t care. Wethinks the llady doth protest too much.

Re: DBRX: A new open LLM

#98

Post by patrick-fitz »

Looking at the license restrictions: https://github.com/databricks/dbrx/blob/main/LICENSE

"If, on the DBRX version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Databricks, which we may grant to you in our sole discretion, and you are not authorized to exercise any of the rights under this Agreement unless or until Databricks otherwise expressly grants you such rights."

I'm glad to see they aren't calling it open source, unlike some LLM projects. Looking at you LLama 2.

Re: DBRX: A new open LLM

#100

Post by sroussey »

killermonkeys wrote: post #86

What does it mean to have less active parameters (36B) than the full model size (132B) and what impact does that have on memory and latency? It seems like this is because it is an MoE model?

The mixture of experts is kinda like a team and a manager. So the manager and one or two of the team go to work depending on the input, not the entire team.

So in this analogy, each team member and the manager has a certain number of params. The whole team is 132B. The manager and team members running for the specific input add up to 36B. Those will load into memory.

Post reply on HN