Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DBRX: A new open LLM

databricks.com

Post reply on HN
Print view View original

251–260 of 360 posts

Re: DBRX: A new open LLM

#251

Post by bboygravity »

mrtranscendence wrote: post #142
bboygravity wrote: post #131

Less than 1 week after Nancy Pelosi bought a 5M USD share in Databricks, this news is published. https://twitter.com/PelosiTracker_/status/177119703064106223... Crime pays in the US.

Are you alleging that Nancy Pelosi invested in Databricks, a private company without a fluctuating share price, because she learned that they would soon release a small, fairly middling LLM that probably won't move the needle in any meaningful way?

Are you suggesting that Nancy Pelosi, who consistently beats the market through obvious insider trading for years in a row, bought a share in Databricks without any insider info? Possible, yet unlikely is my opinion.

https://jacobin.com/2021/12/house-speaker-paul-stocks-inside...

PS: "without a fluctuating share price" is non-sense. Just because the share is of a private company, doesn't mean its price can't fluctuate. Why would anybody buy shares in private companies if the price couldn't fluctuate? What would be the point?

Example of a changing share price of a different (random) private company that has many different share holders over time: https://www.cnbc.com/2023/12/13/spacex-value-climbs-to-180-b...

Re: DBRX: A new open LLM

#252

Post by bboygravity »

lfmunoz4 wrote: post #235
bboygravity wrote: post #131

Less than 1 week after Nancy Pelosi bought a 5M USD share in Databricks, this news is published. https://twitter.com/PelosiTracker_/status/177119703064106223... Crime pays in the US.

I see these types of jokes everywhere. I cannot understand that hints of corruption are so blatant (i.e. a politician consistently beating the market) yet people keep voting for the same politician. Don't see how that is possible, must be these joke are only on internet and mainstream media never mentions this.

People are down-voting this because they refuse to believe this could be reality.

Re: DBRX: A new open LLM

#253

Post by anonymousDan »

alexott wrote: post #150
anonymousDan wrote: post #137

Earlier quoted context omitted.

Do they use spark for the training?

Mosaic AI Training ( https://www.databricks.com/product/machine-learning/mosaic-a... ) as it's mentioned in the announcement blog ( https://www.databricks.com/blog/announcing-dbrx-new-standard... - it's a bit less technical)

Thanks. Is this open source - i.e. can it be used on my own cluster outside of databricks?

Re: DBRX: A new open LLM

#255

Post by cheselnut »

stri8ed wrote: post #127
swalsh wrote: post #74

Earlier quoted context omitted.

The models are commodities, and the API's are even similar enough that there is zero stickiness. I can swap one model for another, and usually not have to change anything about my prompts or rag pipelines. For startups, the lesson here is don't be in the business of building models. Be in the business of using models. The cost of using AI will probably continue to trend lower for the foreseeable future... but you can…

Or be in the business of building infrastructure for AI inference.

Is this not the same argument? There are like 20 startups and cloud providers all focused on AI inference. I'd think application layer receives the most value accretion in the next 10 years vs AI inference. Curious what others think

Re: DBRX: A new open LLM

#256

Post by astrange »

bee_rider wrote: post #50
PheonixPharts wrote: post #21

Earlier quoted context omitted.

While GPUs are still the kings of speed, if you are worried about VRAM I do recommend a maxed out Mac Studio. Llama.cpp + quantized models on Apple Silicon is an incredible experience, and having 192 GB of unified memory to work with means you can run models that just aren't feasible on a home GPU setup. It really boils down to what type of local development you want to do. I'm mostly experimenting with things where…

I know the M?-pro and ultra variants are multiple standard M?’s in a single package. But so the CPUs and GPUs share a die (like a single 4 p-core CPU 10 GPU core is what come in the die, and the more exotic variants are just a result of LEGO-ing out those guys and disabling some cores for market segmentation or because they had defects?) I guess I’m wondering if they technically could throw in their gauntlet and comp…

Ultra is the only one that's made from two smaller SoCs.

Re: DBRX: A new open LLM

#257

Post by c1b »

bevekspldnw wrote: post #145
PheonixPharts wrote: post #21

Earlier quoted context omitted.

While GPUs are still the kings of speed, if you are worried about VRAM I do recommend a maxed out Mac Studio. Llama.cpp + quantized models on Apple Silicon is an incredible experience, and having 192 GB of unified memory to work with means you can run models that just aren't feasible on a home GPU setup. It really boils down to what type of local development you want to do. I'm mostly experimenting with things where…

I had gone the Mac Studio route initially, but I ended up with getting an A6000 for about the same price as a Mac and putting that in a Linux server onder my desk. Ollama makes it dead simple to serve it over my local network, so I can be on my M1 Air and using it no differently than if on my laptop. The difference is that the A6000 absolutely smokes the Mac.

> The difference is that the A6000 absolutely smokes the Mac.

Memory Bandwidth : Mac Studio wins (about the same @ ~800)

VRAM : Mac Studio wins (4x more)

TFLOPs: A6000 wins (32 vs 38)

Re: DBRX: A new open LLM

#258

Post by underlines »

Y_Y wrote: post #13
vlovich123 wrote: post #10

Earlier quoted context omitted.

I thought float4 sacrificed a negligible cost in evaluation quality for a 8x reduction in RAM?

A free lunch? Wouldn't that be nice! Sometimes the quantization process improves the accuracy a little (probably by implicit regularization) but a model that's at or near capacity (as it should be) is necessarily hurt by throwing away most of the information. Language models often quantize well to small fixed-point types like int4, but it's not a magic wand.

This paper partially finds disagreeing evidence: https://arxiv.org/abs/2403.17887

Re: DBRX: A new open LLM

#259

Post by blueblimp »

brucethemoose2 wrote: post #237

I would note the actual leading models right now (IMO) are: - Miqu 70B (General Chat) - Deepseed 33B (Coding) - Yi 34B (for chat over 32K context) And of course, there are finetunes of all these. And there are some others in the 34B-70B range I have not tried (and some I have tried, like Qwen, which I was not impressed with). Point being that Llama 70B, Mixtral and Grok as seen in the charts are not what I would call…

Qwen1.5-72B-Chat is dominant in the Chatbot Arena leaderboard, though. (Miqu isn't on there due to being bootleg, but Qwen outranks Mistral Medium.)

Re: DBRX: A new open LLM

#260

Post by bobbruno »

tartrate wrote: post #214
hintymad wrote: post #121

Earlier quoted context omitted.

Thanks! Why do they not focus on hosting other open models then? I suspect other models will soon catch up with their advantages in faster inference and better benchmark results. That said, maybe the advantage is aligned interests: they want customers to use their platforms, so they can keep their models open. In contrast, Mistral removed their commitment to open source as they found a potential path to profitability…

> Why do they not focus on hosting other open models then? They do host other open models as well (pay-per-token).

https://docs.databricks.com/en/machine-learning/foundation-m...
Post reply on HN