Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DBRX: A new open LLM

databricks.com

Post reply on HN
Print view View original

351–360 of 360 posts

Re: DBRX: A new open LLM

#351

Post by Zambyte »

XCSme wrote: post #335
Zambyte wrote: post #334

Earlier quoted context omitted.

I genuinely recommend considering AMD options. I went with a 7900 XTX because it has the most VRAM for any $1000 card (24 GB). NVIDIA cards at that price point are only 16 GB. Ollama and other inference software works on ROCm, generally with at most setting an environment variable now. I've even run Ollama on my Steam Deck with GPU inferencing :)

I ended up getting a 2nd hand 3090 for 680€. Funnily, I think the card is new (smells new) and unused, most likely a scalper bought it and couldn't sell it.

Nice, that's definitely a sweet deal

Re: DBRX: A new open LLM

#352

Post by XCSme »

Zambyte wrote: post #351
XCSme wrote: post #335

Earlier quoted context omitted.

I ended up getting a 2nd hand 3090 for 680€. Funnily, I think the card is new (smells new) and unused, most likely a scalper bought it and couldn't sell it.

Nice, that's definitely a sweet deal

Thanks, I chose a 3090 instead of 4070ti, it was around $200 cheaper and has 24GB vs 16GB VRAM and a similar performance. The only drawback is the 350W TDP.

I still struggle with the RAM issue on Ollama, where it uses 128GB/128GB RAM for Mixtral 24.6GB, even though Docker limit is set to 90GB.

Docker seems pretty buggy on Windows...

Re: DBRX: A new open LLM

#353

Post by PoignardAzur »

sitkack wrote: post #297
youssefabdelm wrote: post #292

Earlier quoted context omitted.

At this point it's a cliché to share this article, as much as I love gwern lol.

There is always the lucky 10k.

For that reference in particular, feels like you should really share the link as well:

https://xkcd.com/1053/

Re: DBRX: A new open LLM

#354

Post by speedylight »

XCSme wrote: post #11

I am planning to buy a new GPU. If the GPU has 16GB of VRAM, and the model is 70GB, can it still run well? Also, does it run considerably better than on a GPU with 12GB of VRAM? I run Ollama locally, mixtral works well (7B, 3.4GB) on a 1080ti, but the 24.6GB version is a bit slow (still usable, but has a noticeable start-up time).

Quantized models will run well, otherwise inference might be really really slow or the client crashes all together with some CUDA out of memory error.

Re: DBRX: A new open LLM

#356

Post by panarky »

evrial wrote: post #348
panarky wrote: post #340

Earlier quoted context omitted.

Apparently, if you want to avoid "chart crime" when you chart temperatures, then it's deceptive if you don't start at absolute zero.

When was the temperature on Earth at absolute zero?

My point exactly.

In a chart of world gross domestic product for the last 12 months, when was it at zero?

In a chart of ocean salinity, when was it at absolute zero?

Is it inherently deceptive to use a y-axis that doesn't begin at zero?

Re: DBRX: A new open LLM

#358

Post by bboygravity »

mazlix wrote: post #342
bboygravity wrote: post #251

Earlier quoted context omitted.

Are you suggesting that Nancy Pelosi, who consistently beats the market through obvious insider trading for years in a row, bought a share in Databricks without any insider info? Possible, yet unlikely is my opinion. https://jacobin.com/2021/12/house-speaker-paul-stocks-inside... PS: "without a fluctuating share price" is non-sense. Just because the share is of a private company, doesn't mean its price can't fluctuat…

I'm not a lawyer and this isn't investment advice so don't sue me if I'm wrong but I'm not sure this qualifies as insider trading in the way that would be illegal for public markets. Aren't most investors in private companies privy to information that isn't entirely public? I can see how this feels a bit different because DataBricks might be the size where it might trade with a decent amount of liquidity, but certain…

She (and many other senators and other government employees) are very obviously and consistently beating the market as well as beating most investment funds.

There's only 1 explanation for this: they're getting inside info from lobbyists and such.

I don't care whether that's currently illegal or not. I don't care whether other types of investors also engage in that same practice. I just think that that's extremely wrong and corrupt and it blows my mind that both the US government and people think that this is totally OK (or don't know about it, which is even worse).

Re: DBRX: A new open LLM

#359

Post by lobocinza »

dheera wrote: post #249
declaredapple wrote: post #232

Earlier quoted context omitted.

What? Are you asking if the framework automatically quantizes/prunes the model on the fly? Or are you suggesting the LLM itself should realize it's too big to run, and prune/quantize itself? Your references to "intelligent" almost leads me to the conclusion that you think the LLM should prune itself. Not only is this a chicken and egg problem, but LLMs are statistical models, they aren't inherently self bootstraping.

I realize that, but I do think it's doable to bootstrap it on a cluster and teach itself to self-prune, and surprised nobody is actively working on this. I hate software that complains (about dependencies, resources) when you try to run it and I think that should be one of the first use cases for LLMs to get L5 autonomous software installation and execution.

Worst is software that doesn't complain but fails silently.

Re: DBRX: A new open LLM

#360

Post by MP_1729 »

petesergeant wrote: post #287

This makes me bearish on OpenAI as a company. When a cloud company can offer a strong model for free by selling the compute, what competitive advantage does a company who want you to pay for the model have left? Feels like they might get Netscape’d.

OpenAI is not the worst, ChatGPT is used by 100M people weekly, sort of insulated from benchmarks. The best of the rest, Anthropic, should be really scared.
Post reply on HN