Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jamesob's guide to running SOTA LLMs locally

github.com

Post reply on HN
Print view View original

181–190 of 193 posts

Re: Jamesob's guide to running SOTA LLMs locally

#181

Post by maxxxml »

jzer0cool wrote: post #135
maxxxml wrote: post #63

What harness is the best for local LLMs? I've been researching optimizing local LLM agent harness performance with context/ tools. Quite the endeavor and would love to learn what users prefer for this type of workflow.

What's the technical reason we call call these a harness? Seems right but want to understand better.

The model represents intelligence and the harness is toolset which allows the model to create more informed decisions with context. Specifically, loops, subagents, tools, connectors, prompts, skills, and much more. This is why Cursor performs so well.

Re: Jamesob's guide to running SOTA LLMs locally

#182

Post by echelon »

broadsidepicnic wrote: post #171
echelon wrote: post #49

Earlier quoted context omitted.

Stop trying to run them locally, folks. You don't own your fiber connection. So why try to own another rapidly depreciating, expensive, and annoying asset? Rent cloud GPUs! You get to participate in the ownership, data control, price control, and hacking culture without having to Frankenstein some hobbyist box that costs a ton, is distilled down to functional uselessness, and is a PITA to maintain.

> You don't own your fiber connection. So why try to own another rapidly depreciating, expensive, and annoying asset? Single mode fiber can serve for tens of years without problems and push the fastest speeds available today. I do not understand this comparison.

We don't need to own the hardware.

We need to own the software and the models.

Playing around with local models is like playing around with Ubuntu and Arch in the 00's. It's a fun toy, but it doesn't make a big economic dent, and it doesn't ensure we retain our rights and a slim capability gap against the frontier.

Developing software that works with big models, showing up with economic demand - that ensures that capability gets built and that open whittles away at closed at the very frontier.

More customers going to tiny hobbyist models also sucks oxygen out of the room for more large scale open models. We need to put economic demand on the larger open weights.

I see two futures ahead:

Future 1 - Big companies alone have access to the most productive models. Consumers play with API offerings and tiny RTX-scale models that lack the same capability.

Future 2 - A robust assortment of open weights models keep a very slim capability gap against the most mature frontier models. There's a viable economy around using and supporting these big models. Prosumers and enterprise can easily rent spot instances and spin up weights on-demand for a variety of tasks. There are rich model and fine tune marketplaces, a wide assortment of tools that can call these models, and easy tools to train models for any task from any foundation pretrain starting point.

I'd rather we go down path #2.

Re: Jamesob's guide to running SOTA LLMs locally

#183

Post by BoorishBears »

bradfa wrote: post #174
BoorishBears wrote: post #134

Earlier quoted context omitted.

The only thing that diminishes the value of a GPU right now is unsupported features with outsized value during inference and/or training (like FP4 support) and it takes time for those features to actually take off And labs are fully leaning into pricing for intelligence, so their margins are improving very quickly (which allows them to pay even more for existing compute) I'd be shocked if current prices aren't the bo…

Many newer Chinese lab models are releasing with int4 native weights. Latest NVIDIA generation GPUs have a hard time with this and can actually be slower than previous generations. This may make Blackwell depreciate faster than other recent generations.

That's not a real problem, hardly any 3rd party was running native weights anyways so they'll get quantized to NVFP4

Re: Jamesob's guide to running SOTA LLMs locally

#184

Post by DiabloD3 »

ineptech wrote: post #95
DiabloD3 wrote: post #86

Earlier quoted context omitted.

That is incorrect. They both have GDDR6. The B70 has 256 bit it bus at a clock speed of 2375mhz (608 GB/s), the 3090 has a 384 bit bus at a clock speed of 2438mhz (936 GB/s). It isn't slower, it just has less channels, ie, it is less wide.

Whoops thanks, was going from memory. At any rate, the effect is that it's somewhat slower than the 3090, when using a model small enough to fit entirely in nvram, but can fit models the 3090 can't.

Yep, but the flip side is you can get two B70s for the price of a single 3090 (MSRP, obviously; 3090s used go for about the same as a new B70), and they're true 2 slot, so they can fit on x8/x8 consumer boards fine.

The side effect is Intel has fired their entire Arc team, including the driver team, as well have canceled all Celestial products (only low end combined GPU + IO die products will have Celestial, as its too late in production to change). All future Intel products will have Nvidia graphics tiles.

Llama.cpp is, of course, still trying to better support the existing Intel products, but its kinda hard when they might stop working any day due to driver breakage.

Re: Jamesob's guide to running SOTA LLMs locally

#185

Post by mstaoru »

DiabloD3 wrote: post #184
ineptech wrote: post #95

Earlier quoted context omitted.

Whoops thanks, was going from memory. At any rate, the effect is that it's somewhat slower than the 3090, when using a model small enough to fit entirely in nvram, but can fit models the 3090 can't.

Yep, but the flip side is you can get two B70s for the price of a single 3090 (MSRP, obviously; 3090s used go for about the same as a new B70), and they're true 2 slot, so they can fit on x8/x8 consumer boards fine. The side effect is Intel has fired their entire Arc team, including the driver team, as well have canceled all Celestial products (only low end combined GPU + IO die products will have Celestial, as its t…

"Intel has fired their entire Arc team"? "All future Intel products will have Nvidia graphics tiles"?

Can you point to a source of these wild claims?

Re: Jamesob's guide to running SOTA LLMs locally

#186

Post by echelon »

gizajob wrote: post #148
echelon wrote: post #146

Earlier quoted context omitted.

Lobotomized RTX models are playthings. People building this stuff are "year of linux on desktop"ing open weights AI. It's a huge opportunity cost - not just for you, but for the open source community at large. You need to double down on big fat honking models that take multiple H200s to run. That's where the real power lies, and that's where our entire community needs to focus our efforts if we want to keep the delta…

You could be an IBM executive writing about the Apple I.

There's a world of difference between the Apple I and setting up CUDA drivers and Python.

You're in a small community of hobbyists. Cheap hobbyists who mostly don't pay for the stuff.

It's a bad growth market. It's a bad space to develop products. And it's as distracting to brilliant minds as bitcoin.

It's a suboptimal nerd snipe, and all the effort spent there is effort not being spent building actual frontier capabilities in the open.

Re: Jamesob's guide to running SOTA LLMs locally

#187

Post by DiabloD3 »

mstaoru wrote: post #185
DiabloD3 wrote: post #184

Earlier quoted context omitted.

Yep, but the flip side is you can get two B70s for the price of a single 3090 (MSRP, obviously; 3090s used go for about the same as a new B70), and they're true 2 slot, so they can fit on x8/x8 consumer boards fine. The side effect is Intel has fired their entire Arc team, including the driver team, as well have canceled all Celestial products (only low end combined GPU + IO die products will have Celestial, as its t…

"Intel has fired their entire Arc team"? "All future Intel products will have Nvidia graphics tiles"? Can you point to a source of these wild claims?

HN front page circa September through December of last year. Its part of the $5B investment into Nvidia, and the new CEO, Lip-Bu Tan, happily did whatever Nvidia told him to do.

Nova Lake (Series 4) is already too late in development to switch, so its shipping with Celestial, and Titan Lake (Series 5) after it will be essentially a Nova Lake refresh using a hybrid tile that is Celestial's graphics engine with Druid's media, memory, and bus controllers (since Druid development has been halted).

Serpent Lake (Series 6) will be the first that can ship with Nvidia tiles, which should launch in early 2028.

Unless something changes, there will be no DGPU Celestial products.

Unfortunately, the ink on the contract is already dry, so this is the end of Intel's GPU unit.

Re: Jamesob's guide to running SOTA LLMs locally

#189

Post by mstaoru »

DiabloD3 wrote: post #187
mstaoru wrote: post #185

Earlier quoted context omitted.

"Intel has fired their entire Arc team"? "All future Intel products will have Nvidia graphics tiles"? Can you point to a source of these wild claims?

HN front page circa September through December of last year. Its part of the $5B investment into Nvidia, and the new CEO, Lip-Bu Tan, happily did whatever Nvidia told him to do. Nova Lake (Series 4) is already too late in development to switch, so its shipping with Celestial, and Titan Lake (Series 5) after it will be essentially a Nova Lake refresh using a hybrid tile that is Celestial's graphics engine with Druid's…

I see. "My" (double checked) information is:

- Arc discrete (!) gaming (!!) GPUs are canceled,

- consumer gaming (!) SoC Serpent Lake w/RTX in 2028,

- Arc is continued in mobile, professional, iGPU, and DC GPUs,

- Arc team is just fine, Intel cut some jobs but not "the whole team".

Re: Jamesob's guide to running SOTA LLMs locally

#190

Post by DiabloD3 »

mstaoru wrote: post #189
DiabloD3 wrote: post #187

Earlier quoted context omitted.

HN front page circa September through December of last year. Its part of the $5B investment into Nvidia, and the new CEO, Lip-Bu Tan, happily did whatever Nvidia told him to do. Nova Lake (Series 4) is already too late in development to switch, so its shipping with Celestial, and Titan Lake (Series 5) after it will be essentially a Nova Lake refresh using a hybrid tile that is Celestial's graphics engine with Druid's…

I see. "My" (double checked) information is: - Arc discrete (!) gaming (!!) GPUs are canceled, - consumer gaming (!) SoC Serpent Lake w/RTX in 2028, - Arc is continued in mobile, professional, iGPU, and DC GPUs, - Arc team is just fine, Intel cut some jobs but not "the whole team".

Serpent Lake (and its siblings) cover both consumer desktop and laptop.

Arc's team was disproportionately effected by the layoffs. A lot of the major engineers that worked there changed their Linked in details to list Nvidia, AMD, etc.

As for DC and Workstation GPUs, they also seem to be canceled.

We'll see next year, I guess.

Post reply on HN