Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I want everything local – Building my offline AI workspace

instavm.io

Post reply on HN
Print view View original

91–100 of 294 posts

Re: I want everything local – Building my offline AI workspace

#91

Post by cyanydeez »

tcdent wrote: post #11

I'm constantly tempted by the idealism of this experience, but when you factor in the performance of the models you have access to, and the cost of running them on-demand in a cloud, it's really just a fun hobby instead of a viable strategy to benefit your life. As the hardware continues to iterate at a rapid pace, anything you pick up second-hand will still deprecate at that pace, making any real investment in hardw…

Anything you build in the LLM cloud will be. Must be. Rug pulled either via locking success or utter bankruptcy or just a model context prompt change.

Unless you're a billionaire with pull, you're building tools you cant control, cant own and are ephermap wisps.

That's even if you can even trust these large models in consistency.

Re: I want everything local – Building my offline AI workspace

#92

Post by thorncorona »

com2kid wrote: post #55

> Even with help from the "world's best" LLMs, things didn't go quite as smoothly as we had expected. They hallucinated steps, missed platform-specific quirks, and often left us worse off. This shows how little native app training data is even available. People rarely write blog posts about designing native apps, long winded medium tutorials don't exist, heck even the number of open source projects for native desktop…

I mean outside of HPC why would you when the browser is the world’s most ubiquitous VM?

Re: I want everything local – Building my offline AI workspace

#93

Post by kelipso »

bionsystem wrote: post #84
1oooqooq wrote: post #76

Earlier quoted context omitted.

more interesting is the extent apple convinced people a laptop can replace a desktop or server. mind blowing reality distortion field (as will be proven by some twenty comments telling I'm wrong 3... 2... 1).

I'm a desktop guy, considering the switch to a laptop-only setup, what would I miss ?

For $10k, you too can get the power of a $2k desktop, and enjoy burning your lap everyday, or something like that. If I were to do local compute and wanted to use my laptop, I would only consider a setup where I ssh in to my desktop. So I guess only difference from saas llm would be privacy and the cool factor. And rate limits, and paying more if you go over, etc.

Re: I want everything local – Building my offline AI workspace

#94

Post by motorest »

kelnos wrote: post #81
motorest wrote: post #61

Earlier quoted context omitted.

You're using a lot of words to say "I believe yesterday's hardware might not run models as as fast as today's hardware." That's fine. The point is that yesterday's hardware is quite capable of running yesterday's models, and obviously it will also run tomorrow's models. So the question is cost. Capex vs opex. The fact is that buying your own hardware is proven to be far more cost-effective than paying cloud providers…

Not the GP, but my take on this: You're right about the cost question, but I think the added dimension that people are worried about is the current pace of change. To abuse the idiom a bit, yesterday's hardware should be able to run tomorrow's models, as you say, but it might not be able to run next month's models (acceptably or at all). Fast-forward some number of years, as the pace slows. Then-yesterday's hardware…

> Fast-forward some number of years (...)

I repeat: OP's home server costs as much as a few months of a cloud provider's infrastructure.

To put it another way, OP can buy brand new hardware a few times per year and still save money compared with paying a cloud provider for equivalent hardware.

> Regarding the equivalent EC2 instance, I'm not comparing it to the cost of a homelab, I'm comparing it to the cost of an Anthropic Pro or Max subscription.

OP stated quite clearly their goal was to run models locally.

Re: I want everything local – Building my offline AI workspace

#95

Post by esseph »

jauntywundrkind wrote: post #78
Aurornis wrote: post #47

Earlier quoted context omitted.

I think the local LLM scene is very fun and I enjoy following what people do. However every time I run local models on my MacBook Pro with a ton of RAM, I’m reminded of the gap between local hosted models and the frontier models that I can get for $20/month or nominal price per token from different providers. The difference in speed and quality is massive. The current local models are very impressive, but they’re sti…

I agree and disagree. Many of the best models are open source, just too big to run for most people. And there are plenty of ways to fit these models! A Mac Studio M3 Ultra with 512 GB unified memory though has huge capacity, and a decent chunk of bandwidth (800GB/s. Compare vs a 5090's ~1800GB/s). $10k is a lot of money, but that ability to fit these very large models & get quality results is very impressive. Perform…

https://pcisig.com/pci-sig-announces-pcie-80-specification-t...

From 2003-2016, 13 years, we had PCIE 1,2,3.

2017 - PCIE 4.0

2019 - PCIE 5.0

2022 - PCIE 6.0

2025 - PCIE 7.0

2028 - PCIE 8.0

Manufacturing and vendors are having a hard time keeping up. And the PCIE 5.0 memory is.. not always the most stable.

Re: I want everything local – Building my offline AI workspace

#97

Post by eric-burel »

An llm on your computer is a fun hobby, an llm in your SME for 10 people is a business idea. There are not enough resources on this topic at all and the need is growing extremely fast. Local LLMs are needed for many use cases and business where cloud is not possible.

Re: I want everything local – Building my offline AI workspace

#98

Post by ActorNightly »

tcdent wrote: post #11

I'm constantly tempted by the idealism of this experience, but when you factor in the performance of the models you have access to, and the cost of running them on-demand in a cloud, it's really just a fun hobby instead of a viable strategy to benefit your life. As the hardware continues to iterate at a rapid pace, anything you pick up second-hand will still deprecate at that pace, making any real investment in hardw…

>but when you factor in the performance of the models you have access to, and the cost of running them on-demand in a cloud, it's really just a fun hobby instead of a viable strategy to benefit your life.

Its because people are thinking too linearly about this, equating model size with usability.

Without going into too much detail because this may be a viable business plan for me, but I have had very good success with Gemma QAT model that runs quite well on a 3090 wrapped up in a very custom agent format that goes beyond simple prompt->response use. It can do things that even the full size large language models fail to do.

Re: I want everything local – Building my offline AI workspace

#99

Post by m11a »

hombre_fatal wrote: post #25
c-hendricks wrote: post #17

Earlier quoted context omitted.

Why would AI be one of the few areas where locally-hosted options can't reach "good enough"?

For some use-cases, like making big complex changes to big complex important code or doing important research, you're pretty much always going to prefer the best model rather than leave intelligence on the table. For other use-cases, like translations or basic queries, there's a "good enough".

Agree, for now.

But the foundation models will eventually hit a limit, and the open-source ecosystem, which trails by around a year or two, will catch up.

Re: I want everything local – Building my offline AI workspace

#100

Post by baobun »

bionsystem wrote: post #84
1oooqooq wrote: post #76

Earlier quoted context omitted.

more interesting is the extent apple convinced people a laptop can replace a desktop or server. mind blowing reality distortion field (as will be proven by some twenty comments telling I'm wrong 3... 2... 1).

I'm a desktop guy, considering the switch to a laptop-only setup, what would I miss ?

Upgradability, repairability, thermals (translating into widely different performance for the same specs), I/O, connectivity.
Post reply on HN