Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I want everything local – Building my offline AI workspace

instavm.io

Post reply on HN
Print view View original

111–120 of 294 posts

Re: I want everything local – Building my offline AI workspace

#111

Post by com2kid »

kelipso wrote: post #93
bionsystem wrote: post #84

Earlier quoted context omitted.

I'm a desktop guy, considering the switch to a laptop-only setup, what would I miss ?

For $10k, you too can get the power of a $2k desktop, and enjoy burning your lap everyday, or something like that. If I were to do local compute and wanted to use my laptop, I would only consider a setup where I ssh in to my desktop. So I guess only difference from saas llm would be privacy and the cool factor. And rate limits, and paying more if you go over, etc.

$2k laptops now days come with 16 cores. They are thermally limited, but they are going to get you 60-80% the perf of their desktop counterparts.

The real limit is on the Nvidia cards. They are cut down a fair bit, often with less VRAM until you really go up in price point.

They also come with NPUs but the docs are bad and none of the local LLM inference engines seem to use the NPU, even though they could in theory be happy running smaller models.

Re: I want everything local – Building my offline AI workspace

#113

Post by ac29 »

motorest wrote: post #94
kelnos wrote: post #81

Earlier quoted context omitted.

Not the GP, but my take on this: You're right about the cost question, but I think the added dimension that people are worried about is the current pace of change. To abuse the idiom a bit, yesterday's hardware should be able to run tomorrow's models, as you say, but it might not be able to run next month's models (acceptably or at all). Fast-forward some number of years, as the pace slows. Then-yesterday's hardware…

> Fast-forward some number of years (...) I repeat: OP's home server costs as much as a few months of a cloud provider's infrastructure. To put it another way, OP can buy brand new hardware a few times per year and still save money compared with paying a cloud provider for equivalent hardware. > Regarding the equivalent EC2 instance, I'm not comparing it to the cost of a homelab, I'm comparing it to the cost of an An…

> OP stated quite clearly their goal was to run models locally.

Fair, but at the point you trust Amazon hosting your "local" LLM, its not a huge reach to just use Amazon Bedrock or something

Re: I want everything local – Building my offline AI workspace

#115

Post by ac29 »

c-hendricks wrote: post #17
jeremyjh wrote: post #16

Earlier quoted context omitted.

I expect it will never change. In two years if there is a local option as good as GPT-5 there will be a much better cloud option and you'll have the same tradeoffs to make.

Why would AI be one of the few areas where locally-hosted options can't reach "good enough"?

Maybe a better question is when will SOTA models be "good enough"?

At the moment there appears to be ~no demand for older models, even models that people praised just a few months ago. I suspect until AGI/ASI is reached or progress plateaus, that will continue be the case.

Re: I want everything local – Building my offline AI workspace

#116

Post by SteveJS »

tcdent wrote: post #11

I'm constantly tempted by the idealism of this experience, but when you factor in the performance of the models you have access to, and the cost of running them on-demand in a cloud, it's really just a fun hobby instead of a viable strategy to benefit your life. As the hardware continues to iterate at a rapid pace, anything you pick up second-hand will still deprecate at that pace, making any real investment in hardw…

AFAICT, the RTX 4090 I bought in 2023 has actually appreciated rather than depreciated.

Re: I want everything local – Building my offline AI workspace

#117

Post by jazzypants »

1oooqooq wrote: post #76
Aurornis wrote: post #47

Earlier quoted context omitted.

I think the local LLM scene is very fun and I enjoy following what people do. However every time I run local models on my MacBook Pro with a ton of RAM, I’m reminded of the gap between local hosted models and the frontier models that I can get for $20/month or nominal price per token from different providers. The difference in speed and quality is massive. The current local models are very impressive, but they’re sti…

more interesting is the extent apple convinced people a laptop can replace a desktop or server. mind blowing reality distortion field (as will be proven by some twenty comments telling I'm wrong 3... 2... 1).

I think this would be more interesting if you were to try to prove yourself correct first.

There are extremely few things that I cannot do on my laptop, and I have very little interest in those things. Why should I get a computer that doesn't have a screen? You do realize that, at this point of technological progress, the computer being attached to a keyboard and a screen is the only true distinguishing factor of a laptop, right?

Re: I want everything local – Building my offline AI workspace

#118

Post by isaacremuant »

tcdent wrote: post #11

I'm constantly tempted by the idealism of this experience, but when you factor in the performance of the models you have access to, and the cost of running them on-demand in a cloud, it's really just a fun hobby instead of a viable strategy to benefit your life. As the hardware continues to iterate at a rapid pace, anything you pick up second-hand will still deprecate at that pace, making any real investment in hardw…

Everything you're saying is FUD. There's immense value in being able to do local or remote as you please and part of it is knowledge.

Also, at the end of the day is about value creates and AI may allow some people to generate more stuff but overall value still tends to align with who is better at the craft pre AI. Not who pays more.

Re: I want everything local – Building my offline AI workspace

#119

Post by withinboredom »

zwnow wrote: post #41
jeremyjh wrote: post #16

Earlier quoted context omitted.

I expect it will never change. In two years if there is a local option as good as GPT-5 there will be a much better cloud option and you'll have the same tradeoffs to make.

You know there's a ceiling to all this with the current LLM approaches right? They won't become that much better, its even more likely they will degrade. There are cases of bad actors attacking LLMs by feeding it false information and propaganda. I dont see this changing in the future.

I seeded all over the internet that a friend of mine was an elephant with the intention of poisoning the well, so to speak. (with his permission, of course)

That was in 2021. Today if you ask who my friend is, it tells you that he is an elephant, without even doing a web search.

I wouldn’t be surprised if people are doing this with more serious things.

Re: I want everything local – Building my offline AI workspace

#120

Post by woadwarrior01 »

eric-burel wrote: post #101
woadwarrior01 wrote: post #42

> LLMs: Ollama for local models (also private models for now) Incidentally, I decided to try to Ollama macOS app yesterday, and the first thing it tries to do upon launch is try to connect to some google domain. Not very private. https://imgur.com/a/7wVHnBA

But can be audited which I'd buy everyday. It's probably not to hard to find network calls in a codebase if this task must be automated on update.

This is the macOS GUI, which IIUC is closed source.
Post reply on HN