Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can I run AI locally?

canirun.ai

Post reply on HN
Print view View original

301–310 of 382 posts

Re: Can I run AI locally?

#301

Post by verisimi »

colechristensen wrote: post #259
dsr_ wrote: post #256

Earlier quoted context omitted.

I am a professional in the information technology field, which is to say a pedantic extremist who believes that words have meanings derived from consensus, and when people alter the meanings, they alter what they believe. Using "thinking", "feeling", "alive", or otherwise referring to a current generation LLM as a creature is a mistake which encourages being wrong in further thinking about them.

I'd suggest spending more time studying words to relive your extremism. The meanings of words move incredibly quickly and a tremendous number of words have little to no relation to previous meanings. Words such as nice, terrific, awful, manufacture, naughty, decimate, artificial, bully... and on and on.

> I'd suggest spending more time studying words to relive your extremism.

Should one study words to relive extremism? Or should one study words to relieve extremism?

To a doctor of linguistics: "Dr, my extremism... What should I do about it - with words?!? Please help."

That is the question.

Does the doctor answer thusly: "Study the words to relive the extremism! There is your answer!" says he.

or does he say: "Study the words to relieve and soothe the painful, abrasive extremism. Do it twice daily, before meals."

Sage advice in either case methinks.

Re: Can I run AI locally?

#302

Post by dataflow »

lilactown wrote: post #300
dataflow wrote: post #188

Earlier quoted context omitted.

Thanks for sharing this, it's super helpful. I have a question if you don't mind: I want a model that I can feed, say, my entire email mailbox to, so that I can ask it questions later. (Just the text content, which I can clean and preprocess offline for its use.) Have any offline models you've dealt with seemed suitable for that sort of use case, with that volume of content?

If your inbox is as big as mine, you won’t be able to load all the text content into a prompt even with SotA cloud hosted models. Instead you should give it tools to search over the mailbox for terms, labels, addresses, etc. so that the model can do fine grained filters based on the query, read the relevant emails it finds, then answer the question.

Thanks, yeah. I think strong prefiltering is pretty much always doable because, if nothing else, I usually know the time range of the relevant emails and probably the sender/recipient or some keywords, plus I know how to filter out a big chunk of the irrelevant emails (like mailing lists, etc.), so I'm hoping it's not actually that much data for each search. What I don't know is which models would be most suitable even in the case where I can fit the data.

As an example of the kind of query I'm interested in, I want a model that can tell me all the flights I took within a given time range (so that means it'd have to filter out cancellations). Or, for a given flight, the arrival and departure times and time zones (or the city and country so I can look up the time zone). Stuff like that. (Travel is just an example obviously, I have other topics to ask about.) It's not a terribly large number of emails to search through in each query, but the email structures are too heterogeneous across senders to write custom tooling for each case.

Re: Can I run AI locally?

#304

Post by sayamqazi »

dsr_ wrote: post #213
sdrinf wrote: post #151

Earlier quoted context omitted.

Just want to echo the recommendation for qwen3.5:9b. This is a smol, thinking, agentic tool-using, text-image multimodal creature, with very good internal chains of thought. CoT can be sometimes excessive, but it leads to very stable decision-making process, even across very large contexts -something we haven't seen models of this size before. What's also new here, is VRAM-context size trade-off: for 25% of it's atte…

[flagged]

Are insects not creatures?

Re: Can I run AI locally?

#305

Post by sayamqazi »

colechristensen wrote: post #291
hnfong wrote: post #268

Earlier quoted context omitted.

If we accept the Church-Turing thesis, a philosopher can be simulated by a simple Universal Turing machine. If one day we are able to create a philosopher from such a rudimentary machine (and a lot of tape), would you consider that very much towards the "rock" end as well?

Can a Turing machine of any sort truly indistinguishably simulate a nondeterministic system? If a Turing machine can truly simulate a full nondeterministic system as complex as a philosopher but it would take dedicating every gram of matter in the visible universe for a trillion years to simulate one second, is this meaningfully different than saying it cannot? I suggest the answer to both questions are no, but the s…

You are claiming that intelligence and even consciousness are non-deterministic entties at core. This is a huge claim and requires incredible proof.

Re: Can I run AI locally?

#306

Post by mghfreud »

colechristensen wrote: post #291
hnfong wrote: post #268

Earlier quoted context omitted.

If we accept the Church-Turing thesis, a philosopher can be simulated by a simple Universal Turing machine. If one day we are able to create a philosopher from such a rudimentary machine (and a lot of tape), would you consider that very much towards the "rock" end as well?

Can a Turing machine of any sort truly indistinguishably simulate a nondeterministic system? If a Turing machine can truly simulate a full nondeterministic system as complex as a philosopher but it would take dedicating every gram of matter in the visible universe for a trillion years to simulate one second, is this meaningfully different than saying it cannot? I suggest the answer to both questions are no, but the s…

I am not following what we are talking about here. I am a basic human being, I cannot truly simulate a nondeterministic system. Does it mean “I am not thinking”?

Re: Can I run AI locally?

#307

Post by rwmj »

xnzakg wrote: post #202
the_pwner224 wrote: post #178

Earlier quoted context omitted.

Arch with KDE, it works perfectly out of the box. I configured/disabled RGB lighting in Windows before wiping and the settings carried over to Linux. On Arch, install & enable power-profiles-daemon and you can switch between quiet/balanced/performance fan & TDP profiles. It uses the same profiles & fan curves as the options in Asus's Windows software. KDE has native integration for this in the GUI in the battery menu…

You may wanna see if openrgb isn't able to configure the RGB. Could even do some fun stuff like changing the color once done with a training run or something

I use openrgb to turn off all the RGB crap on my desktop machine. Unfortunately you have to leave openrgb running and it takes a constant 0.5% of CPU. I wish there was a "norgb" program that would simply turn off RGB everywhere and not use any CPU while doing it.

Re: Can I run AI locally?

#308

Post by throwdbaaway »

pants2 wrote: post #156

This really highlights the impracticality of local models: My $3k Macbook can run `GPT-OSS 20B` at ~16 tok/s according to this guide. Or I can run `GPT-OSS 120B` (a 6X larger model) at 360 tok/s (30X faster) on Groq at $0.60/Mtok output tokens. To generate $3k worth of output tokens on my local Mac at that pricing it would have to run 10 years continuously without stopping. There's virtually no economic break-even to…

90% of what you pay in agentic coding is for cached reads, which are free with local inference serving one user. This is well known in r/LocalLLaMA for ages, and an article about this also hit HN front page few weeks ago.

Re: Can I run AI locally?

#310

Post by tobr »

dsr_ wrote: post #213
sdrinf wrote: post #151

Earlier quoted context omitted.

Just want to echo the recommendation for qwen3.5:9b. This is a smol, thinking, agentic tool-using, text-image multimodal creature, with very good internal chains of thought. CoT can be sometimes excessive, but it leads to very stable decision-making process, even across very large contexts -something we haven't seen models of this size before. What's also new here, is VRAM-context size trade-off: for 25% of it's atte…

[flagged]

Describing what computers do as ”thinking” is not new. It’s a useful and obvious metaphor. https://www.gutenberg.org/ebooks/68991
Post reply on HN