Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can I run AI locally?

canirun.ai

Post reply on HN
Print view View original

181–190 of 382 posts

Re: Can I run AI locally?

#181

Post by mopierotti »

comboy wrote: post #145
mopierotti wrote: post #95

This (+ llmfit) are great attempts, but I've been generally frustrated by how it feels so hard to find any sort of guidance about what I would expect to be the most straightforward/common question: "What is the highest-quality model that I can run on my hardware, with tok/s greater than , and context limit greater than " (My personal approach has just devolved into guess-and-check, which is time consuming.) When usin…

What is the $/Mtok that would make you choose your time vs savings of running stuff locally? Just to be clear, it may sound like a snarky comment but I'm really curious from you or others how do you see it. I mean there are some batches long running tasks where ignoring electricity it's kind of free but usually local generation is slower (and worse quality) and we all kind of want some stuff to get done. Or is it not…

Good question. I agree with what I think you're implying, which is that local generation is not the right choice if you want to maximize results per time/$ spent. In my experience, hosted models like Claude Opus 4.6 are just so effective that it's hard to justify using much else.

Nevertheless, I spend a lot of time with local models because of:

1. Pure engineering/academic curiosity. It's a blast to experiment with low-level settings/finetunes/lora's/etc. (I have a Cog Sci/ML/software eng background.)

2. I prefer not to share my data with 3rd party services, and it's also nice to not have to worry too much about accidentally pasting sensitive data into prompts (like personal health notes), or if I'm wasting $ with silly experiments, or if I'm accidentally poisoning some stateful cross-session 'memories' linked to an account.

3. It's nice to be able solve simple tasks without having to reason about any external 'side-effects' outside my machine.

Re: Can I run AI locally?

#182

Post by philipkglass »

mandeepj wrote: post #167
philipkglass wrote: post #110

Earlier quoted context omitted.

It really depends on the tasks you have to perform. I am using specialized OCR models running locally to extract page layout information and text from scanned legal documents. The quality isn't perfect, but it is really good compared to desktop/server OCR software that I formerly used that cost hundreds or thousands of dollars for a license. If you have similar needs and the time to try just one model, start with GLM…

> I formerly used that cost hundreds or thousands of dollars for a license Azure Doc Intelligence charges $1.50 for 1000 pages. Was that an annual/recurring license? Would you mind sharing your OCR model? I'm using Azure for now, as I want to focus on building the functionality first, but would later opt for a local model.

I took a long break from document processing after working on it heavily 20 years ago. The tools I used before were ABBYY FineReader and PrimeOCR. I haven't tried any of the commercial cloud based solutions. I'm currently using GLM-OCR, Chandra OCR, and Apple's LiveText in conjunction with each other (plus custom code for glue functionality and downstream processing).

Try just GLM-OCR if you want to get started quickly. It has good layout recognition quality, good text recognition quality, and they actually tested it on Apple Silicon laptops. It works easily out-of-the-box without the yak shaving I encountered with some other models. Chandra is even more accurate on text but its layout bounding boxes are worse and it runs very slowly unless you can set up batched inference with vLLM on CUDA. (I tried to get batching to run with vllm-mlx so it could work entirely on macOS, but a day spent shaving the yak with Claude Opus's help went nowhere.)

If you just want to transcribe documents, you can also try end-to-end models like olmOCR 2. I need pipeline models that expose inner details of document layout because I need to segment and restructure page contents for further processing. The end-to-end models just "magically" turn page scans into complete Markdown or HTML documents, which is more convenient for some uses but not mine.

Re: Can I run AI locally?

#183

Post by comrade1234 »

mind_heist wrote: post #180
comrade1234 wrote: post #172

I can't tell at a glance what this page is showing, but I am curious about the licenses on the various models that let me run it locally and make money off it. Awhile ago only deepseek let you do that - not sure now.

nice, this is an interesting idea. Can you elaborate on the licensing issue ? how do you get blocked for using the models commercially ?

Just read the license agreement. Last time I looked into this the only model I could run locally and do what I want was deepseek. I think it was the MIT license. The others had various restrictions that just didn't make it worth it.

I stopped researching this because buying the hardware to run deepseek full model just isn't practical right now. Our customers will have to be happy with us sending data to OpenAI/deepseek/etc if they want to use those features.

Re: Can I run AI locally?

#184

Post by girvo »

echelon wrote: post #144
radlad wrote: post #104

Earlier quoted context omitted.

I'd love to know how you fit smaller models into your workflow. I have an M4 Macbook Pro w/ 128GB RAM and while I have toyed with some models via ollama, I haven't really found a nice workflow for them yet.

Shouldn't we prioritize large scale open weights and open source cloud infra? An OpenRunPod with decent usage might encourage more non-leading labs to dump foundation models into the commons. We just need infra to run it. Distilling them down to desktop is a fool's errand. They're meant to run on DC compute. I'm fine with running everything in the cloud as long as we own the software infra and the weights. This is co…

I run Qwen3.5-plus through Alibaba’s coding plan (Model Studio): incredibly cheap, pretty fast, and decent. I can’t compare it to the highest released weight one though.

Re: Can I run AI locally?

#185

Post by girvo »

manmal wrote: post #96
mark_l_watson wrote: post #78

I have spent a HUGE amount of time the last two years experimenting with local models. A few lessons learned: 1. small models like the new qwen3.5:9b can be fantastic for local tool use, information extraction, and many other embedded applications. 2. For coding tools, just use Google Antigravity and gemini-cli, or, Anthropic Claude, or... Now to be clear, I have spent perhaps 100 hours in the last year configuring l…

What about running e.g. Qwen3.5 128B on a rented RTX Pro 6000?

IMO you’re better off using qwen3.5-plus through the model studio coding plan, but ymmv

Re: Can I run AI locally?

#186

Post by phillmv »

comboy wrote: post #145
mopierotti wrote: post #95

This (+ llmfit) are great attempts, but I've been generally frustrated by how it feels so hard to find any sort of guidance about what I would expect to be the most straightforward/common question: "What is the highest-quality model that I can run on my hardware, with tok/s greater than , and context limit greater than " (My personal approach has just devolved into guess-and-check, which is time consuming.) When usin…

What is the $/Mtok that would make you choose your time vs savings of running stuff locally? Just to be clear, it may sound like a snarky comment but I'm really curious from you or others how do you see it. I mean there are some batches long running tasks where ignoring electricity it's kind of free but usually local generation is slower (and worse quality) and we all kind of want some stuff to get done. Or is it not…

i can think of some tasks (classification, structured info extraction) that i _imagine_ even small meh models could do quite well at

on data i would never ever want to upload to any vendor if i can avoid it

Re: Can I run AI locally?

#187

Post by xandrius »

pants2 wrote: post #156

This really highlights the impracticality of local models: My $3k Macbook can run `GPT-OSS 20B` at ~16 tok/s according to this guide. Or I can run `GPT-OSS 120B` (a 6X larger model) at 360 tok/s (30X faster) on Groq at $0.60/Mtok output tokens. To generate $3k worth of output tokens on my local Mac at that pricing it would have to run 10 years continuously without stopping. There's virtually no economic break-even to…

You're saying it as if privacy was worthless? Also not many people would consider the price of buying a macbook and put it strictly towards running a local model.

Instead if you wanted to get a macbook anyway, you get to run local models for free on top. Very different story.

Re: Can I run AI locally?

#188

Post by dataflow »

mark_l_watson wrote: post #78

I have spent a HUGE amount of time the last two years experimenting with local models. A few lessons learned: 1. small models like the new qwen3.5:9b can be fantastic for local tool use, information extraction, and many other embedded applications. 2. For coding tools, just use Google Antigravity and gemini-cli, or, Anthropic Claude, or... Now to be clear, I have spent perhaps 100 hours in the last year configuring l…

Thanks for sharing this, it's super helpful. I have a question if you don't mind: I want a model that I can feed, say, my entire email mailbox to, so that I can ask it questions later. (Just the text content, which I can clean and preprocess offline for its use.) Have any offline models you've dealt with seemed suitable for that sort of use case, with that volume of content?

Re: Can I run AI locally?

#189

Post by torginus »

Jaxan wrote: post #171
torginus wrote: post #170

Huh, I never knew my browser just volunteers my exact hardware specs to any website without so much as even notifying me about it.

It doesn’t really. The website thinks I’m on a iPhone 19 pro, although I’m actually on a iPhone SE 1st gen. So it’s off by roughly a decade.

Maybe that's one of Safari's numerous 'quirks' our frontend devs keep bitching about.

Which in this case Im thankful that Apple isn't too keen on following standards like these.

Re: Can I run AI locally?

#190

Post by D-Machine »

philipkglass wrote: post #182
mandeepj wrote: post #167

Earlier quoted context omitted.

> I formerly used that cost hundreds or thousands of dollars for a license Azure Doc Intelligence charges $1.50 for 1000 pages. Was that an annual/recurring license? Would you mind sharing your OCR model? I'm using Azure for now, as I want to focus on building the functionality first, but would later opt for a local model.

I took a long break from document processing after working on it heavily 20 years ago. The tools I used before were ABBYY FineReader and PrimeOCR. I haven't tried any of the commercial cloud based solutions. I'm currently using GLM-OCR, Chandra OCR, and Apple's LiveText in conjunction with each other (plus custom code for glue functionality and downstream processing). Try just GLM-OCR if you want to get started quick…

These are some really great explicit examples and links, much appreciated.
Post reply on HN