Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can I run AI locally?

canirun.ai

Post reply on HN
Print view View original

261–270 of 382 posts

Re: Can I run AI locally?

#261

Post by AnimalMuppet »

dsr_ wrote: post #256
scronkfinkle wrote: post #248

Earlier quoted context omitted.

Do you also require computers to grow legs when they "run"? "Thinking" is just a term to describe a process in generative AI where you generate additional tokens in a manner similar to thinking a problem through. It's kind of a tired point to argue against the verb since it's meaning is well understood at this point

I am a professional in the information technology field, which is to say a pedantic extremist who believes that words have meanings derived from consensus, and when people alter the meanings, they alter what they believe. Using "thinking", "feeling", "alive", or otherwise referring to a current generation LLM as a creature is a mistake which encourages being wrong in further thinking about them.

When people alter the meanings, you need to start using different words to describe what you believe.

Re: Can I run AI locally?

#262

Post by colechristensen »

peddling-brink wrote: post #241
dsr_ wrote: post #213

Earlier quoted context omitted.

[flagged]

Rebooting a machine running an LLM isn’t noticed by the LLM. Would you feel comfortable digitally torturing it? Giving it a persona and telling it terrible things? Acts of violence against its persona? I’m not confident it’s not “feeling” in a way. Yes its circuitry is ones and zeros, we understand the mechanics. But at some point, there’s mechanics and meat circuitry behind our thoughts and feelings too. It is hubri…

I'm not entirely opposed to the kind of animism that assigns a certain amount of soul, consciousness, or being to everything in a spectrum between a rock and a philosopher... but even so.

Multiplying large matrices over and over is very much towards the "rock" end of that scale.

Re: Can I run AI locally?

#265

Post by scorpioxy »

Besides trying to run on your own hardware, anybody have recommendations for running some decent models on one of the many "AI clouds" providers? This is for sporadic use and so maybe one of the "serverless" providers that bill by the hour or minute or similar as opposed to monthly renting GPUs.

There are quite a few of them but their marketing is just confusing and full of buzz words. I've been tinkering with OpenRouter that acts as a middleman.

Re: Can I run AI locally?

#266

Post by raiph_ai »

Great site, I have an M2 and M3pro and was thinking about getting and Ultra M4 and wanted to know if it was going to be worth it. Now I can see exactly what models I can run locally.

Re: Can I run AI locally?

#267

Post by metrix »

scorpioxy wrote: post #265

Besides trying to run on your own hardware, anybody have recommendations for running some decent models on one of the many "AI clouds" providers? This is for sporadic use and so maybe one of the "serverless" providers that bill by the hour or minute or similar as opposed to monthly renting GPUs. There are quite a few of them but their marketing is just confusing and full of buzz words. I've been tinkering with OpenRo…

use openrouter, and call it a day. auto switching between providers, connectivity to all clouds and even works with free models

Re: Can I run AI locally?

#268

Post by hnfong »

colechristensen wrote: post #262
peddling-brink wrote: post #241

Earlier quoted context omitted.

Rebooting a machine running an LLM isn’t noticed by the LLM. Would you feel comfortable digitally torturing it? Giving it a persona and telling it terrible things? Acts of violence against its persona? I’m not confident it’s not “feeling” in a way. Yes its circuitry is ones and zeros, we understand the mechanics. But at some point, there’s mechanics and meat circuitry behind our thoughts and feelings too. It is hubri…

I'm not entirely opposed to the kind of animism that assigns a certain amount of soul, consciousness, or being to everything in a spectrum between a rock and a philosopher... but even so. Multiplying large matrices over and over is very much towards the "rock" end of that scale.

If we accept the Church-Turing thesis, a philosopher can be simulated by a simple Universal Turing machine.

If one day we are able to create a philosopher from such a rudimentary machine (and a lot of tape), would you consider that very much towards the "rock" end as well?

Re: Can I run AI locally?

#270

Post by rahimnathwani »

This site presents models in an incomplete and misleading way.

When I visit the site with an Apple M1 Max with 32GB RAM, the first model that's listed is Llama 3.1 8B, which is listed as needing 4.1GB RAM.

But the weights for Llama 3.1 8B are over 16GB. You can see that here in the official HF repo: https://huggingface.co/meta-llama/Llama-3.1-8B/tree/main

The model this site calls 'Llama 3.1 8B' is actually a 4-bit quantized version ( Q4_K_M) available on ollama.com/library: https://ollama.com/library/llama3.1:8b

If you're going to recommend a model to someone based on their hardware, you have to recommend not only a specific model, but a specific version of that model (either the original, or some specific quantized version).

This matters because different quantized versions of the model will have different RAM requirements and different performance characteristics.

Another thing I don't like is that the model names are sometimes misleading. For example, there's a model with the name 'DeepSeek R1 1.5B'. There's only one architecture for DeepSeek R1, and it has 671B parameters. The model they call 'DeepSeek R1 1.5B' does not use that architecture. It's a qwen2 1.5B model that's been finetuned on DeepSeek R1's outputs. (And it's a Q4_K_M quantized version.)

Post reply on HN