Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MacBook M5 Pro and Qwen3.5 = Local AI Security System

sharpai.org

Post reply on HN
Print view View original

81–90 of 162 posts

Re: MacBook M5 Pro and Qwen3.5 = Local AI Security System

#81

Post by camdenreslink »

zamadatix wrote: post #72
psyclobe wrote: post #28

I have always envisioned a ai server being part of a family's major purchases e.g. when they buy a house, appliance, etc. they also buy a 'ai system'. Machine hardware evolution is slowing down, pretty soon you can buy one big ass server that will last potentially decades as it would be purpose built for ai. Things like 'context based home security' yeah thats just, automatic, free, part of the ai system. Everyone wi…

If you bought a big ass server for your home 10 years ago it probably wouldn't have even have had a GPU/AI accelerator at all. If it did, it would have been something with wimpy compute and VRAM because you needed the video encoder/decoder for security cameras or the like. I'm not sure that really gives confidence hardware has really slowed down enough to invest in it for decades. Single core CPU performance has but…

It really just depends on if the hardware is "good enough" for whatever its purpose is. If the hardware today can locally run whatever models for your security cameras, it's likely they will still be "good enough" in 10 years.

Of course, similar to a 10 year old car or appliance, you will be missing any new features or bells and whistles that have become available in the meantime.

Re: MacBook M5 Pro and Qwen3.5 = Local AI Security System

#84

Post by wtallis »

camdenreslink wrote: post #81
zamadatix wrote: post #72

Earlier quoted context omitted.

If you bought a big ass server for your home 10 years ago it probably wouldn't have even have had a GPU/AI accelerator at all. If it did, it would have been something with wimpy compute and VRAM because you needed the video encoder/decoder for security cameras or the like. I'm not sure that really gives confidence hardware has really slowed down enough to invest in it for decades. Single core CPU performance has but…

It really just depends on if the hardware is "good enough" for whatever its purpose is. If the hardware today can locally run whatever models for your security cameras, it's likely they will still be "good enough" in 10 years. Of course, similar to a 10 year old car or appliance, you will be missing any new features or bells and whistles that have become available in the meantime.

I agree; it's important to recognize that there are lots of use cases where computers have long since reached "good enough" and aren't really going obsolete anymore for those use cases.

My NAS is about 13 years old, the network switches it connects through are even older, and while 2.5GbE now exists I have no need throw out my "good enough" equipment to replace with something marginally faster or more power efficient. I don't even really need to expand the storage of that NAS anytime soon, because my music collection could never come close to filling it, my movie/TV collection isn't growing much anymore due to the shift to streaming, and the volume of other stuff that I need to back up from my other computers just isn't growing much over the years.

Re: MacBook M5 Pro and Qwen3.5 = Local AI Security System

#85

Post by Octoth0rpe »

jmalicki wrote: post #46
Octoth0rpe wrote: post #43

Earlier quoted context omitted.

> pretty soon you can buy one big ass server that will last potentially decades as it would be purpose built for ai. This feels like a very, very weak prediction (though certainly possible).

Perhaps if we truly run out of steam on the process node front?

Even if that happened tomorrow, I suspect we'd have _at least_ a decade of people tweaking/optimizing designs on the same node to squeeze meaningful performance upgrades out. Eg, coming up with hardware support for new int/float formats that make more sense for the models of 2029, running matrix operators on ram chips directly, etc.

Re: MacBook M5 Pro and Qwen3.5 = Local AI Security System

#86

Post by lm28469 »

psyclobe wrote: post #28

I have always envisioned a ai server being part of a family's major purchases e.g. when they buy a house, appliance, etc. they also buy a 'ai system'. Machine hardware evolution is slowing down, pretty soon you can buy one big ass server that will last potentially decades as it would be purpose built for ai. Things like 'context based home security' yeah thats just, automatic, free, part of the ai system. Everyone wi…

This is your reminder we're in a bubble inside of a bubble...

Most people don't even think about running network cables or mesh wifi when building a house, no one will buy a server to run ai in their physical home

Re: MacBook M5 Pro and Qwen3.5 = Local AI Security System

#87

Post by antiterra »

bigyabai wrote: post #5
aegis_camera wrote: post #4

Earlier quoted context omitted.

Memory is the limitation, M5 has larger memory options. So large language model could be used.

Context is your limitation, on the M5. The larger your model is, the longer you'll be waiting on token prefill. TFTT with 0 tokens of context isn't a real-world benchmark. That's why most professional inference solutions reach for GPU-heavy hardware like the Jetson. Apple Silicon seems like a strange and overly expensive fit for this use cae.

I'm not a hardware expert here but this strikes me as inaccurate, though the actual performance can be scenario dependent.

The Jetson hardware is targeted to low power robotics implementations.

The Jetson Orin is currently marketed as prototyping platform, and I believe it does not generally challenge recent Apple Silicon for inference performance, even considering prefill.

In the latest Blackwell based Jetson Thor, the key advantage over Apple Silicon is its capable FP4 tensor cores, which do indeed help with prefill. However, it also has half the memory bandwidth of an M4 Max, so this puts a big bottleneck on token generation with large context. If your use case did some kind of RAG lookup with very short responses then you might come out ahead using an optimized model, but for straightforward inference you are likely to lag behind Apple Silicon.

At this stage, professional inference solutions ideally use discrete GPUs that are far more capable than either, but those are a different class of monetary expense.

Re: MacBook M5 Pro and Qwen3.5 = Local AI Security System

#88

Post by johndough »

0xbadcafebee wrote: post #29
johndough wrote: post #18

Earlier quoted context omitted.

Perhaps OP was referring to a usable agentic system, for which $2500 sounds about right. I've got a 3060 myself, which is nice to play around with the smaller models for free (minus electricity) and with 100% uptime, but I was not able to program anything with them yet that I didn't want to rewrite completely. A heavily quantized Qwen3.5-27B model is getting close though. Maybe in a few months.

Strix Halo systems were ~$1500. They've gone up in price due to demand, but that is a perfectly usable "agentic system" (whatever that means). If 128GB VRAM and a fast GPU isn't good enough, I don't know what is.

> Strix Halo systems were ~$1500. They've gone up in price due to demand

The price hike has been crazy. The Bosgame M5 Mini is $2400 now. I didn't get one last year when they were $1500 because I thought the memory bandwidth was mediocre. However, it doesn't look like we'll get anything better for that price anytime soon.

Re: MacBook M5 Pro and Qwen3.5 = Local AI Security System

#89

Post by thijson »

hparadiz wrote: post #6

Currently the barrier to entry for local models is about $2500. Funny thing is $2500 is about the amount my parents paid for a 166 MHZ machine in 1995.

I remember my Dad buying a 386 25MHz a few years earlier for a similar amount.

In 1984 he bought a TRS-80 for almost a thousand dollars. 32kB RAM, around 1 MHz 8 bit CPU.

I bought a Pentium 90 in the late 90's for several thousand dollars. It had the FDIV bug in it.

After experiencing a lifetime of high depreciation in electronics, I'm extremely price sensitive when buying it. I feel that if I wait a few years everything will become much cheaper. Maybe that's not the case with the slow down in Moore's law and the AI datacenter build out.

Re: MacBook M5 Pro and Qwen3.5 = Local AI Security System

#90

Post by icedchai »

psyclobe wrote: post #28

I have always envisioned a ai server being part of a family's major purchases e.g. when they buy a house, appliance, etc. they also buy a 'ai system'. Machine hardware evolution is slowing down, pretty soon you can buy one big ass server that will last potentially decades as it would be purpose built for ai. Things like 'context based home security' yeah thats just, automatic, free, part of the ai system. Everyone wi…

Based on our current trajectory, it seems more likely everyone will upload everything to the cloud and pay perpetual royalties to access their own data.
Post reply on HN