Ask HN: Any way to borrow compute from Apple M1 • Hacker News Board

Re: Ask HN: Any way to borrow compute from Apple M1

#31

Post by NBJack » Thu, Jul 18, 2024, 12:46 PM UTC

saberience wrote: post #13

BossingAround wrote: post #6

An off-topic question, are Apple's M-series chips any good at current AI/ML work? How does it compare with dedicated GPUs?

I have an M3 chip in my laptop, it has more memory than my 4090 but it's still way slower when inferencing. So as long as the model fits in memory, Nvidia GPUs are going to be way faster just because they have more/faster compute cores. Of course, if the model fits in memory on your M chip and doesn't in your Nvidia chip, the M chip wins by default. However, I would say, if you load a 70B model in your M chip, while…

The latest Nvidia drivers offer an option to start using system memory when VRAM is insufficient. It certainly slows things down, but it does work. It's not perfect in my experience, but it is an alternative for large models.

Re: Ask HN: Any way to borrow compute from Apple M1

#32

Post by Shadowmist » Thu, Jul 18, 2024, 12:52 PM UTC

moffkalast wrote: post #14

loktarogar wrote: post #11

This was bouncing around the last few days, if you have a few devices as well as the M1 (though i'm not sure it able to work over the internet as opposed to a local network): https://github.com/exo-explore/exo Otherwise set up Ollama's API

You mean the llama.cpp server API right? Ollama keeps taking credit for things they put a thin wrapper around, and it's seriously annoying.

What’s the llama.cpp CLI equivalent of `ollama pull`?

Re: Ask HN: Any way to borrow compute from Apple M1

#33

Post by gkfasdfasdf » Thu, Jul 18, 2024, 12:54 PM UTC

BossingAround wrote: post #6

An off-topic question, are Apple's M-series chips any good at current AI/ML work? How does it compare with dedicated GPUs?

Compared to a GPU like a 4090 with equal vram it probably won't fare well as others point out, however it far outperforms any CPU. On an M1 Ultra MacBook Pro I was seeing like 40 tokens/second with llama3:7B vs 9 tokens/second on various Intel servers/desktops with sufficient ram.

Re: Ask HN: Any way to borrow compute from Apple M1

#34

Post by gkfasdfasdf » Thu, Jul 18, 2024, 1:00 PM UTC

mrweasel wrote: post #20

BossingAround wrote: post #6

An off-topic question, are Apple's M-series chips any good at current AI/ML work? How does it compare with dedicated GPUs?

I've only tried it on my M1, running Llama-3 via Ollama. It works, but it's slow to the point where it's not really usable. Maybe there are smaller models you can run that will perform better.

What size model did you try and how much memory does your M1 have? See my other comment, my experience has been that llama3 was very fast on an M1.

Re: Ask HN: Any way to borrow compute from Apple M1

#35

Post by neom » Thu, Jul 18, 2024, 1:00 PM UTC

I feel like Ollama just came out and now y'all are doing model based laptop resource sharing.

Should I take this as an indicator that embedded GenAI is moving quite quickly?

(Also just wanted to say I find this thread incredibly cool generally, some very interesting stuff going on!!! :D )

Re: Ask HN: Any way to borrow compute from Apple M1

#36

Post by paxys » Thu, Jul 18, 2024, 1:05 PM UTC

The entire point of embedded models is that you can run them locally. If it'll anyways take an internet roundtrip then what's the point of connecting to your friend's laptop over a cloud GPU or a managed service like ChatGPT-4o?

Re: Ask HN: Any way to borrow compute from Apple M1

#37

Post by crtasm » Thu, Jul 18, 2024, 1:44 PM UTC

paxys wrote: post #36

The entire point of embedded models is that you can run them locally. If it'll anyways take an internet roundtrip then what's the point of connecting to your friend's laptop over a cloud GPU or a managed service like ChatGPT-4o?

Presumably a cloud GPU is not $0 ?

Re: Ask HN: Any way to borrow compute from Apple M1

#38

Post by drivingmenuts » Thu, Jul 18, 2024, 1:45 PM UTC

paxys wrote: post #36

The entire point of embedded models is that you can run them locally. If it'll anyways take an internet roundtrip then what's the point of connecting to your friend's laptop over a cloud GPU or a managed service like ChatGPT-4o?

It won't cost actual money?

Re: Ask HN: Any way to borrow compute from Apple M1

#39

Post by wiradikusuma » Thu, Jul 18, 2024, 2:33 PM UTC

NBJack wrote: post #31

saberience wrote: post #13

Earlier quoted context omitted.

I have an M3 chip in my laptop, it has more memory than my 4090 but it's still way slower when inferencing. So as long as the model fits in memory, Nvidia GPUs are going to be way faster just because they have more/faster compute cores. Of course, if the model fits in memory on your M chip and doesn't in your Nvidia chip, the M chip wins by default. However, I would say, if you load a 70B model in your M chip, while…

The latest Nvidia drivers offer an option to start using system memory when VRAM is insufficient. It certainly slows things down, but it does work. It's not perfect in my experience, but it is an alternative for large models.

Sounds great! Do you have source maybe a wiki?

Re: Ask HN: Any way to borrow compute from Apple M1

#40

Post by moffkalast » Thu, Jul 18, 2024, 3:19 PM UTC

Shadowmist wrote: post #32

moffkalast wrote: post #14

Earlier quoted context omitted.

You mean the llama.cpp server API right? Ollama keeps taking credit for things they put a thin wrapper around, and it's seriously annoying.

What’s the llama.cpp CLI equivalent of `ollama pull`?

wget ?