Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Any way to borrow compute from Apple M1

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 47 posts

Re: Ask HN: Any way to borrow compute from Apple M1

#13

Post by saberience »

BossingAround wrote: post #6

An off-topic question, are Apple's M-series chips any good at current AI/ML work? How does it compare with dedicated GPUs?

I have an M3 chip in my laptop, it has more memory than my 4090 but it's still way slower when inferencing. So as long as the model fits in memory, Nvidia GPUs are going to be way faster just because they have more/faster compute cores.

Of course, if the model fits in memory on your M chip and doesn't in your Nvidia chip, the M chip wins by default. However, I would say, if you load a 70B model in your M chip, while it WILL work, the tokens/sec will be slow as fuck... so it kinda doesn't matter anyway.

Re: Ask HN: Any way to borrow compute from Apple M1

#14

Post by moffkalast »

loktarogar wrote: post #11

This was bouncing around the last few days, if you have a few devices as well as the M1 (though i'm not sure it able to work over the internet as opposed to a local network): https://github.com/exo-explore/exo Otherwise set up Ollama's API

You mean the llama.cpp server API right? Ollama keeps taking credit for things they put a thin wrapper around, and it's seriously annoying.

Re: Ask HN: Any way to borrow compute from Apple M1

#16

Post by whywhywhywhy »

M1 can’t really handle SD the inference times are closer to a minute and with SDXL you can feel the machine straining under it, battery depletes quick and the machine often completely freezes up for a second if you’re trying to do other things at the same time (M1 Max 32gb).

Think you’d be way better off just paying for a service designed for this or renting a GPU from a service set up for this cost won’t be that significant.

Re: Ask HN: Any way to borrow compute from Apple M1

#17

Post by enoch2090 »

For llama3 just ask him to install ollama and serve the model. Ollama has auto memory management and will free the model when not used, and whenever you make a call to the API (do let your friend know before you do this) ollama will reload the model back to memory again.

Not sure whether there are anything similar for SD though.

Re: Ask HN: Any way to borrow compute from Apple M1

#18

Post by Keyframe »

yieldcrv wrote: post #12
BossingAround wrote: post #6

An off-topic question, are Apple's M-series chips any good at current AI/ML work? How does it compare with dedicated GPUs?

If apple offers these in a data center that’s open access it’s game over for NVDA

Based on which fantasy premise?

Re: Ask HN: Any way to borrow compute from Apple M1

#20

Post by mrweasel »

BossingAround wrote: post #6

An off-topic question, are Apple's M-series chips any good at current AI/ML work? How does it compare with dedicated GPUs?

I've only tried it on my M1, running Llama-3 via Ollama. It works, but it's slow to the point where it's not really usable. Maybe there are smaller models you can run that will perform better.
Post reply on HN