If you are in the same network try https://pinokio.computer
Ask HN: Any way to borrow compute from Apple M1
41–47 of 47 posts
Re: Ask HN: Any way to borrow compute from Apple M1
#42Earlier quoted context omitted.
I've only tried it on my M1, running Llama-3 via Ollama. It works, but it's slow to the point where it's not really usable. Maybe there are smaller models you can run that will perform better.
What size model did you try and how much memory does your M1 have? See my other comment, my experience has been that llama3 was very fast on an M1.
I was just running: ollama run llama3, So that would be 8B parameters, on a 8GB M1 Air.
Maybe it's just my expectations, but it seems rather slow to process queries. Depending on the prompt somewhere between the 10 - 40 tokens per second, but that very much depends on the prompt.
My complaint is the time between the prompt is entered and output starts.
Re: Ask HN: Any way to borrow compute from Apple M1
#43Earlier quoted context omitted.
Based on which fantasy premise?
The premise where these are readily available for mass purchase, have a hardware and software stack that already works reliably, and have a lower energy footprint than other offerings and somewhat competitive on cost, but that wont be the main selling point, just availability
lower energy but at least they're slow? You need to add up to the same performance level and then consider the cost and running cost. I bet it's not close on that.
Availability maybe, but as you've noted - zero availability for data center environments. Those volumes would also then fall onto TSMC/Samsung/Whatever where Nvidia is stuck as well.
Re: Ask HN: Any way to borrow compute from Apple M1
#44Earlier quoted context omitted.
The premise where these are readily available for mass purchase, have a hardware and software stack that already works reliably, and have a lower energy footprint than other offerings and somewhat competitive on cost, but that wont be the main selling point, just availability
lower energy but at least they're slow? You need to add up to the same performance level and then consider the cost and running cost. I bet it's not close on that. Availability maybe, but as you've noted - zero availability for data center environments. Those volumes would also then fall onto TSMC/Samsung/Whatever where Nvidia is stuck as well.
I’m aware Apple is also beholden to TSMC’s capacity too
sadly
Re: Ask HN: Any way to borrow compute from Apple M1
#45Earlier quoted context omitted.
Probably just because in the full phrasing, ``i would like to "borrow" his gpu``, omitting the scare quotes would paint the picture of their friend unplugging/unsoldering their GPU and lending it to the author.
My friend Kevin was once building a graphics engine, and to test it on various GPUs he’d borrow them from Best Buy and return them within the 30 day window. Since there was a restocking fee, it seemed like a non-harmful and clever way to test a bunch of configurations.
so Kevin is why we can't have nice things....
Re: Ask HN: Any way to borrow compute from Apple M1
#46Earlier quoted context omitted.
The latest Nvidia drivers offer an option to start using system memory when VRAM is insufficient. It certainly slows things down, but it does work. It's not perfect in my experience, but it is an alternative for large models.
Sounds great! Do you have source maybe a wiki?
This might be the best place to start:
https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/s...
Re: Ask HN: Any way to borrow compute from Apple M1
#47This was bouncing around the last few days, if you have a few devices as well as the M1 (though i'm not sure it able to work over the internet as opposed to a local network): https://github.com/exo-explore/exo Otherwise set up Ollama's API
You mean the llama.cpp server API right? Ollama keeps taking credit for things they put a thin wrapper around, and it's seriously annoying.
I don't, in the same way I don't say that i'm taking an internal combustion engine bus to get somewhere - what powers the bus is not relevant to the solution