I wonder if any of these tricks would be applicable to make a version of Stable Diffusion which could run on the Steam Deck.
Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
41–50 of 184 posts
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#42This is super cool. I just tried the default prompt on my iPhone 13 with the image size set to 768x512 and using the 3D Model (Redshift v1) and it just crashed the whole phone and restarted. Just like when I get BSOD's at work on my Windows GPU desktop :)
[deleted]
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#43liuliu has always been a fucking god
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#44Could in-memory compression used to bring down the RAM requirements? There are some performance compressors like Blosch tuned for this: https://www.blosc.org/pages/blosc-in-depth/ “Faster than memcpy” is the slogan.
In-memory compression means the memory is inherently dirty memory
On Apple platforms if you mmap a read-only file into the process address space, then it is "clean" memory. It is clean because the kernel can drop it at any time because it already exists on disk. You essentially can offload the memory management to the kernel page cache.
The downside is that if you run up to the limit and the "working set" can't fit entirely in memory, then you run into page faults which incur an I/O cost.
The advantage is that the kernel will drop the page cache before it considers killing your process to reclaim memory.
That said, I don't know the typical access patterns for neural network inference, so I don't know how the page faults would effect performance
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#45This is super cool. I just tried the default prompt on my iPhone 13 with the image size set to 768x512 and using the 3D Model (Redshift v1) and it just crashed the whole phone and restarted. Just like when I get BSOD's at work on my Windows GPU desktop :)
You should see a warning when selecting that size? 4GiB model cannot run at that resolution until someone implemented FlashAttention on Metal :)
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#46Earlier quoted context omitted.
Memory compression? I can't find any good resources to read about it, any hints? I'm having trouble imaging how could it possibly work without totally destroying performance.
On Mac, you can find Compressed memory in Activity monitor. It’s something similar to swap - apps do not need to have built in support for it.
Many operations are actually a lot faster with compressed memory than without. It's all about where the bottleneck is.
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#47[deleted]
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#48This is some impressive work.
You might like to look at the work HuggingFace has been doing (on non-iOS versions). They can run it in under 1GB RAM:
> If is also possible to chain it with attention slicing for minimal memory consumption, running it in as little as https://huggingface.co/docs/diffusers/optimization/fp16#offl...
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#49the developer is about to have a MASSIVE hosting bill
the download restarts from 0% if the app is sent to the background, as there does not seem to be a download manager. This is especially problematic for the large 1.5gb file.
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#50the developer is about to have a MASSIVE hosting bill the download restarts from 0% if the app is sent to the background, as there does not seem to be a download manager. This is especially problematic for the large 1.5gb file.
I am using Cloudflare R2, which doesn't have egress fee and I am getting about 5k Class B operations right now. Unless Cloudflare changes their end of the deal, I think it is OK.