On a related note, has anyone been able to utilize Apple silicon GPUs? Running CPU-only is incredibly slow (and sad, since i've got these Apple accelerators idle!)
Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
151–160 of 184 posts
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#152here's a direct app store link, if anyone wants to try the iPhone app immediately: https://apps.apple.com/us/app/draw-things-ai-generation/id64... congratulations to liuliu on the launch!
Crashes on my 13 Mini, which is unexpected! Anything we need to do to prep the phones?
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#153On a related note, has anyone been able to utilize Apple silicon GPUs? Running CPU-only is incredibly slow (and sad, since i've got these Apple accelerators idle!)
This repo is working well on my M1 Mini.
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#154here's a direct app store link, if anyone wants to try the iPhone app immediately: https://apps.apple.com/us/app/draw-things-ai-generation/id64... congratulations to liuliu on the launch!
Crashes on my 13 Mini, which is unexpected! Anything we need to do to prep the phones?
I think that probably is to be expected, unfortunately. The model is 2GB; per the article iOS will kill apps that use 2GB on 4GB RAM devices (and the 13 Mini is 4GB).
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#155I wonder how long before the appstore policies are updated to require models be embedded in apps.
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#156Could in-memory compression used to bring down the RAM requirements? There are some performance compressors like Blosch tuned for this: https://www.blosc.org/pages/blosc-in-depth/ “Faster than memcpy” is the slogan.
Should be doable for parameters but at that point, you don't need compression rather just LLM.Int8 tricks would be sufficient. For activations, I wrote about it a while back: https://liuliu.me/eyes/reduce-another-70-memory-usage-for-de...
It is not as useful for this case (inference) because the activations holds long (UNet holds downsampling passes' activations and use that for upsampling) is not that much of a memory (in the range of a few megabytes). If it is for training, it is probably more useful.
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#157Earlier quoted context omitted.
You should see a warning when selecting that size? 4GiB model cannot run at that resolution until someone implemented FlashAttention on Metal :)
Nice work :) Porting FlashAttention to Metal will be quite hard. Because for performance reasons, they did a lot of shenanigans to respect the memory hierarchy. Thankfully, you can probably do something slower but more adapted to your memory constraints. If you relax this need for performance and allow some re-computations, you can write a qkvatt function which takes q,k,v and a buffer to store the resulting attentio…
[deleted]
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#158Earlier quoted context omitted.
I have a Twitter thread on this: https://twitter.com/liuliu/status/1587978815208407041?s=46&t... (Note that at that time, there is an implemention bug in inpainting model that caused the weirdness that I need to manually fix)
liuliu, this is simply incredible. Were you focused on just making it work on the iPhone, or do you think you will keep adding functionalities to the app? Do you think it will ever be possible to train one's own model on an iPhone?
I think that fine-tuning the whole model (a.k.a. Dreambooth) on iPhone would require more RAM / processing power than it currently has. More viable path is to implement Hypernetwork + Textual Inversion, that is within possibility of today's hardware.
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#159Update: Draw Things uses “One-time photo selection” which according to Settings > Privacy & Security > Photos “Even if your photos were recently shown to you to select from, the app did not have access to your photo library.” Still, I didn’t realize apps could save to Photos without explicitly asking permission. I don’t recall giving “Draw Things” permissions to access my photo library, yet the app is able to save to…
I use PHPickerViewController: https://developer.apple.com/documentation/photokit/phpickerv... which runs out of the process such that when you want to select a photo into the app, I have no access to any information about your other photos and the location info is erased from what PHPickerViewController passed to me.
When save the photo, I only use UIImageWriteToSavedPhotosAlbum (https://developer.apple.com/documentation/uikit/1619125-uiim...) which asks you permission to write to the album, not read permission (they are separate). There are more things I can do if I have read permissions (like create a "Draw Things" collections and save to that, rather than save to generic Camera Roll). Ultimately decided to not do that because I don't want more permissions than I minimally absolutely need.
Re: Stretch iPhone to its limit: 2GiB Stable Diffusion model runs locally on device
#160I wonder how long before the appstore policies are updated to require models be embedded in apps.
App Store has a CDN-like feature that allows uploading large resources separately and download them after running the app for the first time.