SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds
snap-research.github.io
SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds
1–10 of 54 posts
Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds
#2This is insane! But it makes me wonder if we've reached a local maximum in AI where the current methods are great at generating still images but they're pretty much uncontrollable. Like if you ask the AI to generate a dog, is it really possible to prompt every single detail so it creates exactly what you have in mind, or is it more like a trust situation where you just accept whatever the AI generates for you?
Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds
#3Really impressive. The abstract claims sub 2 second generation times, but the youtube demo seems to show generations taking ~6 seconds. Not that I would complain about 6 second generations, my 12gb 3060 probably takes 3-4x as long running SD1.5; perhaps they're not counting the time to load the model, just the active inference time?
Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds
#4From the paper:
> In this work, we present the first text-to-image diffusion model that generates an image on mobile devices in less than 2 seconds. To achieve this, we mainly focus on improving the slow inference speed of the UNet and reducing the number of necessary denoising steps.
As a layman, it's impressive and surprising that there's so much room for optimization here, given the number of hands on folks in the OSS space.
> We propose a novel evolving training framework to obtain an efficient UNet that performs better than the original Stable Diffusion v1.52 while being significantly faster. We also introduce a data distillation pipeline to compress and accelerate the image decoder.
Pretty impressive.
Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds
#5This is insane! But it makes me wonder if we've reached a local maximum in AI where the current methods are great at generating still images but they're pretty much uncontrollable. Like if you ask the AI to generate a dog, is it really possible to prompt every single detail so it creates exactly what you have in mind, or is it more like a trust situation where you just accept whatever the AI generates for you?
Pure text->image is impossible to get exactly, given there's 10000 possibilities for a dog. Even if text prompts eliminate 99.9% of probabilities, it there's still 10 possible images.
However, with stuff like controlnet, it's already possible, and will be solved within a year. Yes you can specify every exact detail, but you need to feed it a sketch, or a skeletal pose, or a reference image of the dog...
Also, you can train a LORA on the subject before hand, if you want to consistently regenerate the subject, with just text.
Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds
#6This is insane! But it makes me wonder if we've reached a local maximum in AI where the current methods are great at generating still images but they're pretty much uncontrollable. Like if you ask the AI to generate a dog, is it really possible to prompt every single detail so it creates exactly what you have in mind, or is it more like a trust situation where you just accept whatever the AI generates for you?
https://vcai.mpi-inf.mpg.de/projects/DragGAN/ Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold
Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds
#7still pretty slow
Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds
#8Really impressive. The abstract claims sub 2 second generation times, but the youtube demo seems to show generations taking ~6 seconds. Not that I would complain about 6 second generations, my 12gb 3060 probably takes 3-4x as long running SD1.5; perhaps they're not counting the time to load the model, just the active inference time?
Yeah stable diffusion has a stage before generation where it transforms the text prompt into data for the model to use, called CLIP Encoding, it runs before every generation, and its probably the stage in the video where you see a spinner in place of the step bar.
Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds
#9This is insane! But it makes me wonder if we've reached a local maximum in AI where the current methods are great at generating still images but they're pretty much uncontrollable. Like if you ask the AI to generate a dog, is it really possible to prompt every single detail so it creates exactly what you have in mind, or is it more like a trust situation where you just accept whatever the AI generates for you?
ControlNet begs to differ
Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds
#10This is insane! But it makes me wonder if we've reached a local maximum in AI where the current methods are great at generating still images but they're pretty much uncontrollable. Like if you ask the AI to generate a dog, is it really possible to prompt every single detail so it creates exactly what you have in mind, or is it more like a trust situation where you just accept whatever the AI generates for you?
It's a human problem really. If you asked an artist to draw a dog you'd have to "trust" them? To control every detail you'd have to tell them every detail, either upfront (whereby the artist night struggle to achieve) or as a series of edits. In the latter case both artist and AI would struggle to keep the look consistent the more edits you make.