Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds

snap-research.github.io

Post reply on HN
Print view View original

41–50 of 54 posts

Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds

#41

Post by bee_rider »

detrites wrote: post #33
prepend wrote: post #29

Earlier quoted context omitted.

The issue is that it’s a controlled video from the author. I can still get Wi-Fi and Bluetooth in airplane mode so the airplane mode sign isn’t “proof” enough for me to accept the paper conclusions. I hope it’s real, but posting a YouTube to assuage the “show me the implementation” isn’t going to help with my cynicism.

The paper describes the implementation including a detailed breakdown of the optimisation algorithm itself. It's also plausible an iPhone 14 Pro could do it given its memory b/w, ops/s and that it can fit the SD model in RAM.

Hopefully someone will replicate it, and then we’ll know if they were telling the truth.

Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds

#42

Post by renonce »

sdflhasjd wrote: post #38
pedrovhb wrote: post #35

I've seen at least a couple of papers before with similar claims, and still nothing I can run on my phone, so I'm not holding my breath yet. It seems a bit disingenuous to compare with Stable Diffusion taking 50 steps, though; with the newer schedulers you can consistently get great images in 12 steps of diffusion, probably less if you're a bit careful with exact parameters/model fine tuning choice.

Steps isn't an apples-to-apples comparison though because some schedulers have longer steps (time-wise) than others.

At least let’s compare quality within the same amount of time, regardless of steps.

Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds

#43

Post by MattRix »

renonce wrote: post #42
sdflhasjd wrote: post #38

Earlier quoted context omitted.

Steps isn't an apples-to-apples comparison though because some schedulers have longer steps (time-wise) than others.

At least let’s compare quality within the same amount of time, regardless of steps.

The first graph on the page compares the quality of it, showing that it has higher quality results despite taking only 2 seconds vs results from Stable Diffusion that take 1.4 minutes.

Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds

#44

Post by bee_rider »

TeMPOraL wrote: post #30
devadvance wrote: post #4

From the paper: > In this work, we present the first text-to-image diffusion model that generates an image on mobile devices in less than 2 seconds. To achieve this, we mainly focus on improving the slow inference speed of the UNet and reducing the number of necessary denoising steps. As a layman, it's impressive and surprising that there's so much room for optimization here, given the number of hands on folks in the…

> As a layman, it's impressive and surprising that there's so much room for optimization here, given the number of hands on folks in the OSS space. There's only so many folks in OSS space that are capable of doing work from this angle. There are more who could be micro-optimizing code, but the most end up developing GUIs and app prototypes and ad-hoc Python scripts that use the models. At the same time, the whole fie…

It isn’t obvious to me that these models produce something similar to our brains’ output. We can imagine images of course, but the level of quality is hard to define, and it is really hard and time consuming to save the output of an imagined image.

People paint or draw imagined images, but that’s a slow process and there’s a feedback loop going on throughout the whole thing (paint a bit, see how it looks, try a little happy tree, didn’t work out, turn it into a cloud). If we include the time spent painting and reconsidering, image generation using humans is pretty expensive.

An iPhone battery holds tens of watt-hours. A painting might take hours to make (I don’t paint. A couple hours is quick, right?), so if the brain is burning tens of watts in that time, the total cost could be in the same ballpark as generating images until your battery dies. But of course it is really hard to make an apples-to-apples comparison here because the human spends a lot of energy just keeping the lights on while bandwidth is limited by the rate of arm-movement.

Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds

#45

Post by MattRix »

bredren wrote: post #11

> Text-to-image diffusion models can create stunning images from natural language descriptions that rival the work of professional artists and photographers. I’m all for the continued advance of diffusion models. If this paper offered evidence of quantitative and qualities measurement techniques for determining human preference for art or photos based on a prompt, I’d get it the phrasing. But having the first sentenc…

You’re reading something into the text that just isn’t there. “Rival” means it’s good enough to be competitive, it doesn’t mean it’s better. The statement is hardly surprising or controversial. It’s fairly obvious to anyone who has looked at the output of these kinds of models.

Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds

#46

Post by MattRix »

bufferoverflow wrote: post #39

I would rather have more quality than speed. The output of this model reminds me of Midjourney 3.

There are lots of models/approaches to look at if you want improved speed. It seems bizarre to not also want the speed to increase.

Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds

#47

Post by tpmx »

asutekku wrote: post #26
suyash wrote: post #23

Earlier quoted context omitted.

I'll believe thier hypothesis when I can run the open source code on my iPhone in 2 seconds, doubt it's that fast.

They included a video: https://www.youtube.com/watch?v=zK5PQ3Oj_L8

In that video it takes 5 seconds.

Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds

#48

Post by lukasb »

pedrovhb wrote: post #35

I've seen at least a couple of papers before with similar claims, and still nothing I can run on my phone, so I'm not holding my breath yet. It seems a bit disingenuous to compare with Stable Diffusion taking 50 steps, though; with the newer schedulers you can consistently get great images in 12 steps of diffusion, probably less if you're a bit careful with exact parameters/model fine tuning choice.

What newer schedulers?

Re: SnapFusion: Text-to-Image Diffusion Model on Mobile Devices Within Two Seconds

#50

Post by digging »

anonylizard wrote: post #5
behnamoh wrote: post #2

This is insane! But it makes me wonder if we've reached a local maximum in AI where the current methods are great at generating still images but they're pretty much uncontrollable. Like if you ask the AI to generate a dog, is it really possible to prompt every single detail so it creates exactly what you have in mind, or is it more like a trust situation where you just accept whatever the AI generates for you?

Pure text->image is impossible to get exactly, given there's 10000 possibilities for a dog. Even if text prompts eliminate 99.9% of probabilities, it there's still 10 possible images. However, with stuff like controlnet, it's already possible, and will be solved within a year. Yes you can specify every exact detail, but you need to feed it a sketch, or a skeletal pose, or a reference image of the dog... Also, you can…

> 10000 possibilities for a dog

I suppose it's not your main point, but that number is off by... probably about 10,000 orders of magnitude.

Post reply on HN