Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

minimaxir.com

Post reply on HN
Print view View original

41–50 of 66 posts

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#41

Post by davely »

theptip wrote: post #24

In general I'm really interested by the concept of personalized RLHF. As we have more and more interactions with a given generative AI system, it seems we'll start to have enough interaction data to meaningfully steer the output towards our personal preferences. I hope the UIs improve to make this as transparent as possible. Just thinking about how to productize this flow, it should be quite easy to implement the "th…

> Just thinking about how to productize this flow, it should be quite easy to implement the "thumbs up/down" feedback option on every image generated in the UI, plus an optional text label to override "wrong". Then when you have enough HF (or nightly) you could have a batch job to re-train a new LoRA with your updated preferences.

The AI Horde [1] (an open source distributed cluster of GPUs contributed by volunteers) has a partnership with Stability.ai to effectively do this [2]. They are contributing some GPU resources to AI Horde to run an A/B test.

If a user of one of the AI Horde UIs (Lucid Creations[3] or ArtBot[4]... made by me) requests an image using an SDXL model, they get 2 images back. One was created using SDXL v1.0. The other was created using an updated model (you don't know which is which).

You're asked to pick which image you like better of the two. That's pretty much it. The result is sent back to Stability.ai for analysis and incorporation into future image models.

EDIT: There is a similar partnership between the AI Horde and LAION to provide user-defined aesthetics ratings for the same thing[5].

[1] https://aihorde.net/

[2] https://dbzer0.com/blog/stable-diffusion-xl-beta-on-the-ai-h...

[3] https://dbzer0.itch.io/lucid-creations

[4] https://tinybots.net/artbot

[5] https://laion.ai/blog/laion-stable-horde/

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#42

Post by Footnote7341 »

It became a trend among some data scientists maybe 5 years ago to start recording every keystroke they made on their PC. I'm kind of jealous now when that data is actually kind of useful.

I have a large 30,000 image collection of anime art that I like, that I even competitively ranked for aesthetic score 5 years ago that would come in useful for something like this.

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#44

Post by CuriouslyC »

Sharlin wrote: post #30
hospitalJail wrote: post #25

Earlier quoted context omitted.

>The difference between SD 1.5/2.0 and SDXL is massive, Can you explain? I havent used SDXL yet, but I spent a ton of time in 1.5. So far I gathered: >Higher res >higher 'quality' But given I was using realistic vision 3 for so long, I never had a quality issue. With upscaling, I never needed higher res.

Yes, currently SDXL doesn't really beat the best SD1.5 checkpoints quality-wise. But it (and the currently available checkpoints) shows awesome promise, so give it a six months or so.

The best 1.5 checkpoints are constrained in their output flexibility to achieve the quality they get though, and they don't follow prompts nearly as well as SDXL, so if the model doesn't naturally gravitate towards doing what you want it's very hard to steer it anywhere. SDXL also does a better job with full anatomy, which is the reason shared 1.5 generations tend to be torso up or portrait shots.

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#45

Post by 3abiton »

zirgs wrote: post #32
hospitalJail wrote: post #25

Earlier quoted context omitted.

>The difference between SD 1.5/2.0 and SDXL is massive, Can you explain? I havent used SDXL yet, but I spent a ton of time in 1.5. So far I gathered: >Higher res >higher 'quality' But given I was using realistic vision 3 for so long, I never had a quality issue. With upscaling, I never needed higher res.

From my experiments it seems that SD XL understands prompts much better. While SD 1.5 is great at generating your typical "anime girl with big boobs" stuff - if you try to generate something a little bit more unusual - it usually doesn't generate exactly what you want and seems to straight up ignore large parts of the prompt. SD XL seems to understand weird and unusual prompts a lot better. SD XL is capable of genera…

> While SD 1.5 is great at generating your typical "anime girl with big boobs" stuff - if you try to generate something a little bit more unusual - it usually doesn't generate exactly what you want and seems to straight up ignore large parts of the prompt.

Pretty much experience with SD 1.5, but I'll give XL a try.

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#46

Post by Auracle »

Sharlin wrote: post #30
hospitalJail wrote: post #25

Earlier quoted context omitted.

>The difference between SD 1.5/2.0 and SDXL is massive, Can you explain? I havent used SDXL yet, but I spent a ton of time in 1.5. So far I gathered: >Higher res >higher 'quality' But given I was using realistic vision 3 for so long, I never had a quality issue. With upscaling, I never needed higher res.

Yes, currently SDXL doesn't really beat the best SD1.5 checkpoints quality-wise. But it (and the currently available checkpoints) shows awesome promise, so give it a six months or so.

Currently SDXL is better than SD1.5 checkpoints at pretty much everything other than portraits (or anime drawings) of pretty women.

Unfortunately it seems that's all people want to generate, as is evident when you search for SD on Twitter.

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#47

Post by Auracle »

hospitalJail wrote: post #25
rabuse wrote: post #9

Creating art with stable diffusion has become such a fun hobby of mine. The difference between SD 1.5/2.0 and SDXL is massive, and it's impressive how quickly the quality is improving with this stuff.

>The difference between SD 1.5/2.0 and SDXL is massive, Can you explain? I havent used SDXL yet, but I spent a ton of time in 1.5. So far I gathered: >Higher res >higher 'quality' But given I was using realistic vision 3 for so long, I never had a quality issue. With upscaling, I never needed higher res.

Here's an example using my dog - a trained checkpoint on one of the nicer SD 1.5 models and a LoRA for the SDXL ones: https://imgur.com/a/PklEKwC

The first 3 images are some of my attempts at making her into a Pokemon. Some turned out pretty good (after generating 50+ per type), but I struggled with water in particular. It was hard to get her to have a fin, especially with no additional tail.

I haven't done many in SDXL, but that's the point. I've probably generated..10 images of her as a Pokemon, just when I first trying out the LoRA. The next 2 images are from that, and that was before I had a good ComfyUI workflow to boot.

The rest are various sample images from SDXL showing how versatile it is. In most of those, I only had to generate a few images per prompt to get something pretty darn great. In the Halo 2 one the prompt was literally "an xbox 360 screenshot of cinderdog in Halo 2, multiplayer."

And it made her into a freaking Elite, and it worked wonderfully. I previously tried to generate ones like those candyland images in 1.5 models and the foreground and background just didn't look good. In SDXL it just works.

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#48

Post by jononor »

Auracle wrote: post #47
hospitalJail wrote: post #25

Earlier quoted context omitted.

>The difference between SD 1.5/2.0 and SDXL is massive, Can you explain? I havent used SDXL yet, but I spent a ton of time in 1.5. So far I gathered: >Higher res >higher 'quality' But given I was using realistic vision 3 for so long, I never had a quality issue. With upscaling, I never needed higher res.

Here's an example using my dog - a trained checkpoint on one of the nicer SD 1.5 models and a LoRA for the SDXL ones: https://imgur.com/a/PklEKwC The first 3 images are some of my attempts at making her into a Pokemon. Some turned out pretty good (after generating 50+ per type), but I struggled with water in particular. It was hard to get her to have a fin, especially with no additional tail. I haven't done many in S…

Very cool! How many images did you use to create the LoRa of your dog? Do you have any guide to recommend?

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#49

Post by ChatGTP »

Auracle wrote: post #46
Sharlin wrote: post #30

Earlier quoted context omitted.

Yes, currently SDXL doesn't really beat the best SD1.5 checkpoints quality-wise. But it (and the currently available checkpoints) shows awesome promise, so give it a six months or so.

Currently SDXL is better than SD1.5 checkpoints at pretty much everything other than portraits (or anime drawings) of pretty women. Unfortunately it seems that's all people want to generate, as is evident when you search for SD on Twitter.

Stable diffusion doesn’t grant the user a good imagination or taste unfortunately

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#50

Post by 0cf8612b2e1e »

Zetobal wrote: post #26
minimaxir wrote: post #12

Earlier quoted context omitted.

I posted the original release to /r/StableDiffusion but all the comments are "why not compatable with A1111?" and I can't find a good script to do the conversion: https://www.reddit.com/r/StableDiffusion/comments/15r5k3i/i_... Civitai has syndicated the LoRA: https://civitai.com/models/128708/sdxl-wrong-lora

You will get more users if you provide a safetensors file instead of bin and pickletensors a lot of people have gotten really scared by the malware scare that was going through social media a few months ago.

I would also ask that sha hashes are posted somewhere. It annoys me to know end how difficult it can be to confirm you are using the real model.
Post reply on HN