Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

minimaxir.com

Post reply on HN
Print view View original

61–66 of 66 posts

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#61

Post by minimaxir »

astrange wrote: post #60
minimaxir wrote: post #27

Earlier quoted context omitted.

> I didn't spot in the article, how many `wrong` images were there? From a quick skim of the code it looks like maybe 6 per keyword with 13 keywords, so not many at all. ~100 is surprisingly little feedback to steer the model this well. Correct: 6 CFG values * 13 keywords = 78 images. Some of them aren't as useful though; apparently "random text" results in old-school SMS applications sometimes! LoRAs only need 4-5 i…

I noticed some of your bad prompts are a little "wishcasted", although that's pretty common. People put stuff like "bad hands" into every model assuming it'll work, but it only works on NovelAI descendents because that's based on Danbooru which has a "bad hands" tag.

Some of the generated hands are really bad: I opted not to include them to avoid disturbing imagery.

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#62

Post by carbocation »

Auracle wrote: post #52
jononor wrote: post #48

Earlier quoted context omitted.

Very cool! How many images did you use to create the LoRa of your dog? Do you have any guide to recommend?

It was about 30 images, though I'm planning on adding more and training again sometime. Either that or splitting it up between when her hair is short and when it's long, as it really changes how she looks. This isn't what I used for my dog's LoRa but I used it for my wife and it worked better than what I was doing before (Adafactor): https://civitai.notion.site/SDXL-1-0-Training-Overview-4fb03... I'd recommend increa…

That's a very low learning rate -- between 2-3 orders of magnitude lower than what I've seen for that number of steps. I'll have to give it a try.

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#63

Post by SkyPuncher »

hospitalJail wrote: post #25
rabuse wrote: post #9

Creating art with stable diffusion has become such a fun hobby of mine. The difference between SD 1.5/2.0 and SDXL is massive, and it's impressive how quickly the quality is improving with this stuff.

>The difference between SD 1.5/2.0 and SDXL is massive, Can you explain? I havent used SDXL yet, but I spent a ton of time in 1.5. So far I gathered: >Higher res >higher 'quality' But given I was using realistic vision 3 for so long, I never had a quality issue. With upscaling, I never needed higher res.

For simplicity, it feels like SDXL has better "defaults". You don't have to include a bunch boilerplate keywords to wrangle it into generating good images.

The flip side is I've found it a bit harder to tweak prompts

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#64

Post by esperent »

SkyPuncher wrote: post #63
hospitalJail wrote: post #25

Earlier quoted context omitted.

>The difference between SD 1.5/2.0 and SDXL is massive, Can you explain? I havent used SDXL yet, but I spent a ton of time in 1.5. So far I gathered: >Higher res >higher 'quality' But given I was using realistic vision 3 for so long, I never had a quality issue. With upscaling, I never needed higher res.

For simplicity, it feels like SDXL has better "defaults". You don't have to include a bunch boilerplate keywords to wrangle it into generating good images. The flip side is I've found it a bit harder to tweak prompts

I've found it very hard to create different styles with SDXL. If you want photorealism, anime, sci-fi, or somewhere in between, it's amazing.

But I've been trying to get it to generate equivalent quality in other styles, e.g. watercolor, abstract painting etc. It doesn't seem to be easy - the quality drops a lot and it's harder to avoid weird results like people wearing enormous hats or distorted perspective.

Admittedly I haven't spent a huge amount of time on this because generation is just a bit too slow to be enjoyable on my machine. Has anyone else had success here?

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#65

Post by Auracle »

carbocation wrote: post #62
Auracle wrote: post #52

Earlier quoted context omitted.

It was about 30 images, though I'm planning on adding more and training again sometime. Either that or splitting it up between when her hair is short and when it's long, as it really changes how she looks. This isn't what I used for my dog's LoRa but I used it for my wife and it worked better than what I was doing before (Adafactor): https://civitai.notion.site/SDXL-1-0-Training-Overview-4fb03... I'd recommend increa…

That's a very low learning rate -- between 2-3 orders of magnitude lower than what I've seen for that number of steps. I'll have to give it a try.

I should have been clear - I'm using the Prodigy settings on that page, not the Adafactor one. You set the learning rate to 1 and the scheduler to cosine, but the real learning rate is figured out by the optimizer.

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#66

Post by pedrosorio »

MuffinFlavored wrote: post #51
theptip wrote: post #24

In general I'm really interested by the concept of personalized RLHF. As we have more and more interactions with a given generative AI system, it seems we'll start to have enough interaction data to meaningfully steer the output towards our personal preferences. I hope the UIs improve to make this as transparent as possible. Just thinking about how to productize this flow, it should be quite easy to implement the "th…

> RLHF Reinforcement Learning from Human Feedback Aren't these systems already trained to score good things higher and bad things worse dictated by human feedback?

personalized RLHF is the keyword
Post reply on HN