Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

minimaxir.com

Post reply on HN
Print view View original

31–40 of 66 posts

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#31

Post by Sharlin »

Zetobal wrote: post #26
minimaxir wrote: post #12

Earlier quoted context omitted.

I posted the original release to /r/StableDiffusion but all the comments are "why not compatable with A1111?" and I can't find a good script to do the conversion: https://www.reddit.com/r/StableDiffusion/comments/15r5k3i/i_... Civitai has syndicated the LoRA: https://civitai.com/models/128708/sdxl-wrong-lora

You will get more users if you provide a safetensors file instead of bin and pickletensors a lot of people have gotten really scared by the malware scare that was going through social media a few months ago.

And for a good reason. A big hunk of floating-point numbers really shouldn't be able to execute arbitrary code. Or any code at all.

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#32

Post by zirgs »

hospitalJail wrote: post #25
rabuse wrote: post #9

Creating art with stable diffusion has become such a fun hobby of mine. The difference between SD 1.5/2.0 and SDXL is massive, and it's impressive how quickly the quality is improving with this stuff.

>The difference between SD 1.5/2.0 and SDXL is massive, Can you explain? I havent used SDXL yet, but I spent a ton of time in 1.5. So far I gathered: >Higher res >higher 'quality' But given I was using realistic vision 3 for so long, I never had a quality issue. With upscaling, I never needed higher res.

From my experiments it seems that SD XL understands prompts much better. While SD 1.5 is great at generating your typical "anime girl with big boobs" stuff - if you try to generate something a little bit more unusual - it usually doesn't generate exactly what you want and seems to straight up ignore large parts of the prompt.

SD XL seems to understand weird and unusual prompts a lot better.

SD XL is capable of generating 1024x1024 images without hacks like "hires fix". That's a very good thing, because hires fix sometimes introduces additional glitches while upscaling. Especially at higher denoising strength. Hires fix fixed the broken face - yay, but the subject now has 3 legs instead of two. Things like that happen far less often with SD XL.

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#33

Post by BoorishBears »

theptip wrote: post #24

In general I'm really interested by the concept of personalized RLHF. As we have more and more interactions with a given generative AI system, it seems we'll start to have enough interaction data to meaningfully steer the output towards our personal preferences. I hope the UIs improve to make this as transparent as possible. Just thinking about how to productize this flow, it should be quite easy to implement the "th…

Implicit RLHF works better than explicit.

It's just like the Mom test: if you ask people to rate you affect their rating

You can have the upscale flow, but you're not limited like Discord based Midjourney was: you can even show all the full sized images and detect that the person copied/saved/right clicked for example

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#35

Post by Sharlin »

chankstein38 wrote: post #21
politelemon wrote: post #10

Please consider posting the LoRa on civitai.com as well as the stable diffusion Reddit. These results look pretty good, looking forward to trying it out. I hadn't realized that the generative images buzz was dying out, since I'm using it regularly I guess it is always in buzz for me.

Agreed I feel like, and I do this a lot as well, people have a tendency to track their habits and assume everyone follows that. From my perspective, the gen image buzz is still as hot as ever! If I lacked excitement for SDXL it was because it felt like the there was no massive jump in image quality to me. Sure the size doubling is great, but it also presents a problem, as I don't always want to generate 1024x1024 ima…

SDXL is more resolution-agnostic than SD1x, 768x768 works fine, but admittedly going down to 512x512 does tend to produce cropped images.

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#36

Post by araes »

Zetobal wrote: post #26
minimaxir wrote: post #12

Earlier quoted context omitted.

I posted the original release to /r/StableDiffusion but all the comments are "why not compatable with A1111?" and I can't find a good script to do the conversion: https://www.reddit.com/r/StableDiffusion/comments/15r5k3i/i_... Civitai has syndicated the LoRA: https://civitai.com/models/128708/sdxl-wrong-lora

You will get more users if you provide a safetensors file instead of bin and pickletensors a lot of people have gotten really scared by the malware scare that was going through social media a few months ago.

Thank you for note on this. I had not heard there were already trojan horse malware being slipped into tensor files as python scripts. Apparently torch pickle uses eval on the tensor file with no filter.

Heard surprisingly little commentary on this topic. The full explanation of how Safetensors are "Safe" can be found from the developer at: https://github.com/huggingface/safetensors/discussions/111

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#37

Post by theptip »

minimaxir wrote: post #27
theptip wrote: post #24

In general I'm really interested by the concept of personalized RLHF. As we have more and more interactions with a given generative AI system, it seems we'll start to have enough interaction data to meaningfully steer the output towards our personal preferences. I hope the UIs improve to make this as transparent as possible. Just thinking about how to productize this flow, it should be quite easy to implement the "th…

> I didn't spot in the article, how many `wrong` images were there? From a quick skim of the code it looks like maybe 6 per keyword with 13 keywords, so not many at all. ~100 is surprisingly little feedback to steer the model this well. Correct: 6 CFG values * 13 keywords = 78 images. Some of them aren't as useful though; apparently "random text" results in old-school SMS applications sometimes! LoRAs only need 4-5 i…

It's really weird that this works. I can see how LoRA on a specific fine-grained concept like Ugly Sonic can work with so few samples, but naively I'd think such a diffuse concept as "!wrong" should require more bits to specify! Like, isn't the loss function already penalizing the model for being "wrong" on all generated images?

(I wonder if there is a follow-up experiment to test if this LoRA'd model actually has better loss on the original training dataset? There's a very interesting interpretability question here I think. Maybe it's just doing much better on a small subset of possible images, but is slightly worse on the remainder of the training data distribution.)

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#38

Post by leopoldhaller »

theptip wrote: post #24

In general I'm really interested by the concept of personalized RLHF. As we have more and more interactions with a given generative AI system, it seems we'll start to have enough interaction data to meaningfully steer the output towards our personal preferences. I hope the UIs improve to make this as transparent as possible. Just thinking about how to productize this flow, it should be quite easy to implement the "th…

You may be interested in the open source framework we're developing at https://github.com/agentic-ai/enact

It's still early, but the core insight is that a lot of these generative AI flows (whether text, image, single models, model chains, etc) will need to be fit via some form of feedback signal, so it makes sense to build some fundamental infrastructure to support that. One of the early demos (not currently live, but I plan on bringing it back soon) was precisely the type of flow you're talking about, although we used 'prompt refinement' as a cheap proxy for tuning the actual model weights.

Roughly, we aim to build out core python-level infra that makes it easy to write flows in mostly native python and then allows you track executions of your generative flows, including executions of 'human components' such as raters. We also support time travel / rewind / replay, automatic gradio UIs, fastAPI (the latter two very experimental atm).

Medium term we want to make it easy to take any generative flow, wrap it in a 'human rating' flow, auto-deploy as an API or gradio UI and then fit using a number of techniques, e.g., RLHF, finetuning, A/B testing of generative subcomponents, etc, so stay tuned.

At the moment, we're focused on getting the 'bones' right, but between the quickstart (https://github.com/agentic-ai/enact/blob/main/examples/quick...) and our readme (https://github.com/agentic-ai/enact/tree/main#why-enact) you get a decent idea of where we're headed.

We're looking for people to kick the tires / contribute, so if this sounds interesting, please check it out.

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#39

Post by davely »

hospitalJail wrote: post #25
rabuse wrote: post #9

Creating art with stable diffusion has become such a fun hobby of mine. The difference between SD 1.5/2.0 and SDXL is massive, and it's impressive how quickly the quality is improving with this stuff.

>The difference between SD 1.5/2.0 and SDXL is massive, Can you explain? I havent used SDXL yet, but I spent a ton of time in 1.5. So far I gathered: >Higher res >higher 'quality' But given I was using realistic vision 3 for so long, I never had a quality issue. With upscaling, I never needed higher res.

I hope you'll forgive me for a bit of a self promotion here, but I think I have an interesting example of SD 1.5 (what most people are familiar with and what most models are based off of) vs SDXL.

Before Phony Stark shut down the Twitter API, I was running a bot that created landscape images with Stable Diffusion v1.5. Its name is Mr. RossBot [1]. Check out the Twitter page for some examples of the quality.

This weekend, I finally updated the code to get it running on Mastodon. In the process, I updated the model to use SDXL [2]. It's running the exact same code otherwise to randomly generate prompts.

The image caption is a simplified version of the prompt. e.g., "Snowcapped mountain peaks with an oxbow lake at golden hour."

Behind the scenes, a whole bunch of extra descriptive stuff is added, so the prompt that SD v1.5 / SDXL get is: "beautiful painting of snowcapped mountain peaks with an oxbow lake at golden hour, concept art, trending on artstation, 8k, very sharp, extremely detailed, volumetric, beautiful lighting, serene, oil painting, wet-on-wet brush strokes, bob ross style"

Anyway, I feel like the quality of SDXL is sharper and it just nails subjects a lot better. It also tries to add reflections and shadows (not always correctly), whereas that didn't happen as much with SD v1.5.

I'm pretty impressed! Especially because Stability.ai had released an update model of Stable Diffusion before SDXL: SD v2.0 and SD v2.1. The results (IMHO) were absolute garbage using the same prompts.

[1] https://twitter.com/mrrossbot

[2] https://botsin.space/@MrRossBot

Re: I Made Stable Diffusion XL Smarter by Finetuning It on Bad AI-Generated Images

#40

Post by homarp »

araes wrote: post #36
Zetobal wrote: post #26

Earlier quoted context omitted.

You will get more users if you provide a safetensors file instead of bin and pickletensors a lot of people have gotten really scared by the malware scare that was going through social media a few months ago.

Thank you for note on this. I had not heard there were already trojan horse malware being slipped into tensor files as python scripts. Apparently torch pickle uses eval on the tensor file with no filter. Heard surprisingly little commentary on this topic. The full explanation of how Safetensors are "Safe" can be found from the developer at: https://github.com/huggingface/safetensors/discussions/111

also safetensors security audit: https://huggingface.co/blog/safetensors-security-audit
Post reply on HN