Viewing profile — erwannmillon

hahahah i'm doing well tianpei good to hear from you!

yeah data really is everything that was the number one lesson from this whole project

yoo i'm also a researcher on the krea 1 project and happy to answer any questions :)

LCMs are actually exactly SD architecture. LCM is initialized from a regular SD unet and finetuned on a new objective. We are already compiling to get to these times. A lot of othe…

i worked on the gpu/infra side of this, so feel free to AMA Ultimately the LCM is just a SD Unet trained with a new objective, so a lot of SD optimizations are transferable to LCMs…

I actually worked on this feature at krea, happy to answer any technical questions about how this works / is trained

In the decoder, the features from the unet blocks get concatenated with features from the encoder layer through 'skip connections'. The paper discusses how rescaling the backbone f…

Improve SD image quality and reduce artefacts without any additional training, simply by reweighting skip connections in the decoder stage of a diffusion Unet decoder

"Native refiner swap inside one single k-sampler. The advantage is that now the refiner model can reuse the base model's momentum (or ODE's history parameters) collected from k-sam…

I work at krea.ai. We are for sure making art extremely accessible, but we consider it enhancing creativity rather than replacing. I fully agree that being able to generate an aest…

So we're using a color space that has two channels dedicated entirely to color, which is the only thing the model needs to learn. The model doesn't need to touch the lightness chan…

Think inference time was on the order of 4-5seconds per image on a v100, which you can rent for like .80 cents an hour, though you can get way better gpus like a100s for ~1.1 usd/h…

Yeah, if you have a high res image, you can get color info at super low-res and then regenerate the colors at high res with another model. (though this isn't an efficient approach …

Depends, given the low res, the 3x64x64 pixel space image is smaller than the latents you would get from encoding a higher-res image with models like VQGAN or the stablediff VAE at…

Think the final training run was only a couple hours on a Colab V100

Took a lot of failed experiments, the model would keep converging to greyscale / sepia images. Think one of the ways I fixed was by adding an greyscale encoder to the arch. Used it…

Btw, I did this in pixel space for simplicity, cool animations, and compute costs. Would be really interesting to do this as an LDM (though of course you can't really do the LAB co…

Fair enough. Honestly this was just a fun side project. I actually coded this up last october when I was doing a deep dive to learn about diffusion models, and saw that no one had …

temporal coherence is def an issue with these types of models, though I haven't tested it out with ColorDiffusion. Assuming you're not doing anything autoregressive (from frame to …

Technically yes, the encoder and unet are convolutional and support arbitrary input sizes, but the model was trained at 64x64px bc of compute limitations. You could probably resume…

Yeah, the model is racist for sure. That's a limitation of the dataset though (celeb A is not known for its diversity, but it was easy for me to work with, I trained this model on …

hahaha it reminded me of some "zoom and enhance" stuff when I was making the animations

You can do this with spatial palette t2i or controlnet. Give a super lores spatial palette as conditioning like this: https://camo.githubusercontent.com/8e488996fd309165fb065b0cd..…

touché, nevertheless, colors go brrrrrrrr