Colorizing black and white photos with deep learning • Hacker News Board

Re: Colorizing black and white photos with deep learning

#31

Post by zan2434 » Fri, Oct 13, 2017, 6:47 PM UTC

iluvmylife wrote: post #2

The averaging problem in colorization is interesting. If it learns that an apple can be red, green and even yellow - how does it know how to color it? A HN user in an earlier thread suggested to use a fake/real colorization classifiers as a loss function. [1] But I still feel that it would not solve the averaging problem. It would hop between different colors and probably converge to brown. I haven’t come across a pl…

Although it is quite egregious here - this is not a problem inherent to colorization but rather to generative models in general.

Using something akin to a variational autoencoder would solve this problem, because it learns a distributional approximation rather than a single point estimate of the color, and then the random noise vector input allows one to sample from this output distribution. Similarly, Mixture Density Networks allow you to model a distribution and then sample from it.

Re: Colorizing black and white photos with deep learning

#32

Post by perturbation » Fri, Oct 13, 2017, 7:43 PM UTC

terrabytes wrote: post #3

This is refreshing. I’ve been learning machine learning through Kaggle. Recently and I’m a bit tired with the “tuning hyperparameter” culture. It rewards people that have the pockets to spend on computing power and the time to try every parameter. I’m starting to find problems that don’t have a simple accuracy metric more interesting. It forces me to understand the problem and think in new ways, instead of going down…

You can be a little less brute force if you use something like hyperopt (http://hyperopt.github.io/hyperopt/) or hyperband (https://github.com/zygmuntz/hyperband) for tuning hyperparameters (Bayesian and multi-armed bandit optimization, respectively). If you're more comfortable with R, then caret supports some of these types of techniques as well, and mlr has a model-based optimization (https://github.com/mlr-org/mlrMBO) package as well.

These types of techniques should let you explore the hyperparameter space much more quickly (and cheaply!), but I agree - having money to burn on EC2 (or access to powerful GPUs) will still be a major factor in tuning models.

Re: Colorizing black and white photos with deep learning

#33

Post by WalterBright » Fri, Oct 13, 2017, 7:54 PM UTC

vwcx wrote: post #22

As a professional photo editor and historian, colorized photos really agitate me. I'm all for the creation of new ways to get people to engage with historical primary documentation, but the nuance that these colorizations are interpretations gets lost immediately. Do an image search for "D-Day in color" and try to tell me which results are original color negatives and which are colorizations made by teenagers. I'm al…

I like colorized historical photos. It brings them to life in a remarkable way. I'd like to see the old B+W movies colorized (the Ted Turner ones don't count, as they were done very poorly).

Sure, the colors will never be exact, because we don't know what the original colors were. But that doesn't matter in any material way.

Re: Colorizing black and white photos with deep learning

#34

Post by WalterBright » Fri, Oct 13, 2017, 7:57 PM UTC

vwcx wrote: post #22

As a professional photo editor and historian, colorized photos really agitate me. I'm all for the creation of new ways to get people to engage with historical primary documentation, but the nuance that these colorizations are interpretations gets lost immediately. Do an image search for "D-Day in color" and try to tell me which results are original color negatives and which are colorizations made by teenagers. I'm al…

> I'm also a little confused as to why colorizations always aim to restore color to the equivalent of a faded color negative, with muted tonality and grain.

Many movies set in the past, or that have flashbacks to the past, will often mute the colors. Most modern movies muck around with the colors in post production, too. The worst is when they go for the blue/orange palette.

Re: Colorizing black and white photos with deep learning

#35

Post by WalterBright » Fri, Oct 13, 2017, 8:02 PM UTC

flsantos wrote: post #5

Interesting! If nowadays, pictures are colorized by hand in photoshop, it wouldn't be practical to colorize a full black and white movie. I guess this deep learning approach would solve this problem and colorize old black and white classic movies.

I'd like to see more than colorization. Consider the silent movie "Wings". Very high quality blu-rays are available of it. I would colorize it, remove the dialog cards and dub the dialog, then add foley sound effects and a music soundtrack!

Re: Colorizing black and white photos with deep learning

#36

Post by jacobolus » Fri, Oct 13, 2017, 8:39 PM UTC

vwcx wrote: post #30

justinator wrote: post #28

Earlier quoted context omitted.

I'm also a little confused as to why colorizations always aim to restore color to the equivalent of a faded color negative, with muted tonality and grain. Human logic is funny. Well, as a professional photo editor, you know that if the original b+w photo captures an image with say, a 50% grey value, you don't know if the original color was bright red, or closer to that 50% gray. Bright red has a much higher chroma va…

Thanks. That's a perspective that didn't really dawn on me until your comment. I guess in some ways, the muted colors could be seen less as the colorist's stylistic choice than an appeal to "safe" representation of chroma values that don't offer information about vibrancy.

I don’t think that’s quite it (i.e. that modern amateur photo colorists are intentionally aiming for a 19th century tinted photo style).

The bigger problem is that real images have varying chroma and hue within single shapes, but actually mimicking that when coloring a black and white photograph takes a huge amount of skill, attention to detail, and work. You have to think about what the lighting was like, what material it was striking and at what angle, what lens filters the photographer might have used to capture the image, etc. and then you have to go in and painstakingly paint all of those fine gradations and textures in.

It’s especially difficult to do a convincing job with skin, but most materials are hard to color convincingly.

It’s much easier to apply color to whole shapes as a blob, but this looks terrible (very obviously wrong) when you make the colors very strong.

If you want to try for yourself, get a Photoshop expert friend to find some color photographs (without showing them to you first) and convert them to black and white, applying whatever kind of intermediate processing he/she desires as long as it results in a roughly photorealistic looking black and white image.

Then you try to photorealistically colorize the photo, spending as much time and effort on it as you want. When you have something you are satisfied with, compare to the colored original. It’s very likely that the colorized version will look pretty bad in comparison, even if it looked vaguely okay on its own.

Re: Colorizing black and white photos with deep learning

#37

Post by tzahola » Fri, Oct 13, 2017, 9:15 PM UTC

saip wrote: post #9

flsantos wrote: post #5

Interesting! If nowadays, pictures are colorized by hand in photoshop, it wouldn't be practical to colorize a full black and white movie. I guess this deep learning approach would solve this problem and colorize old black and white classic movies.

Agreed. I imagine this has applications in compression as well. You could stream a movie (or a football game) in black and white and enable each device to color it on the spot. A similar technique could also be done for HD/3D/VR.

Or you can just broadcast the audio from the game and a neural net will synthesize the video on the fly. The possibilities are _endless_!

Re: Colorizing black and white photos with deep learning

#38

Post by quotemstr » Fri, Oct 13, 2017, 9:24 PM UTC

The suggestion of using a classification network as a loss function is brilliant!

I love how we can, in general, elevate the sophistication of ML models by having different models interact and train each other.

Re: Colorizing black and white photos with deep learning

#39

Post by dahart » Fri, Oct 13, 2017, 9:25 PM UTC

vwcx wrote: post #22

As a professional photo editor and historian, colorized photos really agitate me. I'm all for the creation of new ways to get people to engage with historical primary documentation, but the nuance that these colorizations are interpretations gets lost immediately. Do an image search for "D-Day in color" and try to tell me which results are original color negatives and which are colorizations made by teenagers. I'm al…

> the nuance that these colorizations are interpretations gets lost immediately.

True for all AI; neural networks are doing amazing things, but the output is a synthesis, it's a complex interpolation of it's training inputs that may seem "good" or reliable, but it is never to be taken as truth or fact, and it can be arbitrarily wrong with unbounded errors.

> I'm also a little confused as to why colorization always aim to restore color to the equivalent of a faded color negative, with muted tonality and grain. Human logic is funny.

This isn't a human logic problem. Normally colorizations don't affect tonality and grain much, they are putting color splashes on top of a B/W image. This is true of hand-painted colorization, as well as the digital colorization here. You can't get rid of grain or adjust tone by adding color.

One can adjust tone and grain, but then you're doing more than colorizing, and going even further down the road of "interpretation" you're concerned about.

In this particular case, the author did mention "A more diverse dataset makes the pictures brownish". Brown is the average color in natural photos, so minimizing error tends to make things browish. That is separate from leaving faded tone & grain in tact, but it's a second reason why AI based colorization will tend toward muted color.

Re: Colorizing black and white photos with deep learning

#40

Post by web007 » Fri, Oct 13, 2017, 9:36 PM UTC

whatrocks wrote: post #14

I'd like to train this on color comic strips and then run something traditionally black and white like xkcd through it. Seems like it could make the colorization part of hand drawn animation much easier.

You'll probably want something closer to a GAN like pix2pix - https://phillipi.github.io/pix2pix/

An example implementation would look something like edges2cats https://affinelayer.com/pixsrv/