Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Colorizing and restoring old images with deep learning

github.com

Post reply on HN
Print view View original

91–100 of 107 posts

Re: Colorizing and restoring old images with deep learning

#91

Post by citnaj »

sytelus wrote: post #88
citnaj wrote: post #37

Earlier quoted context omitted.

Author here- So I'll just be brutally honest on that one- not all renders are doing that. I cherry picked the one that did that because yeah, it's amazing. There's a simple explanation for why it sometimes doesn't pick up on the guy on tv to color it- The source material is fuzzy and small. I wish I could claim it was something more awesome than that but that's the truth! I'm treating these outputs as an art of selec…

If you had like to brutely honest, you should put randomly selected set, along with the hand-picked set - labeling each set how it was selected. This is a cancer in current deep learning research. You see paper with such a glowing cool examples but in reality they are just hiding all problematic cases while being fully aware of it. If this happened anywhere else in any other domain people would say they got ripped of…

I understand the frustration and in fact share it to a certain extent with science in general. Keep in mind that this wasn't intended to be published as a paper or anything like that. I'm just a software engineer who picked a problem and found a pretty cool solution.

Primarily I thought it was cool because it should be useful in many other image modification domains. And then it blew up in popularity today (didn't expect that). But yeah in the notes in the readme at github I do say this:

>To expand on the above- Getting the best images really boils down to the art of selection.

I added that after getting some feedback similar to yours, because before that, this disclaimer wasn't quite cutting it apparently:

>You'll have to play around with the size of the image a bit to get the best result output.

So yeah I'm trying to stay honest here. I'm not going as far as picking completely random samples, admittedly, but really what I'm trying to drive at here is you can produce cool results with this tool. It's not perfect, but it's a tool. And even if you pick at random, they still look pretty damn good. Just sometimes it renders the tv as color and sometimes it doesn't, and i picked the cool option.

Re: Colorizing and restoring old images with deep learning

#92

Post by citnaj »

sytelus wrote: post #88
citnaj wrote: post #37

Earlier quoted context omitted.

Author here- So I'll just be brutally honest on that one- not all renders are doing that. I cherry picked the one that did that because yeah, it's amazing. There's a simple explanation for why it sometimes doesn't pick up on the guy on tv to color it- The source material is fuzzy and small. I wish I could claim it was something more awesome than that but that's the truth! I'm treating these outputs as an art of selec…

If you had like to brutely honest, you should put randomly selected set, along with the hand-picked set - labeling each set how it was selected. This is a cancer in current deep learning research. You see paper with such a glowing cool examples but in reality they are just hiding all problematic cases while being fully aware of it. If this happened anywhere else in any other domain people would say they got ripped of…

I should add too, just to be clear- I'm not at all involved in academia.

Re: Colorizing and restoring old images with deep learning

#93

Post by citnaj »

md5wasp wrote: post #85
citnaj wrote: post #60

Earlier quoted context omitted.

Author here. That's actually what I find quite fascinating myself about the results- that they look almost perfect at first glance, yet you drill down a bit closer and you see another "zombie hand". The resolution issue you mention is definitely something I'm painfully aware of- it just comes down to lack of memory resources to support bigger renderings. That's going to be something I'm going to try to attack next.

Thanks for being so present in the comments :) I don't know much about ML, but would it be possible to use some kind of attention model to iteratively construct the final colouring? The memory limit of the GPU would then limit the attention region size, but not the maximum image size. Talkin' outta my rear here, though.

I was actually thinking along the same lines because yeah...if you could break this problem down into smaller pieces, it would probably be the most effective way to reduce memory requirements. But I do think that's easier said than done. This is where I think I'll have to rely on Ian Goodfellow and others to come up with another something brilliant for me to stick in the code lol

Re: Colorizing and restoring old images with deep learning

#94

Post by citnaj »

userbinator wrote: post #90

One of the reasons why these photos look so convincingly realistic is the same reason https://en.wikipedia.org/wiki/Chroma_subsampling is done --- the human eye has less sensitivity to colour resolution, and so even relatively vague blobs of colour can evoke the right perception as long as there is sufficient luma detail (provided by the original monochrome image); but if you inspect the photos closely, you'll see th…

That's very interesting to point out Thank you!

Re: Colorizing and restoring old images with deep learning

#95

Post by Mtinie »

ekianjo wrote: post #82
anigbrowl wrote: post #61

Earlier quoted context omitted.

Shooting from the hip here, but I'd much rather you optimized for speed and allowed selection from a rotating palette of local maxima. I noticed your sadness about the limitations on the picture of the indian woman leaning against a tree whose hand came out red, presumably because of vignetting or some chemical inconsistency in the film substrate. But that superposition of possible interpretations on noisy data is so…

> I noticed your sadness about the limitations on the picture of the indian woman leaning against a tree whose hand came out red, I think the biggest problem is that picture is not the hand (its very visible and it could be easily fixed in post processing), it's the blue shade in the clothes that just should not be there. Otherwise, the colors are great (skin and all look very real).

Are we discounting the possibility of the NN calculations resolving to show her hand as it really existed? Tonally, it may have been differentiated from the general population in such a way the algorithm amplified this difference?

Re: Colorizing and restoring old images with deep learning

#96

Post by sixdimensional »

userbinator wrote: post #90

One of the reasons why these photos look so convincingly realistic is the same reason https://en.wikipedia.org/wiki/Chroma_subsampling is done --- the human eye has less sensitivity to colour resolution, and so even relatively vague blobs of colour can evoke the right perception as long as there is sufficient luma detail (provided by the original monochrome image); but if you inspect the photos closely, you'll see th…

This was my thought too - it may not matter if the colors are 100% accurate as long they are enough to trick the human eye and brain into filling in what’s missing. Besides, the reality is, these are not color source photos and will never be. A black and white photo does not contain the color information, it was never captured. All we really can do is use historically accurate colors and afaik, that is the same thing professionally recolorists do as well.

Really neat work!

Re: Colorizing and restoring old images with deep learning

#97

Post by m3at »

This is a cool application of ML. Not to diminish the work, just to point out that humans are more sensitive to luminance than color (hence YUV encoding [1] and others), so it might make inaccuracies less visible.

For example, in "Interior of Miller and Shoemaker Soda Fountain, 1899" the colors from the counter and chairs blend, but the luma help our eyes to separate it.

[1] https://en.m.wikipedia.org/wiki/YUV

Re: Colorizing and restoring old images with deep learning

#99

Post by MayeulC »

citnaj wrote: post #60
MayeulC wrote: post #55

This is quite interesting! The pictures were basically perfect to myeyes, until I scrolled down to the "gotchas" section, at which point I started to notice a lot of details that are wrong, mostly fading colors, on clothes or otherwise. Now, there seems to be a distinct loss of details in the restored images. The network being resolution-limited, is the black-and-white image displayed at full resolution besides the r…

Author here. That's actually what I find quite fascinating myself about the results- that they look almost perfect at first glance, yet you drill down a bit closer and you see another "zombie hand". The resolution issue you mention is definitely something I'm painfully aware of- it just comes down to lack of memory resources to support bigger renderings. That's going to be something I'm going to try to attack next.

Hey, thanks for replying!

However, I feel like you glossed over the proposed workaround, which I feel is appropriate (though more complicated if you want to implement"defade"), and extremely easy to implement.

I took a couple minutes to write an octave script that implement the workaround [1], it would have been even easier if both images had already been distinct files, and perfectly aligned.

The basic idea here is the same as the one behind the YUV transform: our brains are much less sensitive to the chroma channels than the luma channel. So I separate those, and keep the original luma channel, while I use the reconstructed chroma, which is lower-resolution.

Judge the results by yourself, but it seems to me that the end results are a whole lot better: https://imgur.com/a/n2sBYCi

And it could still be improved a lot more (by using the original high-resolution image, and avoiding to hand-align the images).

Edit: also, ironically, the Indigo dye (thus blue clothes) didn't become common before the 1900s [2], so the bias might produce historically-inaccurate images!

[1] https://gist.github.com/MayeulC/626bafbaf925fb3a3c80fdba76b7...

[2] https://en.wikipedia.org/wiki/Indigo_dye#Synthetic_indigo

Re: Colorizing and restoring old images with deep learning

#100

Post by samstave »

citnaj wrote: post #37
nine_k wrote: post #2

The most interesting exhibit for me is "People watching a television set for the first time", where everything is colorized except the TV image , which correctly remains B&W. I wonder what kind of a training set provided the neural network with this notion.

Author here- So I'll just be brutally honest on that one- not all renders are doing that. I cherry picked the one that did that because yeah, it's amazing. There's a simple explanation for why it sometimes doesn't pick up on the guy on tv to color it- The source material is fuzzy and small. I wish I could claim it was something more awesome than that but that's the truth! I'm treating these outputs as an art of selec…

Do you have test pics of a known color pic, then save a black and white version, then colorize the black and white one with this and compare?
Post reply on HN