Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PixelNN – Example-Based Image Synthesis

cs.cmu.edu

Post reply on HN
Print view View original

61–70 of 155 posts

Re: PixelNN – Example-Based Image Synthesis

#61

Post by jampekka »

avian wrote: post #25

I found the title somewhat misleading. I was expecting some clever application of the nearest-neighbor interpolation. But this seems to involve neural nets and appears far from "simple" to me (I'm not in the image processing field though).

AFAIU it actually seems to be sort of "just" a clever application of the nearest-neighbor interpolation. The CNN is used to come up with the feature space for the pixels (weights of the CNN), and then each pixel is "copy-pasted" from the training set based on the nearest match.

It seems that this could be used in theory with any feature descriptors, such as local color histograms, although the results wouldn't probably be as good.

Edit: Being a nearest neighbor probably also carries the usual computational complexity problems of the method. If I understand it correctly, they ease this by actually first finding just subset of best matching full images using the CNN features and then do a local nearest neighbor search just in those images.

Re: PixelNN – Example-Based Image Synthesis

#64

Post by adrianN »

IshKebab wrote: post #40
omtinez wrote: post #19

Earlier quoted context omitted.

I haven't read the paper in full detail, but reading between the lines I'm guessing that there's a significant portion of manual processing and hand waving involved. From the abstract, emphasis mine: > the second stage uses a pixel-wise nearest neighbor method to map the smoothed output to multiple high-quality, high-frequency outputs in a controllable manner. My interpretation is that they select training data by ha…

Could be pretty great for police sketch artists. (Although pretty misleading for juries too.)

Just train the model with the suspect's Facebook photostream and presto you have convincing evidence.

Re: PixelNN – Example-Based Image Synthesis

#65

Post by adrianN »

maho wrote: post #21

I hope some day this will generalize to video. I don't care about the exact shape of background trees in an action movie - with this approach, they could be compressed to just a few bytes, regardless of resolution.

Plug in the script and some artist's impressions of the sets and generate the whole movie on the fly.

Re: PixelNN – Example-Based Image Synthesis

#67

Post by yorwba »

sweezyjeezy wrote: post #42
ACow_Adonis wrote: post #4

Earlier quoted context omitted.

Except, and this is really the fundamental catch, it's not so much "enhance" as it is "project a believable substitute/interpretation". You fundamentally can't get back information that has been destroyed/or never captured in the first place. What you can do is fill in the gaps/information with plausible values. I don't know whether this sounds like I'm splitting hairs, but it's really important that the general publ…

> Except, and this is really the fundamental catch, it's not so much "enhance" as it is "project a believable substitute/interpretation". I would argue that this is a form of enhancement though, and in some cases will be enough to completely reconstruct the original image. For example, if I give you a scanned PDF, and you know for a fact that it was size 12 black Ariel text on a white background, this can feasibly le…

See also https://dheera.net/projects/blur

Re: PixelNN – Example-Based Image Synthesis

#68

Post by chrismorgan »

ComputerGuru wrote: post #12
gambiting wrote: post #10

Earlier quoted context omitted.

No, but think of these blurred images as a "hash" - in an ideal situation, you only have one value that encodes to a certain hash value, right? So If you are given a hash X you technically can work out that it was derived from value Y - you're not getting back information that was lost - in a way it was merely encoded into the blurred image, and it should be possible to produce a real image which, when blurred, will…

> you only have one value that encodes to a certain hash value, right? Errr wrong. A perfect hash, yes. But they're never perfect. You have a collision domain and you hope that you don't have enough inputs to trigger a birthday paradox. Look at the pictures on the article. It's an outline of the shoe. That's your hash. ANY shoe with that general outline resolves to that same hash. If your input is objects found in th…

FYI (not because it’s particularly relevant to the sort of hashing that is being talked about, but because it’s a useful piece of info that might interest people, and corrects what I think is a misunderstanding in the parent comment): perfect hash functions are a thing, and are useful: https://en.wikipedia.org/wiki/Perfect_hash_function. So long as you’re dealing with a known, finite set of values, you can craft a useful perfect hash function. As an example of how this can be useful, there’s a set of crates in Rust that make it easy to generate efficient string lookup tables using the magic of perfect hash functions: https://github.com/sfackler/rust-phf#phf_macros. (A regular hash map for such a thing would be substantially less efficient.)

Crafting a perfect hash function with keys being the set of words from the OED is perfectly reasonable. It’ll take a short while to produce it, but it’ll work just fine. (rust-phf says that it “can generate a 100,000 entry map in roughly .4 seconds when compiling with optimizations”, and the OED word count is in the hundreds of thousands.)

Re: PixelNN – Example-Based Image Synthesis

#69

Post by gus_massa »

ACow_Adonis wrote: post #4
otp124 wrote: post #3

I used to roll my eyes at crime television shows, whenever they said "Enhance" for a low quality image. Now it seems the possibility of that becoming realistic are increasing with a steady clip, based on this paper and other enhancement techniques I've seen posted here.

Except, and this is really the fundamental catch, it's not so much "enhance" as it is "project a believable substitute/interpretation". You fundamentally can't get back information that has been destroyed/or never captured in the first place. What you can do is fill in the gaps/information with plausible values. I don't know whether this sounds like I'm splitting hairs, but it's really important that the general publ…

In comparison of output vs original it is clear that the skin color is not accurate.

Re: PixelNN – Example-Based Image Synthesis

#70

Post by radarsat1 »

jeeceebees wrote: post #54
magnat wrote: post #30

Earlier quoted context omitted.

There kind of already is audio equivalent: MIDI. It supplies low resolution timing and pitch information and it's up to synthesizer to produce audio output matching those data.

I think the interesting part would be example based audio synthesis. Could you replace a synthesizer with a neural network which, when fed examples, would allow you to generate sounds / explore some latent space between the examples. For example an approach similar to https://gauthamzz.github.io/2017/09/23/AudioStyleTransfer/ but then using the methods described in the PixelNN paper.

I'll just plug my recent work on my sound synthesis "copier": http://gitlab.com/sinclairs/sounderfeit

It more or less attempts to be what you describe. Not very polished yet, but I had some basic success in modeling the parameter space of a synth, and adding new latent spaces with regularization.

Post reply on HN