Live data from Hacker News

Depixelizing Pixel Art

research.microsoft.com

71–80 of 164 posts

Re: Depixelizing Pixel Art

#71
The issue with depixelizing pixel art is that you need an AI that can identify semantically what the features in the sprite are. When you look at Mario you know you're looking at a stout little man with a hat, you can identify his eyes, ears, nose, hair, hat, moustache, shirt, gloves, overalls and shoes. Given that knowledge, if I were to depixelize Mario's sprite, I'll know the rough shape of his shoes and where the shading should go and what shape it should have, i.e. I can reason what the 3D shape is and how the 2D shape was derived from that. I'll also be able to identify which lines were put there to make sure separate elements can be legible and not make those thicker than needed. Or that his eyes blend with his hat simply because there aren't enough pixels to make them separate.

For simple shapes, like the background, their algorithm works really well, but for complex objects it fails, because it distorts details that were put in with very careful thought and completely depend on the resolution. Such small sprites rely a lot on being looked at by someone who can identify semantically what they're looking at, and any really successful depixelization solution will need to be able to understand what basic shapes the sprite is made of based on what it's supposed to represent.

Re: Depixelizing Pixel Art

#72
post #36

Earlier quoted context omitted.

Only if you look at the first ones. All of Boo, Dolphin, all of SoM, Invaders, Golden Axe, Windows assets and Lost Vikings look great and have vastly reduced artifacts compared to hqx. Also, according to the paper abstract[0], it outputs vector, not scaled up bitmap. [0]: http://research.microsoft.com/en-us/um/people/kopf/pixelart/

I agreed with you, but then I realized they're representing hqx deceptively. There was a video posted here, with them rendered at the same resolution side by side. https://news.ycombinator.com/item?id=7307975 It's really a matter of taste in this video.

> they're representing hqx deceptively.

A big difference in the techniques is that hq4x is a fix-rate upscaling, whereas Microsoft's can handle arbitrary target resolutions. At the same rez, they are pretty equivalent. But, MS has the option to go higher. HQ8x has not yet been developed. I doubt that HQ(arbitrary)x is possible without completely changing the algorithm.

Re: Depixelizing Pixel Art

#73
post #59
post #36

Earlier quoted context omitted.

Only if you look at the first ones. All of Boo, Dolphin, all of SoM, Invaders, Golden Axe, Windows assets and Lost Vikings look great and have vastly reduced artifacts compared to hqx. Also, according to the paper abstract[0], it outputs vector, not scaled up bitmap. [0]: http://research.microsoft.com/en-us/um/people/kopf/pixelart/

No, hq4x is still better on almost all of them. Microsoft's makes everything too round and blobby. It pretty much only works really well when everything is round, like in Boo. But look at what it does to the Golden Axe characters' faces.

hq4x keeps angles when it should not, degrading many features in the process, and that bothers me much more than the overall change to a bit more blobbiness. Look at the outline of Bowser's horns and mouth, Toad's cap outline, Fake Sage's moustache, or Axe Battler 2 sword. The mouths of invaders are also faithfully reproduced. In the Ports icon, the cable is correctly represented, and the port itself has volume instead of becoming two planes as in hq4x. On the Keyboard one, hq4x produces a good icon, although it rounds the keys a little, while "ours" produces a non-faithful in style but nonetheless correct result. Control Panel is interesting, as "ours" looks like the hq4x version has melted, but the hq4x version produces a result that is semantically similar (piano keys, weird mouse shape, angles in letter) leading me to think the original icon is not exactly the best pixel art material (maybe due to palette constraints).

Maybe the results of "ours" aren't exactly faithful in style in some cases, but they're globally consistent, and I find the shapes semantic more truthful. As for hq4x by itself, I actually prefer the original sprites overall.

Re: Depixelizing Pixel Art

#74

Earlier quoted context omitted.

I agreed with you, but then I realized they're representing hqx deceptively. There was a video posted here, with them rendered at the same resolution side by side. https://news.ycombinator.com/item?id=7307975 It's really a matter of taste in this video.

> they're representing hqx deceptively. A big difference in the techniques is that hq4x is a fix-rate upscaling, whereas Microsoft's can handle arbitrary target resolutions. At the same rez, they are pretty equivalent. But, MS has the option to go higher. HQ8x has not yet been developed. I doubt that HQ(arbitrary)x is possible without completely changing the algorithm.

Not just higher, but non integer/power of two.

Re: Depixelizing Pixel Art

#75
post #62
post #57

Earlier quoted context omitted.

As you say: It depends. E.g. the sword looks better with hq4x, because the result isn't rounded.

The Super Mario and Secret of Mana stuff works quite good, except it changes the facial expression on "Peach 2". I'd like to see Donkey Kong Country 1/2/3, Megaman (NES and SNES) and Chrono Trigger processed by this. And not just the sprites cut out of the background but embedded in a scene.

"ours" Peach 1 suddenly looks really sad.

Re: Depixelizing Pixel Art

#76
post #43

I wish they would use this in Windows 8. As far as I can tell, their scaling algorithm for 4k displays is: scale up 2x using nearest neighbor, then randomly drop pixels to get back to 125% or 150%. The result is horrifyingly bad. Not just kind of bad, but unusably bad. (Adding insult to injury, they ClearType the text before scaling.)

While something similar would probably be better, this is heavily optimized for pixel artwork. Lots of curves, shading, and most importantly - limited colors. UI is generally much straighter, less shaded, with far more colors. Here's an example of UI that's scaled with potrace (a similar algorithm): http://i.imgur.com/jDq4M7e.png (left side is the converted side, right side is nearest-neighbor. Top is 3x, bottom is 2…

> That's with the highest amount of colors/depth that potrace allows. On an i7 processor, it takes a LONG time to compute all of that.

OTOH, for icons, you just need to compute that once and then cache the png.

Re: Depixelizing Pixel Art

#77
post #36

Earlier quoted context omitted.

Only if you look at the first ones. All of Boo, Dolphin, all of SoM, Invaders, Golden Axe, Windows assets and Lost Vikings look great and have vastly reduced artifacts compared to hqx. Also, according to the paper abstract[0], it outputs vector, not scaled up bitmap. [0]: http://research.microsoft.com/en-us/um/people/kopf/pixelart/

I agreed with you, but then I realized they're representing hqx deceptively. There was a video posted here, with them rendered at the same resolution side by side. https://news.ycombinator.com/item?id=7307975 It's really a matter of taste in this video.

I took great care to compare 4x, only considering 16x as a zoom, especially since "zoomed in", it blows out the details too much so you lose the global picture. Still, 16x means HiDPI will make it looks crisp as hell for the same physical size.

As for the video, the dent in Mario's cap outline and the similar angles in the background outlines ruins hq4x for me, while I could live with the "200"/"400" lettering (which looks better on hq4x) because it disappears quickly (but does not bode well for other cases where text appears)

Re: Depixelizing Pixel Art

#79

Earlier quoted context omitted.

I would guess it's for performance reasons.

I have 4 i7 cores and a $500 graphics card. I feel like it's possible :) TF2, for example, plays beautifully at 3840x2160@60Hz. Scaling up the graphics for an 800x600 visual novel that plays at 1 frame per 30 seconds is computationally possible. (The good news is that someone has written an emulator for one VN that I'm working through, so I should be able to play with scaling in that way. If I can get it to compile.)

What do you use for displaying 3840x2160@60Hz?
Post reply on HN