Live data from Hacker News

Content-aware image resize library

github.com

41–50 of 84 posts

Re: Content-aware image resize library

#41

I wrote a GUI for another seam carving library back in 2009[1], and it looks like although in archive mode, you can still access the source as well as the windows / mac binaries. Just tested it and it still works! Not as fancy as photoshop I'm sure, but does have the ability to paint a mask of regions to keep / remove to aid the algorithm and get the desired result. Multi-threaded too! [1] https://code.google.com/arc…

Thanks for that. I used it way back when(mid 2010) to make it look like a a 30' cliff jump was a 60' one. Really impressed my friends with it.

Re: Content-aware image resize library

#42
How well can it remove objects from images? Can it remove an object in a cluttered environment? I'm thinking of image augmentation for deep CNN's. Normally, augmentation can help with invariability to small rotations, translations and flips, but if we could remove objects from images, it might be useful for creating many examples from one image, for image based reasoning.

Re: Content-aware image resize library

#44
post #17

I remember first seeing content aware image scaling at Adobe MAX in Barcelona I think it was. We were completely dumbfounded by what was surely magic happening on stage. When they showed removing objects we just lost it, jaws on the floor and all. That was a fun conference.

I'm the author of this library. The algorithm permits to remove an object from the image, only it needs somehow to be localized. After localization it can be applied to that area a higher energy map, which means that part will be avoided by the seam carver. I will implement a face detection algorithm to automatically exclude faces to get not altered by the carver.

Re: Content-aware image resize library

#46
Happy to see this here! Seam carving is one of those neat little algorithms that hit a sweet spot, being a sum of a few moderately complex algorithms, comes from a readable paper, and gives very satisfying result. I had a lot of fun implementing it in undergrad.

One thing that I always wondered is how Photoshop managed to make it so fast that you can resize in real-time. If n is the width or height of the image, then the dynamic programming part is O(n^2) and needs to be recomputed after every seam removed. Since every seam is a single pixel wide, resizing the image by a non-trivial amount (say half) is O(n^3). There are other papers that remove multiple seams at a time but the quality isn't as good. GPU acceleration perhaps?

Another thing I learned while testing seam carving extensively is that it works nicely in certain scenes/situations, but tends to break down most of the time. The two most common scenarios are: 1) lines that are off by the horizontal by more than a few degrees get cutoff 2) objects loose their proportions and even when the manipulation is not directly obvious, it tends to feel off (in the uncanny valley sense).

I expect some interesting work to use deep learning for content-aware resizing, since neural nets could theoretically be more semantically and holistically aware of objects in the image.

Re: Content-aware image resize library

#47
post #44
post #17

I remember first seeing content aware image scaling at Adobe MAX in Barcelona I think it was. We were completely dumbfounded by what was surely magic happening on stage. When they showed removing objects we just lost it, jaws on the floor and all. That was a fun conference.

I'm the author of this library. The algorithm permits to remove an object from the image, only it needs somehow to be localized. After localization it can be applied to that area a higher energy map, which means that part will be avoided by the seam carver. I will implement a face detection algorithm to automatically exclude faces to get not altered by the carver.

There are a few interesting things to explore in that field, with different twists:

https://algorithmia.com/algorithms/opencv/SmartThumbnail

https://blog.twitter.com/engineering/en_us/topics/infrastruc...

Re: Content-aware image resize library

#48
post #46

Happy to see this here! Seam carving is one of those neat little algorithms that hit a sweet spot, being a sum of a few moderately complex algorithms, comes from a readable paper, and gives very satisfying result. I had a lot of fun implementing it in undergrad. One thing that I always wondered is how Photoshop managed to make it so fast that you can resize in real-time. If n is the width or height of the image, then…

I also thought about augmenting this algorithm with the semantic information output of a neural network, but couldn't come up with a good way to generate training data for semantic segmentation which wouldn't distort objects.

For example, if you consider an image of a roof, the roof pixels will all be semantically similar, but if you remove any of them just based on that, the regular structure will be distorted. Do you have an idea how to solve this?

Re: Content-aware image resize library

#49
post #46

Happy to see this here! Seam carving is one of those neat little algorithms that hit a sweet spot, being a sum of a few moderately complex algorithms, comes from a readable paper, and gives very satisfying result. I had a lot of fun implementing it in undergrad. One thing that I always wondered is how Photoshop managed to make it so fast that you can resize in real-time. If n is the width or height of the image, then…

Do you really need to regenerate the whole dynamic programming table though? It seems to me that it could be kept more conservative and only recalculate the needed parts.

Re: Content-aware image resize library

#50
post #42

How well can it remove objects from images? Can it remove an object in a cluttered environment? I'm thinking of image augmentation for deep CNN's. Normally, augmentation can help with invariability to small rotations, translations and flips, but if we could remove objects from images, it might be useful for creating many examples from one image, for image based reasoning.

I read the paper long time ago, but if I remember correctly, it does not remove objects, the way it works is it iteratively finds a "seam" - a pixel-level path that goes perpendicular to the direction of resizing - that has the lowest "energy" and removes it. So objects are preserved and "empty space" is reduced.
Post reply on HN