Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Segmenting comic book frames

vrroom.github.io

Post reply on HN
Print view View original

41–50 of 53 posts

Re: Segmenting comic book frames

#41

Post by astrange »

TheAceOfHearts wrote: post #18
jameshart wrote: post #7

Next AI challenge: try to infer the intended panel reading sequence, and the flow of speech bubbles/narrative. Would potentially be a useful augmentation to a digital comic book reader, refocusing from panel to panel in sequence. Not to mention making comic book content more accessible.

Crunchyroll used to offer a guided reading experience for some of their manga. You could probably build a tool that tags each panel and attempts to figure out the order, and then have a human editor do a validation pass. If you have enough people reading a series you can probably crowdsource the panel sequence.

Kindle just added one. I find it annoying because I can't remember how to edit the single-panel mode once you get into it.

Re: Segmenting comic book frames

#42

Post by hanlec »

tehnub wrote: post #37

There’s a high quality, free comic reader app that doesn’t collect any data that implements this feature quite nicely: https://apps.apple.com/us/app/smart-comic-reader/id151117521...

I haven't Smart Comic before. Thanks for the recommendation. Definitely going to give it a try. (I admit that panel handling is a nice to have feature, but what makes or breaks a comic reader for me is how easy is to sync comics---but this is a long departure from the thread).

YAC Reader [1] has great panel recognition. My other favorite comic reader has attempted but never got something that I could use.

[1]: https://www.yacreader.com/

Re: Segmenting comic book frames

#43

Post by Sardtok »

yjftsjthsd-h wrote: post #10
jameshart wrote: post #7

Next AI challenge: try to infer the intended panel reading sequence, and the flow of speech bubbles/narrative. Would potentially be a useful augmentation to a digital comic book reader, refocusing from panel to panel in sequence. Not to mention making comic book content more accessible.

Is that solvable? It's my personal experience that humans can't reliably do that; I could imagine a machine doing better, but I'm not sure what information it would be working with.

I feel the problem is you often read some panels in the wrong order, but when you read them you realize the order is wrong after the fact, based on the text. The AI could probably use the text as additional information for determining the order.

Would probably never be perfect, but if it gets it right in all but a few outlying cases, that should be good enough.

Re: Segmenting comic book frames

#45

Post by Cthulhu_ »

dylan604 wrote: post #19
doesnt_know wrote: post #17

Earlier quoted context omitted.

I'm more of a manga reader then a western comic one, but why would there be sentiment against this? In what way is the reading order of panels up to interpretation?

You've now focused too much on manga/comics. Stepping back a bit and looking at art more in general and using the provided example of looking at a painting, who are you to tell me where I should be looking. yeah yeah, i see the obvious thing your AI is trying to tell me where to look, but I'm looking at this less obvious thing that really strikes my fancy. maybe it's a blemish. maybe it's a unique brush stroke/techni…

Yeah but isn't telling you where to look for and what to think contrary to the idea of art? Sure, a guide telling you why Van Gogh's brushstroke was interesting may give you a newfound appreciation of what from a distance is a simple looking painting, but treating looking at art as something that should be min/maxed and optimized sounds depressing.

Re: Segmenting comic book frames

#46

Post by Cthulhu_ »

Sardtok wrote: post #43
yjftsjthsd-h wrote: post #10

Earlier quoted context omitted.

Is that solvable? It's my personal experience that humans can't reliably do that; I could imagine a machine doing better, but I'm not sure what information it would be working with.

I feel the problem is you often read some panels in the wrong order, but when you read them you realize the order is wrong after the fact, based on the text. The AI could probably use the text as additional information for determining the order. Would probably never be perfect, but if it gets it right in all but a few outlying cases, that should be good enough.

Counterpoint; isn't reading and re-reading the page after figuring out the intended order part of the experience?

Take video games. From Software games tell stories through breadcrumbs; locations, speech lines, item descriptions, and only over time can you start to connect the dots.

Sure, you can watch a youtuber connect the dots for you, show you what you've missed, and make you aware of your own mental limitations. But it's not the same.

Re: Segmenting comic book frames

#47

Post by bigbillheck »

awdii wrote: post #23

Awesome stuff! We're also working on comic segmentation @ https://toona.io and other stuff for motion comic generation. The synthetic dataset approaches are really interesting, I'm curious if you could use an algorithm like https://en.wikipedia.org/wiki/Flood_fill to aid in segmentation (especially for manga).

This is, as they say, an insult to life itself.

Re: Segmenting comic book frames

#48

Post by GaggiX »

Cthulhu_ wrote: post #46
Sardtok wrote: post #43

Earlier quoted context omitted.

I feel the problem is you often read some panels in the wrong order, but when you read them you realize the order is wrong after the fact, based on the text. The AI could probably use the text as additional information for determining the order. Would probably never be perfect, but if it gets it right in all but a few outlying cases, that should be good enough.

Counterpoint; isn't reading and re-reading the page after figuring out the intended order part of the experience? Take video games. From Software games tell stories through breadcrumbs; locations, speech lines, item descriptions, and only over time can you start to connect the dots. Sure, you can watch a youtuber connect the dots for you, show you what you've missed, and make you aware of your own mental limitations.…

These programs would probably be useful to speed up the translation of comics, for example there are hundreds of thousands of Japanese self-published works and many of them do not have a translation.

Re: Segmenting comic book frames

#49

Post by tehnub »

thih9 wrote: post #38
tehnub wrote: post #37

There’s a high quality, free comic reader app that doesn’t collect any data that implements this feature quite nicely: https://apps.apple.com/us/app/smart-comic-reader/id151117521...

> free comic reader app Your link points to a paid app store entry. Is this the correct app?

Oh I didn't realize, sorry about that. I think it's free on iOS, or it was at least when I downloaded it a few years ago.

Re: Segmenting comic book frames

#50

Post by matroid »

famouswaffles wrote: post #9
jameshart wrote: post #7

Next AI challenge: try to infer the intended panel reading sequence, and the flow of speech bubbles/narrative. Would potentially be a useful augmentation to a digital comic book reader, refocusing from panel to panel in sequence. Not to mention making comic book content more accessible.

https://github.com/ragavsachdeva/magi

This is such a good resource! Thank you!
Post reply on HN