Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientific Breakthrough Lets SnappyCam App Take 20 Full-Res Photos Per Second

techcrunch.com

Post reply on HN
Print view View original

21–30 of 262 posts

Re: Scientific Breakthrough Lets SnappyCam App Take 20 Full-Res Photos Per Second

#21

Post by jpap »

0x09 wrote: post #13
jpap wrote: post #5

Earlier quoted context omitted.

I've given a bit more background to the fast JPEG codec on my engineering blog: http://www.snappylabs.com/blog/snappycam/2013/07/31/iphone-k... If you like signal processing, fixed point arithmetic, SIMD cores, and assembly, then this is for you. :-)

So the summary is "JPEG encoder written in assembly with NEON instructions saves images faster than Apple's encoder." That's a cool feat and is a little damning for Accelerate.framework, although the way techcrunch writes it I expected a new kind of fast cosine transform.

Don't forget that SnappyCam pumps both CPU cores when available.

The actual DCT algorithm created and used in the app is different to the typical AAN (Arai, Agui, Nakajima) DCT algorithm that's used in JPEG codecs, at least all the ones I've seen.

It's all about doing as little work as possible to achieve the end result. That's why there's so much asm implementation, with carefully chosen NEON instructions for each step.

Think of it as a cross-layer optimization between algorithm and implementation... done by hand. :-)

Re: Scientific Breakthrough Lets SnappyCam App Take 20 Full-Res Photos Per Second

#22

Post by vidarh »

jpap wrote: post #9
peter_l_downs wrote: post #6

Any chance of this coming to Android soonish? This is seriously cool!

The fast JPEG codec was written for the ARM NEON SIMD coprocessor found in the iPhone. Most Android devices also sport the same architecture, so it is indeed possible. The code for the codec is written in mixed C and assembly, so it can be "easily" ported to Android by making use the JNI. While the R&D for the fast JPEG codec took about a year to perfect, the iOS app took just about the same time to get polished (inc…

If I was you, if you're unsure it might pay off, I'd go to the phone makers and offer them licenses for just the encoder.

Re: Scientific Breakthrough Lets SnappyCam App Take 20 Full-Res Photos Per Second

#26

Post by jpap »

[deleted] wrote: post #20

[deleted]

1. Instragram is only shown if you actually have Instagram installed on your device. ;-) As you might know, Instagram guard their API carefully: we don't yet have general access to it.

2. E-Mail is also only shown if your device has built-in e-mail accounts set up.

3. iTunes App File Sharing is accessible by connecting your device to your Mac/PC via USB and using the iTunes app.

Drop me a line jpap {at} snappylabs.com if you're still having issues. I'd love to help! :D

Re: Scientific Breakthrough Lets SnappyCam App Take 20 Full-Res Photos Per Second

#27

Post by revelation »

jpap wrote: post #16
revelation wrote: post #11

DCT is already lossy [1], so the statements around 8 megapixels are completely pointless, and worst of all, its 1990 lossy technology. Wavelet transformations completely destroy any DCT. That said, if their emphasis is on producing pictures with minimal time delta at highest resolution, algorithms used for still pictures are out of place. Video compression algorithms still use DCT and wavelets, but they do so only af…

You are right on the loss: it's purposefully introduced as a quantization step after performing the DCT, and before losslessly compressing the resulting coefficients with Huffman and encoding to the final JPEG bitstream. Despite all of that, JPEG has now become computationally tractable. I remember the days where it took tens of seconds to encode a JPEG on a commodity machine. Now, with the help of SIMD, we can encod…

It's just bizarre that you would be doing the complete JPEG process at the instant you get the image from the sensor. As you note, there are a plethora of steps that JPEG performs, from color space conversion, to DCT transformation (essentially a gigantic matrix multiplication), Huffman coding, quantization, arithmetic coding and encoding as JPEG bitstream.

The only reason would be that you are pressed for memory or bandwidth, but certainly you have the resources to store one full frame and produce deltas, or just apply part of the JPEG chain, enough to remedy memory pressure. You can always encode it to an actual JPEG after the process.

And yes, pulling single frames from a completely encoded video isn't helpful, because they can get away with more compression. But there are very sophisticated algorithms for eliminating the redundancy between frames, which would have been my first avenue in attempting to do something like this.

Re: Scientific Breakthrough Lets SnappyCam App Take 20 Full-Res Photos Per Second

#28

Post by [deleted] »

Strilanc wrote: post #14
oakwhiz wrote: post #8

Earlier quoted context omitted.

"discrete cosine transform JPG science" Here's a more interesting link directly from the app developers: http://www.snappylabs.com/blog/snappycam/2013/07/31/iphone-k...

Thanks, that's a much better article. > extended some of that research to create a new algorithm > 10,000 lines of hand-tuned assembly code > optimized out pipeline bubbles using a cycle counter tool Color me impressed. It sounds like they really pulled out all the stops.

[deleted]

Re: Scientific Breakthrough Lets SnappyCam App Take 20 Full-Res Photos Per Second

#29

Post by jpap »

pdog wrote: post #23

Any papers on the subject? I'd love to dig deep into some of the technical details behind this.

I'd love to disclose the implementation details, and even release it on GitHub, but unfortunately I have to keep it as a trade secret for obvious reasons.

Perhaps one day! :D It was a tonne of work that I'd love for fellow engineers to take a look at. I learned a massive amount from reading the likes of libTurboJpeg and other OSS implementations; though none of them are using the same DCT as the one I developed for SnappyCam. (They weren't a good fit for the ARM NEON ISA.)

Re: Scientific Breakthrough Lets SnappyCam App Take 20 Full-Res Photos Per Second

#30

Post by dvt »

Why not have a deferred compressor? I assume that just straight-up saving the raw data in memory would be much faster than compressing every frame as you get it.

Couldn't you get significant FPS increases (given that you still had free space/memory available)?

Post reply on HN