Pixel Visual Core: Google’s first custom-designed co-processor • Hacker News Board

Re: Pixel Visual Core: Google’s first custom-designed co-processor

#81

Post by londons_explore » Wed, Oct 18, 2017, 5:12 PM UTC

bradleyjg wrote: post #73

bitL wrote: post #53

I wish my Nikon pro camera had a similar chip inside. Now we have two worlds - awesome pro cameras with incapable post-processing chips (I mean beyond RAW and initial processing) and horrible mobile cameras with outstanding post-processing chips.

If you have burst mode and can get the raw frames, then shouldn't you be able to recreate the effect in post-processing? With the severe power constraints of a mobile phone a co-processor makes sense, but I'd think a general purpose desktop processor could easily handle this algorithm.

Can you burst 80 frames @ 60fps of 12 bit 12 megapixel RAW and save all that output, together with the output of gyroscopes and accelerometers timed to the start and end of exposure of every row of the image?

If you can, you might be able to postprocess it... But as far as I know, no DSLR cameras even have gyros, nor sufficiently high burst rates, so that kind of photography is out of the question.

It comes out to over a gigabyte of data per image... and you can't compress it before processing...

Re: Pixel Visual Core: Google’s first custom-designed co-processor

#82

Post by londons_explore » Wed, Oct 18, 2017, 5:19 PM UTC

j_s wrote: post #43

sandGorgon wrote: post #2

This is pretty much Tensorflow on metal. Considering that tensorflow works pretty well on most mobile phone GPU, im betting this feature can be turned on for most phones if Google wants. I would love to see the apples to apples comparison of the Pixel tensorflow computation running on a vanilla snapdragon GPU vs this custom core. Also will be interesting to know why halide and not opencv.

Curious if anyone here knows anything about this alternative for existing phones: Ask HN: What happened to tensorflow lite? | https://news.ycombinator.com/item?id=15494368 (Oct 2017, no comments) >zitterbewegung: They announced it at Google I/O in May See : https://techcrunch.com/2017/05/17/googles-tensorflow-lite-br... I can't find another announcement is this vaporware?

It became tensorflow slim: https://github.com/tensorflow/tensorflow/tree/master/tensorf...

Re: Pixel Visual Core: Google’s first custom-designed co-processor

#83

Post by londons_explore » Wed, Oct 18, 2017, 5:21 PM UTC

TheAdamist wrote: post #74

ggambetta wrote: post #47

> HDR+ addresses this problem by taking a burst of shots with short exposure times, aligning them algorithmically, and replacing each pixel with the average color at that position across all the shots. This is almost exactly how you'd make super-resolution pictures (adding a nearest-point scaling up step at some point! Yet their super-resolution app (Photoscan) has some sort of artificial upper resolution limit. If t…

Super resolution would want to utilize the OIS to artificially shift the lens around instead of cancelling movements. I dont think they are exposing anything like that.

Turns out one doesn't have to. The users hand movements are at least one pixel as long as the phone is handheld. Just use natural movement to do superresolution.

Everything you need is available in the camera2 RAW API.

Using the users hand movements help cancel out various types of per-physical-pixel noise too.

Re: Pixel Visual Core: Google’s first custom-designed co-processor

#84

Post by londons_explore » Wed, Oct 18, 2017, 5:39 PM UTC

sp332 wrote: post #58

GuB-42 wrote: post #46

So what is that IPU exactly? Lots of buzzwords but few actual details about the architecture. Is it like a integer GPU? The massive amount of ALUs suggests something like that. But later, I see "Notably, because Pixel Visual Core is programmable, we’re already preparing the next set of applications.". So, is it FPGA-like?

It's an Image Processing Unit. Architecturally I'd expect it to look like a GPU, maybe with something like the new matrix-multiply unit from their TPU. https://www.slideshare.net/AntoniosKatsarakis/tensor-process... Edit: wrong name

It does not have a TPU matrix multiply unit.

That unit is massive, and wouldn't fit in a small die like this.

It seems likely it's optimized for the same kind of operations though, even if it doesn't have a matrix multiply unit.

I would guess the main intended use for this silicon is running neural networks, even though the initial use case is for photos.

Nearly all AI things on phones (voice recognition, google assistant's local features, keyboard predictive language model, offline translation, etc.) are severely compute limited, and could perform much better with this silicon.

Some features are obvious candidates to put on-device, like realtime recognition of the contents of a photo, realtime wavenet voice synthesis, yet compute limitations preclude it.

Re: Pixel Visual Core: Google’s first custom-designed co-processor

#85

Post by matwood » Wed, Oct 18, 2017, 5:40 PM UTC

ggus wrote: post #60

matwood wrote: post #55

Earlier quoted context omitted.

> This thread seems to be filled with amateur professionals thinking that their iPhone or Pixel will magically do something optically it can't (yet?). This is key. I own some decent camera gear and I'm amazed at what is coming out of all of the top end phones. Of course I can critique and find faults, but the pictures that non-photographers can take with their phones in a matter of seconds are down right amazing. I a…

..or for the day that we have all those magic baked into our "decent camera gear"!

And the problem is that it's mostly software. The main camera people (Cannon/Nikon) are terrible at software. From what I understand Cannon is a bit better, but all of the software I use from Nikon is horrible. I love the lenses, and my Nikon body, but anytime I have to use their software I cringe. I also don't see how they fix this anytime soon. It's just not part of their DNA.

Re: Pixel Visual Core: Google’s first custom-designed co-processor

#86

Post by jrs95 » Wed, Oct 18, 2017, 5:57 PM UTC

MBCook wrote: post #75

jrs95 wrote: post #67

Maybe Google should start designing it's own CPUs/SoCs as well, since Qualcomm doesn't really seem capable of keeping up with what Apple is doing. Snapdragon 835 came out 6 months after the A10, and scored about 2/3 as well at single core, and less than 10% on Geekbench 4. A11 scored twice what the 835 did on single core, and over 50% more on multi core, and the 835 is going to be the best chip available in the U.S.…

I don’t think google has enough volume to make that worthwhile. Wouldn’t you need to sell tens of millions of chips for that to pay off?

If they only used it in their own devices, it would probably be too expensive. If they sold those devices to other manufacturers, they could probably get enough volume to at least avoid taking big losses. It's less about making money from selling the chips, and more about keeping the hardware behind the Android ecosystem competitive.

Re: Pixel Visual Core: Google’s first custom-designed co-processor

#87

Post by ksk » Wed, Oct 18, 2017, 6:06 PM UTC

windexh8er wrote: post #44

jacobolus wrote: post #26

Earlier quoted context omitted.

Their results are quite bad compared to a passably competent Photoshop user willing to spend 2 minutes per picture. But if you need the computer to do all the color adjustment for you, so you can post your snapshot on facebook or print it at the local drug store with no manual intervention, this might be a good enough result.

I think you are missing the point with improvements of this nature. Take for example: you have kids. You're at an event with said kids. While I still own a mirror-less micro 4/3rds, I often time don't have it with me. It's easy to snap hundreds of photos in random life events with a good smart phone camera. I personally have almost 2TB of family photo and video and I've got a long way to go. Can you imagine me spendi…

You're missing who this technology focused article is for, and making a vague argument about a grandmom usecase in a retail product. Sorry, but this article is clearly intended to be viewed by people interested in photography, and also photography-technology. If you claim your tech is awesome, you will get criticized if people don't think it is all that great.

> You shouldn't have an expectation that your $1000 phone competes with a few thousand in camera, glass and filters in the right hands. It won't.

IMHO, it doesn't even beat $500 worth of camera equipment.

Re: Pixel Visual Core: Google’s first custom-designed co-processor

#88

Post by ksk » Wed, Oct 18, 2017, 6:24 PM UTC

matwood wrote: post #55

windexh8er wrote: post #44

Earlier quoted context omitted.

I think you are missing the point with improvements of this nature. Take for example: you have kids. You're at an event with said kids. While I still own a mirror-less micro 4/3rds, I often time don't have it with me. It's easy to snap hundreds of photos in random life events with a good smart phone camera. I personally have almost 2TB of family photo and video and I've got a long way to go. Can you imagine me spendi…

> This thread seems to be filled with amateur professionals thinking that their iPhone or Pixel will magically do something optically it can't (yet?). This is key. I own some decent camera gear and I'm amazed at what is coming out of all of the top end phones. Of course I can critique and find faults, but the pictures that non-photographers can take with their phones in a matter of seconds are down right amazing. I a…

> but the pictures that non-photographers can take with their phones in a matter of seconds are down right amazing.

But that simply highlights that simply owning good gear doesn't mean you can shoot good pictures. You still need to learn lighting, composition, posing your subjects, etc etc, which was always the case!

Re: Pixel Visual Core: Google’s first custom-designed co-processor

#89

Post by joshuamorton » Wed, Oct 18, 2017, 6:57 PM UTC

sandGorgon wrote: post #78

joshuamorton wrote: post #71

Earlier quoted context omitted.

I'm really confused why you keep bringing up opencv. What does opencv have to do with the (presumably) TF ML model running on the IPU that you want to benchmark against the TF ML model running on a GPU?

Because halide is positioned as a high performance improvement over opencv. While I believe it is just an architectural decision to enable a non-portable ML model.

Where did I mention halide?

My question is why do you care about opencv when benchmarking neural network performance on various hardware?

Re: Pixel Visual Core: Google’s first custom-designed co-processor

#90

Post by awalton » Wed, Oct 18, 2017, 7:10 PM UTC

jrs95 wrote: post #67

Maybe Google should start designing it's own CPUs/SoCs as well, since Qualcomm doesn't really seem capable of keeping up with what Apple is doing. Snapdragon 835 came out 6 months after the A10, and scored about 2/3 as well at single core, and less than 10% on Geekbench 4. A11 scored twice what the 835 did on single core, and over 50% more on multi core, and the 835 is going to be the best chip available in the U.S.…

The problem is that basically Qualcomm is strangling the entire cellphone space. So what if you built a completely new, much faster, much better-in-every-way SoC... if you still have to pay Qualcomm for the entire chipset because they refuse to sell you just the modem without you paying the full bill.

Multiple designers could run circles around Qualcomm, but the marginal cost of doing so combined with the number of CPUs you'd have to mint to do so makes it a losing proposition. Apple's in the unique position of being able to leverage Intel to bend Qualcomm over a barrel, and Qualcomm fires back with a gazillion lawsuits and New York Times articles that cry "woe is me, Apple is eating my lunch!"

On a different topic, benchmark navalgazing is probably not going to help anyone. Every company that's been designing CPUs from the mid 80s on has been writing benchmark defeating code, and I'm 100% certain PA SEMI/Apple is no different; they certainly have the silicon space to waste writing Application-specific-accelerators for acing benchmarking tasks, especially on a platform with as much opacity as Apple holds over the iPhone.