Live data from Hacker News

Apple Support: How to hear image descriptions in the Camera app

youtube.com

31–40 of 52 posts

Re: Apple Support: How to hear image descriptions in the Camera app

#31
post #29

Neat. Is my phone classifying all this stuff, or is it sending all my photos to some Apple server to do it? I watched the video in the OP but it didn't go into any technical details.

I don't know of any ML stuff that Apple does serverside. That is why they have invested so much in ML processing on their devices.

Re: Apple Support: How to hear image descriptions in the Camera app

#32
post #29

Neat. Is my phone classifying all this stuff, or is it sending all my photos to some Apple server to do it? I watched the video in the OP but it didn't go into any technical details.

It runs on the phone, no doubt using whatever model they use to enable the keyword searching in the camera-roll. Just verified it all works in airplane mode.

Re: Apple Support: How to hear image descriptions in the Camera app

#34

The descriptions are shockingly detailed. I wish there was a way you could add these descriptions to the metadata of all the photos in your library, it would be an amazing way to add rich searchable data. In particular, asking Siri to "show me photos of " where x is a thing that the photo descriptions does recognize, doesn't always work.

[deleted]

Re: Apple Support: How to hear image descriptions in the Camera app

#35
post #29

Neat. Is my phone classifying all this stuff, or is it sending all my photos to some Apple server to do it? I watched the video in the OP but it didn't go into any technical details.

I don't know of any ML stuff that Apple does serverside. That is why they have invested so much in ML processing on their devices.

I guess Siri does work at least partially serverside?

Re: Apple Support: How to hear image descriptions in the Camera app

#36

I tried to get Voice Over working but scroll broke once enabled. It says three fingers to scroll but that just won't work for me.

VoiceOver is a completely different UI. You need to learn how it works before attempting to use it.

Re: Apple Support: How to hear image descriptions in the Camera app

#37

Sound recognition is also another very cool accessibility feature: https://www.loopinsight.com/2021/05/11/apple-support-how-to-... Though I'm not sure if it comes at the cost of increased battery usage.

In my anecdotal experience it makes no difference on my 11 Pro. It presumably uses the same hardware that’s already awake all the time listening for “Hey Siri”.

Re: Apple Support: How to hear image descriptions in the Camera app

#39
post #5

The YouTube channel by Kristy Viers has been helpful in understanding how they use an iPhone and other devices. Here’s a video on VoiceOver in the Camera app: https://youtu.be/8CAafjodkyE

That’s amazing, I wonder if it constantly doing ML classification is more draining on the battery.

Re: Apple Support: How to hear image descriptions in the Camera app

#40
post #29

Neat. Is my phone classifying all this stuff, or is it sending all my photos to some Apple server to do it? I watched the video in the OP but it didn't go into any technical details.

I don't know of any ML stuff that Apple does serverside. That is why they have invested so much in ML processing on their devices.

All ML on device definitely seems like a goal for Apple[1]. However, right now Siri doesn't work in airplane mode.

Maybe its still too power hungry to run locally? Maybe they still need the data from the requests server side to improve the model?

I hope both Siri and Maps, and especially their integration, i.e. "Siri navigate to ____." "Siri find restaurants nearby (or along the way)" works on device in the near future. Its annoying for the CarPlay integration to stop working when you're in a remote area and need the navigational help.

[1] https://appleinsider.com/articles/18/11/15/apple-considering...

Post reply on HN