Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple Support: How to hear image descriptions in the Camera app

youtube.com

Post reply on HN
Print view View original

41–50 of 52 posts

Re: Apple Support: How to hear image descriptions in the Camera app

#41

Post by sarsway »

cbovis wrote: post #25

Turning on voice over to investigate this has to have been one of my worst decisions in life. Please do not repeat.

Haha I remember having to test apps, and being completely confused by it in the beginning too. First thing I did was add VoiceOver toggle to Control Center.

Once you get used to it, it's actually kinda amazing. Apple has put so much effort into iOS to make this work well, and there are a ton of features packed into VoiceOver mode.

Most have no idea this even exists, if you're bored you should play around with it a bit more.

Re: Apple Support: How to hear image descriptions in the Camera app

#43

Post by KMnO4 »

judge2020 wrote: post #39
mimischi wrote: post #5

The YouTube channel by Kristy Viers has been helpful in understanding how they use an iPhone and other devices. Here’s a video on VoiceOver in the Camera app: https://youtu.be/8CAafjodkyE

That’s amazing, I wonder if it constantly doing ML classification is more draining on the battery.

Probably balanced by the fact that you can turn the screen off completely when navigating with VO.

https://support.apple.com/en-ca/HT201443

Re: Apple Support: How to hear image descriptions in the Camera app

#44

Post by Waterluvian »

user-the-name wrote: post #36
Waterluvian wrote: post #13

I tried to get Voice Over working but scroll broke once enabled. It says three fingers to scroll but that just won't work for me.

VoiceOver is a completely different UI. You need to learn how it works before attempting to use it.

Turns out iOS was broken and a reboot made the three finger scrolling work.

Re: Apple Support: How to hear image descriptions in the Camera app

#45

Post by etaioinshrdlu »

It seems to basically be using an image captioning model like this one: https://deepai.org/machine-learning-model/neuraltalk , but possibly trained on more data? I wonder what dataset they’ve used, or if they created it themselves and won’t share it.

Re: Apple Support: How to hear image descriptions in the Camera app

#47

Post by cknoxrun »

cochne wrote: post #26

I just tried this as it was really bad. Kept saying everything was a picture of the night sky, only one of many many descriptions was correct.

Interesting, I pointed it at my son and it immediately accurately described "a child sitting at a white table doing a jigsaw puzzle". I pointed at one of his Lego creations, and immediately "a multi-colour lego construction sitting on a wooden shelf". I was blown away.

Re: Apple Support: How to hear image descriptions in the Camera app

#48

Post by propogandist »

spullara wrote: post #31
Lammy wrote: post #29

Neat. Is my phone classifying all this stuff, or is it sending all my photos to some Apple server to do it? I watched the video in the OP but it didn't go into any technical details.

I don't know of any ML stuff that Apple does serverside. That is why they have invested so much in ML processing on their devices.

yet they can't manage offline Maps and directions without internet connectivity.

Re: Apple Support: How to hear image descriptions in the Camera app

#49

Post by alblue »

yazaddaruvala wrote: post #40
spullara wrote: post #31

Earlier quoted context omitted.

I don't know of any ML stuff that Apple does serverside. That is why they have invested so much in ML processing on their devices.

All ML on device definitely seems like a goal for Apple[1]. However, right now Siri doesn't work in airplane mode. Maybe its still too power hungry to run locally? Maybe they still need the data from the requests server side to improve the model? I hope both Siri and Maps, and especially their integration, i.e. "Siri navigate to ____." "Siri find restaurants nearby (or along the way)" works on device in the near futu…

Siri’s voice translation works in your phone but the answers it needs are all calculated from the server (on the whole). Maps, for example, are all stored server side and route calculation are server side processes as well, as are the locations of restaurants.

There’s a difference been decoding voice requests and being able to answer any questions you may have; you wouldn’t expect Google web search or Wikipedia to work in offline mode, so why do you expect it to work for Siri?

Re: Apple Support: How to hear image descriptions in the Camera app

#50

Post by spullara »

user-the-name wrote: post #35
spullara wrote: post #31

Earlier quoted context omitted.

I don't know of any ML stuff that Apple does serverside. That is why they have invested so much in ML processing on their devices.

I guess Siri does work at least partially serverside?

The ML part is local for the voice transcription but the questions almost always need to be answered from the server. But if you ask to play music or set a timer for example that is all local. Also works in Airplane mode.
Post reply on HN