Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple rejected my dictation app for using the accessibility API

mitmllc.com

Post reply on HN
Print view View original

151–160 of 200 posts

Re: Apple rejected my dictation app for using the accessibility API

#151

Post by alnwlsn »

john_strinlai wrote: post #78
stonogo wrote: post #69

Earlier quoted context omitted.

Nobody in my life even notices when they change their 'primary operating system.' They buy a phone based on what looks cool at the time, sometimes it's android, sometimes it's iphone. They move freely between chromebooks, windows, and mac os, because everything is online anyway. It's only 'experts' who have trouble with this.

you are telling me that everyone in your life freely swaps between windows and mac without even noticing a difference? no problems? i call bullshit. i have worked in very big orgs. changing a single icon can cause a deluge of support tickets.

I have family that would gladly use FacebookOS if such a thing existed and automatically loaded that and only that website as soon as you turn on the computer or phone.

Re: Apple rejected my dictation app for using the accessibility API

#153

Post by int0x29 »

marvin-hansen wrote: post #118

I actually had the almost same situation by building an offline voice dictation app for macOS and iOS, and in macOS I was confronted with the exact same situation. However, I would like to point out that Apple isn't totally wrong here because the accessibility API unfortunately is way too broadly scoped, and because of that you literally get access to everything on the computer like you you can screenshot listen and…

> However, I would like to point out that Apple isn't totally wrong here because the accessibility API unfortunately is way too broadly scoped, and because of that you literally get access to everything on the computer like you you can screenshot listen and and move the cursor... This is completely ridiculous and the proper engineering solution would actually be to phase out the accessibility API and replace it with something that is narrowly scoped so you can grant specific permissions individually

If you don't have use of your hands you want that. The whole point of accessibility APIs is allowing arbitrary control of your computer via novel means. One of the big selling points of Dragon Natually Speaking is the ability to tell your computer to do things based on descriptions without a mouse. "open outlook", "click compose", "select subject", "type foo", etc.

And no the solution here is not computer vision with an LLM. Text and buttons rendered on my computer exist in memory somewhere as text and buttons. We should not need to convert them to pixels and back lossily to recover text and buttons. We should just expose things to the accessibility API and not guess.

Re: Apple rejected my dictation app for using the accessibility API

#154

Post by Barbing »

Wowfunhappy wrote: post #144
marvin-hansen wrote: post #118

I actually had the almost same situation by building an offline voice dictation app for macOS and iOS, and in macOS I was confronted with the exact same situation. However, I would like to point out that Apple isn't totally wrong here because the accessibility API unfortunately is way too broadly scoped, and because of that you literally get access to everything on the computer like you you can screenshot listen and…

> However, I would like to point out that Apple isn't totally wrong here because the accessibility API unfortunately is way too broadly scoped, and because of that you literally get access to everything on the computer like you you can screenshot listen and and move the cursor... I want apps to be able to do that!

Yes but miffing to open Privacy & Security & see dozens of apps pretending to need “accessibility” features. Apple has a dozen+ categories there but many poweruser apps I want specifically need accessibility.

Is there an opinionated reason not to break out capabilities?

Re: Apple rejected my dictation app for using the accessibility API

#155

Post by int0x29 »

phainopepla2 wrote: post #148
Wowfunhappy wrote: post #144

Earlier quoted context omitted.

> However, I would like to point out that Apple isn't totally wrong here because the accessibility API unfortunately is way too broadly scoped, and because of that you literally get access to everything on the computer like you you can screenshot listen and and move the cursor... I want apps to be able to do that!

Then they should use an appropriately scoped API, as OP suggested.

Controlling my computer is appropriate scope for an accessibility tool

Re: Apple rejected my dictation app for using the accessibility API

#156

Post by int0x29 »

Barbing wrote: post #154
Wowfunhappy wrote: post #144

Earlier quoted context omitted.

> However, I would like to point out that Apple isn't totally wrong here because the accessibility API unfortunately is way too broadly scoped, and because of that you literally get access to everything on the computer like you you can screenshot listen and and move the cursor... I want apps to be able to do that!

Yes but miffing to open Privacy & Security & see dozens of apps pretending to need “accessibility” features. Apple has a dozen+ categories there but many poweruser apps I want specifically need accessibility. Is there an opinionated reason not to break out capabilities?

> Is there an opinionated reason not to break out capabilities?

If you have a disability and need tools to use your computer the last thing you want to do is have those things not only off by default but complicated and involved to turn on.

Re: Apple rejected my dictation app for using the accessibility API

#157

Post by Wowfunhappy »

Barbing wrote: post #154
Wowfunhappy wrote: post #144

Earlier quoted context omitted.

> However, I would like to point out that Apple isn't totally wrong here because the accessibility API unfortunately is way too broadly scoped, and because of that you literally get access to everything on the computer like you you can screenshot listen and and move the cursor... I want apps to be able to do that!

Yes but miffing to open Privacy & Security & see dozens of apps pretending to need “accessibility” features. Apple has a dozen+ categories there but many poweruser apps I want specifically need accessibility. Is there an opinionated reason not to break out capabilities?

Apple doesn’t provide another API for this, so apps have to use the one that’s available.

Re: Apple rejected my dictation app for using the accessibility API

#159

Post by Wowfunhappy »

int0x29 wrote: post #153
marvin-hansen wrote: post #118

I actually had the almost same situation by building an offline voice dictation app for macOS and iOS, and in macOS I was confronted with the exact same situation. However, I would like to point out that Apple isn't totally wrong here because the accessibility API unfortunately is way too broadly scoped, and because of that you literally get access to everything on the computer like you you can screenshot listen and…

> However, I would like to point out that Apple isn't totally wrong here because the accessibility API unfortunately is way too broadly scoped, and because of that you literally get access to everything on the computer like you you can screenshot listen and and move the cursor... This is completely ridiculous and the proper engineering solution would actually be to phase out the accessibility API and replace it with…

> And no the solution here is not computer vision with an LLM.

Also, even if you hypothetically wanted to use computer vision with an LLM… what API is that LLM going to use to take screenshots and click on stuff?

Post reply on HN