141–150 of 295 posts
Earlier quoted context omitted.
If agents is what it finally takes to get good a11y I'll take it. I'll bitch about it, but I'll take it.
Very real risk of this going in reverse: people building inaccessible websites to prevent AI use.
Earlier quoted context omitted.
Those people probably aren't working on anything useful anyways, so its no big deal.
I've found that by far the most useful websites as a programmer are also the ones most resistant to AI. This would be a huge loss for anyone vision impaired
I'm building something that fixes this exact problem[1]. The landing page doesn't advertise it yet, but essentially, I give agents a small set of tools to explore apps' surfaces, and then an API over common macOS functions, especially those related to accessibility. The agent explores the app, then writes a repeatable workflow for it. Then it can run that workflow through CLI: `invoke chrome pinTab` Why accessibility…
I think this is very fertile ground - big labs need to use approaches that can work on multiple platforms and arbitrary workflows, and full-page vision is the lowest common denominator. Platform-specific approaches are a really exciting open space!
Earlier quoted context omitted.
If agents is what it finally takes to get good a11y I'll take it. I'll bitch about it, but I'll take it.
Playwright, the end-to-end testing framework for the web, provides a strong incentive to give sites good a11y: Playwright tests are an absolute delight to read, write and maintain on properly accessible sites, when using the accessibility locators. Somewhat less so when using a soup of CSS selector and getByText()-style locators. One thing I am curious about is a hybrid approach where LLMs work in conjunction with vi…
https://playwright.dev/docs/getting-started-mcp#accessibilit...
I've mentioned several times and gotten snarky remarks about how rewriting your code so it fits in your head, and in the LLM's context helps the LLM code better, to which people complain about rewriting code just for an LLM, not realizing that the suggestion is to follow better coding principles to let the LLM code better, which has the net benefit of letting humans code better! Well looks like, if you support accessibility in your web apps correctly, Playwright MCP will work correctly for you.
Amazing.
IMO, this is the argument for doing work in the first place.
I'm always skeptical of the whole "computer use" concept. It's like hiring someone and inviting him to your house and telling him to go ahead, feel free to sleep on the bed, use the toilet, eat whatever is in the fridge, watch the TV, and oh here are the combinations for the safe... and that someone you hire is a monkey.
Tokens a resource and should be managed as such.
In an agentic world, the OS needs to be completely rethought. For example, every single app functionality should be exposable via an API while remaining human friendly. I think OpenAI designing their own phone is the next logical step. I hope they succeed which should bring major competition to Apple and Android.
"In an agentic world, the OS needs to be completely rethought" - if AI is progressing as fast as we think it is, I don't think we'll be interested in waiting for the world to rebuild all the legacy tooling from the OS up. For new stuff, that'd be great. I imagine the AIs will get a lot better at intercepting things at an intermediate level - API calls under the hood, etc. Probably much better (and cheaper) vision abi…
Open source research/project I have been exploring on the topic: https://aevum.build/learn/architecture/
Great guidance hidden in here for making it expensive for agents to navigate your website. Move elements on screen as the mouse moves, force natural mouse movement to make the UI work, change the button labels in the JS to be randomly named every visit, force scrolling to the bottom of the screen to check for hidden extra tasks... Hang on, that sounds like common corporate SaaS apps.
Ah damn it, we invented Jira
Almost sounds like an Orielly book