241–250 of 295 posts
In an agentic world, the OS needs to be completely rethought. For example, every single app functionality should be exposable via an API while remaining human friendly. I think OpenAI designing their own phone is the next logical step. I hope they succeed which should bring major competition to Apple and Android.
We gave up on structured APIs 20+ years ago when JSON RPC largely replaced XML REST. You can do REST in many different formats, it mainly just needs to be structured data and self-discoverable. Had we not made that wrong turn, LLMs and humans would have a much easier time reasoning about APIs they don't directly control.
REST has nothing to do with structured data or discoverability.
Earlier quoted context omitted.
This will not happen. None of the existing apps people use daily on their phones have any incentive to support this. Social media wants the people to doomscroll, shopping apps and booking sites want to use their own dark patterns to make people believe they get a special discount if they buy _now_ and everything else just wants users to see the ads. Why on earth would they offer convenient hooks for AI chatbots?
Actually this more or less describes how accessibility APIs work.
A closer analogue would be AppleScript, or rather, the underlying Apple Event and Open Scripting Architecture functionality supplied by the OS to support AppleScript, that allowed applications to expose these interfaces along with metadata documenting them, and for external tools to record manually performed tasks across applications as programs expressed in terms of these interfaces to make them easier to use (this last bit, while not strictly required, is convenient, and especially useful for less technical users).
If you're familiar with VBA in Microsoft Office applications, sort of like that, except with support provided by OS APIs that could be used by any application that chose to implement scripting support, official guidance from Apple suggesting that all well-designed applications should be scriptable and recordable, and application design patterns and frameworks designed with scriptability and recordability in mind.
Note that I use the past tense here, despite AppleScript still being available in macOS, because it is not well-supported by modern applications.
Earlier quoted context omitted.
How firm is the boundary between a dev doc and a user doc in your opinion? I have found that the overlap can be quite large if the users are also technically proficient. Right now I'm trying to balance "how X works so you can use the app better" with "how X works so you can contribute or build your own plugin". DeepWiki really helps as a backstop for anything not already covered though it's not without its own caveat…
Not OP but I think you have the right intuition in making a difference between using the app / contribute to the app. You may want to read https://diataxis.fr/ which elaborate on this idea and add another dimension (action / cognition) to this.
In fact, the only area I've been struggling with are "Concepts" because they have less clear boundaries for the right amount of detail.
Here is what I've been working on: https://github.com/super-productivity/super-productivity/wik...
Earlier quoted context omitted.
Why on earth would they offer convenient hooks for AI chatbots? Competition. If I ask my OS-level AI assistant to find a social media reel about a elephant dancing, the social media app that exposes a set of APIs for an AI agent might get used more. Watch how fast Meta adds this if a new hot shot social media app succeeds by designing for AI agents controlled by users.
>Competition. If I ask my OS-level AI assistant to find a social media reel about a elephant dancing, the social media app that exposes a set of APIs for an AI agent might get used more. This is the exact opposite of what will happen (and in fact what has happened). Reddit is suing Perplexity right now for scraping. Meta will not serve content to some other app for free - for what benefit? They will not see advertisi…
Advertising isn't the only possible business model.
And profit isn't the only possible motive to provide a service.
Is it possible to ask the vision agent to "map" the UI and expose it to another agent as a set of interfaces that resemble an API better? From what I understand the vision agent now should both know that "next page" shows more results and that they need to get more results in the first place. If one agent just explores the UI, maybe in a test environment, and outputs a somewhat-structured description of the various U…
>Is it possible to ask the vision agent to "map" No most vision models focus on subset of an image at a time when using image -> text image -> image uses whole image.
Is this true? Where can I read more about it?
Well, if your backend was sufficiently decoupled from your frontend, and the server-side operations were designed thoughtfully and generically, it need not be an engineering project.
Earlier quoted context omitted.
> It's really weird, I'm seeing across the board that people who never believed in them before are suddenly all into good software eng practices (starting with writing a spec) because of AI. > It's kind of fascinating that we never were willing to do these things for humans but now that AI needs it ... we are all in. A bit depressing in the sense that I think mostly the reason we happy to do it for AI is that we perc…
Yeah I think a lot of pushback to best practices is basic cost/benefit; I like writing documentation, but I'm also often feeling a bit depressed that nobody will actually read it in as much detail as I wrote it. But LLMs do / can. Actually there's a lot of projection there too; I don't read documentation in detail. And nowadays, I point an LLM at documentation so that it can find the details I would otherwise skip ov…
Imagine how crippled you would be if you felt compelled to follow every comment thread to its end.
We're just monkeys looking for the good bits among a pile of rotten fruit.
Earlier quoted context omitted.
It's really weird, I'm seeing across the board that people who never believed in them before are suddenly all into good software eng practices (starting with writing a spec) because of AI. It's kind of fascinating that we never were willing to do these things for humans but now that AI needs it ... we are all in. A bit depressing in the sense that I think mostly the reason we happy to do it for AI is that we perceive…
My friend at a faang was talking about the "massive overhauls to make everything ready for ai". I asked for an example. He said "basically just documenting the shit out of everything" I guess that just never occurred to anybody before.