Apparently the API of my kitchen is very bad.
If an AI agent can't figure out how your API works, neither can your users
31–40 of 55 posts
Re: If an AI agent can't figure out how your API works, neither can your users
#32In one of my first uses of a LLM for helping me code, was to use a not popular Python library API. It completely hallucinated the API calls, but what was interesting is that the invented API was much more pythonic than the real one. It created a better API. Now I'm waiting for the change to use LLMs for creating an API for a package of mine. It will be averaged from all other apis and won't have unexpected calls.
well, yes, because the imaginary world has no real world constraints
Re: If an AI agent can't figure out how your API works, neither can your users
#33AI agents can't even do very basic things without causing absolute mayhem. Do you really think it makes sense for your basic assumption to be that your user is dumber than a token generator?
> AI agents can't even do very basic things without causing absolute mayhem.
Agreed.
> Do you really think it makes sense for your basic assumption to be that your user is dumber than a token generator?
Absolutely. Users will fuck up basic things. I wrote a simple inotify wrapper and half of the issues I got were that it didn't work on Windows and MacOS.
Re: If an AI agent can't figure out how your API works, neither can your users
#34We used to have things called WSDLs and XSD schemas that made it _extraordinarily_ easy to make remote calls. Granted, a bunch of ding dongs never loaded their own WSDL to look at it, creating a bad rap. We do have: * WADLs: https://en.wikipedia.org/wiki/Web_Application_Description_La... * JSON Schema: https://json-schema.org/learn/miscellaneous-examples And when they are available they're incredible, but nobody uses…
Swagger/OpenAPI has been reasonably popular for a while: https://swagger.io/docs/specification/v3_0/about/
Re: If an AI agent can't figure out how your API works, neither can your users
#35We used to have things called WSDLs and XSD schemas that made it _extraordinarily_ easy to make remote calls. Granted, a bunch of ding dongs never loaded their own WSDL to look at it, creating a bad rap. We do have: * WADLs: https://en.wikipedia.org/wiki/Web_Application_Description_La... * JSON Schema: https://json-schema.org/learn/miscellaneous-examples And when they are available they're incredible, but nobody uses…
Json schema is great, we used it at my last job and I've built a fuzzer generating data based on it. So the data passes basic validation, but possibly crashes the backend afterwards. Found lots of problems the first run.
Re: If an AI agent can't figure out how your API works, neither can your users
#36A good API is upfront with its caveats, e.g. saying that you should do an exponential backoff X times with Y initial delay. But the only way to verify that claim is through usage. I wouldn't expect a junior or an AI agent to be able to assess that in the initial client implementation, and I really wouldn't want a junior or AI agent to hit the prod instance of this service in myriad different ways to try and unearth these quirks.
Re: If an AI agent can't figure out how your API works, neither can your users
#37We used to have things called WSDLs and XSD schemas that made it _extraordinarily_ easy to make remote calls. Granted, a bunch of ding dongs never loaded their own WSDL to look at it, creating a bad rap. We do have: * WADLs: https://en.wikipedia.org/wiki/Web_Application_Description_La... * JSON Schema: https://json-schema.org/learn/miscellaneous-examples And when they are available they're incredible, but nobody uses…
If you ever tried to read an actual WSDL file, you'd have noticed it was not great.
Yes, having an API type declaration is really important. And yes, somehow a lot of people just don't use those things. But the WSDL was one of the worst standards for that in all times. (And also, it inherited all of the shitness from XML, even the allowing non-deterministic processing and side effects while reading the file.)
Anyway, I'm not really disagreeing on your main point. Read the JSON Schema docs, people, use it.
Re: If an AI agent can't figure out how your API works, neither can your users
#38In one of my first uses of a LLM for helping me code, was to use a not popular Python library API. It completely hallucinated the API calls, but what was interesting is that the invented API was much more pythonic than the real one. It created a better API. Now I'm waiting for the change to use LLMs for creating an API for a package of mine. It will be averaged from all other apis and won't have unexpected calls.
> It created a better API.
Did it though? It didn't create an API, it created the appearance of an API. The reality of the API, something the library author had to wrestle with and the LLM didn't, is probably much more complex and nuanced than the LLM is hallucinating.
Maybe the API it's hinting at would be better if made real. But it pains me you're telling us the LLM, a tool known for making things up and being wrong, despite not even having done it, can do it better than a person who actually spent their time to make and put a real thing out into the world. Maybe one day, but just making up BS is not creating a better API.
Re: If an AI agent can't figure out how your API works, neither can your users
#39In one of my first uses of a LLM for helping me code, was to use a not popular Python library API. It completely hallucinated the API calls, but what was interesting is that the invented API was much more pythonic than the real one. It created a better API. Now I'm waiting for the change to use LLMs for creating an API for a package of mine. It will be averaged from all other apis and won't have unexpected calls.
well, yes, because the imaginary world has no real world constraints
But it's not necessarily constraints that drive most library API design. I'd say most of it is arbitrary. Kinda of like which classes you decided to make in an OOP system. We tend to pick something that works and roll with it. Maybe only refactoring it if it's too hard to test.
Re: If an AI agent can't figure out how your API works, neither can your users
#40I did an experiment a few weeks back as a bit of a litmus test for a new data format my team and I have been working on. I fed the spec documentation into ChatGPT and asked it to write a file reader for it. It spat out a mostly-correct implementation almost immediately, and with some extra coaxing managed to fix the small bugs in it too.
To be fair, it's a very simple format, but it made me feel good about the quality of the documentation.