Live data from Hacker News

How to Design Better APIs

r.bluethl.net

61–70 of 238 posts

Re: How to Design Better APIs

#61

I’m gonna say it: Many rest apis are lazy and developer friendly, not consumer friendly. If you have related resources, let’s say, product and product options as two distinct endpoints: - /api/product - /api/options Then, and I want to be clear here, it is impossible for a client to perform an atomic operation on multiple distinct objects types. Let’s say the client needs to add a product with a single option or fail…

Everyone has different experiences but my view of the problem is that it's not so much down to developer laziness as it is technological ignorance and reactive direction from higher up - the people who actually fund and approve the work.

Outside of the bigger providers I don't think most APIs are designed with general consumption in mind. The impetus usually seems to be reactive - "we've got this idea for a mobile app using data from our legacy WMS/CRM/whatever, and it needs to be finished yesterday!"

So generally (not always but usually) what I've seen is whatever the underlying system supports and requires gets reflected into the API, and there's minimal proper engineering or sensible abstraction, partly because of time limits and also because the requirements are discovered mid-flight.

"Oh... Products have options in a separate entity? Oops. Uh, we'll add an options endpoint. Easy."

Several years later, the app is mildly successful, the business decides to whitelabel the API for some of their B2B clients, there's no budget for a redesign, "It works doesn't it? If it ain't broke don't fix it..."

Where possible I try to design and build APIs with some forethought and push for as much information on requirements and capabilities of existing systems beforehand but quite often the stakeholders don't really know and don't want to pay for a few weeks of in-depth analysis.

The endless facepalming of hearing the cycle of "Don't overengineer it" when I ask too many questions people can't answer, through to "Why didn't we pick up on this requirement" and having to listen to "we can all learn from this" when something fails because of some if/else statement buried in the legacy business logic, right back to "Don't overengineer it" when you propose the clean way of encapsulating that in the API, has left a permanent indent on my skull. So much leakage of black-box logic into the clients which bloats frontend logic and provides plenty of double handling and scope for unknown business logic to be missed, and no one in charge really gives a shit other than the developers and engineers, and that's fine because we can deal with it and it's their fault for not thinking of it anyway... they're the smart ones right?

You'd think this is just a problem with SMEs but some of the biggest national and multinationals I've worked with have a ton of horrid shit in their API designs. One industry-wide government-mandated reporting spec you'd post a SOAP envelope that wraps an undocumented XML document (they had an example document and the envelope was documented, but no doctype or anything to go off for the contents), a major real estate interchange format has you polling a REST API for an update flag, then loading and saving CSVs over FTP. Some godawful affiliate marketing API essentially just exposed their nth normal form relational database as an API. One government department just used HTSQL for their API using a python script to reflect some of their silos into PGSQL and called it a day (even though whether or not your interactions would work or made sense depended entirely on their internal application logic which wasn't represented or documented).

And too many projects where it turns out the promised APIs didn't even exist until the app was well into development. "Don't worry about it, that's their problem and gives us an excuse when the inevitable delays occur..."

No wonder we're hearing of massive breaches daily, from top to bottom the entire industry is barely held together by glue and sticky tape. It seems like an engineering problem but I think it goes much higher than that, it's a cross-industry failure in planning and management. Engineers are not given the time and budget to do the job that the data deserves. The market self corrects though, massive breaches have closed more than one business down for good.

It feels futile but I do support and encourage people to work towards just reasonable and well-documented endpoints, I doubt there's one true standard, but just some level above slapping it together at the last minute would be nice. I don't care if it's JSON or XML, or whether paths follow some semantics or another. As long as the relationships, types, and mandatory vs optional fields are explained clearly and succinctly, and the auth mechanism isn't a nightmare, and I can work with it.

Sorry for the rant this is just one area that triggers me. I've seen too much bad practice, and fighting against it for decades and still hearing the same lack of concern in the architecture phase has left me more than a little jaded.

Re: How to Design Better APIs

#62
post #8
post #4

Some nice tips in here. However, tip 15, I strongly disagree with: > 15. Allow expanding resources I would suggest the opposite. A REST API should not return nested resources at all. Instead, and to stay with the example provided on the website, to obtain the "orders", the /users/:id/orders endpoint should be called. It might be tempting to return nested resources, because clients would only have to make a single cal…

Dear PAM69, this is great advice if you want to end up with a slow-to-load, low-performing web application that your customers complain about and hate using. But at least it'll adhere to a specific notion of architectural "purity", right? /s Anytime clients need to make 15 async calls before the UI can be displayed, you're headed up the creek. Generally speaking, this is an anti-pattern. There are exceptions, but the…

I think this is an unnecessarily harsh and sarcastic tone to take here.

The comment you're replying to set out specific reasons why they disagree with expanding/bundling sub-resources. It obviously depends on your use case - and in fact they say use GraphQL, which I heartily agree with - but the point is that you don't always know how the API is going to evolve over time, and keeping things unbundled tends to be a "no regrets" path, while bundling resources by default, in my experience, can lead to trouble later.

When the API evolves - as it probably will - using bundled resources ends up running the risk of either an inconsistent API (where some stuff is bundled and some isn't, and the consumer has to work out which), a slow API (because over time the bundled data becomes a performance burden), or a complicated API (where you need to embed multiple, backward-compatible options for bundling and/or pagination in a single operation). In addition, the bundling of resources commits you to I/O and backend resource consumption based only on an assumption that the data is required. None of this makes sense to me.

In practice, if you can keep your API latency reasonably low and take a little bit of care on the client side, there's no reason a user should notice the few milliseconds of additional latency caused by a couple more API calls during the page draw of an app.

It's not about architectural purity, it's about decomposing your application in a way that balances multiple conflicting needs. I agree with pan69, after many years of doing this in multiple contexts, my default has become to not bundle resources when responding to an API request.

Re: How to Design Better APIs

#63
post #55

Earlier quoted context omitted.

Sort of, but is it any better if the GraphQL layer still has to make 15 requests in order to serve a single useful response?

But those 15 requests are then all occurring over a local network (in the data center), not over the Internet. The true power with GraphQL is that it might not even make all 15 calls because it will entirely depend on what you are querying for. E.g. if you query for a User but not the Orders for that User, then the request to retrieve the orders is simply skipped by GraphQL.

Also, of course, those 15 calls are occurring in parallel. I love how GraphQL makes all the complexity of marshalling data go away. Even when a GraphQL server is directly fronting an SQL database, I found the latency to be better than what I'd probably get if I was to code the calls manually.

Re: How to Design Better APIs

#64

Sometimes, I feel that we ought to have a simple protocol, on top of HTTP, to simply do remote procedure calls and throw out all this HTTP verbs crap. Every request is a http POST, with or without any body and the data transfer is in binary. So that objects can be passed back and forth between client and server. Sure, there is gRPC, but it requires another API specification (the proto files). There I said it. HTTP Ve…

So the problem with “Data transfer is in binary” is that it really requires both the source and the recipients to be running the same executable, otherwise you run into some really weird problems. If you just embrace parsing you of course don't have those problems, but that's what you are saying not to do... Another great idea is for a binary blob to begin with the program necessary to interrogate it and get your values out, this has existed on CDs and DVDs and floppies and tape forever but the problem is that those media have a separate chain of trust, the internet does not, so webassembly (plus, say, a distributed hash table) really has a chance to shine here, as the language which allows the web to do this quickly and safely. But it hasn't been mature.

The basic reason you need binary identicality is the problem that a parser gives you an error state, by foregoing a parser you lose the ability to detect errors. And like you think you have the ability to detect those errors because you both depend on a shared library or something, and then you get hit by it anyway because you both depend on different versions of that shared library to interpret the thing. So you implement a version string or something, and that turns out to not play well with rollbacks, so the first time you roll back everything breaks... You finally solve this problem, then someone finds a way to route a Foo object to the Bar service via the Baz service, which (because Baz doesn't parse it) downgrades the version number but does not change the rest of the blob, due to library mismatches... Turns out when they do this they can get RCE in Bar service. There's just a lot of side cases. If you're not a fan of Whack-a-Mole it becomes easier to bundle all your services into one binary plus a flag, “I should operate as a Bar service,” to solve these problems once and for all.

Re: How to Design Better APIs

#65

Earlier quoted context omitted.

True enough, but I would still recommend that API responses normalize to UTC (Z suffix) in the general case and document as much, and if actually returning a timestamp with a specific timezone, document the intended meaning.

+1, if your application cares about time then you should just tell your users all dates will be normalized to UTC and they are responsible for displaying them in a preferred time zone.

Time zone rules change relatively frequently - future dates may be better stored in the appropriate time zone, optionally along with the UTC offset at the time of recording, so you don't report incorrect information when the rules do change on you.

Past dates should always be in UTC, for the same reason - timezone rule changes are sometimes even retroactive.

Re: How to Design Better APIs

#67
a) Use standardized error codes, not standardized error messages. Clients are responsible for internationalization, which includes presenting error messages in the user's language. If you document a set of error codes as an enum, the client can present a user-friendly error message in the user's language based on the error code. If there are dynamic parts of the error message, i.e. "404: There is no user with ID 123456 in the system", then the user ID should be extracted into the error response body, so that it can be provided correctly to the user in the user's language.

b) Pagination is the devil. The state of the server can change while the user is paginating, leading to fragile clients. Don't paginate your API. If you think you have a need to paginate, have one API call return a list of IDs, and have a separate API call return a list of resources for a given list of IDs, where the second API call accepts some maximum number of IDs as a query parameter. This ensures consistency from one API call to the next.

Re: How to Design Better APIs

#68
post #4

Some nice tips in here. However, tip 15, I strongly disagree with: > 15. Allow expanding resources I would suggest the opposite. A REST API should not return nested resources at all. Instead, and to stay with the example provided on the website, to obtain the "orders", the /users/:id/orders endpoint should be called. It might be tempting to return nested resources, because clients would only have to make a single cal…

> you will find that the interface will become increasingly muddled.

Will I? For example The Stripe API uses expand parameters and I prefer that approach to "atomic" REST or being forced to use GraphQL. There is a missed standardization opportunity for incremental graph APIs built on REST.

Re: How to Design Better APIs

#69
post #44

Sometimes, I feel that we ought to have a simple protocol, on top of HTTP, to simply do remote procedure calls and throw out all this HTTP verbs crap. Every request is a http POST, with or without any body and the data transfer is in binary. So that objects can be passed back and forth between client and server. Sure, there is gRPC, but it requires another API specification (the proto files). There I said it. HTTP Ve…

I don't think I've ever come across any third party actually implementing HATEOAS ( https://en.wikipedia.org/wiki/HATEOAS )

HATEOAS have always sounded like a delicious part of a healthy breakfast

Re: How to Design Better APIs

#70
Can someone share how they handle versioning in their API when it comes to data model changes? For example `POST /users` now takes a required field `avatar_url` but it was not part of `v1`.

Since this field is validated in the DB, merely having `v1` `v2` distinction at the API layer is not sufficient. So I was thinking we will have to either 1) disable DB validations and rely on app validations or 2) run two separate systems (e.g., one DB per version) and let people 'upgrade' to the new version (once you upgrade you cannot go back).

Even though people refer to Stripe's API versioning blog, I don't recall any mention of actual data model changes and how it is actually managed

Post reply on HN