Building Protocols with HTTP

By the way, server (and client, if needed) auth is implemented by the TLS layer. Do you propose using something else to secure other protocols?

[1]: http://openid.net/specs/openid-connect-discovery-1_0.html

This is also good to combine with requiring users retry and use multiple endpoints. This way you don't have to take risks with DNS and global routing failures potentially breaking your application.

If you use TLS to protect your application, I recommend further (randomly) generating a bad SSL certificate to make sure clients retry (and get a good one) so that you can detect anyone stupid enough to take advice like this[2]

[2]: http://web.archive.org/web/20120317165131/http://forum.devel...

> How is that different than having only 1 version of the API and updating it?

Using a service discovery document makes it easier to put /v1 and /v2 on separate servers/domains instead of putting both on all the servers/domains. Serving the service discovery document is cheaper than having your front-end server route all requests to backend systems (like using nginx or an Amazon ELB in application mode).

Putting applications on different domains is desirable if /v2 has performance gains (or cost gains) obtained by using the new protocol. Or if they're separate codebases.

> Clients code will still break if a structural data change is pushed (for example, removing a field from the responses)

It's very difficult to predict how a client will react if you haven't seen/vetted their implementation, but there are things we can do that go beyond service discovery.

For example:

My advertising service allows publishers (people who make web pages and content) to choose how to render my advertisers' ads in those pages. They do this via a server-side call -- I don't do any tracking of this, but I want to be sure that the client is doing the right thing because I'm still representing the brands.

I can look at the web page and sign off on it, but I want to make sure that it'll support a wide range of ads -- including ones I haven't thought of.

So I put a lot of energy into patterns like fuzzing the clients, and doing things that I hope will require they do things securely:

- I require retries so users always get an ad

- I randomly giving them a bad SSL certificate -- one that if they accept, I block their account. I usually only do this during onboarding or for my crawlers (recognised by IP) which check for sneaky stuff.

- I require certain utf8 characters to make sure the query is utf8 encoded

- I randomly insert random fields to make sure I can add new ones later

- I randomly reorder fields in the response to make sure they're not counting

I think that if you do all these things, and perhaps other things, and clients are used to it then you probably can have just one version of the API and update it. But this is hard work, and the enterprise is not (in my experience) very fond of hard work.

It is just a protocol for stateless server-client communication. Nothing more. What you build on top of it, is your choice. And you don't have to use it. But it is a mature protocol and it is well understood and it has a huge ecosystem of technology around it, that is just good enough for many use cases.

> 1. It's a verbose protocol with more often than not a really bad ratio payload over metadata;

Its a stateless protocol. You have to provide all the required meta data (Headers) in a conversation. Otherwise, it would not be able to be stateless. And yes, it does use text, because it would be extremely difficult to create a evolving protocol just by using some more memory-efficient binary tricks. Text is just better for comprehensible semantics. And that's even a good thing in pure machine-to-machine interaction.

There is also HTTP/2 which addresses efficiency and latency issues. But it trades that for simplicity of the transport method.

> 2. Client authentication is hard;

It depends. It can be as easy as sending an Authorization header. Or you can defer authentication to the transport layer (client certificates over TLS). Otherwise, I'd say: Best security is hard. And it won't get easier in other stateless protocols.

> 3. Server authentication is expensive;

That's just too generic. What does "expensive" mean? For what use case is it too expensive?

> 4. It's bad at streaming;

It depends what you mean with streaming. I think, you mean legacy binary streams? Yes, HTTP is not meant for that. But you can use HTTP for initiating your custom streaming use case. There is the "Upgrade" header. Or you just provide an URL to your streaming endpoint that uses something different than HTTP or even TCP, as long as the client can handle this stuff (e.g. RTMP).

Then there is something that is often misconceived as a utility for streaming: using chunked streams or using the Range header. Chunked streaming is just for fetching content of arbitrary size. And having a server support "Range" allows for fail-over / retries. Its not meant to support continuous streams of data.

> 5. It does not help a bit for RPC (no support for fan-out, quotas, fail-overs, retries, ...).

You can do RPC if you design your payload around it. See SOAP or even something like RMI over HTTP. Fan-out, quotas, fail-overs and retries. Everything can be done using HTTP. Does the specification itself solve those issues? No and it should not, because it would make it too complex and harm adaptation of HTTP.

The only times I know that non-HTTP works is video games and that is sorta hit-or-miss on one ISP I frequent, they block protocols they don't know for "security".

Making it HTTPS is an easy way to get around any potential firewall problems and have it work easily in corporate settings.