> but no sane client will go around and request random server-provided urls to discover what is available
Compare and contrast: what SQL admin GUI clients do to discover the DBMS schema. They essentially spider it.
> Under HATEOAS the client would need to associate the knowledge of resource structure with a particular resource received. A promising identifier for this association would be the resource collection path, i.e. the URL. If the client needs to know the URLs, why have them in the response?
The client does not need to know the URL; the client needs to know how to get to the URL.
Have you ever written a web scraper, for a site that doesn't really enjoy being scraped, and so uses stuff like CSRF protection "synchronizer tokens"? To do/get anything on such a site, you can't just directly go to the thing; you have to start where a real browser would start, and click links / submit forms like a browser would do, to get there.
HATEOAS is the idea that APIs should also work that way: "clicking links" and "submitting forms" from some well-known root resource to get to the API you know about.
As with a scraper, it's the "link text" or the "form field names" — the delivered API — that you're assuming to be stable; not the URL structure.
> Other problems include creating new resource - how the client is supposed to know the structure of to-be created resource, if there is none yet? The client has nothing to request to discover the resource structure and associations.
What do you think HTML forms are, if not descriptions of resource templates? When you GET /foos, why do you think the browser result is a form? It's a description of how to make a foo (and, through the submit attribute, a place to send it to get it made.)
Alternately, compare/contrast what Kubernetes does — exposes its resource schemas as documents, living in the same resource-hierarchy as the documents themselves live.
> Also hypertext does not map well to JSON. In JSON you can not differenciate between data and metadata (i.e. links to other resources)
It's right in the name: in HATEOAS, hypertext is the engine of application state. Hypertext as in, say, HTML. JSON is not hypertext, because it's not text — it's not a markup language like SGML/XML/HTML are.
You realize that HTML is machine-readable, right? That you can respond to an XHR with HTML, and then parse it using the browser (or API client's) HTML parsing capabilities, just like you can receive and parse JSON? That there's nothing stopping you from using a to send tabular data, etc.? And that if you do so, then debugging your API becomes as simple as just visiting the same link the API client is visiting, and seeing the data rendered as HTML in your browser?