Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

API pagination design

solovyov.net

Post reply on HN
Print view View original

141–150 of 150 posts

Re: API pagination design

#141

Post by sonthonax »

This is one of those obvious things that people have a blockheaded aversion towards. In every job I’ve consulted on, there has been a component where pagination is the bottleneck. And it’s rarely where pagination could be useful (like a static list of products). I typically see it when someone builds an aggregating (and aggravating) microservice that pulls all of the data from another service; that blocks the entire database.

Perhaps it’s the world of ‘Django developers’ I find myself in. But the fact this simple design pattern reaches the front page of Hackernews worries me.

I’ve personally had backend developers give confused looks and diatribes about simplicity when I’ve suggested this approach. Frontend developers frequently give the greatest resistance, which I think is because they most often act in concert with product managers who are welded to certain UX idioms that they chose without fully imagining the engineering consequences of.

One approach which I try to push people towards is fetching a big list (N≈5000) and paginating it however the UX designer wants. The response of the big list should be sparse, that is without including additional fields that aren’t displayed by the front end (and fetching more data when the user navigates). You’ll frequently get puzzled looks suggesting this, but benchmarks will usually show that fetching 5000 rows and a small number of columns takes a few milliseconds, and can be serialised into 150kb. On balance it ends up being faster: as you’ll get fewer network requests; fewer round trips to the database; fewer fetches; and less time spend serialising (which is a major bottleneck itself that Python developers ignore).

Re: API pagination design

#142

Post by scrollaway »

callesgg wrote: post #134
hellcow wrote: post #124

Earlier quoted context omitted.

"Woah, our database is slowing to a crawl, requests are deadlocked everywhere. Yeah, some asshole is hitting GET /messages and pulling 2TB in a single request again." "Surely that's what he wanted."

Given that is what the person is doing, that is likly what they are trying to do. Most of the times when I do things, I do it cause I want to do it, not cause I did an opsie.

"let's allow users to DOS us in a single request if they want to" is not an acceptable design decision, I'm sure you are aware of that.

Re: API pagination design

#143

Post by londons_explore »

twic wrote: post #107
egeozcan wrote: post #89

Earlier quoted context omitted.

If it is updated, though (in this case, renamed to Istanbul), it can appear later once again, and newly inserted rows behind "Constantinople" will never appear. I mean, these problems happen if you use offsets too, and my point is that they don't matter. The most hardcore solution to those would be pre-computing pages but I doubt it makes sense to bring even more state to the search results.

Yes, pagination over a consistent view is a whole other problem! Either you store search results server side, in which case now you need to think about expiration and running out of space, or you are able to regenerate them on the fly somehow. A database which supports as-of queries might make that quite easy, but not many databases do.

> but not many databases do.

Really? The underlying storage engine of most databases stores all data necessary for an "as of" query. Simply ignore all data pages newer than the timestamp cutoff, and prevent the garbage collection of any page that could be part of such a query.

In fact thats the way [eg. postgres] can do a long-running query on a table that someone else is modifying. You in effect are looking at a snapshot of the table at the moment you started the query, even if it takes an hour to produce all the results.

Re: API pagination design

#144

Post by atom_arranger »

jiji33i wrote: post #138
atom_arranger wrote: post #27

I'm a fan of the "GraphQL Cursor Connections Specification", which could be applied even if not using GraphQL: https://relay.dev/graphql/connections.htm

"If all you have is a hammer, everything looks like a nail." Are GraphQL-fans so uncapable you have to put your GraphQL-spamming in every single post? Didn't you learn anything else in your life?

No, and yes.

Re: API pagination design

#145

Post by User23 »

mmkos wrote: post #59
User23 wrote: post #32

I'm pretty sure pagination APIs are the result of people who haven't ever read Stevens writing services.

Can you elaborate? I tried looking it up but nothing meaningful comes up.

The TCP book. As in Transmission Control Protocol and it solves the problem way better than any of the janky hacks called pagination.

Re: API pagination design

#146

Post by Can_Not »

jiji33i wrote: post #138
atom_arranger wrote: post #27

I'm a fan of the "GraphQL Cursor Connections Specification", which could be applied even if not using GraphQL: https://relay.dev/graphql/connections.htm

"If all you have is a hammer, everything looks like a nail." Are GraphQL-fans so uncapable you have to put your GraphQL-spamming in every single post? Didn't you learn anything else in your life?

I mentally filter out "GraphQL spam" with the same rigor and aptitude I mentally filter out "bitcoin spam", but this unsubstantiated ad hominem does nothing to convince me that I'm smart or in-the-right for it.

Re: API pagination design

#147

Post by Can_Not »

callesgg wrote: post #123

I think is better to not use pagination at all. Design your api or page so that you receive all or more entries than you could possibly need. If the user wants them all pagination is just in the way. If the user does not want them all give them as many as they are likely to be able to handle.

This is terrible advice, sending the client as little as reasonably possible (therefore use pagination) is the most sane, performant, affordable, and secure default assumption. Any rails/laravel/feathersjs style framework have multiple types of pagination built in for free.

Re: API pagination design

#148

Post by raihansaputra »

sonthonax wrote: post #141

This is one of those obvious things that people have a blockheaded aversion towards. In every job I’ve consulted on, there has been a component where pagination is the bottleneck. And it’s rarely where pagination could be useful (like a static list of products). I typically see it when someone builds an aggregating (and aggravating) microservice that pulls all of the data from another service; that blocks the entire…

Sometimes it's the case of overoptimising/early optimization too. BE doesn't want to create another endpoint with simplified fields, FE being overly concerned with size of JSON responses over the wire / being processed client side (while the image for each item is not optimized themselves). Thanks for sharing the Big N approach, I have to try that.

Re: API pagination design

#149

Post by callesgg »

scrollaway wrote: post #142
callesgg wrote: post #134

Earlier quoted context omitted.

Given that is what the person is doing, that is likly what they are trying to do. Most of the times when I do things, I do it cause I want to do it, not cause I did an opsie.

"let's allow users to DOS us in a single request if they want to" is not an acceptable design decision, I'm sure you are aware of that.

Most services don’t handle ddos amounts of data, a few 1000 rows of data is fine.

Don’t design like you are Twitter if you are not. It is a waste of man hours. If I as a client need 10 000 rows of data that is what I will ask for either in one request or in a 100 consecutive.

Re: API pagination design

#150

Post by callesgg »

Can_Not wrote: post #147
callesgg wrote: post #123

I think is better to not use pagination at all. Design your api or page so that you receive all or more entries than you could possibly need. If the user wants them all pagination is just in the way. If the user does not want them all give them as many as they are likely to be able to handle.

This is terrible advice, sending the client as little as reasonably possible (therefore use pagination) is the most sane, performant, affordable, and secure default assumption. Any rails/laravel/feathersjs style framework have multiple types of pagination built in for free.

Performant for you assuming the client gives up after the first page. Less performant for you and the world as a hole if the client actually needs the data it ask for. Cause if it needs it the client will just make the request 100 times until it has all the data. What is more performant one query or a hundred.
Post reply on HN