Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

API pagination design

solovyov.net

Post reply on HN
Print view View original

81–90 of 150 posts

Re: API pagination design

#81

Post by londons_explore »

Too wrote: post #52
et1337 wrote: post #8

Another reason to use cursors is to avoid the problem of repeated or skipped elements caused by concurrent edits. If you use offsets and you’re on page 10, and someone deletes an item on page 1, the whole list shifts and you can accidentally skip an item on page 11. Likewise if someone adds an item on page 1 and you’re on page 10, one of the page 10 items will also show up on page 11. Cursors elegantly sidestep these…

One way to get around this is to drop the notion of page number. Instead do greater than last element on last page. But this has the drawback of only working as long as the sorting field doesn't have many duplicates. You have to beware if timestamps are high resolution enough for your application or if you have 100+ entries with same family name. Usually users are smart(lazy) enough to realize they got more than 100…

What if the last element on a page is deleted when you click "next"?

Re: API pagination design

#82

Post by twic »

londons_explore wrote: post #81
Too wrote: post #52

Earlier quoted context omitted.

One way to get around this is to drop the notion of page number. Instead do greater than last element on last page. But this has the drawback of only working as long as the sorting field doesn't have many duplicates. You have to beware if timestamps are high resolution enough for your application or if you have 100+ entries with same family name. Usually users are smart(lazy) enough to realize they got more than 100…

What if the last element on a page is deleted when you click "next"?

You use the sort key as the page selector. So if it's cities sorted by name, the query is "give me the first 20 cities with names after 'Constantinople'". If Constantinople gets the works, it doesn't matter, because there are still 20 cities whose names come after that lexicographically.

Re: API pagination design

#83

Post by knorker »

londons_explore wrote: post #81
Too wrote: post #52

Earlier quoted context omitted.

One way to get around this is to drop the notion of page number. Instead do greater than last element on last page. But this has the drawback of only working as long as the sorting field doesn't have many duplicates. You have to beware if timestamps are high resolution enough for your application or if you have 100+ entries with same family name. Usually users are smart(lazy) enough to realize they got more than 100…

What if the last element on a page is deleted when you click "next"?

A B C D

2 items per page

first page is A&B. Next query is "WHERE ID>'B'".

What's the problem?

Re: API pagination design

#84

Post by knorker »

graderjs wrote: post #49
boulos wrote: post #34

Given the confusion over the reuse of the term “cursor”, I prefer the terms we (Google) have settled on for Pagination [1]: page token and page size. It’s often awkward to argue about “really, do I need to support pagination from the outset”, but the examples given in that AIP are from real-world battle scars. Even if you just set the page size to a big number, at least you prepare your callers for “one day, you’ll n…

Not contradicting, just adding that GCP (Datastore) has had cursors for at least 8 years. When I came across that using App Engine many years ago I really liked the cursor thing, and it stuck with me. I found it a little funny this article sounded a bit like "Here's this new thing"

It's definitely not new, but worth repeating.

But yes, I'm surprised it would show up on hackerNEWS.

Re: API pagination design

#87

Post by knorker »

pram wrote: post #40

Is an auto increment sequential number good enough for a 'cursor' or is there something I'm missing here.

As long as it can be used to "seek" directly into where the last page ended.

The problem with offset and limit is that you have to actually find the previous N results in order to get the N+1th. But with a cursor you can just "continue".

(when I say "seek" I mean use indexes. I'm comparing to listing a tar file (requires reading the whole thing to find the file in the middle) vs zipfile (seek to where the file is, and read))

Re: API pagination design

#89

Post by egeozcan »

twic wrote: post #82
londons_explore wrote: post #81

Earlier quoted context omitted.

What if the last element on a page is deleted when you click "next"?

You use the sort key as the page selector. So if it's cities sorted by name, the query is "give me the first 20 cities with names after 'Constantinople'". If Constantinople gets the works, it doesn't matter, because there are still 20 cities whose names come after that lexicographically.

If it is updated, though (in this case, renamed to Istanbul), it can appear later once again, and newly inserted rows behind "Constantinople" will never appear.

I mean, these problems happen if you use offsets too, and my point is that they don't matter.

The most hardcore solution to those would be pre-computing pages but I doubt it makes sense to bring even more state to the search results.

Re: API pagination design

#90

Post by flqn »

In addition to using cursors or whatever pagination method suits your backend in query params, it's also a good idea to include pagination URLs in the response itself if you can. Greatly simplifies what the client has to do and reduces coupling since now the client doesn't need to care what your query param structure is
Post reply on HN