Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

API pagination design

solovyov.net

Post reply on HN
Print view View original

41–50 of 150 posts

Re: API pagination design

#41

Post by ajitid »

contravariant wrote: post #22
et1337 wrote: post #8

Another reason to use cursors is to avoid the problem of repeated or skipped elements caused by concurrent edits. If you use offsets and you’re on page 10, and someone deletes an item on page 1, the whole list shifts and you can accidentally skip an item on page 11. Likewise if someone adds an item on page 1 and you’re on page 10, one of the page 10 items will also show up on page 11. Cursors elegantly sidestep these…

Though you can't avoid seeing an item just before it is deleted and skipping an item just before it is created. This isn't a huge problem but does mean that the list you end up with may not be a coherent view of the database. But if you require guarantees like that then you might be better of just using SQL directly. Or just include everything in a single response, HTTPS was designed to handle arbitrary length payloa…

> use a more streamable format than JSON

Did you mean we should stream data over WebSocket or use HTTP/2 or do we need to do something different altogether?

Re: API pagination design

#42

Post by alexeldeib »

ajitid wrote: post #41
contravariant wrote: post #22

Earlier quoted context omitted.

Though you can't avoid seeing an item just before it is deleted and skipping an item just before it is created. This isn't a huge problem but does mean that the list you end up with may not be a coherent view of the database. But if you require guarantees like that then you might be better of just using SQL directly. Or just include everything in a single response, HTTPS was designed to handle arbitrary length payloa…

> use a more streamable format than JSON Did you mean we should stream data over WebSocket or use HTTP/2 or do we need to do something different altogether?

AFAIK chunked transfer works specifically with HTTP 1.1, think Kubernetes for example uses this with JSON in a few prominent places for things like WATCH. But that setup is hardly ideal for the use case.

Re: API pagination design

#44

Post by ajitid »

et1337 wrote: post #8

Another reason to use cursors is to avoid the problem of repeated or skipped elements caused by concurrent edits. If you use offsets and you’re on page 10, and someone deletes an item on page 1, the whole list shifts and you can accidentally skip an item on page 11. Likewise if someone adds an item on page 1 and you’re on page 10, one of the page 10 items will also show up on page 11. Cursors elegantly sidestep these…

I have question, lets say we have

page 1 elements: A B C D E

page 2 elements: F G H I J

so 5 elements in each page.

Suppose I'm on page 2. If I insert a new element Q and it gets pushed as first then page 1 will have Q A B C D. Now if I go back to page 1, I'll get A B C D E and also a token/pointer to go back one more time only to retrieve Q.

So while cursor solved the issues you mentioned, it still will have this case in which pagination gets broken. I'm interested how we can tackle this.

Re: API pagination design

#45

Post by dmlittle »

dmitryminkovsky wrote: post #17
amelius wrote: post #6

How long will the cursor be valid?

Indefinitely. It's not the kind of cursor that a database gives out, which is a resource. A cursor in this context is just a pointer to some position in a set of results, something like "all results where date created is greater than $X". For SQL implementations, check out: https://github.com/hasura/graphql-engine/issues/141#issuecom... or https://stackoverflow.com/questions/38017054/mysql-cursor-ba...

It depends on the implementation. For example, the post mentions using Elasticsearch in production "which naturally supports this cursor stuff" but it has a max keep alive context duration of 24 hours[1] by default. This limit can be extended on a cluster level but it's unlikely they have it set to indefinite. Which means that their search context is likely only valid for 24 hours.

[1] https://www.elastic.co/guide/en/elasticsearch/reference/curr...

Re: API pagination design

#46

Post by Andys »

Centigonal wrote: post #2

I've worked with APIs (mainly based on ElasticSearch backends) that implement something like this pattern. There, the "cursor" is called a "ScrollId." Here's what that looks like in ES: https://www.elastic.co/guide/en/elasticsearch/client/java-re... The APIs I generally work will abstract away most of the scroll context configuration stuff, and just return a page of results, plus the ScrollId for the next page.

Also, they have a concurrent version called "Sliced Scroll" which lets you do multi-threaded fetch in N streams, each with their own cursor.

Re: API pagination design

#48

Post by Andys »

blueline wrote: post #4
Centigonal wrote: post #2

I've worked with APIs (mainly based on ElasticSearch backends) that implement something like this pattern. There, the "cursor" is called a "ScrollId." Here's what that looks like in ES: https://www.elastic.co/guide/en/elasticsearch/client/java-re... The APIs I generally work will abstract away most of the scroll context configuration stuff, and just return a page of results, plus the ScrollId for the next page.

Elastic search now recommends using “search_after” instead of scrolling for deep pagination, although mechanically they’re pretty similar to use

Yeah - search_after is the stateless version

Re: API pagination design

#49

Post by graderjs »

boulos wrote: post #34

Given the confusion over the reuse of the term “cursor”, I prefer the terms we (Google) have settled on for Pagination [1]: page token and page size. It’s often awkward to argue about “really, do I need to support pagination from the outset”, but the examples given in that AIP are from real-world battle scars. Even if you just set the page size to a big number, at least you prepare your callers for “one day, you’ll n…

Not contradicting, just adding that GCP (Datastore) has had cursors for at least 8 years. When I came across that using App Engine many years ago I really liked the cursor thing, and it stuck with me. I found it a little funny this article sounded a bit like "Here's this new thing"

Re: API pagination design

#50

Post by cheschire »

ajitid wrote: post #41
contravariant wrote: post #22

Earlier quoted context omitted.

Though you can't avoid seeing an item just before it is deleted and skipping an item just before it is created. This isn't a huge problem but does mean that the list you end up with may not be a coherent view of the database. But if you require guarantees like that then you might be better of just using SQL directly. Or just include everything in a single response, HTTPS was designed to handle arbitrary length payloa…

> use a more streamable format than JSON Did you mean we should stream data over WebSocket or use HTTP/2 or do we need to do something different altogether?

Check out MessagePack.

https://msgpack.org/

Post reply on HN