From the code sample in the article I didn’t know you could append to a slice from within a go func
What's good about offset pagination; designing parallel cursor-based web APIs
11–20 of 51 posts
Re: What's good about offset pagination; designing parallel cursor-based web APIs
#12I think keeping temporal history and restricting paginated results to the data at the point in time where the first page was retrieved would be a pretty decent way to solve offset based interfaces (regardless of the complexity of making the query implementation efficient). Data with a lot of churn could churn on, but clients would see a consistent view until they return to the point of entry. Obviously this has some…
I think for most use cases, as a user i'd rather see the newest items in a list, then consistency of pagination. If i forget to manually refresh, i might miss out on important new items.
Why do you think it is important for users to have temporal consistency?
Re: What's good about offset pagination; designing parallel cursor-based web APIs
#13> it uses offsets for pagination... understood to be bad practice by today’s standards. Although convenient to use, offsets are difficult to keep performant in the backend This is funny. Using offsets is known to be bad practice because.... it’s hard to do. Look I’m just a UI guy so what do I know. But this argument gets old because I’m sorry, but people want a paginated list and to know how many pages are in the lis…
How do people even figure out they need to be on page 10? Sounds like a filter of some kind (e.g. before a date) would be better, except they trained themselves to use x amount of pages to approximate. Either way, 10 pages isn't so bad but tens of thousands can become troublesome as explained on https://shopify.engineering/pagination-relative-cursors
Maybe they've been there before and happen to remember "page 10".
Maybe someone else has been there before and told them "Go to page ten".
Maybe you know that there are 20 pages, and are looking to find the median, or are just informally playing around, looking at the distribution.
Same as you'd do with an interesting page of a book. I don't think I'd stop using page numbers in dead tree books if they somehow came with filters.
Re: What's good about offset pagination; designing parallel cursor-based web APIs
#14I think keeping temporal history and restricting paginated results to the data at the point in time where the first page was retrieved would be a pretty decent way to solve offset based interfaces (regardless of the complexity of making the query implementation efficient). Data with a lot of churn could churn on, but clients would see a consistent view until they return to the point of entry. Obviously this has some…
I think for most use cases, as a user i'd rather see the newest items in a list, then consistency of pagination. If i forget to manually refresh, i might miss out on important new items. Why do you think it is important for users to have temporal consistency?
Well I’ll use a recent example I encountered that was actually very frustrating. I was looking for a font to use for a logo for a personal project. The site I was using (won’t name and shame, and I can’t recall the site now anyway) had no sorting options, items were ordered by whatever “popularity” formula they use. As I paginated, many of the fonts I’d previously viewed would appear on subsequent pages, often in a different order. It was frustrating not just because I could tell that I was probably missing fonts that were being bumped up to previous pages, but also because it made me doubt my mental model of my own browsing history: “Did I navigate back too far? Did I forget a tangential click and end up on a different search path?”
It’s not a great UX. And in some ways I suspect that my own views were at least partially causing it, which made me more hesitant to even click on anything unless I was sure it was worth the disruption.
Re: What's good about offset pagination; designing parallel cursor-based web APIs
#15Earlier quoted context omitted.
I think for most use cases, as a user i'd rather see the newest items in a list, then consistency of pagination. If i forget to manually refresh, i might miss out on important new items. Why do you think it is important for users to have temporal consistency?
Well I’ll use a recent example I encountered that was actually very frustrating. I was looking for a font to use for a logo for a personal project. The site I was using (won’t name and shame, and I can’t recall the site now anyway) had no sorting options, items were ordered by whatever “popularity” formula they use. As I paginated, many of the fonts I’d previously viewed would appear on subsequent pages, often in a d…
This doesn't sound like a "temporal consistency" problem, rather an inconsistent and untransparent ordering issue.
Re: What's good about offset pagination; designing parallel cursor-based web APIs
#16To point out the obvious: generally API providers don’t particularly want you to pararelize your request (they even implement rate limiting to make it harder on purpose). If they wanted to make it easy to get all the results, they would allow you to access the data without pagination - just download all the data in one go.
> If they wanted to make it easy to get all the results Speaking from experience...we want to make it easy but also want to keep it performant. Getting the data all in one go is generally not performant and is easy to abuse as an API consumer. For example, always asking for all of the data rather than maintaining a cursor and secondary index (which is so much more performant for everyone involved).
We provide (internal) access to data where we provide interactive access via GraphQL-based APIs and bulk access via CSV or RDF dumps - I feel like dump files are grossly undervalued these days.
Re: What's good about offset pagination; designing parallel cursor-based web APIs
#17> it uses offsets for pagination... understood to be bad practice by today’s standards. Although convenient to use, offsets are difficult to keep performant in the backend This is funny. Using offsets is known to be bad practice because.... it’s hard to do. Look I’m just a UI guy so what do I know. But this argument gets old because I’m sorry, but people want a paginated list and to know how many pages are in the lis…
I can't find one right now, but I feel like there must be an algorithm that can identify the key that can be used for each page, yet cheaper than a full sort (ie cheaper than offset pagination in generality). Of course, a skip list would work for a secondary index.
Re: What's good about offset pagination; designing parallel cursor-based web APIs
#18Earlier quoted context omitted.
> If they wanted to make it easy to get all the results Speaking from experience...we want to make it easy but also want to keep it performant. Getting the data all in one go is generally not performant and is easy to abuse as an API consumer. For example, always asking for all of the data rather than maintaining a cursor and secondary index (which is so much more performant for everyone involved).
We provide (internal) access to data where we provide interactive access via GraphQL-based APIs and bulk access via CSV or RDF dumps - I feel like dump files are grossly undervalued these days.
I agree. I am going to reflect on this and see if there's a way to support dump files long term in our app. We sorta support it today but it's ad hoc implementation since an export can range from a few hundred of a thing to tens of millions of a thing.
Is there any good literature or patterns on supporting dumps in the tens of millions or larger?
I wrote a sheets plug-in that uses our cursor API to provide a full dump into a spreadsheet. Our professional services team is in love with it, so I bet they'd love generic data export capability.
Re: What's good about offset pagination; designing parallel cursor-based web APIs
#19From the code sample in the article I didn’t know you could append to a slice from within a go func
As long as you use the mutex locks
Re: What's good about offset pagination; designing parallel cursor-based web APIs
#20Earlier quoted context omitted.
Well I’ll use a recent example I encountered that was actually very frustrating. I was looking for a font to use for a logo for a personal project. The site I was using (won’t name and shame, and I can’t recall the site now anyway) had no sorting options, items were ordered by whatever “popularity” formula they use. As I paginated, many of the fonts I’d previously viewed would appear on subsequent pages, often in a d…
This doesn't sound like a "temporal consistency" problem, rather an inconsistent and untransparent ordering issue.
Huh? It’s absolutely a temporal consistency problem. The ordering was absolutely clear and consistent (most->least popular at time of request). But the popularity scores were changing so rapidly that “at time of request” makes the ordering unstable. If the ordering was determined by popularity at the time I accessed page 1, the ordering would have been stable.
Sure, that popularity score would be stale. But who cares?
Think of it this way. Suppose you’re viewing your Twitter timeline in recent order, and suppose the pagination (lazy loaded on scroll) worked this way, and suppose you have new Tweets arriving at the same rate you scroll. What you would see is the same page of Tweets repeat forever (well, until you hit the pagination cap).
This is why people come up with solutions like cursors. But what I was suggesting is that you can instead continue to use offsets (for the benefits discussed in the article like parallelism and predictability) if you paginate on the state of the data at the time you began (edit: or on the state of your sorting criteria at the time you began, which allows for the mitigations I described upthread).
That’s not to suggest that once you begin a pagination, you’ll forever access stale data. It’s to suggest that a set of pagination requests can be treated as a session accessing a stable snapshot.
This can also be totally transparent to the client, and entirely optional (eg pagination is performed with an offset and an optional validAt token).