Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We switched to cursor-based pagination

moderntreasury.com

Post reply on HN
Print view View original

111–118 of 118 posts

Re: We switched to cursor-based pagination

#111

Post by friendzis »

VWWHFSfQ wrote: post #97
andy800 wrote: post #78

Why does every web site default to aggressively paginate their information? Pagination sucks, it's a waste of time and of clicks, and should be a last resort. Sure, when Google returns millions of search results, paginate. But: For instance, if you have 40 entries and specify that there should be 10 items per page 40 entries??? Just show them all to me. My browser has a scrollbar, CTRL-F is much faster than your sear…

> Why does every web site default to aggressively paginate their information? Because data retrieval costs money and resources. Unbounded API responses are a susceptible to denial-of-service. Well-architected APIs will always cap the number of results they return per-call.

Do you perhaps conflate API level pagination with display pagination?

Re: We switched to cursor-based pagination

#112

Post by redditor98654 »

hirundo wrote: post #38

My sympathies to the coders downstream of this large change for the amount of work required. I would like to add cursor based pagination to the APIs I manage. It would not be an option to go to our clients and explain that offset-pagination will be removed. There seems to be very little cost in supporting both.

AWS does the pagination thing for all LIST APIs and they encode the token in a way they clients cannot deconstruct anything out of it. For them it is just an opaque string. In fact it is cryptographically encrypted and not something simple like a Base64 encoding of some Json data.

And the token has enough metadata to reconstruct the query from where it left behind in the previous call and perform additional security checks so that it cannot be used by other customers to get results from a different account.

I have actually migrated databases from Aurora to DynamoDb behind the scenes where customers continued to call these APIs with these tokens and the calls would flip to the new DB and resume from the last place. No downtime or "maintenance window" which would be a non-starter at AWS anyway.

I recommend this for any external service. For internal services, if you work closely enough with your users, may be you can opt for something more transparent.

Re: We switched to cursor-based pagination

#113

Post by Khoth »

zaroth wrote: post #71
robertknight wrote: post #59

Earlier quoted context omitted.

> Google does it with millions of search results Google cheats, but in a way that very few users will notice. You can only access the first 20 pages of search results. Depending on user behavior, this is one way to offer navigation via page number while limiting worst-case cost.

I doubt that 21st page actually exists, even on the back end. I would bet the top-line number of results is some sort of extrapolated estimation thing, a hand-wavey way to represent how deep the hypothetical result set is for that query.

It definitely is an estimate. If you search for a rare enough term that there aren't 20 pages of results, it can still estimate thousands of results until you call its bluff by going to a later page, at which point the estimate is replaced by the actual number.

Re: We switched to cursor-based pagination

#114

Post by justsomeuser »

Aeolun wrote: post #94
andy800 wrote: post #78

Why does every web site default to aggressively paginate their information? Pagination sucks, it's a waste of time and of clicks, and should be a last resort. Sure, when Google returns millions of search results, paginate. But: For instance, if you have 40 entries and specify that there should be 10 items per page 40 entries??? Just show them all to me. My browser has a scrollbar, CTRL-F is much faster than your sear…

And this kind of thinking is how you end up with 2mb per request. It’ll be like Jira, pulling in literal megabytes of data and still slow.

I don't think that is a bad thing, considering that a YouTube video in HD would be 500MB and people flick through them like web pages.

I agree with OP that if you are on a powerful machine with fast internet it would be better just to load a massive HTML document of list items.

Re: We switched to cursor-based pagination

#115

Post by pwinnski »

andy800 wrote: post #99
VWWHFSfQ wrote: post #97

Earlier quoted context omitted.

> Why does every web site default to aggressively paginate their information? Because data retrieval costs money and resources. Unbounded API responses are a susceptible to denial-of-service. Well-architected APIs will always cap the number of results they return per-call.

Nothing needs to be "unbounded." Set the cap at 100 or 200 records instead of 10-20. Better yet, plenty of results can be cached. A custom search might require a database query, but Newegg's initial listing of MicroSD cards, graphics cards, hard drives, etc, -- queries that are fulfilled identically hundreds of times per day -- can be cached.

At which point you're not actually arguing against pagination, you're suggesting a higher page size.

I agree, page sizes are usually far too short by default! Especially when I'm on a desktop computer and a fiber connection. I think people using phones and cellular connections probably appreciate short page sizes, so it would be nice if sites had different page sizes for different clients.

Re: We switched to cursor-based pagination

#116

Post by pwinnski »

djbusby wrote: post #13
ReactiveJelly wrote: post #9

Earlier quoted context omitted.

So basically do it like Reddit? https://old.reddit.com/?count=25&after=t3_wtpvdp I noticed Reddit's pagination has that "after" parameter, which points to the last post on the current page. It glitches out if the last item is deleted by moderators, but otherwise it works smoothly.

On Reddit I frequently see the "next" page having the same posts as the previous page. Not all the same but many of the same. Like, maybe after is being respected but the sorting is different or something.

The problem with Reddit is that the sorting isn't by time, so the content order changes.

The only solution I can think of for that is to track which individual posts have been shown to which users, which is quite a lot of work to do.

Re: We switched to cursor-based pagination

#117

Post by erikpukinskis »

simonw wrote: post #53
erikpukinskis wrote: post #47

Earlier quoted context omitted.

Interesting… what happens if new records are added with that 15 value? Do you need an implied secondary sort with the record created time? Also what if there are more than $page_size records with that 15 value?

Yes, if you want to support new records being added it's up to you to include something like a created date as part of your specified sort order. More than page_size records with that value works fine - that's why the primary key is included as a tie-breaker.

Doesn’t that take you back to the original problem though: if you delete the primary key the query fails?

Would the created_at be a better option than the primary key? That can’t be deleted or changed, and that would give you a stable secondary sort?

Re: We switched to cursor-based pagination

#118

Post by erikpukinskis »

imbusy111 wrote: post #58
erikpukinskis wrote: post #44

Earlier quoted context omitted.

Does that mean you have to scan the entire results set to get the right page? So if I am on page 100, I have to query pages 1-99 and discard them? Or is there a trick here I’m missing?

There are no pages anymore. You fetch a record by ID and next N records.

Right, but in your database query doesn’t that require you to get a monster result set and filter it down in your application layer?

There’s no SQL clause for “WHERE IT WOULD COME AFTER id=ah73d IN THE RESULT SET” as far as I’m aware.

Post reply on HN