Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

API pagination design

solovyov.net

Post reply on HN
Print view View original

111–120 of 150 posts

Re: API pagination design

#111

Post by bmn__ »

djeiasbsbo wrote: post #67
altdatathrow wrote: post #51

Earlier quoted context omitted.

fyi that mysql multiple column answer is incorrect, mysql does indeed support row/tuple comparisons [1]: > For row comparisons, (a, b) > (x, y) is equivalent to: > (a > x) OR ((a = x) AND (b > y)) [1] https://dev.mysql.com/doc/refman/8.0/en/comparison-operators...

Just in case you might be able to help me out.. is there any SQL flavor that supports specifying any except certain columns? For example something like: SELECT * EXCEPT FOO_ID FROM FOO; I have always wanted this but have never vome across it...

The correct answer is to just apply a view and enumerate the wanted column names once.

Otherwise, dynamic column names (that's your search engine fodder) require introspection…

    -- "Chinook" sample database
    select column_name
    from information_schema.columns
    where table_name = 'Track'
    except
    select 'UnitPrice'
… coupled with the moral equivalent of PL/pgSQL `execute` (i.e. run-time eval) statement.

Re: API pagination design

#112

Post by bswinnerton »

cj wrote: post #104
mewpmewp2 wrote: post #97

I think that cursor should be used more with realtime data and more stable data like products in an e-commerce shop should use paginated queries. There's many UX benefits for paginated queries. It gives customer a clear overview of how many products there are and customer can also navigate faster to get a better understanding of what general prices are. If the table/list doesn't have good filters, customer can easily…

A while ago, github replaced the pagination on their commits page to be based on a git commit ID instead of a page number. After that change, it made it a lot more difficult to find the oldest commit in a large repo - frustrating. That said, I think the percent of average ability internet users (ie. people visiting an ecommerce shop) will virtually never attempt to navigate by changing values in a query string param.…

The move from page-based pagination to cursor based pagination on the commits page was so that the underlying queries could be moved to GraphQL, which use cursors (specifically the GraphQL Relay specification). That was one of the first pages that was moved over, and these days I think it would be relatively trivial to add a “last” page, since the Relay connection has the concept of totalCount and we have a few tricks around first/last cursors. Theoretically the ordering could also be flipped to view the first commit on the first page.

Re: API pagination design

#113

Post by dmitryminkovsky »

dmlittle wrote: post #45
dmitryminkovsky wrote: post #17

Earlier quoted context omitted.

Indefinitely. It's not the kind of cursor that a database gives out, which is a resource. A cursor in this context is just a pointer to some position in a set of results, something like "all results where date created is greater than $X". For SQL implementations, check out: https://github.com/hasura/graphql-engine/issues/141#issuecom... or https://stackoverflow.com/questions/38017054/mysql-cursor-ba...

It depends on the implementation. For example, the post mentions using Elasticsearch in production "which naturally supports this cursor stuff" but it has a max keep alive context duration of 24 hours[1] by default. This limit can be extended on a cluster level but it's unlikely they have it set to indefinite. Which means that their search context is likely only valid for 24 hours. [1] https://www.elastic.co/guide/en…

Yep, that’s the case if you use the scroll parameter, but their search_after API is not stateful and that’s the sort of thing this article is about.

Re: API pagination design

#114

Post by mewpmewp2 »

bswinnerton wrote: post #112
cj wrote: post #104

Earlier quoted context omitted.

A while ago, github replaced the pagination on their commits page to be based on a git commit ID instead of a page number. After that change, it made it a lot more difficult to find the oldest commit in a large repo - frustrating. That said, I think the percent of average ability internet users (ie. people visiting an ecommerce shop) will virtually never attempt to navigate by changing values in a query string param.…

The move from page-based pagination to cursor based pagination on the commits page was so that the underlying queries could be moved to GraphQL, which use cursors (specifically the GraphQL Relay specification). That was one of the first pages that was moved over, and these days I think it would be relatively trivial to add a “last” page, since the Relay connection has the concept of totalCount and we have a few trick…

Interesting, but right now it seems they return HTML and use some sort of Turbolinks like logic to make it seem single page app? Or is GraphQL not yet implemented there / used on some other layer in backend for example?

Re: API pagination design

#115

Post by u801e »

cj wrote: post #104
mewpmewp2 wrote: post #97

I think that cursor should be used more with realtime data and more stable data like products in an e-commerce shop should use paginated queries. There's many UX benefits for paginated queries. It gives customer a clear overview of how many products there are and customer can also navigate faster to get a better understanding of what general prices are. If the table/list doesn't have good filters, customer can easily…

A while ago, github replaced the pagination on their commits page to be based on a git commit ID instead of a page number. After that change, it made it a lot more difficult to find the oldest commit in a large repo - frustrating. That said, I think the percent of average ability internet users (ie. people visiting an ecommerce shop) will virtually never attempt to navigate by changing values in a query string param.…

> A while ago, github replaced the pagination on their commits page to be based on a git commit ID instead of a page number. After that change, it made it a lot more difficult to find the oldest commit in a large repo

git log has a reverse option to reverse the order of commits displayed. You can also provide a commit reference like HEAD~10 to get the 10th ancestor from the head commit.

Does Github have a way to specify URL parameters to achieve something similar?

Re: API pagination design

#116

Post by [deleted] »

mewpmewp2 wrote: post #98
momokoko wrote: post #38

This is trumpeted around and actually put into production every once in a while. The reason opaque pagination is an antipattern is because you can’t optimistically fetch resources. So your customer, the person that paying you for your product, needs to wait for some number of synchronous reads. With non-opaque offsets these can be done in parallel. If the typical request requires 4 pages, these can be done 4 at a tim…

I think that cursor should be used more with realtime data and more stable data like products in an e-commerce shop should use paginated queries. There's many UX benefits for paginated queries. It gives customer a clear overview of how many products there are and customer can also navigate faster to get a better understanding of what general prices are. If the table/list doesn't have good filters, customer can find w…

[deleted]

Re: API pagination design

#117

Post by bswinnerton »

mewpmewp2 wrote: post #114
bswinnerton wrote: post #112

Earlier quoted context omitted.

The move from page-based pagination to cursor based pagination on the commits page was so that the underlying queries could be moved to GraphQL, which use cursors (specifically the GraphQL Relay specification). That was one of the first pages that was moved over, and these days I think it would be relatively trivial to add a “last” page, since the Relay connection has the concept of totalCount and we have a few trick…

Interesting, but right now it seems they return HTML and use some sort of Turbolinks like logic to make it seem single page app? Or is GraphQL not yet implemented there / used on some other layer in backend for example?

Oh sorry, I should have clarified it’s GraphQL “under the hood” (backend). Most of GitHub is a Rails app, and in the case of the commits page, the controller makes a GraphQL query, and then takes the results and injects it into the view, all server side. The original idea was that if GitHub was using its own GraphQL API to build its pages, our users would have the exact same access as GitHub engineers.

Re: API pagination design

#118

Post by bswinnerton »

u801e wrote: post #115
cj wrote: post #104

Earlier quoted context omitted.

A while ago, github replaced the pagination on their commits page to be based on a git commit ID instead of a page number. After that change, it made it a lot more difficult to find the oldest commit in a large repo - frustrating. That said, I think the percent of average ability internet users (ie. people visiting an ecommerce shop) will virtually never attempt to navigate by changing values in a query string param.…

> A while ago, github replaced the pagination on their commits page to be based on a git commit ID instead of a page number. After that change, it made it a lot more difficult to find the oldest commit in a large repo git log has a reverse option to reverse the order of commits displayed. You can also provide a commit reference like HEAD~10 to get the 10th ancestor from the head commit. Does Github have a way to spec…

Not on that page, no, but you could easily fetch this information via the GraphQL API.

Re: API pagination design

#119

Post by djeiasbsbo »

bmn__ wrote: post #111
djeiasbsbo wrote: post #67

Earlier quoted context omitted.

Just in case you might be able to help me out.. is there any SQL flavor that supports specifying any except certain columns? For example something like: SELECT * EXCEPT FOO_ID FROM FOO; I have always wanted this but have never vome across it...

The correct answer is to just apply a view and enumerate the wanted column names once. Otherwise, dynamic column names (that's your search engine fodder) require introspection… -- "Chinook" sample database select column_name from information_schema.columns where table_name = 'Track' except select 'UnitPrice' … coupled with the moral equivalent of PL/pgSQL `execute` (i.e. run-time eval) statement.

Thanks for answering! Currently I do use views but it's just something I had wondered about.

I mean, the information from the information_schema must be updated anyway when one deletes a column or table, so I thought maybe a function like that which looks it up could exist.

I will try with PL/pgSQL, have long wanted to familiarise myself with it anyway.

Re: API pagination design

#120

Post by anamexis »

knorker wrote: post #83
londons_explore wrote: post #81

Earlier quoted context omitted.

What if the last element on a page is deleted when you click "next"?

A B C D 2 items per page first page is A&B. Next query is "WHERE ID>'B'". What's the problem?

Aa and Ab are inserted after loading the first page.
Post reply on HN