Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UI considerations for designing large data tables

coyleandrew.medium.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: UI considerations for designing large data tables

#21

Post by warpspin »

taeric wrote: post #14
whalesalad wrote: post #13

Earlier quoted context omitted.

I don't really know how to interpret this comment. 3.2M rows are not loading into the browser at once, only about 10k. The page size is configurable. The frontend and backend have a contract to agree on how this works, so as the user scrolls (and frontend needs another page) it asks the backend for more. The frontend will keep up to N pages (also configurable) cached in the client. Works shockingly well without any f…

Apologies, it was not the technical side of it that had me musing on it. I was genuinely curious why and how I would "scroll" through that much data in a meaningful way. And I don't mean this as a heavy criticism of the idea. I'm assuming it is useful to you. Always fun to hear about how this sort of thing is used.

You don't scroll through such a list - you use filters, pivoting etc. For end users, it's often quite comfortable to unify this in an Excel-like interface, as they're used to it. In most CRUD projects I had, users had an aversion against paged/pre-filtered displays and rather would have everything in one list where they can dynamically filter it if necessary.

Re: UI considerations for designing large data tables

#22

Post by bzzzt »

wolverine876 wrote: post #9

I feel the tension when just looking at those screenshots, thinking of trying to utilize a large table in a website UI - JavaScript, scrolling, dynamic updates, the pauses and hangups, Internet latency, etc., plus the very limited capabilities. Just give me my local spreadsheet app. Your web UI is never going to come close to that speed and functionality. I understand that might not fit every use case, but I guess th…

Think of multi user databases where you want the latest version of the data. Local spreadsheets would be a synchronization nightmare. Also, web applications are seldom one single sheet, but an intertwined view on lots of tables combined with custom functionality you can't replicate easily in a spreadsheet. And that custom functionality needs to be updated once in a while. The web has become the easiest software delivery platform: you only have to update the server side. Even if a web interface is a bit janky it's a better alternative than maintaining lots of native apps (only Windows, Mac, how many Linux distributions?) or one of the cross platform toolkits that don't ever work 100% right and still require you to test on every platform.

Re: UI considerations for designing large data tables

#23

Post by simion314 »

taeric wrote: post #15
simion314 wrote: post #10

Earlier quoted context omitted.

Hope I do not misunderstand your comment, but I think the point is not that you use millions of rows but that you should be able to use all of your rows/data without having to use pagination as a workaround to hide the problem that the GUI is incapable to render too many rows at once. Imagine having such a terrible UX while editing a large source code files, the editor loads only 20 lines then you need to click a but…

Apologies, that was not what I meant. I meant it more directly in assuming the technical side works and you are able to scroll just fine, how do you meaningfully do so with that much data? And I don't mean this as an attack. I presume there are some techniques that I just don't know. Or data sets I just don't typically interact with. For the ones I am used to, aggregates are key to working with them. That and graphic…

Say in my web app I can save "projects" and I have 200 projects in total. Because of DOM performance the devs implemented pagination say 40 items at max so I have 5 pages of projects. We all know the issues with pagination.

If all the projects are loaded I can :

1 scroll all of them , exactly like I scroll a document or how I scroll in my File Manager or Image Viewer

2 I can do Instant Filter and Search, no backend requests, don't you hate when say you open a YouTube account video list and you just can't use the browser Find function because the content is not in the page

One example is YouTube in a specific account Videos section , you are forced to scroll and wait, scroll and wait , scroll and wait , they can try and hack this to be smoother but this could be instant since the JSON for all the videos could be returned at once, it is just text and is more efficient then returning it in chunks , then like in competent GUI toolkits you have a big grid with all the results and the toolkit/framework Grid does the work for you in making sure that everything is rendered efficiently and smooth.

I would prefer at least an option in this apps to offer the customer a setting to decide the Page size , maybe I want a slower initial load but have everything loaded in one screen , imagine paginated File Manager

Re: UI considerations for designing large data tables

#24

Post by taeric »

simion314 wrote: post #23
taeric wrote: post #15

Earlier quoted context omitted.

Apologies, that was not what I meant. I meant it more directly in assuming the technical side works and you are able to scroll just fine, how do you meaningfully do so with that much data? And I don't mean this as an attack. I presume there are some techniques that I just don't know. Or data sets I just don't typically interact with. For the ones I am used to, aggregates are key to working with them. That and graphic…

Say in my web app I can save "projects" and I have 200 projects in total. Because of DOM performance the devs implemented pagination say 40 items at max so I have 5 pages of projects. We all know the issues with pagination. If all the projects are loaded I can : 1 scroll all of them , exactly like I scroll a document or how I scroll in my File Manager or Image Viewer 2 I can do Instant Filter and Search, no backend r…

Agreed on most points. But 200 is a very different number from 3.2m. I would expect I could load all 200 in the client and get fast local operations. For 3.2m, I'd expect sending the filter off client would be faster.

Essentially, for performance there is a tradeoff between sending the data to where you will perform operations, and sending operations to the data. I don't know where the cutoff is, but I'd expect 3.2m to be on the "send operations to the data" side.

Re: UI considerations for designing large data tables

#25

Post by taeric »

warpspin wrote: post #21
taeric wrote: post #14

Earlier quoted context omitted.

Apologies, it was not the technical side of it that had me musing on it. I was genuinely curious why and how I would "scroll" through that much data in a meaningful way. And I don't mean this as a heavy criticism of the idea. I'm assuming it is useful to you. Always fun to hear about how this sort of thing is used.

You don't scroll through such a list - you use filters, pivoting etc. For end users, it's often quite comfortable to unify this in an Excel-like interface, as they're used to it. In most CRUD projects I had, users had an aversion against paged/pre-filtered displays and rather would have everything in one list where they can dynamically filter it if necessary.

Right. But that just brings us back to "how do people work with a scrolling list of 3.2m records?"

I get the point of wanting it locally to use power tools. And I get that the browser is probably capable of implementing a lot of power tools. Seems silly to insist on doing it all "in memory" on the browser, though?

That is, if the idea is you are doing pivots and filters, I don't know why a server side hit wouldn't be better for that. Similarly, when I look at something like a stock ticker for the day, I don't expect every single transaction was sent to my browser to create the graph. It /could/ be done that way, but why?

More directly to the question I had here, why and how would someone need a scroll list of every market transaction? For fine audits, I would get it, but even then I'd expect some sort of search or anomaly detection?

Still, I think if the answer is to "get it in the users hand and let them do what they will with the data," I can accept that. Goal isn't necessarily to let the users scroll the data endlessly, but for them to use any bespoke tooling they are already using.

Re: UI considerations for designing large data tables

#26

Post by ceolter »

mondrian wrote: post #8
whalesalad wrote: post #2

ag-grid is pretty great at all of this. it will get you 90% of the way there, then you just need to ensure you are ordering columns correctly and using the right types, and potentially using a few custom cell renderers to display novel data. we're using it to infinite scroll / sort / filter a table with 3.2M rows of data and it 'just works'

I've tried ag-grid but trying to make it look consistent across light and dark mode was a pain. The available light and dark themes have a lot of discrepancies. To make it look right you have to dive deep into its styling system which is complex.

Have you looked at the AG Grid release from two weeks ago? We released a new theme called Quartz, which I understand is much better. Eg it has automatic switching between dark and bright mode based on users desktop preferance.

Re: UI considerations for designing large data tables

#27

Post by ceolter »

This is exactly what AG Grid Server Side Row Model is designed for. I would recommend anyone considering putting a grid on a large dataset to at least understand what AG Grid has to offer here. The trick is only load what the user needs to see, AG Grid achieves this with Grouping and lazy loading children as the user expands, and also infinite scrolling to present more rows as the user scrolls down.

Re: UI considerations for designing large data tables

#28

Post by simion314 »

taeric wrote: post #24
simion314 wrote: post #23

Earlier quoted context omitted.

Say in my web app I can save "projects" and I have 200 projects in total. Because of DOM performance the devs implemented pagination say 40 items at max so I have 5 pages of projects. We all know the issues with pagination. If all the projects are loaded I can : 1 scroll all of them , exactly like I scroll a document or how I scroll in my File Manager or Image Viewer 2 I can do Instant Filter and Search, no backend r…

Agreed on most points. But 200 is a very different number from 3.2m. I would expect I could load all 200 in the client and get fast local operations. For 3.2m, I'd expect sending the filter off client would be faster. Essentially, for performance there is a tradeoff between sending the data to where you will perform operations, and sending operations to the data. I don't know where the cutoff is, but I'd expect 3.2m…

if there are 3 million simple objects like title, url, thumbnail url, why would be faster to send the request, to the filtering in the backed db and then send the response back? I suspect that an Array.filter would be as fast with a db search + the request overhead.

I just wish that the DOM would have some built in List,Table, Grid components, like you have in Qt,.Net WPF or Flex4 . Today we just have divs in divs in divs.

Re: UI considerations for designing large data tables

#29

Post by wolverine876 »

bzzzt wrote: post #22
wolverine876 wrote: post #9

I feel the tension when just looking at those screenshots, thinking of trying to utilize a large table in a website UI - JavaScript, scrolling, dynamic updates, the pauses and hangups, Internet latency, etc., plus the very limited capabilities. Just give me my local spreadsheet app. Your web UI is never going to come close to that speed and functionality. I understand that might not fit every use case, but I guess th…

Think of multi user databases where you want the latest version of the data. Local spreadsheets would be a synchronization nightmare. Also, web applications are seldom one single sheet, but an intertwined view on lots of tables combined with custom functionality you can't replicate easily in a spreadsheet. And that custom functionality needs to be updated once in a while. The web has become the easiest software deliv…

Those are good points. I was thinking only of one use case, but obviously there are many more. It makes me think someone might try to create a high-performance spreadsheet-like UI for web applications. I've seen some spreadsheet-like UIs, but none I looked forward to using - though Google Sheets isn't horrible. (Probably, if it could be done easily, everyone would be using it.)

Re: UI considerations for designing large data tables

#30

Post by taeric »

simion314 wrote: post #28
taeric wrote: post #24

Earlier quoted context omitted.

Agreed on most points. But 200 is a very different number from 3.2m. I would expect I could load all 200 in the client and get fast local operations. For 3.2m, I'd expect sending the filter off client would be faster. Essentially, for performance there is a tradeoff between sending the data to where you will perform operations, and sending operations to the data. I don't know where the cutoff is, but I'd expect 3.2m…

if there are 3 million simple objects like title, url, thumbnail url, why would be faster to send the request, to the filtering in the backed db and then send the response back? I suspect that an Array.filter would be as fast with a db search + the request overhead. I just wish that the DOM would have some built in List,Table, Grid components, like you have in Qt,.Net WPF or Flex4 . Today we just have divs in divs in…

I mean, fair, you could have tiny objects? My assertion was to not send all 3m rows to the frontend, but only the results of any aggregations and such. Want to know the top N? That should require sending N rows, period. Same for filtering and such. This is basically just a restatement of the "send the operations to the data" approach. Is a big part of why doing joins in the database is so much faster than doing them in the application layer.

And, as you allude, the shuffling of all of the DOM overhead to manage what is visible is non-trivial. Yes, you can basically flyweight it to save memory, but it is only a matter of time before the user wants Ctrl-F to work and then you try to find a way to put the whole object in a place for the user to directly work against. (Yeah, you would probably try to capture Ctrl-F in the application and fake the native search. But then case folding and other concerns now have to be reimplemented by you.)

Post reply on HN