One of the rare times HN can come together and agree on something. Which is: that this is a loose weekend project thrown together quickly that doesnt solve any problem in particular
Show HN: 1 Million Rows
31–40 of 44 posts
Re: Show HN: 1 Million Rows
#32Earlier quoted context omitted.
Loading more than a few thousand rows on a web page will make unusably slow. Especially when you add a lot more features to it. This is such that only what's seen or about to be seen is put on the page. The rest is kept ready on the server on local memory depending on what the user is doing. This allows for a scalable solution that allows you to view thousands of records and interact with them
I'm not so sure that loading "more than a few thousand rows" is as bad as it used to be. I did some quick benchmarks a couple years back. It's been a while but I want to say that Chrome was drawing 10k rows of a decent size each (10 columns of real world data and about 500b/row iirc) in about 300ms on a 10 year old MBP. I'll do a little benchmarking later today if I get a chance.
10 columns is pretty tame. Add in another 5 or 10 to introduce horizontal scrolling, add in some fixed position columns, html inputs like a checkbox in each row, apply styling or custom rendering to individual cells, and it can starts to show a little more quickly.
Re: Show HN: 1 Million Rows
#33Not 100% sure what I'm looking at here? Am I missing something or is it just table of data with an "infinite scroll" that loads 200 records at a time?
Loading more than a few thousand rows on a web page will make unusably slow. Especially when you add a lot more features to it. This is such that only what's seen or about to be seen is put on the page. The rest is kept ready on the server on local memory depending on what the user is doing. This allows for a scalable solution that allows you to view thousands of records and interact with them
I know why you're doing it; I'm just not sure what I'm looking at. I'm not sure if this is supposed to be a product, but right now it's literally just a paginated HTML table?
Re: Show HN: 1 Million Rows
#34I tried the live demo with synthetic data. To whom it may concern: I scrolled a bit with the scrollbar on iOS and the page immediately crashed.
On Android the rows just didn't load
On Android Firefox they do load. I'm on my tablet. It fills the first half of the page with rows. The bottom half is empty.
Re: Show HN: 1 Million Rows
#35I had hoped it might be like one of those "1 million pixels" websites, where anyone can run queries to update the rows.
Re: Show HN: 1 Million Rows
#36i dabbled into this last year and wrote about the challenge here https://newbeelearn.com/blog/million-rows-csv-debug-story/
Re: Show HN: 1 Million Rows
#37You basically reinvented something Elixir Streams already nails out of the box.
Streams in Elixir are lazy, chunked, and backpressure-friendly, so you can process any size dataset without loading it all into memory...whether it's a million or a trillion rows. The trick is you never try to render them all in the browser (that's where virtualization comes in).
So yeah...neat work, but battle-tested versions of this have been around for a while.
Re: Show HN: 1 Million Rows
#38You basically reinvented something Elixir Streams already nails out of the box. Streams in Elixir are lazy, chunked, and backpressure-friendly, so you can process any size dataset without loading it all into memory...whether it's a million or a trillion rows. The trick is you never try to render them all in the browser (that's where virtualization comes in). So yeah...neat work, but battle-tested versions of this hav…
above 50k and the UI needs to change because 1) you can’t count the collection, and 2) the scrollheight becomes too unwieldy to use the mouse, slight adjustments to the handle will skip forward many pages. And if you try at some point you exceed the pixel height browser scroll bars can support, needing a custom non-native scroll bar. anyway well before that point, roughly at 50k records you need to switch to “search” UX (much smaller result sets) because there is no way to actually access page 99910 of your million record collection.
tldr: “show me the demo”
Re: Show HN: 1 Million Rows
#39What the heck is this app doing?
Re: Show HN: 1 Million Rows
#40Not 100% sure what I'm looking at here? Am I missing something or is it just table of data with an "infinite scroll" that loads 200 records at a time?
Loading more than a few thousand rows on a web page will make unusably slow. Especially when you add a lot more features to it. This is such that only what's seen or about to be seen is put on the page. The rest is kept ready on the server on local memory depending on what the user is doing. This allows for a scalable solution that allows you to view thousands of records and interact with them
We can't view at thousands of records at the same time, that is why we have pagination, filters and sorting. Any library that can display 50 rows per page is good to go. The real work is on the backend.
1M rows in memory, with pagination or infiniscroll, is interesting only if we load that data and go offline, and all of filtering and sorting is up to the browser. I'd say that it's a niche use case. Furthermore 1M row x 1kB each is 1GB so we enter an order of magnitude ridden with troubles.