Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CSV Explorer (YC F1) - Explore CSVs with Millions of Rows

csvexplorer.com

Post reply on HN
Print view View original

21–30 of 35 posts

Re: Show HN: CSV Explorer (YC F1) - Explore CSVs with Millions of Rows

#21

Post by jastr »

seanp2k2 wrote: post #19
jastr wrote: post #16

Earlier quoted context omitted.

Thanks - having worked with health data for a few years I can relate. Unfortunately, I don't have plans for a desktop app right now.

Think not of a desktop app, but of doing everything client-side in JS. That way, it's still a web app, but you're not schlepping [sensitive|large] data between front and back-ends. Also, by offloading the work onto clients, it scales much better - you could host the app on a CDN and have no real back-end.

I originally wrote it in entirely client side JS, but it didn't scale nicely past a few hundred thousand rows. For the really big datasets, CSV Explorer loads them into Redshift - queries takes a few seconds!

Re: Show HN: CSV Explorer (YC F1) - Explore CSVs with Millions of Rows

#22

Post by lomereiter »

jastr wrote: post #9
xaybey wrote: post #6

I usually use csvkit ( https://csvkit.readthedocs.io/en/1.0.1/index.html ). There are commands to list the columns, filter, browse the data in a somewhat formatted way with less. Typing the commands is a huge pain though, and I would be very interested in a tool that could instantly pop open and let me peruse. Excel can take minutes to load and eagerly does a lot of unhelpful formatting on things like dates and decim…

Csvkit is great, and I use it a lot. The target users for CSV Explorer though are mostly non-engineers ie. people who don't use the command line or code.

Kibana has quite a few rough edges but it takes non-technical users very far in terms of data exploration. Local setup also alleviates privacy concerns. The bummer is import/export: 'Upload CSV' feature has been almost introduced in 5.0 release but removed in the end [1], and CSV export has been asked for many times [2] but is still not there. So a user-friendly fork might be a worthy business idea.

[1]: https://github.com/elastic/kibana/pull/8497 [2]: https://github.com/elastic/kibana/issues/1992

Re: Show HN: CSV Explorer (YC F1) - Explore CSVs with Millions of Rows

#24

Post by jastr »

emodendroket wrote: post #23

Is there any reason this tool couldn't be 100% client-side?

Perhaps - I tried a client-side implementation, but I had issues scaling past a few hundred thousand rows. I understand hesitations of uploading data to the cloud, but I now have users looking at hundreds of millions of rows in seconds thanks to Postgres!

Re: Show HN: CSV Explorer (YC F1) - Explore CSVs with Millions of Rows

#25

Post by Zaheer »

jastr wrote: post #24
emodendroket wrote: post #23

Is there any reason this tool couldn't be 100% client-side?

Perhaps - I tried a client-side implementation, but I had issues scaling past a few hundred thousand rows. I understand hesitations of uploading data to the cloud, but I now have users looking at hundreds of millions of rows in seconds thanks to Postgres!

You can run PostGres natively as well and do the same thing no?

Re: Show HN: CSV Explorer (YC F1) - Explore CSVs with Millions of Rows

#26

Post by emodendroket »

Zaheer wrote: post #25
jastr wrote: post #24

Earlier quoted context omitted.

Perhaps - I tried a client-side implementation, but I had issues scaling past a few hundred thousand rows. I understand hesitations of uploading data to the cloud, but I now have users looking at hundreds of millions of rows in seconds thanks to Postgres!

You can run PostGres natively as well and do the same thing no?

Not really an all-in-browser solution then.

Re: Show HN: CSV Explorer (YC F1) - Explore CSVs with Millions of Rows

#28

Post by donquichotte »

a3n wrote: post #20
xaybey wrote: post #6

I usually use csvkit ( https://csvkit.readthedocs.io/en/1.0.1/index.html ). There are commands to list the columns, filter, browse the data in a somewhat formatted way with less. Typing the commands is a huge pain though, and I would be very interested in a tool that could instantly pop open and let me peruse. Excel can take minutes to load and eagerly does a lot of unhelpful formatting on things like dates and decim…

At a former job, our embedded device logs decoded to csv. Some of them were too large for Excel. Pandas handled them without a burp. Pandas in Jupyter (Ipython Notebook) was a godsend. There's a minimal amount of variable setup, but once you've done that once or twice it's easy. Of course, any analysis or manipulation takes a bit of python code, but I see that as a feature, not the least because you can read it right…

Same here. I load the data using pandas or parsing the rows by hand in go. It's interesting to watch your RAM getting filled up while the data is loaded.

Looks like this tool is for non-programmers, it's interesting to see that there seems to be a market here.

Re: Show HN: CSV Explorer (YC F1) - Explore CSVs with Millions of Rows

#29

Post by infinite8s »

xaybey wrote: post #6

I usually use csvkit ( https://csvkit.readthedocs.io/en/1.0.1/index.html ). There are commands to list the columns, filter, browse the data in a somewhat formatted way with less. Typing the commands is a huge pain though, and I would be very interested in a tool that could instantly pop open and let me peruse. Excel can take minutes to load and eagerly does a lot of unhelpful formatting on things like dates and decim…

I've been prototyping simple desktop GUI tools on top of dask/pandas and PyQt that let you lazily load large CSVs (and other types supported by pandas) and interactively filter based on smart histograms (the per column histograms are fully interactive and provide crossfiltering across the attributes):

http://imgur.com/a/vfAmV

The idea is to map a lot of the basic functionality of dataframes onto simple GUI interactions (for example, changing column types, stacking and unstacking columns, pivoting) and couple that with an ipython console for more complicated data manipulation. And then maybe even adding adding Tableau like charting functionality:

http://imgur.com/a/z8d1w

For quick throwaway exploration/analysis. It can easily handle about a million rows just using generic pandas and a bit of memory. There's lots of cool database techniques that can also be used on small local data (for example, compressed bitmaps using EWAHBool for interactive filtering).

Post reply on HN