This tool would be great were it not for me (rightfully) getting fired and possibly sued by any of my clients if I uploaded even a single file. In fact, what kind of business with millions-of-rows kind of files would entrust said datasets to a service that to me seems strangely vague [1] on how the data is secured, or where it is actually going to be physically stored.
Show HN: CSV Explorer (YC F1) - Explore CSVs with Millions of Rows
31–35 of 35 posts
Re: Show HN: CSV Explorer (YC F1) - Explore CSVs with Millions of Rows
#32I usually use csvkit ( https://csvkit.readthedocs.io/en/1.0.1/index.html ). There are commands to list the columns, filter, browse the data in a somewhat formatted way with less. Typing the commands is a huge pain though, and I would be very interested in a tool that could instantly pop open and let me peruse. Excel can take minutes to load and eagerly does a lot of unhelpful formatting on things like dates and decim…
I regularly test applications that generate big CSV reports. As I don't always have influence on the input data (we want to test on real datasets pulled from production servers of our partners), I fall back to defining constraints that must be satisfied. I ensure that they remain satisfied by analysing the csv files with powershell (of all the tools).
I just find the magic of "import-csv bla.csv | where some condition | select some existing or calculated value | group | format-list | out-file output.txt" to be extremely helpful, it's like SQL for the csv files.
And the ability to question live services and parse JSONs, and cross-check with other reports, or merge multiple reports into one... It's indispensable.
Re: Show HN: CSV Explorer (YC F1) - Explore CSVs with Millions of Rows
#33I usually use csvkit ( https://csvkit.readthedocs.io/en/1.0.1/index.html ). There are commands to list the columns, filter, browse the data in a somewhat formatted way with less. Typing the commands is a huge pain though, and I would be very interested in a tool that could instantly pop open and let me peruse. Excel can take minutes to load and eagerly does a lot of unhelpful formatting on things like dates and decim…
I've been prototyping simple desktop GUI tools on top of dask/pandas and PyQt that let you lazily load large CSVs (and other types supported by pandas) and interactively filter based on smart histograms (the per column histograms are fully interactive and provide crossfiltering across the attributes): http://imgur.com/a/vfAmV The idea is to map a lot of the basic functionality of dataframes onto simple GUI interactio…
Do you plan to release something soon? Even just a way to visualize the rows by loading them lazily would be a huge improvement. I personally use pandas but some of my colleagues are not familiar with it, and it pains me when they try to inspect a large dataset by opening it on Excel on our small university-provided desktops instead of spending a couple of minutes writing a few python lines to extract what they need.
Re: Show HN: CSV Explorer (YC F1) - Explore CSVs with Millions of Rows
#34Earlier quoted context omitted.
I've been prototyping simple desktop GUI tools on top of dask/pandas and PyQt that let you lazily load large CSVs (and other types supported by pandas) and interactively filter based on smart histograms (the per column histograms are fully interactive and provide crossfiltering across the attributes): http://imgur.com/a/vfAmV The idea is to map a lot of the basic functionality of dataframes onto simple GUI interactio…
Do you plan to release something soon? Even just a way to visualize the rows by loading them lazily would be a huge improvement. I personally use pandas but some of my colleagues are not familiar with it, and it pains me when they try to inspect a large dataset by opening it on Excel on our small university-provided desktops instead of spending a couple of minutes writing a few python lines to extract what they need.
It sounds like CSV Explorer might work well for them.
Re: Show HN: CSV Explorer (YC F1) - Explore CSVs with Millions of Rows
#35Is there any reason this tool couldn't be 100% client-side?
Perhaps - I tried a client-side implementation, but I had issues scaling past a few hundred thousand rows. I understand hesitations of uploading data to the cloud, but I now have users looking at hundreds of millions of rows in seconds thanks to Postgres!
I suggest exposing a config variable in the js client to set the hostname of the Postgres server. This way you can provide a self hosted version of the backend, and customers can self host it either locally on their machine, or within an internal network. Same benefits of the browser, just need to change config.