11–20 of 35 posts
Hi HN! A few months ago, a college friend reached out because his consulting company had just received a 60 Gb spreadsheet, and they didn't know what to do with it! They actually tried opening it in Excel. I'm excited to Show HN CSV Explorer - a simple web tool for opening really big CSVs! Try it out, and let me know how it goes!
My solution to this problem, so far, has been ipython and pandas. (and maybe jupyter notebook for visualization and sharing). What's the value-add here?
Hi HN! A few months ago, a college friend reached out because his consulting company had just received a 60 Gb spreadsheet, and they didn't know what to do with it! They actually tried opening it in Excel. I'm excited to Show HN CSV Explorer - a simple web tool for opening really big CSVs! Try it out, and let me know how it goes!
Open it with a programming language like R, Julia or Python that supports the data frames. No reason to upload 60 gigs to the web or Excel.
This is really awesome. What sparked this idea? Not sure if you have this feature, but I think it would be really cool if you could open APIs to your own datasets. This could be really useful for enterprise applications that do a lot of flat file imports/exports to push/pull data.
Cool idea - Having to work with sensitive data, it'd be great to have this functionality without importing/uploading to the internet.
I usually use csvkit ( https://csvkit.readthedocs.io/en/1.0.1/index.html ). There are commands to list the columns, filter, browse the data in a somewhat formatted way with less. Typing the commands is a huge pain though, and I would be very interested in a tool that could instantly pop open and let me peruse. Excel can take minutes to load and eagerly does a lot of unhelpful formatting on things like dates and decim…
That should give you a 10k-line random sample to play with that should open quite quickly in Excel.
This is really awesome. What sparked this idea? Not sure if you have this feature, but I think it would be really cool if you could open APIs to your own datasets. This could be really useful for enterprise applications that do a lot of flat file imports/exports to push/pull data.
Some consultant friends asked for help when they got a 60Gb CSV file from their client! Since then, I've also found that lots of tech companies share CSV internally. I've also worked with journalists attempting to look at large public government datasets.
Cool idea - Having to work with sensitive data, it'd be great to have this functionality without importing/uploading to the internet.
Thanks - having worked with health data for a few years I can relate. Unfortunately, I don't have plans for a desktop app right now.
I usually use csvkit ( https://csvkit.readthedocs.io/en/1.0.1/index.html ). There are commands to list the columns, filter, browse the data in a somewhat formatted way with less. Typing the commands is a huge pain though, and I would be very interested in a tool that could instantly pop open and let me peruse. Excel can take minutes to load and eagerly does a lot of unhelpful formatting on things like dates and decim…
Pandas handled them without a burp. Pandas in Jupyter (Ipython Notebook) was a godsend.
There's a minimal amount of variable setup, but once you've done that once or twice it's easy.
Of course, any analysis or manipulation takes a bit of python code, but I see that as a feature, not the least because you can read it right there in the open instead of having to hunt for formulas in cells.