Tad, a tabular data viewer • Hacker News Board

antonycourtney wrote: post #9

fjert wrote: post #8

Wow, this seems simple but I got excited immediately when seeing this. Since I have been writing code to export data for clients, I have grown so tired of Kingsoft Spreadsheets and Google Sheets lagging like crazy with any sizable amounts of data. This will be a cool new tool to show my coworkers tomorrow and I'll be using it. Performance seems very snappy so far!

I invested some effort to keep it performant even with fairly large CSV files, including a custom port of some C++ code for fast CSV import. My current favorite example is the Met Museum's 228MB 450k row collection data set; takes about 12 sec. to open in Tad on my 2013 MacBook Pro. Definitely not lag free (and hard to achieve that without going to some serious column store data warehouse like Amazon Redshift), but s…

Thanks for putting this out there!

There are some projects out there using memory mapped files to do fast CSV parsing. Could be a nice way to speed up the memory loading and scroll it in real time. Can't find the link to the library I saw it used in, but it might be an interesting venue to consider. Another library that does it seems to be astropy fast ascii IO module [1].

[1]: http://docs.astropy.org/en/stable/io/ascii/fast_ascii_io.htm...

Re: Tad, a tabular data viewer

#24

Post by beefsack » Wed, May 31, 2017, 4:09 AM UTC

vortico wrote: post #4

Looks fantastic! I love the combination of SQLite fixed schema with the hierarchical view. Some advice on the website: Do some browser sniffing (I know) to display a screenshot of the software on the user's operating system. This immediately answers the question "Does the software work on my computer?" Also, the source code link should be near the Download section. Not everyone is trained that the triangle GitHub ico…

The screenshot suggestion is a good one, whenever I open a GUI software page and see a screenshot of Windows or MacOS I often just close the tab and move on.

Re: Tad, a tabular data viewer

#25

Post by rabidrat » Wed, May 31, 2017, 5:02 AM UTC

If you prefer working from the terminal, you should check out VisiData (https://github.com/saulpw/visidata). It's a curses tabular data tool that can start browsing terabyte .csv files immediately, with few dependencies and no clicking.

Re: Tad, a tabular data viewer

#26

Post by nerdponx » Wed, May 31, 2017, 5:57 AM UTC

antonycourtney wrote: post #9

fjert wrote: post #8

Wow, this seems simple but I got excited immediately when seeing this. Since I have been writing code to export data for clients, I have grown so tired of Kingsoft Spreadsheets and Google Sheets lagging like crazy with any sizable amounts of data. This will be a cool new tool to show my coworkers tomorrow and I'll be using it. Performance seems very snappy so far!

I invested some effort to keep it performant even with fairly large CSV files, including a custom port of some C++ code for fast CSV import. My current favorite example is the Met Museum's 228MB 450k row collection data set; takes about 12 sec. to open in Tad on my 2013 MacBook Pro. Definitely not lag free (and hard to achieve that without going to some serious column store data warehouse like Amazon Redshift), but s…

Are you familiar with the R package data.table? Its CSV parser is blazing fast. Pandas (the Python tabular data library) also implements a speedy CSV parser. Both are written in C under the hood.

Re: Tad, a tabular data viewer

#27

Post by gravelc » Wed, May 31, 2017, 6:45 AM UTC

Just gave it a quick test run on a 6 million row by 8 column CSV my software produces. Fairly slow to open on an iMac, but got there in the end. Looks nice. Sorting fast. Have got an additional column called 'Rec' for some reason.

Re: Tad, a tabular data viewer

#28

Post by MrMattWright » Wed, May 31, 2017, 6:53 AM UTC

This is absolutely terrific! I often work with csv or tsv and large data, and I just want to quickly look at it without loading excel or exporting to google sheets. https://atom.io/packages/tablr Tablr for Atom is good too. One feature request. I very often get data in JSON format (then convert to csv), it would be amazing to also support JSON. Thanks for sharing.

Re: Tad, a tabular data viewer

#29

Post by Cogito » Wed, May 31, 2017, 6:55 AM UTC

I couldn't see any way to export data, once I had filtered or pivoted it.

Is this supported at all, or do I need to use the copy mechanism? I have a ~4GB file to filter down and analyse, and this almost looks like a nice tool for business users to use to explore the data themselves, but they need to be able to export to excel at some point.

Re: Tad, a tabular data viewer

#30

Post by wx196 » Wed, May 31, 2017, 6:56 AM UTC

fjert wrote: post #8

Wow, this seems simple but I got excited immediately when seeing this. Since I have been writing code to export data for clients, I have grown so tired of Kingsoft Spreadsheets and Google Sheets lagging like crazy with any sizable amounts of data. This will be a cool new tool to show my coworkers tomorrow and I'll be using it. Performance seems very snappy so far!

You should definitely try Delimit (http://www.delimitware.com), I was impressed how easy I could handle >1 mil rows with it.