CSV as a Data Source

#1

Post by thingsilearned » Thu, Aug 01, 2013, 5:25 PM UTC

Re: CSV as a Data Source

#2

Post by bdon » Thu, Aug 01, 2013, 5:35 PM UTC

The type/error detection would be really cool paired with something like DataWrangler: http://vis.stanford.edu/wrangler/

Re: CSV as a Data Source

#3

Post by michaelrbock » Thu, Aug 01, 2013, 5:44 PM UTC

As someone who has worked in an industry where FTP'ing massive CSV files was the norm, I must say this looks really nice.

Re: CSV as a Data Source

#4

Post by thingsilearned » Thu, Aug 01, 2013, 5:53 PM UTC

michaelrbock wrote: post #3

As someone who has worked in an industry where FTP'ing massive CSV files was the norm, I must say this looks really nice.

Thanks!

Re: CSV as a Data Source

#5

Post by thingsilearned » Thu, Aug 01, 2013, 5:55 PM UTC

bdon wrote: post #2

The type/error detection would be really cool paired with something like DataWrangler: http://vis.stanford.edu/wrangler/

Yeah, DataWrangler is an awesome project. There are actually a few great companies coming out with products based around it. We may just integrate with one of those.

Re: CSV as a Data Source

#6

Post by netvarun » Thu, Aug 01, 2013, 6:08 PM UTC

This is really cool! Are there any open source libraries to do steps 2 and 3 - auto type detection and error correction/detection?

Re: CSV as a Data Source

#7

Post by tbrownaw » Thu, Aug 01, 2013, 6:24 PM UTC

The biggest issue I have with csv files, is when the people sending them don't get the $%^&@ quotes right. The second biggest issue, is when they say "sorry, we're not able to do that" after I tell them exactly how to fix what they're sending.

Re: CSV as a Data Source

#8

Post by jeremysmyth » Thu, Aug 01, 2013, 6:26 PM UTC

MySQL has supported CSV as a data source for quite some time: http://dev.mysql.com/doc/refman/5.6/en/csv-storage-engine.ht...

You can run full SQL queries directly against a text file as if it was a table.

Re: CSV as a Data Source

#9

Post by caissy » Thu, Aug 01, 2013, 6:30 PM UTC

That is really nice. I remember implementing something similar to step 2 and 3 on AppEngine a few years back for a project. It turned out quite nice and non tech-savvy users used it without much problems. We managed to have something that could easily work with different CSV formats (MS Excel and others).

Re: CSV as a Data Source

#10

Post by akinder » Thu, Aug 01, 2013, 6:45 PM UTC

tbrownaw wrote: post #7

The biggest issue I have with csv files, is when the people sending them don't get the $%^&@ quotes right . The second biggest issue, is when they say "sorry, we're not able to do that" after I tell them exactly how to fix what they're sending.

What is the proper way to encode the quotes? I'm playing around with writing a Livejournal data dump reader and will have to do the quote dance pretty soon. In LJ files they are encoded like so:

col1,col2,col3,"Longer data and somethin ""with"" quotes",col5