Visualize any public CSV on GitHub in a few clicks

#1

Post by glaugh » Tue, Apr 29, 2014, 5:43 PM UTC

Visualize any public CSV on GitHub in a few clicks
blog.statwing.com

Re: Visualize any public CSV on GitHub in a few clicks

#2

Post by caio1982 » Tue, Apr 29, 2014, 6:42 PM UTC

Couldn't quickly find the limitations when parsing these .csv files. How many lines in them would be still ok?

Re: Visualize any public CSV on GitHub in a few clicks

#3

Post by glaugh » Tue, Apr 29, 2014, 6:44 PM UTC

caio1982 wrote: post #2

Couldn't quickly find the limitations when parsing these .csv files. How many lines in them would be still ok?

Things will definitely start slowing down pretty linearly after 100k lines, but we often see millions, and most files shouldn't break us as long as they're not over ~500MB.

Edit: Fleshed out explanation

Re: Visualize any public CSV on GitHub in a few clicks

#4

Post by sureshv » Tue, Apr 29, 2014, 6:55 PM UTC

Looks like you handle CSV's and non-Github links to datasets. You should add transparent handling of gz/zip files as well as xls. This would be useful for poking around government datasets.

Re: Visualize any public CSV on GitHub in a few clicks

#5

Post by glaugh » Tue, Apr 29, 2014, 6:59 PM UTC

sureshv wrote: post #4

Looks like you handle CSV's and non-Github links to datasets. You should add transparent handling of gz/zip files as well as xls. This would be useful for poking around government datasets.

Yeah, agreed. We do actually handle xls files in our main product (this is sort of a demo of an API connection). Probably should have enabled that for this little implementation.

Can't yet take gz/zip files, probably should though.

Thanks/cheers

Re: Visualize any public CSV on GitHub in a few clicks

#6

Post by toomuchtodo » Tue, Apr 29, 2014, 7:39 PM UTC

glaugh wrote: post #3

caio1982 wrote: post #2

Couldn't quickly find the limitations when parsing these .csv files. How many lines in them would be still ok?

Things will definitely start slowing down pretty linearly after 100k lines, but we often see millions, and most files shouldn't break us as long as they're not over ~500MB. Edit: Fleshed out explanation

Do you try to check a resource with an http head request to ensure its under 500MB before ingesting?

Re: Visualize any public CSV on GitHub in a few clicks

#7

Post by glaugh » Tue, Apr 29, 2014, 7:54 PM UTC

toomuchtodo wrote: post #6

glaugh wrote: post #3

Earlier quoted context omitted.

Things will definitely start slowing down pretty linearly after 100k lines, but we often see millions, and most files shouldn't break us as long as they're not over ~500MB. Edit: Fleshed out explanation

Do you try to check a resource with an http head request to ensure its under 500MB before ingesting?

Nope. Folks who sign up for the API generally have some awareness of what size files work and what don't. And since they're uploading for their users, we're aligned around wanting those users to have a good experience.

We haven't worried about it in this particular implementation around the API because we didn't run across many raw github files that were big. And even when we do get the odd big one, we just refuse to process it once we receive it, so it doesn't hurt us much if someone sends us a few GBs.