Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Dataset Search

toolbox.google.com

Post reply on HN
Print view View original

11–20 of 81 posts

Re: Google Dataset Search

#11

Post by philipashlock »

danso wrote: post #2

Very nice. Worth keeping in mind prior examples for comparison's sake. My favorites so far: - https://www.opendatanetwork.com -- what I would call the "Google, for Socrata datasets" - https://public.enigma.com/ -- One of the best collections of U.S. federal data, with good taxonomy and lots of useful options for refining a search, such as filtering by dataset size. - https://www.data.gov/ -- Not as useful as what mos…

@danso thanks for the feedback on data.gov. I'm part of the small (3 person) team that helps to manage it. If you have a moment to chat I'll see if I can reach out to you to see if you'd be interested in participating in some more in-depth user research in the future. Folks can also always leave feedback via email, github, twitter, and other means - https://www.data.gov/contact

The Federal Data Strategy will also be opening up for comments again in October - https://strategy.data.gov/feedback/

Data.gov and Federal agencies use the same metadata standard (DCAT) that Google Dataset Search is using so much of our metadata is also being syndicated there.

Re: Google Dataset Search

#12

Post by neuromantik8086 »

Dangeranger wrote: post #7

Nice. I'm glad Google is making it easier to find public data sets. I wish that these could be filtered by format, so that you could narrow them to CSV, XML, JSON, KML, etc. Another nice resource that I've used in the past is 'toddmotto/public-apis' on Github [0]. In the end I would prefer all public data sets to be available over the DAT protocol [1] instead of being hosted only on government or organization website…

Dat's pretty cool, but it's not the only game out there. The efforts of git-annex/Datalad [0], Academic Torrents [1], Quilt[2], DVC[3], and Pachyderm [4] are also notable in this space. My hopes are broader in the sense that I just hope that dataset versioning happens in the first place.

[0] https://www.datalad.org/

[1] http://academictorrents.com/

[2] https://quiltdata.com/

[3] https://dvc.org/

[4] http://www.pachyderm.io/

Re: Google Dataset Search

#13

Post by neuromantik8086 »

Looks like academic institutional repositories and figshare are doing the heavy lifting here. It's still neat to see Google aggregate everything, but it's not that different from what they do with other services relying on these sources already, and is largely dependent upon how rich these upstream sources are in the first place.

Re: Google Dataset Search

#16

Post by MrEfficiency »

What is the best way for a website to format data to the public?

I already have my presentation, but I can also provide it as a .xls, .csv, sql, or html table.

What would be best to help programmers/data scientists use my data?

Re: Google Dataset Search

#17

Post by polkapolka »

MrEfficiency wrote: post #16

What is the best way for a website to format data to the public? I already have my presentation, but I can also provide it as a .xls, .csv, sql, or html table. What would be best to help programmers/data scientists use my data?

Csv and a permitive license like CC0. Provide good meta data to make provenance easier. Mark up your dataset with schema.org.

Re: Google Dataset Search

#20

Post by heinrichhartman »

The state of data sharing seems to be still quite sad.

* Hosting problems. The first link I tried was already broken.

* Format problems. Also the presented data is in all kinds of formats, some "data sets" even require me to read data off images: https://www.ceicdata.com/en/indicator/germany/gdp-per-capita And even if it's JSON, this is not particularly great either (Unicode support? Large (64bit) integers?).

* Update problems. Many data-sets change over time (e.g. GDP). How can I subscribe to updates? "git pull" would be nice.

* Provenance problems. I want to know who put which record into the dataset, when and why? "git log" would be nice.

* Presentation problems. (This is OK sometimes) I necessarily want to download 5Gb file before I looked into it. The first few rows of the dataset should be presented on the page, with information about it.

Wrote down a few more thoughts a while ago here: https://github.com/HeinrichHartmann/data-sharing#in-the-idea...

Approaches I have seen so far in the wild:

* figshare.com -- Addresses Hosting and Presentation.

* https://quiltdata.com/ -- (!) looks great. Still exploring.

* github.com -- works fine for small datasets (* packaging (yum, pkg, pip) -- (?) Not sure if that works, but at least they solve: Hosting, Update, Provenance.

This seems to be a wide open problem to me.

Post reply on HN