Interestingly enough, I built and released something very similar [1] about a month ago using a Google Custom Search Engine.
Here is the Show HN for it: https://news.ycombinator.com/item?id=17789119
Google Dataset Search
21–30 of 81 posts
Re: Google Dataset Search
#22Can we list some good data searches to suggest below this comment?
I would love to see some cool data we might be able to use.
Re: Google Dataset Search
#23Of note is the link below which indicates how you can have your dataset indexed.
https://developers.google.com/search/docs/data-types/dataset
Re: Google Dataset Search
#24Very nice. Worth keeping in mind prior examples for comparison's sake. My favorites so far: - https://www.opendatanetwork.com -- what I would call the "Google, for Socrata datasets" - https://public.enigma.com/ -- One of the best collections of U.S. federal data, with good taxonomy and lots of useful options for refining a search, such as filtering by dataset size. - https://www.data.gov/ -- Not as useful as what mos…
I really like NYC Open Data: https://opendata.cityofnewyork.us/
Re: Google Dataset Search
#25Nice. I'm glad Google is making it easier to find public data sets. I wish that these could be filtered by format, so that you could narrow them to CSV, XML, JSON, KML, etc. Another nice resource that I've used in the past is 'toddmotto/public-apis' on Github [0]. In the end I would prefer all public data sets to be available over the DAT protocol [1] instead of being hosted only on government or organization website…
Yeah, if they had added an attribute for formats available for each data set, and then added a filter by type to the search (e.g. "type:csv") or similar, it would be great.
Sometimes you really want a specific format for a dataset.
Re: Google Dataset Search
#26Nice. I'm glad Google is making it easier to find public data sets. I wish that these could be filtered by format, so that you could narrow them to CSV, XML, JSON, KML, etc. Another nice resource that I've used in the past is 'toddmotto/public-apis' on Github [0]. In the end I would prefer all public data sets to be available over the DAT protocol [1] instead of being hosted only on government or organization website…
Dat's pretty cool, but it's not the only game out there. The efforts of git-annex/Datalad [0], Academic Torrents [1], Quilt[2], DVC[3], and Pachyderm [4] are also notable in this space. My hopes are broader in the sense that I just hope that dataset versioning happens in the first place. [0] https://www.datalad.org/ [1] http://academictorrents.com/ [2] https://quiltdata.com/ [3] https://dvc.org/ [4] http://www.pachyd…
[5] https://qri.io (we're tackling the dataset versioning problem head on.
Re: Google Dataset Search
#27The state of data sharing seems to be still quite sad. * Hosting problems. The first link I tried was already broken. * Format problems. Also the presented data is in all kinds of formats, some "data sets" even require me to read data off images: https://www.ceicdata.com/en/indicator/germany/gdp-per-capita And even if it's JSON, this is not particularly great either (Unicode support? Large (64bit) integers?). * Updat…
Totally agree! At Qri (https://qri.io) we're working on many of these problems together - hosting, formatting (interoperability), provenance and sync. It's an open source project - we'd love to have your feedback as we design it!
Re: Google Dataset Search
#28The state of data sharing seems to be still quite sad. * Hosting problems. The first link I tried was already broken. * Format problems. Also the presented data is in all kinds of formats, some "data sets" even require me to read data off images: https://www.ceicdata.com/en/indicator/germany/gdp-per-capita And even if it's JSON, this is not particularly great either (Unicode support? Large (64bit) integers?). * Updat…
I agree and I have an idea for one possible solution that I've wanted to implement for years. It could be a business but, I think maybe it would be better for the world if it was open source. I just haven't had the time or support to do it, as it would need full-time effort even to get it started.
It's one of those... I know I should just do it kind of things, even just to get it out there, but I haven't found the inertia.
Seeing things like dat, quiltdata, public data sets, etc. made me think what I wanted to do was unnecessary, but I also agree with your comment.
I think a core problem is data democracy / control / politics of data. Too much we still act silo'ed instead of benefiting from massive data sharing, for a multitude of reasons (especially but not limited to $$$).
Re: Google Dataset Search
#29Very nice. Worth keeping in mind prior examples for comparison's sake. My favorites so far: - https://www.opendatanetwork.com -- what I would call the "Google, for Socrata datasets" - https://public.enigma.com/ -- One of the best collections of U.S. federal data, with good taxonomy and lots of useful options for refining a search, such as filtering by dataset size. - https://www.data.gov/ -- Not as useful as what mos…
https://data.gov.ie —- this is a repository of a bunch of the open data produced in the Irish public sector. (Disclosure, I work on this)
I'm a particular fan of the boundary "dataset" you have that's a low-resolution TIFF file.
(edit to add something more productive: the site is littered to the tune of at least 25% and maybe even a third with junk "data", all obviously added to get the number of records as high as possible, with no regard to whether that data is either useful to anybody, is machine-readable in any way at all, or even -- in the example above -- even qualifies as "data". Data.gov.ie would be moderately interesting if all the shit in it was removed.)
Re: Google Dataset Search
#30The state of data sharing seems to be still quite sad. * Hosting problems. The first link I tried was already broken. * Format problems. Also the presented data is in all kinds of formats, some "data sets" even require me to read data off images: https://www.ceicdata.com/en/indicator/germany/gdp-per-capita And even if it's JSON, this is not particularly great either (Unicode support? Large (64bit) integers?). * Updat…
Totally agree! At Qri ( https://qri.io ) we're working on many of these problems together - hosting, formatting (interoperability), provenance and sync. It's an open source project - we'd love to have your feedback as we design it!
This is cool and is a perfect case for IPFS for public datasets. I've not heard of it before, though, and I think naming / branding is something that makes finding these things / building momentum more difficult.
For example, someone else mentioned engima.com. I would have no idea that is related to data sources / sets unless I knew what it was.
Certainly wish you the best of luck though and will keep an eye on Qri! Cool project!