I have a lot to read before I get excited but if the team is here: Can we get DCAT for sets that are otherwise only discoverable with OAI-PMH? Seems like a divide between govt and academic repos that hinders harvesting.
Google Dataset Search
11–20 of 44 posts
Re: Google Dataset Search
#12Do they have a deprecation notice up already?
Re: Google Dataset Search
#13Also in case the team is here... the updated date for ERA5 back extension to 1950-1978 (Preliminary version - https://datasetsearch.research.google.com/search?query=ERA5%...) is incorrect as this was only released last year (2020) but is stated as 2011.
Re: Google Dataset Search
#14This is a great resource. At Splitgraph, we index ~40k open data sets, and we make sure to include structured metadata for each one, so we show up in these results. (example [0])
One cool aspect of this metadata is that it allows a dataset to have multiple sources. So if two sites index the same dataset, there is no duplicate content penalty like there might be with textual content. If you search for a dataset, it will include links to all its sources (whether canonical or otherwise).
For most of the data we index at Splitgraph, the canonical source is an open government data portal powered by Socrata (e.g. data.cdc.gov). We noticed that Socrata powered a lot of portals, so we wrote a Socrata plugin for Splitgraph, along with a scraper to index the metadata. The plugin basically implements a Postgres FDW so that Splitgraph can translate from SQL to the upstream query language. In this case, the plugin translates to Socrata's bespoke API language. But for private deployments we also have plugins for Snowflake, Postgres, some SaaS services, etc.
If you find some data on Google Dataset Search with Splitgraph listed as a source, please take a look! Our "Data Delivery Network" (DDN) is implemented on top of the Postgres wire protocol, so you can connect with any Postgres client (or use our web editor). All the Postgres query syntax is available to you; you can even JOIN across any of the other 40k+ datasets indexed at Splitgraph. That includes "live data" like Socrata portals, but also versioned snapshots of data called "data images." Here's an example of a point-in-time query across two snapshots (basically a diff) [1], and another query that joins across tables at data.cityofchicago.org and data.cambridgema.gov [2].
[0] https://www.splitgraph.com/cdc-gov/distribution-of-covid19-d... – "View Source" to see the Schema.org metadata
(Sorry for the bit.ly links. The URL for our query editor includes the full SQL string, and I don't want to mess up HN formatting.)
Re: Google Dataset Search
#15This dataset search engine has been around for years! We created DataMarket ( https://datamarket.es ) inspired by this site (and Auren Hoffman's SafeGraph).
Absolutely. This dataset search was first introduced in September of 2018. It was out of beta in January last year: https://www.kdnuggets.com/2020/01/google-dataset-search.html.
Re: Google Dataset Search
#16This is every data scientists’ dream.
It’s ok, but surprisingly feature poor since they only index datasets with structured metadata. I kind of wish they would compile all their metadata into a structured mega-catalog and allowed searching by api. Or just dumped it out as a dataset itself.
Then you'll love what we're doing at Splitgraph: https://www.splitgraph.com/connect
As far as your SQL client is concerned, data.splitgraph.com:5432 is a giant Postgres database with ~40,000 tables in it. You can query and join across them using your existing tools. Behind the curtain, we'll forward your query to the upstream data source, translating it from SQL to whatever language it expects. (We can also ingest delta-compressed versioned snapshots).
Re: Google Dataset Search
#17Do they have a deprecation notice up already?
Everyone's hyped but Vanilla Google is even better in term of design: Google.com is more readable, just like Hackernews.
Re: Google Dataset Search
#18Dataset with 9,000 annotated cat images! => https://datasetsearch.research.google.com/search?query=cat&d...
Re: Google Dataset Search
#19This is every data scientists’ dream.
This is not new, though. So it may be a dream in the sense of people have been asleep?
Why the condescension? Do you mean that Google has been offering this service for a while? Or do you mean that similar services have previously been offered by other organizations? In which case, perhaps you could link to them?
Re: Google Dataset Search
#20Stop, you're making the barrier to entry too low! /s This is really really cool. Between this and Hugginfaces Dataset and models hubs, AI/ML is really getting easier to use.
I've actually been on the lookout for model hubs lately. Any that you've seen or reccomend?
I've found https://modelzoo.co/ but it seems more like a currated list of models (some incomplete) rather than a community where users share trained models.