I’ve come across this a few different times over the years... always seems enticing and potentially useful, but I’ve never found a real use for it. I suppose it provides a library of well-prepped datasets to test ML models on? Anyone ever used this for any practical purpose beyond a sandbox-type use case?
Google Dataset Search
31–40 of 44 posts
Re: Google Dataset Search
#32I'm not a scientist, so not the most scholarly first lookup, but tried searching for penis data[0]. The first link sent me to a site that requires signup to use [1]. No fun. Won't use again. [0] - https://datasetsearch.research.google.com/search?query=penis... [1] - https://data.world/jemus42/world-penis-data
I'll keep using it because the inconvenience of the occasional sign-up is trumped by the convenience of searchable datasets.
Re: Google Dataset Search
#33Earlier quoted context omitted.
Then you'll love what we're doing at Splitgraph: https://www.splitgraph.com/connect As far as your SQL client is concerned, data.splitgraph.com:5432 is a giant Postgres database with ~40,000 tables in it. You can query and join across them using your existing tools. Behind the curtain, we'll forward your query to the upstream data source, translating it from SQL to whatever language it expects. (We can also ingest de…
What are the largest datasets in Splitgraph? Can I list the datasets sorted by size? We have the need for large public datasets for testing ClickHouse: https://clickhouse.tech/docs/en/getting-started/example-data...
On the public DDN (data.splitgraph.com:5432), we enforce a (currently arbitrary) 10k row limit on responses. You can construct multiple queries using LIMIT and OFFSET, or you can run a local Splitgraph engine without a limit. We also have a private beta program if you want a managed or self-hosted cloud deployment with the full catalog and DDN features. And we are planning to ship some "export to..." type workflows for exporting to CSV and potentially other formats.
For live/external data, we proxy the query to the data source, so there is no theoretical data size limit except for any defined by the upstream.
For snapshotted data, we store the data as fragments in object storage. Any size limit depends on the machine where Splitgraph's Postgres engine is running, and how you choose to materialize the data when downloading it from object storage. You can "check out" an entire image to materialize it locally, at which point it will be like any other Postgres schema. Or you can use "layered querying" which will return a result set while only materializing the fragments necessary to answer the query.
Regarding ClickHouse, you could watch this presentation [0] my co-founder Artjoms gave at a recent ClickHouse meet-up on the topic of your question. We also have specific documentation for using the ClickHouse ODBC client with the DDN [1], as well as an example reference implementation. [2]
[0] https://www.youtube.com/watch?v=44CDs7hJTho
[1] https://www.splitgraph.com/connect
[2] https://github.com/splitgraph/splitgraph/tree/master/example...
Re: Google Dataset Search
#34The privacy problem should be considered
Re: Google Dataset Search
#35The privacy problem should be considered
Can you explain what you mean by "the privacy problem"?
Re: Google Dataset Search
#36Discussion from Jan 2020: https://news.ycombinator.com/item?id=22130874 | 32 comments
Discussion from Sept 2018: https://news.ycombinator.com/item?id=17919297 | 76 comments
Re: Google Dataset Search
#37They should add a "I'm Feeling lucky" button
Re: Google Dataset Search
#38Information on how to annotate datasets: https://developers.google.com/search/docs/data-types/dataset > We can understand structured data in Web pages about datasets, using either schema.org Dataset markup, or equivalent structures represented in W3C's Data Catalog Vocabulary (DCAT) format. We also are exploring experimental support for structured data based on W3C CSVW, and expect to evolve and adapt our approach as…
It’s funny because Google does not use these standards to validate. I keep getting errors from Google that some of my dataset’s descriptions are over 5,000 characters even though dcat:description does not have a size limit. Of course it’s impossible for me to report a bug in how they index.
You could submit a dataset containing the bug report :-)
Re: Google Dataset Search
#39The privacy problem should be considered
Privacy is the liability of those who collect data and share it in a form of dataset not of those who search for datasets or who crawl them.
Re: Google Dataset Search
#40The lab I work in has a project that helps annotate datasets with metadata and register their schemas: https://discovery.biothings.io/
A common barrier to making FAIR datasets is that not all data lends itself to be schema.org compliant. The idea is that instead of enforcing one schema to rule them all, we allow people to make their own schemas by extending existing ones, and register them in an API to be easily discoverable.