Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Dataset Search

toolbox.google.com

Post reply on HN
Print view View original

31–40 of 81 posts

Re: Google Dataset Search

#31

Post by heinrichhartman »

rgardaphe wrote: post #27
heinrichhartman wrote: post #20

The state of data sharing seems to be still quite sad. * Hosting problems. The first link I tried was already broken. * Format problems. Also the presented data is in all kinds of formats, some "data sets" even require me to read data off images: https://www.ceicdata.com/en/indicator/germany/gdp-per-capita And even if it's JSON, this is not particularly great either (Unicode support? Large (64bit) integers?). * Updat…

Totally agree! At Qri ( https://qri.io ) we're working on many of these problems together - hosting, formatting (interoperability), provenance and sync. It's an open source project - we'd love to have your feedback as we design it!

That looks interesting. IPFS backed -- nice.

A web interface would be nice so that I don't have to install a tool to browse the content. Going to give it a shot non-theless.

Re: Google Dataset Search

#32

Post by heinrichhartman »

heinrichhartman wrote: post #31
rgardaphe wrote: post #27

Earlier quoted context omitted.

Totally agree! At Qri ( https://qri.io ) we're working on many of these problems together - hosting, formatting (interoperability), provenance and sync. It's an open source project - we'd love to have your feedback as we design it!

That looks interesting. IPFS backed -- nice. A web interface would be nice so that I don't have to install a tool to browse the content. Going to give it a shot non-theless.

Hmm: "Qri backend is unavailable". Going to come back tomorrow.

Re: Google Dataset Search

#33

Post by combatentropy »

MrEfficiency wrote: post #16

What is the best way for a website to format data to the public? I already have my presentation, but I can also provide it as a .xls, .csv, sql, or html table. What would be best to help programmers/data scientists use my data?

I vote CSV.

- simple

- lightweight

- open

It's easy for the consumer of your data to convert a CSV to whatever format they need.

- spreadsheet, for personal analysis

- SQL database, for industrial-strength analysis

- HTML, for pretty output to their users

Re: Google Dataset Search

#34

Post by eruci »

wiredfool wrote: post #5
danso wrote: post #2

Very nice. Worth keeping in mind prior examples for comparison's sake. My favorites so far: - https://www.opendatanetwork.com -- what I would call the "Google, for Socrata datasets" - https://public.enigma.com/ -- One of the best collections of U.S. federal data, with good taxonomy and lots of useful options for refining a search, such as filtering by dataset size. - https://www.data.gov/ -- Not as useful as what mos…

https://data.gov.ie —- this is a repository of a bunch of the open data produced in the Irish public sector. (Disclosure, I work on this)

Are you going to release any street address datasets at some point?

Re: Google Dataset Search

#35

Post by fnord123 »

wiredfool wrote: post #5
danso wrote: post #2

Very nice. Worth keeping in mind prior examples for comparison's sake. My favorites so far: - https://www.opendatanetwork.com -- what I would call the "Google, for Socrata datasets" - https://public.enigma.com/ -- One of the best collections of U.S. federal data, with good taxonomy and lots of useful options for refining a search, such as filtering by dataset size. - https://www.data.gov/ -- Not as useful as what mos…

https://data.gov.ie —- this is a repository of a bunch of the open data produced in the Irish public sector. (Disclosure, I work on this)

Playing around, a lot of countries use this url scheme:

https://data.gov.be

http://data.gov.ro/

https://www.dati.gov.it/ (note, Italy redirects data. to dati.)

https://data.gov.pl/

https://data.gov.za/ (South Africa's has a cert problem)

https://data.gov.in

https://data.gov.au/

https://datos.gob.mx/

Re: Google Dataset Search

#36

Post by heinrichhartman »

MrEfficiency wrote: post #16

What is the best way for a website to format data to the public? I already have my presentation, but I can also provide it as a .xls, .csv, sql, or html table. What would be best to help programmers/data scientists use my data?

1) Put it on figshare or github. Your website might be dead next year. Those services have a better chance of survival.

2) CSV, JSON works fine if your dataset is just a few numbers and strings. GitHub will preview csv files as HTML tables.

3) If you need the efficiency of binary data and more robust data containers I would look into - Parquet https://parquet.apache.org/documentation/latest/ and - Arvo http://avro.apache.org/ http://blog.cloudera.com/blog/2009/11/avro-a-new-format-for-...

4) DataScientists work with R/Pandas "DataFrames". If you are familiar with either one, import the data into a data frame and use an export method to do the serialization for you: https://pandas.pydata.org/pandas-docs/stable/api.html#id12

Re: Google Dataset Search

#37

Post by thebooglebooski »

philipashlock wrote: post #11
danso wrote: post #2

Very nice. Worth keeping in mind prior examples for comparison's sake. My favorites so far: - https://www.opendatanetwork.com -- what I would call the "Google, for Socrata datasets" - https://public.enigma.com/ -- One of the best collections of U.S. federal data, with good taxonomy and lots of useful options for refining a search, such as filtering by dataset size. - https://www.data.gov/ -- Not as useful as what mos…

@danso thanks for the feedback on data.gov. I'm part of the small (3 person) team that helps to manage it. If you have a moment to chat I'll see if I can reach out to you to see if you'd be interested in participating in some more in-depth user research in the future. Folks can also always leave feedback via email, github, twitter, and other means - https://www.data.gov/contact The Federal Data Strategy will also be…

I think the biggest blocker with using publicly available datasets is stale data.

If you, or anyone else who aggregates these datasets could make it EASY to find the FREQUENCY of updates, rather than just the LAST UPDATED timestamp, it'd incentivize people to consume APIs more.

I realize having a snapshot from 2014 is better than what was publicly available before. But I feel no one's really talked about why they would or wouldn't use particular data.

Re: Google Dataset Search

#38

Post by sbr464 »

This is nice. I'm working on a similar open source project that is releasing soon called DataLibrary[0]

It goes further by bringing this kind of data together into a single API, converting/cleaning into a similar schema where possible.

A small write up can be found on github [1]. Any feedback/ideas would be appreciated!

[0] https://www.datalibrary.com (not online currently)

[1] https://github.com/reactual/datalibrary/blob/master/README.m...

Re: Google Dataset Search

#39

Post by rgardaphe »

sixdimensional wrote: post #30
rgardaphe wrote: post #27

Earlier quoted context omitted.

Totally agree! At Qri ( https://qri.io ) we're working on many of these problems together - hosting, formatting (interoperability), provenance and sync. It's an open source project - we'd love to have your feedback as we design it!

This is cool and is a perfect case for IPFS for public datasets. I've not heard of it before, though, and I think naming / branding is something that makes finding these things / building momentum more difficult. For example, someone else mentioned engima.com. I would have no idea that is related to data sources / sets unless I knew what it was. Certainly wish you the best of luck though and will keep an eye on Qri!…

Thanks! you can follow us on twitter where we announce most updates: @qri_io

Re: Google Dataset Search

#40

Post by matt4077 »

sixdimensional wrote: post #25
Dangeranger wrote: post #7

Nice. I'm glad Google is making it easier to find public data sets. I wish that these could be filtered by format, so that you could narrow them to CSV, XML, JSON, KML, etc. Another nice resource that I've used in the past is 'toddmotto/public-apis' on Github [0]. In the end I would prefer all public data sets to be available over the DAT protocol [1] instead of being hosted only on government or organization website…

Yeah, if they had added an attribute for formats available for each data set, and then added a filter by type to the search (e.g. "type:csv") or similar, it would be great. Sometimes you really want a specific format for a dataset.

Having dabbled a bit in open data, I think "this data is available in too many formats and going through all the options manually is tiresome" counts as a problem you'd love to have.
Post reply on HN