Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Dataset Search

toolbox.google.com

Post reply on HN
Print view View original

71–80 of 81 posts

Re: Google Dataset Search

#71

Post by rufuspollock »

heinrichhartman wrote: post #20

The state of data sharing seems to be still quite sad. * Hosting problems. The first link I tried was already broken. * Format problems. Also the presented data is in all kinds of formats, some "data sets" even require me to read data off images: https://www.ceicdata.com/en/indicator/germany/gdp-per-capita And even if it's JSON, this is not particularly great either (Unicode support? Large (64bit) integers?). * Updat…

Yes, state is still mediocre. Back in 2011 i did Chaos Computer Congress talk on "apt-get for Debian of Data". This itself came out of building the original data portal and open data catalog/hosting site in 2006 https://datahub.io/ (originally ckan.net).

The ideas there are now getting realised in Frictionless Data https://frictionlessdata.io/

This an initiative providing a simple way of "packaging" data like software plus an ecosystem of tools including a package manager etc - https://frictionlessdata.io/data-packages/.

Aims to be minimal, easy to adopt etc (e.g. based on CSV). It has got significant traction with integration and adoption into Pandas, OpenRefine etc.

https://datahub.io/ itself is entirely rebuilt around Data Packages and includes a package manager tool "data".

If you're interested to talk more please come chat on http://gitter.im/datahubio/chat

Re: Google Dataset Search

#73

Post by rufuspollock »

danso wrote: post #2

Very nice. Worth keeping in mind prior examples for comparison's sake. My favorites so far: - https://www.opendatanetwork.com -- what I would call the "Google, for Socrata datasets" - https://public.enigma.com/ -- One of the best collections of U.S. federal data, with good taxonomy and lots of useful options for refining a search, such as filtering by dataset size. - https://www.data.gov/ -- Not as useful as what mos…

Interesting. There have been a lot of attempts at "meta data portals" that search across portals. Most of them have struggled.

At Open Knowledge we built a really early one called opendatasearch.org in 2011/2012 - now defunct - and were involved in the first version of the pan EU open data portal. We also had the original https://ckan.net/ (and subsites) which is now https://datahub.io/ and has become much more focused on quality data and data deployment. [Disclosure: I was/am involved in many of these projects]

The challenge, as others have mentioned, is that data quality is very variable and searching for datasets is complicated (think of software as an analogy - searching for good code libraries is a bit of an art).

I imagine Google are trying this out before making datasets another "special type" of search result -- after all you can already search google for datasets. In addition, Google are already Google so including datasets will have a level of comprehensiveness and exposure you struggle with elsewhere (part of the power of monopoly in a sense!).

PS: for those looking for data gov sites https://dataportals.org/ has most of these.

Re: Google Dataset Search

#74

Post by dandare »

heinrichhartman wrote: post #20

The state of data sharing seems to be still quite sad. * Hosting problems. The first link I tried was already broken. * Format problems. Also the presented data is in all kinds of formats, some "data sets" even require me to read data off images: https://www.ceicdata.com/en/indicator/germany/gdp-per-capita And even if it's JSON, this is not particularly great either (Unicode support? Large (64bit) integers?). * Updat…

Shameless plug. Please give me feedback.

https://us.wikibudgets.org/w/united-states-budget-2016

I would like to create laymen oriented central repository for all public spending data of the world.

* Hosting problems - I make my own copy of the data.

* Format problems - clearing and formatting data from different sources is a real pain. Once it is on my website, I offer CSV download or COPY/PASTE tabulated data.

* Update problems - no versioning or public API yet.

* Provenance problems - there is a link to the source of the data.

* Presentation problems - tailored to displaying budgets. Not cross-browser or full mobile support yet.

Re: Google Dataset Search

#76

Post by divbyzer0 »

fnord123 wrote: post #35
wiredfool wrote: post #5

Earlier quoted context omitted.

https://data.gov.ie —- this is a repository of a bunch of the open data produced in the Irish public sector. (Disclosure, I work on this)

Playing around, a lot of countries use this url scheme: https://data.gov.be http://data.gov.ro/ https://www.dati.gov.it/ (note, Italy redirects data. to dati.) https://data.gov.pl/ https://data.gov.za/ (South Africa's has a cert problem) https://data.gov.in https://data.gov.au/ https://datos.gob.mx/

https://data.gov.ie/

Re: Google Dataset Search

#78

Post by MR4D »

Maybe I’m missing something but this strikes me as underwhelming. To the point of something that I could do as opposed to something that the firm that created maps and gmail could do.

Is it just me?

Re: Google Dataset Search

#80

Post by handelaar »

wiredfool wrote: post #42
handelaar wrote: post #29

Earlier quoted context omitted.

I'm a particular fan of the boundary "dataset" you have that's a low-resolution TIFF file. (edit to add something more productive: the site is littered to the tune of at least 25% and maybe even a third with junk "data", all obviously added to get the number of records as high as possible, with no regard to whether that data is either useful to anybody, is machine-readable in any way at all, or even -- in the example…

The quality of the datasets varies greatly depending on the source. Some work well, some, less so. There are data sources that are undergoing active development to harvest them more accurately. None of them were added to pump up the numbers. The biggest numbers bump recently was ca 1600 Met Eireann rainfall records datasets from all around the country, some of them daily rainfall dating back 60 years. (Spoiler, there…

> None of them were added to pump up the numbers.

Then kindly explain these

https://data.gov.ie/advanced_search?query=&texttype=anywhere...

Or these

https://data.gov.ie/advanced_search?query=&texttype=anywhere...

(I'd show only the PDF-only ones but your search doesn't work.)

Oh and look two of these also contain no machine-readable data whatsoever

https://data.gov.ie/advanced_search?query=&texttype=anywhere...

Post reply on HN