Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

City of Boulder Open Data

open-data.bouldercolorado.gov

Post reply on HN
Print view View original

51–60 of 67 posts

Re: City of Boulder Open Data

#51

Post by didgetmaster »

chaps wrote: post #35

Just be VERY careful when using any of these open data sets. Datasets will undoubtedly be filtered by some unwritten criteria. It's best to use these datasets as a sanity check for some things, with follow-up FOIA requests for the serious work. And even that will be filled with similar filtering. Source: spent the last ten years working with open data and FOIA in Chicago. Also, here's a tweet-fitting one liner I wrot…

It's not just the filtering. It is also that many datasets are full of errors. I wrote a data analytics tool that I would test out on a bunch of open data sets like this one. One of my favorites was the Chicago crime data. The tool could quickly find (and fix) a bunch of errors. I tried contacting the people who published the crime data and even offered to show them how to use my free tool to clean up the data. They could not have cared less that their data set had large numbers of errors in it.

Re: City of Boulder Open Data

#52

Post by zlurker »

didgetmaster wrote: post #51
chaps wrote: post #35

Just be VERY careful when using any of these open data sets. Datasets will undoubtedly be filtered by some unwritten criteria. It's best to use these datasets as a sanity check for some things, with follow-up FOIA requests for the serious work. And even that will be filled with similar filtering. Source: spent the last ten years working with open data and FOIA in Chicago. Also, here's a tweet-fitting one liner I wrot…

It's not just the filtering. It is also that many datasets are full of errors. I wrote a data analytics tool that I would test out on a bunch of open data sets like this one. One of my favorites was the Chicago crime data. The tool could quickly find (and fix) a bunch of errors. I tried contacting the people who published the crime data and even offered to show them how to use my free tool to clean up the data. They…

interesting! what sort of errors did you find and were there any sort of take aways your had other than lack of care by the data provider?

Re: City of Boulder Open Data

#53

Post by bigstrat2003 »

mp05 wrote: post #13
edrxty wrote: post #3

Earlier quoted context omitted.

Yes but it's Boulder, the promised land, according to here

Boulder is the worst best small town in America. You know what I mean.

It's pretty, and it would be nice to be that close to the mountains, but I can't stand the political climate in Boulder. Denver is fairly blue but still diverse enough that I feel comfortable here even though I don't personally share a lot of those views. Boulder, though... it's like people are trying to build the Berkeley of Colorado there. Which is fine, live and let live. But I would never want to live there.

Re: City of Boulder Open Data

#54

Post by toomuchtodo »

chaps wrote: post #45
NoToP wrote: post #42

Earlier quoted context omitted.

This strikes me as a Wikipedia Article type caveat. Sure the warning applies to certain categories of article/data, namely the ones where people are jockeying to have the authority of an encyclopaedia / data to reinforce their political stance, but on any sort of non political topic neutrality is a non issue. I fully anticipate that say, police and crime data by demographic shouldn't be trusted. Other data, like say…

I mean, sure? I'm very much saying this for police datasets, where for example, a "crime" database might not include homicide information. And no, I don't think it's fair to say that the amount of pounds of recycled garbage collected per census tract would be remotely accurate. Too many private companies with private incentives. Just a taste, for example: https://www.wbez.org/stories/bga-investigation-reveals-major..…

What FOIA request methodologies would you recommend to ensure you’re getting the most comprehensive data possible out of an agency or government body response?

Re: City of Boulder Open Data

#55

Post by didgetmaster »

zlurker wrote: post #52
didgetmaster wrote: post #51

Earlier quoted context omitted.

It's not just the filtering. It is also that many datasets are full of errors. I wrote a data analytics tool that I would test out on a bunch of open data sets like this one. One of my favorites was the Chicago crime data. The tool could quickly find (and fix) a bunch of errors. I tried contacting the people who published the crime data and even offered to show them how to use my free tool to clean up the data. They…

interesting! what sort of errors did you find and were there any sort of take aways your had other than lack of care by the data provider?

Each crime report would specify (among other things) what crime was reported and what city block it had occurred. Using my tool, it was easy to analyze specific crimes over time for specific areas. One of the problems was their database would have multiple spellings for the same location which made it much harder to do the analysis since you had to account for that.

Re: City of Boulder Open Data

#56

Post by lowpro »

Mistletoe wrote: post #30
Eumenes wrote: post #14

Earlier quoted context omitted.

Too commercial, too many rich brats, expensive, nice access to nature but too crowded. The western side of the Rockies are superior imo.

What cities do you suggest west of the Rockies?

They're talking about the western slope, which would mean roughly Grand Junction to Durango in this context.

Re: City of Boulder Open Data

#57

Post by iAMkenough »

chaps wrote: post #35

Just be VERY careful when using any of these open data sets. Datasets will undoubtedly be filtered by some unwritten criteria. It's best to use these datasets as a sanity check for some things, with follow-up FOIA requests for the serious work. And even that will be filled with similar filtering. Source: spent the last ten years working with open data and FOIA in Chicago. Also, here's a tweet-fitting one liner I wrot…

Just curious: for a municipality with limited resources and very minimal understanding of data governance, why should a city government follow in the footsteps of Boulder?

Do the missteps of Boulder make the case for other municipalities to not pursue open data in a similar way, given FOIA is sufficient?

Re: City of Boulder Open Data

#58

Post by chaps »

toomuchtodo wrote: post #54
chaps wrote: post #45

Earlier quoted context omitted.

I mean, sure? I'm very much saying this for police datasets, where for example, a "crime" database might not include homicide information. And no, I don't think it's fair to say that the amount of pounds of recycled garbage collected per census tract would be remotely accurate. Too many private companies with private incentives. Just a taste, for example: https://www.wbez.org/stories/bga-investigation-reveals-major..…

What FOIA request methodologies would you recommend to ensure you’re getting the most comprehensive data possible out of an agency or government body response?

One of the most productive things I've seen is to request for "data sufficient to show" rather than describing the data you're looking for. That way you have a bit more wiggle room in the space of interpretation. Another way is to just look up forms ("filetype:pdf site:yourmunicipality.gov" in google works wonders) that are online -- if filled out reports have text filled in by a computer, then that information was likely filled into a database, and the form you're looking at is a product of a multitude of SQL queries. In Illinois, Hites v. Waubonese compels an agency to run a SQL query if data is represented in a database, so being explicit about the existence of that law can be helpful. I'm sure similar can be said about other states. Also, be sure to explicitly request your data in an excel compatible format (some states legally require it).

Also, have a look at the Reporters Committee for the Freedom of Press's FOIA guides per-state. https://www.rcfp.org/open-government-guide/

Re: City of Boulder Open Data

#59

Post by CPLX »

justinator wrote: post #46
CPLX wrote: post #32

Earlier quoted context omitted.

Brooklyn and Manhattan are in different counties too. Ask yourself questions like what airport do you fly to and where would you take your kids to the children’s museum if you lived there and you’ll realize what city Boulder is a part of.

>Brooklyn and Manhattan are in different counties too. You're going to use the most densely populated city in the US for your example? >Ask yourself questions like what airport do you fly to Colorado has one international airport - are you saying ALL of Colorado is Denver? Are you saying Newark NJ is in the same city as Brooklyn? I don't know where you're heading, but it's just logical gymnastics.

Colorado has commercial service to half a dozen airports including Colorado Springs, Aspen, Vail/Eagle, and Grand Junction.

And Newark is definitely a part of greater NYC, absolutely nobody is confused by that at all.

I lived in Colorado for years. Boulder is definitely a suburb of Denver. In 20 years Fort Collins probably will be too at this rate.

Re: City of Boulder Open Data

#60

Post by toomuchtodo »

chaps wrote: post #58
toomuchtodo wrote: post #54

Earlier quoted context omitted.

What FOIA request methodologies would you recommend to ensure you’re getting the most comprehensive data possible out of an agency or government body response?

One of the most productive things I've seen is to request for "data sufficient to show" rather than describing the data you're looking for. That way you have a bit more wiggle room in the space of interpretation. Another way is to just look up forms ("filetype:pdf site:yourmunicipality.gov" in google works wonders) that are online -- if filled out reports have text filled in by a computer, then that information was l…

Incredibly helpful, thanks for the reply. These tips will make their way into my requests immediately.
Post reply on HN