Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Congress votes to make open government data the default in the United States

e-pluribusunum.org

Post reply on HN
Print view View original

111–120 of 161 posts

Re: Congress votes to make open government data the default in the United States

#111

Post by akudha »

yxhuvud wrote: post #59
tehlike wrote: post #42

With all the problems its government has, united states is still at it fostering innovation. Open data will bring innovation and accountability. Great move

Innovation? Other nations (like Sweden) have had its data public for many decades now.

UK is another great example.

There is no "innovation" going on here. It is the data owned by the citizens, doesn't take that much effort to release it - they are collecting the data anyway, all they have to do is make it available to the general public. Still, a step in the right direction

Re: Congress votes to make open government data the default in the United States

#112

Post by gizmodo59 »

org3432 wrote: post #75
newleaf wrote: post #37

Earlier quoted context omitted.

In case you haven't seen it, I think https://www.data.gov/ is an attempt to answer your point about making it "accessible and useful to people." There's room for improvement, but it's a start.

I thought there would be a flood of projects analysing the data when it came out, but it seems like the idea everyone applauded but not much came out of it. Steve Ballmer's http://usafacts.org seems like the first real attempt though.

+1 for this. This site is beautifully designed and loads very fast in mobile.

Re: Congress votes to make open government data the default in the United States

#113

Post by lingrino »

newleaf wrote: post #37
compute_me wrote: post #23

Making public data open by default can arguable be an imporant step towards fostering societal equity. However, it needs to be not only "open", which typically means stashed away in some corner as a spreadsheet or database file, but accessible and useful to people. The UK has been pushing open data for years now and more and more institutions are now realizing this. Shameless plug for a research project that is aimin…

In case you haven't seen it, I think https://www.data.gov/ is an attempt to answer your point about making it "accessible and useful to people." There's room for improvement, but it's a start.

https://public.enigma.com is another good way to browse public government data.

Re: Congress votes to make open government data the default in the United States

#114

Post by _bohm »

ams6110 wrote: post #69
_bohm wrote: post #49

Earlier quoted context omitted.

I've spent a great deal of time parsing data out of government PDFs that isn't attainable by any other means as a part of my job. In the process I've learned how difficult this information can be to access even for people who don't require it to be in a machine readable format. It certainly has been an interesting exercise in how far simple web scraping tools can be pushed, though.

Amazon Textract was recently announced, sounds like it might be good for that. Haven't tried it myself. https://aws.amazon.com/about-aws/whats-new/2018/11/introduci...

I applied to the beta, but they never got back to me :\

Re: Congress votes to make open government data the default in the United States

#115

Post by philipashlock »

rocketflumes wrote: post #84
newleaf wrote: post #37

Earlier quoted context omitted.

In case you haven't seen it, I think https://www.data.gov/ is an attempt to answer your point about making it "accessible and useful to people." There's room for improvement, but it's a start.

data.gov is great but it seems like the most recent activities/updates were prior to 2016...

Current activity on GitHub, eg:

https://github.com/gsa/datagov-deploy

https://github.com/ViderumGlobal/datagov-ckan-cloud-provisio...

Re: Congress votes to make open government data the default in the United States

#116

Post by amyboyd »

erentz wrote: post #110

"The Open, Public, Electronic, and Necessary Government Data Act (AKA the OPEN Government Data Act)" Please, could we end this obsession with backronyms in congress. Perhaps some congressperson could create a suitably titled backronym act to rid us of backronym acts. This should just be the one singular "Government Data Act" and it should be amended as needed to cover any law changes around "Government Data".

BACKRONYM Act: Banning Acronym Cleverness Keeping Representatives Occupied Nutting Your Mom Act

(struggled with the NYM so gave up and went with something childish...)

Re: Congress votes to make open government data the default in the United States

#117

Post by _bohm »

TheAceOfHearts wrote: post #93
_bohm wrote: post #49

Earlier quoted context omitted.

I've spent a great deal of time parsing data out of government PDFs that isn't attainable by any other means as a part of my job. In the process I've learned how difficult this information can be to access even for people who don't require it to be in a machine readable format. It certainly has been an interesting exercise in how far simple web scraping tools can be pushed, though.

Do you have any tool suggestions or general advice for someone trying to do this? A while back I was trying to extract text from some government PDFs in order to make the information more accessible for others, but I became a bit overwhelmed when I started reading up on PDFs.

Sure! In terms of raw text extraction (for documents that don't require OCR), the most useful tools I've worked with have been pdftotext [0] and PyMuPDF [1]. For extracting useful details, really, my best advice is to make sure that your regex skills are sharp. I've been meaning to explore the possibility of using NLP tools for named entity recognition, but unfortunately I don't have much of a background there.

The rest kind of it kind of just comes down to using good software engineering practices to help keep yourself sane. Find useful abstractions for common tasks you need to perform and build a library around them, make sure that your data processing pipeline is designed with enough flexibility to handle inputs in different formats so that adding or modifying parsing logic becomes trivial, etc.

[0] https://www.xpdfreader.com/pdftotext-man.html [1] https://pymupdf.readthedocs.io/en/latest/

Re: Congress votes to make open government data the default in the United States

#118

Post by bpchaps »

_bohm wrote: post #114
ams6110 wrote: post #69

Earlier quoted context omitted.

Amazon Textract was recently announced, sounds like it might be good for that. Haven't tried it myself. https://aws.amazon.com/about-aws/whats-new/2018/11/introduci...

I applied to the beta, but they never got back to me :\

Have you tried Apache's tika? It's pretty decent.

Re: Congress votes to make open government data the default in the United States

#119

Post by _bohm »

bpchaps wrote: post #118
_bohm wrote: post #114

Earlier quoted context omitted.

I applied to the beta, but they never got back to me :\

Have you tried Apache's tika? It's pretty decent.

Nope, I'll have to give it a spin. Thanks for the recommendation!

Re: Congress votes to make open government data the default in the United States

#120

Post by danso »

mancerayder wrote: post #108
compute_me wrote: post #23

Making public data open by default can arguable be an imporant step towards fostering societal equity. However, it needs to be not only "open", which typically means stashed away in some corner as a spreadsheet or database file, but accessible and useful to people. The UK has been pushing open data for years now and more and more institutions are now realizing this. Shameless plug for a research project that is aimin…

Making public data open by default can arguable be an imporant step towards fostering societal equity. Define equity here?

Not the parent commenter, but open data policies could contribute to levelingn the playing field for society. The vast majority of records requests are currently made by corporations and other business interests.
Post reply on HN