Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Congress votes to make open government data the default in the United States

e-pluribusunum.org

Post reply on HN
Print view View original

91–100 of 161 posts

Re: Congress votes to make open government data the default in the United States

#91

Post by jtloong »

scarejunba wrote: post #90
compute_me wrote: post #23

Making public data open by default can arguable be an imporant step towards fostering societal equity. However, it needs to be not only "open", which typically means stashed away in some corner as a spreadsheet or database file, but accessible and useful to people. The UK has been pushing open data for years now and more and more institutions are now realizing this. Shameless plug for a research project that is aimin…

I prefer when it is available as a spreadsheet or database file. The crucial element is my having access to the data. After that, any differential amount of information I have over everyone else is an improvement. For instance, consider the statistics on accidents between cars and bikes in California. You can get the numbers yourself from the government. When someone says that more accidents are adjudged to be becaus…

I think it needs to be both. Accessible in it's raw format as well as approachable for non-technical users to interact with public information.

Re: Congress votes to make open government data the default in the United States

#92

Post by cantthinkofone »

compute_me wrote: post #23

Making public data open by default can arguable be an imporant step towards fostering societal equity. However, it needs to be not only "open", which typically means stashed away in some corner as a spreadsheet or database file, but accessible and useful to people. The UK has been pushing open data for years now and more and more institutions are now realizing this. Shameless plug for a research project that is aimin…

I agree that the government will need to work on a transparent front-end to make this data universally accessible. Nonprofits without the budgets for advanced tech workers and without volunteers will need clearly organized links to download. They may not know how to do shell scripting.

The more data everyone can use, rather than data that can be owned and commoditized or utilized only by specialists is a good thing.

Re: Congress votes to make open government data the default in the United States

#93

Post by TheAceOfHearts »

_bohm wrote: post #49
kerkeslager wrote: post #33

Earlier quoted context omitted.

As long as those spreadsheets/database files are accessible to someone with technical skill, people can pull in the data and use tools to make it more accessible and useful. Ideally, yes, the data is useful to begin with, but as long as it's available, there's nothing stopping individuals with the skills from making it useful. Of course, there are exceptions: the PDFs that are often provided by the prosecution as par…

I've spent a great deal of time parsing data out of government PDFs that isn't attainable by any other means as a part of my job. In the process I've learned how difficult this information can be to access even for people who don't require it to be in a machine readable format. It certainly has been an interesting exercise in how far simple web scraping tools can be pushed, though.

Do you have any tool suggestions or general advice for someone trying to do this? A while back I was trying to extract text from some government PDFs in order to make the information more accessible for others, but I became a bit overwhelmed when I started reading up on PDFs.

Re: Congress votes to make open government data the default in the United States

#94

Post by vertexFarm »

compute_me wrote: post #23

Making public data open by default can arguable be an imporant step towards fostering societal equity. However, it needs to be not only "open", which typically means stashed away in some corner as a spreadsheet or database file, but accessible and useful to people. The UK has been pushing open data for years now and more and more institutions are now realizing this. Shameless plug for a research project that is aimin…

I was kind of concerned right off the bat with that numbered list:

1. public information should be open by default to the public in a machine-readable format, where such publication doesn’t harm privacy or security

I'm sure literally everything that they wish to keep opaque will declared to be covered under one or both of these incredibly vague categories and nothing will significantly change. Is there any elaboration in the bill that defines what they can call a matter of privacy or security? Even if there is, it wouldn't matter much because how are people going to tell if they keep it locked down in the first place? And they would not risk any sort of real blowback for abusing this and getting caught. Tell me there have not been far, far worse scandals that resulted in no consequences for the perps and cowed silence from the public. I don't think they're hiding the X-Files in there or anything, but this won't magically cause a more transparent, just, or equitable government unless it has serious teeth and tight language.

And 2. federal agencies should use evidence when they make public policy

Somehow I wonder if the data from the Kansas experiment will be taken into consideration and turned into public policy by this current administration, or if they will cherry-pick evidence selectively to justify only wildly unpopular legislation because someone (possibly an industry with a conflict of interest?) contrives some p-hacked research to back it up. Just because something is scientific doesn't necessarily mean it's good government. It is often so, but I'm always very wary when they trot out a bill with lots of bold language touting justice and democracy, truth, stuff like that. If the US legislature passes a bill called "protect innocent puppies from being kicked in the name of god and freedom" you can be 95% sure that this bill will enable a great wave of puppy-kicking despite its holy name.

Re: Congress votes to make open government data the default in the United States

#95

Post by dandare »

All I want for this Christmass is an authoritative list of all US federal agencies. I kid you not, there is no such list and the number of federal agencies is uncertain. From Wikipedia:

>Legislative definitions of a federal agency are varied, and even contradictory, and the official United States Government Manual offers no definition. While the Administrative Procedure Act definition of "agency" applies to most executive branch agencies, Congress may define an agency however it chooses in enabling legislation, and subsequent litigation, often involving the Freedom of Information Act and the Government in the Sunshine Act. These further cloud attempts to enumerate a list of agencies.

Re: Congress votes to make open government data the default in the United States

#96

Post by mehrdadn »

dandare wrote: post #95

All I want for this Christmass is an authoritative list of all US federal agencies. I kid you not, there is no such list and the number of federal agencies is uncertain. From Wikipedia: >Legislative definitions of a federal agency are varied, and even contradictory, and the official United States Government Manual offers no definition. While the Administrative Procedure Act definition of "agency" applies to most exec…

What do you mean by federal agency (given that the definition itself is unclear)?

Re: Congress votes to make open government data the default in the United States

#97

Post by acjohnson55 »

dandare wrote: post #95

All I want for this Christmass is an authoritative list of all US federal agencies. I kid you not, there is no such list and the number of federal agencies is uncertain. From Wikipedia: >Legislative definitions of a federal agency are varied, and even contradictory, and the official United States Government Manual offers no definition. While the Administrative Procedure Act definition of "agency" applies to most exec…

That seems like the issue is that the answer varies with chosen semantics, not unavailability of information.

Re: Congress votes to make open government data the default in the United States

#98

Post by killjoywashere »

dandare wrote: post #95

All I want for this Christmass is an authoritative list of all US federal agencies. I kid you not, there is no such list and the number of federal agencies is uncertain. From Wikipedia: >Legislative definitions of a federal agency are varied, and even contradictory, and the official United States Government Manual offers no definition. While the Administrative Procedure Act definition of "agency" applies to most exec…

Check the budget. Bills.

Re: Congress votes to make open government data the default in the United States

#99

Post by koboll »

newleaf wrote: post #37
compute_me wrote: post #23

Making public data open by default can arguable be an imporant step towards fostering societal equity. However, it needs to be not only "open", which typically means stashed away in some corner as a spreadsheet or database file, but accessible and useful to people. The UK has been pushing open data for years now and more and more institutions are now realizing this. Shameless plug for a research project that is aimin…

In case you haven't seen it, I think https://www.data.gov/ is an attempt to answer your point about making it "accessible and useful to people." There's room for improvement, but it's a start.

USDS/18F have done such incredible work and I'm glad the Trump admin hasn't killed them off completely. Given Trump's total-war attitude toward all Obama initiatives I'm somewhat surprised they still exist and haven't been gutted like CFPB.

Re: Congress votes to make open government data the default in the United States

#100

Post by prepend »

paulryanrogers wrote: post #64
prepend wrote: post #29

Earlier quoted context omitted.

It the only, but maybe the easiest. Having a data review process with automated integrity, confidentiality, and quality checks is not terribly difficult. But having a prototocol to export the pdf to csv is also dead easy for confirming only the data relevant is included. ASCII is just as “easy” as scan, but it requires training clerks to be data-oriented rather than document.

_ugh_ if only CSVs were standardized sooner and more completely. There are many encoding, delimiter, escaping and truncation conventions to deal with in real world data.

Definitely. They are better than PDFs, but still have lots of room for improvement.
Post reply on HN