Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Congress votes to make open government data the default in the United States

e-pluribusunum.org

Post reply on HN
Print view View original

61–70 of 161 posts

Re: Congress votes to make open government data the default in the United States

#61

Post by grawprog »

compute_me wrote: post #23

Making public data open by default can arguable be an imporant step towards fostering societal equity. However, it needs to be not only "open", which typically means stashed away in some corner as a spreadsheet or database file, but accessible and useful to people. The UK has been pushing open data for years now and more and more institutions are now realizing this. Shameless plug for a research project that is aimin…

>Making public data open by default can arguable be an imporant step towards fostering societal equity

I think this was one of my biggest shocks doing work for the government, collecting public data, payed by tax funded grants. Public data isn't for the public.

We went into this project with all these starry eyed dreams of making a public online database and freely posting everything we collected, with maps and interactive tools, status reports. It was part of our grant proposal.

Then reality came and we found out public data meant a government password protected database with access fees where our data would be available to people willing to pay for it or we'd lose our funding. The data were for companies or individuals willing to pay the government not for the public.

This still doesn't sit well with me nearly 6 years later. That was never what we wanted out of that project and it wasn't what was planned or accepted when we wrote our proposal.

Re: Congress votes to make open government data the default in the United States

#63

Post by Procrastes »

rjf72 wrote: post #30
astazangasta wrote: post #24

Earlier quoted context omitted.

I'm not convinced there are really any important national security secrets other than passcodes and so on. The rest seems to be excuses for hiding corruption of various kinds. Secrecy in government is incompatible with democracy by definition.

You have to think about how people respond to things. Social media in its current state has basically turned into a system that takes an event, removes all context, spins it in the most negative way imaginable, and then dogpiles on it to no end -- virtue signaling for imaginary points. There are often lots of things in government, and in life in general, where you must choose between two very negative choices. And th…

When arguing against the fundamental ability of people to rule themselves, here the ability to eventually make good decisions based on the available data, it's a good idea to consider the alternatives.

For this specific case, a loud and public debate over our country's disastrous relationship with the Saudi royal family is long overdue. Embarrassment, and unseemly noise are a pretty low price to pay to do the right thing and save more lives in, for instance, Yemen.

Our elected officials and government employees don't suddenly gain insight they did not have as common citizens. They just have access to more information.

Re: Congress votes to make open government data the default in the United States

#64

Post by paulryanrogers »

prepend wrote: post #29
wilde wrote: post #21

Earlier quoted context omitted.

I’ll defend this practice. It’s the only way of knowing for sure that you’re transmitting exactly the information you intend to send. Even copy/paste often picks up other stuff you don’t intend.

It the only, but maybe the easiest. Having a data review process with automated integrity, confidentiality, and quality checks is not terribly difficult. But having a prototocol to export the pdf to csv is also dead easy for confirming only the data relevant is included. ASCII is just as “easy” as scan, but it requires training clerks to be data-oriented rather than document.

_ugh_ if only CSVs were standardized sooner and more completely. There are many encoding, delimiter, escaping and truncation conventions to deal with in real world data.

Re: Congress votes to make open government data the default in the United States

#65

Post by TheCoelacanth »

wilde wrote: post #21
pratheekrebala wrote: post #3

FOIA officers will still find a way to send me scanned PDFs of spreadsheets.

I’ll defend this practice. It’s the only way of knowing for sure that you’re transmitting exactly the information you intend to send. Even copy/paste often picks up other stuff you don’t intend.

It's more of a way to prevent transmitting any easily accessible data at all. Using a human-auditable but still machine-readable format like CSV is what should be done.

Re: Congress votes to make open government data the default in the United States

#67

Post by bobwaycott »

compute_me wrote: post #23

Making public data open by default can arguable be an imporant step towards fostering societal equity. However, it needs to be not only "open", which typically means stashed away in some corner as a spreadsheet or database file, but accessible and useful to people. The UK has been pushing open data for years now and more and more institutions are now realizing this. Shameless plug for a research project that is aimin…

> Making public data open by default can arguable be an imporant step towards fostering societal equity.

Out of curiosity, what’s the argument here for how public data being open by default is an important step toward fostering societal equity?

Re: Congress votes to make open government data the default in the United States

#68

Post by indigochill »

jcriddle4 wrote: post #8

Interesting that the Federal Reserve was exempted from this legislation.

Is it? My understanding is that although the board is appointed by the President and Congress, once appointed they don't answer to the government, so their data isn't really "government data".

But I'll be the first to admit I know very little about how the Federal Reserve works.

Re: Congress votes to make open government data the default in the United States

#69

Post by ams6110 »

_bohm wrote: post #49
kerkeslager wrote: post #33

Earlier quoted context omitted.

As long as those spreadsheets/database files are accessible to someone with technical skill, people can pull in the data and use tools to make it more accessible and useful. Ideally, yes, the data is useful to begin with, but as long as it's available, there's nothing stopping individuals with the skills from making it useful. Of course, there are exceptions: the PDFs that are often provided by the prosecution as par…

I've spent a great deal of time parsing data out of government PDFs that isn't attainable by any other means as a part of my job. In the process I've learned how difficult this information can be to access even for people who don't require it to be in a machine readable format. It certainly has been an interesting exercise in how far simple web scraping tools can be pushed, though.

Amazon Textract was recently announced, sounds like it might be good for that. Haven't tried it myself.

https://aws.amazon.com/about-aws/whats-new/2018/11/introduci...

Re: Congress votes to make open government data the default in the United States

#70

Post by hartator »

compute_me wrote: post #23

Making public data open by default can arguable be an imporant step towards fostering societal equity. However, it needs to be not only "open", which typically means stashed away in some corner as a spreadsheet or database file, but accessible and useful to people. The UK has been pushing open data for years now and more and more institutions are now realizing this. Shameless plug for a research project that is aimin…

I actually rather have the raw data accessible instead of doctored version.
Post reply on HN